Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [........................XX....X..................X..........] duration: 6.013499185s trace: https://jaeger.dev.storj.io/trace/1a181df13358808f FAIL (2025-10-31 21:09:59 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Oct 31 21:03:40.187891701 (33h23m11.883740444s) [upload=29297]: Starting upload Oct 31 21:03:40.187972845 (33h23m11.883821598s) [upload=29297]: Done starting upload Oct 31 21:03:40.188163167 (33h23m11.88401191s) [upload=29297]: Uploading segments... Oct 31 21:03:40.188240786 (33h23m11.884089559s) [upload=29297]: Got next segment. Inline: false Oct 31 21:03:40.188281714 (33h23m11.884130467s) [upload=29297]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 31 21:03:40.188288065 (33h23m11.884136808s) [upload=29297]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 31 21:03:40.439495977 (33h23m12.13534474s) [seg_pos={0 0}, upload=29297]: Begin upload segment... Oct 31 21:03:40.439552094 (33h23m12.135400847s) [seg_pos={0 0}, upload=29297]: Done begin upload segment. Oct 31 21:03:40.439606758 (33h23m12.135455511s) [noise=true , address=145.79.227.21:4473 , piece_id=XBSRRHW5MAWDHZV3YFV7BWHL7VWX7C2YWB2DTNCFQJTXE43LAAYQ, seg_pos={0 0}, upload=29297]: Uploading piece... Oct 31 21:03:40.439643338 (33h23m12.135492081s) [noise=true , address=77.90.31.15:4698 , piece_id=2SJEFFLOU7LPMQZV5DLW5Z4BUQNJFZ347K656TFXV47LTCODGA3A, seg_pos={0 0}, upload=29297]: Uploading piece... Oct 31 21:03:40.440209786 (33h23m12.136058539s) [noise=true , address=145.79.227.14:4837 , piece_id=E3NUXQAOQNOCNTMUGO2NX2ELYKMII4J37DQPUBSQ6ABGBHIC3WSA, seg_pos={0 0}, upload=29297]: Uploading piece... Oct 31 21:03:40.440607473 (33h23m12.136456216s) [noise=true , address=145.79.227.13:4802 , piece_id=GZK6DPTPN4X5D4NRJABMLW6HNPXMH7IR25UE347P2BQZYKCJY3IQ, seg_pos={0 0}, upload=29297]: Uploading piece... Oct 31 21:03:40.441017394 (33h23m12.136866147s) [noise=true , address=77.90.31.15:4699 , piece_id=DLH6733XEDD5Z7AIO7NT4UC4TMTM57RUDVUHYOYONTXTGUJWJSYA, seg_pos={0 0}, upload=29297]: Uploading piece... Oct 31 21:03:40.441619029 (33h23m12.137467772s) [noise=true , address=77.90.31.17:4772 , piece_id=BFR46U4SPW6VNCIDV3XNIEOVYWGCXMQPDZ4JCJO5EAY5VSY4X56A, seg_pos={0 0}, upload=29297]: Uploading piece... Oct 31 21:03:40.442305126 (33h23m12.138153879s) [noise=true , address=145.79.227.17:4970 , piece_id=6LJTN7QUSUSHS6DSKRHUUA3XB5ICEZ2ACFOSU4SK5HMUDRSKMEYA, seg_pos={0 0}, upload=29297]: Uploading piece... Oct 31 21:03:40.443621452 (33h23m12.139470205s) [noise=true , address=145.79.227.15:4874 , piece_id=VK3CG2KERHGJQSKLJKOT7V46GRU6XEJN7T5S4HMTW2M76E5HOUHQ, seg_pos={0 0}, upload=29297]: Uploading piece... Oct 31 21:03:40.443864906 (33h23m12.139713659s) [noise=true , address=145.79.227.11:4732 , piece_id=E4WNX7JXJZHMPKBPEJPSXJAYL4BP47HLXLO6DBHPICX6ESIK767Q, seg_pos={0 0}, upload=29297]: Uploading piece... Oct 31 21:03:40.444991701 (33h23m12.140840444s) [noise=true , address=145.79.227.24:4583 , piece_id=KSDYUXPSUXPNBT7J7Q6RW7HNPILNVUWSX7STJXDBI7YKR7MA7XMA, seg_pos={0 0}, upload=29297]: Uploading piece... Oct 31 21:03:40.445358190 (33h23m12.141206933s) [noise=true , address=145.79.227.5:4223 , piece_id=CAMTFWQE5GQIDZMJDJXSO6SIHKQXKI6HRNQAFI46JICNCIMEGACA, seg_pos={0 0}, upload=29297]: Uploading piece... Oct 31 21:03:40.446480987 (33h23m12.14232974s) [noise=true , address=145.79.227.30:4762 , piece_id=HDRX3AY2PGE2UDY4SLN4A7T27KGLZQRTOYGKHCDC4S7EDVJOIWXA, seg_pos={0 0}, upload=29297]: Uploading piece... Oct 31 21:03:40.446948418 (33h23m12.142797171s) [noise=true , address=77.90.31.10:4517 , piece_id=BDAUXG53XX42I72C6BIIC3VCLA6GZEFD3IDVSLYCC73AS2TG626Q, seg_pos={0 0}, upload=29297]: Uploading piece... Oct 31 21:03:40.448429257 (33h23m12.14427801s) [noise=true , address=145.79.227.17:4971 , piece_id=3P7N3ERAPOY62SAKEBV7CHWOWWNUPWLSLL3RZ4RWCPF3VHI3C4FQ, seg_pos={0 0}, upload=29297]: Uploading piece... Oct 31 21:03:40.449960433 (33h23m12.145809176s) [noise=true , address=77.90.31.11:4555 , piece_id=BMTMJ4QDYLWDSMNIP64YUDUBZA2DABKYYC5JK72VJSUBTBZ67YTA, seg_pos={0 0}, upload=29297]: Uploading piece... Oct 31 21:03:40.451429571 (33h23m12.147278324s) [noise=true , address=145.79.227.4:4185 , piece_id=7C536MM2MA3EC44OTMXDUGNW5YDD3OOK5ODAU2FOFM3YTQ7VU6PQ, seg_pos={0 0}, upload=29297]: Uploading piece... Oct 31 21:03:42.182232662 (33h23m13.878081416s) [upload=29297]: Next returned nil segment Oct 31 21:03:42.182251107 (33h23m13.87809985s) [upload=29297]: Waiting for error group managing segments... Oct 31 21:03:42.460580042 (33h23m14.156428785s) [noise=true , address=145.79.227.24:4583 , piece_id=KSDYUXPSUXPNBT7J7Q6RW7HNPILNVUWSX7STJXDBI7YKR7MA7XMA, seg_pos={0 0}, upload=29297]: Done uploading piece. err: <nil> Oct 31 21:03:42.470437052 (33h23m14.166285805s) [noise=true , address=145.79.227.5:4223 , piece_id=CAMTFWQE5GQIDZMJDJXSO6SIHKQXKI6HRNQAFI46JICNCIMEGACA, seg_pos={0 0}, upload=29297]: Done uploading piece. err: <nil> Oct 31 21:03:42.478522949 (33h23m14.174371692s) [noise=true , address=145.79.227.4:4185 , piece_id=7C536MM2MA3EC44OTMXDUGNW5YDD3OOK5ODAU2FOFM3YTQ7VU6PQ, seg_pos={0 0}, upload=29297]: Done uploading piece. err: <nil> Oct 31 21:03:42.479322582 (33h23m14.175171325s) [noise=true , address=145.79.227.15:4874 , piece_id=VK3CG2KERHGJQSKLJKOT7V46GRU6XEJN7T5S4HMTW2M76E5HOUHQ, seg_pos={0 0}, upload=29297]: Done uploading piece. err: <nil> Oct 31 21:03:42.479617323 (33h23m14.175466076s) [noise=true , address=145.79.227.21:4473 , piece_id=XBSRRHW5MAWDHZV3YFV7BWHL7VWX7C2YWB2DTNCFQJTXE43LAAYQ, seg_pos={0 0}, upload=29297]: Done uploading piece. err: <nil> Oct 31 21:03:42.481690871 (33h23m14.177539614s) [noise=true , address=145.79.227.30:4762 , piece_id=HDRX3AY2PGE2UDY4SLN4A7T27KGLZQRTOYGKHCDC4S7EDVJOIWXA, seg_pos={0 0}, upload=29297]: Done uploading piece. err: <nil> Oct 31 21:03:42.513679364 (33h23m14.209528107s) [noise=true , address=145.79.227.13:4802 , piece_id=GZK6DPTPN4X5D4NRJABMLW6HNPXMH7IR25UE347P2BQZYKCJY3IQ, seg_pos={0 0}, upload=29297]: Done uploading piece. err: <nil> Oct 31 21:03:42.569840746 (33h23m14.265689489s) [noise=true , address=145.79.227.14:4837 , piece_id=E3NUXQAOQNOCNTMUGO2NX2ELYKMII4J37DQPUBSQ6ABGBHIC3WSA, seg_pos={0 0}, upload=29297]: Done uploading piece. err: <nil> Oct 31 21:03:42.571444861 (33h23m14.267293614s) [noise=true , address=145.79.227.17:4970 , piece_id=6LJTN7QUSUSHS6DSKRHUUA3XB5ICEZ2ACFOSU4SK5HMUDRSKMEYA, seg_pos={0 0}, upload=29297]: Done uploading piece. err: <nil> Oct 31 21:03:42.707369620 (33h23m14.403218373s) [noise=true , address=145.79.227.11:4732 , piece_id=E4WNX7JXJZHMPKBPEJPSXJAYL4BP47HLXLO6DBHPICX6ESIK767Q, seg_pos={0 0}, upload=29297]: Done uploading piece. err: <nil> Oct 31 21:03:42.745077775 (33h23m14.440926518s) [noise=true , address=145.79.227.17:4971 , piece_id=3P7N3ERAPOY62SAKEBV7CHWOWWNUPWLSLL3RZ4RWCPF3VHI3C4FQ, seg_pos={0 0}, upload=29297]: Done uploading piece. err: <nil> Oct 31 21:04:00.023801048 (33h23m31.719649801s) [noise=true , address=77.90.31.17:4772 , piece_id=BFR46U4SPW6VNCIDV3XNIEOVYWGCXMQPDZ4JCJO5EAY5VSY4X56A, seg_pos={0 0}, upload=29297]: Done uploading piece. err: <nil> Oct 31 21:04:09.488974106 (33h23m41.184822859s) [noise=true , address=77.90.31.15:4698 , piece_id=2SJEFFLOU7LPMQZV5DLW5Z4BUQNJFZ347K656TFXV47LTCODGA3A, seg_pos={0 0}, upload=29297]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: write tcp 188.166.225.170:35474->77.90.31.15:4698: use of closed network connection; write tcp 188.166.225.170:35474->77.90.31.15:4698: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35474->77.90.31.15:4698: use of closed network connection Oct 31 21:04:09.489128110 (33h23m41.184976863s) [noise=true , address=77.90.31.10:4517 , piece_id=BDAUXG53XX42I72C6BIIC3VCLA6GZEFD3IDVSLYCC73AS2TG626Q, seg_pos={0 0}, upload=29297]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 188.166.225.170:58092->77.90.31.10:4517: use of closed network connection; write tcp 188.166.225.170:58092->77.90.31.10:4517: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58092->77.90.31.10:4517: use of closed network connection Oct 31 21:04:09.489250382 (33h23m41.185099135s) [noise=true , address=77.90.31.11:4555 , piece_id=BMTMJ4QDYLWDSMNIP64YUDUBZA2DABKYYC5JK72VJSUBTBZ67YTA, seg_pos={0 0}, upload=29297]: Done uploading piece. err: ecclient: upload failed (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, address:77.90.31.11:4555): protocol: write tcp 188.166.225.170:40898->77.90.31.11:4555: use of closed network connection; write tcp 188.166.225.170:40898->77.90.31.11:4555: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40898->77.90.31.11:4555: use of closed network connection Oct 31 21:04:09.489384528 (33h23m41.185233281s) [noise=true , address=77.90.31.15:4699 , piece_id=DLH6733XEDD5Z7AIO7NT4UC4TMTM57RUDVUHYOYONTXTGUJWJSYA, seg_pos={0 0}, upload=29297]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: write tcp 188.166.225.170:38634->77.90.31.15:4699: use of closed network connection; write tcp 188.166.225.170:38634->77.90.31.15:4699: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38634->77.90.31.15:4699: use of closed network connection Oct 31 21:04:09.489415327 (33h23m41.18526408s) [seg_pos={0 0}, upload=29297]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Oct 31 21:04:09.733073582 (33h23m41.428922335s) [upload=29297]: Done uploading segments... Oct 31 21:04:09.733121814 (33h23m41.428970557s) [upload=29297]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.013548257s core uplink us-select-1
checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect) antiblip: 1 out of 15 is failed (threshold=6) [....X..........] duration: 310.473195ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/dcs-satellites trace: https://jaeger.dev.storj.io/trace/571fef006088dd70 FAIL (2025-10-31 21:09:59 UTC) orbiter warning: Get "https://www.storj.io/dcs-satellites": read tcp [2400:6180:0:d0::f70:8001]:42806->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 310.499416ms core website
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [............................X....X..........................] duration: 6.028708327s trace: https://jaeger.dev.storj.io/trace/35569b98f076d00a FAIL (2025-10-31 21:09:43 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.028737161s edge gateway us-select-1
checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect) antiblip: 1 out of 15 is failed (threshold=6) [.....X.........] duration: 308.634394ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/dcs-satellites trace: https://jaeger.dev.storj.io/trace/571fef006088dd70 FAIL (2025-10-31 21:09:19 UTC) orbiter warning: Get "https://www.storj.io/dcs-satellites": read tcp [2400:6180:0:d0::f70:8001]:42806->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 308.665383ms core website
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.........................XX....X..................X.........] duration: 6.023485292s trace: https://jaeger.dev.storj.io/trace/1a181df13358808f FAIL (2025-10-31 21:09:14 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Oct 31 21:03:40.187891701 (33h23m11.883740444s) [upload=29297]: Starting upload Oct 31 21:03:40.187972845 (33h23m11.883821598s) [upload=29297]: Done starting upload Oct 31 21:03:40.188163167 (33h23m11.88401191s) [upload=29297]: Uploading segments... Oct 31 21:03:40.188240786 (33h23m11.884089559s) [upload=29297]: Got next segment. Inline: false Oct 31 21:03:40.188281714 (33h23m11.884130467s) [upload=29297]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 31 21:03:40.188288065 (33h23m11.884136808s) [upload=29297]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 31 21:03:40.439495977 (33h23m12.13534474s) [seg_pos={0 0}, upload=29297]: Begin upload segment... Oct 31 21:03:40.439552094 (33h23m12.135400847s) [seg_pos={0 0}, upload=29297]: Done begin upload segment. Oct 31 21:03:40.439606758 (33h23m12.135455511s) [noise=true , address=145.79.227.21:4473 , piece_id=XBSRRHW5MAWDHZV3YFV7BWHL7VWX7C2YWB2DTNCFQJTXE43LAAYQ, seg_pos={0 0}, upload=29297]: Uploading piece... Oct 31 21:03:40.439643338 (33h23m12.135492081s) [noise=true , address=77.90.31.15:4698 , piece_id=2SJEFFLOU7LPMQZV5DLW5Z4BUQNJFZ347K656TFXV47LTCODGA3A, seg_pos={0 0}, upload=29297]: Uploading piece... Oct 31 21:03:40.440209786 (33h23m12.136058539s) [noise=true , address=145.79.227.14:4837 , piece_id=E3NUXQAOQNOCNTMUGO2NX2ELYKMII4J37DQPUBSQ6ABGBHIC3WSA, seg_pos={0 0}, upload=29297]: Uploading piece... Oct 31 21:03:40.440607473 (33h23m12.136456216s) [noise=true , address=145.79.227.13:4802 , piece_id=GZK6DPTPN4X5D4NRJABMLW6HNPXMH7IR25UE347P2BQZYKCJY3IQ, seg_pos={0 0}, upload=29297]: Uploading piece... Oct 31 21:03:40.441017394 (33h23m12.136866147s) [noise=true , address=77.90.31.15:4699 , piece_id=DLH6733XEDD5Z7AIO7NT4UC4TMTM57RUDVUHYOYONTXTGUJWJSYA, seg_pos={0 0}, upload=29297]: Uploading piece... Oct 31 21:03:40.441619029 (33h23m12.137467772s) [noise=true , address=77.90.31.17:4772 , piece_id=BFR46U4SPW6VNCIDV3XNIEOVYWGCXMQPDZ4JCJO5EAY5VSY4X56A, seg_pos={0 0}, upload=29297]: Uploading piece... Oct 31 21:03:40.442305126 (33h23m12.138153879s) [noise=true , address=145.79.227.17:4970 , piece_id=6LJTN7QUSUSHS6DSKRHUUA3XB5ICEZ2ACFOSU4SK5HMUDRSKMEYA, seg_pos={0 0}, upload=29297]: Uploading piece... Oct 31 21:03:40.443621452 (33h23m12.139470205s) [noise=true , address=145.79.227.15:4874 , piece_id=VK3CG2KERHGJQSKLJKOT7V46GRU6XEJN7T5S4HMTW2M76E5HOUHQ, seg_pos={0 0}, upload=29297]: Uploading piece... Oct 31 21:03:40.443864906 (33h23m12.139713659s) [noise=true , address=145.79.227.11:4732 , piece_id=E4WNX7JXJZHMPKBPEJPSXJAYL4BP47HLXLO6DBHPICX6ESIK767Q, seg_pos={0 0}, upload=29297]: Uploading piece... Oct 31 21:03:40.444991701 (33h23m12.140840444s) [noise=true , address=145.79.227.24:4583 , piece_id=KSDYUXPSUXPNBT7J7Q6RW7HNPILNVUWSX7STJXDBI7YKR7MA7XMA, seg_pos={0 0}, upload=29297]: Uploading piece... Oct 31 21:03:40.445358190 (33h23m12.141206933s) [noise=true , address=145.79.227.5:4223 , piece_id=CAMTFWQE5GQIDZMJDJXSO6SIHKQXKI6HRNQAFI46JICNCIMEGACA, seg_pos={0 0}, upload=29297]: Uploading piece... Oct 31 21:03:40.446480987 (33h23m12.14232974s) [noise=true , address=145.79.227.30:4762 , piece_id=HDRX3AY2PGE2UDY4SLN4A7T27KGLZQRTOYGKHCDC4S7EDVJOIWXA, seg_pos={0 0}, upload=29297]: Uploading piece... Oct 31 21:03:40.446948418 (33h23m12.142797171s) [noise=true , address=77.90.31.10:4517 , piece_id=BDAUXG53XX42I72C6BIIC3VCLA6GZEFD3IDVSLYCC73AS2TG626Q, seg_pos={0 0}, upload=29297]: Uploading piece... Oct 31 21:03:40.448429257 (33h23m12.14427801s) [noise=true , address=145.79.227.17:4971 , piece_id=3P7N3ERAPOY62SAKEBV7CHWOWWNUPWLSLL3RZ4RWCPF3VHI3C4FQ, seg_pos={0 0}, upload=29297]: Uploading piece... Oct 31 21:03:40.449960433 (33h23m12.145809176s) [noise=true , address=77.90.31.11:4555 , piece_id=BMTMJ4QDYLWDSMNIP64YUDUBZA2DABKYYC5JK72VJSUBTBZ67YTA, seg_pos={0 0}, upload=29297]: Uploading piece... Oct 31 21:03:40.451429571 (33h23m12.147278324s) [noise=true , address=145.79.227.4:4185 , piece_id=7C536MM2MA3EC44OTMXDUGNW5YDD3OOK5ODAU2FOFM3YTQ7VU6PQ, seg_pos={0 0}, upload=29297]: Uploading piece... Oct 31 21:03:42.182232662 (33h23m13.878081416s) [upload=29297]: Next returned nil segment Oct 31 21:03:42.182251107 (33h23m13.87809985s) [upload=29297]: Waiting for error group managing segments... Oct 31 21:03:42.460580042 (33h23m14.156428785s) [noise=true , address=145.79.227.24:4583 , piece_id=KSDYUXPSUXPNBT7J7Q6RW7HNPILNVUWSX7STJXDBI7YKR7MA7XMA, seg_pos={0 0}, upload=29297]: Done uploading piece. err: <nil> Oct 31 21:03:42.470437052 (33h23m14.166285805s) [noise=true , address=145.79.227.5:4223 , piece_id=CAMTFWQE5GQIDZMJDJXSO6SIHKQXKI6HRNQAFI46JICNCIMEGACA, seg_pos={0 0}, upload=29297]: Done uploading piece. err: <nil> Oct 31 21:03:42.478522949 (33h23m14.174371692s) [noise=true , address=145.79.227.4:4185 , piece_id=7C536MM2MA3EC44OTMXDUGNW5YDD3OOK5ODAU2FOFM3YTQ7VU6PQ, seg_pos={0 0}, upload=29297]: Done uploading piece. err: <nil> Oct 31 21:03:42.479322582 (33h23m14.175171325s) [noise=true , address=145.79.227.15:4874 , piece_id=VK3CG2KERHGJQSKLJKOT7V46GRU6XEJN7T5S4HMTW2M76E5HOUHQ, seg_pos={0 0}, upload=29297]: Done uploading piece. err: <nil> Oct 31 21:03:42.479617323 (33h23m14.175466076s) [noise=true , address=145.79.227.21:4473 , piece_id=XBSRRHW5MAWDHZV3YFV7BWHL7VWX7C2YWB2DTNCFQJTXE43LAAYQ, seg_pos={0 0}, upload=29297]: Done uploading piece. err: <nil> Oct 31 21:03:42.481690871 (33h23m14.177539614s) [noise=true , address=145.79.227.30:4762 , piece_id=HDRX3AY2PGE2UDY4SLN4A7T27KGLZQRTOYGKHCDC4S7EDVJOIWXA, seg_pos={0 0}, upload=29297]: Done uploading piece. err: <nil> Oct 31 21:03:42.513679364 (33h23m14.209528107s) [noise=true , address=145.79.227.13:4802 , piece_id=GZK6DPTPN4X5D4NRJABMLW6HNPXMH7IR25UE347P2BQZYKCJY3IQ, seg_pos={0 0}, upload=29297]: Done uploading piece. err: <nil> Oct 31 21:03:42.569840746 (33h23m14.265689489s) [noise=true , address=145.79.227.14:4837 , piece_id=E3NUXQAOQNOCNTMUGO2NX2ELYKMII4J37DQPUBSQ6ABGBHIC3WSA, seg_pos={0 0}, upload=29297]: Done uploading piece. err: <nil> Oct 31 21:03:42.571444861 (33h23m14.267293614s) [noise=true , address=145.79.227.17:4970 , piece_id=6LJTN7QUSUSHS6DSKRHUUA3XB5ICEZ2ACFOSU4SK5HMUDRSKMEYA, seg_pos={0 0}, upload=29297]: Done uploading piece. err: <nil> Oct 31 21:03:42.707369620 (33h23m14.403218373s) [noise=true , address=145.79.227.11:4732 , piece_id=E4WNX7JXJZHMPKBPEJPSXJAYL4BP47HLXLO6DBHPICX6ESIK767Q, seg_pos={0 0}, upload=29297]: Done uploading piece. err: <nil> Oct 31 21:03:42.745077775 (33h23m14.440926518s) [noise=true , address=145.79.227.17:4971 , piece_id=3P7N3ERAPOY62SAKEBV7CHWOWWNUPWLSLL3RZ4RWCPF3VHI3C4FQ, seg_pos={0 0}, upload=29297]: Done uploading piece. err: <nil> Oct 31 21:04:00.023801048 (33h23m31.719649801s) [noise=true , address=77.90.31.17:4772 , piece_id=BFR46U4SPW6VNCIDV3XNIEOVYWGCXMQPDZ4JCJO5EAY5VSY4X56A, seg_pos={0 0}, upload=29297]: Done uploading piece. err: <nil> Oct 31 21:04:09.488974106 (33h23m41.184822859s) [noise=true , address=77.90.31.15:4698 , piece_id=2SJEFFLOU7LPMQZV5DLW5Z4BUQNJFZ347K656TFXV47LTCODGA3A, seg_pos={0 0}, upload=29297]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: write tcp 188.166.225.170:35474->77.90.31.15:4698: use of closed network connection; write tcp 188.166.225.170:35474->77.90.31.15:4698: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35474->77.90.31.15:4698: use of closed network connection Oct 31 21:04:09.489128110 (33h23m41.184976863s) [noise=true , address=77.90.31.10:4517 , piece_id=BDAUXG53XX42I72C6BIIC3VCLA6GZEFD3IDVSLYCC73AS2TG626Q, seg_pos={0 0}, upload=29297]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 188.166.225.170:58092->77.90.31.10:4517: use of closed network connection; write tcp 188.166.225.170:58092->77.90.31.10:4517: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58092->77.90.31.10:4517: use of closed network connection Oct 31 21:04:09.489250382 (33h23m41.185099135s) [noise=true , address=77.90.31.11:4555 , piece_id=BMTMJ4QDYLWDSMNIP64YUDUBZA2DABKYYC5JK72VJSUBTBZ67YTA, seg_pos={0 0}, upload=29297]: Done uploading piece. err: ecclient: upload failed (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, address:77.90.31.11:4555): protocol: write tcp 188.166.225.170:40898->77.90.31.11:4555: use of closed network connection; write tcp 188.166.225.170:40898->77.90.31.11:4555: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40898->77.90.31.11:4555: use of closed network connection Oct 31 21:04:09.489384528 (33h23m41.185233281s) [noise=true , address=77.90.31.15:4699 , piece_id=DLH6733XEDD5Z7AIO7NT4UC4TMTM57RUDVUHYOYONTXTGUJWJSYA, seg_pos={0 0}, upload=29297]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: write tcp 188.166.225.170:38634->77.90.31.15:4699: use of closed network connection; write tcp 188.166.225.170:38634->77.90.31.15:4699: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38634->77.90.31.15:4699: use of closed network connection Oct 31 21:04:09.489415327 (33h23m41.18526408s) [seg_pos={0 0}, upload=29297]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Oct 31 21:04:09.733073582 (33h23m41.428922335s) [upload=29297]: Done uploading segments... Oct 31 21:04:09.733121814 (33h23m41.428970557s) [upload=29297]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.023513466s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [..........................XX....X..................X........] duration: 5.933118536s trace: https://jaeger.dev.storj.io/trace/1a181df13358808f FAIL (2025-10-31 21:08:33 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Oct 31 21:03:40.187891701 (33h23m11.883740444s) [upload=29297]: Starting upload Oct 31 21:03:40.187972845 (33h23m11.883821598s) [upload=29297]: Done starting upload Oct 31 21:03:40.188163167 (33h23m11.88401191s) [upload=29297]: Uploading segments... Oct 31 21:03:40.188240786 (33h23m11.884089559s) [upload=29297]: Got next segment. Inline: false Oct 31 21:03:40.188281714 (33h23m11.884130467s) [upload=29297]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 31 21:03:40.188288065 (33h23m11.884136808s) [upload=29297]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 31 21:03:40.439495977 (33h23m12.13534474s) [seg_pos={0 0}, upload=29297]: Begin upload segment... Oct 31 21:03:40.439552094 (33h23m12.135400847s) [seg_pos={0 0}, upload=29297]: Done begin upload segment. Oct 31 21:03:40.439606758 (33h23m12.135455511s) [noise=true , address=145.79.227.21:4473 , piece_id=XBSRRHW5MAWDHZV3YFV7BWHL7VWX7C2YWB2DTNCFQJTXE43LAAYQ, seg_pos={0 0}, upload=29297]: Uploading piece... Oct 31 21:03:40.439643338 (33h23m12.135492081s) [noise=true , address=77.90.31.15:4698 , piece_id=2SJEFFLOU7LPMQZV5DLW5Z4BUQNJFZ347K656TFXV47LTCODGA3A, seg_pos={0 0}, upload=29297]: Uploading piece... Oct 31 21:03:40.440209786 (33h23m12.136058539s) [noise=true , address=145.79.227.14:4837 , piece_id=E3NUXQAOQNOCNTMUGO2NX2ELYKMII4J37DQPUBSQ6ABGBHIC3WSA, seg_pos={0 0}, upload=29297]: Uploading piece... Oct 31 21:03:40.440607473 (33h23m12.136456216s) [noise=true , address=145.79.227.13:4802 , piece_id=GZK6DPTPN4X5D4NRJABMLW6HNPXMH7IR25UE347P2BQZYKCJY3IQ, seg_pos={0 0}, upload=29297]: Uploading piece... Oct 31 21:03:40.441017394 (33h23m12.136866147s) [noise=true , address=77.90.31.15:4699 , piece_id=DLH6733XEDD5Z7AIO7NT4UC4TMTM57RUDVUHYOYONTXTGUJWJSYA, seg_pos={0 0}, upload=29297]: Uploading piece... Oct 31 21:03:40.441619029 (33h23m12.137467772s) [noise=true , address=77.90.31.17:4772 , piece_id=BFR46U4SPW6VNCIDV3XNIEOVYWGCXMQPDZ4JCJO5EAY5VSY4X56A, seg_pos={0 0}, upload=29297]: Uploading piece... Oct 31 21:03:40.442305126 (33h23m12.138153879s) [noise=true , address=145.79.227.17:4970 , piece_id=6LJTN7QUSUSHS6DSKRHUUA3XB5ICEZ2ACFOSU4SK5HMUDRSKMEYA, seg_pos={0 0}, upload=29297]: Uploading piece... Oct 31 21:03:40.443621452 (33h23m12.139470205s) [noise=true , address=145.79.227.15:4874 , piece_id=VK3CG2KERHGJQSKLJKOT7V46GRU6XEJN7T5S4HMTW2M76E5HOUHQ, seg_pos={0 0}, upload=29297]: Uploading piece... Oct 31 21:03:40.443864906 (33h23m12.139713659s) [noise=true , address=145.79.227.11:4732 , piece_id=E4WNX7JXJZHMPKBPEJPSXJAYL4BP47HLXLO6DBHPICX6ESIK767Q, seg_pos={0 0}, upload=29297]: Uploading piece... Oct 31 21:03:40.444991701 (33h23m12.140840444s) [noise=true , address=145.79.227.24:4583 , piece_id=KSDYUXPSUXPNBT7J7Q6RW7HNPILNVUWSX7STJXDBI7YKR7MA7XMA, seg_pos={0 0}, upload=29297]: Uploading piece... Oct 31 21:03:40.445358190 (33h23m12.141206933s) [noise=true , address=145.79.227.5:4223 , piece_id=CAMTFWQE5GQIDZMJDJXSO6SIHKQXKI6HRNQAFI46JICNCIMEGACA, seg_pos={0 0}, upload=29297]: Uploading piece... Oct 31 21:03:40.446480987 (33h23m12.14232974s) [noise=true , address=145.79.227.30:4762 , piece_id=HDRX3AY2PGE2UDY4SLN4A7T27KGLZQRTOYGKHCDC4S7EDVJOIWXA, seg_pos={0 0}, upload=29297]: Uploading piece... Oct 31 21:03:40.446948418 (33h23m12.142797171s) [noise=true , address=77.90.31.10:4517 , piece_id=BDAUXG53XX42I72C6BIIC3VCLA6GZEFD3IDVSLYCC73AS2TG626Q, seg_pos={0 0}, upload=29297]: Uploading piece... Oct 31 21:03:40.448429257 (33h23m12.14427801s) [noise=true , address=145.79.227.17:4971 , piece_id=3P7N3ERAPOY62SAKEBV7CHWOWWNUPWLSLL3RZ4RWCPF3VHI3C4FQ, seg_pos={0 0}, upload=29297]: Uploading piece... Oct 31 21:03:40.449960433 (33h23m12.145809176s) [noise=true , address=77.90.31.11:4555 , piece_id=BMTMJ4QDYLWDSMNIP64YUDUBZA2DABKYYC5JK72VJSUBTBZ67YTA, seg_pos={0 0}, upload=29297]: Uploading piece... Oct 31 21:03:40.451429571 (33h23m12.147278324s) [noise=true , address=145.79.227.4:4185 , piece_id=7C536MM2MA3EC44OTMXDUGNW5YDD3OOK5ODAU2FOFM3YTQ7VU6PQ, seg_pos={0 0}, upload=29297]: Uploading piece... Oct 31 21:03:42.182232662 (33h23m13.878081416s) [upload=29297]: Next returned nil segment Oct 31 21:03:42.182251107 (33h23m13.87809985s) [upload=29297]: Waiting for error group managing segments... Oct 31 21:03:42.460580042 (33h23m14.156428785s) [noise=true , address=145.79.227.24:4583 , piece_id=KSDYUXPSUXPNBT7J7Q6RW7HNPILNVUWSX7STJXDBI7YKR7MA7XMA, seg_pos={0 0}, upload=29297]: Done uploading piece. err: <nil> Oct 31 21:03:42.470437052 (33h23m14.166285805s) [noise=true , address=145.79.227.5:4223 , piece_id=CAMTFWQE5GQIDZMJDJXSO6SIHKQXKI6HRNQAFI46JICNCIMEGACA, seg_pos={0 0}, upload=29297]: Done uploading piece. err: <nil> Oct 31 21:03:42.478522949 (33h23m14.174371692s) [noise=true , address=145.79.227.4:4185 , piece_id=7C536MM2MA3EC44OTMXDUGNW5YDD3OOK5ODAU2FOFM3YTQ7VU6PQ, seg_pos={0 0}, upload=29297]: Done uploading piece. err: <nil> Oct 31 21:03:42.479322582 (33h23m14.175171325s) [noise=true , address=145.79.227.15:4874 , piece_id=VK3CG2KERHGJQSKLJKOT7V46GRU6XEJN7T5S4HMTW2M76E5HOUHQ, seg_pos={0 0}, upload=29297]: Done uploading piece. err: <nil> Oct 31 21:03:42.479617323 (33h23m14.175466076s) [noise=true , address=145.79.227.21:4473 , piece_id=XBSRRHW5MAWDHZV3YFV7BWHL7VWX7C2YWB2DTNCFQJTXE43LAAYQ, seg_pos={0 0}, upload=29297]: Done uploading piece. err: <nil> Oct 31 21:03:42.481690871 (33h23m14.177539614s) [noise=true , address=145.79.227.30:4762 , piece_id=HDRX3AY2PGE2UDY4SLN4A7T27KGLZQRTOYGKHCDC4S7EDVJOIWXA, seg_pos={0 0}, upload=29297]: Done uploading piece. err: <nil> Oct 31 21:03:42.513679364 (33h23m14.209528107s) [noise=true , address=145.79.227.13:4802 , piece_id=GZK6DPTPN4X5D4NRJABMLW6HNPXMH7IR25UE347P2BQZYKCJY3IQ, seg_pos={0 0}, upload=29297]: Done uploading piece. err: <nil> Oct 31 21:03:42.569840746 (33h23m14.265689489s) [noise=true , address=145.79.227.14:4837 , piece_id=E3NUXQAOQNOCNTMUGO2NX2ELYKMII4J37DQPUBSQ6ABGBHIC3WSA, seg_pos={0 0}, upload=29297]: Done uploading piece. err: <nil> Oct 31 21:03:42.571444861 (33h23m14.267293614s) [noise=true , address=145.79.227.17:4970 , piece_id=6LJTN7QUSUSHS6DSKRHUUA3XB5ICEZ2ACFOSU4SK5HMUDRSKMEYA, seg_pos={0 0}, upload=29297]: Done uploading piece. err: <nil> Oct 31 21:03:42.707369620 (33h23m14.403218373s) [noise=true , address=145.79.227.11:4732 , piece_id=E4WNX7JXJZHMPKBPEJPSXJAYL4BP47HLXLO6DBHPICX6ESIK767Q, seg_pos={0 0}, upload=29297]: Done uploading piece. err: <nil> Oct 31 21:03:42.745077775 (33h23m14.440926518s) [noise=true , address=145.79.227.17:4971 , piece_id=3P7N3ERAPOY62SAKEBV7CHWOWWNUPWLSLL3RZ4RWCPF3VHI3C4FQ, seg_pos={0 0}, upload=29297]: Done uploading piece. err: <nil> Oct 31 21:04:00.023801048 (33h23m31.719649801s) [noise=true , address=77.90.31.17:4772 , piece_id=BFR46U4SPW6VNCIDV3XNIEOVYWGCXMQPDZ4JCJO5EAY5VSY4X56A, seg_pos={0 0}, upload=29297]: Done uploading piece. err: <nil> Oct 31 21:04:09.488974106 (33h23m41.184822859s) [noise=true , address=77.90.31.15:4698 , piece_id=2SJEFFLOU7LPMQZV5DLW5Z4BUQNJFZ347K656TFXV47LTCODGA3A, seg_pos={0 0}, upload=29297]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: write tcp 188.166.225.170:35474->77.90.31.15:4698: use of closed network connection; write tcp 188.166.225.170:35474->77.90.31.15:4698: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35474->77.90.31.15:4698: use of closed network connection Oct 31 21:04:09.489128110 (33h23m41.184976863s) [noise=true , address=77.90.31.10:4517 , piece_id=BDAUXG53XX42I72C6BIIC3VCLA6GZEFD3IDVSLYCC73AS2TG626Q, seg_pos={0 0}, upload=29297]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 188.166.225.170:58092->77.90.31.10:4517: use of closed network connection; write tcp 188.166.225.170:58092->77.90.31.10:4517: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58092->77.90.31.10:4517: use of closed network connection Oct 31 21:04:09.489250382 (33h23m41.185099135s) [noise=true , address=77.90.31.11:4555 , piece_id=BMTMJ4QDYLWDSMNIP64YUDUBZA2DABKYYC5JK72VJSUBTBZ67YTA, seg_pos={0 0}, upload=29297]: Done uploading piece. err: ecclient: upload failed (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, address:77.90.31.11:4555): protocol: write tcp 188.166.225.170:40898->77.90.31.11:4555: use of closed network connection; write tcp 188.166.225.170:40898->77.90.31.11:4555: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40898->77.90.31.11:4555: use of closed network connection Oct 31 21:04:09.489384528 (33h23m41.185233281s) [noise=true , address=77.90.31.15:4699 , piece_id=DLH6733XEDD5Z7AIO7NT4UC4TMTM57RUDVUHYOYONTXTGUJWJSYA, seg_pos={0 0}, upload=29297]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: write tcp 188.166.225.170:38634->77.90.31.15:4699: use of closed network connection; write tcp 188.166.225.170:38634->77.90.31.15:4699: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38634->77.90.31.15:4699: use of closed network connection Oct 31 21:04:09.489415327 (33h23m41.18526408s) [seg_pos={0 0}, upload=29297]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Oct 31 21:04:09.733073582 (33h23m41.428922335s) [upload=29297]: Done uploading segments... Oct 31 21:04:09.733121814 (33h23m41.428970557s) [upload=29297]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.933151298s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.............................X....X.........................] duration: 6.097044876s trace: https://jaeger.dev.storj.io/trace/35569b98f076d00a FAIL (2025-10-31 21:08:30 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.097068521s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [...........................XX....X..................X.......] duration: 5.894725527s trace: https://jaeger.dev.storj.io/trace/1a181df13358808f FAIL (2025-10-31 21:08:16 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Oct 31 21:03:40.187891701 (33h23m11.883740444s) [upload=29297]: Starting upload Oct 31 21:03:40.187972845 (33h23m11.883821598s) [upload=29297]: Done starting upload Oct 31 21:03:40.188163167 (33h23m11.88401191s) [upload=29297]: Uploading segments... Oct 31 21:03:40.188240786 (33h23m11.884089559s) [upload=29297]: Got next segment. Inline: false Oct 31 21:03:40.188281714 (33h23m11.884130467s) [upload=29297]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 31 21:03:40.188288065 (33h23m11.884136808s) [upload=29297]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 31 21:03:40.439495977 (33h23m12.13534474s) [seg_pos={0 0}, upload=29297]: Begin upload segment... Oct 31 21:03:40.439552094 (33h23m12.135400847s) [seg_pos={0 0}, upload=29297]: Done begin upload segment. Oct 31 21:03:40.439606758 (33h23m12.135455511s) [noise=true , address=145.79.227.21:4473 , piece_id=XBSRRHW5MAWDHZV3YFV7BWHL7VWX7C2YWB2DTNCFQJTXE43LAAYQ, seg_pos={0 0}, upload=29297]: Uploading piece... Oct 31 21:03:40.439643338 (33h23m12.135492081s) [noise=true , address=77.90.31.15:4698 , piece_id=2SJEFFLOU7LPMQZV5DLW5Z4BUQNJFZ347K656TFXV47LTCODGA3A, seg_pos={0 0}, upload=29297]: Uploading piece... Oct 31 21:03:40.440209786 (33h23m12.136058539s) [noise=true , address=145.79.227.14:4837 , piece_id=E3NUXQAOQNOCNTMUGO2NX2ELYKMII4J37DQPUBSQ6ABGBHIC3WSA, seg_pos={0 0}, upload=29297]: Uploading piece... Oct 31 21:03:40.440607473 (33h23m12.136456216s) [noise=true , address=145.79.227.13:4802 , piece_id=GZK6DPTPN4X5D4NRJABMLW6HNPXMH7IR25UE347P2BQZYKCJY3IQ, seg_pos={0 0}, upload=29297]: Uploading piece... Oct 31 21:03:40.441017394 (33h23m12.136866147s) [noise=true , address=77.90.31.15:4699 , piece_id=DLH6733XEDD5Z7AIO7NT4UC4TMTM57RUDVUHYOYONTXTGUJWJSYA, seg_pos={0 0}, upload=29297]: Uploading piece... Oct 31 21:03:40.441619029 (33h23m12.137467772s) [noise=true , address=77.90.31.17:4772 , piece_id=BFR46U4SPW6VNCIDV3XNIEOVYWGCXMQPDZ4JCJO5EAY5VSY4X56A, seg_pos={0 0}, upload=29297]: Uploading piece... Oct 31 21:03:40.442305126 (33h23m12.138153879s) [noise=true , address=145.79.227.17:4970 , piece_id=6LJTN7QUSUSHS6DSKRHUUA3XB5ICEZ2ACFOSU4SK5HMUDRSKMEYA, seg_pos={0 0}, upload=29297]: Uploading piece... Oct 31 21:03:40.443621452 (33h23m12.139470205s) [noise=true , address=145.79.227.15:4874 , piece_id=VK3CG2KERHGJQSKLJKOT7V46GRU6XEJN7T5S4HMTW2M76E5HOUHQ, seg_pos={0 0}, upload=29297]: Uploading piece... Oct 31 21:03:40.443864906 (33h23m12.139713659s) [noise=true , address=145.79.227.11:4732 , piece_id=E4WNX7JXJZHMPKBPEJPSXJAYL4BP47HLXLO6DBHPICX6ESIK767Q, seg_pos={0 0}, upload=29297]: Uploading piece... Oct 31 21:03:40.444991701 (33h23m12.140840444s) [noise=true , address=145.79.227.24:4583 , piece_id=KSDYUXPSUXPNBT7J7Q6RW7HNPILNVUWSX7STJXDBI7YKR7MA7XMA, seg_pos={0 0}, upload=29297]: Uploading piece... Oct 31 21:03:40.445358190 (33h23m12.141206933s) [noise=true , address=145.79.227.5:4223 , piece_id=CAMTFWQE5GQIDZMJDJXSO6SIHKQXKI6HRNQAFI46JICNCIMEGACA, seg_pos={0 0}, upload=29297]: Uploading piece... Oct 31 21:03:40.446480987 (33h23m12.14232974s) [noise=true , address=145.79.227.30:4762 , piece_id=HDRX3AY2PGE2UDY4SLN4A7T27KGLZQRTOYGKHCDC4S7EDVJOIWXA, seg_pos={0 0}, upload=29297]: Uploading piece... Oct 31 21:03:40.446948418 (33h23m12.142797171s) [noise=true , address=77.90.31.10:4517 , piece_id=BDAUXG53XX42I72C6BIIC3VCLA6GZEFD3IDVSLYCC73AS2TG626Q, seg_pos={0 0}, upload=29297]: Uploading piece... Oct 31 21:03:40.448429257 (33h23m12.14427801s) [noise=true , address=145.79.227.17:4971 , piece_id=3P7N3ERAPOY62SAKEBV7CHWOWWNUPWLSLL3RZ4RWCPF3VHI3C4FQ, seg_pos={0 0}, upload=29297]: Uploading piece... Oct 31 21:03:40.449960433 (33h23m12.145809176s) [noise=true , address=77.90.31.11:4555 , piece_id=BMTMJ4QDYLWDSMNIP64YUDUBZA2DABKYYC5JK72VJSUBTBZ67YTA, seg_pos={0 0}, upload=29297]: Uploading piece... Oct 31 21:03:40.451429571 (33h23m12.147278324s) [noise=true , address=145.79.227.4:4185 , piece_id=7C536MM2MA3EC44OTMXDUGNW5YDD3OOK5ODAU2FOFM3YTQ7VU6PQ, seg_pos={0 0}, upload=29297]: Uploading piece... Oct 31 21:03:42.182232662 (33h23m13.878081416s) [upload=29297]: Next returned nil segment Oct 31 21:03:42.182251107 (33h23m13.87809985s) [upload=29297]: Waiting for error group managing segments... Oct 31 21:03:42.460580042 (33h23m14.156428785s) [noise=true , address=145.79.227.24:4583 , piece_id=KSDYUXPSUXPNBT7J7Q6RW7HNPILNVUWSX7STJXDBI7YKR7MA7XMA, seg_pos={0 0}, upload=29297]: Done uploading piece. err: <nil> Oct 31 21:03:42.470437052 (33h23m14.166285805s) [noise=true , address=145.79.227.5:4223 , piece_id=CAMTFWQE5GQIDZMJDJXSO6SIHKQXKI6HRNQAFI46JICNCIMEGACA, seg_pos={0 0}, upload=29297]: Done uploading piece. err: <nil> Oct 31 21:03:42.478522949 (33h23m14.174371692s) [noise=true , address=145.79.227.4:4185 , piece_id=7C536MM2MA3EC44OTMXDUGNW5YDD3OOK5ODAU2FOFM3YTQ7VU6PQ, seg_pos={0 0}, upload=29297]: Done uploading piece. err: <nil> Oct 31 21:03:42.479322582 (33h23m14.175171325s) [noise=true , address=145.79.227.15:4874 , piece_id=VK3CG2KERHGJQSKLJKOT7V46GRU6XEJN7T5S4HMTW2M76E5HOUHQ, seg_pos={0 0}, upload=29297]: Done uploading piece. err: <nil> Oct 31 21:03:42.479617323 (33h23m14.175466076s) [noise=true , address=145.79.227.21:4473 , piece_id=XBSRRHW5MAWDHZV3YFV7BWHL7VWX7C2YWB2DTNCFQJTXE43LAAYQ, seg_pos={0 0}, upload=29297]: Done uploading piece. err: <nil> Oct 31 21:03:42.481690871 (33h23m14.177539614s) [noise=true , address=145.79.227.30:4762 , piece_id=HDRX3AY2PGE2UDY4SLN4A7T27KGLZQRTOYGKHCDC4S7EDVJOIWXA, seg_pos={0 0}, upload=29297]: Done uploading piece. err: <nil> Oct 31 21:03:42.513679364 (33h23m14.209528107s) [noise=true , address=145.79.227.13:4802 , piece_id=GZK6DPTPN4X5D4NRJABMLW6HNPXMH7IR25UE347P2BQZYKCJY3IQ, seg_pos={0 0}, upload=29297]: Done uploading piece. err: <nil> Oct 31 21:03:42.569840746 (33h23m14.265689489s) [noise=true , address=145.79.227.14:4837 , piece_id=E3NUXQAOQNOCNTMUGO2NX2ELYKMII4J37DQPUBSQ6ABGBHIC3WSA, seg_pos={0 0}, upload=29297]: Done uploading piece. err: <nil> Oct 31 21:03:42.571444861 (33h23m14.267293614s) [noise=true , address=145.79.227.17:4970 , piece_id=6LJTN7QUSUSHS6DSKRHUUA3XB5ICEZ2ACFOSU4SK5HMUDRSKMEYA, seg_pos={0 0}, upload=29297]: Done uploading piece. err: <nil> Oct 31 21:03:42.707369620 (33h23m14.403218373s) [noise=true , address=145.79.227.11:4732 , piece_id=E4WNX7JXJZHMPKBPEJPSXJAYL4BP47HLXLO6DBHPICX6ESIK767Q, seg_pos={0 0}, upload=29297]: Done uploading piece. err: <nil> Oct 31 21:03:42.745077775 (33h23m14.440926518s) [noise=true , address=145.79.227.17:4971 , piece_id=3P7N3ERAPOY62SAKEBV7CHWOWWNUPWLSLL3RZ4RWCPF3VHI3C4FQ, seg_pos={0 0}, upload=29297]: Done uploading piece. err: <nil> Oct 31 21:04:00.023801048 (33h23m31.719649801s) [noise=true , address=77.90.31.17:4772 , piece_id=BFR46U4SPW6VNCIDV3XNIEOVYWGCXMQPDZ4JCJO5EAY5VSY4X56A, seg_pos={0 0}, upload=29297]: Done uploading piece. err: <nil> Oct 31 21:04:09.488974106 (33h23m41.184822859s) [noise=true , address=77.90.31.15:4698 , piece_id=2SJEFFLOU7LPMQZV5DLW5Z4BUQNJFZ347K656TFXV47LTCODGA3A, seg_pos={0 0}, upload=29297]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: write tcp 188.166.225.170:35474->77.90.31.15:4698: use of closed network connection; write tcp 188.166.225.170:35474->77.90.31.15:4698: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35474->77.90.31.15:4698: use of closed network connection Oct 31 21:04:09.489128110 (33h23m41.184976863s) [noise=true , address=77.90.31.10:4517 , piece_id=BDAUXG53XX42I72C6BIIC3VCLA6GZEFD3IDVSLYCC73AS2TG626Q, seg_pos={0 0}, upload=29297]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 188.166.225.170:58092->77.90.31.10:4517: use of closed network connection; write tcp 188.166.225.170:58092->77.90.31.10:4517: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58092->77.90.31.10:4517: use of closed network connection Oct 31 21:04:09.489250382 (33h23m41.185099135s) [noise=true , address=77.90.31.11:4555 , piece_id=BMTMJ4QDYLWDSMNIP64YUDUBZA2DABKYYC5JK72VJSUBTBZ67YTA, seg_pos={0 0}, upload=29297]: Done uploading piece. err: ecclient: upload failed (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, address:77.90.31.11:4555): protocol: write tcp 188.166.225.170:40898->77.90.31.11:4555: use of closed network connection; write tcp 188.166.225.170:40898->77.90.31.11:4555: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40898->77.90.31.11:4555: use of closed network connection Oct 31 21:04:09.489384528 (33h23m41.185233281s) [noise=true , address=77.90.31.15:4699 , piece_id=DLH6733XEDD5Z7AIO7NT4UC4TMTM57RUDVUHYOYONTXTGUJWJSYA, seg_pos={0 0}, upload=29297]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: write tcp 188.166.225.170:38634->77.90.31.15:4699: use of closed network connection; write tcp 188.166.225.170:38634->77.90.31.15:4699: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38634->77.90.31.15:4699: use of closed network connection Oct 31 21:04:09.489415327 (33h23m41.18526408s) [seg_pos={0 0}, upload=29297]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Oct 31 21:04:09.733073582 (33h23m41.428922335s) [upload=29297]: Done uploading segments... Oct 31 21:04:09.733121814 (33h23m41.428970557s) [upload=29297]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.894752679s core uplink us-select-1
checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect) antiblip: 1 out of 15 is failed (threshold=6) [......X........] duration: 318.554874ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/dcs-satellites trace: https://jaeger.dev.storj.io/trace/571fef006088dd70 FAIL (2025-10-31 21:08:07 UTC) orbiter warning: Get "https://www.storj.io/dcs-satellites": read tcp [2400:6180:0:d0::f70:8001]:42806->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 318.585582ms core website
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [............................XX....X..................X......] duration: 6.011092542s trace: https://jaeger.dev.storj.io/trace/1a181df13358808f FAIL (2025-10-31 21:07:55 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Oct 31 21:03:40.187891701 (33h23m11.883740444s) [upload=29297]: Starting upload Oct 31 21:03:40.187972845 (33h23m11.883821598s) [upload=29297]: Done starting upload Oct 31 21:03:40.188163167 (33h23m11.88401191s) [upload=29297]: Uploading segments... Oct 31 21:03:40.188240786 (33h23m11.884089559s) [upload=29297]: Got next segment. Inline: false Oct 31 21:03:40.188281714 (33h23m11.884130467s) [upload=29297]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 31 21:03:40.188288065 (33h23m11.884136808s) [upload=29297]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 31 21:03:40.439495977 (33h23m12.13534474s) [seg_pos={0 0}, upload=29297]: Begin upload segment... Oct 31 21:03:40.439552094 (33h23m12.135400847s) [seg_pos={0 0}, upload=29297]: Done begin upload segment. Oct 31 21:03:40.439606758 (33h23m12.135455511s) [noise=true , address=145.79.227.21:4473 , piece_id=XBSRRHW5MAWDHZV3YFV7BWHL7VWX7C2YWB2DTNCFQJTXE43LAAYQ, seg_pos={0 0}, upload=29297]: Uploading piece... Oct 31 21:03:40.439643338 (33h23m12.135492081s) [noise=true , address=77.90.31.15:4698 , piece_id=2SJEFFLOU7LPMQZV5DLW5Z4BUQNJFZ347K656TFXV47LTCODGA3A, seg_pos={0 0}, upload=29297]: Uploading piece... Oct 31 21:03:40.440209786 (33h23m12.136058539s) [noise=true , address=145.79.227.14:4837 , piece_id=E3NUXQAOQNOCNTMUGO2NX2ELYKMII4J37DQPUBSQ6ABGBHIC3WSA, seg_pos={0 0}, upload=29297]: Uploading piece... Oct 31 21:03:40.440607473 (33h23m12.136456216s) [noise=true , address=145.79.227.13:4802 , piece_id=GZK6DPTPN4X5D4NRJABMLW6HNPXMH7IR25UE347P2BQZYKCJY3IQ, seg_pos={0 0}, upload=29297]: Uploading piece... Oct 31 21:03:40.441017394 (33h23m12.136866147s) [noise=true , address=77.90.31.15:4699 , piece_id=DLH6733XEDD5Z7AIO7NT4UC4TMTM57RUDVUHYOYONTXTGUJWJSYA, seg_pos={0 0}, upload=29297]: Uploading piece... Oct 31 21:03:40.441619029 (33h23m12.137467772s) [noise=true , address=77.90.31.17:4772 , piece_id=BFR46U4SPW6VNCIDV3XNIEOVYWGCXMQPDZ4JCJO5EAY5VSY4X56A, seg_pos={0 0}, upload=29297]: Uploading piece... Oct 31 21:03:40.442305126 (33h23m12.138153879s) [noise=true , address=145.79.227.17:4970 , piece_id=6LJTN7QUSUSHS6DSKRHUUA3XB5ICEZ2ACFOSU4SK5HMUDRSKMEYA, seg_pos={0 0}, upload=29297]: Uploading piece... Oct 31 21:03:40.443621452 (33h23m12.139470205s) [noise=true , address=145.79.227.15:4874 , piece_id=VK3CG2KERHGJQSKLJKOT7V46GRU6XEJN7T5S4HMTW2M76E5HOUHQ, seg_pos={0 0}, upload=29297]: Uploading piece... Oct 31 21:03:40.443864906 (33h23m12.139713659s) [noise=true , address=145.79.227.11:4732 , piece_id=E4WNX7JXJZHMPKBPEJPSXJAYL4BP47HLXLO6DBHPICX6ESIK767Q, seg_pos={0 0}, upload=29297]: Uploading piece... Oct 31 21:03:40.444991701 (33h23m12.140840444s) [noise=true , address=145.79.227.24:4583 , piece_id=KSDYUXPSUXPNBT7J7Q6RW7HNPILNVUWSX7STJXDBI7YKR7MA7XMA, seg_pos={0 0}, upload=29297]: Uploading piece... Oct 31 21:03:40.445358190 (33h23m12.141206933s) [noise=true , address=145.79.227.5:4223 , piece_id=CAMTFWQE5GQIDZMJDJXSO6SIHKQXKI6HRNQAFI46JICNCIMEGACA, seg_pos={0 0}, upload=29297]: Uploading piece... Oct 31 21:03:40.446480987 (33h23m12.14232974s) [noise=true , address=145.79.227.30:4762 , piece_id=HDRX3AY2PGE2UDY4SLN4A7T27KGLZQRTOYGKHCDC4S7EDVJOIWXA, seg_pos={0 0}, upload=29297]: Uploading piece... Oct 31 21:03:40.446948418 (33h23m12.142797171s) [noise=true , address=77.90.31.10:4517 , piece_id=BDAUXG53XX42I72C6BIIC3VCLA6GZEFD3IDVSLYCC73AS2TG626Q, seg_pos={0 0}, upload=29297]: Uploading piece... Oct 31 21:03:40.448429257 (33h23m12.14427801s) [noise=true , address=145.79.227.17:4971 , piece_id=3P7N3ERAPOY62SAKEBV7CHWOWWNUPWLSLL3RZ4RWCPF3VHI3C4FQ, seg_pos={0 0}, upload=29297]: Uploading piece... Oct 31 21:03:40.449960433 (33h23m12.145809176s) [noise=true , address=77.90.31.11:4555 , piece_id=BMTMJ4QDYLWDSMNIP64YUDUBZA2DABKYYC5JK72VJSUBTBZ67YTA, seg_pos={0 0}, upload=29297]: Uploading piece... Oct 31 21:03:40.451429571 (33h23m12.147278324s) [noise=true , address=145.79.227.4:4185 , piece_id=7C536MM2MA3EC44OTMXDUGNW5YDD3OOK5ODAU2FOFM3YTQ7VU6PQ, seg_pos={0 0}, upload=29297]: Uploading piece... Oct 31 21:03:42.182232662 (33h23m13.878081416s) [upload=29297]: Next returned nil segment Oct 31 21:03:42.182251107 (33h23m13.87809985s) [upload=29297]: Waiting for error group managing segments... Oct 31 21:03:42.460580042 (33h23m14.156428785s) [noise=true , address=145.79.227.24:4583 , piece_id=KSDYUXPSUXPNBT7J7Q6RW7HNPILNVUWSX7STJXDBI7YKR7MA7XMA, seg_pos={0 0}, upload=29297]: Done uploading piece. err: <nil> Oct 31 21:03:42.470437052 (33h23m14.166285805s) [noise=true , address=145.79.227.5:4223 , piece_id=CAMTFWQE5GQIDZMJDJXSO6SIHKQXKI6HRNQAFI46JICNCIMEGACA, seg_pos={0 0}, upload=29297]: Done uploading piece. err: <nil> Oct 31 21:03:42.478522949 (33h23m14.174371692s) [noise=true , address=145.79.227.4:4185 , piece_id=7C536MM2MA3EC44OTMXDUGNW5YDD3OOK5ODAU2FOFM3YTQ7VU6PQ, seg_pos={0 0}, upload=29297]: Done uploading piece. err: <nil> Oct 31 21:03:42.479322582 (33h23m14.175171325s) [noise=true , address=145.79.227.15:4874 , piece_id=VK3CG2KERHGJQSKLJKOT7V46GRU6XEJN7T5S4HMTW2M76E5HOUHQ, seg_pos={0 0}, upload=29297]: Done uploading piece. err: <nil> Oct 31 21:03:42.479617323 (33h23m14.175466076s) [noise=true , address=145.79.227.21:4473 , piece_id=XBSRRHW5MAWDHZV3YFV7BWHL7VWX7C2YWB2DTNCFQJTXE43LAAYQ, seg_pos={0 0}, upload=29297]: Done uploading piece. err: <nil> Oct 31 21:03:42.481690871 (33h23m14.177539614s) [noise=true , address=145.79.227.30:4762 , piece_id=HDRX3AY2PGE2UDY4SLN4A7T27KGLZQRTOYGKHCDC4S7EDVJOIWXA, seg_pos={0 0}, upload=29297]: Done uploading piece. err: <nil> Oct 31 21:03:42.513679364 (33h23m14.209528107s) [noise=true , address=145.79.227.13:4802 , piece_id=GZK6DPTPN4X5D4NRJABMLW6HNPXMH7IR25UE347P2BQZYKCJY3IQ, seg_pos={0 0}, upload=29297]: Done uploading piece. err: <nil> Oct 31 21:03:42.569840746 (33h23m14.265689489s) [noise=true , address=145.79.227.14:4837 , piece_id=E3NUXQAOQNOCNTMUGO2NX2ELYKMII4J37DQPUBSQ6ABGBHIC3WSA, seg_pos={0 0}, upload=29297]: Done uploading piece. err: <nil> Oct 31 21:03:42.571444861 (33h23m14.267293614s) [noise=true , address=145.79.227.17:4970 , piece_id=6LJTN7QUSUSHS6DSKRHUUA3XB5ICEZ2ACFOSU4SK5HMUDRSKMEYA, seg_pos={0 0}, upload=29297]: Done uploading piece. err: <nil> Oct 31 21:03:42.707369620 (33h23m14.403218373s) [noise=true , address=145.79.227.11:4732 , piece_id=E4WNX7JXJZHMPKBPEJPSXJAYL4BP47HLXLO6DBHPICX6ESIK767Q, seg_pos={0 0}, upload=29297]: Done uploading piece. err: <nil> Oct 31 21:03:42.745077775 (33h23m14.440926518s) [noise=true , address=145.79.227.17:4971 , piece_id=3P7N3ERAPOY62SAKEBV7CHWOWWNUPWLSLL3RZ4RWCPF3VHI3C4FQ, seg_pos={0 0}, upload=29297]: Done uploading piece. err: <nil> Oct 31 21:04:00.023801048 (33h23m31.719649801s) [noise=true , address=77.90.31.17:4772 , piece_id=BFR46U4SPW6VNCIDV3XNIEOVYWGCXMQPDZ4JCJO5EAY5VSY4X56A, seg_pos={0 0}, upload=29297]: Done uploading piece. err: <nil> Oct 31 21:04:09.488974106 (33h23m41.184822859s) [noise=true , address=77.90.31.15:4698 , piece_id=2SJEFFLOU7LPMQZV5DLW5Z4BUQNJFZ347K656TFXV47LTCODGA3A, seg_pos={0 0}, upload=29297]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: write tcp 188.166.225.170:35474->77.90.31.15:4698: use of closed network connection; write tcp 188.166.225.170:35474->77.90.31.15:4698: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35474->77.90.31.15:4698: use of closed network connection Oct 31 21:04:09.489128110 (33h23m41.184976863s) [noise=true , address=77.90.31.10:4517 , piece_id=BDAUXG53XX42I72C6BIIC3VCLA6GZEFD3IDVSLYCC73AS2TG626Q, seg_pos={0 0}, upload=29297]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 188.166.225.170:58092->77.90.31.10:4517: use of closed network connection; write tcp 188.166.225.170:58092->77.90.31.10:4517: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58092->77.90.31.10:4517: use of closed network connection Oct 31 21:04:09.489250382 (33h23m41.185099135s) [noise=true , address=77.90.31.11:4555 , piece_id=BMTMJ4QDYLWDSMNIP64YUDUBZA2DABKYYC5JK72VJSUBTBZ67YTA, seg_pos={0 0}, upload=29297]: Done uploading piece. err: ecclient: upload failed (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, address:77.90.31.11:4555): protocol: write tcp 188.166.225.170:40898->77.90.31.11:4555: use of closed network connection; write tcp 188.166.225.170:40898->77.90.31.11:4555: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40898->77.90.31.11:4555: use of closed network connection Oct 31 21:04:09.489384528 (33h23m41.185233281s) [noise=true , address=77.90.31.15:4699 , piece_id=DLH6733XEDD5Z7AIO7NT4UC4TMTM57RUDVUHYOYONTXTGUJWJSYA, seg_pos={0 0}, upload=29297]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: write tcp 188.166.225.170:38634->77.90.31.15:4699: use of closed network connection; write tcp 188.166.225.170:38634->77.90.31.15:4699: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38634->77.90.31.15:4699: use of closed network connection Oct 31 21:04:09.489415327 (33h23m41.18526408s) [seg_pos={0 0}, upload=29297]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Oct 31 21:04:09.733073582 (33h23m41.428922335s) [upload=29297]: Done uploading segments... Oct 31 21:04:09.733121814 (33h23m41.428970557s) [upload=29297]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.011123442s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..............................X....X........................] duration: 6.060060596s trace: https://jaeger.dev.storj.io/trace/35569b98f076d00a FAIL (2025-10-31 21:07:35 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.060092106s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...............................X....X.......................] duration: 6.100056749s trace: https://jaeger.dev.storj.io/trace/35569b98f076d00a FAIL (2025-10-31 21:07:17 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.100082719s edge gateway us-select-1
checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect) antiblip: 1 out of 15 is failed (threshold=6) [.......X.......] duration: 370.358347ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/dcs-satellites trace: https://jaeger.dev.storj.io/trace/571fef006088dd70 FAIL (2025-10-31 21:07:10 UTC) orbiter warning: Get "https://www.storj.io/dcs-satellites": read tcp [2400:6180:0:d0::f70:8001]:42806->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 370.38649ms core website
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.............................XX....X..................X.....] duration: 5.991592049s trace: https://jaeger.dev.storj.io/trace/1a181df13358808f FAIL (2025-10-31 21:07:07 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Oct 31 21:03:40.187891701 (33h23m11.883740444s) [upload=29297]: Starting upload Oct 31 21:03:40.187972845 (33h23m11.883821598s) [upload=29297]: Done starting upload Oct 31 21:03:40.188163167 (33h23m11.88401191s) [upload=29297]: Uploading segments... Oct 31 21:03:40.188240786 (33h23m11.884089559s) [upload=29297]: Got next segment. Inline: false Oct 31 21:03:40.188281714 (33h23m11.884130467s) [upload=29297]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 31 21:03:40.188288065 (33h23m11.884136808s) [upload=29297]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 31 21:03:40.439495977 (33h23m12.13534474s) [seg_pos={0 0}, upload=29297]: Begin upload segment... Oct 31 21:03:40.439552094 (33h23m12.135400847s) [seg_pos={0 0}, upload=29297]: Done begin upload segment. Oct 31 21:03:40.439606758 (33h23m12.135455511s) [noise=true , address=145.79.227.21:4473 , piece_id=XBSRRHW5MAWDHZV3YFV7BWHL7VWX7C2YWB2DTNCFQJTXE43LAAYQ, seg_pos={0 0}, upload=29297]: Uploading piece... Oct 31 21:03:40.439643338 (33h23m12.135492081s) [noise=true , address=77.90.31.15:4698 , piece_id=2SJEFFLOU7LPMQZV5DLW5Z4BUQNJFZ347K656TFXV47LTCODGA3A, seg_pos={0 0}, upload=29297]: Uploading piece... Oct 31 21:03:40.440209786 (33h23m12.136058539s) [noise=true , address=145.79.227.14:4837 , piece_id=E3NUXQAOQNOCNTMUGO2NX2ELYKMII4J37DQPUBSQ6ABGBHIC3WSA, seg_pos={0 0}, upload=29297]: Uploading piece... Oct 31 21:03:40.440607473 (33h23m12.136456216s) [noise=true , address=145.79.227.13:4802 , piece_id=GZK6DPTPN4X5D4NRJABMLW6HNPXMH7IR25UE347P2BQZYKCJY3IQ, seg_pos={0 0}, upload=29297]: Uploading piece... Oct 31 21:03:40.441017394 (33h23m12.136866147s) [noise=true , address=77.90.31.15:4699 , piece_id=DLH6733XEDD5Z7AIO7NT4UC4TMTM57RUDVUHYOYONTXTGUJWJSYA, seg_pos={0 0}, upload=29297]: Uploading piece... Oct 31 21:03:40.441619029 (33h23m12.137467772s) [noise=true , address=77.90.31.17:4772 , piece_id=BFR46U4SPW6VNCIDV3XNIEOVYWGCXMQPDZ4JCJO5EAY5VSY4X56A, seg_pos={0 0}, upload=29297]: Uploading piece... Oct 31 21:03:40.442305126 (33h23m12.138153879s) [noise=true , address=145.79.227.17:4970 , piece_id=6LJTN7QUSUSHS6DSKRHUUA3XB5ICEZ2ACFOSU4SK5HMUDRSKMEYA, seg_pos={0 0}, upload=29297]: Uploading piece... Oct 31 21:03:40.443621452 (33h23m12.139470205s) [noise=true , address=145.79.227.15:4874 , piece_id=VK3CG2KERHGJQSKLJKOT7V46GRU6XEJN7T5S4HMTW2M76E5HOUHQ, seg_pos={0 0}, upload=29297]: Uploading piece... Oct 31 21:03:40.443864906 (33h23m12.139713659s) [noise=true , address=145.79.227.11:4732 , piece_id=E4WNX7JXJZHMPKBPEJPSXJAYL4BP47HLXLO6DBHPICX6ESIK767Q, seg_pos={0 0}, upload=29297]: Uploading piece... Oct 31 21:03:40.444991701 (33h23m12.140840444s) [noise=true , address=145.79.227.24:4583 , piece_id=KSDYUXPSUXPNBT7J7Q6RW7HNPILNVUWSX7STJXDBI7YKR7MA7XMA, seg_pos={0 0}, upload=29297]: Uploading piece... Oct 31 21:03:40.445358190 (33h23m12.141206933s) [noise=true , address=145.79.227.5:4223 , piece_id=CAMTFWQE5GQIDZMJDJXSO6SIHKQXKI6HRNQAFI46JICNCIMEGACA, seg_pos={0 0}, upload=29297]: Uploading piece... Oct 31 21:03:40.446480987 (33h23m12.14232974s) [noise=true , address=145.79.227.30:4762 , piece_id=HDRX3AY2PGE2UDY4SLN4A7T27KGLZQRTOYGKHCDC4S7EDVJOIWXA, seg_pos={0 0}, upload=29297]: Uploading piece... Oct 31 21:03:40.446948418 (33h23m12.142797171s) [noise=true , address=77.90.31.10:4517 , piece_id=BDAUXG53XX42I72C6BIIC3VCLA6GZEFD3IDVSLYCC73AS2TG626Q, seg_pos={0 0}, upload=29297]: Uploading piece... Oct 31 21:03:40.448429257 (33h23m12.14427801s) [noise=true , address=145.79.227.17:4971 , piece_id=3P7N3ERAPOY62SAKEBV7CHWOWWNUPWLSLL3RZ4RWCPF3VHI3C4FQ, seg_pos={0 0}, upload=29297]: Uploading piece... Oct 31 21:03:40.449960433 (33h23m12.145809176s) [noise=true , address=77.90.31.11:4555 , piece_id=BMTMJ4QDYLWDSMNIP64YUDUBZA2DABKYYC5JK72VJSUBTBZ67YTA, seg_pos={0 0}, upload=29297]: Uploading piece... Oct 31 21:03:40.451429571 (33h23m12.147278324s) [noise=true , address=145.79.227.4:4185 , piece_id=7C536MM2MA3EC44OTMXDUGNW5YDD3OOK5ODAU2FOFM3YTQ7VU6PQ, seg_pos={0 0}, upload=29297]: Uploading piece... Oct 31 21:03:42.182232662 (33h23m13.878081416s) [upload=29297]: Next returned nil segment Oct 31 21:03:42.182251107 (33h23m13.87809985s) [upload=29297]: Waiting for error group managing segments... Oct 31 21:03:42.460580042 (33h23m14.156428785s) [noise=true , address=145.79.227.24:4583 , piece_id=KSDYUXPSUXPNBT7J7Q6RW7HNPILNVUWSX7STJXDBI7YKR7MA7XMA, seg_pos={0 0}, upload=29297]: Done uploading piece. err: <nil> Oct 31 21:03:42.470437052 (33h23m14.166285805s) [noise=true , address=145.79.227.5:4223 , piece_id=CAMTFWQE5GQIDZMJDJXSO6SIHKQXKI6HRNQAFI46JICNCIMEGACA, seg_pos={0 0}, upload=29297]: Done uploading piece. err: <nil> Oct 31 21:03:42.478522949 (33h23m14.174371692s) [noise=true , address=145.79.227.4:4185 , piece_id=7C536MM2MA3EC44OTMXDUGNW5YDD3OOK5ODAU2FOFM3YTQ7VU6PQ, seg_pos={0 0}, upload=29297]: Done uploading piece. err: <nil> Oct 31 21:03:42.479322582 (33h23m14.175171325s) [noise=true , address=145.79.227.15:4874 , piece_id=VK3CG2KERHGJQSKLJKOT7V46GRU6XEJN7T5S4HMTW2M76E5HOUHQ, seg_pos={0 0}, upload=29297]: Done uploading piece. err: <nil> Oct 31 21:03:42.479617323 (33h23m14.175466076s) [noise=true , address=145.79.227.21:4473 , piece_id=XBSRRHW5MAWDHZV3YFV7BWHL7VWX7C2YWB2DTNCFQJTXE43LAAYQ, seg_pos={0 0}, upload=29297]: Done uploading piece. err: <nil> Oct 31 21:03:42.481690871 (33h23m14.177539614s) [noise=true , address=145.79.227.30:4762 , piece_id=HDRX3AY2PGE2UDY4SLN4A7T27KGLZQRTOYGKHCDC4S7EDVJOIWXA, seg_pos={0 0}, upload=29297]: Done uploading piece. err: <nil> Oct 31 21:03:42.513679364 (33h23m14.209528107s) [noise=true , address=145.79.227.13:4802 , piece_id=GZK6DPTPN4X5D4NRJABMLW6HNPXMH7IR25UE347P2BQZYKCJY3IQ, seg_pos={0 0}, upload=29297]: Done uploading piece. err: <nil> Oct 31 21:03:42.569840746 (33h23m14.265689489s) [noise=true , address=145.79.227.14:4837 , piece_id=E3NUXQAOQNOCNTMUGO2NX2ELYKMII4J37DQPUBSQ6ABGBHIC3WSA, seg_pos={0 0}, upload=29297]: Done uploading piece. err: <nil> Oct 31 21:03:42.571444861 (33h23m14.267293614s) [noise=true , address=145.79.227.17:4970 , piece_id=6LJTN7QUSUSHS6DSKRHUUA3XB5ICEZ2ACFOSU4SK5HMUDRSKMEYA, seg_pos={0 0}, upload=29297]: Done uploading piece. err: <nil> Oct 31 21:03:42.707369620 (33h23m14.403218373s) [noise=true , address=145.79.227.11:4732 , piece_id=E4WNX7JXJZHMPKBPEJPSXJAYL4BP47HLXLO6DBHPICX6ESIK767Q, seg_pos={0 0}, upload=29297]: Done uploading piece. err: <nil> Oct 31 21:03:42.745077775 (33h23m14.440926518s) [noise=true , address=145.79.227.17:4971 , piece_id=3P7N3ERAPOY62SAKEBV7CHWOWWNUPWLSLL3RZ4RWCPF3VHI3C4FQ, seg_pos={0 0}, upload=29297]: Done uploading piece. err: <nil> Oct 31 21:04:00.023801048 (33h23m31.719649801s) [noise=true , address=77.90.31.17:4772 , piece_id=BFR46U4SPW6VNCIDV3XNIEOVYWGCXMQPDZ4JCJO5EAY5VSY4X56A, seg_pos={0 0}, upload=29297]: Done uploading piece. err: <nil> Oct 31 21:04:09.488974106 (33h23m41.184822859s) [noise=true , address=77.90.31.15:4698 , piece_id=2SJEFFLOU7LPMQZV5DLW5Z4BUQNJFZ347K656TFXV47LTCODGA3A, seg_pos={0 0}, upload=29297]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: write tcp 188.166.225.170:35474->77.90.31.15:4698: use of closed network connection; write tcp 188.166.225.170:35474->77.90.31.15:4698: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35474->77.90.31.15:4698: use of closed network connection Oct 31 21:04:09.489128110 (33h23m41.184976863s) [noise=true , address=77.90.31.10:4517 , piece_id=BDAUXG53XX42I72C6BIIC3VCLA6GZEFD3IDVSLYCC73AS2TG626Q, seg_pos={0 0}, upload=29297]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 188.166.225.170:58092->77.90.31.10:4517: use of closed network connection; write tcp 188.166.225.170:58092->77.90.31.10:4517: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58092->77.90.31.10:4517: use of closed network connection Oct 31 21:04:09.489250382 (33h23m41.185099135s) [noise=true , address=77.90.31.11:4555 , piece_id=BMTMJ4QDYLWDSMNIP64YUDUBZA2DABKYYC5JK72VJSUBTBZ67YTA, seg_pos={0 0}, upload=29297]: Done uploading piece. err: ecclient: upload failed (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, address:77.90.31.11:4555): protocol: write tcp 188.166.225.170:40898->77.90.31.11:4555: use of closed network connection; write tcp 188.166.225.170:40898->77.90.31.11:4555: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40898->77.90.31.11:4555: use of closed network connection Oct 31 21:04:09.489384528 (33h23m41.185233281s) [noise=true , address=77.90.31.15:4699 , piece_id=DLH6733XEDD5Z7AIO7NT4UC4TMTM57RUDVUHYOYONTXTGUJWJSYA, seg_pos={0 0}, upload=29297]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: write tcp 188.166.225.170:38634->77.90.31.15:4699: use of closed network connection; write tcp 188.166.225.170:38634->77.90.31.15:4699: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38634->77.90.31.15:4699: use of closed network connection Oct 31 21:04:09.489415327 (33h23m41.18526408s) [seg_pos={0 0}, upload=29297]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Oct 31 21:04:09.733073582 (33h23m41.428922335s) [upload=29297]: Done uploading segments... Oct 31 21:04:09.733121814 (33h23m41.428970557s) [upload=29297]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.991627528s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [................................X....X......................] duration: 6.321728507s trace: https://jaeger.dev.storj.io/trace/35569b98f076d00a FAIL (2025-10-31 21:06:59 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.321755939s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [..............................XX....X..................X....] duration: 5.81240329s trace: https://jaeger.dev.storj.io/trace/1a181df13358808f FAIL (2025-10-31 21:06:41 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Oct 31 21:03:40.187891701 (33h23m11.883740444s) [upload=29297]: Starting upload Oct 31 21:03:40.187972845 (33h23m11.883821598s) [upload=29297]: Done starting upload Oct 31 21:03:40.188163167 (33h23m11.88401191s) [upload=29297]: Uploading segments... Oct 31 21:03:40.188240786 (33h23m11.884089559s) [upload=29297]: Got next segment. Inline: false Oct 31 21:03:40.188281714 (33h23m11.884130467s) [upload=29297]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 31 21:03:40.188288065 (33h23m11.884136808s) [upload=29297]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 31 21:03:40.439495977 (33h23m12.13534474s) [seg_pos={0 0}, upload=29297]: Begin upload segment... Oct 31 21:03:40.439552094 (33h23m12.135400847s) [seg_pos={0 0}, upload=29297]: Done begin upload segment. Oct 31 21:03:40.439606758 (33h23m12.135455511s) [noise=true , address=145.79.227.21:4473 , piece_id=XBSRRHW5MAWDHZV3YFV7BWHL7VWX7C2YWB2DTNCFQJTXE43LAAYQ, seg_pos={0 0}, upload=29297]: Uploading piece... Oct 31 21:03:40.439643338 (33h23m12.135492081s) [noise=true , address=77.90.31.15:4698 , piece_id=2SJEFFLOU7LPMQZV5DLW5Z4BUQNJFZ347K656TFXV47LTCODGA3A, seg_pos={0 0}, upload=29297]: Uploading piece... Oct 31 21:03:40.440209786 (33h23m12.136058539s) [noise=true , address=145.79.227.14:4837 , piece_id=E3NUXQAOQNOCNTMUGO2NX2ELYKMII4J37DQPUBSQ6ABGBHIC3WSA, seg_pos={0 0}, upload=29297]: Uploading piece... Oct 31 21:03:40.440607473 (33h23m12.136456216s) [noise=true , address=145.79.227.13:4802 , piece_id=GZK6DPTPN4X5D4NRJABMLW6HNPXMH7IR25UE347P2BQZYKCJY3IQ, seg_pos={0 0}, upload=29297]: Uploading piece... Oct 31 21:03:40.441017394 (33h23m12.136866147s) [noise=true , address=77.90.31.15:4699 , piece_id=DLH6733XEDD5Z7AIO7NT4UC4TMTM57RUDVUHYOYONTXTGUJWJSYA, seg_pos={0 0}, upload=29297]: Uploading piece... Oct 31 21:03:40.441619029 (33h23m12.137467772s) [noise=true , address=77.90.31.17:4772 , piece_id=BFR46U4SPW6VNCIDV3XNIEOVYWGCXMQPDZ4JCJO5EAY5VSY4X56A, seg_pos={0 0}, upload=29297]: Uploading piece... Oct 31 21:03:40.442305126 (33h23m12.138153879s) [noise=true , address=145.79.227.17:4970 , piece_id=6LJTN7QUSUSHS6DSKRHUUA3XB5ICEZ2ACFOSU4SK5HMUDRSKMEYA, seg_pos={0 0}, upload=29297]: Uploading piece... Oct 31 21:03:40.443621452 (33h23m12.139470205s) [noise=true , address=145.79.227.15:4874 , piece_id=VK3CG2KERHGJQSKLJKOT7V46GRU6XEJN7T5S4HMTW2M76E5HOUHQ, seg_pos={0 0}, upload=29297]: Uploading piece... Oct 31 21:03:40.443864906 (33h23m12.139713659s) [noise=true , address=145.79.227.11:4732 , piece_id=E4WNX7JXJZHMPKBPEJPSXJAYL4BP47HLXLO6DBHPICX6ESIK767Q, seg_pos={0 0}, upload=29297]: Uploading piece... Oct 31 21:03:40.444991701 (33h23m12.140840444s) [noise=true , address=145.79.227.24:4583 , piece_id=KSDYUXPSUXPNBT7J7Q6RW7HNPILNVUWSX7STJXDBI7YKR7MA7XMA, seg_pos={0 0}, upload=29297]: Uploading piece... Oct 31 21:03:40.445358190 (33h23m12.141206933s) [noise=true , address=145.79.227.5:4223 , piece_id=CAMTFWQE5GQIDZMJDJXSO6SIHKQXKI6HRNQAFI46JICNCIMEGACA, seg_pos={0 0}, upload=29297]: Uploading piece... Oct 31 21:03:40.446480987 (33h23m12.14232974s) [noise=true , address=145.79.227.30:4762 , piece_id=HDRX3AY2PGE2UDY4SLN4A7T27KGLZQRTOYGKHCDC4S7EDVJOIWXA, seg_pos={0 0}, upload=29297]: Uploading piece... Oct 31 21:03:40.446948418 (33h23m12.142797171s) [noise=true , address=77.90.31.10:4517 , piece_id=BDAUXG53XX42I72C6BIIC3VCLA6GZEFD3IDVSLYCC73AS2TG626Q, seg_pos={0 0}, upload=29297]: Uploading piece... Oct 31 21:03:40.448429257 (33h23m12.14427801s) [noise=true , address=145.79.227.17:4971 , piece_id=3P7N3ERAPOY62SAKEBV7CHWOWWNUPWLSLL3RZ4RWCPF3VHI3C4FQ, seg_pos={0 0}, upload=29297]: Uploading piece... Oct 31 21:03:40.449960433 (33h23m12.145809176s) [noise=true , address=77.90.31.11:4555 , piece_id=BMTMJ4QDYLWDSMNIP64YUDUBZA2DABKYYC5JK72VJSUBTBZ67YTA, seg_pos={0 0}, upload=29297]: Uploading piece... Oct 31 21:03:40.451429571 (33h23m12.147278324s) [noise=true , address=145.79.227.4:4185 , piece_id=7C536MM2MA3EC44OTMXDUGNW5YDD3OOK5ODAU2FOFM3YTQ7VU6PQ, seg_pos={0 0}, upload=29297]: Uploading piece... Oct 31 21:03:42.182232662 (33h23m13.878081416s) [upload=29297]: Next returned nil segment Oct 31 21:03:42.182251107 (33h23m13.87809985s) [upload=29297]: Waiting for error group managing segments... Oct 31 21:03:42.460580042 (33h23m14.156428785s) [noise=true , address=145.79.227.24:4583 , piece_id=KSDYUXPSUXPNBT7J7Q6RW7HNPILNVUWSX7STJXDBI7YKR7MA7XMA, seg_pos={0 0}, upload=29297]: Done uploading piece. err: <nil> Oct 31 21:03:42.470437052 (33h23m14.166285805s) [noise=true , address=145.79.227.5:4223 , piece_id=CAMTFWQE5GQIDZMJDJXSO6SIHKQXKI6HRNQAFI46JICNCIMEGACA, seg_pos={0 0}, upload=29297]: Done uploading piece. err: <nil> Oct 31 21:03:42.478522949 (33h23m14.174371692s) [noise=true , address=145.79.227.4:4185 , piece_id=7C536MM2MA3EC44OTMXDUGNW5YDD3OOK5ODAU2FOFM3YTQ7VU6PQ, seg_pos={0 0}, upload=29297]: Done uploading piece. err: <nil> Oct 31 21:03:42.479322582 (33h23m14.175171325s) [noise=true , address=145.79.227.15:4874 , piece_id=VK3CG2KERHGJQSKLJKOT7V46GRU6XEJN7T5S4HMTW2M76E5HOUHQ, seg_pos={0 0}, upload=29297]: Done uploading piece. err: <nil> Oct 31 21:03:42.479617323 (33h23m14.175466076s) [noise=true , address=145.79.227.21:4473 , piece_id=XBSRRHW5MAWDHZV3YFV7BWHL7VWX7C2YWB2DTNCFQJTXE43LAAYQ, seg_pos={0 0}, upload=29297]: Done uploading piece. err: <nil> Oct 31 21:03:42.481690871 (33h23m14.177539614s) [noise=true , address=145.79.227.30:4762 , piece_id=HDRX3AY2PGE2UDY4SLN4A7T27KGLZQRTOYGKHCDC4S7EDVJOIWXA, seg_pos={0 0}, upload=29297]: Done uploading piece. err: <nil> Oct 31 21:03:42.513679364 (33h23m14.209528107s) [noise=true , address=145.79.227.13:4802 , piece_id=GZK6DPTPN4X5D4NRJABMLW6HNPXMH7IR25UE347P2BQZYKCJY3IQ, seg_pos={0 0}, upload=29297]: Done uploading piece. err: <nil> Oct 31 21:03:42.569840746 (33h23m14.265689489s) [noise=true , address=145.79.227.14:4837 , piece_id=E3NUXQAOQNOCNTMUGO2NX2ELYKMII4J37DQPUBSQ6ABGBHIC3WSA, seg_pos={0 0}, upload=29297]: Done uploading piece. err: <nil> Oct 31 21:03:42.571444861 (33h23m14.267293614s) [noise=true , address=145.79.227.17:4970 , piece_id=6LJTN7QUSUSHS6DSKRHUUA3XB5ICEZ2ACFOSU4SK5HMUDRSKMEYA, seg_pos={0 0}, upload=29297]: Done uploading piece. err: <nil> Oct 31 21:03:42.707369620 (33h23m14.403218373s) [noise=true , address=145.79.227.11:4732 , piece_id=E4WNX7JXJZHMPKBPEJPSXJAYL4BP47HLXLO6DBHPICX6ESIK767Q, seg_pos={0 0}, upload=29297]: Done uploading piece. err: <nil> Oct 31 21:03:42.745077775 (33h23m14.440926518s) [noise=true , address=145.79.227.17:4971 , piece_id=3P7N3ERAPOY62SAKEBV7CHWOWWNUPWLSLL3RZ4RWCPF3VHI3C4FQ, seg_pos={0 0}, upload=29297]: Done uploading piece. err: <nil> Oct 31 21:04:00.023801048 (33h23m31.719649801s) [noise=true , address=77.90.31.17:4772 , piece_id=BFR46U4SPW6VNCIDV3XNIEOVYWGCXMQPDZ4JCJO5EAY5VSY4X56A, seg_pos={0 0}, upload=29297]: Done uploading piece. err: <nil> Oct 31 21:04:09.488974106 (33h23m41.184822859s) [noise=true , address=77.90.31.15:4698 , piece_id=2SJEFFLOU7LPMQZV5DLW5Z4BUQNJFZ347K656TFXV47LTCODGA3A, seg_pos={0 0}, upload=29297]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: write tcp 188.166.225.170:35474->77.90.31.15:4698: use of closed network connection; write tcp 188.166.225.170:35474->77.90.31.15:4698: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35474->77.90.31.15:4698: use of closed network connection Oct 31 21:04:09.489128110 (33h23m41.184976863s) [noise=true , address=77.90.31.10:4517 , piece_id=BDAUXG53XX42I72C6BIIC3VCLA6GZEFD3IDVSLYCC73AS2TG626Q, seg_pos={0 0}, upload=29297]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 188.166.225.170:58092->77.90.31.10:4517: use of closed network connection; write tcp 188.166.225.170:58092->77.90.31.10:4517: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58092->77.90.31.10:4517: use of closed network connection Oct 31 21:04:09.489250382 (33h23m41.185099135s) [noise=true , address=77.90.31.11:4555 , piece_id=BMTMJ4QDYLWDSMNIP64YUDUBZA2DABKYYC5JK72VJSUBTBZ67YTA, seg_pos={0 0}, upload=29297]: Done uploading piece. err: ecclient: upload failed (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, address:77.90.31.11:4555): protocol: write tcp 188.166.225.170:40898->77.90.31.11:4555: use of closed network connection; write tcp 188.166.225.170:40898->77.90.31.11:4555: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40898->77.90.31.11:4555: use of closed network connection Oct 31 21:04:09.489384528 (33h23m41.185233281s) [noise=true , address=77.90.31.15:4699 , piece_id=DLH6733XEDD5Z7AIO7NT4UC4TMTM57RUDVUHYOYONTXTGUJWJSYA, seg_pos={0 0}, upload=29297]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: write tcp 188.166.225.170:38634->77.90.31.15:4699: use of closed network connection; write tcp 188.166.225.170:38634->77.90.31.15:4699: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38634->77.90.31.15:4699: use of closed network connection Oct 31 21:04:09.489415327 (33h23m41.18526408s) [seg_pos={0 0}, upload=29297]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Oct 31 21:04:09.733073582 (33h23m41.428922335s) [upload=29297]: Done uploading segments... Oct 31 21:04:09.733121814 (33h23m41.428970557s) [upload=29297]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.812428167s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.................................X....X.....................] duration: 6.133328707s trace: https://jaeger.dev.storj.io/trace/35569b98f076d00a FAIL (2025-10-31 21:06:34 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.133355377s edge gateway us-select-1
checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect) antiblip: 1 out of 15 is failed (threshold=6) [........X......] duration: 316.469682ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/dcs-satellites trace: https://jaeger.dev.storj.io/trace/571fef006088dd70 FAIL (2025-10-31 21:06:31 UTC) orbiter warning: Get "https://www.storj.io/dcs-satellites": read tcp [2400:6180:0:d0::f70:8001]:42806->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 316.506393ms core website
checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect) antiblip: 1 out of 15 is failed (threshold=6) [.........X.....] duration: 311.868932ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/dcs-satellites trace: https://jaeger.dev.storj.io/trace/571fef006088dd70 FAIL (2025-10-31 21:05:44 UTC) orbiter warning: Get "https://www.storj.io/dcs-satellites": read tcp [2400:6180:0:d0::f70:8001]:42806->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 311.899119ms core website
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..................................X....X....................] duration: 6.434578558s trace: https://jaeger.dev.storj.io/trace/35569b98f076d00a FAIL (2025-10-31 21:05:40 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.434608022s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [...............................XX....X..................X...] duration: 5.955445411s trace: https://jaeger.dev.storj.io/trace/1a181df13358808f FAIL (2025-10-31 21:05:33 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Oct 31 21:03:40.187891701 (33h23m11.883740444s) [upload=29297]: Starting upload Oct 31 21:03:40.187972845 (33h23m11.883821598s) [upload=29297]: Done starting upload Oct 31 21:03:40.188163167 (33h23m11.88401191s) [upload=29297]: Uploading segments... Oct 31 21:03:40.188240786 (33h23m11.884089559s) [upload=29297]: Got next segment. Inline: false Oct 31 21:03:40.188281714 (33h23m11.884130467s) [upload=29297]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 31 21:03:40.188288065 (33h23m11.884136808s) [upload=29297]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 31 21:03:40.439495977 (33h23m12.13534474s) [seg_pos={0 0}, upload=29297]: Begin upload segment... Oct 31 21:03:40.439552094 (33h23m12.135400847s) [seg_pos={0 0}, upload=29297]: Done begin upload segment. Oct 31 21:03:40.439606758 (33h23m12.135455511s) [noise=true , address=145.79.227.21:4473 , piece_id=XBSRRHW5MAWDHZV3YFV7BWHL7VWX7C2YWB2DTNCFQJTXE43LAAYQ, seg_pos={0 0}, upload=29297]: Uploading piece... Oct 31 21:03:40.439643338 (33h23m12.135492081s) [noise=true , address=77.90.31.15:4698 , piece_id=2SJEFFLOU7LPMQZV5DLW5Z4BUQNJFZ347K656TFXV47LTCODGA3A, seg_pos={0 0}, upload=29297]: Uploading piece... Oct 31 21:03:40.440209786 (33h23m12.136058539s) [noise=true , address=145.79.227.14:4837 , piece_id=E3NUXQAOQNOCNTMUGO2NX2ELYKMII4J37DQPUBSQ6ABGBHIC3WSA, seg_pos={0 0}, upload=29297]: Uploading piece... Oct 31 21:03:40.440607473 (33h23m12.136456216s) [noise=true , address=145.79.227.13:4802 , piece_id=GZK6DPTPN4X5D4NRJABMLW6HNPXMH7IR25UE347P2BQZYKCJY3IQ, seg_pos={0 0}, upload=29297]: Uploading piece... Oct 31 21:03:40.441017394 (33h23m12.136866147s) [noise=true , address=77.90.31.15:4699 , piece_id=DLH6733XEDD5Z7AIO7NT4UC4TMTM57RUDVUHYOYONTXTGUJWJSYA, seg_pos={0 0}, upload=29297]: Uploading piece... Oct 31 21:03:40.441619029 (33h23m12.137467772s) [noise=true , address=77.90.31.17:4772 , piece_id=BFR46U4SPW6VNCIDV3XNIEOVYWGCXMQPDZ4JCJO5EAY5VSY4X56A, seg_pos={0 0}, upload=29297]: Uploading piece... Oct 31 21:03:40.442305126 (33h23m12.138153879s) [noise=true , address=145.79.227.17:4970 , piece_id=6LJTN7QUSUSHS6DSKRHUUA3XB5ICEZ2ACFOSU4SK5HMUDRSKMEYA, seg_pos={0 0}, upload=29297]: Uploading piece... Oct 31 21:03:40.443621452 (33h23m12.139470205s) [noise=true , address=145.79.227.15:4874 , piece_id=VK3CG2KERHGJQSKLJKOT7V46GRU6XEJN7T5S4HMTW2M76E5HOUHQ, seg_pos={0 0}, upload=29297]: Uploading piece... Oct 31 21:03:40.443864906 (33h23m12.139713659s) [noise=true , address=145.79.227.11:4732 , piece_id=E4WNX7JXJZHMPKBPEJPSXJAYL4BP47HLXLO6DBHPICX6ESIK767Q, seg_pos={0 0}, upload=29297]: Uploading piece... Oct 31 21:03:40.444991701 (33h23m12.140840444s) [noise=true , address=145.79.227.24:4583 , piece_id=KSDYUXPSUXPNBT7J7Q6RW7HNPILNVUWSX7STJXDBI7YKR7MA7XMA, seg_pos={0 0}, upload=29297]: Uploading piece... Oct 31 21:03:40.445358190 (33h23m12.141206933s) [noise=true , address=145.79.227.5:4223 , piece_id=CAMTFWQE5GQIDZMJDJXSO6SIHKQXKI6HRNQAFI46JICNCIMEGACA, seg_pos={0 0}, upload=29297]: Uploading piece... Oct 31 21:03:40.446480987 (33h23m12.14232974s) [noise=true , address=145.79.227.30:4762 , piece_id=HDRX3AY2PGE2UDY4SLN4A7T27KGLZQRTOYGKHCDC4S7EDVJOIWXA, seg_pos={0 0}, upload=29297]: Uploading piece... Oct 31 21:03:40.446948418 (33h23m12.142797171s) [noise=true , address=77.90.31.10:4517 , piece_id=BDAUXG53XX42I72C6BIIC3VCLA6GZEFD3IDVSLYCC73AS2TG626Q, seg_pos={0 0}, upload=29297]: Uploading piece... Oct 31 21:03:40.448429257 (33h23m12.14427801s) [noise=true , address=145.79.227.17:4971 , piece_id=3P7N3ERAPOY62SAKEBV7CHWOWWNUPWLSLL3RZ4RWCPF3VHI3C4FQ, seg_pos={0 0}, upload=29297]: Uploading piece... Oct 31 21:03:40.449960433 (33h23m12.145809176s) [noise=true , address=77.90.31.11:4555 , piece_id=BMTMJ4QDYLWDSMNIP64YUDUBZA2DABKYYC5JK72VJSUBTBZ67YTA, seg_pos={0 0}, upload=29297]: Uploading piece... Oct 31 21:03:40.451429571 (33h23m12.147278324s) [noise=true , address=145.79.227.4:4185 , piece_id=7C536MM2MA3EC44OTMXDUGNW5YDD3OOK5ODAU2FOFM3YTQ7VU6PQ, seg_pos={0 0}, upload=29297]: Uploading piece... Oct 31 21:03:42.182232662 (33h23m13.878081416s) [upload=29297]: Next returned nil segment Oct 31 21:03:42.182251107 (33h23m13.87809985s) [upload=29297]: Waiting for error group managing segments... Oct 31 21:03:42.460580042 (33h23m14.156428785s) [noise=true , address=145.79.227.24:4583 , piece_id=KSDYUXPSUXPNBT7J7Q6RW7HNPILNVUWSX7STJXDBI7YKR7MA7XMA, seg_pos={0 0}, upload=29297]: Done uploading piece. err: <nil> Oct 31 21:03:42.470437052 (33h23m14.166285805s) [noise=true , address=145.79.227.5:4223 , piece_id=CAMTFWQE5GQIDZMJDJXSO6SIHKQXKI6HRNQAFI46JICNCIMEGACA, seg_pos={0 0}, upload=29297]: Done uploading piece. err: <nil> Oct 31 21:03:42.478522949 (33h23m14.174371692s) [noise=true , address=145.79.227.4:4185 , piece_id=7C536MM2MA3EC44OTMXDUGNW5YDD3OOK5ODAU2FOFM3YTQ7VU6PQ, seg_pos={0 0}, upload=29297]: Done uploading piece. err: <nil> Oct 31 21:03:42.479322582 (33h23m14.175171325s) [noise=true , address=145.79.227.15:4874 , piece_id=VK3CG2KERHGJQSKLJKOT7V46GRU6XEJN7T5S4HMTW2M76E5HOUHQ, seg_pos={0 0}, upload=29297]: Done uploading piece. err: <nil> Oct 31 21:03:42.479617323 (33h23m14.175466076s) [noise=true , address=145.79.227.21:4473 , piece_id=XBSRRHW5MAWDHZV3YFV7BWHL7VWX7C2YWB2DTNCFQJTXE43LAAYQ, seg_pos={0 0}, upload=29297]: Done uploading piece. err: <nil> Oct 31 21:03:42.481690871 (33h23m14.177539614s) [noise=true , address=145.79.227.30:4762 , piece_id=HDRX3AY2PGE2UDY4SLN4A7T27KGLZQRTOYGKHCDC4S7EDVJOIWXA, seg_pos={0 0}, upload=29297]: Done uploading piece. err: <nil> Oct 31 21:03:42.513679364 (33h23m14.209528107s) [noise=true , address=145.79.227.13:4802 , piece_id=GZK6DPTPN4X5D4NRJABMLW6HNPXMH7IR25UE347P2BQZYKCJY3IQ, seg_pos={0 0}, upload=29297]: Done uploading piece. err: <nil> Oct 31 21:03:42.569840746 (33h23m14.265689489s) [noise=true , address=145.79.227.14:4837 , piece_id=E3NUXQAOQNOCNTMUGO2NX2ELYKMII4J37DQPUBSQ6ABGBHIC3WSA, seg_pos={0 0}, upload=29297]: Done uploading piece. err: <nil> Oct 31 21:03:42.571444861 (33h23m14.267293614s) [noise=true , address=145.79.227.17:4970 , piece_id=6LJTN7QUSUSHS6DSKRHUUA3XB5ICEZ2ACFOSU4SK5HMUDRSKMEYA, seg_pos={0 0}, upload=29297]: Done uploading piece. err: <nil> Oct 31 21:03:42.707369620 (33h23m14.403218373s) [noise=true , address=145.79.227.11:4732 , piece_id=E4WNX7JXJZHMPKBPEJPSXJAYL4BP47HLXLO6DBHPICX6ESIK767Q, seg_pos={0 0}, upload=29297]: Done uploading piece. err: <nil> Oct 31 21:03:42.745077775 (33h23m14.440926518s) [noise=true , address=145.79.227.17:4971 , piece_id=3P7N3ERAPOY62SAKEBV7CHWOWWNUPWLSLL3RZ4RWCPF3VHI3C4FQ, seg_pos={0 0}, upload=29297]: Done uploading piece. err: <nil> Oct 31 21:04:00.023801048 (33h23m31.719649801s) [noise=true , address=77.90.31.17:4772 , piece_id=BFR46U4SPW6VNCIDV3XNIEOVYWGCXMQPDZ4JCJO5EAY5VSY4X56A, seg_pos={0 0}, upload=29297]: Done uploading piece. err: <nil> Oct 31 21:04:09.488974106 (33h23m41.184822859s) [noise=true , address=77.90.31.15:4698 , piece_id=2SJEFFLOU7LPMQZV5DLW5Z4BUQNJFZ347K656TFXV47LTCODGA3A, seg_pos={0 0}, upload=29297]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: write tcp 188.166.225.170:35474->77.90.31.15:4698: use of closed network connection; write tcp 188.166.225.170:35474->77.90.31.15:4698: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35474->77.90.31.15:4698: use of closed network connection Oct 31 21:04:09.489128110 (33h23m41.184976863s) [noise=true , address=77.90.31.10:4517 , piece_id=BDAUXG53XX42I72C6BIIC3VCLA6GZEFD3IDVSLYCC73AS2TG626Q, seg_pos={0 0}, upload=29297]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 188.166.225.170:58092->77.90.31.10:4517: use of closed network connection; write tcp 188.166.225.170:58092->77.90.31.10:4517: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58092->77.90.31.10:4517: use of closed network connection Oct 31 21:04:09.489250382 (33h23m41.185099135s) [noise=true , address=77.90.31.11:4555 , piece_id=BMTMJ4QDYLWDSMNIP64YUDUBZA2DABKYYC5JK72VJSUBTBZ67YTA, seg_pos={0 0}, upload=29297]: Done uploading piece. err: ecclient: upload failed (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, address:77.90.31.11:4555): protocol: write tcp 188.166.225.170:40898->77.90.31.11:4555: use of closed network connection; write tcp 188.166.225.170:40898->77.90.31.11:4555: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40898->77.90.31.11:4555: use of closed network connection Oct 31 21:04:09.489384528 (33h23m41.185233281s) [noise=true , address=77.90.31.15:4699 , piece_id=DLH6733XEDD5Z7AIO7NT4UC4TMTM57RUDVUHYOYONTXTGUJWJSYA, seg_pos={0 0}, upload=29297]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: write tcp 188.166.225.170:38634->77.90.31.15:4699: use of closed network connection; write tcp 188.166.225.170:38634->77.90.31.15:4699: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38634->77.90.31.15:4699: use of closed network connection Oct 31 21:04:09.489415327 (33h23m41.18526408s) [seg_pos={0 0}, upload=29297]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Oct 31 21:04:09.733073582 (33h23m41.428922335s) [upload=29297]: Done uploading segments... Oct 31 21:04:09.733121814 (33h23m41.428970557s) [upload=29297]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.955477081s core uplink us-select-1
checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect) antiblip: 1 out of 15 is failed (threshold=6) [..........X....] duration: 305.512087ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/dcs-satellites trace: https://jaeger.dev.storj.io/trace/571fef006088dd70 FAIL (2025-10-31 21:05:19 UTC) orbiter warning: Get "https://www.storj.io/dcs-satellites": read tcp [2400:6180:0:d0::f70:8001]:42806->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 305.542745ms core website
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...................................X....X...................] duration: 6.150934127s trace: https://jaeger.dev.storj.io/trace/35569b98f076d00a FAIL (2025-10-31 21:05:15 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.151015943s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [................................XX....X..................X..] duration: 6.091447135s trace: https://jaeger.dev.storj.io/trace/1a181df13358808f FAIL (2025-10-31 21:04:59 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Oct 31 21:03:40.187891701 (33h23m11.883740444s) [upload=29297]: Starting upload Oct 31 21:03:40.187972845 (33h23m11.883821598s) [upload=29297]: Done starting upload Oct 31 21:03:40.188163167 (33h23m11.88401191s) [upload=29297]: Uploading segments... Oct 31 21:03:40.188240786 (33h23m11.884089559s) [upload=29297]: Got next segment. Inline: false Oct 31 21:03:40.188281714 (33h23m11.884130467s) [upload=29297]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 31 21:03:40.188288065 (33h23m11.884136808s) [upload=29297]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 31 21:03:40.439495977 (33h23m12.13534474s) [seg_pos={0 0}, upload=29297]: Begin upload segment... Oct 31 21:03:40.439552094 (33h23m12.135400847s) [seg_pos={0 0}, upload=29297]: Done begin upload segment. Oct 31 21:03:40.439606758 (33h23m12.135455511s) [noise=true , address=145.79.227.21:4473 , piece_id=XBSRRHW5MAWDHZV3YFV7BWHL7VWX7C2YWB2DTNCFQJTXE43LAAYQ, seg_pos={0 0}, upload=29297]: Uploading piece... Oct 31 21:03:40.439643338 (33h23m12.135492081s) [noise=true , address=77.90.31.15:4698 , piece_id=2SJEFFLOU7LPMQZV5DLW5Z4BUQNJFZ347K656TFXV47LTCODGA3A, seg_pos={0 0}, upload=29297]: Uploading piece... Oct 31 21:03:40.440209786 (33h23m12.136058539s) [noise=true , address=145.79.227.14:4837 , piece_id=E3NUXQAOQNOCNTMUGO2NX2ELYKMII4J37DQPUBSQ6ABGBHIC3WSA, seg_pos={0 0}, upload=29297]: Uploading piece... Oct 31 21:03:40.440607473 (33h23m12.136456216s) [noise=true , address=145.79.227.13:4802 , piece_id=GZK6DPTPN4X5D4NRJABMLW6HNPXMH7IR25UE347P2BQZYKCJY3IQ, seg_pos={0 0}, upload=29297]: Uploading piece... Oct 31 21:03:40.441017394 (33h23m12.136866147s) [noise=true , address=77.90.31.15:4699 , piece_id=DLH6733XEDD5Z7AIO7NT4UC4TMTM57RUDVUHYOYONTXTGUJWJSYA, seg_pos={0 0}, upload=29297]: Uploading piece... Oct 31 21:03:40.441619029 (33h23m12.137467772s) [noise=true , address=77.90.31.17:4772 , piece_id=BFR46U4SPW6VNCIDV3XNIEOVYWGCXMQPDZ4JCJO5EAY5VSY4X56A, seg_pos={0 0}, upload=29297]: Uploading piece... Oct 31 21:03:40.442305126 (33h23m12.138153879s) [noise=true , address=145.79.227.17:4970 , piece_id=6LJTN7QUSUSHS6DSKRHUUA3XB5ICEZ2ACFOSU4SK5HMUDRSKMEYA, seg_pos={0 0}, upload=29297]: Uploading piece... Oct 31 21:03:40.443621452 (33h23m12.139470205s) [noise=true , address=145.79.227.15:4874 , piece_id=VK3CG2KERHGJQSKLJKOT7V46GRU6XEJN7T5S4HMTW2M76E5HOUHQ, seg_pos={0 0}, upload=29297]: Uploading piece... Oct 31 21:03:40.443864906 (33h23m12.139713659s) [noise=true , address=145.79.227.11:4732 , piece_id=E4WNX7JXJZHMPKBPEJPSXJAYL4BP47HLXLO6DBHPICX6ESIK767Q, seg_pos={0 0}, upload=29297]: Uploading piece... Oct 31 21:03:40.444991701 (33h23m12.140840444s) [noise=true , address=145.79.227.24:4583 , piece_id=KSDYUXPSUXPNBT7J7Q6RW7HNPILNVUWSX7STJXDBI7YKR7MA7XMA, seg_pos={0 0}, upload=29297]: Uploading piece... Oct 31 21:03:40.445358190 (33h23m12.141206933s) [noise=true , address=145.79.227.5:4223 , piece_id=CAMTFWQE5GQIDZMJDJXSO6SIHKQXKI6HRNQAFI46JICNCIMEGACA, seg_pos={0 0}, upload=29297]: Uploading piece... Oct 31 21:03:40.446480987 (33h23m12.14232974s) [noise=true , address=145.79.227.30:4762 , piece_id=HDRX3AY2PGE2UDY4SLN4A7T27KGLZQRTOYGKHCDC4S7EDVJOIWXA, seg_pos={0 0}, upload=29297]: Uploading piece... Oct 31 21:03:40.446948418 (33h23m12.142797171s) [noise=true , address=77.90.31.10:4517 , piece_id=BDAUXG53XX42I72C6BIIC3VCLA6GZEFD3IDVSLYCC73AS2TG626Q, seg_pos={0 0}, upload=29297]: Uploading piece... Oct 31 21:03:40.448429257 (33h23m12.14427801s) [noise=true , address=145.79.227.17:4971 , piece_id=3P7N3ERAPOY62SAKEBV7CHWOWWNUPWLSLL3RZ4RWCPF3VHI3C4FQ, seg_pos={0 0}, upload=29297]: Uploading piece... Oct 31 21:03:40.449960433 (33h23m12.145809176s) [noise=true , address=77.90.31.11:4555 , piece_id=BMTMJ4QDYLWDSMNIP64YUDUBZA2DABKYYC5JK72VJSUBTBZ67YTA, seg_pos={0 0}, upload=29297]: Uploading piece... Oct 31 21:03:40.451429571 (33h23m12.147278324s) [noise=true , address=145.79.227.4:4185 , piece_id=7C536MM2MA3EC44OTMXDUGNW5YDD3OOK5ODAU2FOFM3YTQ7VU6PQ, seg_pos={0 0}, upload=29297]: Uploading piece... Oct 31 21:03:42.182232662 (33h23m13.878081416s) [upload=29297]: Next returned nil segment Oct 31 21:03:42.182251107 (33h23m13.87809985s) [upload=29297]: Waiting for error group managing segments... Oct 31 21:03:42.460580042 (33h23m14.156428785s) [noise=true , address=145.79.227.24:4583 , piece_id=KSDYUXPSUXPNBT7J7Q6RW7HNPILNVUWSX7STJXDBI7YKR7MA7XMA, seg_pos={0 0}, upload=29297]: Done uploading piece. err: <nil> Oct 31 21:03:42.470437052 (33h23m14.166285805s) [noise=true , address=145.79.227.5:4223 , piece_id=CAMTFWQE5GQIDZMJDJXSO6SIHKQXKI6HRNQAFI46JICNCIMEGACA, seg_pos={0 0}, upload=29297]: Done uploading piece. err: <nil> Oct 31 21:03:42.478522949 (33h23m14.174371692s) [noise=true , address=145.79.227.4:4185 , piece_id=7C536MM2MA3EC44OTMXDUGNW5YDD3OOK5ODAU2FOFM3YTQ7VU6PQ, seg_pos={0 0}, upload=29297]: Done uploading piece. err: <nil> Oct 31 21:03:42.479322582 (33h23m14.175171325s) [noise=true , address=145.79.227.15:4874 , piece_id=VK3CG2KERHGJQSKLJKOT7V46GRU6XEJN7T5S4HMTW2M76E5HOUHQ, seg_pos={0 0}, upload=29297]: Done uploading piece. err: <nil> Oct 31 21:03:42.479617323 (33h23m14.175466076s) [noise=true , address=145.79.227.21:4473 , piece_id=XBSRRHW5MAWDHZV3YFV7BWHL7VWX7C2YWB2DTNCFQJTXE43LAAYQ, seg_pos={0 0}, upload=29297]: Done uploading piece. err: <nil> Oct 31 21:03:42.481690871 (33h23m14.177539614s) [noise=true , address=145.79.227.30:4762 , piece_id=HDRX3AY2PGE2UDY4SLN4A7T27KGLZQRTOYGKHCDC4S7EDVJOIWXA, seg_pos={0 0}, upload=29297]: Done uploading piece. err: <nil> Oct 31 21:03:42.513679364 (33h23m14.209528107s) [noise=true , address=145.79.227.13:4802 , piece_id=GZK6DPTPN4X5D4NRJABMLW6HNPXMH7IR25UE347P2BQZYKCJY3IQ, seg_pos={0 0}, upload=29297]: Done uploading piece. err: <nil> Oct 31 21:03:42.569840746 (33h23m14.265689489s) [noise=true , address=145.79.227.14:4837 , piece_id=E3NUXQAOQNOCNTMUGO2NX2ELYKMII4J37DQPUBSQ6ABGBHIC3WSA, seg_pos={0 0}, upload=29297]: Done uploading piece. err: <nil> Oct 31 21:03:42.571444861 (33h23m14.267293614s) [noise=true , address=145.79.227.17:4970 , piece_id=6LJTN7QUSUSHS6DSKRHUUA3XB5ICEZ2ACFOSU4SK5HMUDRSKMEYA, seg_pos={0 0}, upload=29297]: Done uploading piece. err: <nil> Oct 31 21:03:42.707369620 (33h23m14.403218373s) [noise=true , address=145.79.227.11:4732 , piece_id=E4WNX7JXJZHMPKBPEJPSXJAYL4BP47HLXLO6DBHPICX6ESIK767Q, seg_pos={0 0}, upload=29297]: Done uploading piece. err: <nil> Oct 31 21:03:42.745077775 (33h23m14.440926518s) [noise=true , address=145.79.227.17:4971 , piece_id=3P7N3ERAPOY62SAKEBV7CHWOWWNUPWLSLL3RZ4RWCPF3VHI3C4FQ, seg_pos={0 0}, upload=29297]: Done uploading piece. err: <nil> Oct 31 21:04:00.023801048 (33h23m31.719649801s) [noise=true , address=77.90.31.17:4772 , piece_id=BFR46U4SPW6VNCIDV3XNIEOVYWGCXMQPDZ4JCJO5EAY5VSY4X56A, seg_pos={0 0}, upload=29297]: Done uploading piece. err: <nil> Oct 31 21:04:09.488974106 (33h23m41.184822859s) [noise=true , address=77.90.31.15:4698 , piece_id=2SJEFFLOU7LPMQZV5DLW5Z4BUQNJFZ347K656TFXV47LTCODGA3A, seg_pos={0 0}, upload=29297]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: write tcp 188.166.225.170:35474->77.90.31.15:4698: use of closed network connection; write tcp 188.166.225.170:35474->77.90.31.15:4698: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35474->77.90.31.15:4698: use of closed network connection Oct 31 21:04:09.489128110 (33h23m41.184976863s) [noise=true , address=77.90.31.10:4517 , piece_id=BDAUXG53XX42I72C6BIIC3VCLA6GZEFD3IDVSLYCC73AS2TG626Q, seg_pos={0 0}, upload=29297]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 188.166.225.170:58092->77.90.31.10:4517: use of closed network connection; write tcp 188.166.225.170:58092->77.90.31.10:4517: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58092->77.90.31.10:4517: use of closed network connection Oct 31 21:04:09.489250382 (33h23m41.185099135s) [noise=true , address=77.90.31.11:4555 , piece_id=BMTMJ4QDYLWDSMNIP64YUDUBZA2DABKYYC5JK72VJSUBTBZ67YTA, seg_pos={0 0}, upload=29297]: Done uploading piece. err: ecclient: upload failed (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, address:77.90.31.11:4555): protocol: write tcp 188.166.225.170:40898->77.90.31.11:4555: use of closed network connection; write tcp 188.166.225.170:40898->77.90.31.11:4555: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40898->77.90.31.11:4555: use of closed network connection Oct 31 21:04:09.489384528 (33h23m41.185233281s) [noise=true , address=77.90.31.15:4699 , piece_id=DLH6733XEDD5Z7AIO7NT4UC4TMTM57RUDVUHYOYONTXTGUJWJSYA, seg_pos={0 0}, upload=29297]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: write tcp 188.166.225.170:38634->77.90.31.15:4699: use of closed network connection; write tcp 188.166.225.170:38634->77.90.31.15:4699: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38634->77.90.31.15:4699: use of closed network connection Oct 31 21:04:09.489415327 (33h23m41.18526408s) [seg_pos={0 0}, upload=29297]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Oct 31 21:04:09.733073582 (33h23m41.428922335s) [upload=29297]: Done uploading segments... Oct 31 21:04:09.733121814 (33h23m41.428970557s) [upload=29297]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.091475328s core uplink us-select-1
checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect) antiblip: 1 out of 15 is failed (threshold=6) [...........X...] duration: 310.752046ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/dcs-satellites trace: https://jaeger.dev.storj.io/trace/571fef006088dd70 FAIL (2025-10-31 21:04:38 UTC) orbiter warning: Get "https://www.storj.io/dcs-satellites": read tcp [2400:6180:0:d0::f70:8001]:42806->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 310.779708ms core website
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.................................XX....X..................X.] duration: 6.101229722s trace: https://jaeger.dev.storj.io/trace/1a181df13358808f FAIL (2025-10-31 21:04:16 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Oct 31 21:03:40.187891701 (33h23m11.883740444s) [upload=29297]: Starting upload Oct 31 21:03:40.187972845 (33h23m11.883821598s) [upload=29297]: Done starting upload Oct 31 21:03:40.188163167 (33h23m11.88401191s) [upload=29297]: Uploading segments... Oct 31 21:03:40.188240786 (33h23m11.884089559s) [upload=29297]: Got next segment. Inline: false Oct 31 21:03:40.188281714 (33h23m11.884130467s) [upload=29297]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 31 21:03:40.188288065 (33h23m11.884136808s) [upload=29297]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 31 21:03:40.439495977 (33h23m12.13534474s) [seg_pos={0 0}, upload=29297]: Begin upload segment... Oct 31 21:03:40.439552094 (33h23m12.135400847s) [seg_pos={0 0}, upload=29297]: Done begin upload segment. Oct 31 21:03:40.439606758 (33h23m12.135455511s) [noise=true , address=145.79.227.21:4473 , piece_id=XBSRRHW5MAWDHZV3YFV7BWHL7VWX7C2YWB2DTNCFQJTXE43LAAYQ, seg_pos={0 0}, upload=29297]: Uploading piece... Oct 31 21:03:40.439643338 (33h23m12.135492081s) [noise=true , address=77.90.31.15:4698 , piece_id=2SJEFFLOU7LPMQZV5DLW5Z4BUQNJFZ347K656TFXV47LTCODGA3A, seg_pos={0 0}, upload=29297]: Uploading piece... Oct 31 21:03:40.440209786 (33h23m12.136058539s) [noise=true , address=145.79.227.14:4837 , piece_id=E3NUXQAOQNOCNTMUGO2NX2ELYKMII4J37DQPUBSQ6ABGBHIC3WSA, seg_pos={0 0}, upload=29297]: Uploading piece... Oct 31 21:03:40.440607473 (33h23m12.136456216s) [noise=true , address=145.79.227.13:4802 , piece_id=GZK6DPTPN4X5D4NRJABMLW6HNPXMH7IR25UE347P2BQZYKCJY3IQ, seg_pos={0 0}, upload=29297]: Uploading piece... Oct 31 21:03:40.441017394 (33h23m12.136866147s) [noise=true , address=77.90.31.15:4699 , piece_id=DLH6733XEDD5Z7AIO7NT4UC4TMTM57RUDVUHYOYONTXTGUJWJSYA, seg_pos={0 0}, upload=29297]: Uploading piece... Oct 31 21:03:40.441619029 (33h23m12.137467772s) [noise=true , address=77.90.31.17:4772 , piece_id=BFR46U4SPW6VNCIDV3XNIEOVYWGCXMQPDZ4JCJO5EAY5VSY4X56A, seg_pos={0 0}, upload=29297]: Uploading piece... Oct 31 21:03:40.442305126 (33h23m12.138153879s) [noise=true , address=145.79.227.17:4970 , piece_id=6LJTN7QUSUSHS6DSKRHUUA3XB5ICEZ2ACFOSU4SK5HMUDRSKMEYA, seg_pos={0 0}, upload=29297]: Uploading piece... Oct 31 21:03:40.443621452 (33h23m12.139470205s) [noise=true , address=145.79.227.15:4874 , piece_id=VK3CG2KERHGJQSKLJKOT7V46GRU6XEJN7T5S4HMTW2M76E5HOUHQ, seg_pos={0 0}, upload=29297]: Uploading piece... Oct 31 21:03:40.443864906 (33h23m12.139713659s) [noise=true , address=145.79.227.11:4732 , piece_id=E4WNX7JXJZHMPKBPEJPSXJAYL4BP47HLXLO6DBHPICX6ESIK767Q, seg_pos={0 0}, upload=29297]: Uploading piece... Oct 31 21:03:40.444991701 (33h23m12.140840444s) [noise=true , address=145.79.227.24:4583 , piece_id=KSDYUXPSUXPNBT7J7Q6RW7HNPILNVUWSX7STJXDBI7YKR7MA7XMA, seg_pos={0 0}, upload=29297]: Uploading piece... Oct 31 21:03:40.445358190 (33h23m12.141206933s) [noise=true , address=145.79.227.5:4223 , piece_id=CAMTFWQE5GQIDZMJDJXSO6SIHKQXKI6HRNQAFI46JICNCIMEGACA, seg_pos={0 0}, upload=29297]: Uploading piece... Oct 31 21:03:40.446480987 (33h23m12.14232974s) [noise=true , address=145.79.227.30:4762 , piece_id=HDRX3AY2PGE2UDY4SLN4A7T27KGLZQRTOYGKHCDC4S7EDVJOIWXA, seg_pos={0 0}, upload=29297]: Uploading piece... Oct 31 21:03:40.446948418 (33h23m12.142797171s) [noise=true , address=77.90.31.10:4517 , piece_id=BDAUXG53XX42I72C6BIIC3VCLA6GZEFD3IDVSLYCC73AS2TG626Q, seg_pos={0 0}, upload=29297]: Uploading piece... Oct 31 21:03:40.448429257 (33h23m12.14427801s) [noise=true , address=145.79.227.17:4971 , piece_id=3P7N3ERAPOY62SAKEBV7CHWOWWNUPWLSLL3RZ4RWCPF3VHI3C4FQ, seg_pos={0 0}, upload=29297]: Uploading piece... Oct 31 21:03:40.449960433 (33h23m12.145809176s) [noise=true , address=77.90.31.11:4555 , piece_id=BMTMJ4QDYLWDSMNIP64YUDUBZA2DABKYYC5JK72VJSUBTBZ67YTA, seg_pos={0 0}, upload=29297]: Uploading piece... Oct 31 21:03:40.451429571 (33h23m12.147278324s) [noise=true , address=145.79.227.4:4185 , piece_id=7C536MM2MA3EC44OTMXDUGNW5YDD3OOK5ODAU2FOFM3YTQ7VU6PQ, seg_pos={0 0}, upload=29297]: Uploading piece... Oct 31 21:03:42.182232662 (33h23m13.878081416s) [upload=29297]: Next returned nil segment Oct 31 21:03:42.182251107 (33h23m13.87809985s) [upload=29297]: Waiting for error group managing segments... Oct 31 21:03:42.460580042 (33h23m14.156428785s) [noise=true , address=145.79.227.24:4583 , piece_id=KSDYUXPSUXPNBT7J7Q6RW7HNPILNVUWSX7STJXDBI7YKR7MA7XMA, seg_pos={0 0}, upload=29297]: Done uploading piece. err: <nil> Oct 31 21:03:42.470437052 (33h23m14.166285805s) [noise=true , address=145.79.227.5:4223 , piece_id=CAMTFWQE5GQIDZMJDJXSO6SIHKQXKI6HRNQAFI46JICNCIMEGACA, seg_pos={0 0}, upload=29297]: Done uploading piece. err: <nil> Oct 31 21:03:42.478522949 (33h23m14.174371692s) [noise=true , address=145.79.227.4:4185 , piece_id=7C536MM2MA3EC44OTMXDUGNW5YDD3OOK5ODAU2FOFM3YTQ7VU6PQ, seg_pos={0 0}, upload=29297]: Done uploading piece. err: <nil> Oct 31 21:03:42.479322582 (33h23m14.175171325s) [noise=true , address=145.79.227.15:4874 , piece_id=VK3CG2KERHGJQSKLJKOT7V46GRU6XEJN7T5S4HMTW2M76E5HOUHQ, seg_pos={0 0}, upload=29297]: Done uploading piece. err: <nil> Oct 31 21:03:42.479617323 (33h23m14.175466076s) [noise=true , address=145.79.227.21:4473 , piece_id=XBSRRHW5MAWDHZV3YFV7BWHL7VWX7C2YWB2DTNCFQJTXE43LAAYQ, seg_pos={0 0}, upload=29297]: Done uploading piece. err: <nil> Oct 31 21:03:42.481690871 (33h23m14.177539614s) [noise=true , address=145.79.227.30:4762 , piece_id=HDRX3AY2PGE2UDY4SLN4A7T27KGLZQRTOYGKHCDC4S7EDVJOIWXA, seg_pos={0 0}, upload=29297]: Done uploading piece. err: <nil> Oct 31 21:03:42.513679364 (33h23m14.209528107s) [noise=true , address=145.79.227.13:4802 , piece_id=GZK6DPTPN4X5D4NRJABMLW6HNPXMH7IR25UE347P2BQZYKCJY3IQ, seg_pos={0 0}, upload=29297]: Done uploading piece. err: <nil> Oct 31 21:03:42.569840746 (33h23m14.265689489s) [noise=true , address=145.79.227.14:4837 , piece_id=E3NUXQAOQNOCNTMUGO2NX2ELYKMII4J37DQPUBSQ6ABGBHIC3WSA, seg_pos={0 0}, upload=29297]: Done uploading piece. err: <nil> Oct 31 21:03:42.571444861 (33h23m14.267293614s) [noise=true , address=145.79.227.17:4970 , piece_id=6LJTN7QUSUSHS6DSKRHUUA3XB5ICEZ2ACFOSU4SK5HMUDRSKMEYA, seg_pos={0 0}, upload=29297]: Done uploading piece. err: <nil> Oct 31 21:03:42.707369620 (33h23m14.403218373s) [noise=true , address=145.79.227.11:4732 , piece_id=E4WNX7JXJZHMPKBPEJPSXJAYL4BP47HLXLO6DBHPICX6ESIK767Q, seg_pos={0 0}, upload=29297]: Done uploading piece. err: <nil> Oct 31 21:03:42.745077775 (33h23m14.440926518s) [noise=true , address=145.79.227.17:4971 , piece_id=3P7N3ERAPOY62SAKEBV7CHWOWWNUPWLSLL3RZ4RWCPF3VHI3C4FQ, seg_pos={0 0}, upload=29297]: Done uploading piece. err: <nil> Oct 31 21:04:00.023801048 (33h23m31.719649801s) [noise=true , address=77.90.31.17:4772 , piece_id=BFR46U4SPW6VNCIDV3XNIEOVYWGCXMQPDZ4JCJO5EAY5VSY4X56A, seg_pos={0 0}, upload=29297]: Done uploading piece. err: <nil> Oct 31 21:04:09.488974106 (33h23m41.184822859s) [noise=true , address=77.90.31.15:4698 , piece_id=2SJEFFLOU7LPMQZV5DLW5Z4BUQNJFZ347K656TFXV47LTCODGA3A, seg_pos={0 0}, upload=29297]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: write tcp 188.166.225.170:35474->77.90.31.15:4698: use of closed network connection; write tcp 188.166.225.170:35474->77.90.31.15:4698: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35474->77.90.31.15:4698: use of closed network connection Oct 31 21:04:09.489128110 (33h23m41.184976863s) [noise=true , address=77.90.31.10:4517 , piece_id=BDAUXG53XX42I72C6BIIC3VCLA6GZEFD3IDVSLYCC73AS2TG626Q, seg_pos={0 0}, upload=29297]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 188.166.225.170:58092->77.90.31.10:4517: use of closed network connection; write tcp 188.166.225.170:58092->77.90.31.10:4517: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58092->77.90.31.10:4517: use of closed network connection Oct 31 21:04:09.489250382 (33h23m41.185099135s) [noise=true , address=77.90.31.11:4555 , piece_id=BMTMJ4QDYLWDSMNIP64YUDUBZA2DABKYYC5JK72VJSUBTBZ67YTA, seg_pos={0 0}, upload=29297]: Done uploading piece. err: ecclient: upload failed (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, address:77.90.31.11:4555): protocol: write tcp 188.166.225.170:40898->77.90.31.11:4555: use of closed network connection; write tcp 188.166.225.170:40898->77.90.31.11:4555: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40898->77.90.31.11:4555: use of closed network connection Oct 31 21:04:09.489384528 (33h23m41.185233281s) [noise=true , address=77.90.31.15:4699 , piece_id=DLH6733XEDD5Z7AIO7NT4UC4TMTM57RUDVUHYOYONTXTGUJWJSYA, seg_pos={0 0}, upload=29297]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: write tcp 188.166.225.170:38634->77.90.31.15:4699: use of closed network connection; write tcp 188.166.225.170:38634->77.90.31.15:4699: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38634->77.90.31.15:4699: use of closed network connection Oct 31 21:04:09.489415327 (33h23m41.18526408s) [seg_pos={0 0}, upload=29297]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Oct 31 21:04:09.733073582 (33h23m41.428922335s) [upload=29297]: Done uploading segments... Oct 31 21:04:09.733121814 (33h23m41.428970557s) [upload=29297]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.101248439s core uplink us-select-1
checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect) antiblip: 1 out of 15 is failed (threshold=6) [............X..] duration: 303.908754ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/dcs-satellites trace: https://jaeger.dev.storj.io/trace/571fef006088dd70 FAIL (2025-10-31 21:04:16 UTC) orbiter warning: Get "https://www.storj.io/dcs-satellites": read tcp [2400:6180:0:d0::f70:8001]:42806->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 303.93357ms core website
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [..................................XX....X..................X] duration: 30.246208208s trace: https://jaeger.dev.storj.io/trace/1a181df13358808f FAIL (2025-10-31 21:04:09 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Oct 31 21:03:40.187891701 (33h23m11.883740444s) [upload=29297]: Starting upload Oct 31 21:03:40.187972845 (33h23m11.883821598s) [upload=29297]: Done starting upload Oct 31 21:03:40.188163167 (33h23m11.88401191s) [upload=29297]: Uploading segments... Oct 31 21:03:40.188240786 (33h23m11.884089559s) [upload=29297]: Got next segment. Inline: false Oct 31 21:03:40.188281714 (33h23m11.884130467s) [upload=29297]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 31 21:03:40.188288065 (33h23m11.884136808s) [upload=29297]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 31 21:03:40.439495977 (33h23m12.13534474s) [seg_pos={0 0}, upload=29297]: Begin upload segment... Oct 31 21:03:40.439552094 (33h23m12.135400847s) [seg_pos={0 0}, upload=29297]: Done begin upload segment. Oct 31 21:03:40.439606758 (33h23m12.135455511s) [noise=true , address=145.79.227.21:4473 , piece_id=XBSRRHW5MAWDHZV3YFV7BWHL7VWX7C2YWB2DTNCFQJTXE43LAAYQ, seg_pos={0 0}, upload=29297]: Uploading piece... Oct 31 21:03:40.439643338 (33h23m12.135492081s) [noise=true , address=77.90.31.15:4698 , piece_id=2SJEFFLOU7LPMQZV5DLW5Z4BUQNJFZ347K656TFXV47LTCODGA3A, seg_pos={0 0}, upload=29297]: Uploading piece... Oct 31 21:03:40.440209786 (33h23m12.136058539s) [noise=true , address=145.79.227.14:4837 , piece_id=E3NUXQAOQNOCNTMUGO2NX2ELYKMII4J37DQPUBSQ6ABGBHIC3WSA, seg_pos={0 0}, upload=29297]: Uploading piece... Oct 31 21:03:40.440607473 (33h23m12.136456216s) [noise=true , address=145.79.227.13:4802 , piece_id=GZK6DPTPN4X5D4NRJABMLW6HNPXMH7IR25UE347P2BQZYKCJY3IQ, seg_pos={0 0}, upload=29297]: Uploading piece... Oct 31 21:03:40.441017394 (33h23m12.136866147s) [noise=true , address=77.90.31.15:4699 , piece_id=DLH6733XEDD5Z7AIO7NT4UC4TMTM57RUDVUHYOYONTXTGUJWJSYA, seg_pos={0 0}, upload=29297]: Uploading piece... Oct 31 21:03:40.441619029 (33h23m12.137467772s) [noise=true , address=77.90.31.17:4772 , piece_id=BFR46U4SPW6VNCIDV3XNIEOVYWGCXMQPDZ4JCJO5EAY5VSY4X56A, seg_pos={0 0}, upload=29297]: Uploading piece... Oct 31 21:03:40.442305126 (33h23m12.138153879s) [noise=true , address=145.79.227.17:4970 , piece_id=6LJTN7QUSUSHS6DSKRHUUA3XB5ICEZ2ACFOSU4SK5HMUDRSKMEYA, seg_pos={0 0}, upload=29297]: Uploading piece... Oct 31 21:03:40.443621452 (33h23m12.139470205s) [noise=true , address=145.79.227.15:4874 , piece_id=VK3CG2KERHGJQSKLJKOT7V46GRU6XEJN7T5S4HMTW2M76E5HOUHQ, seg_pos={0 0}, upload=29297]: Uploading piece... Oct 31 21:03:40.443864906 (33h23m12.139713659s) [noise=true , address=145.79.227.11:4732 , piece_id=E4WNX7JXJZHMPKBPEJPSXJAYL4BP47HLXLO6DBHPICX6ESIK767Q, seg_pos={0 0}, upload=29297]: Uploading piece... Oct 31 21:03:40.444991701 (33h23m12.140840444s) [noise=true , address=145.79.227.24:4583 , piece_id=KSDYUXPSUXPNBT7J7Q6RW7HNPILNVUWSX7STJXDBI7YKR7MA7XMA, seg_pos={0 0}, upload=29297]: Uploading piece... Oct 31 21:03:40.445358190 (33h23m12.141206933s) [noise=true , address=145.79.227.5:4223 , piece_id=CAMTFWQE5GQIDZMJDJXSO6SIHKQXKI6HRNQAFI46JICNCIMEGACA, seg_pos={0 0}, upload=29297]: Uploading piece... Oct 31 21:03:40.446480987 (33h23m12.14232974s) [noise=true , address=145.79.227.30:4762 , piece_id=HDRX3AY2PGE2UDY4SLN4A7T27KGLZQRTOYGKHCDC4S7EDVJOIWXA, seg_pos={0 0}, upload=29297]: Uploading piece... Oct 31 21:03:40.446948418 (33h23m12.142797171s) [noise=true , address=77.90.31.10:4517 , piece_id=BDAUXG53XX42I72C6BIIC3VCLA6GZEFD3IDVSLYCC73AS2TG626Q, seg_pos={0 0}, upload=29297]: Uploading piece... Oct 31 21:03:40.448429257 (33h23m12.14427801s) [noise=true , address=145.79.227.17:4971 , piece_id=3P7N3ERAPOY62SAKEBV7CHWOWWNUPWLSLL3RZ4RWCPF3VHI3C4FQ, seg_pos={0 0}, upload=29297]: Uploading piece... Oct 31 21:03:40.449960433 (33h23m12.145809176s) [noise=true , address=77.90.31.11:4555 , piece_id=BMTMJ4QDYLWDSMNIP64YUDUBZA2DABKYYC5JK72VJSUBTBZ67YTA, seg_pos={0 0}, upload=29297]: Uploading piece... Oct 31 21:03:40.451429571 (33h23m12.147278324s) [noise=true , address=145.79.227.4:4185 , piece_id=7C536MM2MA3EC44OTMXDUGNW5YDD3OOK5ODAU2FOFM3YTQ7VU6PQ, seg_pos={0 0}, upload=29297]: Uploading piece... Oct 31 21:03:42.182232662 (33h23m13.878081416s) [upload=29297]: Next returned nil segment Oct 31 21:03:42.182251107 (33h23m13.87809985s) [upload=29297]: Waiting for error group managing segments... Oct 31 21:03:42.460580042 (33h23m14.156428785s) [noise=true , address=145.79.227.24:4583 , piece_id=KSDYUXPSUXPNBT7J7Q6RW7HNPILNVUWSX7STJXDBI7YKR7MA7XMA, seg_pos={0 0}, upload=29297]: Done uploading piece. err: <nil> Oct 31 21:03:42.470437052 (33h23m14.166285805s) [noise=true , address=145.79.227.5:4223 , piece_id=CAMTFWQE5GQIDZMJDJXSO6SIHKQXKI6HRNQAFI46JICNCIMEGACA, seg_pos={0 0}, upload=29297]: Done uploading piece. err: <nil> Oct 31 21:03:42.478522949 (33h23m14.174371692s) [noise=true , address=145.79.227.4:4185 , piece_id=7C536MM2MA3EC44OTMXDUGNW5YDD3OOK5ODAU2FOFM3YTQ7VU6PQ, seg_pos={0 0}, upload=29297]: Done uploading piece. err: <nil> Oct 31 21:03:42.479322582 (33h23m14.175171325s) [noise=true , address=145.79.227.15:4874 , piece_id=VK3CG2KERHGJQSKLJKOT7V46GRU6XEJN7T5S4HMTW2M76E5HOUHQ, seg_pos={0 0}, upload=29297]: Done uploading piece. err: <nil> Oct 31 21:03:42.479617323 (33h23m14.175466076s) [noise=true , address=145.79.227.21:4473 , piece_id=XBSRRHW5MAWDHZV3YFV7BWHL7VWX7C2YWB2DTNCFQJTXE43LAAYQ, seg_pos={0 0}, upload=29297]: Done uploading piece. err: <nil> Oct 31 21:03:42.481690871 (33h23m14.177539614s) [noise=true , address=145.79.227.30:4762 , piece_id=HDRX3AY2PGE2UDY4SLN4A7T27KGLZQRTOYGKHCDC4S7EDVJOIWXA, seg_pos={0 0}, upload=29297]: Done uploading piece. err: <nil> Oct 31 21:03:42.513679364 (33h23m14.209528107s) [noise=true , address=145.79.227.13:4802 , piece_id=GZK6DPTPN4X5D4NRJABMLW6HNPXMH7IR25UE347P2BQZYKCJY3IQ, seg_pos={0 0}, upload=29297]: Done uploading piece. err: <nil> Oct 31 21:03:42.569840746 (33h23m14.265689489s) [noise=true , address=145.79.227.14:4837 , piece_id=E3NUXQAOQNOCNTMUGO2NX2ELYKMII4J37DQPUBSQ6ABGBHIC3WSA, seg_pos={0 0}, upload=29297]: Done uploading piece. err: <nil> Oct 31 21:03:42.571444861 (33h23m14.267293614s) [noise=true , address=145.79.227.17:4970 , piece_id=6LJTN7QUSUSHS6DSKRHUUA3XB5ICEZ2ACFOSU4SK5HMUDRSKMEYA, seg_pos={0 0}, upload=29297]: Done uploading piece. err: <nil> Oct 31 21:03:42.707369620 (33h23m14.403218373s) [noise=true , address=145.79.227.11:4732 , piece_id=E4WNX7JXJZHMPKBPEJPSXJAYL4BP47HLXLO6DBHPICX6ESIK767Q, seg_pos={0 0}, upload=29297]: Done uploading piece. err: <nil> Oct 31 21:03:42.745077775 (33h23m14.440926518s) [noise=true , address=145.79.227.17:4971 , piece_id=3P7N3ERAPOY62SAKEBV7CHWOWWNUPWLSLL3RZ4RWCPF3VHI3C4FQ, seg_pos={0 0}, upload=29297]: Done uploading piece. err: <nil> Oct 31 21:04:00.023801048 (33h23m31.719649801s) [noise=true , address=77.90.31.17:4772 , piece_id=BFR46U4SPW6VNCIDV3XNIEOVYWGCXMQPDZ4JCJO5EAY5VSY4X56A, seg_pos={0 0}, upload=29297]: Done uploading piece. err: <nil> Oct 31 21:04:09.488974106 (33h23m41.184822859s) [noise=true , address=77.90.31.15:4698 , piece_id=2SJEFFLOU7LPMQZV5DLW5Z4BUQNJFZ347K656TFXV47LTCODGA3A, seg_pos={0 0}, upload=29297]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: write tcp 188.166.225.170:35474->77.90.31.15:4698: use of closed network connection; write tcp 188.166.225.170:35474->77.90.31.15:4698: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35474->77.90.31.15:4698: use of closed network connection Oct 31 21:04:09.489128110 (33h23m41.184976863s) [noise=true , address=77.90.31.10:4517 , piece_id=BDAUXG53XX42I72C6BIIC3VCLA6GZEFD3IDVSLYCC73AS2TG626Q, seg_pos={0 0}, upload=29297]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 188.166.225.170:58092->77.90.31.10:4517: use of closed network connection; write tcp 188.166.225.170:58092->77.90.31.10:4517: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58092->77.90.31.10:4517: use of closed network connection Oct 31 21:04:09.489250382 (33h23m41.185099135s) [noise=true , address=77.90.31.11:4555 , piece_id=BMTMJ4QDYLWDSMNIP64YUDUBZA2DABKYYC5JK72VJSUBTBZ67YTA, seg_pos={0 0}, upload=29297]: Done uploading piece. err: ecclient: upload failed (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, address:77.90.31.11:4555): protocol: write tcp 188.166.225.170:40898->77.90.31.11:4555: use of closed network connection; write tcp 188.166.225.170:40898->77.90.31.11:4555: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40898->77.90.31.11:4555: use of closed network connection Oct 31 21:04:09.489384528 (33h23m41.185233281s) [noise=true , address=77.90.31.15:4699 , piece_id=DLH6733XEDD5Z7AIO7NT4UC4TMTM57RUDVUHYOYONTXTGUJWJSYA, seg_pos={0 0}, upload=29297]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: write tcp 188.166.225.170:38634->77.90.31.15:4699: use of closed network connection; write tcp 188.166.225.170:38634->77.90.31.15:4699: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38634->77.90.31.15:4699: use of closed network connection Oct 31 21:04:09.489415327 (33h23m41.18526408s) [seg_pos={0 0}, upload=29297]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Oct 31 21:04:09.733073582 (33h23m41.428922335s) [upload=29297]: Done uploading segments... Oct 31 21:04:09.733121814 (33h23m41.428970557s) [upload=29297]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.246225601s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [....................................X....X..................] duration: 6.337560327s trace: https://jaeger.dev.storj.io/trace/35569b98f076d00a FAIL (2025-10-31 21:04:04 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.337590814s edge gateway us-select-1
checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect) antiblip: 1 out of 15 is failed (threshold=6) [.............X.] duration: 319.80812ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/dcs-satellites trace: https://jaeger.dev.storj.io/trace/571fef006088dd70 FAIL (2025-10-31 21:03:30 UTC) orbiter warning: Get "https://www.storj.io/dcs-satellites": read tcp [2400:6180:0:d0::f70:8001]:42806->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 319.835743ms core website
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...................................XX....X..................] duration: 5.937935318s trace: https://jaeger.dev.storj.io/trace/61feae49d47bc60e FAIL (2025-10-31 21:03:01 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Oct 31 20:51:11.706455020 (33h10m43.402303773s) [upload=29117]: Starting upload Oct 31 20:51:11.706524262 (33h10m43.402373005s) [upload=29117]: Done starting upload Oct 31 20:51:11.706833080 (33h10m43.402681823s) [upload=29117]: Uploading segments... Oct 31 20:51:11.707027190 (33h10m43.402875943s) [upload=29117]: Got next segment. Inline: false Oct 31 20:51:11.707074009 (33h10m43.402922762s) [upload=29117]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 31 20:51:11.707095790 (33h10m43.402944544s) [upload=29117]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 31 20:51:11.959151951 (33h10m43.655000704s) [seg_pos={0 0}, upload=29117]: Begin upload segment... Oct 31 20:51:11.959219510 (33h10m43.655068263s) [seg_pos={0 0}, upload=29117]: Done begin upload segment. Oct 31 20:51:11.959257482 (33h10m43.655106235s) [noise=true , address=145.79.227.13:4803 , piece_id=U3TRIALHSTLPGF44OTFLLTHPYHM2LU57DYF2ONNQSPFZSFJT3JGA, seg_pos={0 0}, upload=29117]: Uploading piece... Oct 31 20:51:11.959352193 (33h10m43.655200936s) [noise=true , address=145.79.227.31:4797 , piece_id=RTY2YTW7PA34HHO6NTN7WDGFUZTU5SF3B2TPDH76Z5HLZRRPRXUA, seg_pos={0 0}, upload=29117]: Uploading piece... Oct 31 20:51:11.959984507 (33h10m43.65583325s) [noise=true , address=145.79.227.21:4475 , piece_id=X47C7EP73DAXLRBDFDNLYLBRVTUJ7E25D6HE2BKNNFJCYWE2GETA, seg_pos={0 0}, upload=29117]: Uploading piece... Oct 31 20:51:11.960349932 (33h10m43.656198685s) [noise=true , address=145.79.227.28:4725 , piece_id=X2NUNP6LLBGDB2B2ZIT5MBW77PFJ3AZHJEPUDW3JWZ4CXTYXN6PQ, seg_pos={0 0}, upload=29117]: Uploading piece... Oct 31 20:51:11.960434654 (33h10m43.656283397s) [noise=true , address=145.79.227.31:4799 , piece_id=M2TGJ7FO5UPK5F32KJWOB4J2YAWF5QYB2MLI3CSLVZQXLSOLPOZQ, seg_pos={0 0}, upload=29117]: Uploading piece... Oct 31 20:51:11.960753571 (33h10m43.656602314s) [noise=true , address=145.79.227.8:4331 , piece_id=WBJEFDSQWI57LUVOFU46SW55X5A4TMAQDQYQUJBSL7PUK76BEWFA, seg_pos={0 0}, upload=29117]: Uploading piece... Oct 31 20:51:11.960782867 (33h10m43.65663162s) [noise=true , address=145.79.227.4:4185 , piece_id=F7WTGPH6LJR7KKPKLEITSPS24KRXGBYAPVBH2A7RVCPCQZQKVKFA, seg_pos={0 0}, upload=29117]: Uploading piece... Oct 31 20:51:11.961093809 (33h10m43.656942562s) [noise=true , address=77.90.31.18:4807 , piece_id=7GNYW2VEL4YS5PTCRGICZEVNDGLVIBNATAU6DGLMLQWVBHAVJQRQ, seg_pos={0 0}, upload=29117]: Uploading piece... Oct 31 20:51:11.961223666 (33h10m43.657072409s) [noise=true , address=145.79.227.32:4836 , piece_id=PYCFWHO5ACF33RG5CXXTUURHU5D3XHXFH7GWQXGCULB3LQCABCRQ, seg_pos={0 0}, upload=29117]: Uploading piece... Oct 31 20:51:11.961396354 (33h10m43.657245097s) [noise=true , address=77.90.31.13:4626 , piece_id=LP26J4GFG7AMZALJX6UVH4RBYCLWESEDNVF2ZJSNLEDYOFCSDSIQ, seg_pos={0 0}, upload=29117]: Uploading piece... Oct 31 20:51:11.961606384 (33h10m43.657455127s) [noise=true , address=145.79.227.9:4366 , piece_id=B3MDET2NK75LH2PFOTQQQKMTOORYPKBIPYMVDMRLAPALMFHMMTUQ, seg_pos={0 0}, upload=29117]: Uploading piece... Oct 31 20:51:11.961727265 (33h10m43.657576008s) [noise=true , address=145.79.227.23:4546 , piece_id=XUIN6VBEE64LCSA5UNYXSN3BR74IL6D5VVEPA2KELRN4HYFRPIIA, seg_pos={0 0}, upload=29117]: Uploading piece... Oct 31 20:51:11.962038738 (33h10m43.657887481s) [noise=true , address=77.90.31.12:4592 , piece_id=PW52BFRZZDXQWEXJUBDYNPSU6QGDV7MEJKUM32XKHJIQPQUHAH7A, seg_pos={0 0}, upload=29117]: Uploading piece... Oct 31 20:51:11.962095516 (33h10m43.657944259s) [noise=true , address=145.79.227.24:4581 , piece_id=5QMNPMLJAQXITYKVAMORL5C2ETAVDLAGYUFMFLXI4SJ3HK67LXKA, seg_pos={0 0}, upload=29117]: Uploading piece... Oct 31 20:51:11.962430774 (33h10m43.658279517s) [noise=true , address=77.90.31.10:4519 , piece_id=XSDLPRAYDE7OPRTJBYSMM3YZNGEPLVAPN5YYKBJNBDGXALNM2ISA, seg_pos={0 0}, upload=29117]: Uploading piece... Oct 31 20:51:11.962491260 (33h10m43.658340013s) [noise=true , address=145.79.227.14:4840 , piece_id=NSQMPI3HHFYPKRTICMDIAKLP4OYYXK3PUKWAQWGDUKXIMMAET26Q, seg_pos={0 0}, upload=29117]: Uploading piece... Oct 31 20:51:13.589873545 (33h10m45.285722288s) [upload=29117]: Next returned nil segment Oct 31 20:51:13.589889816 (33h10m45.285738569s) [upload=29117]: Waiting for error group managing segments... Oct 31 20:51:13.973997709 (33h10m45.669846462s) [noise=true , address=145.79.227.9:4366 , piece_id=B3MDET2NK75LH2PFOTQQQKMTOORYPKBIPYMVDMRLAPALMFHMMTUQ, seg_pos={0 0}, upload=29117]: Done uploading piece. err: <nil> Oct 31 20:51:13.977395970 (33h10m45.673244713s) [noise=true , address=145.79.227.8:4331 , piece_id=WBJEFDSQWI57LUVOFU46SW55X5A4TMAQDQYQUJBSL7PUK76BEWFA, seg_pos={0 0}, upload=29117]: Done uploading piece. err: <nil> Oct 31 20:51:13.979447918 (33h10m45.675296671s) [noise=true , address=145.79.227.14:4840 , piece_id=NSQMPI3HHFYPKRTICMDIAKLP4OYYXK3PUKWAQWGDUKXIMMAET26Q, seg_pos={0 0}, upload=29117]: Done uploading piece. err: <nil> Oct 31 20:51:13.980901436 (33h10m45.676750189s) [noise=true , address=145.79.227.31:4797 , piece_id=RTY2YTW7PA34HHO6NTN7WDGFUZTU5SF3B2TPDH76Z5HLZRRPRXUA, seg_pos={0 0}, upload=29117]: Done uploading piece. err: <nil> Oct 31 20:51:13.986457284 (33h10m45.682306088s) [noise=true , address=145.79.227.24:4581 , piece_id=5QMNPMLJAQXITYKVAMORL5C2ETAVDLAGYUFMFLXI4SJ3HK67LXKA, seg_pos={0 0}, upload=29117]: Done uploading piece. err: <nil> Oct 31 20:51:14.004274321 (33h10m45.700123084s) [noise=true , address=145.79.227.13:4803 , piece_id=U3TRIALHSTLPGF44OTFLLTHPYHM2LU57DYF2ONNQSPFZSFJT3JGA, seg_pos={0 0}, upload=29117]: Done uploading piece. err: <nil> Oct 31 20:51:14.012149448 (33h10m45.707998201s) [noise=true , address=145.79.227.4:4185 , piece_id=F7WTGPH6LJR7KKPKLEITSPS24KRXGBYAPVBH2A7RVCPCQZQKVKFA, seg_pos={0 0}, upload=29117]: Done uploading piece. err: <nil> Oct 31 20:51:14.027225746 (33h10m45.723074499s) [noise=true , address=145.79.227.32:4836 , piece_id=PYCFWHO5ACF33RG5CXXTUURHU5D3XHXFH7GWQXGCULB3LQCABCRQ, seg_pos={0 0}, upload=29117]: Done uploading piece. err: <nil> Oct 31 20:51:14.043308651 (33h10m45.739157404s) [noise=true , address=145.79.227.21:4475 , piece_id=X47C7EP73DAXLRBDFDNLYLBRVTUJ7E25D6HE2BKNNFJCYWE2GETA, seg_pos={0 0}, upload=29117]: Done uploading piece. err: <nil> Oct 31 20:51:14.050118539 (33h10m45.745967292s) [noise=true , address=145.79.227.23:4546 , piece_id=XUIN6VBEE64LCSA5UNYXSN3BR74IL6D5VVEPA2KELRN4HYFRPIIA, seg_pos={0 0}, upload=29117]: Done uploading piece. err: <nil> Oct 31 20:51:14.065628894 (33h10m45.761477637s) [noise=true , address=145.79.227.28:4725 , piece_id=X2NUNP6LLBGDB2B2ZIT5MBW77PFJ3AZHJEPUDW3JWZ4CXTYXN6PQ, seg_pos={0 0}, upload=29117]: Done uploading piece. err: <nil> Oct 31 20:51:14.079898897 (33h10m45.77574765s) [noise=true , address=145.79.227.31:4799 , piece_id=M2TGJ7FO5UPK5F32KJWOB4J2YAWF5QYB2MLI3CSLVZQXLSOLPOZQ, seg_pos={0 0}, upload=29117]: Done uploading piece. err: <nil> Oct 31 20:51:41.002915508 (33h11m12.698764281s) [noise=true , address=77.90.31.12:4592 , piece_id=PW52BFRZZDXQWEXJUBDYNPSU6QGDV7MEJKUM32XKHJIQPQUHAH7A, seg_pos={0 0}, upload=29117]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: write tcp 188.166.225.170:47064->77.90.31.12:4592: use of closed network connection; write tcp 188.166.225.170:47064->77.90.31.12:4592: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47064->77.90.31.12:4592: use of closed network connection Oct 31 20:51:41.003131980 (33h11m12.698980723s) [noise=true , address=77.90.31.10:4519 , piece_id=XSDLPRAYDE7OPRTJBYSMM3YZNGEPLVAPN5YYKBJNBDGXALNM2ISA, seg_pos={0 0}, upload=29117]: Done uploading piece. err: ecclient: upload failed (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, address:77.90.31.10:4519): protocol: write tcp 188.166.225.170:37294->77.90.31.10:4519: use of closed network connection; write tcp 188.166.225.170:37294->77.90.31.10:4519: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37294->77.90.31.10:4519: use of closed network connection Oct 31 20:51:41.003350967 (33h11m12.69919973s) [noise=true , address=77.90.31.13:4626 , piece_id=LP26J4GFG7AMZALJX6UVH4RBYCLWESEDNVF2ZJSNLEDYOFCSDSIQ, seg_pos={0 0}, upload=29117]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: write tcp 188.166.225.170:59286->77.90.31.13:4626: use of closed network connection; write tcp 188.166.225.170:59286->77.90.31.13:4626: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59286->77.90.31.13:4626: use of closed network connection Oct 31 20:51:41.003851250 (33h11m12.699700003s) [noise=true , address=77.90.31.18:4807 , piece_id=7GNYW2VEL4YS5PTCRGICZEVNDGLVIBNATAU6DGLMLQWVBHAVJQRQ, seg_pos={0 0}, upload=29117]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: read tcp 188.166.225.170:49810->77.90.31.18:4807: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 20:51:41.003903128 (33h11m12.699751891s) [seg_pos={0 0}, upload=29117]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Oct 31 20:51:41.248997100 (33h11m12.944845853s) [upload=29117]: Done uploading segments... Oct 31 20:51:41.249037957 (33h11m12.944886711s) [upload=29117]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.937966297s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....................................X....X.................] duration: 6.125113237s trace: https://jaeger.dev.storj.io/trace/35569b98f076d00a FAIL (2025-10-31 21:02:52 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.125175866s edge gateway us-select-1
checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect) antiblip: 1 out of 15 is failed (threshold=6) [..............X] duration: 435.022641ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/dcs-satellites trace: https://jaeger.dev.storj.io/trace/571fef006088dd70 FAIL (2025-10-31 21:02:30 UTC) orbiter warning: Get "https://www.storj.io/dcs-satellites": read tcp [2400:6180:0:d0::f70:8001]:42806->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 435.045275ms core website
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [....................................XX....X.................] duration: 6.14379136s trace: https://jaeger.dev.storj.io/trace/61feae49d47bc60e FAIL (2025-10-31 21:02:03 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Oct 31 20:51:11.706455020 (33h10m43.402303773s) [upload=29117]: Starting upload Oct 31 20:51:11.706524262 (33h10m43.402373005s) [upload=29117]: Done starting upload Oct 31 20:51:11.706833080 (33h10m43.402681823s) [upload=29117]: Uploading segments... Oct 31 20:51:11.707027190 (33h10m43.402875943s) [upload=29117]: Got next segment. Inline: false Oct 31 20:51:11.707074009 (33h10m43.402922762s) [upload=29117]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 31 20:51:11.707095790 (33h10m43.402944544s) [upload=29117]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 31 20:51:11.959151951 (33h10m43.655000704s) [seg_pos={0 0}, upload=29117]: Begin upload segment... Oct 31 20:51:11.959219510 (33h10m43.655068263s) [seg_pos={0 0}, upload=29117]: Done begin upload segment. Oct 31 20:51:11.959257482 (33h10m43.655106235s) [noise=true , address=145.79.227.13:4803 , piece_id=U3TRIALHSTLPGF44OTFLLTHPYHM2LU57DYF2ONNQSPFZSFJT3JGA, seg_pos={0 0}, upload=29117]: Uploading piece... Oct 31 20:51:11.959352193 (33h10m43.655200936s) [noise=true , address=145.79.227.31:4797 , piece_id=RTY2YTW7PA34HHO6NTN7WDGFUZTU5SF3B2TPDH76Z5HLZRRPRXUA, seg_pos={0 0}, upload=29117]: Uploading piece... Oct 31 20:51:11.959984507 (33h10m43.65583325s) [noise=true , address=145.79.227.21:4475 , piece_id=X47C7EP73DAXLRBDFDNLYLBRVTUJ7E25D6HE2BKNNFJCYWE2GETA, seg_pos={0 0}, upload=29117]: Uploading piece... Oct 31 20:51:11.960349932 (33h10m43.656198685s) [noise=true , address=145.79.227.28:4725 , piece_id=X2NUNP6LLBGDB2B2ZIT5MBW77PFJ3AZHJEPUDW3JWZ4CXTYXN6PQ, seg_pos={0 0}, upload=29117]: Uploading piece... Oct 31 20:51:11.960434654 (33h10m43.656283397s) [noise=true , address=145.79.227.31:4799 , piece_id=M2TGJ7FO5UPK5F32KJWOB4J2YAWF5QYB2MLI3CSLVZQXLSOLPOZQ, seg_pos={0 0}, upload=29117]: Uploading piece... Oct 31 20:51:11.960753571 (33h10m43.656602314s) [noise=true , address=145.79.227.8:4331 , piece_id=WBJEFDSQWI57LUVOFU46SW55X5A4TMAQDQYQUJBSL7PUK76BEWFA, seg_pos={0 0}, upload=29117]: Uploading piece... Oct 31 20:51:11.960782867 (33h10m43.65663162s) [noise=true , address=145.79.227.4:4185 , piece_id=F7WTGPH6LJR7KKPKLEITSPS24KRXGBYAPVBH2A7RVCPCQZQKVKFA, seg_pos={0 0}, upload=29117]: Uploading piece... Oct 31 20:51:11.961093809 (33h10m43.656942562s) [noise=true , address=77.90.31.18:4807 , piece_id=7GNYW2VEL4YS5PTCRGICZEVNDGLVIBNATAU6DGLMLQWVBHAVJQRQ, seg_pos={0 0}, upload=29117]: Uploading piece... Oct 31 20:51:11.961223666 (33h10m43.657072409s) [noise=true , address=145.79.227.32:4836 , piece_id=PYCFWHO5ACF33RG5CXXTUURHU5D3XHXFH7GWQXGCULB3LQCABCRQ, seg_pos={0 0}, upload=29117]: Uploading piece... Oct 31 20:51:11.961396354 (33h10m43.657245097s) [noise=true , address=77.90.31.13:4626 , piece_id=LP26J4GFG7AMZALJX6UVH4RBYCLWESEDNVF2ZJSNLEDYOFCSDSIQ, seg_pos={0 0}, upload=29117]: Uploading piece... Oct 31 20:51:11.961606384 (33h10m43.657455127s) [noise=true , address=145.79.227.9:4366 , piece_id=B3MDET2NK75LH2PFOTQQQKMTOORYPKBIPYMVDMRLAPALMFHMMTUQ, seg_pos={0 0}, upload=29117]: Uploading piece... Oct 31 20:51:11.961727265 (33h10m43.657576008s) [noise=true , address=145.79.227.23:4546 , piece_id=XUIN6VBEE64LCSA5UNYXSN3BR74IL6D5VVEPA2KELRN4HYFRPIIA, seg_pos={0 0}, upload=29117]: Uploading piece... Oct 31 20:51:11.962038738 (33h10m43.657887481s) [noise=true , address=77.90.31.12:4592 , piece_id=PW52BFRZZDXQWEXJUBDYNPSU6QGDV7MEJKUM32XKHJIQPQUHAH7A, seg_pos={0 0}, upload=29117]: Uploading piece... Oct 31 20:51:11.962095516 (33h10m43.657944259s) [noise=true , address=145.79.227.24:4581 , piece_id=5QMNPMLJAQXITYKVAMORL5C2ETAVDLAGYUFMFLXI4SJ3HK67LXKA, seg_pos={0 0}, upload=29117]: Uploading piece... Oct 31 20:51:11.962430774 (33h10m43.658279517s) [noise=true , address=77.90.31.10:4519 , piece_id=XSDLPRAYDE7OPRTJBYSMM3YZNGEPLVAPN5YYKBJNBDGXALNM2ISA, seg_pos={0 0}, upload=29117]: Uploading piece... Oct 31 20:51:11.962491260 (33h10m43.658340013s) [noise=true , address=145.79.227.14:4840 , piece_id=NSQMPI3HHFYPKRTICMDIAKLP4OYYXK3PUKWAQWGDUKXIMMAET26Q, seg_pos={0 0}, upload=29117]: Uploading piece... Oct 31 20:51:13.589873545 (33h10m45.285722288s) [upload=29117]: Next returned nil segment Oct 31 20:51:13.589889816 (33h10m45.285738569s) [upload=29117]: Waiting for error group managing segments... Oct 31 20:51:13.973997709 (33h10m45.669846462s) [noise=true , address=145.79.227.9:4366 , piece_id=B3MDET2NK75LH2PFOTQQQKMTOORYPKBIPYMVDMRLAPALMFHMMTUQ, seg_pos={0 0}, upload=29117]: Done uploading piece. err: <nil> Oct 31 20:51:13.977395970 (33h10m45.673244713s) [noise=true , address=145.79.227.8:4331 , piece_id=WBJEFDSQWI57LUVOFU46SW55X5A4TMAQDQYQUJBSL7PUK76BEWFA, seg_pos={0 0}, upload=29117]: Done uploading piece. err: <nil> Oct 31 20:51:13.979447918 (33h10m45.675296671s) [noise=true , address=145.79.227.14:4840 , piece_id=NSQMPI3HHFYPKRTICMDIAKLP4OYYXK3PUKWAQWGDUKXIMMAET26Q, seg_pos={0 0}, upload=29117]: Done uploading piece. err: <nil> Oct 31 20:51:13.980901436 (33h10m45.676750189s) [noise=true , address=145.79.227.31:4797 , piece_id=RTY2YTW7PA34HHO6NTN7WDGFUZTU5SF3B2TPDH76Z5HLZRRPRXUA, seg_pos={0 0}, upload=29117]: Done uploading piece. err: <nil> Oct 31 20:51:13.986457284 (33h10m45.682306088s) [noise=true , address=145.79.227.24:4581 , piece_id=5QMNPMLJAQXITYKVAMORL5C2ETAVDLAGYUFMFLXI4SJ3HK67LXKA, seg_pos={0 0}, upload=29117]: Done uploading piece. err: <nil> Oct 31 20:51:14.004274321 (33h10m45.700123084s) [noise=true , address=145.79.227.13:4803 , piece_id=U3TRIALHSTLPGF44OTFLLTHPYHM2LU57DYF2ONNQSPFZSFJT3JGA, seg_pos={0 0}, upload=29117]: Done uploading piece. err: <nil> Oct 31 20:51:14.012149448 (33h10m45.707998201s) [noise=true , address=145.79.227.4:4185 , piece_id=F7WTGPH6LJR7KKPKLEITSPS24KRXGBYAPVBH2A7RVCPCQZQKVKFA, seg_pos={0 0}, upload=29117]: Done uploading piece. err: <nil> Oct 31 20:51:14.027225746 (33h10m45.723074499s) [noise=true , address=145.79.227.32:4836 , piece_id=PYCFWHO5ACF33RG5CXXTUURHU5D3XHXFH7GWQXGCULB3LQCABCRQ, seg_pos={0 0}, upload=29117]: Done uploading piece. err: <nil> Oct 31 20:51:14.043308651 (33h10m45.739157404s) [noise=true , address=145.79.227.21:4475 , piece_id=X47C7EP73DAXLRBDFDNLYLBRVTUJ7E25D6HE2BKNNFJCYWE2GETA, seg_pos={0 0}, upload=29117]: Done uploading piece. err: <nil> Oct 31 20:51:14.050118539 (33h10m45.745967292s) [noise=true , address=145.79.227.23:4546 , piece_id=XUIN6VBEE64LCSA5UNYXSN3BR74IL6D5VVEPA2KELRN4HYFRPIIA, seg_pos={0 0}, upload=29117]: Done uploading piece. err: <nil> Oct 31 20:51:14.065628894 (33h10m45.761477637s) [noise=true , address=145.79.227.28:4725 , piece_id=X2NUNP6LLBGDB2B2ZIT5MBW77PFJ3AZHJEPUDW3JWZ4CXTYXN6PQ, seg_pos={0 0}, upload=29117]: Done uploading piece. err: <nil> Oct 31 20:51:14.079898897 (33h10m45.77574765s) [noise=true , address=145.79.227.31:4799 , piece_id=M2TGJ7FO5UPK5F32KJWOB4J2YAWF5QYB2MLI3CSLVZQXLSOLPOZQ, seg_pos={0 0}, upload=29117]: Done uploading piece. err: <nil> Oct 31 20:51:41.002915508 (33h11m12.698764281s) [noise=true , address=77.90.31.12:4592 , piece_id=PW52BFRZZDXQWEXJUBDYNPSU6QGDV7MEJKUM32XKHJIQPQUHAH7A, seg_pos={0 0}, upload=29117]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: write tcp 188.166.225.170:47064->77.90.31.12:4592: use of closed network connection; write tcp 188.166.225.170:47064->77.90.31.12:4592: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47064->77.90.31.12:4592: use of closed network connection Oct 31 20:51:41.003131980 (33h11m12.698980723s) [noise=true , address=77.90.31.10:4519 , piece_id=XSDLPRAYDE7OPRTJBYSMM3YZNGEPLVAPN5YYKBJNBDGXALNM2ISA, seg_pos={0 0}, upload=29117]: Done uploading piece. err: ecclient: upload failed (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, address:77.90.31.10:4519): protocol: write tcp 188.166.225.170:37294->77.90.31.10:4519: use of closed network connection; write tcp 188.166.225.170:37294->77.90.31.10:4519: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37294->77.90.31.10:4519: use of closed network connection Oct 31 20:51:41.003350967 (33h11m12.69919973s) [noise=true , address=77.90.31.13:4626 , piece_id=LP26J4GFG7AMZALJX6UVH4RBYCLWESEDNVF2ZJSNLEDYOFCSDSIQ, seg_pos={0 0}, upload=29117]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: write tcp 188.166.225.170:59286->77.90.31.13:4626: use of closed network connection; write tcp 188.166.225.170:59286->77.90.31.13:4626: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59286->77.90.31.13:4626: use of closed network connection Oct 31 20:51:41.003851250 (33h11m12.699700003s) [noise=true , address=77.90.31.18:4807 , piece_id=7GNYW2VEL4YS5PTCRGICZEVNDGLVIBNATAU6DGLMLQWVBHAVJQRQ, seg_pos={0 0}, upload=29117]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: read tcp 188.166.225.170:49810->77.90.31.18:4807: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 20:51:41.003903128 (33h11m12.699751891s) [seg_pos={0 0}, upload=29117]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Oct 31 20:51:41.248997100 (33h11m12.944845853s) [upload=29117]: Done uploading segments... Oct 31 20:51:41.249037957 (33h11m12.944886711s) [upload=29117]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.143841456s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [......................................X....X................] duration: 6.146819876s trace: https://jaeger.dev.storj.io/trace/35569b98f076d00a FAIL (2025-10-31 21:02:02 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.146859612s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.....................................XX....X................] duration: 5.990608286s trace: https://jaeger.dev.storj.io/trace/61feae49d47bc60e FAIL (2025-10-31 21:01:48 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Oct 31 20:51:11.706455020 (33h10m43.402303773s) [upload=29117]: Starting upload Oct 31 20:51:11.706524262 (33h10m43.402373005s) [upload=29117]: Done starting upload Oct 31 20:51:11.706833080 (33h10m43.402681823s) [upload=29117]: Uploading segments... Oct 31 20:51:11.707027190 (33h10m43.402875943s) [upload=29117]: Got next segment. Inline: false Oct 31 20:51:11.707074009 (33h10m43.402922762s) [upload=29117]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 31 20:51:11.707095790 (33h10m43.402944544s) [upload=29117]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 31 20:51:11.959151951 (33h10m43.655000704s) [seg_pos={0 0}, upload=29117]: Begin upload segment... Oct 31 20:51:11.959219510 (33h10m43.655068263s) [seg_pos={0 0}, upload=29117]: Done begin upload segment. Oct 31 20:51:11.959257482 (33h10m43.655106235s) [noise=true , address=145.79.227.13:4803 , piece_id=U3TRIALHSTLPGF44OTFLLTHPYHM2LU57DYF2ONNQSPFZSFJT3JGA, seg_pos={0 0}, upload=29117]: Uploading piece... Oct 31 20:51:11.959352193 (33h10m43.655200936s) [noise=true , address=145.79.227.31:4797 , piece_id=RTY2YTW7PA34HHO6NTN7WDGFUZTU5SF3B2TPDH76Z5HLZRRPRXUA, seg_pos={0 0}, upload=29117]: Uploading piece... Oct 31 20:51:11.959984507 (33h10m43.65583325s) [noise=true , address=145.79.227.21:4475 , piece_id=X47C7EP73DAXLRBDFDNLYLBRVTUJ7E25D6HE2BKNNFJCYWE2GETA, seg_pos={0 0}, upload=29117]: Uploading piece... Oct 31 20:51:11.960349932 (33h10m43.656198685s) [noise=true , address=145.79.227.28:4725 , piece_id=X2NUNP6LLBGDB2B2ZIT5MBW77PFJ3AZHJEPUDW3JWZ4CXTYXN6PQ, seg_pos={0 0}, upload=29117]: Uploading piece... Oct 31 20:51:11.960434654 (33h10m43.656283397s) [noise=true , address=145.79.227.31:4799 , piece_id=M2TGJ7FO5UPK5F32KJWOB4J2YAWF5QYB2MLI3CSLVZQXLSOLPOZQ, seg_pos={0 0}, upload=29117]: Uploading piece... Oct 31 20:51:11.960753571 (33h10m43.656602314s) [noise=true , address=145.79.227.8:4331 , piece_id=WBJEFDSQWI57LUVOFU46SW55X5A4TMAQDQYQUJBSL7PUK76BEWFA, seg_pos={0 0}, upload=29117]: Uploading piece... Oct 31 20:51:11.960782867 (33h10m43.65663162s) [noise=true , address=145.79.227.4:4185 , piece_id=F7WTGPH6LJR7KKPKLEITSPS24KRXGBYAPVBH2A7RVCPCQZQKVKFA, seg_pos={0 0}, upload=29117]: Uploading piece... Oct 31 20:51:11.961093809 (33h10m43.656942562s) [noise=true , address=77.90.31.18:4807 , piece_id=7GNYW2VEL4YS5PTCRGICZEVNDGLVIBNATAU6DGLMLQWVBHAVJQRQ, seg_pos={0 0}, upload=29117]: Uploading piece... Oct 31 20:51:11.961223666 (33h10m43.657072409s) [noise=true , address=145.79.227.32:4836 , piece_id=PYCFWHO5ACF33RG5CXXTUURHU5D3XHXFH7GWQXGCULB3LQCABCRQ, seg_pos={0 0}, upload=29117]: Uploading piece... Oct 31 20:51:11.961396354 (33h10m43.657245097s) [noise=true , address=77.90.31.13:4626 , piece_id=LP26J4GFG7AMZALJX6UVH4RBYCLWESEDNVF2ZJSNLEDYOFCSDSIQ, seg_pos={0 0}, upload=29117]: Uploading piece... Oct 31 20:51:11.961606384 (33h10m43.657455127s) [noise=true , address=145.79.227.9:4366 , piece_id=B3MDET2NK75LH2PFOTQQQKMTOORYPKBIPYMVDMRLAPALMFHMMTUQ, seg_pos={0 0}, upload=29117]: Uploading piece... Oct 31 20:51:11.961727265 (33h10m43.657576008s) [noise=true , address=145.79.227.23:4546 , piece_id=XUIN6VBEE64LCSA5UNYXSN3BR74IL6D5VVEPA2KELRN4HYFRPIIA, seg_pos={0 0}, upload=29117]: Uploading piece... Oct 31 20:51:11.962038738 (33h10m43.657887481s) [noise=true , address=77.90.31.12:4592 , piece_id=PW52BFRZZDXQWEXJUBDYNPSU6QGDV7MEJKUM32XKHJIQPQUHAH7A, seg_pos={0 0}, upload=29117]: Uploading piece... Oct 31 20:51:11.962095516 (33h10m43.657944259s) [noise=true , address=145.79.227.24:4581 , piece_id=5QMNPMLJAQXITYKVAMORL5C2ETAVDLAGYUFMFLXI4SJ3HK67LXKA, seg_pos={0 0}, upload=29117]: Uploading piece... Oct 31 20:51:11.962430774 (33h10m43.658279517s) [noise=true , address=77.90.31.10:4519 , piece_id=XSDLPRAYDE7OPRTJBYSMM3YZNGEPLVAPN5YYKBJNBDGXALNM2ISA, seg_pos={0 0}, upload=29117]: Uploading piece... Oct 31 20:51:11.962491260 (33h10m43.658340013s) [noise=true , address=145.79.227.14:4840 , piece_id=NSQMPI3HHFYPKRTICMDIAKLP4OYYXK3PUKWAQWGDUKXIMMAET26Q, seg_pos={0 0}, upload=29117]: Uploading piece... Oct 31 20:51:13.589873545 (33h10m45.285722288s) [upload=29117]: Next returned nil segment Oct 31 20:51:13.589889816 (33h10m45.285738569s) [upload=29117]: Waiting for error group managing segments... Oct 31 20:51:13.973997709 (33h10m45.669846462s) [noise=true , address=145.79.227.9:4366 , piece_id=B3MDET2NK75LH2PFOTQQQKMTOORYPKBIPYMVDMRLAPALMFHMMTUQ, seg_pos={0 0}, upload=29117]: Done uploading piece. err: <nil> Oct 31 20:51:13.977395970 (33h10m45.673244713s) [noise=true , address=145.79.227.8:4331 , piece_id=WBJEFDSQWI57LUVOFU46SW55X5A4TMAQDQYQUJBSL7PUK76BEWFA, seg_pos={0 0}, upload=29117]: Done uploading piece. err: <nil> Oct 31 20:51:13.979447918 (33h10m45.675296671s) [noise=true , address=145.79.227.14:4840 , piece_id=NSQMPI3HHFYPKRTICMDIAKLP4OYYXK3PUKWAQWGDUKXIMMAET26Q, seg_pos={0 0}, upload=29117]: Done uploading piece. err: <nil> Oct 31 20:51:13.980901436 (33h10m45.676750189s) [noise=true , address=145.79.227.31:4797 , piece_id=RTY2YTW7PA34HHO6NTN7WDGFUZTU5SF3B2TPDH76Z5HLZRRPRXUA, seg_pos={0 0}, upload=29117]: Done uploading piece. err: <nil> Oct 31 20:51:13.986457284 (33h10m45.682306088s) [noise=true , address=145.79.227.24:4581 , piece_id=5QMNPMLJAQXITYKVAMORL5C2ETAVDLAGYUFMFLXI4SJ3HK67LXKA, seg_pos={0 0}, upload=29117]: Done uploading piece. err: <nil> Oct 31 20:51:14.004274321 (33h10m45.700123084s) [noise=true , address=145.79.227.13:4803 , piece_id=U3TRIALHSTLPGF44OTFLLTHPYHM2LU57DYF2ONNQSPFZSFJT3JGA, seg_pos={0 0}, upload=29117]: Done uploading piece. err: <nil> Oct 31 20:51:14.012149448 (33h10m45.707998201s) [noise=true , address=145.79.227.4:4185 , piece_id=F7WTGPH6LJR7KKPKLEITSPS24KRXGBYAPVBH2A7RVCPCQZQKVKFA, seg_pos={0 0}, upload=29117]: Done uploading piece. err: <nil> Oct 31 20:51:14.027225746 (33h10m45.723074499s) [noise=true , address=145.79.227.32:4836 , piece_id=PYCFWHO5ACF33RG5CXXTUURHU5D3XHXFH7GWQXGCULB3LQCABCRQ, seg_pos={0 0}, upload=29117]: Done uploading piece. err: <nil> Oct 31 20:51:14.043308651 (33h10m45.739157404s) [noise=true , address=145.79.227.21:4475 , piece_id=X47C7EP73DAXLRBDFDNLYLBRVTUJ7E25D6HE2BKNNFJCYWE2GETA, seg_pos={0 0}, upload=29117]: Done uploading piece. err: <nil> Oct 31 20:51:14.050118539 (33h10m45.745967292s) [noise=true , address=145.79.227.23:4546 , piece_id=XUIN6VBEE64LCSA5UNYXSN3BR74IL6D5VVEPA2KELRN4HYFRPIIA, seg_pos={0 0}, upload=29117]: Done uploading piece. err: <nil> Oct 31 20:51:14.065628894 (33h10m45.761477637s) [noise=true , address=145.79.227.28:4725 , piece_id=X2NUNP6LLBGDB2B2ZIT5MBW77PFJ3AZHJEPUDW3JWZ4CXTYXN6PQ, seg_pos={0 0}, upload=29117]: Done uploading piece. err: <nil> Oct 31 20:51:14.079898897 (33h10m45.77574765s) [noise=true , address=145.79.227.31:4799 , piece_id=M2TGJ7FO5UPK5F32KJWOB4J2YAWF5QYB2MLI3CSLVZQXLSOLPOZQ, seg_pos={0 0}, upload=29117]: Done uploading piece. err: <nil> Oct 31 20:51:41.002915508 (33h11m12.698764281s) [noise=true , address=77.90.31.12:4592 , piece_id=PW52BFRZZDXQWEXJUBDYNPSU6QGDV7MEJKUM32XKHJIQPQUHAH7A, seg_pos={0 0}, upload=29117]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: write tcp 188.166.225.170:47064->77.90.31.12:4592: use of closed network connection; write tcp 188.166.225.170:47064->77.90.31.12:4592: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47064->77.90.31.12:4592: use of closed network connection Oct 31 20:51:41.003131980 (33h11m12.698980723s) [noise=true , address=77.90.31.10:4519 , piece_id=XSDLPRAYDE7OPRTJBYSMM3YZNGEPLVAPN5YYKBJNBDGXALNM2ISA, seg_pos={0 0}, upload=29117]: Done uploading piece. err: ecclient: upload failed (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, address:77.90.31.10:4519): protocol: write tcp 188.166.225.170:37294->77.90.31.10:4519: use of closed network connection; write tcp 188.166.225.170:37294->77.90.31.10:4519: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37294->77.90.31.10:4519: use of closed network connection Oct 31 20:51:41.003350967 (33h11m12.69919973s) [noise=true , address=77.90.31.13:4626 , piece_id=LP26J4GFG7AMZALJX6UVH4RBYCLWESEDNVF2ZJSNLEDYOFCSDSIQ, seg_pos={0 0}, upload=29117]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: write tcp 188.166.225.170:59286->77.90.31.13:4626: use of closed network connection; write tcp 188.166.225.170:59286->77.90.31.13:4626: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59286->77.90.31.13:4626: use of closed network connection Oct 31 20:51:41.003851250 (33h11m12.699700003s) [noise=true , address=77.90.31.18:4807 , piece_id=7GNYW2VEL4YS5PTCRGICZEVNDGLVIBNATAU6DGLMLQWVBHAVJQRQ, seg_pos={0 0}, upload=29117]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: read tcp 188.166.225.170:49810->77.90.31.18:4807: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 20:51:41.003903128 (33h11m12.699751891s) [seg_pos={0 0}, upload=29117]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Oct 31 20:51:41.248997100 (33h11m12.944845853s) [upload=29117]: Done uploading segments... Oct 31 20:51:41.249037957 (33h11m12.944886711s) [upload=29117]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.990641952s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......................................X....X...............] duration: 6.262630107s trace: https://jaeger.dev.storj.io/trace/35569b98f076d00a FAIL (2025-10-31 21:01:44 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.262715639s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [........................................X....X..............] duration: 6.192365628s trace: https://jaeger.dev.storj.io/trace/35569b98f076d00a FAIL (2025-10-31 21:01:08 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.19239303s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........................................X....X.............] duration: 6.153120012s trace: https://jaeger.dev.storj.io/trace/35569b98f076d00a FAIL (2025-10-31 21:00:50 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.153151762s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [......................................XX....X...............] duration: 5.856754817s trace: https://jaeger.dev.storj.io/trace/61feae49d47bc60e FAIL (2025-10-31 21:00:43 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Oct 31 20:51:11.706455020 (33h10m43.402303773s) [upload=29117]: Starting upload Oct 31 20:51:11.706524262 (33h10m43.402373005s) [upload=29117]: Done starting upload Oct 31 20:51:11.706833080 (33h10m43.402681823s) [upload=29117]: Uploading segments... Oct 31 20:51:11.707027190 (33h10m43.402875943s) [upload=29117]: Got next segment. Inline: false Oct 31 20:51:11.707074009 (33h10m43.402922762s) [upload=29117]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 31 20:51:11.707095790 (33h10m43.402944544s) [upload=29117]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 31 20:51:11.959151951 (33h10m43.655000704s) [seg_pos={0 0}, upload=29117]: Begin upload segment... Oct 31 20:51:11.959219510 (33h10m43.655068263s) [seg_pos={0 0}, upload=29117]: Done begin upload segment. Oct 31 20:51:11.959257482 (33h10m43.655106235s) [noise=true , address=145.79.227.13:4803 , piece_id=U3TRIALHSTLPGF44OTFLLTHPYHM2LU57DYF2ONNQSPFZSFJT3JGA, seg_pos={0 0}, upload=29117]: Uploading piece... Oct 31 20:51:11.959352193 (33h10m43.655200936s) [noise=true , address=145.79.227.31:4797 , piece_id=RTY2YTW7PA34HHO6NTN7WDGFUZTU5SF3B2TPDH76Z5HLZRRPRXUA, seg_pos={0 0}, upload=29117]: Uploading piece... Oct 31 20:51:11.959984507 (33h10m43.65583325s) [noise=true , address=145.79.227.21:4475 , piece_id=X47C7EP73DAXLRBDFDNLYLBRVTUJ7E25D6HE2BKNNFJCYWE2GETA, seg_pos={0 0}, upload=29117]: Uploading piece... Oct 31 20:51:11.960349932 (33h10m43.656198685s) [noise=true , address=145.79.227.28:4725 , piece_id=X2NUNP6LLBGDB2B2ZIT5MBW77PFJ3AZHJEPUDW3JWZ4CXTYXN6PQ, seg_pos={0 0}, upload=29117]: Uploading piece... Oct 31 20:51:11.960434654 (33h10m43.656283397s) [noise=true , address=145.79.227.31:4799 , piece_id=M2TGJ7FO5UPK5F32KJWOB4J2YAWF5QYB2MLI3CSLVZQXLSOLPOZQ, seg_pos={0 0}, upload=29117]: Uploading piece... Oct 31 20:51:11.960753571 (33h10m43.656602314s) [noise=true , address=145.79.227.8:4331 , piece_id=WBJEFDSQWI57LUVOFU46SW55X5A4TMAQDQYQUJBSL7PUK76BEWFA, seg_pos={0 0}, upload=29117]: Uploading piece... Oct 31 20:51:11.960782867 (33h10m43.65663162s) [noise=true , address=145.79.227.4:4185 , piece_id=F7WTGPH6LJR7KKPKLEITSPS24KRXGBYAPVBH2A7RVCPCQZQKVKFA, seg_pos={0 0}, upload=29117]: Uploading piece... Oct 31 20:51:11.961093809 (33h10m43.656942562s) [noise=true , address=77.90.31.18:4807 , piece_id=7GNYW2VEL4YS5PTCRGICZEVNDGLVIBNATAU6DGLMLQWVBHAVJQRQ, seg_pos={0 0}, upload=29117]: Uploading piece... Oct 31 20:51:11.961223666 (33h10m43.657072409s) [noise=true , address=145.79.227.32:4836 , piece_id=PYCFWHO5ACF33RG5CXXTUURHU5D3XHXFH7GWQXGCULB3LQCABCRQ, seg_pos={0 0}, upload=29117]: Uploading piece... Oct 31 20:51:11.961396354 (33h10m43.657245097s) [noise=true , address=77.90.31.13:4626 , piece_id=LP26J4GFG7AMZALJX6UVH4RBYCLWESEDNVF2ZJSNLEDYOFCSDSIQ, seg_pos={0 0}, upload=29117]: Uploading piece... Oct 31 20:51:11.961606384 (33h10m43.657455127s) [noise=true , address=145.79.227.9:4366 , piece_id=B3MDET2NK75LH2PFOTQQQKMTOORYPKBIPYMVDMRLAPALMFHMMTUQ, seg_pos={0 0}, upload=29117]: Uploading piece... Oct 31 20:51:11.961727265 (33h10m43.657576008s) [noise=true , address=145.79.227.23:4546 , piece_id=XUIN6VBEE64LCSA5UNYXSN3BR74IL6D5VVEPA2KELRN4HYFRPIIA, seg_pos={0 0}, upload=29117]: Uploading piece... Oct 31 20:51:11.962038738 (33h10m43.657887481s) [noise=true , address=77.90.31.12:4592 , piece_id=PW52BFRZZDXQWEXJUBDYNPSU6QGDV7MEJKUM32XKHJIQPQUHAH7A, seg_pos={0 0}, upload=29117]: Uploading piece... Oct 31 20:51:11.962095516 (33h10m43.657944259s) [noise=true , address=145.79.227.24:4581 , piece_id=5QMNPMLJAQXITYKVAMORL5C2ETAVDLAGYUFMFLXI4SJ3HK67LXKA, seg_pos={0 0}, upload=29117]: Uploading piece... Oct 31 20:51:11.962430774 (33h10m43.658279517s) [noise=true , address=77.90.31.10:4519 , piece_id=XSDLPRAYDE7OPRTJBYSMM3YZNGEPLVAPN5YYKBJNBDGXALNM2ISA, seg_pos={0 0}, upload=29117]: Uploading piece... Oct 31 20:51:11.962491260 (33h10m43.658340013s) [noise=true , address=145.79.227.14:4840 , piece_id=NSQMPI3HHFYPKRTICMDIAKLP4OYYXK3PUKWAQWGDUKXIMMAET26Q, seg_pos={0 0}, upload=29117]: Uploading piece... Oct 31 20:51:13.589873545 (33h10m45.285722288s) [upload=29117]: Next returned nil segment Oct 31 20:51:13.589889816 (33h10m45.285738569s) [upload=29117]: Waiting for error group managing segments... Oct 31 20:51:13.973997709 (33h10m45.669846462s) [noise=true , address=145.79.227.9:4366 , piece_id=B3MDET2NK75LH2PFOTQQQKMTOORYPKBIPYMVDMRLAPALMFHMMTUQ, seg_pos={0 0}, upload=29117]: Done uploading piece. err: <nil> Oct 31 20:51:13.977395970 (33h10m45.673244713s) [noise=true , address=145.79.227.8:4331 , piece_id=WBJEFDSQWI57LUVOFU46SW55X5A4TMAQDQYQUJBSL7PUK76BEWFA, seg_pos={0 0}, upload=29117]: Done uploading piece. err: <nil> Oct 31 20:51:13.979447918 (33h10m45.675296671s) [noise=true , address=145.79.227.14:4840 , piece_id=NSQMPI3HHFYPKRTICMDIAKLP4OYYXK3PUKWAQWGDUKXIMMAET26Q, seg_pos={0 0}, upload=29117]: Done uploading piece. err: <nil> Oct 31 20:51:13.980901436 (33h10m45.676750189s) [noise=true , address=145.79.227.31:4797 , piece_id=RTY2YTW7PA34HHO6NTN7WDGFUZTU5SF3B2TPDH76Z5HLZRRPRXUA, seg_pos={0 0}, upload=29117]: Done uploading piece. err: <nil> Oct 31 20:51:13.986457284 (33h10m45.682306088s) [noise=true , address=145.79.227.24:4581 , piece_id=5QMNPMLJAQXITYKVAMORL5C2ETAVDLAGYUFMFLXI4SJ3HK67LXKA, seg_pos={0 0}, upload=29117]: Done uploading piece. err: <nil> Oct 31 20:51:14.004274321 (33h10m45.700123084s) [noise=true , address=145.79.227.13:4803 , piece_id=U3TRIALHSTLPGF44OTFLLTHPYHM2LU57DYF2ONNQSPFZSFJT3JGA, seg_pos={0 0}, upload=29117]: Done uploading piece. err: <nil> Oct 31 20:51:14.012149448 (33h10m45.707998201s) [noise=true , address=145.79.227.4:4185 , piece_id=F7WTGPH6LJR7KKPKLEITSPS24KRXGBYAPVBH2A7RVCPCQZQKVKFA, seg_pos={0 0}, upload=29117]: Done uploading piece. err: <nil> Oct 31 20:51:14.027225746 (33h10m45.723074499s) [noise=true , address=145.79.227.32:4836 , piece_id=PYCFWHO5ACF33RG5CXXTUURHU5D3XHXFH7GWQXGCULB3LQCABCRQ, seg_pos={0 0}, upload=29117]: Done uploading piece. err: <nil> Oct 31 20:51:14.043308651 (33h10m45.739157404s) [noise=true , address=145.79.227.21:4475 , piece_id=X47C7EP73DAXLRBDFDNLYLBRVTUJ7E25D6HE2BKNNFJCYWE2GETA, seg_pos={0 0}, upload=29117]: Done uploading piece. err: <nil> Oct 31 20:51:14.050118539 (33h10m45.745967292s) [noise=true , address=145.79.227.23:4546 , piece_id=XUIN6VBEE64LCSA5UNYXSN3BR74IL6D5VVEPA2KELRN4HYFRPIIA, seg_pos={0 0}, upload=29117]: Done uploading piece. err: <nil> Oct 31 20:51:14.065628894 (33h10m45.761477637s) [noise=true , address=145.79.227.28:4725 , piece_id=X2NUNP6LLBGDB2B2ZIT5MBW77PFJ3AZHJEPUDW3JWZ4CXTYXN6PQ, seg_pos={0 0}, upload=29117]: Done uploading piece. err: <nil> Oct 31 20:51:14.079898897 (33h10m45.77574765s) [noise=true , address=145.79.227.31:4799 , piece_id=M2TGJ7FO5UPK5F32KJWOB4J2YAWF5QYB2MLI3CSLVZQXLSOLPOZQ, seg_pos={0 0}, upload=29117]: Done uploading piece. err: <nil> Oct 31 20:51:41.002915508 (33h11m12.698764281s) [noise=true , address=77.90.31.12:4592 , piece_id=PW52BFRZZDXQWEXJUBDYNPSU6QGDV7MEJKUM32XKHJIQPQUHAH7A, seg_pos={0 0}, upload=29117]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: write tcp 188.166.225.170:47064->77.90.31.12:4592: use of closed network connection; write tcp 188.166.225.170:47064->77.90.31.12:4592: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47064->77.90.31.12:4592: use of closed network connection Oct 31 20:51:41.003131980 (33h11m12.698980723s) [noise=true , address=77.90.31.10:4519 , piece_id=XSDLPRAYDE7OPRTJBYSMM3YZNGEPLVAPN5YYKBJNBDGXALNM2ISA, seg_pos={0 0}, upload=29117]: Done uploading piece. err: ecclient: upload failed (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, address:77.90.31.10:4519): protocol: write tcp 188.166.225.170:37294->77.90.31.10:4519: use of closed network connection; write tcp 188.166.225.170:37294->77.90.31.10:4519: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37294->77.90.31.10:4519: use of closed network connection Oct 31 20:51:41.003350967 (33h11m12.69919973s) [noise=true , address=77.90.31.13:4626 , piece_id=LP26J4GFG7AMZALJX6UVH4RBYCLWESEDNVF2ZJSNLEDYOFCSDSIQ, seg_pos={0 0}, upload=29117]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: write tcp 188.166.225.170:59286->77.90.31.13:4626: use of closed network connection; write tcp 188.166.225.170:59286->77.90.31.13:4626: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59286->77.90.31.13:4626: use of closed network connection Oct 31 20:51:41.003851250 (33h11m12.699700003s) [noise=true , address=77.90.31.18:4807 , piece_id=7GNYW2VEL4YS5PTCRGICZEVNDGLVIBNATAU6DGLMLQWVBHAVJQRQ, seg_pos={0 0}, upload=29117]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: read tcp 188.166.225.170:49810->77.90.31.18:4807: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 20:51:41.003903128 (33h11m12.699751891s) [seg_pos={0 0}, upload=29117]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Oct 31 20:51:41.248997100 (33h11m12.944845853s) [upload=29117]: Done uploading segments... Oct 31 20:51:41.249037957 (33h11m12.944886711s) [upload=29117]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.856784122s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........................................X....X............] duration: 6.20819668s trace: https://jaeger.dev.storj.io/trace/35569b98f076d00a FAIL (2025-10-31 21:00:29 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.208218643s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.......................................XX....X..............] duration: 5.893328831s trace: https://jaeger.dev.storj.io/trace/61feae49d47bc60e FAIL (2025-10-31 20:59:57 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Oct 31 20:51:11.706455020 (33h10m43.402303773s) [upload=29117]: Starting upload Oct 31 20:51:11.706524262 (33h10m43.402373005s) [upload=29117]: Done starting upload Oct 31 20:51:11.706833080 (33h10m43.402681823s) [upload=29117]: Uploading segments... Oct 31 20:51:11.707027190 (33h10m43.402875943s) [upload=29117]: Got next segment. Inline: false Oct 31 20:51:11.707074009 (33h10m43.402922762s) [upload=29117]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 31 20:51:11.707095790 (33h10m43.402944544s) [upload=29117]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 31 20:51:11.959151951 (33h10m43.655000704s) [seg_pos={0 0}, upload=29117]: Begin upload segment... Oct 31 20:51:11.959219510 (33h10m43.655068263s) [seg_pos={0 0}, upload=29117]: Done begin upload segment. Oct 31 20:51:11.959257482 (33h10m43.655106235s) [noise=true , address=145.79.227.13:4803 , piece_id=U3TRIALHSTLPGF44OTFLLTHPYHM2LU57DYF2ONNQSPFZSFJT3JGA, seg_pos={0 0}, upload=29117]: Uploading piece... Oct 31 20:51:11.959352193 (33h10m43.655200936s) [noise=true , address=145.79.227.31:4797 , piece_id=RTY2YTW7PA34HHO6NTN7WDGFUZTU5SF3B2TPDH76Z5HLZRRPRXUA, seg_pos={0 0}, upload=29117]: Uploading piece... Oct 31 20:51:11.959984507 (33h10m43.65583325s) [noise=true , address=145.79.227.21:4475 , piece_id=X47C7EP73DAXLRBDFDNLYLBRVTUJ7E25D6HE2BKNNFJCYWE2GETA, seg_pos={0 0}, upload=29117]: Uploading piece... Oct 31 20:51:11.960349932 (33h10m43.656198685s) [noise=true , address=145.79.227.28:4725 , piece_id=X2NUNP6LLBGDB2B2ZIT5MBW77PFJ3AZHJEPUDW3JWZ4CXTYXN6PQ, seg_pos={0 0}, upload=29117]: Uploading piece... Oct 31 20:51:11.960434654 (33h10m43.656283397s) [noise=true , address=145.79.227.31:4799 , piece_id=M2TGJ7FO5UPK5F32KJWOB4J2YAWF5QYB2MLI3CSLVZQXLSOLPOZQ, seg_pos={0 0}, upload=29117]: Uploading piece... Oct 31 20:51:11.960753571 (33h10m43.656602314s) [noise=true , address=145.79.227.8:4331 , piece_id=WBJEFDSQWI57LUVOFU46SW55X5A4TMAQDQYQUJBSL7PUK76BEWFA, seg_pos={0 0}, upload=29117]: Uploading piece... Oct 31 20:51:11.960782867 (33h10m43.65663162s) [noise=true , address=145.79.227.4:4185 , piece_id=F7WTGPH6LJR7KKPKLEITSPS24KRXGBYAPVBH2A7RVCPCQZQKVKFA, seg_pos={0 0}, upload=29117]: Uploading piece... Oct 31 20:51:11.961093809 (33h10m43.656942562s) [noise=true , address=77.90.31.18:4807 , piece_id=7GNYW2VEL4YS5PTCRGICZEVNDGLVIBNATAU6DGLMLQWVBHAVJQRQ, seg_pos={0 0}, upload=29117]: Uploading piece... Oct 31 20:51:11.961223666 (33h10m43.657072409s) [noise=true , address=145.79.227.32:4836 , piece_id=PYCFWHO5ACF33RG5CXXTUURHU5D3XHXFH7GWQXGCULB3LQCABCRQ, seg_pos={0 0}, upload=29117]: Uploading piece... Oct 31 20:51:11.961396354 (33h10m43.657245097s) [noise=true , address=77.90.31.13:4626 , piece_id=LP26J4GFG7AMZALJX6UVH4RBYCLWESEDNVF2ZJSNLEDYOFCSDSIQ, seg_pos={0 0}, upload=29117]: Uploading piece... Oct 31 20:51:11.961606384 (33h10m43.657455127s) [noise=true , address=145.79.227.9:4366 , piece_id=B3MDET2NK75LH2PFOTQQQKMTOORYPKBIPYMVDMRLAPALMFHMMTUQ, seg_pos={0 0}, upload=29117]: Uploading piece... Oct 31 20:51:11.961727265 (33h10m43.657576008s) [noise=true , address=145.79.227.23:4546 , piece_id=XUIN6VBEE64LCSA5UNYXSN3BR74IL6D5VVEPA2KELRN4HYFRPIIA, seg_pos={0 0}, upload=29117]: Uploading piece... Oct 31 20:51:11.962038738 (33h10m43.657887481s) [noise=true , address=77.90.31.12:4592 , piece_id=PW52BFRZZDXQWEXJUBDYNPSU6QGDV7MEJKUM32XKHJIQPQUHAH7A, seg_pos={0 0}, upload=29117]: Uploading piece... Oct 31 20:51:11.962095516 (33h10m43.657944259s) [noise=true , address=145.79.227.24:4581 , piece_id=5QMNPMLJAQXITYKVAMORL5C2ETAVDLAGYUFMFLXI4SJ3HK67LXKA, seg_pos={0 0}, upload=29117]: Uploading piece... Oct 31 20:51:11.962430774 (33h10m43.658279517s) [noise=true , address=77.90.31.10:4519 , piece_id=XSDLPRAYDE7OPRTJBYSMM3YZNGEPLVAPN5YYKBJNBDGXALNM2ISA, seg_pos={0 0}, upload=29117]: Uploading piece... Oct 31 20:51:11.962491260 (33h10m43.658340013s) [noise=true , address=145.79.227.14:4840 , piece_id=NSQMPI3HHFYPKRTICMDIAKLP4OYYXK3PUKWAQWGDUKXIMMAET26Q, seg_pos={0 0}, upload=29117]: Uploading piece... Oct 31 20:51:13.589873545 (33h10m45.285722288s) [upload=29117]: Next returned nil segment Oct 31 20:51:13.589889816 (33h10m45.285738569s) [upload=29117]: Waiting for error group managing segments... Oct 31 20:51:13.973997709 (33h10m45.669846462s) [noise=true , address=145.79.227.9:4366 , piece_id=B3MDET2NK75LH2PFOTQQQKMTOORYPKBIPYMVDMRLAPALMFHMMTUQ, seg_pos={0 0}, upload=29117]: Done uploading piece. err: <nil> Oct 31 20:51:13.977395970 (33h10m45.673244713s) [noise=true , address=145.79.227.8:4331 , piece_id=WBJEFDSQWI57LUVOFU46SW55X5A4TMAQDQYQUJBSL7PUK76BEWFA, seg_pos={0 0}, upload=29117]: Done uploading piece. err: <nil> Oct 31 20:51:13.979447918 (33h10m45.675296671s) [noise=true , address=145.79.227.14:4840 , piece_id=NSQMPI3HHFYPKRTICMDIAKLP4OYYXK3PUKWAQWGDUKXIMMAET26Q, seg_pos={0 0}, upload=29117]: Done uploading piece. err: <nil> Oct 31 20:51:13.980901436 (33h10m45.676750189s) [noise=true , address=145.79.227.31:4797 , piece_id=RTY2YTW7PA34HHO6NTN7WDGFUZTU5SF3B2TPDH76Z5HLZRRPRXUA, seg_pos={0 0}, upload=29117]: Done uploading piece. err: <nil> Oct 31 20:51:13.986457284 (33h10m45.682306088s) [noise=true , address=145.79.227.24:4581 , piece_id=5QMNPMLJAQXITYKVAMORL5C2ETAVDLAGYUFMFLXI4SJ3HK67LXKA, seg_pos={0 0}, upload=29117]: Done uploading piece. err: <nil> Oct 31 20:51:14.004274321 (33h10m45.700123084s) [noise=true , address=145.79.227.13:4803 , piece_id=U3TRIALHSTLPGF44OTFLLTHPYHM2LU57DYF2ONNQSPFZSFJT3JGA, seg_pos={0 0}, upload=29117]: Done uploading piece. err: <nil> Oct 31 20:51:14.012149448 (33h10m45.707998201s) [noise=true , address=145.79.227.4:4185 , piece_id=F7WTGPH6LJR7KKPKLEITSPS24KRXGBYAPVBH2A7RVCPCQZQKVKFA, seg_pos={0 0}, upload=29117]: Done uploading piece. err: <nil> Oct 31 20:51:14.027225746 (33h10m45.723074499s) [noise=true , address=145.79.227.32:4836 , piece_id=PYCFWHO5ACF33RG5CXXTUURHU5D3XHXFH7GWQXGCULB3LQCABCRQ, seg_pos={0 0}, upload=29117]: Done uploading piece. err: <nil> Oct 31 20:51:14.043308651 (33h10m45.739157404s) [noise=true , address=145.79.227.21:4475 , piece_id=X47C7EP73DAXLRBDFDNLYLBRVTUJ7E25D6HE2BKNNFJCYWE2GETA, seg_pos={0 0}, upload=29117]: Done uploading piece. err: <nil> Oct 31 20:51:14.050118539 (33h10m45.745967292s) [noise=true , address=145.79.227.23:4546 , piece_id=XUIN6VBEE64LCSA5UNYXSN3BR74IL6D5VVEPA2KELRN4HYFRPIIA, seg_pos={0 0}, upload=29117]: Done uploading piece. err: <nil> Oct 31 20:51:14.065628894 (33h10m45.761477637s) [noise=true , address=145.79.227.28:4725 , piece_id=X2NUNP6LLBGDB2B2ZIT5MBW77PFJ3AZHJEPUDW3JWZ4CXTYXN6PQ, seg_pos={0 0}, upload=29117]: Done uploading piece. err: <nil> Oct 31 20:51:14.079898897 (33h10m45.77574765s) [noise=true , address=145.79.227.31:4799 , piece_id=M2TGJ7FO5UPK5F32KJWOB4J2YAWF5QYB2MLI3CSLVZQXLSOLPOZQ, seg_pos={0 0}, upload=29117]: Done uploading piece. err: <nil> Oct 31 20:51:41.002915508 (33h11m12.698764281s) [noise=true , address=77.90.31.12:4592 , piece_id=PW52BFRZZDXQWEXJUBDYNPSU6QGDV7MEJKUM32XKHJIQPQUHAH7A, seg_pos={0 0}, upload=29117]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: write tcp 188.166.225.170:47064->77.90.31.12:4592: use of closed network connection; write tcp 188.166.225.170:47064->77.90.31.12:4592: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47064->77.90.31.12:4592: use of closed network connection Oct 31 20:51:41.003131980 (33h11m12.698980723s) [noise=true , address=77.90.31.10:4519 , piece_id=XSDLPRAYDE7OPRTJBYSMM3YZNGEPLVAPN5YYKBJNBDGXALNM2ISA, seg_pos={0 0}, upload=29117]: Done uploading piece. err: ecclient: upload failed (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, address:77.90.31.10:4519): protocol: write tcp 188.166.225.170:37294->77.90.31.10:4519: use of closed network connection; write tcp 188.166.225.170:37294->77.90.31.10:4519: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37294->77.90.31.10:4519: use of closed network connection Oct 31 20:51:41.003350967 (33h11m12.69919973s) [noise=true , address=77.90.31.13:4626 , piece_id=LP26J4GFG7AMZALJX6UVH4RBYCLWESEDNVF2ZJSNLEDYOFCSDSIQ, seg_pos={0 0}, upload=29117]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: write tcp 188.166.225.170:59286->77.90.31.13:4626: use of closed network connection; write tcp 188.166.225.170:59286->77.90.31.13:4626: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59286->77.90.31.13:4626: use of closed network connection Oct 31 20:51:41.003851250 (33h11m12.699700003s) [noise=true , address=77.90.31.18:4807 , piece_id=7GNYW2VEL4YS5PTCRGICZEVNDGLVIBNATAU6DGLMLQWVBHAVJQRQ, seg_pos={0 0}, upload=29117]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: read tcp 188.166.225.170:49810->77.90.31.18:4807: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 20:51:41.003903128 (33h11m12.699751891s) [seg_pos={0 0}, upload=29117]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Oct 31 20:51:41.248997100 (33h11m12.944845853s) [upload=29117]: Done uploading segments... Oct 31 20:51:41.249037957 (33h11m12.944886711s) [upload=29117]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.893367584s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...........................................X....X...........] duration: 6.158110858s trace: https://jaeger.dev.storj.io/trace/35569b98f076d00a FAIL (2025-10-31 20:59:30 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.158218573s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [............................................X....X..........] duration: 6.19793947s trace: https://jaeger.dev.storj.io/trace/35569b98f076d00a FAIL (2025-10-31 20:59:09 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.197969037s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [........................................XX....X.............] duration: 6.000984815s trace: https://jaeger.dev.storj.io/trace/61feae49d47bc60e FAIL (2025-10-31 20:59:05 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Oct 31 20:51:11.706455020 (33h10m43.402303773s) [upload=29117]: Starting upload Oct 31 20:51:11.706524262 (33h10m43.402373005s) [upload=29117]: Done starting upload Oct 31 20:51:11.706833080 (33h10m43.402681823s) [upload=29117]: Uploading segments... Oct 31 20:51:11.707027190 (33h10m43.402875943s) [upload=29117]: Got next segment. Inline: false Oct 31 20:51:11.707074009 (33h10m43.402922762s) [upload=29117]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 31 20:51:11.707095790 (33h10m43.402944544s) [upload=29117]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 31 20:51:11.959151951 (33h10m43.655000704s) [seg_pos={0 0}, upload=29117]: Begin upload segment... Oct 31 20:51:11.959219510 (33h10m43.655068263s) [seg_pos={0 0}, upload=29117]: Done begin upload segment. Oct 31 20:51:11.959257482 (33h10m43.655106235s) [noise=true , address=145.79.227.13:4803 , piece_id=U3TRIALHSTLPGF44OTFLLTHPYHM2LU57DYF2ONNQSPFZSFJT3JGA, seg_pos={0 0}, upload=29117]: Uploading piece... Oct 31 20:51:11.959352193 (33h10m43.655200936s) [noise=true , address=145.79.227.31:4797 , piece_id=RTY2YTW7PA34HHO6NTN7WDGFUZTU5SF3B2TPDH76Z5HLZRRPRXUA, seg_pos={0 0}, upload=29117]: Uploading piece... Oct 31 20:51:11.959984507 (33h10m43.65583325s) [noise=true , address=145.79.227.21:4475 , piece_id=X47C7EP73DAXLRBDFDNLYLBRVTUJ7E25D6HE2BKNNFJCYWE2GETA, seg_pos={0 0}, upload=29117]: Uploading piece... Oct 31 20:51:11.960349932 (33h10m43.656198685s) [noise=true , address=145.79.227.28:4725 , piece_id=X2NUNP6LLBGDB2B2ZIT5MBW77PFJ3AZHJEPUDW3JWZ4CXTYXN6PQ, seg_pos={0 0}, upload=29117]: Uploading piece... Oct 31 20:51:11.960434654 (33h10m43.656283397s) [noise=true , address=145.79.227.31:4799 , piece_id=M2TGJ7FO5UPK5F32KJWOB4J2YAWF5QYB2MLI3CSLVZQXLSOLPOZQ, seg_pos={0 0}, upload=29117]: Uploading piece... Oct 31 20:51:11.960753571 (33h10m43.656602314s) [noise=true , address=145.79.227.8:4331 , piece_id=WBJEFDSQWI57LUVOFU46SW55X5A4TMAQDQYQUJBSL7PUK76BEWFA, seg_pos={0 0}, upload=29117]: Uploading piece... Oct 31 20:51:11.960782867 (33h10m43.65663162s) [noise=true , address=145.79.227.4:4185 , piece_id=F7WTGPH6LJR7KKPKLEITSPS24KRXGBYAPVBH2A7RVCPCQZQKVKFA, seg_pos={0 0}, upload=29117]: Uploading piece... Oct 31 20:51:11.961093809 (33h10m43.656942562s) [noise=true , address=77.90.31.18:4807 , piece_id=7GNYW2VEL4YS5PTCRGICZEVNDGLVIBNATAU6DGLMLQWVBHAVJQRQ, seg_pos={0 0}, upload=29117]: Uploading piece... Oct 31 20:51:11.961223666 (33h10m43.657072409s) [noise=true , address=145.79.227.32:4836 , piece_id=PYCFWHO5ACF33RG5CXXTUURHU5D3XHXFH7GWQXGCULB3LQCABCRQ, seg_pos={0 0}, upload=29117]: Uploading piece... Oct 31 20:51:11.961396354 (33h10m43.657245097s) [noise=true , address=77.90.31.13:4626 , piece_id=LP26J4GFG7AMZALJX6UVH4RBYCLWESEDNVF2ZJSNLEDYOFCSDSIQ, seg_pos={0 0}, upload=29117]: Uploading piece... Oct 31 20:51:11.961606384 (33h10m43.657455127s) [noise=true , address=145.79.227.9:4366 , piece_id=B3MDET2NK75LH2PFOTQQQKMTOORYPKBIPYMVDMRLAPALMFHMMTUQ, seg_pos={0 0}, upload=29117]: Uploading piece... Oct 31 20:51:11.961727265 (33h10m43.657576008s) [noise=true , address=145.79.227.23:4546 , piece_id=XUIN6VBEE64LCSA5UNYXSN3BR74IL6D5VVEPA2KELRN4HYFRPIIA, seg_pos={0 0}, upload=29117]: Uploading piece... Oct 31 20:51:11.962038738 (33h10m43.657887481s) [noise=true , address=77.90.31.12:4592 , piece_id=PW52BFRZZDXQWEXJUBDYNPSU6QGDV7MEJKUM32XKHJIQPQUHAH7A, seg_pos={0 0}, upload=29117]: Uploading piece... Oct 31 20:51:11.962095516 (33h10m43.657944259s) [noise=true , address=145.79.227.24:4581 , piece_id=5QMNPMLJAQXITYKVAMORL5C2ETAVDLAGYUFMFLXI4SJ3HK67LXKA, seg_pos={0 0}, upload=29117]: Uploading piece... Oct 31 20:51:11.962430774 (33h10m43.658279517s) [noise=true , address=77.90.31.10:4519 , piece_id=XSDLPRAYDE7OPRTJBYSMM3YZNGEPLVAPN5YYKBJNBDGXALNM2ISA, seg_pos={0 0}, upload=29117]: Uploading piece... Oct 31 20:51:11.962491260 (33h10m43.658340013s) [noise=true , address=145.79.227.14:4840 , piece_id=NSQMPI3HHFYPKRTICMDIAKLP4OYYXK3PUKWAQWGDUKXIMMAET26Q, seg_pos={0 0}, upload=29117]: Uploading piece... Oct 31 20:51:13.589873545 (33h10m45.285722288s) [upload=29117]: Next returned nil segment Oct 31 20:51:13.589889816 (33h10m45.285738569s) [upload=29117]: Waiting for error group managing segments... Oct 31 20:51:13.973997709 (33h10m45.669846462s) [noise=true , address=145.79.227.9:4366 , piece_id=B3MDET2NK75LH2PFOTQQQKMTOORYPKBIPYMVDMRLAPALMFHMMTUQ, seg_pos={0 0}, upload=29117]: Done uploading piece. err: <nil> Oct 31 20:51:13.977395970 (33h10m45.673244713s) [noise=true , address=145.79.227.8:4331 , piece_id=WBJEFDSQWI57LUVOFU46SW55X5A4TMAQDQYQUJBSL7PUK76BEWFA, seg_pos={0 0}, upload=29117]: Done uploading piece. err: <nil> Oct 31 20:51:13.979447918 (33h10m45.675296671s) [noise=true , address=145.79.227.14:4840 , piece_id=NSQMPI3HHFYPKRTICMDIAKLP4OYYXK3PUKWAQWGDUKXIMMAET26Q, seg_pos={0 0}, upload=29117]: Done uploading piece. err: <nil> Oct 31 20:51:13.980901436 (33h10m45.676750189s) [noise=true , address=145.79.227.31:4797 , piece_id=RTY2YTW7PA34HHO6NTN7WDGFUZTU5SF3B2TPDH76Z5HLZRRPRXUA, seg_pos={0 0}, upload=29117]: Done uploading piece. err: <nil> Oct 31 20:51:13.986457284 (33h10m45.682306088s) [noise=true , address=145.79.227.24:4581 , piece_id=5QMNPMLJAQXITYKVAMORL5C2ETAVDLAGYUFMFLXI4SJ3HK67LXKA, seg_pos={0 0}, upload=29117]: Done uploading piece. err: <nil> Oct 31 20:51:14.004274321 (33h10m45.700123084s) [noise=true , address=145.79.227.13:4803 , piece_id=U3TRIALHSTLPGF44OTFLLTHPYHM2LU57DYF2ONNQSPFZSFJT3JGA, seg_pos={0 0}, upload=29117]: Done uploading piece. err: <nil> Oct 31 20:51:14.012149448 (33h10m45.707998201s) [noise=true , address=145.79.227.4:4185 , piece_id=F7WTGPH6LJR7KKPKLEITSPS24KRXGBYAPVBH2A7RVCPCQZQKVKFA, seg_pos={0 0}, upload=29117]: Done uploading piece. err: <nil> Oct 31 20:51:14.027225746 (33h10m45.723074499s) [noise=true , address=145.79.227.32:4836 , piece_id=PYCFWHO5ACF33RG5CXXTUURHU5D3XHXFH7GWQXGCULB3LQCABCRQ, seg_pos={0 0}, upload=29117]: Done uploading piece. err: <nil> Oct 31 20:51:14.043308651 (33h10m45.739157404s) [noise=true , address=145.79.227.21:4475 , piece_id=X47C7EP73DAXLRBDFDNLYLBRVTUJ7E25D6HE2BKNNFJCYWE2GETA, seg_pos={0 0}, upload=29117]: Done uploading piece. err: <nil> Oct 31 20:51:14.050118539 (33h10m45.745967292s) [noise=true , address=145.79.227.23:4546 , piece_id=XUIN6VBEE64LCSA5UNYXSN3BR74IL6D5VVEPA2KELRN4HYFRPIIA, seg_pos={0 0}, upload=29117]: Done uploading piece. err: <nil> Oct 31 20:51:14.065628894 (33h10m45.761477637s) [noise=true , address=145.79.227.28:4725 , piece_id=X2NUNP6LLBGDB2B2ZIT5MBW77PFJ3AZHJEPUDW3JWZ4CXTYXN6PQ, seg_pos={0 0}, upload=29117]: Done uploading piece. err: <nil> Oct 31 20:51:14.079898897 (33h10m45.77574765s) [noise=true , address=145.79.227.31:4799 , piece_id=M2TGJ7FO5UPK5F32KJWOB4J2YAWF5QYB2MLI3CSLVZQXLSOLPOZQ, seg_pos={0 0}, upload=29117]: Done uploading piece. err: <nil> Oct 31 20:51:41.002915508 (33h11m12.698764281s) [noise=true , address=77.90.31.12:4592 , piece_id=PW52BFRZZDXQWEXJUBDYNPSU6QGDV7MEJKUM32XKHJIQPQUHAH7A, seg_pos={0 0}, upload=29117]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: write tcp 188.166.225.170:47064->77.90.31.12:4592: use of closed network connection; write tcp 188.166.225.170:47064->77.90.31.12:4592: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47064->77.90.31.12:4592: use of closed network connection Oct 31 20:51:41.003131980 (33h11m12.698980723s) [noise=true , address=77.90.31.10:4519 , piece_id=XSDLPRAYDE7OPRTJBYSMM3YZNGEPLVAPN5YYKBJNBDGXALNM2ISA, seg_pos={0 0}, upload=29117]: Done uploading piece. err: ecclient: upload failed (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, address:77.90.31.10:4519): protocol: write tcp 188.166.225.170:37294->77.90.31.10:4519: use of closed network connection; write tcp 188.166.225.170:37294->77.90.31.10:4519: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37294->77.90.31.10:4519: use of closed network connection Oct 31 20:51:41.003350967 (33h11m12.69919973s) [noise=true , address=77.90.31.13:4626 , piece_id=LP26J4GFG7AMZALJX6UVH4RBYCLWESEDNVF2ZJSNLEDYOFCSDSIQ, seg_pos={0 0}, upload=29117]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: write tcp 188.166.225.170:59286->77.90.31.13:4626: use of closed network connection; write tcp 188.166.225.170:59286->77.90.31.13:4626: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59286->77.90.31.13:4626: use of closed network connection Oct 31 20:51:41.003851250 (33h11m12.699700003s) [noise=true , address=77.90.31.18:4807 , piece_id=7GNYW2VEL4YS5PTCRGICZEVNDGLVIBNATAU6DGLMLQWVBHAVJQRQ, seg_pos={0 0}, upload=29117]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: read tcp 188.166.225.170:49810->77.90.31.18:4807: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 20:51:41.003903128 (33h11m12.699751891s) [seg_pos={0 0}, upload=29117]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Oct 31 20:51:41.248997100 (33h11m12.944845853s) [upload=29117]: Done uploading segments... Oct 31 20:51:41.249037957 (33h11m12.944886711s) [upload=29117]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.001017968s core uplink us-select-1
checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect) antiblip: 1 out of 15 is failed (threshold=6) [X..............] duration: 321.894604ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/dcs-satellites trace: https://jaeger.dev.storj.io/trace/48774610bbd09e70 FAIL (2025-10-31 20:58:47 UTC) orbiter warning: Get "https://www.storj.io/dcs-satellites": read tcp [2400:6180:0:d0::f70:8001]:38346->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 321.923699ms core website
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.............................................X....X.........] duration: 6.173490306s trace: https://jaeger.dev.storj.io/trace/35569b98f076d00a FAIL (2025-10-31 20:58:46 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.173514101s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.........................................XX....X............] duration: 6.103872643s trace: https://jaeger.dev.storj.io/trace/61feae49d47bc60e FAIL (2025-10-31 20:58:06 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Oct 31 20:51:11.706455020 (33h10m43.402303773s) [upload=29117]: Starting upload Oct 31 20:51:11.706524262 (33h10m43.402373005s) [upload=29117]: Done starting upload Oct 31 20:51:11.706833080 (33h10m43.402681823s) [upload=29117]: Uploading segments... Oct 31 20:51:11.707027190 (33h10m43.402875943s) [upload=29117]: Got next segment. Inline: false Oct 31 20:51:11.707074009 (33h10m43.402922762s) [upload=29117]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 31 20:51:11.707095790 (33h10m43.402944544s) [upload=29117]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 31 20:51:11.959151951 (33h10m43.655000704s) [seg_pos={0 0}, upload=29117]: Begin upload segment... Oct 31 20:51:11.959219510 (33h10m43.655068263s) [seg_pos={0 0}, upload=29117]: Done begin upload segment. Oct 31 20:51:11.959257482 (33h10m43.655106235s) [noise=true , address=145.79.227.13:4803 , piece_id=U3TRIALHSTLPGF44OTFLLTHPYHM2LU57DYF2ONNQSPFZSFJT3JGA, seg_pos={0 0}, upload=29117]: Uploading piece... Oct 31 20:51:11.959352193 (33h10m43.655200936s) [noise=true , address=145.79.227.31:4797 , piece_id=RTY2YTW7PA34HHO6NTN7WDGFUZTU5SF3B2TPDH76Z5HLZRRPRXUA, seg_pos={0 0}, upload=29117]: Uploading piece... Oct 31 20:51:11.959984507 (33h10m43.65583325s) [noise=true , address=145.79.227.21:4475 , piece_id=X47C7EP73DAXLRBDFDNLYLBRVTUJ7E25D6HE2BKNNFJCYWE2GETA, seg_pos={0 0}, upload=29117]: Uploading piece... Oct 31 20:51:11.960349932 (33h10m43.656198685s) [noise=true , address=145.79.227.28:4725 , piece_id=X2NUNP6LLBGDB2B2ZIT5MBW77PFJ3AZHJEPUDW3JWZ4CXTYXN6PQ, seg_pos={0 0}, upload=29117]: Uploading piece... Oct 31 20:51:11.960434654 (33h10m43.656283397s) [noise=true , address=145.79.227.31:4799 , piece_id=M2TGJ7FO5UPK5F32KJWOB4J2YAWF5QYB2MLI3CSLVZQXLSOLPOZQ, seg_pos={0 0}, upload=29117]: Uploading piece... Oct 31 20:51:11.960753571 (33h10m43.656602314s) [noise=true , address=145.79.227.8:4331 , piece_id=WBJEFDSQWI57LUVOFU46SW55X5A4TMAQDQYQUJBSL7PUK76BEWFA, seg_pos={0 0}, upload=29117]: Uploading piece... Oct 31 20:51:11.960782867 (33h10m43.65663162s) [noise=true , address=145.79.227.4:4185 , piece_id=F7WTGPH6LJR7KKPKLEITSPS24KRXGBYAPVBH2A7RVCPCQZQKVKFA, seg_pos={0 0}, upload=29117]: Uploading piece... Oct 31 20:51:11.961093809 (33h10m43.656942562s) [noise=true , address=77.90.31.18:4807 , piece_id=7GNYW2VEL4YS5PTCRGICZEVNDGLVIBNATAU6DGLMLQWVBHAVJQRQ, seg_pos={0 0}, upload=29117]: Uploading piece... Oct 31 20:51:11.961223666 (33h10m43.657072409s) [noise=true , address=145.79.227.32:4836 , piece_id=PYCFWHO5ACF33RG5CXXTUURHU5D3XHXFH7GWQXGCULB3LQCABCRQ, seg_pos={0 0}, upload=29117]: Uploading piece... Oct 31 20:51:11.961396354 (33h10m43.657245097s) [noise=true , address=77.90.31.13:4626 , piece_id=LP26J4GFG7AMZALJX6UVH4RBYCLWESEDNVF2ZJSNLEDYOFCSDSIQ, seg_pos={0 0}, upload=29117]: Uploading piece... Oct 31 20:51:11.961606384 (33h10m43.657455127s) [noise=true , address=145.79.227.9:4366 , piece_id=B3MDET2NK75LH2PFOTQQQKMTOORYPKBIPYMVDMRLAPALMFHMMTUQ, seg_pos={0 0}, upload=29117]: Uploading piece... Oct 31 20:51:11.961727265 (33h10m43.657576008s) [noise=true , address=145.79.227.23:4546 , piece_id=XUIN6VBEE64LCSA5UNYXSN3BR74IL6D5VVEPA2KELRN4HYFRPIIA, seg_pos={0 0}, upload=29117]: Uploading piece... Oct 31 20:51:11.962038738 (33h10m43.657887481s) [noise=true , address=77.90.31.12:4592 , piece_id=PW52BFRZZDXQWEXJUBDYNPSU6QGDV7MEJKUM32XKHJIQPQUHAH7A, seg_pos={0 0}, upload=29117]: Uploading piece... Oct 31 20:51:11.962095516 (33h10m43.657944259s) [noise=true , address=145.79.227.24:4581 , piece_id=5QMNPMLJAQXITYKVAMORL5C2ETAVDLAGYUFMFLXI4SJ3HK67LXKA, seg_pos={0 0}, upload=29117]: Uploading piece... Oct 31 20:51:11.962430774 (33h10m43.658279517s) [noise=true , address=77.90.31.10:4519 , piece_id=XSDLPRAYDE7OPRTJBYSMM3YZNGEPLVAPN5YYKBJNBDGXALNM2ISA, seg_pos={0 0}, upload=29117]: Uploading piece... Oct 31 20:51:11.962491260 (33h10m43.658340013s) [noise=true , address=145.79.227.14:4840 , piece_id=NSQMPI3HHFYPKRTICMDIAKLP4OYYXK3PUKWAQWGDUKXIMMAET26Q, seg_pos={0 0}, upload=29117]: Uploading piece... Oct 31 20:51:13.589873545 (33h10m45.285722288s) [upload=29117]: Next returned nil segment Oct 31 20:51:13.589889816 (33h10m45.285738569s) [upload=29117]: Waiting for error group managing segments... Oct 31 20:51:13.973997709 (33h10m45.669846462s) [noise=true , address=145.79.227.9:4366 , piece_id=B3MDET2NK75LH2PFOTQQQKMTOORYPKBIPYMVDMRLAPALMFHMMTUQ, seg_pos={0 0}, upload=29117]: Done uploading piece. err: <nil> Oct 31 20:51:13.977395970 (33h10m45.673244713s) [noise=true , address=145.79.227.8:4331 , piece_id=WBJEFDSQWI57LUVOFU46SW55X5A4TMAQDQYQUJBSL7PUK76BEWFA, seg_pos={0 0}, upload=29117]: Done uploading piece. err: <nil> Oct 31 20:51:13.979447918 (33h10m45.675296671s) [noise=true , address=145.79.227.14:4840 , piece_id=NSQMPI3HHFYPKRTICMDIAKLP4OYYXK3PUKWAQWGDUKXIMMAET26Q, seg_pos={0 0}, upload=29117]: Done uploading piece. err: <nil> Oct 31 20:51:13.980901436 (33h10m45.676750189s) [noise=true , address=145.79.227.31:4797 , piece_id=RTY2YTW7PA34HHO6NTN7WDGFUZTU5SF3B2TPDH76Z5HLZRRPRXUA, seg_pos={0 0}, upload=29117]: Done uploading piece. err: <nil> Oct 31 20:51:13.986457284 (33h10m45.682306088s) [noise=true , address=145.79.227.24:4581 , piece_id=5QMNPMLJAQXITYKVAMORL5C2ETAVDLAGYUFMFLXI4SJ3HK67LXKA, seg_pos={0 0}, upload=29117]: Done uploading piece. err: <nil> Oct 31 20:51:14.004274321 (33h10m45.700123084s) [noise=true , address=145.79.227.13:4803 , piece_id=U3TRIALHSTLPGF44OTFLLTHPYHM2LU57DYF2ONNQSPFZSFJT3JGA, seg_pos={0 0}, upload=29117]: Done uploading piece. err: <nil> Oct 31 20:51:14.012149448 (33h10m45.707998201s) [noise=true , address=145.79.227.4:4185 , piece_id=F7WTGPH6LJR7KKPKLEITSPS24KRXGBYAPVBH2A7RVCPCQZQKVKFA, seg_pos={0 0}, upload=29117]: Done uploading piece. err: <nil> Oct 31 20:51:14.027225746 (33h10m45.723074499s) [noise=true , address=145.79.227.32:4836 , piece_id=PYCFWHO5ACF33RG5CXXTUURHU5D3XHXFH7GWQXGCULB3LQCABCRQ, seg_pos={0 0}, upload=29117]: Done uploading piece. err: <nil> Oct 31 20:51:14.043308651 (33h10m45.739157404s) [noise=true , address=145.79.227.21:4475 , piece_id=X47C7EP73DAXLRBDFDNLYLBRVTUJ7E25D6HE2BKNNFJCYWE2GETA, seg_pos={0 0}, upload=29117]: Done uploading piece. err: <nil> Oct 31 20:51:14.050118539 (33h10m45.745967292s) [noise=true , address=145.79.227.23:4546 , piece_id=XUIN6VBEE64LCSA5UNYXSN3BR74IL6D5VVEPA2KELRN4HYFRPIIA, seg_pos={0 0}, upload=29117]: Done uploading piece. err: <nil> Oct 31 20:51:14.065628894 (33h10m45.761477637s) [noise=true , address=145.79.227.28:4725 , piece_id=X2NUNP6LLBGDB2B2ZIT5MBW77PFJ3AZHJEPUDW3JWZ4CXTYXN6PQ, seg_pos={0 0}, upload=29117]: Done uploading piece. err: <nil> Oct 31 20:51:14.079898897 (33h10m45.77574765s) [noise=true , address=145.79.227.31:4799 , piece_id=M2TGJ7FO5UPK5F32KJWOB4J2YAWF5QYB2MLI3CSLVZQXLSOLPOZQ, seg_pos={0 0}, upload=29117]: Done uploading piece. err: <nil> Oct 31 20:51:41.002915508 (33h11m12.698764281s) [noise=true , address=77.90.31.12:4592 , piece_id=PW52BFRZZDXQWEXJUBDYNPSU6QGDV7MEJKUM32XKHJIQPQUHAH7A, seg_pos={0 0}, upload=29117]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: write tcp 188.166.225.170:47064->77.90.31.12:4592: use of closed network connection; write tcp 188.166.225.170:47064->77.90.31.12:4592: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47064->77.90.31.12:4592: use of closed network connection Oct 31 20:51:41.003131980 (33h11m12.698980723s) [noise=true , address=77.90.31.10:4519 , piece_id=XSDLPRAYDE7OPRTJBYSMM3YZNGEPLVAPN5YYKBJNBDGXALNM2ISA, seg_pos={0 0}, upload=29117]: Done uploading piece. err: ecclient: upload failed (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, address:77.90.31.10:4519): protocol: write tcp 188.166.225.170:37294->77.90.31.10:4519: use of closed network connection; write tcp 188.166.225.170:37294->77.90.31.10:4519: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37294->77.90.31.10:4519: use of closed network connection Oct 31 20:51:41.003350967 (33h11m12.69919973s) [noise=true , address=77.90.31.13:4626 , piece_id=LP26J4GFG7AMZALJX6UVH4RBYCLWESEDNVF2ZJSNLEDYOFCSDSIQ, seg_pos={0 0}, upload=29117]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: write tcp 188.166.225.170:59286->77.90.31.13:4626: use of closed network connection; write tcp 188.166.225.170:59286->77.90.31.13:4626: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59286->77.90.31.13:4626: use of closed network connection Oct 31 20:51:41.003851250 (33h11m12.699700003s) [noise=true , address=77.90.31.18:4807 , piece_id=7GNYW2VEL4YS5PTCRGICZEVNDGLVIBNATAU6DGLMLQWVBHAVJQRQ, seg_pos={0 0}, upload=29117]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: read tcp 188.166.225.170:49810->77.90.31.18:4807: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 20:51:41.003903128 (33h11m12.699751891s) [seg_pos={0 0}, upload=29117]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Oct 31 20:51:41.248997100 (33h11m12.944845853s) [upload=29117]: Done uploading segments... Oct 31 20:51:41.249037957 (33h11m12.944886711s) [upload=29117]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.10391868s core uplink us-select-1
checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect) antiblip: 1 out of 15 is failed (threshold=6) [.X.............] duration: 326.188269ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/dcs-satellites trace: https://jaeger.dev.storj.io/trace/48774610bbd09e70 FAIL (2025-10-31 20:58:03 UTC) orbiter warning: Get "https://www.storj.io/dcs-satellites": read tcp [2400:6180:0:d0::f70:8001]:38346->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 326.23128ms core website
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..............................................X....X........] duration: 6.216801665s trace: https://jaeger.dev.storj.io/trace/35569b98f076d00a FAIL (2025-10-31 20:57:51 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.216875395s edge gateway us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect)

checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect)

checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect)