Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [........X................................X..................] duration: 3.363931593s trace: https://jaeger.dev.storj.io/trace/275e9b01e747ebf4 FAIL (2025-11-29 14:53:16 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.363964446s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 6.000261184s trace: https://jaeger.dev.storj.io/trace/4add906a25eb3f89 FAIL (2025-11-29 14:52:55 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Nov 29 14:34:17.941236229 (486h30m16.49270669s) [upload=428397]: Got next segment. Inline: false Nov 29 14:34:17.941289010 (486h30m16.492759471s) [upload=428397]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 29 14:34:17.941296063 (486h30m16.492766524s) [upload=428397]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 29 14:34:18.189160800 (486h30m16.740631261s) [seg_pos={0 0}, upload=428397]: Begin upload segment... Nov 29 14:34:18.189218760 (486h30m16.740689221s) [seg_pos={0 0}, upload=428397]: Done begin upload segment. Nov 29 14:34:18.189254097 (486h30m16.740724548s) [noise=true , address=145.79.227.40:4049 , piece_id=DIBJ3Y4NA3HRAZD7GLO72STJHS5XHA3OTCAMWL4VCHMEILLCL3UA, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.189472724 (486h30m16.740943185s) [noise=true , address=77.90.31.13:4627 , piece_id=62A76IX5BE7WFBCCEMPSENCE5PWOSEMCUIKEATUPWGSLTCFJ4BGQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.190037069 (486h30m16.74150753s) [noise=true , address=145.79.227.2:4114 , piece_id=ZB7V2MM6KI6VTSGSC4ZD4QI6EA2265OP3MVLWD2AYAHFERAUE7CQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.190276184 (486h30m16.741746645s) [noise=true , address=145.79.227.21:4473 , piece_id=S2V3HONDHED6NZSIFZOTWU2DQ6FKESGGZ4EVMZX36LI4NT32WDWQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.190487406 (486h30m16.741957867s) [noise=true , address=145.79.227.31:4798 , piece_id=NG2JWZTD5NPFNDIVWYSWFHQHEDTURKJOY6YZ7G6LAQRIDI7WD3FQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.190825751 (486h30m16.742296212s) [noise=true , address=45.91.116.5:4297 , piece_id=GD7HY7SUS6XPN6KJDC5AVRP2ECTJSB2MK4YBQXCNUOTUBVKFRGVQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.190967721 (486h30m16.742438192s) [noise=true , address=145.79.227.24:4583 , piece_id=B6GTB3UF3XX64M5U57FFCFAMWNWDJUDPIZDG23KOHXOVCFTGKCLA, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.191308490 (486h30m16.742778951s) [noise=true , address=145.79.227.33:4869 , piece_id=V3JJCDM7EBRRZL2GPDMSCQAEHP2YX2QXDTDZDT5M6FGVLB27G7PQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.191365077 (486h30m16.742835528s) [noise=true , address=145.79.227.6:4259 , piece_id=KIYGYFEAYWG2NUAQZNWG4ZLMRY6GFJADI5LD635JE5M4DLHTX4TQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.191782502 (486h30m16.743252963s) [noise=true , address=145.79.227.22:4509 , piece_id=N2AEKOF7MLRQZ2HS7FSXV3U6WNI7YQLWP3W4PTFZLMAWQQREBTKQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.192449502 (486h30m16.743919953s) [noise=true , address=145.79.227.14:4839 , piece_id=RDZDYLJRCVHDKTTCAF3JB5ULPKFMSC2E4HNLB2RPLPY2GTUC6ZDQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.193091886 (486h30m16.744562347s) [noise=true , address=145.79.227.12:4767 , piece_id=5Q27HBFWOOI6PIIIMXELWG3QYSEA5B5LAOMU5K4FICMUMPDZIQAA, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.193463874 (486h30m16.744934335s) [noise=true , address=145.79.227.39:4013 , piece_id=YFEKNDHNI77M7MG76EISOGVU32UOZ536CK4X3Z6SFGOLJV2E7EHQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.195162279 (486h30m16.74663274s) [noise=true , address=145.79.227.20:4438 , piece_id=VZK574G5G2HXO5JMVMMEKGFCOJKOTU5KUIJDDEROFXRYTBTITO3A, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.197119445 (486h30m16.748589896s) [noise=true , address=145.79.227.7:4295 , piece_id=OLB3PUIQ4RYD2GGVF4I32FEGXO2O4X56PLJMOKNTJVBK53KZ3VYQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.197903478 (486h30m16.749373929s) [noise=true , address=145.79.227.32:4836 , piece_id=4KBSAAZLHC7OYLWN7RBDTQZUUTMK4HFGFUBES7K2PDKBXZST3OBA, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:19.999395950 (486h30m18.550866411s) [upload=428397]: Next returned nil segment Nov 29 14:34:19.999415177 (486h30m18.550885648s) [upload=428397]: Waiting for error group managing segments... Nov 29 14:34:20.285725942 (486h30m18.837196404s) [noise=true , address=77.90.31.13:4627 , piece_id=62A76IX5BE7WFBCCEMPSENCE5PWOSEMCUIKEATUPWGSLTCFJ4BGQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:20.363243793 (486h30m18.914714244s) [noise=true , address=45.91.116.5:4297 , piece_id=GD7HY7SUS6XPN6KJDC5AVRP2ECTJSB2MK4YBQXCNUOTUBVKFRGVQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:22.551752465 (486h30m21.103222926s) [noise=true , address=145.79.227.31:4798 , piece_id=NG2JWZTD5NPFNDIVWYSWFHQHEDTURKJOY6YZ7G6LAQRIDI7WD3FQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:22.666545505 (486h30m21.218015967s) [noise=true , address=145.79.227.22:4509 , piece_id=N2AEKOF7MLRQZ2HS7FSXV3U6WNI7YQLWP3W4PTFZLMAWQQREBTKQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:23.006352209 (486h30m21.557822671s) [noise=true , address=145.79.227.14:4839 , piece_id=RDZDYLJRCVHDKTTCAF3JB5ULPKFMSC2E4HNLB2RPLPY2GTUC6ZDQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:23.023700063 (486h30m21.575170525s) [noise=true , address=145.79.227.33:4869 , piece_id=V3JJCDM7EBRRZL2GPDMSCQAEHP2YX2QXDTDZDT5M6FGVLB27G7PQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:23.426873130 (486h30m21.978343581s) [noise=true , address=145.79.227.32:4836 , piece_id=4KBSAAZLHC7OYLWN7RBDTQZUUTMK4HFGFUBES7K2PDKBXZST3OBA, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:23.722460741 (486h30m22.273931202s) [noise=true , address=145.79.227.40:4049 , piece_id=DIBJ3Y4NA3HRAZD7GLO72STJHS5XHA3OTCAMWL4VCHMEILLCL3UA, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:25.698494096 (486h30m24.249964557s) [noise=true , address=145.79.227.39:4013 , piece_id=YFEKNDHNI77M7MG76EISOGVU32UOZ536CK4X3Z6SFGOLJV2E7EHQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:25.822411189 (486h30m24.373881651s) [noise=true , address=145.79.227.2:4114 , piece_id=ZB7V2MM6KI6VTSGSC4ZD4QI6EA2265OP3MVLWD2AYAHFERAUE7CQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:29.722449985 (486h30m28.273920436s) [noise=true , address=145.79.227.12:4767 , piece_id=5Q27HBFWOOI6PIIIMXELWG3QYSEA5B5LAOMU5K4FICMUMPDZIQAA, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:47.238199596 (486h30m45.789670047s) [noise=true , address=145.79.227.20:4438 , piece_id=VZK574G5G2HXO5JMVMMEKGFCOJKOTU5KUIJDDEROFXRYTBTITO3A, seg_pos={0 0}, upload=428397]: Done uploading piece. err: ecclient: upload failed (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, address:145.79.227.20:4438): protocol: write tcp 188.166.225.170:34106->145.79.227.20:4438: use of closed network connection; write tcp 188.166.225.170:34106->145.79.227.20:4438: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34106->145.79.227.20:4438: use of closed network connection Nov 29 14:34:47.238464621 (486h30m45.789935082s) [noise=true , address=145.79.227.21:4473 , piece_id=S2V3HONDHED6NZSIFZOTWU2DQ6FKESGGZ4EVMZX36LI4NT32WDWQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:46118->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:46118->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46118->145.79.227.21:4473: use of closed network connection Nov 29 14:34:47.238703104 (486h30m45.790173565s) [noise=true , address=145.79.227.24:4583 , piece_id=B6GTB3UF3XX64M5U57FFCFAMWNWDJUDPIZDG23KOHXOVCFTGKCLA, seg_pos={0 0}, upload=428397]: Done uploading piece. err: ecclient: upload failed (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, address:145.79.227.24:4583): protocol: write tcp 188.166.225.170:51236->145.79.227.24:4583: use of closed network connection; write tcp 188.166.225.170:51236->145.79.227.24:4583: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51236->145.79.227.24:4583: use of closed network connection Nov 29 14:34:47.238869100 (486h30m45.790339561s) [noise=true , address=145.79.227.7:4295 , piece_id=OLB3PUIQ4RYD2GGVF4I32FEGXO2O4X56PLJMOKNTJVBK53KZ3VYQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:145.79.227.7:4295): protocol: write tcp 188.166.225.170:46306->145.79.227.7:4295: use of closed network connection; write tcp 188.166.225.170:46306->145.79.227.7:4295: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46306->145.79.227.7:4295: use of closed network connection Nov 29 14:34:47.239032171 (486h30m45.790502632s) [noise=true , address=145.79.227.6:4259 , piece_id=KIYGYFEAYWG2NUAQZNWG4ZLMRY6GFJADI5LD635JE5M4DLHTX4TQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:145.79.227.6:4259): protocol: write tcp 188.166.225.170:48184->145.79.227.6:4259: use of closed network connection; write tcp 188.166.225.170:48184->145.79.227.6:4259: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48184->145.79.227.6:4259: use of closed network connection Nov 29 14:34:47.239076555 (486h30m45.790547027s) [seg_pos={0 0}, upload=428397]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 29 14:34:47.481897829 (486h30m46.0333683s) [upload=428397]: Done uploading segments... Nov 29 14:34:47.481943356 (486h30m46.033413817s) [upload=428397]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 6.000291682s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 6.399687816s trace: https://jaeger.dev.storj.io/trace/61a1d0645153f16 FAIL (2025-11-29 14:52:53 UTC) orbiter warning: context deadline exceeded Duration: 6.399716449s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.........X................................X.................] duration: 3.476643781s trace: https://jaeger.dev.storj.io/trace/275e9b01e747ebf4 FAIL (2025-11-29 14:52:50 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.476801021s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [..........X................................X................] duration: 3.424420302s trace: https://jaeger.dev.storj.io/trace/275e9b01e747ebf4 FAIL (2025-11-29 14:52:20 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.424446301s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 5.955671792s trace: https://jaeger.dev.storj.io/trace/4add906a25eb3f89 FAIL (2025-11-29 14:52:06 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Nov 29 14:34:17.941236229 (486h30m16.49270669s) [upload=428397]: Got next segment. Inline: false Nov 29 14:34:17.941289010 (486h30m16.492759471s) [upload=428397]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 29 14:34:17.941296063 (486h30m16.492766524s) [upload=428397]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 29 14:34:18.189160800 (486h30m16.740631261s) [seg_pos={0 0}, upload=428397]: Begin upload segment... Nov 29 14:34:18.189218760 (486h30m16.740689221s) [seg_pos={0 0}, upload=428397]: Done begin upload segment. Nov 29 14:34:18.189254097 (486h30m16.740724548s) [noise=true , address=145.79.227.40:4049 , piece_id=DIBJ3Y4NA3HRAZD7GLO72STJHS5XHA3OTCAMWL4VCHMEILLCL3UA, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.189472724 (486h30m16.740943185s) [noise=true , address=77.90.31.13:4627 , piece_id=62A76IX5BE7WFBCCEMPSENCE5PWOSEMCUIKEATUPWGSLTCFJ4BGQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.190037069 (486h30m16.74150753s) [noise=true , address=145.79.227.2:4114 , piece_id=ZB7V2MM6KI6VTSGSC4ZD4QI6EA2265OP3MVLWD2AYAHFERAUE7CQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.190276184 (486h30m16.741746645s) [noise=true , address=145.79.227.21:4473 , piece_id=S2V3HONDHED6NZSIFZOTWU2DQ6FKESGGZ4EVMZX36LI4NT32WDWQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.190487406 (486h30m16.741957867s) [noise=true , address=145.79.227.31:4798 , piece_id=NG2JWZTD5NPFNDIVWYSWFHQHEDTURKJOY6YZ7G6LAQRIDI7WD3FQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.190825751 (486h30m16.742296212s) [noise=true , address=45.91.116.5:4297 , piece_id=GD7HY7SUS6XPN6KJDC5AVRP2ECTJSB2MK4YBQXCNUOTUBVKFRGVQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.190967721 (486h30m16.742438192s) [noise=true , address=145.79.227.24:4583 , piece_id=B6GTB3UF3XX64M5U57FFCFAMWNWDJUDPIZDG23KOHXOVCFTGKCLA, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.191308490 (486h30m16.742778951s) [noise=true , address=145.79.227.33:4869 , piece_id=V3JJCDM7EBRRZL2GPDMSCQAEHP2YX2QXDTDZDT5M6FGVLB27G7PQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.191365077 (486h30m16.742835528s) [noise=true , address=145.79.227.6:4259 , piece_id=KIYGYFEAYWG2NUAQZNWG4ZLMRY6GFJADI5LD635JE5M4DLHTX4TQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.191782502 (486h30m16.743252963s) [noise=true , address=145.79.227.22:4509 , piece_id=N2AEKOF7MLRQZ2HS7FSXV3U6WNI7YQLWP3W4PTFZLMAWQQREBTKQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.192449502 (486h30m16.743919953s) [noise=true , address=145.79.227.14:4839 , piece_id=RDZDYLJRCVHDKTTCAF3JB5ULPKFMSC2E4HNLB2RPLPY2GTUC6ZDQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.193091886 (486h30m16.744562347s) [noise=true , address=145.79.227.12:4767 , piece_id=5Q27HBFWOOI6PIIIMXELWG3QYSEA5B5LAOMU5K4FICMUMPDZIQAA, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.193463874 (486h30m16.744934335s) [noise=true , address=145.79.227.39:4013 , piece_id=YFEKNDHNI77M7MG76EISOGVU32UOZ536CK4X3Z6SFGOLJV2E7EHQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.195162279 (486h30m16.74663274s) [noise=true , address=145.79.227.20:4438 , piece_id=VZK574G5G2HXO5JMVMMEKGFCOJKOTU5KUIJDDEROFXRYTBTITO3A, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.197119445 (486h30m16.748589896s) [noise=true , address=145.79.227.7:4295 , piece_id=OLB3PUIQ4RYD2GGVF4I32FEGXO2O4X56PLJMOKNTJVBK53KZ3VYQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.197903478 (486h30m16.749373929s) [noise=true , address=145.79.227.32:4836 , piece_id=4KBSAAZLHC7OYLWN7RBDTQZUUTMK4HFGFUBES7K2PDKBXZST3OBA, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:19.999395950 (486h30m18.550866411s) [upload=428397]: Next returned nil segment Nov 29 14:34:19.999415177 (486h30m18.550885648s) [upload=428397]: Waiting for error group managing segments... Nov 29 14:34:20.285725942 (486h30m18.837196404s) [noise=true , address=77.90.31.13:4627 , piece_id=62A76IX5BE7WFBCCEMPSENCE5PWOSEMCUIKEATUPWGSLTCFJ4BGQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:20.363243793 (486h30m18.914714244s) [noise=true , address=45.91.116.5:4297 , piece_id=GD7HY7SUS6XPN6KJDC5AVRP2ECTJSB2MK4YBQXCNUOTUBVKFRGVQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:22.551752465 (486h30m21.103222926s) [noise=true , address=145.79.227.31:4798 , piece_id=NG2JWZTD5NPFNDIVWYSWFHQHEDTURKJOY6YZ7G6LAQRIDI7WD3FQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:22.666545505 (486h30m21.218015967s) [noise=true , address=145.79.227.22:4509 , piece_id=N2AEKOF7MLRQZ2HS7FSXV3U6WNI7YQLWP3W4PTFZLMAWQQREBTKQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:23.006352209 (486h30m21.557822671s) [noise=true , address=145.79.227.14:4839 , piece_id=RDZDYLJRCVHDKTTCAF3JB5ULPKFMSC2E4HNLB2RPLPY2GTUC6ZDQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:23.023700063 (486h30m21.575170525s) [noise=true , address=145.79.227.33:4869 , piece_id=V3JJCDM7EBRRZL2GPDMSCQAEHP2YX2QXDTDZDT5M6FGVLB27G7PQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:23.426873130 (486h30m21.978343581s) [noise=true , address=145.79.227.32:4836 , piece_id=4KBSAAZLHC7OYLWN7RBDTQZUUTMK4HFGFUBES7K2PDKBXZST3OBA, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:23.722460741 (486h30m22.273931202s) [noise=true , address=145.79.227.40:4049 , piece_id=DIBJ3Y4NA3HRAZD7GLO72STJHS5XHA3OTCAMWL4VCHMEILLCL3UA, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:25.698494096 (486h30m24.249964557s) [noise=true , address=145.79.227.39:4013 , piece_id=YFEKNDHNI77M7MG76EISOGVU32UOZ536CK4X3Z6SFGOLJV2E7EHQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:25.822411189 (486h30m24.373881651s) [noise=true , address=145.79.227.2:4114 , piece_id=ZB7V2MM6KI6VTSGSC4ZD4QI6EA2265OP3MVLWD2AYAHFERAUE7CQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:29.722449985 (486h30m28.273920436s) [noise=true , address=145.79.227.12:4767 , piece_id=5Q27HBFWOOI6PIIIMXELWG3QYSEA5B5LAOMU5K4FICMUMPDZIQAA, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:47.238199596 (486h30m45.789670047s) [noise=true , address=145.79.227.20:4438 , piece_id=VZK574G5G2HXO5JMVMMEKGFCOJKOTU5KUIJDDEROFXRYTBTITO3A, seg_pos={0 0}, upload=428397]: Done uploading piece. err: ecclient: upload failed (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, address:145.79.227.20:4438): protocol: write tcp 188.166.225.170:34106->145.79.227.20:4438: use of closed network connection; write tcp 188.166.225.170:34106->145.79.227.20:4438: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34106->145.79.227.20:4438: use of closed network connection Nov 29 14:34:47.238464621 (486h30m45.789935082s) [noise=true , address=145.79.227.21:4473 , piece_id=S2V3HONDHED6NZSIFZOTWU2DQ6FKESGGZ4EVMZX36LI4NT32WDWQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:46118->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:46118->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46118->145.79.227.21:4473: use of closed network connection Nov 29 14:34:47.238703104 (486h30m45.790173565s) [noise=true , address=145.79.227.24:4583 , piece_id=B6GTB3UF3XX64M5U57FFCFAMWNWDJUDPIZDG23KOHXOVCFTGKCLA, seg_pos={0 0}, upload=428397]: Done uploading piece. err: ecclient: upload failed (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, address:145.79.227.24:4583): protocol: write tcp 188.166.225.170:51236->145.79.227.24:4583: use of closed network connection; write tcp 188.166.225.170:51236->145.79.227.24:4583: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51236->145.79.227.24:4583: use of closed network connection Nov 29 14:34:47.238869100 (486h30m45.790339561s) [noise=true , address=145.79.227.7:4295 , piece_id=OLB3PUIQ4RYD2GGVF4I32FEGXO2O4X56PLJMOKNTJVBK53KZ3VYQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:145.79.227.7:4295): protocol: write tcp 188.166.225.170:46306->145.79.227.7:4295: use of closed network connection; write tcp 188.166.225.170:46306->145.79.227.7:4295: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46306->145.79.227.7:4295: use of closed network connection Nov 29 14:34:47.239032171 (486h30m45.790502632s) [noise=true , address=145.79.227.6:4259 , piece_id=KIYGYFEAYWG2NUAQZNWG4ZLMRY6GFJADI5LD635JE5M4DLHTX4TQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:145.79.227.6:4259): protocol: write tcp 188.166.225.170:48184->145.79.227.6:4259: use of closed network connection; write tcp 188.166.225.170:48184->145.79.227.6:4259: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48184->145.79.227.6:4259: use of closed network connection Nov 29 14:34:47.239076555 (486h30m45.790547027s) [seg_pos={0 0}, upload=428397]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 29 14:34:47.481897829 (486h30m46.0333683s) [upload=428397]: Done uploading segments... Nov 29 14:34:47.481943356 (486h30m46.033413817s) [upload=428397]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.955705836s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 9.333906664s trace: https://jaeger.dev.storj.io/trace/61a1d0645153f16 FAIL (2025-11-29 14:52:00 UTC) orbiter warning: context deadline exceeded Duration: 9.333937784s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [...........X................................X...............] duration: 3.43090865s trace: https://jaeger.dev.storj.io/trace/275e9b01e747ebf4 FAIL (2025-11-29 14:51:47 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.430980307s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 6.014789576s trace: https://jaeger.dev.storj.io/trace/4add906a25eb3f89 FAIL (2025-11-29 14:51:35 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Nov 29 14:34:17.941236229 (486h30m16.49270669s) [upload=428397]: Got next segment. Inline: false Nov 29 14:34:17.941289010 (486h30m16.492759471s) [upload=428397]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 29 14:34:17.941296063 (486h30m16.492766524s) [upload=428397]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 29 14:34:18.189160800 (486h30m16.740631261s) [seg_pos={0 0}, upload=428397]: Begin upload segment... Nov 29 14:34:18.189218760 (486h30m16.740689221s) [seg_pos={0 0}, upload=428397]: Done begin upload segment. Nov 29 14:34:18.189254097 (486h30m16.740724548s) [noise=true , address=145.79.227.40:4049 , piece_id=DIBJ3Y4NA3HRAZD7GLO72STJHS5XHA3OTCAMWL4VCHMEILLCL3UA, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.189472724 (486h30m16.740943185s) [noise=true , address=77.90.31.13:4627 , piece_id=62A76IX5BE7WFBCCEMPSENCE5PWOSEMCUIKEATUPWGSLTCFJ4BGQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.190037069 (486h30m16.74150753s) [noise=true , address=145.79.227.2:4114 , piece_id=ZB7V2MM6KI6VTSGSC4ZD4QI6EA2265OP3MVLWD2AYAHFERAUE7CQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.190276184 (486h30m16.741746645s) [noise=true , address=145.79.227.21:4473 , piece_id=S2V3HONDHED6NZSIFZOTWU2DQ6FKESGGZ4EVMZX36LI4NT32WDWQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.190487406 (486h30m16.741957867s) [noise=true , address=145.79.227.31:4798 , piece_id=NG2JWZTD5NPFNDIVWYSWFHQHEDTURKJOY6YZ7G6LAQRIDI7WD3FQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.190825751 (486h30m16.742296212s) [noise=true , address=45.91.116.5:4297 , piece_id=GD7HY7SUS6XPN6KJDC5AVRP2ECTJSB2MK4YBQXCNUOTUBVKFRGVQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.190967721 (486h30m16.742438192s) [noise=true , address=145.79.227.24:4583 , piece_id=B6GTB3UF3XX64M5U57FFCFAMWNWDJUDPIZDG23KOHXOVCFTGKCLA, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.191308490 (486h30m16.742778951s) [noise=true , address=145.79.227.33:4869 , piece_id=V3JJCDM7EBRRZL2GPDMSCQAEHP2YX2QXDTDZDT5M6FGVLB27G7PQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.191365077 (486h30m16.742835528s) [noise=true , address=145.79.227.6:4259 , piece_id=KIYGYFEAYWG2NUAQZNWG4ZLMRY6GFJADI5LD635JE5M4DLHTX4TQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.191782502 (486h30m16.743252963s) [noise=true , address=145.79.227.22:4509 , piece_id=N2AEKOF7MLRQZ2HS7FSXV3U6WNI7YQLWP3W4PTFZLMAWQQREBTKQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.192449502 (486h30m16.743919953s) [noise=true , address=145.79.227.14:4839 , piece_id=RDZDYLJRCVHDKTTCAF3JB5ULPKFMSC2E4HNLB2RPLPY2GTUC6ZDQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.193091886 (486h30m16.744562347s) [noise=true , address=145.79.227.12:4767 , piece_id=5Q27HBFWOOI6PIIIMXELWG3QYSEA5B5LAOMU5K4FICMUMPDZIQAA, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.193463874 (486h30m16.744934335s) [noise=true , address=145.79.227.39:4013 , piece_id=YFEKNDHNI77M7MG76EISOGVU32UOZ536CK4X3Z6SFGOLJV2E7EHQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.195162279 (486h30m16.74663274s) [noise=true , address=145.79.227.20:4438 , piece_id=VZK574G5G2HXO5JMVMMEKGFCOJKOTU5KUIJDDEROFXRYTBTITO3A, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.197119445 (486h30m16.748589896s) [noise=true , address=145.79.227.7:4295 , piece_id=OLB3PUIQ4RYD2GGVF4I32FEGXO2O4X56PLJMOKNTJVBK53KZ3VYQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.197903478 (486h30m16.749373929s) [noise=true , address=145.79.227.32:4836 , piece_id=4KBSAAZLHC7OYLWN7RBDTQZUUTMK4HFGFUBES7K2PDKBXZST3OBA, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:19.999395950 (486h30m18.550866411s) [upload=428397]: Next returned nil segment Nov 29 14:34:19.999415177 (486h30m18.550885648s) [upload=428397]: Waiting for error group managing segments... Nov 29 14:34:20.285725942 (486h30m18.837196404s) [noise=true , address=77.90.31.13:4627 , piece_id=62A76IX5BE7WFBCCEMPSENCE5PWOSEMCUIKEATUPWGSLTCFJ4BGQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:20.363243793 (486h30m18.914714244s) [noise=true , address=45.91.116.5:4297 , piece_id=GD7HY7SUS6XPN6KJDC5AVRP2ECTJSB2MK4YBQXCNUOTUBVKFRGVQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:22.551752465 (486h30m21.103222926s) [noise=true , address=145.79.227.31:4798 , piece_id=NG2JWZTD5NPFNDIVWYSWFHQHEDTURKJOY6YZ7G6LAQRIDI7WD3FQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:22.666545505 (486h30m21.218015967s) [noise=true , address=145.79.227.22:4509 , piece_id=N2AEKOF7MLRQZ2HS7FSXV3U6WNI7YQLWP3W4PTFZLMAWQQREBTKQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:23.006352209 (486h30m21.557822671s) [noise=true , address=145.79.227.14:4839 , piece_id=RDZDYLJRCVHDKTTCAF3JB5ULPKFMSC2E4HNLB2RPLPY2GTUC6ZDQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:23.023700063 (486h30m21.575170525s) [noise=true , address=145.79.227.33:4869 , piece_id=V3JJCDM7EBRRZL2GPDMSCQAEHP2YX2QXDTDZDT5M6FGVLB27G7PQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:23.426873130 (486h30m21.978343581s) [noise=true , address=145.79.227.32:4836 , piece_id=4KBSAAZLHC7OYLWN7RBDTQZUUTMK4HFGFUBES7K2PDKBXZST3OBA, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:23.722460741 (486h30m22.273931202s) [noise=true , address=145.79.227.40:4049 , piece_id=DIBJ3Y4NA3HRAZD7GLO72STJHS5XHA3OTCAMWL4VCHMEILLCL3UA, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:25.698494096 (486h30m24.249964557s) [noise=true , address=145.79.227.39:4013 , piece_id=YFEKNDHNI77M7MG76EISOGVU32UOZ536CK4X3Z6SFGOLJV2E7EHQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:25.822411189 (486h30m24.373881651s) [noise=true , address=145.79.227.2:4114 , piece_id=ZB7V2MM6KI6VTSGSC4ZD4QI6EA2265OP3MVLWD2AYAHFERAUE7CQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:29.722449985 (486h30m28.273920436s) [noise=true , address=145.79.227.12:4767 , piece_id=5Q27HBFWOOI6PIIIMXELWG3QYSEA5B5LAOMU5K4FICMUMPDZIQAA, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:47.238199596 (486h30m45.789670047s) [noise=true , address=145.79.227.20:4438 , piece_id=VZK574G5G2HXO5JMVMMEKGFCOJKOTU5KUIJDDEROFXRYTBTITO3A, seg_pos={0 0}, upload=428397]: Done uploading piece. err: ecclient: upload failed (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, address:145.79.227.20:4438): protocol: write tcp 188.166.225.170:34106->145.79.227.20:4438: use of closed network connection; write tcp 188.166.225.170:34106->145.79.227.20:4438: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34106->145.79.227.20:4438: use of closed network connection Nov 29 14:34:47.238464621 (486h30m45.789935082s) [noise=true , address=145.79.227.21:4473 , piece_id=S2V3HONDHED6NZSIFZOTWU2DQ6FKESGGZ4EVMZX36LI4NT32WDWQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:46118->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:46118->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46118->145.79.227.21:4473: use of closed network connection Nov 29 14:34:47.238703104 (486h30m45.790173565s) [noise=true , address=145.79.227.24:4583 , piece_id=B6GTB3UF3XX64M5U57FFCFAMWNWDJUDPIZDG23KOHXOVCFTGKCLA, seg_pos={0 0}, upload=428397]: Done uploading piece. err: ecclient: upload failed (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, address:145.79.227.24:4583): protocol: write tcp 188.166.225.170:51236->145.79.227.24:4583: use of closed network connection; write tcp 188.166.225.170:51236->145.79.227.24:4583: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51236->145.79.227.24:4583: use of closed network connection Nov 29 14:34:47.238869100 (486h30m45.790339561s) [noise=true , address=145.79.227.7:4295 , piece_id=OLB3PUIQ4RYD2GGVF4I32FEGXO2O4X56PLJMOKNTJVBK53KZ3VYQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:145.79.227.7:4295): protocol: write tcp 188.166.225.170:46306->145.79.227.7:4295: use of closed network connection; write tcp 188.166.225.170:46306->145.79.227.7:4295: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46306->145.79.227.7:4295: use of closed network connection Nov 29 14:34:47.239032171 (486h30m45.790502632s) [noise=true , address=145.79.227.6:4259 , piece_id=KIYGYFEAYWG2NUAQZNWG4ZLMRY6GFJADI5LD635JE5M4DLHTX4TQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:145.79.227.6:4259): protocol: write tcp 188.166.225.170:48184->145.79.227.6:4259: use of closed network connection; write tcp 188.166.225.170:48184->145.79.227.6:4259: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48184->145.79.227.6:4259: use of closed network connection Nov 29 14:34:47.239076555 (486h30m45.790547027s) [seg_pos={0 0}, upload=428397]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 29 14:34:47.481897829 (486h30m46.0333683s) [upload=428397]: Done uploading segments... Nov 29 14:34:47.481943356 (486h30m46.033413817s) [upload=428397]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 6.014825494s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 10.970821248s trace: https://jaeger.dev.storj.io/trace/61a1d0645153f16 FAIL (2025-11-29 14:51:20 UTC) orbiter warning: context deadline exceeded Duration: 10.970867226s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 6.421135853s trace: https://jaeger.dev.storj.io/trace/4add906a25eb3f89 FAIL (2025-11-29 14:51:13 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Nov 29 14:34:17.941236229 (486h30m16.49270669s) [upload=428397]: Got next segment. Inline: false Nov 29 14:34:17.941289010 (486h30m16.492759471s) [upload=428397]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 29 14:34:17.941296063 (486h30m16.492766524s) [upload=428397]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 29 14:34:18.189160800 (486h30m16.740631261s) [seg_pos={0 0}, upload=428397]: Begin upload segment... Nov 29 14:34:18.189218760 (486h30m16.740689221s) [seg_pos={0 0}, upload=428397]: Done begin upload segment. Nov 29 14:34:18.189254097 (486h30m16.740724548s) [noise=true , address=145.79.227.40:4049 , piece_id=DIBJ3Y4NA3HRAZD7GLO72STJHS5XHA3OTCAMWL4VCHMEILLCL3UA, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.189472724 (486h30m16.740943185s) [noise=true , address=77.90.31.13:4627 , piece_id=62A76IX5BE7WFBCCEMPSENCE5PWOSEMCUIKEATUPWGSLTCFJ4BGQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.190037069 (486h30m16.74150753s) [noise=true , address=145.79.227.2:4114 , piece_id=ZB7V2MM6KI6VTSGSC4ZD4QI6EA2265OP3MVLWD2AYAHFERAUE7CQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.190276184 (486h30m16.741746645s) [noise=true , address=145.79.227.21:4473 , piece_id=S2V3HONDHED6NZSIFZOTWU2DQ6FKESGGZ4EVMZX36LI4NT32WDWQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.190487406 (486h30m16.741957867s) [noise=true , address=145.79.227.31:4798 , piece_id=NG2JWZTD5NPFNDIVWYSWFHQHEDTURKJOY6YZ7G6LAQRIDI7WD3FQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.190825751 (486h30m16.742296212s) [noise=true , address=45.91.116.5:4297 , piece_id=GD7HY7SUS6XPN6KJDC5AVRP2ECTJSB2MK4YBQXCNUOTUBVKFRGVQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.190967721 (486h30m16.742438192s) [noise=true , address=145.79.227.24:4583 , piece_id=B6GTB3UF3XX64M5U57FFCFAMWNWDJUDPIZDG23KOHXOVCFTGKCLA, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.191308490 (486h30m16.742778951s) [noise=true , address=145.79.227.33:4869 , piece_id=V3JJCDM7EBRRZL2GPDMSCQAEHP2YX2QXDTDZDT5M6FGVLB27G7PQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.191365077 (486h30m16.742835528s) [noise=true , address=145.79.227.6:4259 , piece_id=KIYGYFEAYWG2NUAQZNWG4ZLMRY6GFJADI5LD635JE5M4DLHTX4TQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.191782502 (486h30m16.743252963s) [noise=true , address=145.79.227.22:4509 , piece_id=N2AEKOF7MLRQZ2HS7FSXV3U6WNI7YQLWP3W4PTFZLMAWQQREBTKQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.192449502 (486h30m16.743919953s) [noise=true , address=145.79.227.14:4839 , piece_id=RDZDYLJRCVHDKTTCAF3JB5ULPKFMSC2E4HNLB2RPLPY2GTUC6ZDQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.193091886 (486h30m16.744562347s) [noise=true , address=145.79.227.12:4767 , piece_id=5Q27HBFWOOI6PIIIMXELWG3QYSEA5B5LAOMU5K4FICMUMPDZIQAA, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.193463874 (486h30m16.744934335s) [noise=true , address=145.79.227.39:4013 , piece_id=YFEKNDHNI77M7MG76EISOGVU32UOZ536CK4X3Z6SFGOLJV2E7EHQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.195162279 (486h30m16.74663274s) [noise=true , address=145.79.227.20:4438 , piece_id=VZK574G5G2HXO5JMVMMEKGFCOJKOTU5KUIJDDEROFXRYTBTITO3A, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.197119445 (486h30m16.748589896s) [noise=true , address=145.79.227.7:4295 , piece_id=OLB3PUIQ4RYD2GGVF4I32FEGXO2O4X56PLJMOKNTJVBK53KZ3VYQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.197903478 (486h30m16.749373929s) [noise=true , address=145.79.227.32:4836 , piece_id=4KBSAAZLHC7OYLWN7RBDTQZUUTMK4HFGFUBES7K2PDKBXZST3OBA, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:19.999395950 (486h30m18.550866411s) [upload=428397]: Next returned nil segment Nov 29 14:34:19.999415177 (486h30m18.550885648s) [upload=428397]: Waiting for error group managing segments... Nov 29 14:34:20.285725942 (486h30m18.837196404s) [noise=true , address=77.90.31.13:4627 , piece_id=62A76IX5BE7WFBCCEMPSENCE5PWOSEMCUIKEATUPWGSLTCFJ4BGQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:20.363243793 (486h30m18.914714244s) [noise=true , address=45.91.116.5:4297 , piece_id=GD7HY7SUS6XPN6KJDC5AVRP2ECTJSB2MK4YBQXCNUOTUBVKFRGVQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:22.551752465 (486h30m21.103222926s) [noise=true , address=145.79.227.31:4798 , piece_id=NG2JWZTD5NPFNDIVWYSWFHQHEDTURKJOY6YZ7G6LAQRIDI7WD3FQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:22.666545505 (486h30m21.218015967s) [noise=true , address=145.79.227.22:4509 , piece_id=N2AEKOF7MLRQZ2HS7FSXV3U6WNI7YQLWP3W4PTFZLMAWQQREBTKQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:23.006352209 (486h30m21.557822671s) [noise=true , address=145.79.227.14:4839 , piece_id=RDZDYLJRCVHDKTTCAF3JB5ULPKFMSC2E4HNLB2RPLPY2GTUC6ZDQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:23.023700063 (486h30m21.575170525s) [noise=true , address=145.79.227.33:4869 , piece_id=V3JJCDM7EBRRZL2GPDMSCQAEHP2YX2QXDTDZDT5M6FGVLB27G7PQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:23.426873130 (486h30m21.978343581s) [noise=true , address=145.79.227.32:4836 , piece_id=4KBSAAZLHC7OYLWN7RBDTQZUUTMK4HFGFUBES7K2PDKBXZST3OBA, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:23.722460741 (486h30m22.273931202s) [noise=true , address=145.79.227.40:4049 , piece_id=DIBJ3Y4NA3HRAZD7GLO72STJHS5XHA3OTCAMWL4VCHMEILLCL3UA, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:25.698494096 (486h30m24.249964557s) [noise=true , address=145.79.227.39:4013 , piece_id=YFEKNDHNI77M7MG76EISOGVU32UOZ536CK4X3Z6SFGOLJV2E7EHQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:25.822411189 (486h30m24.373881651s) [noise=true , address=145.79.227.2:4114 , piece_id=ZB7V2MM6KI6VTSGSC4ZD4QI6EA2265OP3MVLWD2AYAHFERAUE7CQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:29.722449985 (486h30m28.273920436s) [noise=true , address=145.79.227.12:4767 , piece_id=5Q27HBFWOOI6PIIIMXELWG3QYSEA5B5LAOMU5K4FICMUMPDZIQAA, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:47.238199596 (486h30m45.789670047s) [noise=true , address=145.79.227.20:4438 , piece_id=VZK574G5G2HXO5JMVMMEKGFCOJKOTU5KUIJDDEROFXRYTBTITO3A, seg_pos={0 0}, upload=428397]: Done uploading piece. err: ecclient: upload failed (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, address:145.79.227.20:4438): protocol: write tcp 188.166.225.170:34106->145.79.227.20:4438: use of closed network connection; write tcp 188.166.225.170:34106->145.79.227.20:4438: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34106->145.79.227.20:4438: use of closed network connection Nov 29 14:34:47.238464621 (486h30m45.789935082s) [noise=true , address=145.79.227.21:4473 , piece_id=S2V3HONDHED6NZSIFZOTWU2DQ6FKESGGZ4EVMZX36LI4NT32WDWQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:46118->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:46118->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46118->145.79.227.21:4473: use of closed network connection Nov 29 14:34:47.238703104 (486h30m45.790173565s) [noise=true , address=145.79.227.24:4583 , piece_id=B6GTB3UF3XX64M5U57FFCFAMWNWDJUDPIZDG23KOHXOVCFTGKCLA, seg_pos={0 0}, upload=428397]: Done uploading piece. err: ecclient: upload failed (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, address:145.79.227.24:4583): protocol: write tcp 188.166.225.170:51236->145.79.227.24:4583: use of closed network connection; write tcp 188.166.225.170:51236->145.79.227.24:4583: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51236->145.79.227.24:4583: use of closed network connection Nov 29 14:34:47.238869100 (486h30m45.790339561s) [noise=true , address=145.79.227.7:4295 , piece_id=OLB3PUIQ4RYD2GGVF4I32FEGXO2O4X56PLJMOKNTJVBK53KZ3VYQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:145.79.227.7:4295): protocol: write tcp 188.166.225.170:46306->145.79.227.7:4295: use of closed network connection; write tcp 188.166.225.170:46306->145.79.227.7:4295: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46306->145.79.227.7:4295: use of closed network connection Nov 29 14:34:47.239032171 (486h30m45.790502632s) [noise=true , address=145.79.227.6:4259 , piece_id=KIYGYFEAYWG2NUAQZNWG4ZLMRY6GFJADI5LD635JE5M4DLHTX4TQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:145.79.227.6:4259): protocol: write tcp 188.166.225.170:48184->145.79.227.6:4259: use of closed network connection; write tcp 188.166.225.170:48184->145.79.227.6:4259: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48184->145.79.227.6:4259: use of closed network connection Nov 29 14:34:47.239076555 (486h30m45.790547027s) [seg_pos={0 0}, upload=428397]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 29 14:34:47.481897829 (486h30m46.0333683s) [upload=428397]: Done uploading segments... Nov 29 14:34:47.481943356 (486h30m46.033413817s) [upload=428397]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 6.421164377s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 11.066422308s trace: https://jaeger.dev.storj.io/trace/61a1d0645153f16 FAIL (2025-11-29 14:50:36 UTC) orbiter warning: context deadline exceeded Duration: 11.066450743s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [............X................................X..............] duration: 3.664476116s trace: https://jaeger.dev.storj.io/trace/275e9b01e747ebf4 FAIL (2025-11-29 14:50:33 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.664504388s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.............X................................X.............] duration: 3.328610504s trace: https://jaeger.dev.storj.io/trace/275e9b01e747ebf4 FAIL (2025-11-29 14:50:16 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.328643447s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 6.020301011s trace: https://jaeger.dev.storj.io/trace/4add906a25eb3f89 FAIL (2025-11-29 14:50:03 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Nov 29 14:34:17.941236229 (486h30m16.49270669s) [upload=428397]: Got next segment. Inline: false Nov 29 14:34:17.941289010 (486h30m16.492759471s) [upload=428397]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 29 14:34:17.941296063 (486h30m16.492766524s) [upload=428397]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 29 14:34:18.189160800 (486h30m16.740631261s) [seg_pos={0 0}, upload=428397]: Begin upload segment... Nov 29 14:34:18.189218760 (486h30m16.740689221s) [seg_pos={0 0}, upload=428397]: Done begin upload segment. Nov 29 14:34:18.189254097 (486h30m16.740724548s) [noise=true , address=145.79.227.40:4049 , piece_id=DIBJ3Y4NA3HRAZD7GLO72STJHS5XHA3OTCAMWL4VCHMEILLCL3UA, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.189472724 (486h30m16.740943185s) [noise=true , address=77.90.31.13:4627 , piece_id=62A76IX5BE7WFBCCEMPSENCE5PWOSEMCUIKEATUPWGSLTCFJ4BGQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.190037069 (486h30m16.74150753s) [noise=true , address=145.79.227.2:4114 , piece_id=ZB7V2MM6KI6VTSGSC4ZD4QI6EA2265OP3MVLWD2AYAHFERAUE7CQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.190276184 (486h30m16.741746645s) [noise=true , address=145.79.227.21:4473 , piece_id=S2V3HONDHED6NZSIFZOTWU2DQ6FKESGGZ4EVMZX36LI4NT32WDWQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.190487406 (486h30m16.741957867s) [noise=true , address=145.79.227.31:4798 , piece_id=NG2JWZTD5NPFNDIVWYSWFHQHEDTURKJOY6YZ7G6LAQRIDI7WD3FQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.190825751 (486h30m16.742296212s) [noise=true , address=45.91.116.5:4297 , piece_id=GD7HY7SUS6XPN6KJDC5AVRP2ECTJSB2MK4YBQXCNUOTUBVKFRGVQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.190967721 (486h30m16.742438192s) [noise=true , address=145.79.227.24:4583 , piece_id=B6GTB3UF3XX64M5U57FFCFAMWNWDJUDPIZDG23KOHXOVCFTGKCLA, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.191308490 (486h30m16.742778951s) [noise=true , address=145.79.227.33:4869 , piece_id=V3JJCDM7EBRRZL2GPDMSCQAEHP2YX2QXDTDZDT5M6FGVLB27G7PQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.191365077 (486h30m16.742835528s) [noise=true , address=145.79.227.6:4259 , piece_id=KIYGYFEAYWG2NUAQZNWG4ZLMRY6GFJADI5LD635JE5M4DLHTX4TQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.191782502 (486h30m16.743252963s) [noise=true , address=145.79.227.22:4509 , piece_id=N2AEKOF7MLRQZ2HS7FSXV3U6WNI7YQLWP3W4PTFZLMAWQQREBTKQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.192449502 (486h30m16.743919953s) [noise=true , address=145.79.227.14:4839 , piece_id=RDZDYLJRCVHDKTTCAF3JB5ULPKFMSC2E4HNLB2RPLPY2GTUC6ZDQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.193091886 (486h30m16.744562347s) [noise=true , address=145.79.227.12:4767 , piece_id=5Q27HBFWOOI6PIIIMXELWG3QYSEA5B5LAOMU5K4FICMUMPDZIQAA, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.193463874 (486h30m16.744934335s) [noise=true , address=145.79.227.39:4013 , piece_id=YFEKNDHNI77M7MG76EISOGVU32UOZ536CK4X3Z6SFGOLJV2E7EHQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.195162279 (486h30m16.74663274s) [noise=true , address=145.79.227.20:4438 , piece_id=VZK574G5G2HXO5JMVMMEKGFCOJKOTU5KUIJDDEROFXRYTBTITO3A, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.197119445 (486h30m16.748589896s) [noise=true , address=145.79.227.7:4295 , piece_id=OLB3PUIQ4RYD2GGVF4I32FEGXO2O4X56PLJMOKNTJVBK53KZ3VYQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.197903478 (486h30m16.749373929s) [noise=true , address=145.79.227.32:4836 , piece_id=4KBSAAZLHC7OYLWN7RBDTQZUUTMK4HFGFUBES7K2PDKBXZST3OBA, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:19.999395950 (486h30m18.550866411s) [upload=428397]: Next returned nil segment Nov 29 14:34:19.999415177 (486h30m18.550885648s) [upload=428397]: Waiting for error group managing segments... Nov 29 14:34:20.285725942 (486h30m18.837196404s) [noise=true , address=77.90.31.13:4627 , piece_id=62A76IX5BE7WFBCCEMPSENCE5PWOSEMCUIKEATUPWGSLTCFJ4BGQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:20.363243793 (486h30m18.914714244s) [noise=true , address=45.91.116.5:4297 , piece_id=GD7HY7SUS6XPN6KJDC5AVRP2ECTJSB2MK4YBQXCNUOTUBVKFRGVQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:22.551752465 (486h30m21.103222926s) [noise=true , address=145.79.227.31:4798 , piece_id=NG2JWZTD5NPFNDIVWYSWFHQHEDTURKJOY6YZ7G6LAQRIDI7WD3FQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:22.666545505 (486h30m21.218015967s) [noise=true , address=145.79.227.22:4509 , piece_id=N2AEKOF7MLRQZ2HS7FSXV3U6WNI7YQLWP3W4PTFZLMAWQQREBTKQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:23.006352209 (486h30m21.557822671s) [noise=true , address=145.79.227.14:4839 , piece_id=RDZDYLJRCVHDKTTCAF3JB5ULPKFMSC2E4HNLB2RPLPY2GTUC6ZDQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:23.023700063 (486h30m21.575170525s) [noise=true , address=145.79.227.33:4869 , piece_id=V3JJCDM7EBRRZL2GPDMSCQAEHP2YX2QXDTDZDT5M6FGVLB27G7PQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:23.426873130 (486h30m21.978343581s) [noise=true , address=145.79.227.32:4836 , piece_id=4KBSAAZLHC7OYLWN7RBDTQZUUTMK4HFGFUBES7K2PDKBXZST3OBA, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:23.722460741 (486h30m22.273931202s) [noise=true , address=145.79.227.40:4049 , piece_id=DIBJ3Y4NA3HRAZD7GLO72STJHS5XHA3OTCAMWL4VCHMEILLCL3UA, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:25.698494096 (486h30m24.249964557s) [noise=true , address=145.79.227.39:4013 , piece_id=YFEKNDHNI77M7MG76EISOGVU32UOZ536CK4X3Z6SFGOLJV2E7EHQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:25.822411189 (486h30m24.373881651s) [noise=true , address=145.79.227.2:4114 , piece_id=ZB7V2MM6KI6VTSGSC4ZD4QI6EA2265OP3MVLWD2AYAHFERAUE7CQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:29.722449985 (486h30m28.273920436s) [noise=true , address=145.79.227.12:4767 , piece_id=5Q27HBFWOOI6PIIIMXELWG3QYSEA5B5LAOMU5K4FICMUMPDZIQAA, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:47.238199596 (486h30m45.789670047s) [noise=true , address=145.79.227.20:4438 , piece_id=VZK574G5G2HXO5JMVMMEKGFCOJKOTU5KUIJDDEROFXRYTBTITO3A, seg_pos={0 0}, upload=428397]: Done uploading piece. err: ecclient: upload failed (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, address:145.79.227.20:4438): protocol: write tcp 188.166.225.170:34106->145.79.227.20:4438: use of closed network connection; write tcp 188.166.225.170:34106->145.79.227.20:4438: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34106->145.79.227.20:4438: use of closed network connection Nov 29 14:34:47.238464621 (486h30m45.789935082s) [noise=true , address=145.79.227.21:4473 , piece_id=S2V3HONDHED6NZSIFZOTWU2DQ6FKESGGZ4EVMZX36LI4NT32WDWQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:46118->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:46118->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46118->145.79.227.21:4473: use of closed network connection Nov 29 14:34:47.238703104 (486h30m45.790173565s) [noise=true , address=145.79.227.24:4583 , piece_id=B6GTB3UF3XX64M5U57FFCFAMWNWDJUDPIZDG23KOHXOVCFTGKCLA, seg_pos={0 0}, upload=428397]: Done uploading piece. err: ecclient: upload failed (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, address:145.79.227.24:4583): protocol: write tcp 188.166.225.170:51236->145.79.227.24:4583: use of closed network connection; write tcp 188.166.225.170:51236->145.79.227.24:4583: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51236->145.79.227.24:4583: use of closed network connection Nov 29 14:34:47.238869100 (486h30m45.790339561s) [noise=true , address=145.79.227.7:4295 , piece_id=OLB3PUIQ4RYD2GGVF4I32FEGXO2O4X56PLJMOKNTJVBK53KZ3VYQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:145.79.227.7:4295): protocol: write tcp 188.166.225.170:46306->145.79.227.7:4295: use of closed network connection; write tcp 188.166.225.170:46306->145.79.227.7:4295: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46306->145.79.227.7:4295: use of closed network connection Nov 29 14:34:47.239032171 (486h30m45.790502632s) [noise=true , address=145.79.227.6:4259 , piece_id=KIYGYFEAYWG2NUAQZNWG4ZLMRY6GFJADI5LD635JE5M4DLHTX4TQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:145.79.227.6:4259): protocol: write tcp 188.166.225.170:48184->145.79.227.6:4259: use of closed network connection; write tcp 188.166.225.170:48184->145.79.227.6:4259: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48184->145.79.227.6:4259: use of closed network connection Nov 29 14:34:47.239076555 (486h30m45.790547027s) [seg_pos={0 0}, upload=428397]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 29 14:34:47.481897829 (486h30m46.0333683s) [upload=428397]: Done uploading segments... Nov 29 14:34:47.481943356 (486h30m46.033413817s) [upload=428397]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 6.020331328s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [..............X................................X............] duration: 3.677481198s trace: https://jaeger.dev.storj.io/trace/275e9b01e747ebf4 FAIL (2025-11-29 14:49:54 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.677510252s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 7.313438374s trace: https://jaeger.dev.storj.io/trace/4add906a25eb3f89 FAIL (2025-11-29 14:49:38 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Nov 29 14:34:17.941236229 (486h30m16.49270669s) [upload=428397]: Got next segment. Inline: false Nov 29 14:34:17.941289010 (486h30m16.492759471s) [upload=428397]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 29 14:34:17.941296063 (486h30m16.492766524s) [upload=428397]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 29 14:34:18.189160800 (486h30m16.740631261s) [seg_pos={0 0}, upload=428397]: Begin upload segment... Nov 29 14:34:18.189218760 (486h30m16.740689221s) [seg_pos={0 0}, upload=428397]: Done begin upload segment. Nov 29 14:34:18.189254097 (486h30m16.740724548s) [noise=true , address=145.79.227.40:4049 , piece_id=DIBJ3Y4NA3HRAZD7GLO72STJHS5XHA3OTCAMWL4VCHMEILLCL3UA, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.189472724 (486h30m16.740943185s) [noise=true , address=77.90.31.13:4627 , piece_id=62A76IX5BE7WFBCCEMPSENCE5PWOSEMCUIKEATUPWGSLTCFJ4BGQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.190037069 (486h30m16.74150753s) [noise=true , address=145.79.227.2:4114 , piece_id=ZB7V2MM6KI6VTSGSC4ZD4QI6EA2265OP3MVLWD2AYAHFERAUE7CQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.190276184 (486h30m16.741746645s) [noise=true , address=145.79.227.21:4473 , piece_id=S2V3HONDHED6NZSIFZOTWU2DQ6FKESGGZ4EVMZX36LI4NT32WDWQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.190487406 (486h30m16.741957867s) [noise=true , address=145.79.227.31:4798 , piece_id=NG2JWZTD5NPFNDIVWYSWFHQHEDTURKJOY6YZ7G6LAQRIDI7WD3FQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.190825751 (486h30m16.742296212s) [noise=true , address=45.91.116.5:4297 , piece_id=GD7HY7SUS6XPN6KJDC5AVRP2ECTJSB2MK4YBQXCNUOTUBVKFRGVQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.190967721 (486h30m16.742438192s) [noise=true , address=145.79.227.24:4583 , piece_id=B6GTB3UF3XX64M5U57FFCFAMWNWDJUDPIZDG23KOHXOVCFTGKCLA, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.191308490 (486h30m16.742778951s) [noise=true , address=145.79.227.33:4869 , piece_id=V3JJCDM7EBRRZL2GPDMSCQAEHP2YX2QXDTDZDT5M6FGVLB27G7PQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.191365077 (486h30m16.742835528s) [noise=true , address=145.79.227.6:4259 , piece_id=KIYGYFEAYWG2NUAQZNWG4ZLMRY6GFJADI5LD635JE5M4DLHTX4TQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.191782502 (486h30m16.743252963s) [noise=true , address=145.79.227.22:4509 , piece_id=N2AEKOF7MLRQZ2HS7FSXV3U6WNI7YQLWP3W4PTFZLMAWQQREBTKQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.192449502 (486h30m16.743919953s) [noise=true , address=145.79.227.14:4839 , piece_id=RDZDYLJRCVHDKTTCAF3JB5ULPKFMSC2E4HNLB2RPLPY2GTUC6ZDQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.193091886 (486h30m16.744562347s) [noise=true , address=145.79.227.12:4767 , piece_id=5Q27HBFWOOI6PIIIMXELWG3QYSEA5B5LAOMU5K4FICMUMPDZIQAA, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.193463874 (486h30m16.744934335s) [noise=true , address=145.79.227.39:4013 , piece_id=YFEKNDHNI77M7MG76EISOGVU32UOZ536CK4X3Z6SFGOLJV2E7EHQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.195162279 (486h30m16.74663274s) [noise=true , address=145.79.227.20:4438 , piece_id=VZK574G5G2HXO5JMVMMEKGFCOJKOTU5KUIJDDEROFXRYTBTITO3A, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.197119445 (486h30m16.748589896s) [noise=true , address=145.79.227.7:4295 , piece_id=OLB3PUIQ4RYD2GGVF4I32FEGXO2O4X56PLJMOKNTJVBK53KZ3VYQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.197903478 (486h30m16.749373929s) [noise=true , address=145.79.227.32:4836 , piece_id=4KBSAAZLHC7OYLWN7RBDTQZUUTMK4HFGFUBES7K2PDKBXZST3OBA, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:19.999395950 (486h30m18.550866411s) [upload=428397]: Next returned nil segment Nov 29 14:34:19.999415177 (486h30m18.550885648s) [upload=428397]: Waiting for error group managing segments... Nov 29 14:34:20.285725942 (486h30m18.837196404s) [noise=true , address=77.90.31.13:4627 , piece_id=62A76IX5BE7WFBCCEMPSENCE5PWOSEMCUIKEATUPWGSLTCFJ4BGQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:20.363243793 (486h30m18.914714244s) [noise=true , address=45.91.116.5:4297 , piece_id=GD7HY7SUS6XPN6KJDC5AVRP2ECTJSB2MK4YBQXCNUOTUBVKFRGVQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:22.551752465 (486h30m21.103222926s) [noise=true , address=145.79.227.31:4798 , piece_id=NG2JWZTD5NPFNDIVWYSWFHQHEDTURKJOY6YZ7G6LAQRIDI7WD3FQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:22.666545505 (486h30m21.218015967s) [noise=true , address=145.79.227.22:4509 , piece_id=N2AEKOF7MLRQZ2HS7FSXV3U6WNI7YQLWP3W4PTFZLMAWQQREBTKQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:23.006352209 (486h30m21.557822671s) [noise=true , address=145.79.227.14:4839 , piece_id=RDZDYLJRCVHDKTTCAF3JB5ULPKFMSC2E4HNLB2RPLPY2GTUC6ZDQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:23.023700063 (486h30m21.575170525s) [noise=true , address=145.79.227.33:4869 , piece_id=V3JJCDM7EBRRZL2GPDMSCQAEHP2YX2QXDTDZDT5M6FGVLB27G7PQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:23.426873130 (486h30m21.978343581s) [noise=true , address=145.79.227.32:4836 , piece_id=4KBSAAZLHC7OYLWN7RBDTQZUUTMK4HFGFUBES7K2PDKBXZST3OBA, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:23.722460741 (486h30m22.273931202s) [noise=true , address=145.79.227.40:4049 , piece_id=DIBJ3Y4NA3HRAZD7GLO72STJHS5XHA3OTCAMWL4VCHMEILLCL3UA, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:25.698494096 (486h30m24.249964557s) [noise=true , address=145.79.227.39:4013 , piece_id=YFEKNDHNI77M7MG76EISOGVU32UOZ536CK4X3Z6SFGOLJV2E7EHQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:25.822411189 (486h30m24.373881651s) [noise=true , address=145.79.227.2:4114 , piece_id=ZB7V2MM6KI6VTSGSC4ZD4QI6EA2265OP3MVLWD2AYAHFERAUE7CQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:29.722449985 (486h30m28.273920436s) [noise=true , address=145.79.227.12:4767 , piece_id=5Q27HBFWOOI6PIIIMXELWG3QYSEA5B5LAOMU5K4FICMUMPDZIQAA, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:47.238199596 (486h30m45.789670047s) [noise=true , address=145.79.227.20:4438 , piece_id=VZK574G5G2HXO5JMVMMEKGFCOJKOTU5KUIJDDEROFXRYTBTITO3A, seg_pos={0 0}, upload=428397]: Done uploading piece. err: ecclient: upload failed (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, address:145.79.227.20:4438): protocol: write tcp 188.166.225.170:34106->145.79.227.20:4438: use of closed network connection; write tcp 188.166.225.170:34106->145.79.227.20:4438: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34106->145.79.227.20:4438: use of closed network connection Nov 29 14:34:47.238464621 (486h30m45.789935082s) [noise=true , address=145.79.227.21:4473 , piece_id=S2V3HONDHED6NZSIFZOTWU2DQ6FKESGGZ4EVMZX36LI4NT32WDWQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:46118->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:46118->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46118->145.79.227.21:4473: use of closed network connection Nov 29 14:34:47.238703104 (486h30m45.790173565s) [noise=true , address=145.79.227.24:4583 , piece_id=B6GTB3UF3XX64M5U57FFCFAMWNWDJUDPIZDG23KOHXOVCFTGKCLA, seg_pos={0 0}, upload=428397]: Done uploading piece. err: ecclient: upload failed (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, address:145.79.227.24:4583): protocol: write tcp 188.166.225.170:51236->145.79.227.24:4583: use of closed network connection; write tcp 188.166.225.170:51236->145.79.227.24:4583: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51236->145.79.227.24:4583: use of closed network connection Nov 29 14:34:47.238869100 (486h30m45.790339561s) [noise=true , address=145.79.227.7:4295 , piece_id=OLB3PUIQ4RYD2GGVF4I32FEGXO2O4X56PLJMOKNTJVBK53KZ3VYQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:145.79.227.7:4295): protocol: write tcp 188.166.225.170:46306->145.79.227.7:4295: use of closed network connection; write tcp 188.166.225.170:46306->145.79.227.7:4295: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46306->145.79.227.7:4295: use of closed network connection Nov 29 14:34:47.239032171 (486h30m45.790502632s) [noise=true , address=145.79.227.6:4259 , piece_id=KIYGYFEAYWG2NUAQZNWG4ZLMRY6GFJADI5LD635JE5M4DLHTX4TQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:145.79.227.6:4259): protocol: write tcp 188.166.225.170:48184->145.79.227.6:4259: use of closed network connection; write tcp 188.166.225.170:48184->145.79.227.6:4259: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48184->145.79.227.6:4259: use of closed network connection Nov 29 14:34:47.239076555 (486h30m45.790547027s) [seg_pos={0 0}, upload=428397]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 29 14:34:47.481897829 (486h30m46.0333683s) [upload=428397]: Done uploading segments... Nov 29 14:34:47.481943356 (486h30m46.033413817s) [upload=428397]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 7.313479861s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 10.348541769s trace: https://jaeger.dev.storj.io/trace/61a1d0645153f16 FAIL (2025-11-29 14:49:25 UTC) orbiter warning: context deadline exceeded Duration: 10.348573069s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [...............X................................X...........] duration: 3.669309036s trace: https://jaeger.dev.storj.io/trace/275e9b01e747ebf4 FAIL (2025-11-29 14:48:58 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.669333783s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 6.275571212s trace: https://jaeger.dev.storj.io/trace/4add906a25eb3f89 FAIL (2025-11-29 14:48:31 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Nov 29 14:34:17.941236229 (486h30m16.49270669s) [upload=428397]: Got next segment. Inline: false Nov 29 14:34:17.941289010 (486h30m16.492759471s) [upload=428397]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 29 14:34:17.941296063 (486h30m16.492766524s) [upload=428397]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 29 14:34:18.189160800 (486h30m16.740631261s) [seg_pos={0 0}, upload=428397]: Begin upload segment... Nov 29 14:34:18.189218760 (486h30m16.740689221s) [seg_pos={0 0}, upload=428397]: Done begin upload segment. Nov 29 14:34:18.189254097 (486h30m16.740724548s) [noise=true , address=145.79.227.40:4049 , piece_id=DIBJ3Y4NA3HRAZD7GLO72STJHS5XHA3OTCAMWL4VCHMEILLCL3UA, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.189472724 (486h30m16.740943185s) [noise=true , address=77.90.31.13:4627 , piece_id=62A76IX5BE7WFBCCEMPSENCE5PWOSEMCUIKEATUPWGSLTCFJ4BGQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.190037069 (486h30m16.74150753s) [noise=true , address=145.79.227.2:4114 , piece_id=ZB7V2MM6KI6VTSGSC4ZD4QI6EA2265OP3MVLWD2AYAHFERAUE7CQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.190276184 (486h30m16.741746645s) [noise=true , address=145.79.227.21:4473 , piece_id=S2V3HONDHED6NZSIFZOTWU2DQ6FKESGGZ4EVMZX36LI4NT32WDWQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.190487406 (486h30m16.741957867s) [noise=true , address=145.79.227.31:4798 , piece_id=NG2JWZTD5NPFNDIVWYSWFHQHEDTURKJOY6YZ7G6LAQRIDI7WD3FQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.190825751 (486h30m16.742296212s) [noise=true , address=45.91.116.5:4297 , piece_id=GD7HY7SUS6XPN6KJDC5AVRP2ECTJSB2MK4YBQXCNUOTUBVKFRGVQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.190967721 (486h30m16.742438192s) [noise=true , address=145.79.227.24:4583 , piece_id=B6GTB3UF3XX64M5U57FFCFAMWNWDJUDPIZDG23KOHXOVCFTGKCLA, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.191308490 (486h30m16.742778951s) [noise=true , address=145.79.227.33:4869 , piece_id=V3JJCDM7EBRRZL2GPDMSCQAEHP2YX2QXDTDZDT5M6FGVLB27G7PQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.191365077 (486h30m16.742835528s) [noise=true , address=145.79.227.6:4259 , piece_id=KIYGYFEAYWG2NUAQZNWG4ZLMRY6GFJADI5LD635JE5M4DLHTX4TQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.191782502 (486h30m16.743252963s) [noise=true , address=145.79.227.22:4509 , piece_id=N2AEKOF7MLRQZ2HS7FSXV3U6WNI7YQLWP3W4PTFZLMAWQQREBTKQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.192449502 (486h30m16.743919953s) [noise=true , address=145.79.227.14:4839 , piece_id=RDZDYLJRCVHDKTTCAF3JB5ULPKFMSC2E4HNLB2RPLPY2GTUC6ZDQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.193091886 (486h30m16.744562347s) [noise=true , address=145.79.227.12:4767 , piece_id=5Q27HBFWOOI6PIIIMXELWG3QYSEA5B5LAOMU5K4FICMUMPDZIQAA, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.193463874 (486h30m16.744934335s) [noise=true , address=145.79.227.39:4013 , piece_id=YFEKNDHNI77M7MG76EISOGVU32UOZ536CK4X3Z6SFGOLJV2E7EHQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.195162279 (486h30m16.74663274s) [noise=true , address=145.79.227.20:4438 , piece_id=VZK574G5G2HXO5JMVMMEKGFCOJKOTU5KUIJDDEROFXRYTBTITO3A, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.197119445 (486h30m16.748589896s) [noise=true , address=145.79.227.7:4295 , piece_id=OLB3PUIQ4RYD2GGVF4I32FEGXO2O4X56PLJMOKNTJVBK53KZ3VYQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.197903478 (486h30m16.749373929s) [noise=true , address=145.79.227.32:4836 , piece_id=4KBSAAZLHC7OYLWN7RBDTQZUUTMK4HFGFUBES7K2PDKBXZST3OBA, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:19.999395950 (486h30m18.550866411s) [upload=428397]: Next returned nil segment Nov 29 14:34:19.999415177 (486h30m18.550885648s) [upload=428397]: Waiting for error group managing segments... Nov 29 14:34:20.285725942 (486h30m18.837196404s) [noise=true , address=77.90.31.13:4627 , piece_id=62A76IX5BE7WFBCCEMPSENCE5PWOSEMCUIKEATUPWGSLTCFJ4BGQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:20.363243793 (486h30m18.914714244s) [noise=true , address=45.91.116.5:4297 , piece_id=GD7HY7SUS6XPN6KJDC5AVRP2ECTJSB2MK4YBQXCNUOTUBVKFRGVQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:22.551752465 (486h30m21.103222926s) [noise=true , address=145.79.227.31:4798 , piece_id=NG2JWZTD5NPFNDIVWYSWFHQHEDTURKJOY6YZ7G6LAQRIDI7WD3FQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:22.666545505 (486h30m21.218015967s) [noise=true , address=145.79.227.22:4509 , piece_id=N2AEKOF7MLRQZ2HS7FSXV3U6WNI7YQLWP3W4PTFZLMAWQQREBTKQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:23.006352209 (486h30m21.557822671s) [noise=true , address=145.79.227.14:4839 , piece_id=RDZDYLJRCVHDKTTCAF3JB5ULPKFMSC2E4HNLB2RPLPY2GTUC6ZDQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:23.023700063 (486h30m21.575170525s) [noise=true , address=145.79.227.33:4869 , piece_id=V3JJCDM7EBRRZL2GPDMSCQAEHP2YX2QXDTDZDT5M6FGVLB27G7PQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:23.426873130 (486h30m21.978343581s) [noise=true , address=145.79.227.32:4836 , piece_id=4KBSAAZLHC7OYLWN7RBDTQZUUTMK4HFGFUBES7K2PDKBXZST3OBA, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:23.722460741 (486h30m22.273931202s) [noise=true , address=145.79.227.40:4049 , piece_id=DIBJ3Y4NA3HRAZD7GLO72STJHS5XHA3OTCAMWL4VCHMEILLCL3UA, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:25.698494096 (486h30m24.249964557s) [noise=true , address=145.79.227.39:4013 , piece_id=YFEKNDHNI77M7MG76EISOGVU32UOZ536CK4X3Z6SFGOLJV2E7EHQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:25.822411189 (486h30m24.373881651s) [noise=true , address=145.79.227.2:4114 , piece_id=ZB7V2MM6KI6VTSGSC4ZD4QI6EA2265OP3MVLWD2AYAHFERAUE7CQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:29.722449985 (486h30m28.273920436s) [noise=true , address=145.79.227.12:4767 , piece_id=5Q27HBFWOOI6PIIIMXELWG3QYSEA5B5LAOMU5K4FICMUMPDZIQAA, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:47.238199596 (486h30m45.789670047s) [noise=true , address=145.79.227.20:4438 , piece_id=VZK574G5G2HXO5JMVMMEKGFCOJKOTU5KUIJDDEROFXRYTBTITO3A, seg_pos={0 0}, upload=428397]: Done uploading piece. err: ecclient: upload failed (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, address:145.79.227.20:4438): protocol: write tcp 188.166.225.170:34106->145.79.227.20:4438: use of closed network connection; write tcp 188.166.225.170:34106->145.79.227.20:4438: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34106->145.79.227.20:4438: use of closed network connection Nov 29 14:34:47.238464621 (486h30m45.789935082s) [noise=true , address=145.79.227.21:4473 , piece_id=S2V3HONDHED6NZSIFZOTWU2DQ6FKESGGZ4EVMZX36LI4NT32WDWQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:46118->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:46118->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46118->145.79.227.21:4473: use of closed network connection Nov 29 14:34:47.238703104 (486h30m45.790173565s) [noise=true , address=145.79.227.24:4583 , piece_id=B6GTB3UF3XX64M5U57FFCFAMWNWDJUDPIZDG23KOHXOVCFTGKCLA, seg_pos={0 0}, upload=428397]: Done uploading piece. err: ecclient: upload failed (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, address:145.79.227.24:4583): protocol: write tcp 188.166.225.170:51236->145.79.227.24:4583: use of closed network connection; write tcp 188.166.225.170:51236->145.79.227.24:4583: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51236->145.79.227.24:4583: use of closed network connection Nov 29 14:34:47.238869100 (486h30m45.790339561s) [noise=true , address=145.79.227.7:4295 , piece_id=OLB3PUIQ4RYD2GGVF4I32FEGXO2O4X56PLJMOKNTJVBK53KZ3VYQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:145.79.227.7:4295): protocol: write tcp 188.166.225.170:46306->145.79.227.7:4295: use of closed network connection; write tcp 188.166.225.170:46306->145.79.227.7:4295: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46306->145.79.227.7:4295: use of closed network connection Nov 29 14:34:47.239032171 (486h30m45.790502632s) [noise=true , address=145.79.227.6:4259 , piece_id=KIYGYFEAYWG2NUAQZNWG4ZLMRY6GFJADI5LD635JE5M4DLHTX4TQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:145.79.227.6:4259): protocol: write tcp 188.166.225.170:48184->145.79.227.6:4259: use of closed network connection; write tcp 188.166.225.170:48184->145.79.227.6:4259: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48184->145.79.227.6:4259: use of closed network connection Nov 29 14:34:47.239076555 (486h30m45.790547027s) [seg_pos={0 0}, upload=428397]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 29 14:34:47.481897829 (486h30m46.0333683s) [upload=428397]: Done uploading segments... Nov 29 14:34:47.481943356 (486h30m46.033413817s) [upload=428397]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 6.275600118s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 6.037681602s trace: https://jaeger.dev.storj.io/trace/61a1d0645153f16 FAIL (2025-11-29 14:48:24 UTC) orbiter warning: context deadline exceeded Duration: 6.037700859s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [................X................................X..........] duration: 4.524746956s trace: https://jaeger.dev.storj.io/trace/275e9b01e747ebf4 FAIL (2025-11-29 14:48:14 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.52477546s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 8.660876511s trace: https://jaeger.dev.storj.io/trace/61a1d0645153f16 FAIL (2025-11-29 14:47:58 UTC) orbiter warning: context deadline exceeded Duration: 8.660904824s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 5.834859633s trace: https://jaeger.dev.storj.io/trace/4add906a25eb3f89 FAIL (2025-11-29 14:47:52 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Nov 29 14:34:17.941236229 (486h30m16.49270669s) [upload=428397]: Got next segment. Inline: false Nov 29 14:34:17.941289010 (486h30m16.492759471s) [upload=428397]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 29 14:34:17.941296063 (486h30m16.492766524s) [upload=428397]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 29 14:34:18.189160800 (486h30m16.740631261s) [seg_pos={0 0}, upload=428397]: Begin upload segment... Nov 29 14:34:18.189218760 (486h30m16.740689221s) [seg_pos={0 0}, upload=428397]: Done begin upload segment. Nov 29 14:34:18.189254097 (486h30m16.740724548s) [noise=true , address=145.79.227.40:4049 , piece_id=DIBJ3Y4NA3HRAZD7GLO72STJHS5XHA3OTCAMWL4VCHMEILLCL3UA, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.189472724 (486h30m16.740943185s) [noise=true , address=77.90.31.13:4627 , piece_id=62A76IX5BE7WFBCCEMPSENCE5PWOSEMCUIKEATUPWGSLTCFJ4BGQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.190037069 (486h30m16.74150753s) [noise=true , address=145.79.227.2:4114 , piece_id=ZB7V2MM6KI6VTSGSC4ZD4QI6EA2265OP3MVLWD2AYAHFERAUE7CQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.190276184 (486h30m16.741746645s) [noise=true , address=145.79.227.21:4473 , piece_id=S2V3HONDHED6NZSIFZOTWU2DQ6FKESGGZ4EVMZX36LI4NT32WDWQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.190487406 (486h30m16.741957867s) [noise=true , address=145.79.227.31:4798 , piece_id=NG2JWZTD5NPFNDIVWYSWFHQHEDTURKJOY6YZ7G6LAQRIDI7WD3FQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.190825751 (486h30m16.742296212s) [noise=true , address=45.91.116.5:4297 , piece_id=GD7HY7SUS6XPN6KJDC5AVRP2ECTJSB2MK4YBQXCNUOTUBVKFRGVQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.190967721 (486h30m16.742438192s) [noise=true , address=145.79.227.24:4583 , piece_id=B6GTB3UF3XX64M5U57FFCFAMWNWDJUDPIZDG23KOHXOVCFTGKCLA, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.191308490 (486h30m16.742778951s) [noise=true , address=145.79.227.33:4869 , piece_id=V3JJCDM7EBRRZL2GPDMSCQAEHP2YX2QXDTDZDT5M6FGVLB27G7PQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.191365077 (486h30m16.742835528s) [noise=true , address=145.79.227.6:4259 , piece_id=KIYGYFEAYWG2NUAQZNWG4ZLMRY6GFJADI5LD635JE5M4DLHTX4TQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.191782502 (486h30m16.743252963s) [noise=true , address=145.79.227.22:4509 , piece_id=N2AEKOF7MLRQZ2HS7FSXV3U6WNI7YQLWP3W4PTFZLMAWQQREBTKQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.192449502 (486h30m16.743919953s) [noise=true , address=145.79.227.14:4839 , piece_id=RDZDYLJRCVHDKTTCAF3JB5ULPKFMSC2E4HNLB2RPLPY2GTUC6ZDQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.193091886 (486h30m16.744562347s) [noise=true , address=145.79.227.12:4767 , piece_id=5Q27HBFWOOI6PIIIMXELWG3QYSEA5B5LAOMU5K4FICMUMPDZIQAA, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.193463874 (486h30m16.744934335s) [noise=true , address=145.79.227.39:4013 , piece_id=YFEKNDHNI77M7MG76EISOGVU32UOZ536CK4X3Z6SFGOLJV2E7EHQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.195162279 (486h30m16.74663274s) [noise=true , address=145.79.227.20:4438 , piece_id=VZK574G5G2HXO5JMVMMEKGFCOJKOTU5KUIJDDEROFXRYTBTITO3A, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.197119445 (486h30m16.748589896s) [noise=true , address=145.79.227.7:4295 , piece_id=OLB3PUIQ4RYD2GGVF4I32FEGXO2O4X56PLJMOKNTJVBK53KZ3VYQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.197903478 (486h30m16.749373929s) [noise=true , address=145.79.227.32:4836 , piece_id=4KBSAAZLHC7OYLWN7RBDTQZUUTMK4HFGFUBES7K2PDKBXZST3OBA, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:19.999395950 (486h30m18.550866411s) [upload=428397]: Next returned nil segment Nov 29 14:34:19.999415177 (486h30m18.550885648s) [upload=428397]: Waiting for error group managing segments... Nov 29 14:34:20.285725942 (486h30m18.837196404s) [noise=true , address=77.90.31.13:4627 , piece_id=62A76IX5BE7WFBCCEMPSENCE5PWOSEMCUIKEATUPWGSLTCFJ4BGQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:20.363243793 (486h30m18.914714244s) [noise=true , address=45.91.116.5:4297 , piece_id=GD7HY7SUS6XPN6KJDC5AVRP2ECTJSB2MK4YBQXCNUOTUBVKFRGVQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:22.551752465 (486h30m21.103222926s) [noise=true , address=145.79.227.31:4798 , piece_id=NG2JWZTD5NPFNDIVWYSWFHQHEDTURKJOY6YZ7G6LAQRIDI7WD3FQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:22.666545505 (486h30m21.218015967s) [noise=true , address=145.79.227.22:4509 , piece_id=N2AEKOF7MLRQZ2HS7FSXV3U6WNI7YQLWP3W4PTFZLMAWQQREBTKQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:23.006352209 (486h30m21.557822671s) [noise=true , address=145.79.227.14:4839 , piece_id=RDZDYLJRCVHDKTTCAF3JB5ULPKFMSC2E4HNLB2RPLPY2GTUC6ZDQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:23.023700063 (486h30m21.575170525s) [noise=true , address=145.79.227.33:4869 , piece_id=V3JJCDM7EBRRZL2GPDMSCQAEHP2YX2QXDTDZDT5M6FGVLB27G7PQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:23.426873130 (486h30m21.978343581s) [noise=true , address=145.79.227.32:4836 , piece_id=4KBSAAZLHC7OYLWN7RBDTQZUUTMK4HFGFUBES7K2PDKBXZST3OBA, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:23.722460741 (486h30m22.273931202s) [noise=true , address=145.79.227.40:4049 , piece_id=DIBJ3Y4NA3HRAZD7GLO72STJHS5XHA3OTCAMWL4VCHMEILLCL3UA, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:25.698494096 (486h30m24.249964557s) [noise=true , address=145.79.227.39:4013 , piece_id=YFEKNDHNI77M7MG76EISOGVU32UOZ536CK4X3Z6SFGOLJV2E7EHQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:25.822411189 (486h30m24.373881651s) [noise=true , address=145.79.227.2:4114 , piece_id=ZB7V2MM6KI6VTSGSC4ZD4QI6EA2265OP3MVLWD2AYAHFERAUE7CQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:29.722449985 (486h30m28.273920436s) [noise=true , address=145.79.227.12:4767 , piece_id=5Q27HBFWOOI6PIIIMXELWG3QYSEA5B5LAOMU5K4FICMUMPDZIQAA, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:47.238199596 (486h30m45.789670047s) [noise=true , address=145.79.227.20:4438 , piece_id=VZK574G5G2HXO5JMVMMEKGFCOJKOTU5KUIJDDEROFXRYTBTITO3A, seg_pos={0 0}, upload=428397]: Done uploading piece. err: ecclient: upload failed (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, address:145.79.227.20:4438): protocol: write tcp 188.166.225.170:34106->145.79.227.20:4438: use of closed network connection; write tcp 188.166.225.170:34106->145.79.227.20:4438: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34106->145.79.227.20:4438: use of closed network connection Nov 29 14:34:47.238464621 (486h30m45.789935082s) [noise=true , address=145.79.227.21:4473 , piece_id=S2V3HONDHED6NZSIFZOTWU2DQ6FKESGGZ4EVMZX36LI4NT32WDWQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:46118->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:46118->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46118->145.79.227.21:4473: use of closed network connection Nov 29 14:34:47.238703104 (486h30m45.790173565s) [noise=true , address=145.79.227.24:4583 , piece_id=B6GTB3UF3XX64M5U57FFCFAMWNWDJUDPIZDG23KOHXOVCFTGKCLA, seg_pos={0 0}, upload=428397]: Done uploading piece. err: ecclient: upload failed (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, address:145.79.227.24:4583): protocol: write tcp 188.166.225.170:51236->145.79.227.24:4583: use of closed network connection; write tcp 188.166.225.170:51236->145.79.227.24:4583: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51236->145.79.227.24:4583: use of closed network connection Nov 29 14:34:47.238869100 (486h30m45.790339561s) [noise=true , address=145.79.227.7:4295 , piece_id=OLB3PUIQ4RYD2GGVF4I32FEGXO2O4X56PLJMOKNTJVBK53KZ3VYQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:145.79.227.7:4295): protocol: write tcp 188.166.225.170:46306->145.79.227.7:4295: use of closed network connection; write tcp 188.166.225.170:46306->145.79.227.7:4295: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46306->145.79.227.7:4295: use of closed network connection Nov 29 14:34:47.239032171 (486h30m45.790502632s) [noise=true , address=145.79.227.6:4259 , piece_id=KIYGYFEAYWG2NUAQZNWG4ZLMRY6GFJADI5LD635JE5M4DLHTX4TQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:145.79.227.6:4259): protocol: write tcp 188.166.225.170:48184->145.79.227.6:4259: use of closed network connection; write tcp 188.166.225.170:48184->145.79.227.6:4259: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48184->145.79.227.6:4259: use of closed network connection Nov 29 14:34:47.239076555 (486h30m45.790547027s) [seg_pos={0 0}, upload=428397]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 29 14:34:47.481897829 (486h30m46.0333683s) [upload=428397]: Done uploading segments... Nov 29 14:34:47.481943356 (486h30m46.033413817s) [upload=428397]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.834891913s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.................X................................X.........] duration: 3.760851508s trace: https://jaeger.dev.storj.io/trace/275e9b01e747ebf4 FAIL (2025-11-29 14:47:30 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.760881946s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 6.011875334s trace: https://jaeger.dev.storj.io/trace/61a1d0645153f16 FAIL (2025-11-29 14:47:20 UTC) orbiter warning: context deadline exceeded Duration: 6.011908737s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [..................X................................X........] duration: 3.511037213s trace: https://jaeger.dev.storj.io/trace/275e9b01e747ebf4 FAIL (2025-11-29 14:46:57 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.511228396s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 5.749272172s trace: https://jaeger.dev.storj.io/trace/4add906a25eb3f89 FAIL (2025-11-29 14:46:48 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Nov 29 14:34:17.941236229 (486h30m16.49270669s) [upload=428397]: Got next segment. Inline: false Nov 29 14:34:17.941289010 (486h30m16.492759471s) [upload=428397]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 29 14:34:17.941296063 (486h30m16.492766524s) [upload=428397]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 29 14:34:18.189160800 (486h30m16.740631261s) [seg_pos={0 0}, upload=428397]: Begin upload segment... Nov 29 14:34:18.189218760 (486h30m16.740689221s) [seg_pos={0 0}, upload=428397]: Done begin upload segment. Nov 29 14:34:18.189254097 (486h30m16.740724548s) [noise=true , address=145.79.227.40:4049 , piece_id=DIBJ3Y4NA3HRAZD7GLO72STJHS5XHA3OTCAMWL4VCHMEILLCL3UA, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.189472724 (486h30m16.740943185s) [noise=true , address=77.90.31.13:4627 , piece_id=62A76IX5BE7WFBCCEMPSENCE5PWOSEMCUIKEATUPWGSLTCFJ4BGQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.190037069 (486h30m16.74150753s) [noise=true , address=145.79.227.2:4114 , piece_id=ZB7V2MM6KI6VTSGSC4ZD4QI6EA2265OP3MVLWD2AYAHFERAUE7CQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.190276184 (486h30m16.741746645s) [noise=true , address=145.79.227.21:4473 , piece_id=S2V3HONDHED6NZSIFZOTWU2DQ6FKESGGZ4EVMZX36LI4NT32WDWQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.190487406 (486h30m16.741957867s) [noise=true , address=145.79.227.31:4798 , piece_id=NG2JWZTD5NPFNDIVWYSWFHQHEDTURKJOY6YZ7G6LAQRIDI7WD3FQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.190825751 (486h30m16.742296212s) [noise=true , address=45.91.116.5:4297 , piece_id=GD7HY7SUS6XPN6KJDC5AVRP2ECTJSB2MK4YBQXCNUOTUBVKFRGVQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.190967721 (486h30m16.742438192s) [noise=true , address=145.79.227.24:4583 , piece_id=B6GTB3UF3XX64M5U57FFCFAMWNWDJUDPIZDG23KOHXOVCFTGKCLA, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.191308490 (486h30m16.742778951s) [noise=true , address=145.79.227.33:4869 , piece_id=V3JJCDM7EBRRZL2GPDMSCQAEHP2YX2QXDTDZDT5M6FGVLB27G7PQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.191365077 (486h30m16.742835528s) [noise=true , address=145.79.227.6:4259 , piece_id=KIYGYFEAYWG2NUAQZNWG4ZLMRY6GFJADI5LD635JE5M4DLHTX4TQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.191782502 (486h30m16.743252963s) [noise=true , address=145.79.227.22:4509 , piece_id=N2AEKOF7MLRQZ2HS7FSXV3U6WNI7YQLWP3W4PTFZLMAWQQREBTKQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.192449502 (486h30m16.743919953s) [noise=true , address=145.79.227.14:4839 , piece_id=RDZDYLJRCVHDKTTCAF3JB5ULPKFMSC2E4HNLB2RPLPY2GTUC6ZDQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.193091886 (486h30m16.744562347s) [noise=true , address=145.79.227.12:4767 , piece_id=5Q27HBFWOOI6PIIIMXELWG3QYSEA5B5LAOMU5K4FICMUMPDZIQAA, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.193463874 (486h30m16.744934335s) [noise=true , address=145.79.227.39:4013 , piece_id=YFEKNDHNI77M7MG76EISOGVU32UOZ536CK4X3Z6SFGOLJV2E7EHQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.195162279 (486h30m16.74663274s) [noise=true , address=145.79.227.20:4438 , piece_id=VZK574G5G2HXO5JMVMMEKGFCOJKOTU5KUIJDDEROFXRYTBTITO3A, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.197119445 (486h30m16.748589896s) [noise=true , address=145.79.227.7:4295 , piece_id=OLB3PUIQ4RYD2GGVF4I32FEGXO2O4X56PLJMOKNTJVBK53KZ3VYQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.197903478 (486h30m16.749373929s) [noise=true , address=145.79.227.32:4836 , piece_id=4KBSAAZLHC7OYLWN7RBDTQZUUTMK4HFGFUBES7K2PDKBXZST3OBA, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:19.999395950 (486h30m18.550866411s) [upload=428397]: Next returned nil segment Nov 29 14:34:19.999415177 (486h30m18.550885648s) [upload=428397]: Waiting for error group managing segments... Nov 29 14:34:20.285725942 (486h30m18.837196404s) [noise=true , address=77.90.31.13:4627 , piece_id=62A76IX5BE7WFBCCEMPSENCE5PWOSEMCUIKEATUPWGSLTCFJ4BGQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:20.363243793 (486h30m18.914714244s) [noise=true , address=45.91.116.5:4297 , piece_id=GD7HY7SUS6XPN6KJDC5AVRP2ECTJSB2MK4YBQXCNUOTUBVKFRGVQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:22.551752465 (486h30m21.103222926s) [noise=true , address=145.79.227.31:4798 , piece_id=NG2JWZTD5NPFNDIVWYSWFHQHEDTURKJOY6YZ7G6LAQRIDI7WD3FQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:22.666545505 (486h30m21.218015967s) [noise=true , address=145.79.227.22:4509 , piece_id=N2AEKOF7MLRQZ2HS7FSXV3U6WNI7YQLWP3W4PTFZLMAWQQREBTKQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:23.006352209 (486h30m21.557822671s) [noise=true , address=145.79.227.14:4839 , piece_id=RDZDYLJRCVHDKTTCAF3JB5ULPKFMSC2E4HNLB2RPLPY2GTUC6ZDQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:23.023700063 (486h30m21.575170525s) [noise=true , address=145.79.227.33:4869 , piece_id=V3JJCDM7EBRRZL2GPDMSCQAEHP2YX2QXDTDZDT5M6FGVLB27G7PQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:23.426873130 (486h30m21.978343581s) [noise=true , address=145.79.227.32:4836 , piece_id=4KBSAAZLHC7OYLWN7RBDTQZUUTMK4HFGFUBES7K2PDKBXZST3OBA, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:23.722460741 (486h30m22.273931202s) [noise=true , address=145.79.227.40:4049 , piece_id=DIBJ3Y4NA3HRAZD7GLO72STJHS5XHA3OTCAMWL4VCHMEILLCL3UA, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:25.698494096 (486h30m24.249964557s) [noise=true , address=145.79.227.39:4013 , piece_id=YFEKNDHNI77M7MG76EISOGVU32UOZ536CK4X3Z6SFGOLJV2E7EHQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:25.822411189 (486h30m24.373881651s) [noise=true , address=145.79.227.2:4114 , piece_id=ZB7V2MM6KI6VTSGSC4ZD4QI6EA2265OP3MVLWD2AYAHFERAUE7CQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:29.722449985 (486h30m28.273920436s) [noise=true , address=145.79.227.12:4767 , piece_id=5Q27HBFWOOI6PIIIMXELWG3QYSEA5B5LAOMU5K4FICMUMPDZIQAA, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:47.238199596 (486h30m45.789670047s) [noise=true , address=145.79.227.20:4438 , piece_id=VZK574G5G2HXO5JMVMMEKGFCOJKOTU5KUIJDDEROFXRYTBTITO3A, seg_pos={0 0}, upload=428397]: Done uploading piece. err: ecclient: upload failed (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, address:145.79.227.20:4438): protocol: write tcp 188.166.225.170:34106->145.79.227.20:4438: use of closed network connection; write tcp 188.166.225.170:34106->145.79.227.20:4438: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34106->145.79.227.20:4438: use of closed network connection Nov 29 14:34:47.238464621 (486h30m45.789935082s) [noise=true , address=145.79.227.21:4473 , piece_id=S2V3HONDHED6NZSIFZOTWU2DQ6FKESGGZ4EVMZX36LI4NT32WDWQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:46118->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:46118->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46118->145.79.227.21:4473: use of closed network connection Nov 29 14:34:47.238703104 (486h30m45.790173565s) [noise=true , address=145.79.227.24:4583 , piece_id=B6GTB3UF3XX64M5U57FFCFAMWNWDJUDPIZDG23KOHXOVCFTGKCLA, seg_pos={0 0}, upload=428397]: Done uploading piece. err: ecclient: upload failed (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, address:145.79.227.24:4583): protocol: write tcp 188.166.225.170:51236->145.79.227.24:4583: use of closed network connection; write tcp 188.166.225.170:51236->145.79.227.24:4583: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51236->145.79.227.24:4583: use of closed network connection Nov 29 14:34:47.238869100 (486h30m45.790339561s) [noise=true , address=145.79.227.7:4295 , piece_id=OLB3PUIQ4RYD2GGVF4I32FEGXO2O4X56PLJMOKNTJVBK53KZ3VYQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:145.79.227.7:4295): protocol: write tcp 188.166.225.170:46306->145.79.227.7:4295: use of closed network connection; write tcp 188.166.225.170:46306->145.79.227.7:4295: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46306->145.79.227.7:4295: use of closed network connection Nov 29 14:34:47.239032171 (486h30m45.790502632s) [noise=true , address=145.79.227.6:4259 , piece_id=KIYGYFEAYWG2NUAQZNWG4ZLMRY6GFJADI5LD635JE5M4DLHTX4TQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:145.79.227.6:4259): protocol: write tcp 188.166.225.170:48184->145.79.227.6:4259: use of closed network connection; write tcp 188.166.225.170:48184->145.79.227.6:4259: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48184->145.79.227.6:4259: use of closed network connection Nov 29 14:34:47.239076555 (486h30m45.790547027s) [seg_pos={0 0}, upload=428397]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 29 14:34:47.481897829 (486h30m46.0333683s) [upload=428397]: Done uploading segments... Nov 29 14:34:47.481943356 (486h30m46.033413817s) [upload=428397]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.749301597s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 6.320086437s trace: https://jaeger.dev.storj.io/trace/61a1d0645153f16 FAIL (2025-11-29 14:46:37 UTC) orbiter warning: context deadline exceeded Duration: 6.320111976s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 6.740113424s trace: https://jaeger.dev.storj.io/trace/61a1d0645153f16 FAIL (2025-11-29 14:46:17 UTC) orbiter warning: context deadline exceeded Duration: 6.740143242s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 5.935687139s trace: https://jaeger.dev.storj.io/trace/4add906a25eb3f89 FAIL (2025-11-29 14:46:09 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Nov 29 14:34:17.941236229 (486h30m16.49270669s) [upload=428397]: Got next segment. Inline: false Nov 29 14:34:17.941289010 (486h30m16.492759471s) [upload=428397]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 29 14:34:17.941296063 (486h30m16.492766524s) [upload=428397]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 29 14:34:18.189160800 (486h30m16.740631261s) [seg_pos={0 0}, upload=428397]: Begin upload segment... Nov 29 14:34:18.189218760 (486h30m16.740689221s) [seg_pos={0 0}, upload=428397]: Done begin upload segment. Nov 29 14:34:18.189254097 (486h30m16.740724548s) [noise=true , address=145.79.227.40:4049 , piece_id=DIBJ3Y4NA3HRAZD7GLO72STJHS5XHA3OTCAMWL4VCHMEILLCL3UA, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.189472724 (486h30m16.740943185s) [noise=true , address=77.90.31.13:4627 , piece_id=62A76IX5BE7WFBCCEMPSENCE5PWOSEMCUIKEATUPWGSLTCFJ4BGQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.190037069 (486h30m16.74150753s) [noise=true , address=145.79.227.2:4114 , piece_id=ZB7V2MM6KI6VTSGSC4ZD4QI6EA2265OP3MVLWD2AYAHFERAUE7CQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.190276184 (486h30m16.741746645s) [noise=true , address=145.79.227.21:4473 , piece_id=S2V3HONDHED6NZSIFZOTWU2DQ6FKESGGZ4EVMZX36LI4NT32WDWQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.190487406 (486h30m16.741957867s) [noise=true , address=145.79.227.31:4798 , piece_id=NG2JWZTD5NPFNDIVWYSWFHQHEDTURKJOY6YZ7G6LAQRIDI7WD3FQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.190825751 (486h30m16.742296212s) [noise=true , address=45.91.116.5:4297 , piece_id=GD7HY7SUS6XPN6KJDC5AVRP2ECTJSB2MK4YBQXCNUOTUBVKFRGVQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.190967721 (486h30m16.742438192s) [noise=true , address=145.79.227.24:4583 , piece_id=B6GTB3UF3XX64M5U57FFCFAMWNWDJUDPIZDG23KOHXOVCFTGKCLA, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.191308490 (486h30m16.742778951s) [noise=true , address=145.79.227.33:4869 , piece_id=V3JJCDM7EBRRZL2GPDMSCQAEHP2YX2QXDTDZDT5M6FGVLB27G7PQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.191365077 (486h30m16.742835528s) [noise=true , address=145.79.227.6:4259 , piece_id=KIYGYFEAYWG2NUAQZNWG4ZLMRY6GFJADI5LD635JE5M4DLHTX4TQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.191782502 (486h30m16.743252963s) [noise=true , address=145.79.227.22:4509 , piece_id=N2AEKOF7MLRQZ2HS7FSXV3U6WNI7YQLWP3W4PTFZLMAWQQREBTKQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.192449502 (486h30m16.743919953s) [noise=true , address=145.79.227.14:4839 , piece_id=RDZDYLJRCVHDKTTCAF3JB5ULPKFMSC2E4HNLB2RPLPY2GTUC6ZDQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.193091886 (486h30m16.744562347s) [noise=true , address=145.79.227.12:4767 , piece_id=5Q27HBFWOOI6PIIIMXELWG3QYSEA5B5LAOMU5K4FICMUMPDZIQAA, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.193463874 (486h30m16.744934335s) [noise=true , address=145.79.227.39:4013 , piece_id=YFEKNDHNI77M7MG76EISOGVU32UOZ536CK4X3Z6SFGOLJV2E7EHQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.195162279 (486h30m16.74663274s) [noise=true , address=145.79.227.20:4438 , piece_id=VZK574G5G2HXO5JMVMMEKGFCOJKOTU5KUIJDDEROFXRYTBTITO3A, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.197119445 (486h30m16.748589896s) [noise=true , address=145.79.227.7:4295 , piece_id=OLB3PUIQ4RYD2GGVF4I32FEGXO2O4X56PLJMOKNTJVBK53KZ3VYQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.197903478 (486h30m16.749373929s) [noise=true , address=145.79.227.32:4836 , piece_id=4KBSAAZLHC7OYLWN7RBDTQZUUTMK4HFGFUBES7K2PDKBXZST3OBA, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:19.999395950 (486h30m18.550866411s) [upload=428397]: Next returned nil segment Nov 29 14:34:19.999415177 (486h30m18.550885648s) [upload=428397]: Waiting for error group managing segments... Nov 29 14:34:20.285725942 (486h30m18.837196404s) [noise=true , address=77.90.31.13:4627 , piece_id=62A76IX5BE7WFBCCEMPSENCE5PWOSEMCUIKEATUPWGSLTCFJ4BGQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:20.363243793 (486h30m18.914714244s) [noise=true , address=45.91.116.5:4297 , piece_id=GD7HY7SUS6XPN6KJDC5AVRP2ECTJSB2MK4YBQXCNUOTUBVKFRGVQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:22.551752465 (486h30m21.103222926s) [noise=true , address=145.79.227.31:4798 , piece_id=NG2JWZTD5NPFNDIVWYSWFHQHEDTURKJOY6YZ7G6LAQRIDI7WD3FQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:22.666545505 (486h30m21.218015967s) [noise=true , address=145.79.227.22:4509 , piece_id=N2AEKOF7MLRQZ2HS7FSXV3U6WNI7YQLWP3W4PTFZLMAWQQREBTKQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:23.006352209 (486h30m21.557822671s) [noise=true , address=145.79.227.14:4839 , piece_id=RDZDYLJRCVHDKTTCAF3JB5ULPKFMSC2E4HNLB2RPLPY2GTUC6ZDQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:23.023700063 (486h30m21.575170525s) [noise=true , address=145.79.227.33:4869 , piece_id=V3JJCDM7EBRRZL2GPDMSCQAEHP2YX2QXDTDZDT5M6FGVLB27G7PQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:23.426873130 (486h30m21.978343581s) [noise=true , address=145.79.227.32:4836 , piece_id=4KBSAAZLHC7OYLWN7RBDTQZUUTMK4HFGFUBES7K2PDKBXZST3OBA, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:23.722460741 (486h30m22.273931202s) [noise=true , address=145.79.227.40:4049 , piece_id=DIBJ3Y4NA3HRAZD7GLO72STJHS5XHA3OTCAMWL4VCHMEILLCL3UA, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:25.698494096 (486h30m24.249964557s) [noise=true , address=145.79.227.39:4013 , piece_id=YFEKNDHNI77M7MG76EISOGVU32UOZ536CK4X3Z6SFGOLJV2E7EHQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:25.822411189 (486h30m24.373881651s) [noise=true , address=145.79.227.2:4114 , piece_id=ZB7V2MM6KI6VTSGSC4ZD4QI6EA2265OP3MVLWD2AYAHFERAUE7CQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:29.722449985 (486h30m28.273920436s) [noise=true , address=145.79.227.12:4767 , piece_id=5Q27HBFWOOI6PIIIMXELWG3QYSEA5B5LAOMU5K4FICMUMPDZIQAA, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:47.238199596 (486h30m45.789670047s) [noise=true , address=145.79.227.20:4438 , piece_id=VZK574G5G2HXO5JMVMMEKGFCOJKOTU5KUIJDDEROFXRYTBTITO3A, seg_pos={0 0}, upload=428397]: Done uploading piece. err: ecclient: upload failed (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, address:145.79.227.20:4438): protocol: write tcp 188.166.225.170:34106->145.79.227.20:4438: use of closed network connection; write tcp 188.166.225.170:34106->145.79.227.20:4438: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34106->145.79.227.20:4438: use of closed network connection Nov 29 14:34:47.238464621 (486h30m45.789935082s) [noise=true , address=145.79.227.21:4473 , piece_id=S2V3HONDHED6NZSIFZOTWU2DQ6FKESGGZ4EVMZX36LI4NT32WDWQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:46118->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:46118->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46118->145.79.227.21:4473: use of closed network connection Nov 29 14:34:47.238703104 (486h30m45.790173565s) [noise=true , address=145.79.227.24:4583 , piece_id=B6GTB3UF3XX64M5U57FFCFAMWNWDJUDPIZDG23KOHXOVCFTGKCLA, seg_pos={0 0}, upload=428397]: Done uploading piece. err: ecclient: upload failed (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, address:145.79.227.24:4583): protocol: write tcp 188.166.225.170:51236->145.79.227.24:4583: use of closed network connection; write tcp 188.166.225.170:51236->145.79.227.24:4583: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51236->145.79.227.24:4583: use of closed network connection Nov 29 14:34:47.238869100 (486h30m45.790339561s) [noise=true , address=145.79.227.7:4295 , piece_id=OLB3PUIQ4RYD2GGVF4I32FEGXO2O4X56PLJMOKNTJVBK53KZ3VYQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:145.79.227.7:4295): protocol: write tcp 188.166.225.170:46306->145.79.227.7:4295: use of closed network connection; write tcp 188.166.225.170:46306->145.79.227.7:4295: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46306->145.79.227.7:4295: use of closed network connection Nov 29 14:34:47.239032171 (486h30m45.790502632s) [noise=true , address=145.79.227.6:4259 , piece_id=KIYGYFEAYWG2NUAQZNWG4ZLMRY6GFJADI5LD635JE5M4DLHTX4TQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:145.79.227.6:4259): protocol: write tcp 188.166.225.170:48184->145.79.227.6:4259: use of closed network connection; write tcp 188.166.225.170:48184->145.79.227.6:4259: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48184->145.79.227.6:4259: use of closed network connection Nov 29 14:34:47.239076555 (486h30m45.790547027s) [seg_pos={0 0}, upload=428397]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 29 14:34:47.481897829 (486h30m46.0333683s) [upload=428397]: Done uploading segments... Nov 29 14:34:47.481943356 (486h30m46.033413817s) [upload=428397]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.935774375s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [...................X................................X.......] duration: 3.454230981s trace: https://jaeger.dev.storj.io/trace/275e9b01e747ebf4 FAIL (2025-11-29 14:45:49 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.454280876s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 5.748533404s trace: https://jaeger.dev.storj.io/trace/4add906a25eb3f89 FAIL (2025-11-29 14:45:14 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Nov 29 14:34:17.941236229 (486h30m16.49270669s) [upload=428397]: Got next segment. Inline: false Nov 29 14:34:17.941289010 (486h30m16.492759471s) [upload=428397]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 29 14:34:17.941296063 (486h30m16.492766524s) [upload=428397]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 29 14:34:18.189160800 (486h30m16.740631261s) [seg_pos={0 0}, upload=428397]: Begin upload segment... Nov 29 14:34:18.189218760 (486h30m16.740689221s) [seg_pos={0 0}, upload=428397]: Done begin upload segment. Nov 29 14:34:18.189254097 (486h30m16.740724548s) [noise=true , address=145.79.227.40:4049 , piece_id=DIBJ3Y4NA3HRAZD7GLO72STJHS5XHA3OTCAMWL4VCHMEILLCL3UA, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.189472724 (486h30m16.740943185s) [noise=true , address=77.90.31.13:4627 , piece_id=62A76IX5BE7WFBCCEMPSENCE5PWOSEMCUIKEATUPWGSLTCFJ4BGQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.190037069 (486h30m16.74150753s) [noise=true , address=145.79.227.2:4114 , piece_id=ZB7V2MM6KI6VTSGSC4ZD4QI6EA2265OP3MVLWD2AYAHFERAUE7CQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.190276184 (486h30m16.741746645s) [noise=true , address=145.79.227.21:4473 , piece_id=S2V3HONDHED6NZSIFZOTWU2DQ6FKESGGZ4EVMZX36LI4NT32WDWQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.190487406 (486h30m16.741957867s) [noise=true , address=145.79.227.31:4798 , piece_id=NG2JWZTD5NPFNDIVWYSWFHQHEDTURKJOY6YZ7G6LAQRIDI7WD3FQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.190825751 (486h30m16.742296212s) [noise=true , address=45.91.116.5:4297 , piece_id=GD7HY7SUS6XPN6KJDC5AVRP2ECTJSB2MK4YBQXCNUOTUBVKFRGVQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.190967721 (486h30m16.742438192s) [noise=true , address=145.79.227.24:4583 , piece_id=B6GTB3UF3XX64M5U57FFCFAMWNWDJUDPIZDG23KOHXOVCFTGKCLA, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.191308490 (486h30m16.742778951s) [noise=true , address=145.79.227.33:4869 , piece_id=V3JJCDM7EBRRZL2GPDMSCQAEHP2YX2QXDTDZDT5M6FGVLB27G7PQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.191365077 (486h30m16.742835528s) [noise=true , address=145.79.227.6:4259 , piece_id=KIYGYFEAYWG2NUAQZNWG4ZLMRY6GFJADI5LD635JE5M4DLHTX4TQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.191782502 (486h30m16.743252963s) [noise=true , address=145.79.227.22:4509 , piece_id=N2AEKOF7MLRQZ2HS7FSXV3U6WNI7YQLWP3W4PTFZLMAWQQREBTKQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.192449502 (486h30m16.743919953s) [noise=true , address=145.79.227.14:4839 , piece_id=RDZDYLJRCVHDKTTCAF3JB5ULPKFMSC2E4HNLB2RPLPY2GTUC6ZDQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.193091886 (486h30m16.744562347s) [noise=true , address=145.79.227.12:4767 , piece_id=5Q27HBFWOOI6PIIIMXELWG3QYSEA5B5LAOMU5K4FICMUMPDZIQAA, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.193463874 (486h30m16.744934335s) [noise=true , address=145.79.227.39:4013 , piece_id=YFEKNDHNI77M7MG76EISOGVU32UOZ536CK4X3Z6SFGOLJV2E7EHQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.195162279 (486h30m16.74663274s) [noise=true , address=145.79.227.20:4438 , piece_id=VZK574G5G2HXO5JMVMMEKGFCOJKOTU5KUIJDDEROFXRYTBTITO3A, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.197119445 (486h30m16.748589896s) [noise=true , address=145.79.227.7:4295 , piece_id=OLB3PUIQ4RYD2GGVF4I32FEGXO2O4X56PLJMOKNTJVBK53KZ3VYQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.197903478 (486h30m16.749373929s) [noise=true , address=145.79.227.32:4836 , piece_id=4KBSAAZLHC7OYLWN7RBDTQZUUTMK4HFGFUBES7K2PDKBXZST3OBA, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:19.999395950 (486h30m18.550866411s) [upload=428397]: Next returned nil segment Nov 29 14:34:19.999415177 (486h30m18.550885648s) [upload=428397]: Waiting for error group managing segments... Nov 29 14:34:20.285725942 (486h30m18.837196404s) [noise=true , address=77.90.31.13:4627 , piece_id=62A76IX5BE7WFBCCEMPSENCE5PWOSEMCUIKEATUPWGSLTCFJ4BGQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:20.363243793 (486h30m18.914714244s) [noise=true , address=45.91.116.5:4297 , piece_id=GD7HY7SUS6XPN6KJDC5AVRP2ECTJSB2MK4YBQXCNUOTUBVKFRGVQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:22.551752465 (486h30m21.103222926s) [noise=true , address=145.79.227.31:4798 , piece_id=NG2JWZTD5NPFNDIVWYSWFHQHEDTURKJOY6YZ7G6LAQRIDI7WD3FQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:22.666545505 (486h30m21.218015967s) [noise=true , address=145.79.227.22:4509 , piece_id=N2AEKOF7MLRQZ2HS7FSXV3U6WNI7YQLWP3W4PTFZLMAWQQREBTKQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:23.006352209 (486h30m21.557822671s) [noise=true , address=145.79.227.14:4839 , piece_id=RDZDYLJRCVHDKTTCAF3JB5ULPKFMSC2E4HNLB2RPLPY2GTUC6ZDQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:23.023700063 (486h30m21.575170525s) [noise=true , address=145.79.227.33:4869 , piece_id=V3JJCDM7EBRRZL2GPDMSCQAEHP2YX2QXDTDZDT5M6FGVLB27G7PQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:23.426873130 (486h30m21.978343581s) [noise=true , address=145.79.227.32:4836 , piece_id=4KBSAAZLHC7OYLWN7RBDTQZUUTMK4HFGFUBES7K2PDKBXZST3OBA, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:23.722460741 (486h30m22.273931202s) [noise=true , address=145.79.227.40:4049 , piece_id=DIBJ3Y4NA3HRAZD7GLO72STJHS5XHA3OTCAMWL4VCHMEILLCL3UA, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:25.698494096 (486h30m24.249964557s) [noise=true , address=145.79.227.39:4013 , piece_id=YFEKNDHNI77M7MG76EISOGVU32UOZ536CK4X3Z6SFGOLJV2E7EHQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:25.822411189 (486h30m24.373881651s) [noise=true , address=145.79.227.2:4114 , piece_id=ZB7V2MM6KI6VTSGSC4ZD4QI6EA2265OP3MVLWD2AYAHFERAUE7CQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:29.722449985 (486h30m28.273920436s) [noise=true , address=145.79.227.12:4767 , piece_id=5Q27HBFWOOI6PIIIMXELWG3QYSEA5B5LAOMU5K4FICMUMPDZIQAA, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:47.238199596 (486h30m45.789670047s) [noise=true , address=145.79.227.20:4438 , piece_id=VZK574G5G2HXO5JMVMMEKGFCOJKOTU5KUIJDDEROFXRYTBTITO3A, seg_pos={0 0}, upload=428397]: Done uploading piece. err: ecclient: upload failed (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, address:145.79.227.20:4438): protocol: write tcp 188.166.225.170:34106->145.79.227.20:4438: use of closed network connection; write tcp 188.166.225.170:34106->145.79.227.20:4438: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34106->145.79.227.20:4438: use of closed network connection Nov 29 14:34:47.238464621 (486h30m45.789935082s) [noise=true , address=145.79.227.21:4473 , piece_id=S2V3HONDHED6NZSIFZOTWU2DQ6FKESGGZ4EVMZX36LI4NT32WDWQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:46118->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:46118->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46118->145.79.227.21:4473: use of closed network connection Nov 29 14:34:47.238703104 (486h30m45.790173565s) [noise=true , address=145.79.227.24:4583 , piece_id=B6GTB3UF3XX64M5U57FFCFAMWNWDJUDPIZDG23KOHXOVCFTGKCLA, seg_pos={0 0}, upload=428397]: Done uploading piece. err: ecclient: upload failed (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, address:145.79.227.24:4583): protocol: write tcp 188.166.225.170:51236->145.79.227.24:4583: use of closed network connection; write tcp 188.166.225.170:51236->145.79.227.24:4583: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51236->145.79.227.24:4583: use of closed network connection Nov 29 14:34:47.238869100 (486h30m45.790339561s) [noise=true , address=145.79.227.7:4295 , piece_id=OLB3PUIQ4RYD2GGVF4I32FEGXO2O4X56PLJMOKNTJVBK53KZ3VYQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:145.79.227.7:4295): protocol: write tcp 188.166.225.170:46306->145.79.227.7:4295: use of closed network connection; write tcp 188.166.225.170:46306->145.79.227.7:4295: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46306->145.79.227.7:4295: use of closed network connection Nov 29 14:34:47.239032171 (486h30m45.790502632s) [noise=true , address=145.79.227.6:4259 , piece_id=KIYGYFEAYWG2NUAQZNWG4ZLMRY6GFJADI5LD635JE5M4DLHTX4TQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:145.79.227.6:4259): protocol: write tcp 188.166.225.170:48184->145.79.227.6:4259: use of closed network connection; write tcp 188.166.225.170:48184->145.79.227.6:4259: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48184->145.79.227.6:4259: use of closed network connection Nov 29 14:34:47.239076555 (486h30m45.790547027s) [seg_pos={0 0}, upload=428397]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 29 14:34:47.481897829 (486h30m46.0333683s) [upload=428397]: Done uploading segments... Nov 29 14:34:47.481943356 (486h30m46.033413817s) [upload=428397]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.748563711s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 6.082238737s trace: https://jaeger.dev.storj.io/trace/61a1d0645153f16 FAIL (2025-11-29 14:45:02 UTC) orbiter warning: context deadline exceeded Duration: 6.082268263s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [....................X................................X......] duration: 3.483924571s trace: https://jaeger.dev.storj.io/trace/275e9b01e747ebf4 FAIL (2025-11-29 14:44:55 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.484002791s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 5.960311487s trace: https://jaeger.dev.storj.io/trace/4add906a25eb3f89 FAIL (2025-11-29 14:44:03 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Nov 29 14:34:17.941236229 (486h30m16.49270669s) [upload=428397]: Got next segment. Inline: false Nov 29 14:34:17.941289010 (486h30m16.492759471s) [upload=428397]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 29 14:34:17.941296063 (486h30m16.492766524s) [upload=428397]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 29 14:34:18.189160800 (486h30m16.740631261s) [seg_pos={0 0}, upload=428397]: Begin upload segment... Nov 29 14:34:18.189218760 (486h30m16.740689221s) [seg_pos={0 0}, upload=428397]: Done begin upload segment. Nov 29 14:34:18.189254097 (486h30m16.740724548s) [noise=true , address=145.79.227.40:4049 , piece_id=DIBJ3Y4NA3HRAZD7GLO72STJHS5XHA3OTCAMWL4VCHMEILLCL3UA, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.189472724 (486h30m16.740943185s) [noise=true , address=77.90.31.13:4627 , piece_id=62A76IX5BE7WFBCCEMPSENCE5PWOSEMCUIKEATUPWGSLTCFJ4BGQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.190037069 (486h30m16.74150753s) [noise=true , address=145.79.227.2:4114 , piece_id=ZB7V2MM6KI6VTSGSC4ZD4QI6EA2265OP3MVLWD2AYAHFERAUE7CQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.190276184 (486h30m16.741746645s) [noise=true , address=145.79.227.21:4473 , piece_id=S2V3HONDHED6NZSIFZOTWU2DQ6FKESGGZ4EVMZX36LI4NT32WDWQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.190487406 (486h30m16.741957867s) [noise=true , address=145.79.227.31:4798 , piece_id=NG2JWZTD5NPFNDIVWYSWFHQHEDTURKJOY6YZ7G6LAQRIDI7WD3FQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.190825751 (486h30m16.742296212s) [noise=true , address=45.91.116.5:4297 , piece_id=GD7HY7SUS6XPN6KJDC5AVRP2ECTJSB2MK4YBQXCNUOTUBVKFRGVQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.190967721 (486h30m16.742438192s) [noise=true , address=145.79.227.24:4583 , piece_id=B6GTB3UF3XX64M5U57FFCFAMWNWDJUDPIZDG23KOHXOVCFTGKCLA, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.191308490 (486h30m16.742778951s) [noise=true , address=145.79.227.33:4869 , piece_id=V3JJCDM7EBRRZL2GPDMSCQAEHP2YX2QXDTDZDT5M6FGVLB27G7PQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.191365077 (486h30m16.742835528s) [noise=true , address=145.79.227.6:4259 , piece_id=KIYGYFEAYWG2NUAQZNWG4ZLMRY6GFJADI5LD635JE5M4DLHTX4TQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.191782502 (486h30m16.743252963s) [noise=true , address=145.79.227.22:4509 , piece_id=N2AEKOF7MLRQZ2HS7FSXV3U6WNI7YQLWP3W4PTFZLMAWQQREBTKQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.192449502 (486h30m16.743919953s) [noise=true , address=145.79.227.14:4839 , piece_id=RDZDYLJRCVHDKTTCAF3JB5ULPKFMSC2E4HNLB2RPLPY2GTUC6ZDQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.193091886 (486h30m16.744562347s) [noise=true , address=145.79.227.12:4767 , piece_id=5Q27HBFWOOI6PIIIMXELWG3QYSEA5B5LAOMU5K4FICMUMPDZIQAA, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.193463874 (486h30m16.744934335s) [noise=true , address=145.79.227.39:4013 , piece_id=YFEKNDHNI77M7MG76EISOGVU32UOZ536CK4X3Z6SFGOLJV2E7EHQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.195162279 (486h30m16.74663274s) [noise=true , address=145.79.227.20:4438 , piece_id=VZK574G5G2HXO5JMVMMEKGFCOJKOTU5KUIJDDEROFXRYTBTITO3A, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.197119445 (486h30m16.748589896s) [noise=true , address=145.79.227.7:4295 , piece_id=OLB3PUIQ4RYD2GGVF4I32FEGXO2O4X56PLJMOKNTJVBK53KZ3VYQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.197903478 (486h30m16.749373929s) [noise=true , address=145.79.227.32:4836 , piece_id=4KBSAAZLHC7OYLWN7RBDTQZUUTMK4HFGFUBES7K2PDKBXZST3OBA, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:19.999395950 (486h30m18.550866411s) [upload=428397]: Next returned nil segment Nov 29 14:34:19.999415177 (486h30m18.550885648s) [upload=428397]: Waiting for error group managing segments... Nov 29 14:34:20.285725942 (486h30m18.837196404s) [noise=true , address=77.90.31.13:4627 , piece_id=62A76IX5BE7WFBCCEMPSENCE5PWOSEMCUIKEATUPWGSLTCFJ4BGQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:20.363243793 (486h30m18.914714244s) [noise=true , address=45.91.116.5:4297 , piece_id=GD7HY7SUS6XPN6KJDC5AVRP2ECTJSB2MK4YBQXCNUOTUBVKFRGVQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:22.551752465 (486h30m21.103222926s) [noise=true , address=145.79.227.31:4798 , piece_id=NG2JWZTD5NPFNDIVWYSWFHQHEDTURKJOY6YZ7G6LAQRIDI7WD3FQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:22.666545505 (486h30m21.218015967s) [noise=true , address=145.79.227.22:4509 , piece_id=N2AEKOF7MLRQZ2HS7FSXV3U6WNI7YQLWP3W4PTFZLMAWQQREBTKQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:23.006352209 (486h30m21.557822671s) [noise=true , address=145.79.227.14:4839 , piece_id=RDZDYLJRCVHDKTTCAF3JB5ULPKFMSC2E4HNLB2RPLPY2GTUC6ZDQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:23.023700063 (486h30m21.575170525s) [noise=true , address=145.79.227.33:4869 , piece_id=V3JJCDM7EBRRZL2GPDMSCQAEHP2YX2QXDTDZDT5M6FGVLB27G7PQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:23.426873130 (486h30m21.978343581s) [noise=true , address=145.79.227.32:4836 , piece_id=4KBSAAZLHC7OYLWN7RBDTQZUUTMK4HFGFUBES7K2PDKBXZST3OBA, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:23.722460741 (486h30m22.273931202s) [noise=true , address=145.79.227.40:4049 , piece_id=DIBJ3Y4NA3HRAZD7GLO72STJHS5XHA3OTCAMWL4VCHMEILLCL3UA, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:25.698494096 (486h30m24.249964557s) [noise=true , address=145.79.227.39:4013 , piece_id=YFEKNDHNI77M7MG76EISOGVU32UOZ536CK4X3Z6SFGOLJV2E7EHQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:25.822411189 (486h30m24.373881651s) [noise=true , address=145.79.227.2:4114 , piece_id=ZB7V2MM6KI6VTSGSC4ZD4QI6EA2265OP3MVLWD2AYAHFERAUE7CQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:29.722449985 (486h30m28.273920436s) [noise=true , address=145.79.227.12:4767 , piece_id=5Q27HBFWOOI6PIIIMXELWG3QYSEA5B5LAOMU5K4FICMUMPDZIQAA, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:47.238199596 (486h30m45.789670047s) [noise=true , address=145.79.227.20:4438 , piece_id=VZK574G5G2HXO5JMVMMEKGFCOJKOTU5KUIJDDEROFXRYTBTITO3A, seg_pos={0 0}, upload=428397]: Done uploading piece. err: ecclient: upload failed (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, address:145.79.227.20:4438): protocol: write tcp 188.166.225.170:34106->145.79.227.20:4438: use of closed network connection; write tcp 188.166.225.170:34106->145.79.227.20:4438: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34106->145.79.227.20:4438: use of closed network connection Nov 29 14:34:47.238464621 (486h30m45.789935082s) [noise=true , address=145.79.227.21:4473 , piece_id=S2V3HONDHED6NZSIFZOTWU2DQ6FKESGGZ4EVMZX36LI4NT32WDWQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:46118->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:46118->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46118->145.79.227.21:4473: use of closed network connection Nov 29 14:34:47.238703104 (486h30m45.790173565s) [noise=true , address=145.79.227.24:4583 , piece_id=B6GTB3UF3XX64M5U57FFCFAMWNWDJUDPIZDG23KOHXOVCFTGKCLA, seg_pos={0 0}, upload=428397]: Done uploading piece. err: ecclient: upload failed (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, address:145.79.227.24:4583): protocol: write tcp 188.166.225.170:51236->145.79.227.24:4583: use of closed network connection; write tcp 188.166.225.170:51236->145.79.227.24:4583: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51236->145.79.227.24:4583: use of closed network connection Nov 29 14:34:47.238869100 (486h30m45.790339561s) [noise=true , address=145.79.227.7:4295 , piece_id=OLB3PUIQ4RYD2GGVF4I32FEGXO2O4X56PLJMOKNTJVBK53KZ3VYQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:145.79.227.7:4295): protocol: write tcp 188.166.225.170:46306->145.79.227.7:4295: use of closed network connection; write tcp 188.166.225.170:46306->145.79.227.7:4295: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46306->145.79.227.7:4295: use of closed network connection Nov 29 14:34:47.239032171 (486h30m45.790502632s) [noise=true , address=145.79.227.6:4259 , piece_id=KIYGYFEAYWG2NUAQZNWG4ZLMRY6GFJADI5LD635JE5M4DLHTX4TQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:145.79.227.6:4259): protocol: write tcp 188.166.225.170:48184->145.79.227.6:4259: use of closed network connection; write tcp 188.166.225.170:48184->145.79.227.6:4259: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48184->145.79.227.6:4259: use of closed network connection Nov 29 14:34:47.239076555 (486h30m45.790547027s) [seg_pos={0 0}, upload=428397]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 29 14:34:47.481897829 (486h30m46.0333683s) [upload=428397]: Done uploading segments... Nov 29 14:34:47.481943356 (486h30m46.033413817s) [upload=428397]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.960336385s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 7.477651357s trace: https://jaeger.dev.storj.io/trace/61a1d0645153f16 FAIL (2025-11-29 14:43:57 UTC) orbiter warning: context deadline exceeded Duration: 7.477682227s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.....................X................................X.....] duration: 3.515161717s trace: https://jaeger.dev.storj.io/trace/275e9b01e747ebf4 FAIL (2025-11-29 14:43:51 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.515187777s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 5.927068206s trace: https://jaeger.dev.storj.io/trace/4add906a25eb3f89 FAIL (2025-11-29 14:43:13 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Nov 29 14:34:17.941236229 (486h30m16.49270669s) [upload=428397]: Got next segment. Inline: false Nov 29 14:34:17.941289010 (486h30m16.492759471s) [upload=428397]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 29 14:34:17.941296063 (486h30m16.492766524s) [upload=428397]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 29 14:34:18.189160800 (486h30m16.740631261s) [seg_pos={0 0}, upload=428397]: Begin upload segment... Nov 29 14:34:18.189218760 (486h30m16.740689221s) [seg_pos={0 0}, upload=428397]: Done begin upload segment. Nov 29 14:34:18.189254097 (486h30m16.740724548s) [noise=true , address=145.79.227.40:4049 , piece_id=DIBJ3Y4NA3HRAZD7GLO72STJHS5XHA3OTCAMWL4VCHMEILLCL3UA, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.189472724 (486h30m16.740943185s) [noise=true , address=77.90.31.13:4627 , piece_id=62A76IX5BE7WFBCCEMPSENCE5PWOSEMCUIKEATUPWGSLTCFJ4BGQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.190037069 (486h30m16.74150753s) [noise=true , address=145.79.227.2:4114 , piece_id=ZB7V2MM6KI6VTSGSC4ZD4QI6EA2265OP3MVLWD2AYAHFERAUE7CQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.190276184 (486h30m16.741746645s) [noise=true , address=145.79.227.21:4473 , piece_id=S2V3HONDHED6NZSIFZOTWU2DQ6FKESGGZ4EVMZX36LI4NT32WDWQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.190487406 (486h30m16.741957867s) [noise=true , address=145.79.227.31:4798 , piece_id=NG2JWZTD5NPFNDIVWYSWFHQHEDTURKJOY6YZ7G6LAQRIDI7WD3FQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.190825751 (486h30m16.742296212s) [noise=true , address=45.91.116.5:4297 , piece_id=GD7HY7SUS6XPN6KJDC5AVRP2ECTJSB2MK4YBQXCNUOTUBVKFRGVQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.190967721 (486h30m16.742438192s) [noise=true , address=145.79.227.24:4583 , piece_id=B6GTB3UF3XX64M5U57FFCFAMWNWDJUDPIZDG23KOHXOVCFTGKCLA, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.191308490 (486h30m16.742778951s) [noise=true , address=145.79.227.33:4869 , piece_id=V3JJCDM7EBRRZL2GPDMSCQAEHP2YX2QXDTDZDT5M6FGVLB27G7PQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.191365077 (486h30m16.742835528s) [noise=true , address=145.79.227.6:4259 , piece_id=KIYGYFEAYWG2NUAQZNWG4ZLMRY6GFJADI5LD635JE5M4DLHTX4TQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.191782502 (486h30m16.743252963s) [noise=true , address=145.79.227.22:4509 , piece_id=N2AEKOF7MLRQZ2HS7FSXV3U6WNI7YQLWP3W4PTFZLMAWQQREBTKQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.192449502 (486h30m16.743919953s) [noise=true , address=145.79.227.14:4839 , piece_id=RDZDYLJRCVHDKTTCAF3JB5ULPKFMSC2E4HNLB2RPLPY2GTUC6ZDQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.193091886 (486h30m16.744562347s) [noise=true , address=145.79.227.12:4767 , piece_id=5Q27HBFWOOI6PIIIMXELWG3QYSEA5B5LAOMU5K4FICMUMPDZIQAA, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.193463874 (486h30m16.744934335s) [noise=true , address=145.79.227.39:4013 , piece_id=YFEKNDHNI77M7MG76EISOGVU32UOZ536CK4X3Z6SFGOLJV2E7EHQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.195162279 (486h30m16.74663274s) [noise=true , address=145.79.227.20:4438 , piece_id=VZK574G5G2HXO5JMVMMEKGFCOJKOTU5KUIJDDEROFXRYTBTITO3A, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.197119445 (486h30m16.748589896s) [noise=true , address=145.79.227.7:4295 , piece_id=OLB3PUIQ4RYD2GGVF4I32FEGXO2O4X56PLJMOKNTJVBK53KZ3VYQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.197903478 (486h30m16.749373929s) [noise=true , address=145.79.227.32:4836 , piece_id=4KBSAAZLHC7OYLWN7RBDTQZUUTMK4HFGFUBES7K2PDKBXZST3OBA, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:19.999395950 (486h30m18.550866411s) [upload=428397]: Next returned nil segment Nov 29 14:34:19.999415177 (486h30m18.550885648s) [upload=428397]: Waiting for error group managing segments... Nov 29 14:34:20.285725942 (486h30m18.837196404s) [noise=true , address=77.90.31.13:4627 , piece_id=62A76IX5BE7WFBCCEMPSENCE5PWOSEMCUIKEATUPWGSLTCFJ4BGQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:20.363243793 (486h30m18.914714244s) [noise=true , address=45.91.116.5:4297 , piece_id=GD7HY7SUS6XPN6KJDC5AVRP2ECTJSB2MK4YBQXCNUOTUBVKFRGVQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:22.551752465 (486h30m21.103222926s) [noise=true , address=145.79.227.31:4798 , piece_id=NG2JWZTD5NPFNDIVWYSWFHQHEDTURKJOY6YZ7G6LAQRIDI7WD3FQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:22.666545505 (486h30m21.218015967s) [noise=true , address=145.79.227.22:4509 , piece_id=N2AEKOF7MLRQZ2HS7FSXV3U6WNI7YQLWP3W4PTFZLMAWQQREBTKQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:23.006352209 (486h30m21.557822671s) [noise=true , address=145.79.227.14:4839 , piece_id=RDZDYLJRCVHDKTTCAF3JB5ULPKFMSC2E4HNLB2RPLPY2GTUC6ZDQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:23.023700063 (486h30m21.575170525s) [noise=true , address=145.79.227.33:4869 , piece_id=V3JJCDM7EBRRZL2GPDMSCQAEHP2YX2QXDTDZDT5M6FGVLB27G7PQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:23.426873130 (486h30m21.978343581s) [noise=true , address=145.79.227.32:4836 , piece_id=4KBSAAZLHC7OYLWN7RBDTQZUUTMK4HFGFUBES7K2PDKBXZST3OBA, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:23.722460741 (486h30m22.273931202s) [noise=true , address=145.79.227.40:4049 , piece_id=DIBJ3Y4NA3HRAZD7GLO72STJHS5XHA3OTCAMWL4VCHMEILLCL3UA, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:25.698494096 (486h30m24.249964557s) [noise=true , address=145.79.227.39:4013 , piece_id=YFEKNDHNI77M7MG76EISOGVU32UOZ536CK4X3Z6SFGOLJV2E7EHQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:25.822411189 (486h30m24.373881651s) [noise=true , address=145.79.227.2:4114 , piece_id=ZB7V2MM6KI6VTSGSC4ZD4QI6EA2265OP3MVLWD2AYAHFERAUE7CQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:29.722449985 (486h30m28.273920436s) [noise=true , address=145.79.227.12:4767 , piece_id=5Q27HBFWOOI6PIIIMXELWG3QYSEA5B5LAOMU5K4FICMUMPDZIQAA, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:47.238199596 (486h30m45.789670047s) [noise=true , address=145.79.227.20:4438 , piece_id=VZK574G5G2HXO5JMVMMEKGFCOJKOTU5KUIJDDEROFXRYTBTITO3A, seg_pos={0 0}, upload=428397]: Done uploading piece. err: ecclient: upload failed (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, address:145.79.227.20:4438): protocol: write tcp 188.166.225.170:34106->145.79.227.20:4438: use of closed network connection; write tcp 188.166.225.170:34106->145.79.227.20:4438: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34106->145.79.227.20:4438: use of closed network connection Nov 29 14:34:47.238464621 (486h30m45.789935082s) [noise=true , address=145.79.227.21:4473 , piece_id=S2V3HONDHED6NZSIFZOTWU2DQ6FKESGGZ4EVMZX36LI4NT32WDWQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:46118->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:46118->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46118->145.79.227.21:4473: use of closed network connection Nov 29 14:34:47.238703104 (486h30m45.790173565s) [noise=true , address=145.79.227.24:4583 , piece_id=B6GTB3UF3XX64M5U57FFCFAMWNWDJUDPIZDG23KOHXOVCFTGKCLA, seg_pos={0 0}, upload=428397]: Done uploading piece. err: ecclient: upload failed (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, address:145.79.227.24:4583): protocol: write tcp 188.166.225.170:51236->145.79.227.24:4583: use of closed network connection; write tcp 188.166.225.170:51236->145.79.227.24:4583: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51236->145.79.227.24:4583: use of closed network connection Nov 29 14:34:47.238869100 (486h30m45.790339561s) [noise=true , address=145.79.227.7:4295 , piece_id=OLB3PUIQ4RYD2GGVF4I32FEGXO2O4X56PLJMOKNTJVBK53KZ3VYQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:145.79.227.7:4295): protocol: write tcp 188.166.225.170:46306->145.79.227.7:4295: use of closed network connection; write tcp 188.166.225.170:46306->145.79.227.7:4295: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46306->145.79.227.7:4295: use of closed network connection Nov 29 14:34:47.239032171 (486h30m45.790502632s) [noise=true , address=145.79.227.6:4259 , piece_id=KIYGYFEAYWG2NUAQZNWG4ZLMRY6GFJADI5LD635JE5M4DLHTX4TQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:145.79.227.6:4259): protocol: write tcp 188.166.225.170:48184->145.79.227.6:4259: use of closed network connection; write tcp 188.166.225.170:48184->145.79.227.6:4259: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48184->145.79.227.6:4259: use of closed network connection Nov 29 14:34:47.239076555 (486h30m45.790547027s) [seg_pos={0 0}, upload=428397]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 29 14:34:47.481897829 (486h30m46.0333683s) [upload=428397]: Done uploading segments... Nov 29 14:34:47.481943356 (486h30m46.033413817s) [upload=428397]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.927105046s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [X.....................X................................X....] duration: 3.784112833s trace: https://jaeger.dev.storj.io/trace/275e9b01e747ebf4 FAIL (2025-11-29 14:43:07 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.784162277s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 10.191889929s trace: https://jaeger.dev.storj.io/trace/61a1d0645153f16 FAIL (2025-11-29 14:42:58 UTC) orbiter warning: context deadline exceeded Duration: 10.192482437s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 5.698620193s trace: https://jaeger.dev.storj.io/trace/61a1d0645153f16 FAIL (2025-11-29 14:42:19 UTC) orbiter warning: context deadline exceeded Duration: 5.69864992s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 5.802233916s trace: https://jaeger.dev.storj.io/trace/4add906a25eb3f89 FAIL (2025-11-29 14:42:02 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Nov 29 14:34:17.941236229 (486h30m16.49270669s) [upload=428397]: Got next segment. Inline: false Nov 29 14:34:17.941289010 (486h30m16.492759471s) [upload=428397]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 29 14:34:17.941296063 (486h30m16.492766524s) [upload=428397]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 29 14:34:18.189160800 (486h30m16.740631261s) [seg_pos={0 0}, upload=428397]: Begin upload segment... Nov 29 14:34:18.189218760 (486h30m16.740689221s) [seg_pos={0 0}, upload=428397]: Done begin upload segment. Nov 29 14:34:18.189254097 (486h30m16.740724548s) [noise=true , address=145.79.227.40:4049 , piece_id=DIBJ3Y4NA3HRAZD7GLO72STJHS5XHA3OTCAMWL4VCHMEILLCL3UA, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.189472724 (486h30m16.740943185s) [noise=true , address=77.90.31.13:4627 , piece_id=62A76IX5BE7WFBCCEMPSENCE5PWOSEMCUIKEATUPWGSLTCFJ4BGQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.190037069 (486h30m16.74150753s) [noise=true , address=145.79.227.2:4114 , piece_id=ZB7V2MM6KI6VTSGSC4ZD4QI6EA2265OP3MVLWD2AYAHFERAUE7CQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.190276184 (486h30m16.741746645s) [noise=true , address=145.79.227.21:4473 , piece_id=S2V3HONDHED6NZSIFZOTWU2DQ6FKESGGZ4EVMZX36LI4NT32WDWQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.190487406 (486h30m16.741957867s) [noise=true , address=145.79.227.31:4798 , piece_id=NG2JWZTD5NPFNDIVWYSWFHQHEDTURKJOY6YZ7G6LAQRIDI7WD3FQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.190825751 (486h30m16.742296212s) [noise=true , address=45.91.116.5:4297 , piece_id=GD7HY7SUS6XPN6KJDC5AVRP2ECTJSB2MK4YBQXCNUOTUBVKFRGVQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.190967721 (486h30m16.742438192s) [noise=true , address=145.79.227.24:4583 , piece_id=B6GTB3UF3XX64M5U57FFCFAMWNWDJUDPIZDG23KOHXOVCFTGKCLA, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.191308490 (486h30m16.742778951s) [noise=true , address=145.79.227.33:4869 , piece_id=V3JJCDM7EBRRZL2GPDMSCQAEHP2YX2QXDTDZDT5M6FGVLB27G7PQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.191365077 (486h30m16.742835528s) [noise=true , address=145.79.227.6:4259 , piece_id=KIYGYFEAYWG2NUAQZNWG4ZLMRY6GFJADI5LD635JE5M4DLHTX4TQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.191782502 (486h30m16.743252963s) [noise=true , address=145.79.227.22:4509 , piece_id=N2AEKOF7MLRQZ2HS7FSXV3U6WNI7YQLWP3W4PTFZLMAWQQREBTKQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.192449502 (486h30m16.743919953s) [noise=true , address=145.79.227.14:4839 , piece_id=RDZDYLJRCVHDKTTCAF3JB5ULPKFMSC2E4HNLB2RPLPY2GTUC6ZDQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.193091886 (486h30m16.744562347s) [noise=true , address=145.79.227.12:4767 , piece_id=5Q27HBFWOOI6PIIIMXELWG3QYSEA5B5LAOMU5K4FICMUMPDZIQAA, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.193463874 (486h30m16.744934335s) [noise=true , address=145.79.227.39:4013 , piece_id=YFEKNDHNI77M7MG76EISOGVU32UOZ536CK4X3Z6SFGOLJV2E7EHQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.195162279 (486h30m16.74663274s) [noise=true , address=145.79.227.20:4438 , piece_id=VZK574G5G2HXO5JMVMMEKGFCOJKOTU5KUIJDDEROFXRYTBTITO3A, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.197119445 (486h30m16.748589896s) [noise=true , address=145.79.227.7:4295 , piece_id=OLB3PUIQ4RYD2GGVF4I32FEGXO2O4X56PLJMOKNTJVBK53KZ3VYQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.197903478 (486h30m16.749373929s) [noise=true , address=145.79.227.32:4836 , piece_id=4KBSAAZLHC7OYLWN7RBDTQZUUTMK4HFGFUBES7K2PDKBXZST3OBA, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:19.999395950 (486h30m18.550866411s) [upload=428397]: Next returned nil segment Nov 29 14:34:19.999415177 (486h30m18.550885648s) [upload=428397]: Waiting for error group managing segments... Nov 29 14:34:20.285725942 (486h30m18.837196404s) [noise=true , address=77.90.31.13:4627 , piece_id=62A76IX5BE7WFBCCEMPSENCE5PWOSEMCUIKEATUPWGSLTCFJ4BGQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:20.363243793 (486h30m18.914714244s) [noise=true , address=45.91.116.5:4297 , piece_id=GD7HY7SUS6XPN6KJDC5AVRP2ECTJSB2MK4YBQXCNUOTUBVKFRGVQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:22.551752465 (486h30m21.103222926s) [noise=true , address=145.79.227.31:4798 , piece_id=NG2JWZTD5NPFNDIVWYSWFHQHEDTURKJOY6YZ7G6LAQRIDI7WD3FQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:22.666545505 (486h30m21.218015967s) [noise=true , address=145.79.227.22:4509 , piece_id=N2AEKOF7MLRQZ2HS7FSXV3U6WNI7YQLWP3W4PTFZLMAWQQREBTKQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:23.006352209 (486h30m21.557822671s) [noise=true , address=145.79.227.14:4839 , piece_id=RDZDYLJRCVHDKTTCAF3JB5ULPKFMSC2E4HNLB2RPLPY2GTUC6ZDQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:23.023700063 (486h30m21.575170525s) [noise=true , address=145.79.227.33:4869 , piece_id=V3JJCDM7EBRRZL2GPDMSCQAEHP2YX2QXDTDZDT5M6FGVLB27G7PQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:23.426873130 (486h30m21.978343581s) [noise=true , address=145.79.227.32:4836 , piece_id=4KBSAAZLHC7OYLWN7RBDTQZUUTMK4HFGFUBES7K2PDKBXZST3OBA, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:23.722460741 (486h30m22.273931202s) [noise=true , address=145.79.227.40:4049 , piece_id=DIBJ3Y4NA3HRAZD7GLO72STJHS5XHA3OTCAMWL4VCHMEILLCL3UA, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:25.698494096 (486h30m24.249964557s) [noise=true , address=145.79.227.39:4013 , piece_id=YFEKNDHNI77M7MG76EISOGVU32UOZ536CK4X3Z6SFGOLJV2E7EHQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:25.822411189 (486h30m24.373881651s) [noise=true , address=145.79.227.2:4114 , piece_id=ZB7V2MM6KI6VTSGSC4ZD4QI6EA2265OP3MVLWD2AYAHFERAUE7CQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:29.722449985 (486h30m28.273920436s) [noise=true , address=145.79.227.12:4767 , piece_id=5Q27HBFWOOI6PIIIMXELWG3QYSEA5B5LAOMU5K4FICMUMPDZIQAA, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:47.238199596 (486h30m45.789670047s) [noise=true , address=145.79.227.20:4438 , piece_id=VZK574G5G2HXO5JMVMMEKGFCOJKOTU5KUIJDDEROFXRYTBTITO3A, seg_pos={0 0}, upload=428397]: Done uploading piece. err: ecclient: upload failed (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, address:145.79.227.20:4438): protocol: write tcp 188.166.225.170:34106->145.79.227.20:4438: use of closed network connection; write tcp 188.166.225.170:34106->145.79.227.20:4438: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34106->145.79.227.20:4438: use of closed network connection Nov 29 14:34:47.238464621 (486h30m45.789935082s) [noise=true , address=145.79.227.21:4473 , piece_id=S2V3HONDHED6NZSIFZOTWU2DQ6FKESGGZ4EVMZX36LI4NT32WDWQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:46118->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:46118->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46118->145.79.227.21:4473: use of closed network connection Nov 29 14:34:47.238703104 (486h30m45.790173565s) [noise=true , address=145.79.227.24:4583 , piece_id=B6GTB3UF3XX64M5U57FFCFAMWNWDJUDPIZDG23KOHXOVCFTGKCLA, seg_pos={0 0}, upload=428397]: Done uploading piece. err: ecclient: upload failed (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, address:145.79.227.24:4583): protocol: write tcp 188.166.225.170:51236->145.79.227.24:4583: use of closed network connection; write tcp 188.166.225.170:51236->145.79.227.24:4583: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51236->145.79.227.24:4583: use of closed network connection Nov 29 14:34:47.238869100 (486h30m45.790339561s) [noise=true , address=145.79.227.7:4295 , piece_id=OLB3PUIQ4RYD2GGVF4I32FEGXO2O4X56PLJMOKNTJVBK53KZ3VYQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:145.79.227.7:4295): protocol: write tcp 188.166.225.170:46306->145.79.227.7:4295: use of closed network connection; write tcp 188.166.225.170:46306->145.79.227.7:4295: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46306->145.79.227.7:4295: use of closed network connection Nov 29 14:34:47.239032171 (486h30m45.790502632s) [noise=true , address=145.79.227.6:4259 , piece_id=KIYGYFEAYWG2NUAQZNWG4ZLMRY6GFJADI5LD635JE5M4DLHTX4TQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:145.79.227.6:4259): protocol: write tcp 188.166.225.170:48184->145.79.227.6:4259: use of closed network connection; write tcp 188.166.225.170:48184->145.79.227.6:4259: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48184->145.79.227.6:4259: use of closed network connection Nov 29 14:34:47.239076555 (486h30m45.790547027s) [seg_pos={0 0}, upload=428397]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 29 14:34:47.481897829 (486h30m46.0333683s) [upload=428397]: Done uploading segments... Nov 29 14:34:47.481943356 (486h30m46.033413817s) [upload=428397]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.802345418s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [.X.....................X................................X...] duration: 3.65604174s trace: https://jaeger.dev.storj.io/trace/275e9b01e747ebf4 FAIL (2025-11-29 14:41:54 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.656065445s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 5.914166425s trace: https://jaeger.dev.storj.io/trace/61a1d0645153f16 FAIL (2025-11-29 14:41:08 UTC) orbiter warning: context deadline exceeded Duration: 5.914184599s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [..X.....................X................................X..] duration: 3.594529344s trace: https://jaeger.dev.storj.io/trace/275e9b01e747ebf4 FAIL (2025-11-29 14:41:04 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.594575883s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 5.794277938s trace: https://jaeger.dev.storj.io/trace/4add906a25eb3f89 FAIL (2025-11-29 14:40:48 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Nov 29 14:34:17.941236229 (486h30m16.49270669s) [upload=428397]: Got next segment. Inline: false Nov 29 14:34:17.941289010 (486h30m16.492759471s) [upload=428397]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 29 14:34:17.941296063 (486h30m16.492766524s) [upload=428397]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 29 14:34:18.189160800 (486h30m16.740631261s) [seg_pos={0 0}, upload=428397]: Begin upload segment... Nov 29 14:34:18.189218760 (486h30m16.740689221s) [seg_pos={0 0}, upload=428397]: Done begin upload segment. Nov 29 14:34:18.189254097 (486h30m16.740724548s) [noise=true , address=145.79.227.40:4049 , piece_id=DIBJ3Y4NA3HRAZD7GLO72STJHS5XHA3OTCAMWL4VCHMEILLCL3UA, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.189472724 (486h30m16.740943185s) [noise=true , address=77.90.31.13:4627 , piece_id=62A76IX5BE7WFBCCEMPSENCE5PWOSEMCUIKEATUPWGSLTCFJ4BGQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.190037069 (486h30m16.74150753s) [noise=true , address=145.79.227.2:4114 , piece_id=ZB7V2MM6KI6VTSGSC4ZD4QI6EA2265OP3MVLWD2AYAHFERAUE7CQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.190276184 (486h30m16.741746645s) [noise=true , address=145.79.227.21:4473 , piece_id=S2V3HONDHED6NZSIFZOTWU2DQ6FKESGGZ4EVMZX36LI4NT32WDWQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.190487406 (486h30m16.741957867s) [noise=true , address=145.79.227.31:4798 , piece_id=NG2JWZTD5NPFNDIVWYSWFHQHEDTURKJOY6YZ7G6LAQRIDI7WD3FQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.190825751 (486h30m16.742296212s) [noise=true , address=45.91.116.5:4297 , piece_id=GD7HY7SUS6XPN6KJDC5AVRP2ECTJSB2MK4YBQXCNUOTUBVKFRGVQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.190967721 (486h30m16.742438192s) [noise=true , address=145.79.227.24:4583 , piece_id=B6GTB3UF3XX64M5U57FFCFAMWNWDJUDPIZDG23KOHXOVCFTGKCLA, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.191308490 (486h30m16.742778951s) [noise=true , address=145.79.227.33:4869 , piece_id=V3JJCDM7EBRRZL2GPDMSCQAEHP2YX2QXDTDZDT5M6FGVLB27G7PQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.191365077 (486h30m16.742835528s) [noise=true , address=145.79.227.6:4259 , piece_id=KIYGYFEAYWG2NUAQZNWG4ZLMRY6GFJADI5LD635JE5M4DLHTX4TQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.191782502 (486h30m16.743252963s) [noise=true , address=145.79.227.22:4509 , piece_id=N2AEKOF7MLRQZ2HS7FSXV3U6WNI7YQLWP3W4PTFZLMAWQQREBTKQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.192449502 (486h30m16.743919953s) [noise=true , address=145.79.227.14:4839 , piece_id=RDZDYLJRCVHDKTTCAF3JB5ULPKFMSC2E4HNLB2RPLPY2GTUC6ZDQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.193091886 (486h30m16.744562347s) [noise=true , address=145.79.227.12:4767 , piece_id=5Q27HBFWOOI6PIIIMXELWG3QYSEA5B5LAOMU5K4FICMUMPDZIQAA, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.193463874 (486h30m16.744934335s) [noise=true , address=145.79.227.39:4013 , piece_id=YFEKNDHNI77M7MG76EISOGVU32UOZ536CK4X3Z6SFGOLJV2E7EHQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.195162279 (486h30m16.74663274s) [noise=true , address=145.79.227.20:4438 , piece_id=VZK574G5G2HXO5JMVMMEKGFCOJKOTU5KUIJDDEROFXRYTBTITO3A, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.197119445 (486h30m16.748589896s) [noise=true , address=145.79.227.7:4295 , piece_id=OLB3PUIQ4RYD2GGVF4I32FEGXO2O4X56PLJMOKNTJVBK53KZ3VYQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.197903478 (486h30m16.749373929s) [noise=true , address=145.79.227.32:4836 , piece_id=4KBSAAZLHC7OYLWN7RBDTQZUUTMK4HFGFUBES7K2PDKBXZST3OBA, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:19.999395950 (486h30m18.550866411s) [upload=428397]: Next returned nil segment Nov 29 14:34:19.999415177 (486h30m18.550885648s) [upload=428397]: Waiting for error group managing segments... Nov 29 14:34:20.285725942 (486h30m18.837196404s) [noise=true , address=77.90.31.13:4627 , piece_id=62A76IX5BE7WFBCCEMPSENCE5PWOSEMCUIKEATUPWGSLTCFJ4BGQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:20.363243793 (486h30m18.914714244s) [noise=true , address=45.91.116.5:4297 , piece_id=GD7HY7SUS6XPN6KJDC5AVRP2ECTJSB2MK4YBQXCNUOTUBVKFRGVQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:22.551752465 (486h30m21.103222926s) [noise=true , address=145.79.227.31:4798 , piece_id=NG2JWZTD5NPFNDIVWYSWFHQHEDTURKJOY6YZ7G6LAQRIDI7WD3FQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:22.666545505 (486h30m21.218015967s) [noise=true , address=145.79.227.22:4509 , piece_id=N2AEKOF7MLRQZ2HS7FSXV3U6WNI7YQLWP3W4PTFZLMAWQQREBTKQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:23.006352209 (486h30m21.557822671s) [noise=true , address=145.79.227.14:4839 , piece_id=RDZDYLJRCVHDKTTCAF3JB5ULPKFMSC2E4HNLB2RPLPY2GTUC6ZDQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:23.023700063 (486h30m21.575170525s) [noise=true , address=145.79.227.33:4869 , piece_id=V3JJCDM7EBRRZL2GPDMSCQAEHP2YX2QXDTDZDT5M6FGVLB27G7PQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:23.426873130 (486h30m21.978343581s) [noise=true , address=145.79.227.32:4836 , piece_id=4KBSAAZLHC7OYLWN7RBDTQZUUTMK4HFGFUBES7K2PDKBXZST3OBA, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:23.722460741 (486h30m22.273931202s) [noise=true , address=145.79.227.40:4049 , piece_id=DIBJ3Y4NA3HRAZD7GLO72STJHS5XHA3OTCAMWL4VCHMEILLCL3UA, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:25.698494096 (486h30m24.249964557s) [noise=true , address=145.79.227.39:4013 , piece_id=YFEKNDHNI77M7MG76EISOGVU32UOZ536CK4X3Z6SFGOLJV2E7EHQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:25.822411189 (486h30m24.373881651s) [noise=true , address=145.79.227.2:4114 , piece_id=ZB7V2MM6KI6VTSGSC4ZD4QI6EA2265OP3MVLWD2AYAHFERAUE7CQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:29.722449985 (486h30m28.273920436s) [noise=true , address=145.79.227.12:4767 , piece_id=5Q27HBFWOOI6PIIIMXELWG3QYSEA5B5LAOMU5K4FICMUMPDZIQAA, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:47.238199596 (486h30m45.789670047s) [noise=true , address=145.79.227.20:4438 , piece_id=VZK574G5G2HXO5JMVMMEKGFCOJKOTU5KUIJDDEROFXRYTBTITO3A, seg_pos={0 0}, upload=428397]: Done uploading piece. err: ecclient: upload failed (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, address:145.79.227.20:4438): protocol: write tcp 188.166.225.170:34106->145.79.227.20:4438: use of closed network connection; write tcp 188.166.225.170:34106->145.79.227.20:4438: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34106->145.79.227.20:4438: use of closed network connection Nov 29 14:34:47.238464621 (486h30m45.789935082s) [noise=true , address=145.79.227.21:4473 , piece_id=S2V3HONDHED6NZSIFZOTWU2DQ6FKESGGZ4EVMZX36LI4NT32WDWQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:46118->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:46118->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46118->145.79.227.21:4473: use of closed network connection Nov 29 14:34:47.238703104 (486h30m45.790173565s) [noise=true , address=145.79.227.24:4583 , piece_id=B6GTB3UF3XX64M5U57FFCFAMWNWDJUDPIZDG23KOHXOVCFTGKCLA, seg_pos={0 0}, upload=428397]: Done uploading piece. err: ecclient: upload failed (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, address:145.79.227.24:4583): protocol: write tcp 188.166.225.170:51236->145.79.227.24:4583: use of closed network connection; write tcp 188.166.225.170:51236->145.79.227.24:4583: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51236->145.79.227.24:4583: use of closed network connection Nov 29 14:34:47.238869100 (486h30m45.790339561s) [noise=true , address=145.79.227.7:4295 , piece_id=OLB3PUIQ4RYD2GGVF4I32FEGXO2O4X56PLJMOKNTJVBK53KZ3VYQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:145.79.227.7:4295): protocol: write tcp 188.166.225.170:46306->145.79.227.7:4295: use of closed network connection; write tcp 188.166.225.170:46306->145.79.227.7:4295: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46306->145.79.227.7:4295: use of closed network connection Nov 29 14:34:47.239032171 (486h30m45.790502632s) [noise=true , address=145.79.227.6:4259 , piece_id=KIYGYFEAYWG2NUAQZNWG4ZLMRY6GFJADI5LD635JE5M4DLHTX4TQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:145.79.227.6:4259): protocol: write tcp 188.166.225.170:48184->145.79.227.6:4259: use of closed network connection; write tcp 188.166.225.170:48184->145.79.227.6:4259: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48184->145.79.227.6:4259: use of closed network connection Nov 29 14:34:47.239076555 (486h30m45.790547027s) [seg_pos={0 0}, upload=428397]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 29 14:34:47.481897829 (486h30m46.0333683s) [upload=428397]: Done uploading segments... Nov 29 14:34:47.481943356 (486h30m46.033413817s) [upload=428397]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.79431037s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [...X.....................X................................X.] duration: 3.358631101s trace: https://jaeger.dev.storj.io/trace/275e9b01e747ebf4 FAIL (2025-11-29 14:40:35 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.358659103s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 6.938317722s trace: https://jaeger.dev.storj.io/trace/61a1d0645153f16 FAIL (2025-11-29 14:40:29 UTC) orbiter warning: context deadline exceeded Duration: 6.938445394s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [....X.....................X................................X] duration: 5.001474437s trace: https://jaeger.dev.storj.io/trace/275e9b01e747ebf4 FAIL (2025-11-29 14:40:04 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001503153s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)