Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 5.903102089s trace: https://jaeger.dev.storj.io/trace/4add906a25eb3f89 FAIL (2025-11-29 15:09:44 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Nov 29 14:34:17.941236229 (486h30m16.49270669s) [upload=428397]: Got next segment. Inline: false Nov 29 14:34:17.941289010 (486h30m16.492759471s) [upload=428397]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 29 14:34:17.941296063 (486h30m16.492766524s) [upload=428397]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 29 14:34:18.189160800 (486h30m16.740631261s) [seg_pos={0 0}, upload=428397]: Begin upload segment... Nov 29 14:34:18.189218760 (486h30m16.740689221s) [seg_pos={0 0}, upload=428397]: Done begin upload segment. Nov 29 14:34:18.189254097 (486h30m16.740724548s) [noise=true , address=145.79.227.40:4049 , piece_id=DIBJ3Y4NA3HRAZD7GLO72STJHS5XHA3OTCAMWL4VCHMEILLCL3UA, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.189472724 (486h30m16.740943185s) [noise=true , address=77.90.31.13:4627 , piece_id=62A76IX5BE7WFBCCEMPSENCE5PWOSEMCUIKEATUPWGSLTCFJ4BGQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.190037069 (486h30m16.74150753s) [noise=true , address=145.79.227.2:4114 , piece_id=ZB7V2MM6KI6VTSGSC4ZD4QI6EA2265OP3MVLWD2AYAHFERAUE7CQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.190276184 (486h30m16.741746645s) [noise=true , address=145.79.227.21:4473 , piece_id=S2V3HONDHED6NZSIFZOTWU2DQ6FKESGGZ4EVMZX36LI4NT32WDWQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.190487406 (486h30m16.741957867s) [noise=true , address=145.79.227.31:4798 , piece_id=NG2JWZTD5NPFNDIVWYSWFHQHEDTURKJOY6YZ7G6LAQRIDI7WD3FQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.190825751 (486h30m16.742296212s) [noise=true , address=45.91.116.5:4297 , piece_id=GD7HY7SUS6XPN6KJDC5AVRP2ECTJSB2MK4YBQXCNUOTUBVKFRGVQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.190967721 (486h30m16.742438192s) [noise=true , address=145.79.227.24:4583 , piece_id=B6GTB3UF3XX64M5U57FFCFAMWNWDJUDPIZDG23KOHXOVCFTGKCLA, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.191308490 (486h30m16.742778951s) [noise=true , address=145.79.227.33:4869 , piece_id=V3JJCDM7EBRRZL2GPDMSCQAEHP2YX2QXDTDZDT5M6FGVLB27G7PQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.191365077 (486h30m16.742835528s) [noise=true , address=145.79.227.6:4259 , piece_id=KIYGYFEAYWG2NUAQZNWG4ZLMRY6GFJADI5LD635JE5M4DLHTX4TQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.191782502 (486h30m16.743252963s) [noise=true , address=145.79.227.22:4509 , piece_id=N2AEKOF7MLRQZ2HS7FSXV3U6WNI7YQLWP3W4PTFZLMAWQQREBTKQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.192449502 (486h30m16.743919953s) [noise=true , address=145.79.227.14:4839 , piece_id=RDZDYLJRCVHDKTTCAF3JB5ULPKFMSC2E4HNLB2RPLPY2GTUC6ZDQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.193091886 (486h30m16.744562347s) [noise=true , address=145.79.227.12:4767 , piece_id=5Q27HBFWOOI6PIIIMXELWG3QYSEA5B5LAOMU5K4FICMUMPDZIQAA, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.193463874 (486h30m16.744934335s) [noise=true , address=145.79.227.39:4013 , piece_id=YFEKNDHNI77M7MG76EISOGVU32UOZ536CK4X3Z6SFGOLJV2E7EHQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.195162279 (486h30m16.74663274s) [noise=true , address=145.79.227.20:4438 , piece_id=VZK574G5G2HXO5JMVMMEKGFCOJKOTU5KUIJDDEROFXRYTBTITO3A, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.197119445 (486h30m16.748589896s) [noise=true , address=145.79.227.7:4295 , piece_id=OLB3PUIQ4RYD2GGVF4I32FEGXO2O4X56PLJMOKNTJVBK53KZ3VYQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.197903478 (486h30m16.749373929s) [noise=true , address=145.79.227.32:4836 , piece_id=4KBSAAZLHC7OYLWN7RBDTQZUUTMK4HFGFUBES7K2PDKBXZST3OBA, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:19.999395950 (486h30m18.550866411s) [upload=428397]: Next returned nil segment Nov 29 14:34:19.999415177 (486h30m18.550885648s) [upload=428397]: Waiting for error group managing segments... Nov 29 14:34:20.285725942 (486h30m18.837196404s) [noise=true , address=77.90.31.13:4627 , piece_id=62A76IX5BE7WFBCCEMPSENCE5PWOSEMCUIKEATUPWGSLTCFJ4BGQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:20.363243793 (486h30m18.914714244s) [noise=true , address=45.91.116.5:4297 , piece_id=GD7HY7SUS6XPN6KJDC5AVRP2ECTJSB2MK4YBQXCNUOTUBVKFRGVQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:22.551752465 (486h30m21.103222926s) [noise=true , address=145.79.227.31:4798 , piece_id=NG2JWZTD5NPFNDIVWYSWFHQHEDTURKJOY6YZ7G6LAQRIDI7WD3FQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:22.666545505 (486h30m21.218015967s) [noise=true , address=145.79.227.22:4509 , piece_id=N2AEKOF7MLRQZ2HS7FSXV3U6WNI7YQLWP3W4PTFZLMAWQQREBTKQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:23.006352209 (486h30m21.557822671s) [noise=true , address=145.79.227.14:4839 , piece_id=RDZDYLJRCVHDKTTCAF3JB5ULPKFMSC2E4HNLB2RPLPY2GTUC6ZDQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:23.023700063 (486h30m21.575170525s) [noise=true , address=145.79.227.33:4869 , piece_id=V3JJCDM7EBRRZL2GPDMSCQAEHP2YX2QXDTDZDT5M6FGVLB27G7PQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:23.426873130 (486h30m21.978343581s) [noise=true , address=145.79.227.32:4836 , piece_id=4KBSAAZLHC7OYLWN7RBDTQZUUTMK4HFGFUBES7K2PDKBXZST3OBA, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:23.722460741 (486h30m22.273931202s) [noise=true , address=145.79.227.40:4049 , piece_id=DIBJ3Y4NA3HRAZD7GLO72STJHS5XHA3OTCAMWL4VCHMEILLCL3UA, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:25.698494096 (486h30m24.249964557s) [noise=true , address=145.79.227.39:4013 , piece_id=YFEKNDHNI77M7MG76EISOGVU32UOZ536CK4X3Z6SFGOLJV2E7EHQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:25.822411189 (486h30m24.373881651s) [noise=true , address=145.79.227.2:4114 , piece_id=ZB7V2MM6KI6VTSGSC4ZD4QI6EA2265OP3MVLWD2AYAHFERAUE7CQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:29.722449985 (486h30m28.273920436s) [noise=true , address=145.79.227.12:4767 , piece_id=5Q27HBFWOOI6PIIIMXELWG3QYSEA5B5LAOMU5K4FICMUMPDZIQAA, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:47.238199596 (486h30m45.789670047s) [noise=true , address=145.79.227.20:4438 , piece_id=VZK574G5G2HXO5JMVMMEKGFCOJKOTU5KUIJDDEROFXRYTBTITO3A, seg_pos={0 0}, upload=428397]: Done uploading piece. err: ecclient: upload failed (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, address:145.79.227.20:4438): protocol: write tcp 188.166.225.170:34106->145.79.227.20:4438: use of closed network connection; write tcp 188.166.225.170:34106->145.79.227.20:4438: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34106->145.79.227.20:4438: use of closed network connection Nov 29 14:34:47.238464621 (486h30m45.789935082s) [noise=true , address=145.79.227.21:4473 , piece_id=S2V3HONDHED6NZSIFZOTWU2DQ6FKESGGZ4EVMZX36LI4NT32WDWQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:46118->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:46118->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46118->145.79.227.21:4473: use of closed network connection Nov 29 14:34:47.238703104 (486h30m45.790173565s) [noise=true , address=145.79.227.24:4583 , piece_id=B6GTB3UF3XX64M5U57FFCFAMWNWDJUDPIZDG23KOHXOVCFTGKCLA, seg_pos={0 0}, upload=428397]: Done uploading piece. err: ecclient: upload failed (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, address:145.79.227.24:4583): protocol: write tcp 188.166.225.170:51236->145.79.227.24:4583: use of closed network connection; write tcp 188.166.225.170:51236->145.79.227.24:4583: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51236->145.79.227.24:4583: use of closed network connection Nov 29 14:34:47.238869100 (486h30m45.790339561s) [noise=true , address=145.79.227.7:4295 , piece_id=OLB3PUIQ4RYD2GGVF4I32FEGXO2O4X56PLJMOKNTJVBK53KZ3VYQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:145.79.227.7:4295): protocol: write tcp 188.166.225.170:46306->145.79.227.7:4295: use of closed network connection; write tcp 188.166.225.170:46306->145.79.227.7:4295: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46306->145.79.227.7:4295: use of closed network connection Nov 29 14:34:47.239032171 (486h30m45.790502632s) [noise=true , address=145.79.227.6:4259 , piece_id=KIYGYFEAYWG2NUAQZNWG4ZLMRY6GFJADI5LD635JE5M4DLHTX4TQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:145.79.227.6:4259): protocol: write tcp 188.166.225.170:48184->145.79.227.6:4259: use of closed network connection; write tcp 188.166.225.170:48184->145.79.227.6:4259: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48184->145.79.227.6:4259: use of closed network connection Nov 29 14:34:47.239076555 (486h30m45.790547027s) [seg_pos={0 0}, upload=428397]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 29 14:34:47.481897829 (486h30m46.0333683s) [upload=428397]: Done uploading segments... Nov 29 14:34:47.481943356 (486h30m46.033413817s) [upload=428397]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.903133891s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [..................X..............................X........X.] duration: 4.319782334s trace: https://jaeger.dev.storj.io/trace/66b8eb4790a4cf18 FAIL (2025-11-29 15:09:39 UTC) orbiter warning: context deadline exceeded Duration: 4.319811129s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 5.850389015s trace: https://jaeger.dev.storj.io/trace/4add906a25eb3f89 FAIL (2025-11-29 15:09:29 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Nov 29 14:34:17.941236229 (486h30m16.49270669s) [upload=428397]: Got next segment. Inline: false Nov 29 14:34:17.941289010 (486h30m16.492759471s) [upload=428397]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 29 14:34:17.941296063 (486h30m16.492766524s) [upload=428397]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 29 14:34:18.189160800 (486h30m16.740631261s) [seg_pos={0 0}, upload=428397]: Begin upload segment... Nov 29 14:34:18.189218760 (486h30m16.740689221s) [seg_pos={0 0}, upload=428397]: Done begin upload segment. Nov 29 14:34:18.189254097 (486h30m16.740724548s) [noise=true , address=145.79.227.40:4049 , piece_id=DIBJ3Y4NA3HRAZD7GLO72STJHS5XHA3OTCAMWL4VCHMEILLCL3UA, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.189472724 (486h30m16.740943185s) [noise=true , address=77.90.31.13:4627 , piece_id=62A76IX5BE7WFBCCEMPSENCE5PWOSEMCUIKEATUPWGSLTCFJ4BGQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.190037069 (486h30m16.74150753s) [noise=true , address=145.79.227.2:4114 , piece_id=ZB7V2MM6KI6VTSGSC4ZD4QI6EA2265OP3MVLWD2AYAHFERAUE7CQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.190276184 (486h30m16.741746645s) [noise=true , address=145.79.227.21:4473 , piece_id=S2V3HONDHED6NZSIFZOTWU2DQ6FKESGGZ4EVMZX36LI4NT32WDWQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.190487406 (486h30m16.741957867s) [noise=true , address=145.79.227.31:4798 , piece_id=NG2JWZTD5NPFNDIVWYSWFHQHEDTURKJOY6YZ7G6LAQRIDI7WD3FQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.190825751 (486h30m16.742296212s) [noise=true , address=45.91.116.5:4297 , piece_id=GD7HY7SUS6XPN6KJDC5AVRP2ECTJSB2MK4YBQXCNUOTUBVKFRGVQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.190967721 (486h30m16.742438192s) [noise=true , address=145.79.227.24:4583 , piece_id=B6GTB3UF3XX64M5U57FFCFAMWNWDJUDPIZDG23KOHXOVCFTGKCLA, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.191308490 (486h30m16.742778951s) [noise=true , address=145.79.227.33:4869 , piece_id=V3JJCDM7EBRRZL2GPDMSCQAEHP2YX2QXDTDZDT5M6FGVLB27G7PQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.191365077 (486h30m16.742835528s) [noise=true , address=145.79.227.6:4259 , piece_id=KIYGYFEAYWG2NUAQZNWG4ZLMRY6GFJADI5LD635JE5M4DLHTX4TQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.191782502 (486h30m16.743252963s) [noise=true , address=145.79.227.22:4509 , piece_id=N2AEKOF7MLRQZ2HS7FSXV3U6WNI7YQLWP3W4PTFZLMAWQQREBTKQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.192449502 (486h30m16.743919953s) [noise=true , address=145.79.227.14:4839 , piece_id=RDZDYLJRCVHDKTTCAF3JB5ULPKFMSC2E4HNLB2RPLPY2GTUC6ZDQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.193091886 (486h30m16.744562347s) [noise=true , address=145.79.227.12:4767 , piece_id=5Q27HBFWOOI6PIIIMXELWG3QYSEA5B5LAOMU5K4FICMUMPDZIQAA, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.193463874 (486h30m16.744934335s) [noise=true , address=145.79.227.39:4013 , piece_id=YFEKNDHNI77M7MG76EISOGVU32UOZ536CK4X3Z6SFGOLJV2E7EHQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.195162279 (486h30m16.74663274s) [noise=true , address=145.79.227.20:4438 , piece_id=VZK574G5G2HXO5JMVMMEKGFCOJKOTU5KUIJDDEROFXRYTBTITO3A, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.197119445 (486h30m16.748589896s) [noise=true , address=145.79.227.7:4295 , piece_id=OLB3PUIQ4RYD2GGVF4I32FEGXO2O4X56PLJMOKNTJVBK53KZ3VYQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.197903478 (486h30m16.749373929s) [noise=true , address=145.79.227.32:4836 , piece_id=4KBSAAZLHC7OYLWN7RBDTQZUUTMK4HFGFUBES7K2PDKBXZST3OBA, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:19.999395950 (486h30m18.550866411s) [upload=428397]: Next returned nil segment Nov 29 14:34:19.999415177 (486h30m18.550885648s) [upload=428397]: Waiting for error group managing segments... Nov 29 14:34:20.285725942 (486h30m18.837196404s) [noise=true , address=77.90.31.13:4627 , piece_id=62A76IX5BE7WFBCCEMPSENCE5PWOSEMCUIKEATUPWGSLTCFJ4BGQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:20.363243793 (486h30m18.914714244s) [noise=true , address=45.91.116.5:4297 , piece_id=GD7HY7SUS6XPN6KJDC5AVRP2ECTJSB2MK4YBQXCNUOTUBVKFRGVQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:22.551752465 (486h30m21.103222926s) [noise=true , address=145.79.227.31:4798 , piece_id=NG2JWZTD5NPFNDIVWYSWFHQHEDTURKJOY6YZ7G6LAQRIDI7WD3FQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:22.666545505 (486h30m21.218015967s) [noise=true , address=145.79.227.22:4509 , piece_id=N2AEKOF7MLRQZ2HS7FSXV3U6WNI7YQLWP3W4PTFZLMAWQQREBTKQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:23.006352209 (486h30m21.557822671s) [noise=true , address=145.79.227.14:4839 , piece_id=RDZDYLJRCVHDKTTCAF3JB5ULPKFMSC2E4HNLB2RPLPY2GTUC6ZDQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:23.023700063 (486h30m21.575170525s) [noise=true , address=145.79.227.33:4869 , piece_id=V3JJCDM7EBRRZL2GPDMSCQAEHP2YX2QXDTDZDT5M6FGVLB27G7PQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:23.426873130 (486h30m21.978343581s) [noise=true , address=145.79.227.32:4836 , piece_id=4KBSAAZLHC7OYLWN7RBDTQZUUTMK4HFGFUBES7K2PDKBXZST3OBA, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:23.722460741 (486h30m22.273931202s) [noise=true , address=145.79.227.40:4049 , piece_id=DIBJ3Y4NA3HRAZD7GLO72STJHS5XHA3OTCAMWL4VCHMEILLCL3UA, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:25.698494096 (486h30m24.249964557s) [noise=true , address=145.79.227.39:4013 , piece_id=YFEKNDHNI77M7MG76EISOGVU32UOZ536CK4X3Z6SFGOLJV2E7EHQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:25.822411189 (486h30m24.373881651s) [noise=true , address=145.79.227.2:4114 , piece_id=ZB7V2MM6KI6VTSGSC4ZD4QI6EA2265OP3MVLWD2AYAHFERAUE7CQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:29.722449985 (486h30m28.273920436s) [noise=true , address=145.79.227.12:4767 , piece_id=5Q27HBFWOOI6PIIIMXELWG3QYSEA5B5LAOMU5K4FICMUMPDZIQAA, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:47.238199596 (486h30m45.789670047s) [noise=true , address=145.79.227.20:4438 , piece_id=VZK574G5G2HXO5JMVMMEKGFCOJKOTU5KUIJDDEROFXRYTBTITO3A, seg_pos={0 0}, upload=428397]: Done uploading piece. err: ecclient: upload failed (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, address:145.79.227.20:4438): protocol: write tcp 188.166.225.170:34106->145.79.227.20:4438: use of closed network connection; write tcp 188.166.225.170:34106->145.79.227.20:4438: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34106->145.79.227.20:4438: use of closed network connection Nov 29 14:34:47.238464621 (486h30m45.789935082s) [noise=true , address=145.79.227.21:4473 , piece_id=S2V3HONDHED6NZSIFZOTWU2DQ6FKESGGZ4EVMZX36LI4NT32WDWQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:46118->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:46118->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46118->145.79.227.21:4473: use of closed network connection Nov 29 14:34:47.238703104 (486h30m45.790173565s) [noise=true , address=145.79.227.24:4583 , piece_id=B6GTB3UF3XX64M5U57FFCFAMWNWDJUDPIZDG23KOHXOVCFTGKCLA, seg_pos={0 0}, upload=428397]: Done uploading piece. err: ecclient: upload failed (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, address:145.79.227.24:4583): protocol: write tcp 188.166.225.170:51236->145.79.227.24:4583: use of closed network connection; write tcp 188.166.225.170:51236->145.79.227.24:4583: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51236->145.79.227.24:4583: use of closed network connection Nov 29 14:34:47.238869100 (486h30m45.790339561s) [noise=true , address=145.79.227.7:4295 , piece_id=OLB3PUIQ4RYD2GGVF4I32FEGXO2O4X56PLJMOKNTJVBK53KZ3VYQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:145.79.227.7:4295): protocol: write tcp 188.166.225.170:46306->145.79.227.7:4295: use of closed network connection; write tcp 188.166.225.170:46306->145.79.227.7:4295: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46306->145.79.227.7:4295: use of closed network connection Nov 29 14:34:47.239032171 (486h30m45.790502632s) [noise=true , address=145.79.227.6:4259 , piece_id=KIYGYFEAYWG2NUAQZNWG4ZLMRY6GFJADI5LD635JE5M4DLHTX4TQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:145.79.227.6:4259): protocol: write tcp 188.166.225.170:48184->145.79.227.6:4259: use of closed network connection; write tcp 188.166.225.170:48184->145.79.227.6:4259: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48184->145.79.227.6:4259: use of closed network connection Nov 29 14:34:47.239076555 (486h30m45.790547027s) [seg_pos={0 0}, upload=428397]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 29 14:34:47.481897829 (486h30m46.0333683s) [upload=428397]: Done uploading segments... Nov 29 14:34:47.481943356 (486h30m46.033413817s) [upload=428397]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.850423931s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [...................X..............................X........X] duration: 5.001925541s trace: https://jaeger.dev.storj.io/trace/66b8eb4790a4cf18 FAIL (2025-11-29 15:08:44 UTC) orbiter warning: context deadline exceeded Duration: 5.001949828s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [....................X..............................X........] duration: 3.438841923s trace: https://jaeger.dev.storj.io/trace/1e2e0cb4b27ac060 FAIL (2025-11-29 15:08:21 UTC) orbiter warning: context deadline exceeded Duration: 3.438872872s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 5.855484196s trace: https://jaeger.dev.storj.io/trace/4add906a25eb3f89 FAIL (2025-11-29 15:08:15 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Nov 29 14:34:17.941236229 (486h30m16.49270669s) [upload=428397]: Got next segment. Inline: false Nov 29 14:34:17.941289010 (486h30m16.492759471s) [upload=428397]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 29 14:34:17.941296063 (486h30m16.492766524s) [upload=428397]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 29 14:34:18.189160800 (486h30m16.740631261s) [seg_pos={0 0}, upload=428397]: Begin upload segment... Nov 29 14:34:18.189218760 (486h30m16.740689221s) [seg_pos={0 0}, upload=428397]: Done begin upload segment. Nov 29 14:34:18.189254097 (486h30m16.740724548s) [noise=true , address=145.79.227.40:4049 , piece_id=DIBJ3Y4NA3HRAZD7GLO72STJHS5XHA3OTCAMWL4VCHMEILLCL3UA, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.189472724 (486h30m16.740943185s) [noise=true , address=77.90.31.13:4627 , piece_id=62A76IX5BE7WFBCCEMPSENCE5PWOSEMCUIKEATUPWGSLTCFJ4BGQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.190037069 (486h30m16.74150753s) [noise=true , address=145.79.227.2:4114 , piece_id=ZB7V2MM6KI6VTSGSC4ZD4QI6EA2265OP3MVLWD2AYAHFERAUE7CQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.190276184 (486h30m16.741746645s) [noise=true , address=145.79.227.21:4473 , piece_id=S2V3HONDHED6NZSIFZOTWU2DQ6FKESGGZ4EVMZX36LI4NT32WDWQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.190487406 (486h30m16.741957867s) [noise=true , address=145.79.227.31:4798 , piece_id=NG2JWZTD5NPFNDIVWYSWFHQHEDTURKJOY6YZ7G6LAQRIDI7WD3FQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.190825751 (486h30m16.742296212s) [noise=true , address=45.91.116.5:4297 , piece_id=GD7HY7SUS6XPN6KJDC5AVRP2ECTJSB2MK4YBQXCNUOTUBVKFRGVQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.190967721 (486h30m16.742438192s) [noise=true , address=145.79.227.24:4583 , piece_id=B6GTB3UF3XX64M5U57FFCFAMWNWDJUDPIZDG23KOHXOVCFTGKCLA, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.191308490 (486h30m16.742778951s) [noise=true , address=145.79.227.33:4869 , piece_id=V3JJCDM7EBRRZL2GPDMSCQAEHP2YX2QXDTDZDT5M6FGVLB27G7PQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.191365077 (486h30m16.742835528s) [noise=true , address=145.79.227.6:4259 , piece_id=KIYGYFEAYWG2NUAQZNWG4ZLMRY6GFJADI5LD635JE5M4DLHTX4TQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.191782502 (486h30m16.743252963s) [noise=true , address=145.79.227.22:4509 , piece_id=N2AEKOF7MLRQZ2HS7FSXV3U6WNI7YQLWP3W4PTFZLMAWQQREBTKQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.192449502 (486h30m16.743919953s) [noise=true , address=145.79.227.14:4839 , piece_id=RDZDYLJRCVHDKTTCAF3JB5ULPKFMSC2E4HNLB2RPLPY2GTUC6ZDQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.193091886 (486h30m16.744562347s) [noise=true , address=145.79.227.12:4767 , piece_id=5Q27HBFWOOI6PIIIMXELWG3QYSEA5B5LAOMU5K4FICMUMPDZIQAA, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.193463874 (486h30m16.744934335s) [noise=true , address=145.79.227.39:4013 , piece_id=YFEKNDHNI77M7MG76EISOGVU32UOZ536CK4X3Z6SFGOLJV2E7EHQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.195162279 (486h30m16.74663274s) [noise=true , address=145.79.227.20:4438 , piece_id=VZK574G5G2HXO5JMVMMEKGFCOJKOTU5KUIJDDEROFXRYTBTITO3A, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.197119445 (486h30m16.748589896s) [noise=true , address=145.79.227.7:4295 , piece_id=OLB3PUIQ4RYD2GGVF4I32FEGXO2O4X56PLJMOKNTJVBK53KZ3VYQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.197903478 (486h30m16.749373929s) [noise=true , address=145.79.227.32:4836 , piece_id=4KBSAAZLHC7OYLWN7RBDTQZUUTMK4HFGFUBES7K2PDKBXZST3OBA, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:19.999395950 (486h30m18.550866411s) [upload=428397]: Next returned nil segment Nov 29 14:34:19.999415177 (486h30m18.550885648s) [upload=428397]: Waiting for error group managing segments... Nov 29 14:34:20.285725942 (486h30m18.837196404s) [noise=true , address=77.90.31.13:4627 , piece_id=62A76IX5BE7WFBCCEMPSENCE5PWOSEMCUIKEATUPWGSLTCFJ4BGQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:20.363243793 (486h30m18.914714244s) [noise=true , address=45.91.116.5:4297 , piece_id=GD7HY7SUS6XPN6KJDC5AVRP2ECTJSB2MK4YBQXCNUOTUBVKFRGVQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:22.551752465 (486h30m21.103222926s) [noise=true , address=145.79.227.31:4798 , piece_id=NG2JWZTD5NPFNDIVWYSWFHQHEDTURKJOY6YZ7G6LAQRIDI7WD3FQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:22.666545505 (486h30m21.218015967s) [noise=true , address=145.79.227.22:4509 , piece_id=N2AEKOF7MLRQZ2HS7FSXV3U6WNI7YQLWP3W4PTFZLMAWQQREBTKQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:23.006352209 (486h30m21.557822671s) [noise=true , address=145.79.227.14:4839 , piece_id=RDZDYLJRCVHDKTTCAF3JB5ULPKFMSC2E4HNLB2RPLPY2GTUC6ZDQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:23.023700063 (486h30m21.575170525s) [noise=true , address=145.79.227.33:4869 , piece_id=V3JJCDM7EBRRZL2GPDMSCQAEHP2YX2QXDTDZDT5M6FGVLB27G7PQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:23.426873130 (486h30m21.978343581s) [noise=true , address=145.79.227.32:4836 , piece_id=4KBSAAZLHC7OYLWN7RBDTQZUUTMK4HFGFUBES7K2PDKBXZST3OBA, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:23.722460741 (486h30m22.273931202s) [noise=true , address=145.79.227.40:4049 , piece_id=DIBJ3Y4NA3HRAZD7GLO72STJHS5XHA3OTCAMWL4VCHMEILLCL3UA, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:25.698494096 (486h30m24.249964557s) [noise=true , address=145.79.227.39:4013 , piece_id=YFEKNDHNI77M7MG76EISOGVU32UOZ536CK4X3Z6SFGOLJV2E7EHQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:25.822411189 (486h30m24.373881651s) [noise=true , address=145.79.227.2:4114 , piece_id=ZB7V2MM6KI6VTSGSC4ZD4QI6EA2265OP3MVLWD2AYAHFERAUE7CQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:29.722449985 (486h30m28.273920436s) [noise=true , address=145.79.227.12:4767 , piece_id=5Q27HBFWOOI6PIIIMXELWG3QYSEA5B5LAOMU5K4FICMUMPDZIQAA, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:47.238199596 (486h30m45.789670047s) [noise=true , address=145.79.227.20:4438 , piece_id=VZK574G5G2HXO5JMVMMEKGFCOJKOTU5KUIJDDEROFXRYTBTITO3A, seg_pos={0 0}, upload=428397]: Done uploading piece. err: ecclient: upload failed (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, address:145.79.227.20:4438): protocol: write tcp 188.166.225.170:34106->145.79.227.20:4438: use of closed network connection; write tcp 188.166.225.170:34106->145.79.227.20:4438: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34106->145.79.227.20:4438: use of closed network connection Nov 29 14:34:47.238464621 (486h30m45.789935082s) [noise=true , address=145.79.227.21:4473 , piece_id=S2V3HONDHED6NZSIFZOTWU2DQ6FKESGGZ4EVMZX36LI4NT32WDWQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:46118->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:46118->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46118->145.79.227.21:4473: use of closed network connection Nov 29 14:34:47.238703104 (486h30m45.790173565s) [noise=true , address=145.79.227.24:4583 , piece_id=B6GTB3UF3XX64M5U57FFCFAMWNWDJUDPIZDG23KOHXOVCFTGKCLA, seg_pos={0 0}, upload=428397]: Done uploading piece. err: ecclient: upload failed (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, address:145.79.227.24:4583): protocol: write tcp 188.166.225.170:51236->145.79.227.24:4583: use of closed network connection; write tcp 188.166.225.170:51236->145.79.227.24:4583: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51236->145.79.227.24:4583: use of closed network connection Nov 29 14:34:47.238869100 (486h30m45.790339561s) [noise=true , address=145.79.227.7:4295 , piece_id=OLB3PUIQ4RYD2GGVF4I32FEGXO2O4X56PLJMOKNTJVBK53KZ3VYQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:145.79.227.7:4295): protocol: write tcp 188.166.225.170:46306->145.79.227.7:4295: use of closed network connection; write tcp 188.166.225.170:46306->145.79.227.7:4295: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46306->145.79.227.7:4295: use of closed network connection Nov 29 14:34:47.239032171 (486h30m45.790502632s) [noise=true , address=145.79.227.6:4259 , piece_id=KIYGYFEAYWG2NUAQZNWG4ZLMRY6GFJADI5LD635JE5M4DLHTX4TQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:145.79.227.6:4259): protocol: write tcp 188.166.225.170:48184->145.79.227.6:4259: use of closed network connection; write tcp 188.166.225.170:48184->145.79.227.6:4259: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48184->145.79.227.6:4259: use of closed network connection Nov 29 14:34:47.239076555 (486h30m45.790547027s) [seg_pos={0 0}, upload=428397]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 29 14:34:47.481897829 (486h30m46.0333683s) [upload=428397]: Done uploading segments... Nov 29 14:34:47.481943356 (486h30m46.033413817s) [upload=428397]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.855514524s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 6.07423763s trace: https://jaeger.dev.storj.io/trace/4add906a25eb3f89 FAIL (2025-11-29 15:07:35 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Nov 29 14:34:17.941236229 (486h30m16.49270669s) [upload=428397]: Got next segment. Inline: false Nov 29 14:34:17.941289010 (486h30m16.492759471s) [upload=428397]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 29 14:34:17.941296063 (486h30m16.492766524s) [upload=428397]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 29 14:34:18.189160800 (486h30m16.740631261s) [seg_pos={0 0}, upload=428397]: Begin upload segment... Nov 29 14:34:18.189218760 (486h30m16.740689221s) [seg_pos={0 0}, upload=428397]: Done begin upload segment. Nov 29 14:34:18.189254097 (486h30m16.740724548s) [noise=true , address=145.79.227.40:4049 , piece_id=DIBJ3Y4NA3HRAZD7GLO72STJHS5XHA3OTCAMWL4VCHMEILLCL3UA, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.189472724 (486h30m16.740943185s) [noise=true , address=77.90.31.13:4627 , piece_id=62A76IX5BE7WFBCCEMPSENCE5PWOSEMCUIKEATUPWGSLTCFJ4BGQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.190037069 (486h30m16.74150753s) [noise=true , address=145.79.227.2:4114 , piece_id=ZB7V2MM6KI6VTSGSC4ZD4QI6EA2265OP3MVLWD2AYAHFERAUE7CQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.190276184 (486h30m16.741746645s) [noise=true , address=145.79.227.21:4473 , piece_id=S2V3HONDHED6NZSIFZOTWU2DQ6FKESGGZ4EVMZX36LI4NT32WDWQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.190487406 (486h30m16.741957867s) [noise=true , address=145.79.227.31:4798 , piece_id=NG2JWZTD5NPFNDIVWYSWFHQHEDTURKJOY6YZ7G6LAQRIDI7WD3FQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.190825751 (486h30m16.742296212s) [noise=true , address=45.91.116.5:4297 , piece_id=GD7HY7SUS6XPN6KJDC5AVRP2ECTJSB2MK4YBQXCNUOTUBVKFRGVQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.190967721 (486h30m16.742438192s) [noise=true , address=145.79.227.24:4583 , piece_id=B6GTB3UF3XX64M5U57FFCFAMWNWDJUDPIZDG23KOHXOVCFTGKCLA, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.191308490 (486h30m16.742778951s) [noise=true , address=145.79.227.33:4869 , piece_id=V3JJCDM7EBRRZL2GPDMSCQAEHP2YX2QXDTDZDT5M6FGVLB27G7PQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.191365077 (486h30m16.742835528s) [noise=true , address=145.79.227.6:4259 , piece_id=KIYGYFEAYWG2NUAQZNWG4ZLMRY6GFJADI5LD635JE5M4DLHTX4TQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.191782502 (486h30m16.743252963s) [noise=true , address=145.79.227.22:4509 , piece_id=N2AEKOF7MLRQZ2HS7FSXV3U6WNI7YQLWP3W4PTFZLMAWQQREBTKQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.192449502 (486h30m16.743919953s) [noise=true , address=145.79.227.14:4839 , piece_id=RDZDYLJRCVHDKTTCAF3JB5ULPKFMSC2E4HNLB2RPLPY2GTUC6ZDQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.193091886 (486h30m16.744562347s) [noise=true , address=145.79.227.12:4767 , piece_id=5Q27HBFWOOI6PIIIMXELWG3QYSEA5B5LAOMU5K4FICMUMPDZIQAA, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.193463874 (486h30m16.744934335s) [noise=true , address=145.79.227.39:4013 , piece_id=YFEKNDHNI77M7MG76EISOGVU32UOZ536CK4X3Z6SFGOLJV2E7EHQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.195162279 (486h30m16.74663274s) [noise=true , address=145.79.227.20:4438 , piece_id=VZK574G5G2HXO5JMVMMEKGFCOJKOTU5KUIJDDEROFXRYTBTITO3A, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.197119445 (486h30m16.748589896s) [noise=true , address=145.79.227.7:4295 , piece_id=OLB3PUIQ4RYD2GGVF4I32FEGXO2O4X56PLJMOKNTJVBK53KZ3VYQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.197903478 (486h30m16.749373929s) [noise=true , address=145.79.227.32:4836 , piece_id=4KBSAAZLHC7OYLWN7RBDTQZUUTMK4HFGFUBES7K2PDKBXZST3OBA, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:19.999395950 (486h30m18.550866411s) [upload=428397]: Next returned nil segment Nov 29 14:34:19.999415177 (486h30m18.550885648s) [upload=428397]: Waiting for error group managing segments... Nov 29 14:34:20.285725942 (486h30m18.837196404s) [noise=true , address=77.90.31.13:4627 , piece_id=62A76IX5BE7WFBCCEMPSENCE5PWOSEMCUIKEATUPWGSLTCFJ4BGQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:20.363243793 (486h30m18.914714244s) [noise=true , address=45.91.116.5:4297 , piece_id=GD7HY7SUS6XPN6KJDC5AVRP2ECTJSB2MK4YBQXCNUOTUBVKFRGVQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:22.551752465 (486h30m21.103222926s) [noise=true , address=145.79.227.31:4798 , piece_id=NG2JWZTD5NPFNDIVWYSWFHQHEDTURKJOY6YZ7G6LAQRIDI7WD3FQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:22.666545505 (486h30m21.218015967s) [noise=true , address=145.79.227.22:4509 , piece_id=N2AEKOF7MLRQZ2HS7FSXV3U6WNI7YQLWP3W4PTFZLMAWQQREBTKQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:23.006352209 (486h30m21.557822671s) [noise=true , address=145.79.227.14:4839 , piece_id=RDZDYLJRCVHDKTTCAF3JB5ULPKFMSC2E4HNLB2RPLPY2GTUC6ZDQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:23.023700063 (486h30m21.575170525s) [noise=true , address=145.79.227.33:4869 , piece_id=V3JJCDM7EBRRZL2GPDMSCQAEHP2YX2QXDTDZDT5M6FGVLB27G7PQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:23.426873130 (486h30m21.978343581s) [noise=true , address=145.79.227.32:4836 , piece_id=4KBSAAZLHC7OYLWN7RBDTQZUUTMK4HFGFUBES7K2PDKBXZST3OBA, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:23.722460741 (486h30m22.273931202s) [noise=true , address=145.79.227.40:4049 , piece_id=DIBJ3Y4NA3HRAZD7GLO72STJHS5XHA3OTCAMWL4VCHMEILLCL3UA, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:25.698494096 (486h30m24.249964557s) [noise=true , address=145.79.227.39:4013 , piece_id=YFEKNDHNI77M7MG76EISOGVU32UOZ536CK4X3Z6SFGOLJV2E7EHQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:25.822411189 (486h30m24.373881651s) [noise=true , address=145.79.227.2:4114 , piece_id=ZB7V2MM6KI6VTSGSC4ZD4QI6EA2265OP3MVLWD2AYAHFERAUE7CQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:29.722449985 (486h30m28.273920436s) [noise=true , address=145.79.227.12:4767 , piece_id=5Q27HBFWOOI6PIIIMXELWG3QYSEA5B5LAOMU5K4FICMUMPDZIQAA, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:47.238199596 (486h30m45.789670047s) [noise=true , address=145.79.227.20:4438 , piece_id=VZK574G5G2HXO5JMVMMEKGFCOJKOTU5KUIJDDEROFXRYTBTITO3A, seg_pos={0 0}, upload=428397]: Done uploading piece. err: ecclient: upload failed (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, address:145.79.227.20:4438): protocol: write tcp 188.166.225.170:34106->145.79.227.20:4438: use of closed network connection; write tcp 188.166.225.170:34106->145.79.227.20:4438: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34106->145.79.227.20:4438: use of closed network connection Nov 29 14:34:47.238464621 (486h30m45.789935082s) [noise=true , address=145.79.227.21:4473 , piece_id=S2V3HONDHED6NZSIFZOTWU2DQ6FKESGGZ4EVMZX36LI4NT32WDWQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:46118->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:46118->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46118->145.79.227.21:4473: use of closed network connection Nov 29 14:34:47.238703104 (486h30m45.790173565s) [noise=true , address=145.79.227.24:4583 , piece_id=B6GTB3UF3XX64M5U57FFCFAMWNWDJUDPIZDG23KOHXOVCFTGKCLA, seg_pos={0 0}, upload=428397]: Done uploading piece. err: ecclient: upload failed (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, address:145.79.227.24:4583): protocol: write tcp 188.166.225.170:51236->145.79.227.24:4583: use of closed network connection; write tcp 188.166.225.170:51236->145.79.227.24:4583: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51236->145.79.227.24:4583: use of closed network connection Nov 29 14:34:47.238869100 (486h30m45.790339561s) [noise=true , address=145.79.227.7:4295 , piece_id=OLB3PUIQ4RYD2GGVF4I32FEGXO2O4X56PLJMOKNTJVBK53KZ3VYQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:145.79.227.7:4295): protocol: write tcp 188.166.225.170:46306->145.79.227.7:4295: use of closed network connection; write tcp 188.166.225.170:46306->145.79.227.7:4295: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46306->145.79.227.7:4295: use of closed network connection Nov 29 14:34:47.239032171 (486h30m45.790502632s) [noise=true , address=145.79.227.6:4259 , piece_id=KIYGYFEAYWG2NUAQZNWG4ZLMRY6GFJADI5LD635JE5M4DLHTX4TQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:145.79.227.6:4259): protocol: write tcp 188.166.225.170:48184->145.79.227.6:4259: use of closed network connection; write tcp 188.166.225.170:48184->145.79.227.6:4259: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48184->145.79.227.6:4259: use of closed network connection Nov 29 14:34:47.239076555 (486h30m45.790547027s) [seg_pos={0 0}, upload=428397]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 29 14:34:47.481897829 (486h30m46.0333683s) [upload=428397]: Done uploading segments... Nov 29 14:34:47.481943356 (486h30m46.033413817s) [upload=428397]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 6.074266916s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.....................X..............................X.......] duration: 3.525031739s trace: https://jaeger.dev.storj.io/trace/1e2e0cb4b27ac060 FAIL (2025-11-29 15:07:34 UTC) orbiter warning: context deadline exceeded Duration: 3.52506318s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 6.170082715s trace: https://jaeger.dev.storj.io/trace/4add906a25eb3f89 FAIL (2025-11-29 15:06:56 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Nov 29 14:34:17.941236229 (486h30m16.49270669s) [upload=428397]: Got next segment. Inline: false Nov 29 14:34:17.941289010 (486h30m16.492759471s) [upload=428397]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 29 14:34:17.941296063 (486h30m16.492766524s) [upload=428397]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 29 14:34:18.189160800 (486h30m16.740631261s) [seg_pos={0 0}, upload=428397]: Begin upload segment... Nov 29 14:34:18.189218760 (486h30m16.740689221s) [seg_pos={0 0}, upload=428397]: Done begin upload segment. Nov 29 14:34:18.189254097 (486h30m16.740724548s) [noise=true , address=145.79.227.40:4049 , piece_id=DIBJ3Y4NA3HRAZD7GLO72STJHS5XHA3OTCAMWL4VCHMEILLCL3UA, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.189472724 (486h30m16.740943185s) [noise=true , address=77.90.31.13:4627 , piece_id=62A76IX5BE7WFBCCEMPSENCE5PWOSEMCUIKEATUPWGSLTCFJ4BGQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.190037069 (486h30m16.74150753s) [noise=true , address=145.79.227.2:4114 , piece_id=ZB7V2MM6KI6VTSGSC4ZD4QI6EA2265OP3MVLWD2AYAHFERAUE7CQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.190276184 (486h30m16.741746645s) [noise=true , address=145.79.227.21:4473 , piece_id=S2V3HONDHED6NZSIFZOTWU2DQ6FKESGGZ4EVMZX36LI4NT32WDWQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.190487406 (486h30m16.741957867s) [noise=true , address=145.79.227.31:4798 , piece_id=NG2JWZTD5NPFNDIVWYSWFHQHEDTURKJOY6YZ7G6LAQRIDI7WD3FQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.190825751 (486h30m16.742296212s) [noise=true , address=45.91.116.5:4297 , piece_id=GD7HY7SUS6XPN6KJDC5AVRP2ECTJSB2MK4YBQXCNUOTUBVKFRGVQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.190967721 (486h30m16.742438192s) [noise=true , address=145.79.227.24:4583 , piece_id=B6GTB3UF3XX64M5U57FFCFAMWNWDJUDPIZDG23KOHXOVCFTGKCLA, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.191308490 (486h30m16.742778951s) [noise=true , address=145.79.227.33:4869 , piece_id=V3JJCDM7EBRRZL2GPDMSCQAEHP2YX2QXDTDZDT5M6FGVLB27G7PQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.191365077 (486h30m16.742835528s) [noise=true , address=145.79.227.6:4259 , piece_id=KIYGYFEAYWG2NUAQZNWG4ZLMRY6GFJADI5LD635JE5M4DLHTX4TQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.191782502 (486h30m16.743252963s) [noise=true , address=145.79.227.22:4509 , piece_id=N2AEKOF7MLRQZ2HS7FSXV3U6WNI7YQLWP3W4PTFZLMAWQQREBTKQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.192449502 (486h30m16.743919953s) [noise=true , address=145.79.227.14:4839 , piece_id=RDZDYLJRCVHDKTTCAF3JB5ULPKFMSC2E4HNLB2RPLPY2GTUC6ZDQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.193091886 (486h30m16.744562347s) [noise=true , address=145.79.227.12:4767 , piece_id=5Q27HBFWOOI6PIIIMXELWG3QYSEA5B5LAOMU5K4FICMUMPDZIQAA, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.193463874 (486h30m16.744934335s) [noise=true , address=145.79.227.39:4013 , piece_id=YFEKNDHNI77M7MG76EISOGVU32UOZ536CK4X3Z6SFGOLJV2E7EHQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.195162279 (486h30m16.74663274s) [noise=true , address=145.79.227.20:4438 , piece_id=VZK574G5G2HXO5JMVMMEKGFCOJKOTU5KUIJDDEROFXRYTBTITO3A, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.197119445 (486h30m16.748589896s) [noise=true , address=145.79.227.7:4295 , piece_id=OLB3PUIQ4RYD2GGVF4I32FEGXO2O4X56PLJMOKNTJVBK53KZ3VYQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.197903478 (486h30m16.749373929s) [noise=true , address=145.79.227.32:4836 , piece_id=4KBSAAZLHC7OYLWN7RBDTQZUUTMK4HFGFUBES7K2PDKBXZST3OBA, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:19.999395950 (486h30m18.550866411s) [upload=428397]: Next returned nil segment Nov 29 14:34:19.999415177 (486h30m18.550885648s) [upload=428397]: Waiting for error group managing segments... Nov 29 14:34:20.285725942 (486h30m18.837196404s) [noise=true , address=77.90.31.13:4627 , piece_id=62A76IX5BE7WFBCCEMPSENCE5PWOSEMCUIKEATUPWGSLTCFJ4BGQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:20.363243793 (486h30m18.914714244s) [noise=true , address=45.91.116.5:4297 , piece_id=GD7HY7SUS6XPN6KJDC5AVRP2ECTJSB2MK4YBQXCNUOTUBVKFRGVQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:22.551752465 (486h30m21.103222926s) [noise=true , address=145.79.227.31:4798 , piece_id=NG2JWZTD5NPFNDIVWYSWFHQHEDTURKJOY6YZ7G6LAQRIDI7WD3FQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:22.666545505 (486h30m21.218015967s) [noise=true , address=145.79.227.22:4509 , piece_id=N2AEKOF7MLRQZ2HS7FSXV3U6WNI7YQLWP3W4PTFZLMAWQQREBTKQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:23.006352209 (486h30m21.557822671s) [noise=true , address=145.79.227.14:4839 , piece_id=RDZDYLJRCVHDKTTCAF3JB5ULPKFMSC2E4HNLB2RPLPY2GTUC6ZDQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:23.023700063 (486h30m21.575170525s) [noise=true , address=145.79.227.33:4869 , piece_id=V3JJCDM7EBRRZL2GPDMSCQAEHP2YX2QXDTDZDT5M6FGVLB27G7PQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:23.426873130 (486h30m21.978343581s) [noise=true , address=145.79.227.32:4836 , piece_id=4KBSAAZLHC7OYLWN7RBDTQZUUTMK4HFGFUBES7K2PDKBXZST3OBA, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:23.722460741 (486h30m22.273931202s) [noise=true , address=145.79.227.40:4049 , piece_id=DIBJ3Y4NA3HRAZD7GLO72STJHS5XHA3OTCAMWL4VCHMEILLCL3UA, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:25.698494096 (486h30m24.249964557s) [noise=true , address=145.79.227.39:4013 , piece_id=YFEKNDHNI77M7MG76EISOGVU32UOZ536CK4X3Z6SFGOLJV2E7EHQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:25.822411189 (486h30m24.373881651s) [noise=true , address=145.79.227.2:4114 , piece_id=ZB7V2MM6KI6VTSGSC4ZD4QI6EA2265OP3MVLWD2AYAHFERAUE7CQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:29.722449985 (486h30m28.273920436s) [noise=true , address=145.79.227.12:4767 , piece_id=5Q27HBFWOOI6PIIIMXELWG3QYSEA5B5LAOMU5K4FICMUMPDZIQAA, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:47.238199596 (486h30m45.789670047s) [noise=true , address=145.79.227.20:4438 , piece_id=VZK574G5G2HXO5JMVMMEKGFCOJKOTU5KUIJDDEROFXRYTBTITO3A, seg_pos={0 0}, upload=428397]: Done uploading piece. err: ecclient: upload failed (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, address:145.79.227.20:4438): protocol: write tcp 188.166.225.170:34106->145.79.227.20:4438: use of closed network connection; write tcp 188.166.225.170:34106->145.79.227.20:4438: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34106->145.79.227.20:4438: use of closed network connection Nov 29 14:34:47.238464621 (486h30m45.789935082s) [noise=true , address=145.79.227.21:4473 , piece_id=S2V3HONDHED6NZSIFZOTWU2DQ6FKESGGZ4EVMZX36LI4NT32WDWQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:46118->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:46118->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46118->145.79.227.21:4473: use of closed network connection Nov 29 14:34:47.238703104 (486h30m45.790173565s) [noise=true , address=145.79.227.24:4583 , piece_id=B6GTB3UF3XX64M5U57FFCFAMWNWDJUDPIZDG23KOHXOVCFTGKCLA, seg_pos={0 0}, upload=428397]: Done uploading piece. err: ecclient: upload failed (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, address:145.79.227.24:4583): protocol: write tcp 188.166.225.170:51236->145.79.227.24:4583: use of closed network connection; write tcp 188.166.225.170:51236->145.79.227.24:4583: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51236->145.79.227.24:4583: use of closed network connection Nov 29 14:34:47.238869100 (486h30m45.790339561s) [noise=true , address=145.79.227.7:4295 , piece_id=OLB3PUIQ4RYD2GGVF4I32FEGXO2O4X56PLJMOKNTJVBK53KZ3VYQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:145.79.227.7:4295): protocol: write tcp 188.166.225.170:46306->145.79.227.7:4295: use of closed network connection; write tcp 188.166.225.170:46306->145.79.227.7:4295: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46306->145.79.227.7:4295: use of closed network connection Nov 29 14:34:47.239032171 (486h30m45.790502632s) [noise=true , address=145.79.227.6:4259 , piece_id=KIYGYFEAYWG2NUAQZNWG4ZLMRY6GFJADI5LD635JE5M4DLHTX4TQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:145.79.227.6:4259): protocol: write tcp 188.166.225.170:48184->145.79.227.6:4259: use of closed network connection; write tcp 188.166.225.170:48184->145.79.227.6:4259: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48184->145.79.227.6:4259: use of closed network connection Nov 29 14:34:47.239076555 (486h30m45.790547027s) [seg_pos={0 0}, upload=428397]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 29 14:34:47.481897829 (486h30m46.0333683s) [upload=428397]: Done uploading segments... Nov 29 14:34:47.481943356 (486h30m46.033413817s) [upload=428397]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 6.170106841s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [......................X..............................X......] duration: 3.55844826s trace: https://jaeger.dev.storj.io/trace/1e2e0cb4b27ac060 FAIL (2025-11-29 15:06:48 UTC) orbiter warning: context deadline exceeded Duration: 3.558477286s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.......................X..............................X.....] duration: 3.551154922s trace: https://jaeger.dev.storj.io/trace/1e2e0cb4b27ac060 FAIL (2025-11-29 15:06:20 UTC) orbiter warning: context deadline exceeded Duration: 3.55118494s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 5.83746428s trace: https://jaeger.dev.storj.io/trace/4add906a25eb3f89 FAIL (2025-11-29 15:05:59 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Nov 29 14:34:17.941236229 (486h30m16.49270669s) [upload=428397]: Got next segment. Inline: false Nov 29 14:34:17.941289010 (486h30m16.492759471s) [upload=428397]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 29 14:34:17.941296063 (486h30m16.492766524s) [upload=428397]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 29 14:34:18.189160800 (486h30m16.740631261s) [seg_pos={0 0}, upload=428397]: Begin upload segment... Nov 29 14:34:18.189218760 (486h30m16.740689221s) [seg_pos={0 0}, upload=428397]: Done begin upload segment. Nov 29 14:34:18.189254097 (486h30m16.740724548s) [noise=true , address=145.79.227.40:4049 , piece_id=DIBJ3Y4NA3HRAZD7GLO72STJHS5XHA3OTCAMWL4VCHMEILLCL3UA, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.189472724 (486h30m16.740943185s) [noise=true , address=77.90.31.13:4627 , piece_id=62A76IX5BE7WFBCCEMPSENCE5PWOSEMCUIKEATUPWGSLTCFJ4BGQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.190037069 (486h30m16.74150753s) [noise=true , address=145.79.227.2:4114 , piece_id=ZB7V2MM6KI6VTSGSC4ZD4QI6EA2265OP3MVLWD2AYAHFERAUE7CQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.190276184 (486h30m16.741746645s) [noise=true , address=145.79.227.21:4473 , piece_id=S2V3HONDHED6NZSIFZOTWU2DQ6FKESGGZ4EVMZX36LI4NT32WDWQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.190487406 (486h30m16.741957867s) [noise=true , address=145.79.227.31:4798 , piece_id=NG2JWZTD5NPFNDIVWYSWFHQHEDTURKJOY6YZ7G6LAQRIDI7WD3FQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.190825751 (486h30m16.742296212s) [noise=true , address=45.91.116.5:4297 , piece_id=GD7HY7SUS6XPN6KJDC5AVRP2ECTJSB2MK4YBQXCNUOTUBVKFRGVQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.190967721 (486h30m16.742438192s) [noise=true , address=145.79.227.24:4583 , piece_id=B6GTB3UF3XX64M5U57FFCFAMWNWDJUDPIZDG23KOHXOVCFTGKCLA, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.191308490 (486h30m16.742778951s) [noise=true , address=145.79.227.33:4869 , piece_id=V3JJCDM7EBRRZL2GPDMSCQAEHP2YX2QXDTDZDT5M6FGVLB27G7PQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.191365077 (486h30m16.742835528s) [noise=true , address=145.79.227.6:4259 , piece_id=KIYGYFEAYWG2NUAQZNWG4ZLMRY6GFJADI5LD635JE5M4DLHTX4TQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.191782502 (486h30m16.743252963s) [noise=true , address=145.79.227.22:4509 , piece_id=N2AEKOF7MLRQZ2HS7FSXV3U6WNI7YQLWP3W4PTFZLMAWQQREBTKQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.192449502 (486h30m16.743919953s) [noise=true , address=145.79.227.14:4839 , piece_id=RDZDYLJRCVHDKTTCAF3JB5ULPKFMSC2E4HNLB2RPLPY2GTUC6ZDQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.193091886 (486h30m16.744562347s) [noise=true , address=145.79.227.12:4767 , piece_id=5Q27HBFWOOI6PIIIMXELWG3QYSEA5B5LAOMU5K4FICMUMPDZIQAA, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.193463874 (486h30m16.744934335s) [noise=true , address=145.79.227.39:4013 , piece_id=YFEKNDHNI77M7MG76EISOGVU32UOZ536CK4X3Z6SFGOLJV2E7EHQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.195162279 (486h30m16.74663274s) [noise=true , address=145.79.227.20:4438 , piece_id=VZK574G5G2HXO5JMVMMEKGFCOJKOTU5KUIJDDEROFXRYTBTITO3A, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.197119445 (486h30m16.748589896s) [noise=true , address=145.79.227.7:4295 , piece_id=OLB3PUIQ4RYD2GGVF4I32FEGXO2O4X56PLJMOKNTJVBK53KZ3VYQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.197903478 (486h30m16.749373929s) [noise=true , address=145.79.227.32:4836 , piece_id=4KBSAAZLHC7OYLWN7RBDTQZUUTMK4HFGFUBES7K2PDKBXZST3OBA, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:19.999395950 (486h30m18.550866411s) [upload=428397]: Next returned nil segment Nov 29 14:34:19.999415177 (486h30m18.550885648s) [upload=428397]: Waiting for error group managing segments... Nov 29 14:34:20.285725942 (486h30m18.837196404s) [noise=true , address=77.90.31.13:4627 , piece_id=62A76IX5BE7WFBCCEMPSENCE5PWOSEMCUIKEATUPWGSLTCFJ4BGQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:20.363243793 (486h30m18.914714244s) [noise=true , address=45.91.116.5:4297 , piece_id=GD7HY7SUS6XPN6KJDC5AVRP2ECTJSB2MK4YBQXCNUOTUBVKFRGVQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:22.551752465 (486h30m21.103222926s) [noise=true , address=145.79.227.31:4798 , piece_id=NG2JWZTD5NPFNDIVWYSWFHQHEDTURKJOY6YZ7G6LAQRIDI7WD3FQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:22.666545505 (486h30m21.218015967s) [noise=true , address=145.79.227.22:4509 , piece_id=N2AEKOF7MLRQZ2HS7FSXV3U6WNI7YQLWP3W4PTFZLMAWQQREBTKQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:23.006352209 (486h30m21.557822671s) [noise=true , address=145.79.227.14:4839 , piece_id=RDZDYLJRCVHDKTTCAF3JB5ULPKFMSC2E4HNLB2RPLPY2GTUC6ZDQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:23.023700063 (486h30m21.575170525s) [noise=true , address=145.79.227.33:4869 , piece_id=V3JJCDM7EBRRZL2GPDMSCQAEHP2YX2QXDTDZDT5M6FGVLB27G7PQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:23.426873130 (486h30m21.978343581s) [noise=true , address=145.79.227.32:4836 , piece_id=4KBSAAZLHC7OYLWN7RBDTQZUUTMK4HFGFUBES7K2PDKBXZST3OBA, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:23.722460741 (486h30m22.273931202s) [noise=true , address=145.79.227.40:4049 , piece_id=DIBJ3Y4NA3HRAZD7GLO72STJHS5XHA3OTCAMWL4VCHMEILLCL3UA, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:25.698494096 (486h30m24.249964557s) [noise=true , address=145.79.227.39:4013 , piece_id=YFEKNDHNI77M7MG76EISOGVU32UOZ536CK4X3Z6SFGOLJV2E7EHQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:25.822411189 (486h30m24.373881651s) [noise=true , address=145.79.227.2:4114 , piece_id=ZB7V2MM6KI6VTSGSC4ZD4QI6EA2265OP3MVLWD2AYAHFERAUE7CQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:29.722449985 (486h30m28.273920436s) [noise=true , address=145.79.227.12:4767 , piece_id=5Q27HBFWOOI6PIIIMXELWG3QYSEA5B5LAOMU5K4FICMUMPDZIQAA, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:47.238199596 (486h30m45.789670047s) [noise=true , address=145.79.227.20:4438 , piece_id=VZK574G5G2HXO5JMVMMEKGFCOJKOTU5KUIJDDEROFXRYTBTITO3A, seg_pos={0 0}, upload=428397]: Done uploading piece. err: ecclient: upload failed (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, address:145.79.227.20:4438): protocol: write tcp 188.166.225.170:34106->145.79.227.20:4438: use of closed network connection; write tcp 188.166.225.170:34106->145.79.227.20:4438: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34106->145.79.227.20:4438: use of closed network connection Nov 29 14:34:47.238464621 (486h30m45.789935082s) [noise=true , address=145.79.227.21:4473 , piece_id=S2V3HONDHED6NZSIFZOTWU2DQ6FKESGGZ4EVMZX36LI4NT32WDWQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:46118->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:46118->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46118->145.79.227.21:4473: use of closed network connection Nov 29 14:34:47.238703104 (486h30m45.790173565s) [noise=true , address=145.79.227.24:4583 , piece_id=B6GTB3UF3XX64M5U57FFCFAMWNWDJUDPIZDG23KOHXOVCFTGKCLA, seg_pos={0 0}, upload=428397]: Done uploading piece. err: ecclient: upload failed (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, address:145.79.227.24:4583): protocol: write tcp 188.166.225.170:51236->145.79.227.24:4583: use of closed network connection; write tcp 188.166.225.170:51236->145.79.227.24:4583: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51236->145.79.227.24:4583: use of closed network connection Nov 29 14:34:47.238869100 (486h30m45.790339561s) [noise=true , address=145.79.227.7:4295 , piece_id=OLB3PUIQ4RYD2GGVF4I32FEGXO2O4X56PLJMOKNTJVBK53KZ3VYQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:145.79.227.7:4295): protocol: write tcp 188.166.225.170:46306->145.79.227.7:4295: use of closed network connection; write tcp 188.166.225.170:46306->145.79.227.7:4295: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46306->145.79.227.7:4295: use of closed network connection Nov 29 14:34:47.239032171 (486h30m45.790502632s) [noise=true , address=145.79.227.6:4259 , piece_id=KIYGYFEAYWG2NUAQZNWG4ZLMRY6GFJADI5LD635JE5M4DLHTX4TQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:145.79.227.6:4259): protocol: write tcp 188.166.225.170:48184->145.79.227.6:4259: use of closed network connection; write tcp 188.166.225.170:48184->145.79.227.6:4259: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48184->145.79.227.6:4259: use of closed network connection Nov 29 14:34:47.239076555 (486h30m45.790547027s) [seg_pos={0 0}, upload=428397]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 29 14:34:47.481897829 (486h30m46.0333683s) [upload=428397]: Done uploading segments... Nov 29 14:34:47.481943356 (486h30m46.033413817s) [upload=428397]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.83751136s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [........................X..............................X....] duration: 3.520811653s trace: https://jaeger.dev.storj.io/trace/1e2e0cb4b27ac060 FAIL (2025-11-29 15:05:15 UTC) orbiter warning: context deadline exceeded Duration: 3.52092515s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 6.447036905s trace: https://jaeger.dev.storj.io/trace/4add906a25eb3f89 FAIL (2025-11-29 15:05:10 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Nov 29 14:34:17.941236229 (486h30m16.49270669s) [upload=428397]: Got next segment. Inline: false Nov 29 14:34:17.941289010 (486h30m16.492759471s) [upload=428397]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 29 14:34:17.941296063 (486h30m16.492766524s) [upload=428397]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 29 14:34:18.189160800 (486h30m16.740631261s) [seg_pos={0 0}, upload=428397]: Begin upload segment... Nov 29 14:34:18.189218760 (486h30m16.740689221s) [seg_pos={0 0}, upload=428397]: Done begin upload segment. Nov 29 14:34:18.189254097 (486h30m16.740724548s) [noise=true , address=145.79.227.40:4049 , piece_id=DIBJ3Y4NA3HRAZD7GLO72STJHS5XHA3OTCAMWL4VCHMEILLCL3UA, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.189472724 (486h30m16.740943185s) [noise=true , address=77.90.31.13:4627 , piece_id=62A76IX5BE7WFBCCEMPSENCE5PWOSEMCUIKEATUPWGSLTCFJ4BGQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.190037069 (486h30m16.74150753s) [noise=true , address=145.79.227.2:4114 , piece_id=ZB7V2MM6KI6VTSGSC4ZD4QI6EA2265OP3MVLWD2AYAHFERAUE7CQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.190276184 (486h30m16.741746645s) [noise=true , address=145.79.227.21:4473 , piece_id=S2V3HONDHED6NZSIFZOTWU2DQ6FKESGGZ4EVMZX36LI4NT32WDWQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.190487406 (486h30m16.741957867s) [noise=true , address=145.79.227.31:4798 , piece_id=NG2JWZTD5NPFNDIVWYSWFHQHEDTURKJOY6YZ7G6LAQRIDI7WD3FQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.190825751 (486h30m16.742296212s) [noise=true , address=45.91.116.5:4297 , piece_id=GD7HY7SUS6XPN6KJDC5AVRP2ECTJSB2MK4YBQXCNUOTUBVKFRGVQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.190967721 (486h30m16.742438192s) [noise=true , address=145.79.227.24:4583 , piece_id=B6GTB3UF3XX64M5U57FFCFAMWNWDJUDPIZDG23KOHXOVCFTGKCLA, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.191308490 (486h30m16.742778951s) [noise=true , address=145.79.227.33:4869 , piece_id=V3JJCDM7EBRRZL2GPDMSCQAEHP2YX2QXDTDZDT5M6FGVLB27G7PQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.191365077 (486h30m16.742835528s) [noise=true , address=145.79.227.6:4259 , piece_id=KIYGYFEAYWG2NUAQZNWG4ZLMRY6GFJADI5LD635JE5M4DLHTX4TQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.191782502 (486h30m16.743252963s) [noise=true , address=145.79.227.22:4509 , piece_id=N2AEKOF7MLRQZ2HS7FSXV3U6WNI7YQLWP3W4PTFZLMAWQQREBTKQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.192449502 (486h30m16.743919953s) [noise=true , address=145.79.227.14:4839 , piece_id=RDZDYLJRCVHDKTTCAF3JB5ULPKFMSC2E4HNLB2RPLPY2GTUC6ZDQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.193091886 (486h30m16.744562347s) [noise=true , address=145.79.227.12:4767 , piece_id=5Q27HBFWOOI6PIIIMXELWG3QYSEA5B5LAOMU5K4FICMUMPDZIQAA, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.193463874 (486h30m16.744934335s) [noise=true , address=145.79.227.39:4013 , piece_id=YFEKNDHNI77M7MG76EISOGVU32UOZ536CK4X3Z6SFGOLJV2E7EHQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.195162279 (486h30m16.74663274s) [noise=true , address=145.79.227.20:4438 , piece_id=VZK574G5G2HXO5JMVMMEKGFCOJKOTU5KUIJDDEROFXRYTBTITO3A, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.197119445 (486h30m16.748589896s) [noise=true , address=145.79.227.7:4295 , piece_id=OLB3PUIQ4RYD2GGVF4I32FEGXO2O4X56PLJMOKNTJVBK53KZ3VYQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.197903478 (486h30m16.749373929s) [noise=true , address=145.79.227.32:4836 , piece_id=4KBSAAZLHC7OYLWN7RBDTQZUUTMK4HFGFUBES7K2PDKBXZST3OBA, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:19.999395950 (486h30m18.550866411s) [upload=428397]: Next returned nil segment Nov 29 14:34:19.999415177 (486h30m18.550885648s) [upload=428397]: Waiting for error group managing segments... Nov 29 14:34:20.285725942 (486h30m18.837196404s) [noise=true , address=77.90.31.13:4627 , piece_id=62A76IX5BE7WFBCCEMPSENCE5PWOSEMCUIKEATUPWGSLTCFJ4BGQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:20.363243793 (486h30m18.914714244s) [noise=true , address=45.91.116.5:4297 , piece_id=GD7HY7SUS6XPN6KJDC5AVRP2ECTJSB2MK4YBQXCNUOTUBVKFRGVQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:22.551752465 (486h30m21.103222926s) [noise=true , address=145.79.227.31:4798 , piece_id=NG2JWZTD5NPFNDIVWYSWFHQHEDTURKJOY6YZ7G6LAQRIDI7WD3FQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:22.666545505 (486h30m21.218015967s) [noise=true , address=145.79.227.22:4509 , piece_id=N2AEKOF7MLRQZ2HS7FSXV3U6WNI7YQLWP3W4PTFZLMAWQQREBTKQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:23.006352209 (486h30m21.557822671s) [noise=true , address=145.79.227.14:4839 , piece_id=RDZDYLJRCVHDKTTCAF3JB5ULPKFMSC2E4HNLB2RPLPY2GTUC6ZDQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:23.023700063 (486h30m21.575170525s) [noise=true , address=145.79.227.33:4869 , piece_id=V3JJCDM7EBRRZL2GPDMSCQAEHP2YX2QXDTDZDT5M6FGVLB27G7PQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:23.426873130 (486h30m21.978343581s) [noise=true , address=145.79.227.32:4836 , piece_id=4KBSAAZLHC7OYLWN7RBDTQZUUTMK4HFGFUBES7K2PDKBXZST3OBA, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:23.722460741 (486h30m22.273931202s) [noise=true , address=145.79.227.40:4049 , piece_id=DIBJ3Y4NA3HRAZD7GLO72STJHS5XHA3OTCAMWL4VCHMEILLCL3UA, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:25.698494096 (486h30m24.249964557s) [noise=true , address=145.79.227.39:4013 , piece_id=YFEKNDHNI77M7MG76EISOGVU32UOZ536CK4X3Z6SFGOLJV2E7EHQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:25.822411189 (486h30m24.373881651s) [noise=true , address=145.79.227.2:4114 , piece_id=ZB7V2MM6KI6VTSGSC4ZD4QI6EA2265OP3MVLWD2AYAHFERAUE7CQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:29.722449985 (486h30m28.273920436s) [noise=true , address=145.79.227.12:4767 , piece_id=5Q27HBFWOOI6PIIIMXELWG3QYSEA5B5LAOMU5K4FICMUMPDZIQAA, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:47.238199596 (486h30m45.789670047s) [noise=true , address=145.79.227.20:4438 , piece_id=VZK574G5G2HXO5JMVMMEKGFCOJKOTU5KUIJDDEROFXRYTBTITO3A, seg_pos={0 0}, upload=428397]: Done uploading piece. err: ecclient: upload failed (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, address:145.79.227.20:4438): protocol: write tcp 188.166.225.170:34106->145.79.227.20:4438: use of closed network connection; write tcp 188.166.225.170:34106->145.79.227.20:4438: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34106->145.79.227.20:4438: use of closed network connection Nov 29 14:34:47.238464621 (486h30m45.789935082s) [noise=true , address=145.79.227.21:4473 , piece_id=S2V3HONDHED6NZSIFZOTWU2DQ6FKESGGZ4EVMZX36LI4NT32WDWQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:46118->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:46118->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46118->145.79.227.21:4473: use of closed network connection Nov 29 14:34:47.238703104 (486h30m45.790173565s) [noise=true , address=145.79.227.24:4583 , piece_id=B6GTB3UF3XX64M5U57FFCFAMWNWDJUDPIZDG23KOHXOVCFTGKCLA, seg_pos={0 0}, upload=428397]: Done uploading piece. err: ecclient: upload failed (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, address:145.79.227.24:4583): protocol: write tcp 188.166.225.170:51236->145.79.227.24:4583: use of closed network connection; write tcp 188.166.225.170:51236->145.79.227.24:4583: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51236->145.79.227.24:4583: use of closed network connection Nov 29 14:34:47.238869100 (486h30m45.790339561s) [noise=true , address=145.79.227.7:4295 , piece_id=OLB3PUIQ4RYD2GGVF4I32FEGXO2O4X56PLJMOKNTJVBK53KZ3VYQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:145.79.227.7:4295): protocol: write tcp 188.166.225.170:46306->145.79.227.7:4295: use of closed network connection; write tcp 188.166.225.170:46306->145.79.227.7:4295: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46306->145.79.227.7:4295: use of closed network connection Nov 29 14:34:47.239032171 (486h30m45.790502632s) [noise=true , address=145.79.227.6:4259 , piece_id=KIYGYFEAYWG2NUAQZNWG4ZLMRY6GFJADI5LD635JE5M4DLHTX4TQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:145.79.227.6:4259): protocol: write tcp 188.166.225.170:48184->145.79.227.6:4259: use of closed network connection; write tcp 188.166.225.170:48184->145.79.227.6:4259: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48184->145.79.227.6:4259: use of closed network connection Nov 29 14:34:47.239076555 (486h30m45.790547027s) [seg_pos={0 0}, upload=428397]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 29 14:34:47.481897829 (486h30m46.0333683s) [upload=428397]: Done uploading segments... Nov 29 14:34:47.481943356 (486h30m46.033413817s) [upload=428397]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 6.447070298s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 6.230308507s trace: https://jaeger.dev.storj.io/trace/4add906a25eb3f89 FAIL (2025-11-29 15:04:51 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Nov 29 14:34:17.941236229 (486h30m16.49270669s) [upload=428397]: Got next segment. Inline: false Nov 29 14:34:17.941289010 (486h30m16.492759471s) [upload=428397]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 29 14:34:17.941296063 (486h30m16.492766524s) [upload=428397]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 29 14:34:18.189160800 (486h30m16.740631261s) [seg_pos={0 0}, upload=428397]: Begin upload segment... Nov 29 14:34:18.189218760 (486h30m16.740689221s) [seg_pos={0 0}, upload=428397]: Done begin upload segment. Nov 29 14:34:18.189254097 (486h30m16.740724548s) [noise=true , address=145.79.227.40:4049 , piece_id=DIBJ3Y4NA3HRAZD7GLO72STJHS5XHA3OTCAMWL4VCHMEILLCL3UA, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.189472724 (486h30m16.740943185s) [noise=true , address=77.90.31.13:4627 , piece_id=62A76IX5BE7WFBCCEMPSENCE5PWOSEMCUIKEATUPWGSLTCFJ4BGQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.190037069 (486h30m16.74150753s) [noise=true , address=145.79.227.2:4114 , piece_id=ZB7V2MM6KI6VTSGSC4ZD4QI6EA2265OP3MVLWD2AYAHFERAUE7CQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.190276184 (486h30m16.741746645s) [noise=true , address=145.79.227.21:4473 , piece_id=S2V3HONDHED6NZSIFZOTWU2DQ6FKESGGZ4EVMZX36LI4NT32WDWQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.190487406 (486h30m16.741957867s) [noise=true , address=145.79.227.31:4798 , piece_id=NG2JWZTD5NPFNDIVWYSWFHQHEDTURKJOY6YZ7G6LAQRIDI7WD3FQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.190825751 (486h30m16.742296212s) [noise=true , address=45.91.116.5:4297 , piece_id=GD7HY7SUS6XPN6KJDC5AVRP2ECTJSB2MK4YBQXCNUOTUBVKFRGVQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.190967721 (486h30m16.742438192s) [noise=true , address=145.79.227.24:4583 , piece_id=B6GTB3UF3XX64M5U57FFCFAMWNWDJUDPIZDG23KOHXOVCFTGKCLA, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.191308490 (486h30m16.742778951s) [noise=true , address=145.79.227.33:4869 , piece_id=V3JJCDM7EBRRZL2GPDMSCQAEHP2YX2QXDTDZDT5M6FGVLB27G7PQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.191365077 (486h30m16.742835528s) [noise=true , address=145.79.227.6:4259 , piece_id=KIYGYFEAYWG2NUAQZNWG4ZLMRY6GFJADI5LD635JE5M4DLHTX4TQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.191782502 (486h30m16.743252963s) [noise=true , address=145.79.227.22:4509 , piece_id=N2AEKOF7MLRQZ2HS7FSXV3U6WNI7YQLWP3W4PTFZLMAWQQREBTKQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.192449502 (486h30m16.743919953s) [noise=true , address=145.79.227.14:4839 , piece_id=RDZDYLJRCVHDKTTCAF3JB5ULPKFMSC2E4HNLB2RPLPY2GTUC6ZDQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.193091886 (486h30m16.744562347s) [noise=true , address=145.79.227.12:4767 , piece_id=5Q27HBFWOOI6PIIIMXELWG3QYSEA5B5LAOMU5K4FICMUMPDZIQAA, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.193463874 (486h30m16.744934335s) [noise=true , address=145.79.227.39:4013 , piece_id=YFEKNDHNI77M7MG76EISOGVU32UOZ536CK4X3Z6SFGOLJV2E7EHQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.195162279 (486h30m16.74663274s) [noise=true , address=145.79.227.20:4438 , piece_id=VZK574G5G2HXO5JMVMMEKGFCOJKOTU5KUIJDDEROFXRYTBTITO3A, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.197119445 (486h30m16.748589896s) [noise=true , address=145.79.227.7:4295 , piece_id=OLB3PUIQ4RYD2GGVF4I32FEGXO2O4X56PLJMOKNTJVBK53KZ3VYQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.197903478 (486h30m16.749373929s) [noise=true , address=145.79.227.32:4836 , piece_id=4KBSAAZLHC7OYLWN7RBDTQZUUTMK4HFGFUBES7K2PDKBXZST3OBA, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:19.999395950 (486h30m18.550866411s) [upload=428397]: Next returned nil segment Nov 29 14:34:19.999415177 (486h30m18.550885648s) [upload=428397]: Waiting for error group managing segments... Nov 29 14:34:20.285725942 (486h30m18.837196404s) [noise=true , address=77.90.31.13:4627 , piece_id=62A76IX5BE7WFBCCEMPSENCE5PWOSEMCUIKEATUPWGSLTCFJ4BGQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:20.363243793 (486h30m18.914714244s) [noise=true , address=45.91.116.5:4297 , piece_id=GD7HY7SUS6XPN6KJDC5AVRP2ECTJSB2MK4YBQXCNUOTUBVKFRGVQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:22.551752465 (486h30m21.103222926s) [noise=true , address=145.79.227.31:4798 , piece_id=NG2JWZTD5NPFNDIVWYSWFHQHEDTURKJOY6YZ7G6LAQRIDI7WD3FQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:22.666545505 (486h30m21.218015967s) [noise=true , address=145.79.227.22:4509 , piece_id=N2AEKOF7MLRQZ2HS7FSXV3U6WNI7YQLWP3W4PTFZLMAWQQREBTKQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:23.006352209 (486h30m21.557822671s) [noise=true , address=145.79.227.14:4839 , piece_id=RDZDYLJRCVHDKTTCAF3JB5ULPKFMSC2E4HNLB2RPLPY2GTUC6ZDQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:23.023700063 (486h30m21.575170525s) [noise=true , address=145.79.227.33:4869 , piece_id=V3JJCDM7EBRRZL2GPDMSCQAEHP2YX2QXDTDZDT5M6FGVLB27G7PQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:23.426873130 (486h30m21.978343581s) [noise=true , address=145.79.227.32:4836 , piece_id=4KBSAAZLHC7OYLWN7RBDTQZUUTMK4HFGFUBES7K2PDKBXZST3OBA, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:23.722460741 (486h30m22.273931202s) [noise=true , address=145.79.227.40:4049 , piece_id=DIBJ3Y4NA3HRAZD7GLO72STJHS5XHA3OTCAMWL4VCHMEILLCL3UA, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:25.698494096 (486h30m24.249964557s) [noise=true , address=145.79.227.39:4013 , piece_id=YFEKNDHNI77M7MG76EISOGVU32UOZ536CK4X3Z6SFGOLJV2E7EHQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:25.822411189 (486h30m24.373881651s) [noise=true , address=145.79.227.2:4114 , piece_id=ZB7V2MM6KI6VTSGSC4ZD4QI6EA2265OP3MVLWD2AYAHFERAUE7CQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:29.722449985 (486h30m28.273920436s) [noise=true , address=145.79.227.12:4767 , piece_id=5Q27HBFWOOI6PIIIMXELWG3QYSEA5B5LAOMU5K4FICMUMPDZIQAA, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:47.238199596 (486h30m45.789670047s) [noise=true , address=145.79.227.20:4438 , piece_id=VZK574G5G2HXO5JMVMMEKGFCOJKOTU5KUIJDDEROFXRYTBTITO3A, seg_pos={0 0}, upload=428397]: Done uploading piece. err: ecclient: upload failed (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, address:145.79.227.20:4438): protocol: write tcp 188.166.225.170:34106->145.79.227.20:4438: use of closed network connection; write tcp 188.166.225.170:34106->145.79.227.20:4438: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34106->145.79.227.20:4438: use of closed network connection Nov 29 14:34:47.238464621 (486h30m45.789935082s) [noise=true , address=145.79.227.21:4473 , piece_id=S2V3HONDHED6NZSIFZOTWU2DQ6FKESGGZ4EVMZX36LI4NT32WDWQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:46118->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:46118->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46118->145.79.227.21:4473: use of closed network connection Nov 29 14:34:47.238703104 (486h30m45.790173565s) [noise=true , address=145.79.227.24:4583 , piece_id=B6GTB3UF3XX64M5U57FFCFAMWNWDJUDPIZDG23KOHXOVCFTGKCLA, seg_pos={0 0}, upload=428397]: Done uploading piece. err: ecclient: upload failed (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, address:145.79.227.24:4583): protocol: write tcp 188.166.225.170:51236->145.79.227.24:4583: use of closed network connection; write tcp 188.166.225.170:51236->145.79.227.24:4583: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51236->145.79.227.24:4583: use of closed network connection Nov 29 14:34:47.238869100 (486h30m45.790339561s) [noise=true , address=145.79.227.7:4295 , piece_id=OLB3PUIQ4RYD2GGVF4I32FEGXO2O4X56PLJMOKNTJVBK53KZ3VYQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:145.79.227.7:4295): protocol: write tcp 188.166.225.170:46306->145.79.227.7:4295: use of closed network connection; write tcp 188.166.225.170:46306->145.79.227.7:4295: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46306->145.79.227.7:4295: use of closed network connection Nov 29 14:34:47.239032171 (486h30m45.790502632s) [noise=true , address=145.79.227.6:4259 , piece_id=KIYGYFEAYWG2NUAQZNWG4ZLMRY6GFJADI5LD635JE5M4DLHTX4TQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:145.79.227.6:4259): protocol: write tcp 188.166.225.170:48184->145.79.227.6:4259: use of closed network connection; write tcp 188.166.225.170:48184->145.79.227.6:4259: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48184->145.79.227.6:4259: use of closed network connection Nov 29 14:34:47.239076555 (486h30m45.790547027s) [seg_pos={0 0}, upload=428397]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 29 14:34:47.481897829 (486h30m46.0333683s) [upload=428397]: Done uploading segments... Nov 29 14:34:47.481943356 (486h30m46.033413817s) [upload=428397]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 6.230354856s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.........................X..............................X...] duration: 3.691982569s trace: https://jaeger.dev.storj.io/trace/1e2e0cb4b27ac060 FAIL (2025-11-29 15:04:42 UTC) orbiter warning: context deadline exceeded Duration: 3.692016362s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [..........................X..............................X..] duration: 3.668268561s trace: https://jaeger.dev.storj.io/trace/1e2e0cb4b27ac060 FAIL (2025-11-29 15:04:15 UTC) orbiter warning: context deadline exceeded Duration: 3.66829953s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 6.184133626s trace: https://jaeger.dev.storj.io/trace/4add906a25eb3f89 FAIL (2025-11-29 15:04:03 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Nov 29 14:34:17.941236229 (486h30m16.49270669s) [upload=428397]: Got next segment. Inline: false Nov 29 14:34:17.941289010 (486h30m16.492759471s) [upload=428397]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 29 14:34:17.941296063 (486h30m16.492766524s) [upload=428397]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 29 14:34:18.189160800 (486h30m16.740631261s) [seg_pos={0 0}, upload=428397]: Begin upload segment... Nov 29 14:34:18.189218760 (486h30m16.740689221s) [seg_pos={0 0}, upload=428397]: Done begin upload segment. Nov 29 14:34:18.189254097 (486h30m16.740724548s) [noise=true , address=145.79.227.40:4049 , piece_id=DIBJ3Y4NA3HRAZD7GLO72STJHS5XHA3OTCAMWL4VCHMEILLCL3UA, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.189472724 (486h30m16.740943185s) [noise=true , address=77.90.31.13:4627 , piece_id=62A76IX5BE7WFBCCEMPSENCE5PWOSEMCUIKEATUPWGSLTCFJ4BGQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.190037069 (486h30m16.74150753s) [noise=true , address=145.79.227.2:4114 , piece_id=ZB7V2MM6KI6VTSGSC4ZD4QI6EA2265OP3MVLWD2AYAHFERAUE7CQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.190276184 (486h30m16.741746645s) [noise=true , address=145.79.227.21:4473 , piece_id=S2V3HONDHED6NZSIFZOTWU2DQ6FKESGGZ4EVMZX36LI4NT32WDWQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.190487406 (486h30m16.741957867s) [noise=true , address=145.79.227.31:4798 , piece_id=NG2JWZTD5NPFNDIVWYSWFHQHEDTURKJOY6YZ7G6LAQRIDI7WD3FQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.190825751 (486h30m16.742296212s) [noise=true , address=45.91.116.5:4297 , piece_id=GD7HY7SUS6XPN6KJDC5AVRP2ECTJSB2MK4YBQXCNUOTUBVKFRGVQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.190967721 (486h30m16.742438192s) [noise=true , address=145.79.227.24:4583 , piece_id=B6GTB3UF3XX64M5U57FFCFAMWNWDJUDPIZDG23KOHXOVCFTGKCLA, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.191308490 (486h30m16.742778951s) [noise=true , address=145.79.227.33:4869 , piece_id=V3JJCDM7EBRRZL2GPDMSCQAEHP2YX2QXDTDZDT5M6FGVLB27G7PQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.191365077 (486h30m16.742835528s) [noise=true , address=145.79.227.6:4259 , piece_id=KIYGYFEAYWG2NUAQZNWG4ZLMRY6GFJADI5LD635JE5M4DLHTX4TQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.191782502 (486h30m16.743252963s) [noise=true , address=145.79.227.22:4509 , piece_id=N2AEKOF7MLRQZ2HS7FSXV3U6WNI7YQLWP3W4PTFZLMAWQQREBTKQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.192449502 (486h30m16.743919953s) [noise=true , address=145.79.227.14:4839 , piece_id=RDZDYLJRCVHDKTTCAF3JB5ULPKFMSC2E4HNLB2RPLPY2GTUC6ZDQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.193091886 (486h30m16.744562347s) [noise=true , address=145.79.227.12:4767 , piece_id=5Q27HBFWOOI6PIIIMXELWG3QYSEA5B5LAOMU5K4FICMUMPDZIQAA, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.193463874 (486h30m16.744934335s) [noise=true , address=145.79.227.39:4013 , piece_id=YFEKNDHNI77M7MG76EISOGVU32UOZ536CK4X3Z6SFGOLJV2E7EHQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.195162279 (486h30m16.74663274s) [noise=true , address=145.79.227.20:4438 , piece_id=VZK574G5G2HXO5JMVMMEKGFCOJKOTU5KUIJDDEROFXRYTBTITO3A, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.197119445 (486h30m16.748589896s) [noise=true , address=145.79.227.7:4295 , piece_id=OLB3PUIQ4RYD2GGVF4I32FEGXO2O4X56PLJMOKNTJVBK53KZ3VYQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.197903478 (486h30m16.749373929s) [noise=true , address=145.79.227.32:4836 , piece_id=4KBSAAZLHC7OYLWN7RBDTQZUUTMK4HFGFUBES7K2PDKBXZST3OBA, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:19.999395950 (486h30m18.550866411s) [upload=428397]: Next returned nil segment Nov 29 14:34:19.999415177 (486h30m18.550885648s) [upload=428397]: Waiting for error group managing segments... Nov 29 14:34:20.285725942 (486h30m18.837196404s) [noise=true , address=77.90.31.13:4627 , piece_id=62A76IX5BE7WFBCCEMPSENCE5PWOSEMCUIKEATUPWGSLTCFJ4BGQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:20.363243793 (486h30m18.914714244s) [noise=true , address=45.91.116.5:4297 , piece_id=GD7HY7SUS6XPN6KJDC5AVRP2ECTJSB2MK4YBQXCNUOTUBVKFRGVQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:22.551752465 (486h30m21.103222926s) [noise=true , address=145.79.227.31:4798 , piece_id=NG2JWZTD5NPFNDIVWYSWFHQHEDTURKJOY6YZ7G6LAQRIDI7WD3FQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:22.666545505 (486h30m21.218015967s) [noise=true , address=145.79.227.22:4509 , piece_id=N2AEKOF7MLRQZ2HS7FSXV3U6WNI7YQLWP3W4PTFZLMAWQQREBTKQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:23.006352209 (486h30m21.557822671s) [noise=true , address=145.79.227.14:4839 , piece_id=RDZDYLJRCVHDKTTCAF3JB5ULPKFMSC2E4HNLB2RPLPY2GTUC6ZDQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:23.023700063 (486h30m21.575170525s) [noise=true , address=145.79.227.33:4869 , piece_id=V3JJCDM7EBRRZL2GPDMSCQAEHP2YX2QXDTDZDT5M6FGVLB27G7PQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:23.426873130 (486h30m21.978343581s) [noise=true , address=145.79.227.32:4836 , piece_id=4KBSAAZLHC7OYLWN7RBDTQZUUTMK4HFGFUBES7K2PDKBXZST3OBA, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:23.722460741 (486h30m22.273931202s) [noise=true , address=145.79.227.40:4049 , piece_id=DIBJ3Y4NA3HRAZD7GLO72STJHS5XHA3OTCAMWL4VCHMEILLCL3UA, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:25.698494096 (486h30m24.249964557s) [noise=true , address=145.79.227.39:4013 , piece_id=YFEKNDHNI77M7MG76EISOGVU32UOZ536CK4X3Z6SFGOLJV2E7EHQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:25.822411189 (486h30m24.373881651s) [noise=true , address=145.79.227.2:4114 , piece_id=ZB7V2MM6KI6VTSGSC4ZD4QI6EA2265OP3MVLWD2AYAHFERAUE7CQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:29.722449985 (486h30m28.273920436s) [noise=true , address=145.79.227.12:4767 , piece_id=5Q27HBFWOOI6PIIIMXELWG3QYSEA5B5LAOMU5K4FICMUMPDZIQAA, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:47.238199596 (486h30m45.789670047s) [noise=true , address=145.79.227.20:4438 , piece_id=VZK574G5G2HXO5JMVMMEKGFCOJKOTU5KUIJDDEROFXRYTBTITO3A, seg_pos={0 0}, upload=428397]: Done uploading piece. err: ecclient: upload failed (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, address:145.79.227.20:4438): protocol: write tcp 188.166.225.170:34106->145.79.227.20:4438: use of closed network connection; write tcp 188.166.225.170:34106->145.79.227.20:4438: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34106->145.79.227.20:4438: use of closed network connection Nov 29 14:34:47.238464621 (486h30m45.789935082s) [noise=true , address=145.79.227.21:4473 , piece_id=S2V3HONDHED6NZSIFZOTWU2DQ6FKESGGZ4EVMZX36LI4NT32WDWQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:46118->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:46118->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46118->145.79.227.21:4473: use of closed network connection Nov 29 14:34:47.238703104 (486h30m45.790173565s) [noise=true , address=145.79.227.24:4583 , piece_id=B6GTB3UF3XX64M5U57FFCFAMWNWDJUDPIZDG23KOHXOVCFTGKCLA, seg_pos={0 0}, upload=428397]: Done uploading piece. err: ecclient: upload failed (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, address:145.79.227.24:4583): protocol: write tcp 188.166.225.170:51236->145.79.227.24:4583: use of closed network connection; write tcp 188.166.225.170:51236->145.79.227.24:4583: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51236->145.79.227.24:4583: use of closed network connection Nov 29 14:34:47.238869100 (486h30m45.790339561s) [noise=true , address=145.79.227.7:4295 , piece_id=OLB3PUIQ4RYD2GGVF4I32FEGXO2O4X56PLJMOKNTJVBK53KZ3VYQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:145.79.227.7:4295): protocol: write tcp 188.166.225.170:46306->145.79.227.7:4295: use of closed network connection; write tcp 188.166.225.170:46306->145.79.227.7:4295: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46306->145.79.227.7:4295: use of closed network connection Nov 29 14:34:47.239032171 (486h30m45.790502632s) [noise=true , address=145.79.227.6:4259 , piece_id=KIYGYFEAYWG2NUAQZNWG4ZLMRY6GFJADI5LD635JE5M4DLHTX4TQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:145.79.227.6:4259): protocol: write tcp 188.166.225.170:48184->145.79.227.6:4259: use of closed network connection; write tcp 188.166.225.170:48184->145.79.227.6:4259: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48184->145.79.227.6:4259: use of closed network connection Nov 29 14:34:47.239076555 (486h30m45.790547027s) [seg_pos={0 0}, upload=428397]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 29 14:34:47.481897829 (486h30m46.0333683s) [upload=428397]: Done uploading segments... Nov 29 14:34:47.481943356 (486h30m46.033413817s) [upload=428397]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 6.184162s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [...........................X..............................X.] duration: 3.771370134s trace: https://jaeger.dev.storj.io/trace/1e2e0cb4b27ac060 FAIL (2025-11-29 15:03:13 UTC) orbiter warning: context deadline exceeded Duration: 3.771428997s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 5.985932817s trace: https://jaeger.dev.storj.io/trace/4add906a25eb3f89 FAIL (2025-11-29 15:03:01 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Nov 29 14:34:17.941236229 (486h30m16.49270669s) [upload=428397]: Got next segment. Inline: false Nov 29 14:34:17.941289010 (486h30m16.492759471s) [upload=428397]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 29 14:34:17.941296063 (486h30m16.492766524s) [upload=428397]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 29 14:34:18.189160800 (486h30m16.740631261s) [seg_pos={0 0}, upload=428397]: Begin upload segment... Nov 29 14:34:18.189218760 (486h30m16.740689221s) [seg_pos={0 0}, upload=428397]: Done begin upload segment. Nov 29 14:34:18.189254097 (486h30m16.740724548s) [noise=true , address=145.79.227.40:4049 , piece_id=DIBJ3Y4NA3HRAZD7GLO72STJHS5XHA3OTCAMWL4VCHMEILLCL3UA, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.189472724 (486h30m16.740943185s) [noise=true , address=77.90.31.13:4627 , piece_id=62A76IX5BE7WFBCCEMPSENCE5PWOSEMCUIKEATUPWGSLTCFJ4BGQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.190037069 (486h30m16.74150753s) [noise=true , address=145.79.227.2:4114 , piece_id=ZB7V2MM6KI6VTSGSC4ZD4QI6EA2265OP3MVLWD2AYAHFERAUE7CQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.190276184 (486h30m16.741746645s) [noise=true , address=145.79.227.21:4473 , piece_id=S2V3HONDHED6NZSIFZOTWU2DQ6FKESGGZ4EVMZX36LI4NT32WDWQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.190487406 (486h30m16.741957867s) [noise=true , address=145.79.227.31:4798 , piece_id=NG2JWZTD5NPFNDIVWYSWFHQHEDTURKJOY6YZ7G6LAQRIDI7WD3FQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.190825751 (486h30m16.742296212s) [noise=true , address=45.91.116.5:4297 , piece_id=GD7HY7SUS6XPN6KJDC5AVRP2ECTJSB2MK4YBQXCNUOTUBVKFRGVQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.190967721 (486h30m16.742438192s) [noise=true , address=145.79.227.24:4583 , piece_id=B6GTB3UF3XX64M5U57FFCFAMWNWDJUDPIZDG23KOHXOVCFTGKCLA, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.191308490 (486h30m16.742778951s) [noise=true , address=145.79.227.33:4869 , piece_id=V3JJCDM7EBRRZL2GPDMSCQAEHP2YX2QXDTDZDT5M6FGVLB27G7PQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.191365077 (486h30m16.742835528s) [noise=true , address=145.79.227.6:4259 , piece_id=KIYGYFEAYWG2NUAQZNWG4ZLMRY6GFJADI5LD635JE5M4DLHTX4TQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.191782502 (486h30m16.743252963s) [noise=true , address=145.79.227.22:4509 , piece_id=N2AEKOF7MLRQZ2HS7FSXV3U6WNI7YQLWP3W4PTFZLMAWQQREBTKQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.192449502 (486h30m16.743919953s) [noise=true , address=145.79.227.14:4839 , piece_id=RDZDYLJRCVHDKTTCAF3JB5ULPKFMSC2E4HNLB2RPLPY2GTUC6ZDQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.193091886 (486h30m16.744562347s) [noise=true , address=145.79.227.12:4767 , piece_id=5Q27HBFWOOI6PIIIMXELWG3QYSEA5B5LAOMU5K4FICMUMPDZIQAA, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.193463874 (486h30m16.744934335s) [noise=true , address=145.79.227.39:4013 , piece_id=YFEKNDHNI77M7MG76EISOGVU32UOZ536CK4X3Z6SFGOLJV2E7EHQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.195162279 (486h30m16.74663274s) [noise=true , address=145.79.227.20:4438 , piece_id=VZK574G5G2HXO5JMVMMEKGFCOJKOTU5KUIJDDEROFXRYTBTITO3A, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.197119445 (486h30m16.748589896s) [noise=true , address=145.79.227.7:4295 , piece_id=OLB3PUIQ4RYD2GGVF4I32FEGXO2O4X56PLJMOKNTJVBK53KZ3VYQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.197903478 (486h30m16.749373929s) [noise=true , address=145.79.227.32:4836 , piece_id=4KBSAAZLHC7OYLWN7RBDTQZUUTMK4HFGFUBES7K2PDKBXZST3OBA, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:19.999395950 (486h30m18.550866411s) [upload=428397]: Next returned nil segment Nov 29 14:34:19.999415177 (486h30m18.550885648s) [upload=428397]: Waiting for error group managing segments... Nov 29 14:34:20.285725942 (486h30m18.837196404s) [noise=true , address=77.90.31.13:4627 , piece_id=62A76IX5BE7WFBCCEMPSENCE5PWOSEMCUIKEATUPWGSLTCFJ4BGQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:20.363243793 (486h30m18.914714244s) [noise=true , address=45.91.116.5:4297 , piece_id=GD7HY7SUS6XPN6KJDC5AVRP2ECTJSB2MK4YBQXCNUOTUBVKFRGVQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:22.551752465 (486h30m21.103222926s) [noise=true , address=145.79.227.31:4798 , piece_id=NG2JWZTD5NPFNDIVWYSWFHQHEDTURKJOY6YZ7G6LAQRIDI7WD3FQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:22.666545505 (486h30m21.218015967s) [noise=true , address=145.79.227.22:4509 , piece_id=N2AEKOF7MLRQZ2HS7FSXV3U6WNI7YQLWP3W4PTFZLMAWQQREBTKQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:23.006352209 (486h30m21.557822671s) [noise=true , address=145.79.227.14:4839 , piece_id=RDZDYLJRCVHDKTTCAF3JB5ULPKFMSC2E4HNLB2RPLPY2GTUC6ZDQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:23.023700063 (486h30m21.575170525s) [noise=true , address=145.79.227.33:4869 , piece_id=V3JJCDM7EBRRZL2GPDMSCQAEHP2YX2QXDTDZDT5M6FGVLB27G7PQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:23.426873130 (486h30m21.978343581s) [noise=true , address=145.79.227.32:4836 , piece_id=4KBSAAZLHC7OYLWN7RBDTQZUUTMK4HFGFUBES7K2PDKBXZST3OBA, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:23.722460741 (486h30m22.273931202s) [noise=true , address=145.79.227.40:4049 , piece_id=DIBJ3Y4NA3HRAZD7GLO72STJHS5XHA3OTCAMWL4VCHMEILLCL3UA, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:25.698494096 (486h30m24.249964557s) [noise=true , address=145.79.227.39:4013 , piece_id=YFEKNDHNI77M7MG76EISOGVU32UOZ536CK4X3Z6SFGOLJV2E7EHQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:25.822411189 (486h30m24.373881651s) [noise=true , address=145.79.227.2:4114 , piece_id=ZB7V2MM6KI6VTSGSC4ZD4QI6EA2265OP3MVLWD2AYAHFERAUE7CQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:29.722449985 (486h30m28.273920436s) [noise=true , address=145.79.227.12:4767 , piece_id=5Q27HBFWOOI6PIIIMXELWG3QYSEA5B5LAOMU5K4FICMUMPDZIQAA, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:47.238199596 (486h30m45.789670047s) [noise=true , address=145.79.227.20:4438 , piece_id=VZK574G5G2HXO5JMVMMEKGFCOJKOTU5KUIJDDEROFXRYTBTITO3A, seg_pos={0 0}, upload=428397]: Done uploading piece. err: ecclient: upload failed (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, address:145.79.227.20:4438): protocol: write tcp 188.166.225.170:34106->145.79.227.20:4438: use of closed network connection; write tcp 188.166.225.170:34106->145.79.227.20:4438: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34106->145.79.227.20:4438: use of closed network connection Nov 29 14:34:47.238464621 (486h30m45.789935082s) [noise=true , address=145.79.227.21:4473 , piece_id=S2V3HONDHED6NZSIFZOTWU2DQ6FKESGGZ4EVMZX36LI4NT32WDWQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:46118->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:46118->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46118->145.79.227.21:4473: use of closed network connection Nov 29 14:34:47.238703104 (486h30m45.790173565s) [noise=true , address=145.79.227.24:4583 , piece_id=B6GTB3UF3XX64M5U57FFCFAMWNWDJUDPIZDG23KOHXOVCFTGKCLA, seg_pos={0 0}, upload=428397]: Done uploading piece. err: ecclient: upload failed (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, address:145.79.227.24:4583): protocol: write tcp 188.166.225.170:51236->145.79.227.24:4583: use of closed network connection; write tcp 188.166.225.170:51236->145.79.227.24:4583: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51236->145.79.227.24:4583: use of closed network connection Nov 29 14:34:47.238869100 (486h30m45.790339561s) [noise=true , address=145.79.227.7:4295 , piece_id=OLB3PUIQ4RYD2GGVF4I32FEGXO2O4X56PLJMOKNTJVBK53KZ3VYQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:145.79.227.7:4295): protocol: write tcp 188.166.225.170:46306->145.79.227.7:4295: use of closed network connection; write tcp 188.166.225.170:46306->145.79.227.7:4295: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46306->145.79.227.7:4295: use of closed network connection Nov 29 14:34:47.239032171 (486h30m45.790502632s) [noise=true , address=145.79.227.6:4259 , piece_id=KIYGYFEAYWG2NUAQZNWG4ZLMRY6GFJADI5LD635JE5M4DLHTX4TQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:145.79.227.6:4259): protocol: write tcp 188.166.225.170:48184->145.79.227.6:4259: use of closed network connection; write tcp 188.166.225.170:48184->145.79.227.6:4259: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48184->145.79.227.6:4259: use of closed network connection Nov 29 14:34:47.239076555 (486h30m45.790547027s) [seg_pos={0 0}, upload=428397]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 29 14:34:47.481897829 (486h30m46.0333683s) [upload=428397]: Done uploading segments... Nov 29 14:34:47.481943356 (486h30m46.033413817s) [upload=428397]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.98596571s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [............................X..............................X] duration: 5.001434133s trace: https://jaeger.dev.storj.io/trace/1e2e0cb4b27ac060 FAIL (2025-11-29 15:02:34 UTC) orbiter warning: context deadline exceeded Duration: 5.001460543s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 5.974099143s trace: https://jaeger.dev.storj.io/trace/4add906a25eb3f89 FAIL (2025-11-29 15:02:15 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Nov 29 14:34:17.941236229 (486h30m16.49270669s) [upload=428397]: Got next segment. Inline: false Nov 29 14:34:17.941289010 (486h30m16.492759471s) [upload=428397]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 29 14:34:17.941296063 (486h30m16.492766524s) [upload=428397]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 29 14:34:18.189160800 (486h30m16.740631261s) [seg_pos={0 0}, upload=428397]: Begin upload segment... Nov 29 14:34:18.189218760 (486h30m16.740689221s) [seg_pos={0 0}, upload=428397]: Done begin upload segment. Nov 29 14:34:18.189254097 (486h30m16.740724548s) [noise=true , address=145.79.227.40:4049 , piece_id=DIBJ3Y4NA3HRAZD7GLO72STJHS5XHA3OTCAMWL4VCHMEILLCL3UA, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.189472724 (486h30m16.740943185s) [noise=true , address=77.90.31.13:4627 , piece_id=62A76IX5BE7WFBCCEMPSENCE5PWOSEMCUIKEATUPWGSLTCFJ4BGQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.190037069 (486h30m16.74150753s) [noise=true , address=145.79.227.2:4114 , piece_id=ZB7V2MM6KI6VTSGSC4ZD4QI6EA2265OP3MVLWD2AYAHFERAUE7CQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.190276184 (486h30m16.741746645s) [noise=true , address=145.79.227.21:4473 , piece_id=S2V3HONDHED6NZSIFZOTWU2DQ6FKESGGZ4EVMZX36LI4NT32WDWQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.190487406 (486h30m16.741957867s) [noise=true , address=145.79.227.31:4798 , piece_id=NG2JWZTD5NPFNDIVWYSWFHQHEDTURKJOY6YZ7G6LAQRIDI7WD3FQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.190825751 (486h30m16.742296212s) [noise=true , address=45.91.116.5:4297 , piece_id=GD7HY7SUS6XPN6KJDC5AVRP2ECTJSB2MK4YBQXCNUOTUBVKFRGVQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.190967721 (486h30m16.742438192s) [noise=true , address=145.79.227.24:4583 , piece_id=B6GTB3UF3XX64M5U57FFCFAMWNWDJUDPIZDG23KOHXOVCFTGKCLA, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.191308490 (486h30m16.742778951s) [noise=true , address=145.79.227.33:4869 , piece_id=V3JJCDM7EBRRZL2GPDMSCQAEHP2YX2QXDTDZDT5M6FGVLB27G7PQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.191365077 (486h30m16.742835528s) [noise=true , address=145.79.227.6:4259 , piece_id=KIYGYFEAYWG2NUAQZNWG4ZLMRY6GFJADI5LD635JE5M4DLHTX4TQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.191782502 (486h30m16.743252963s) [noise=true , address=145.79.227.22:4509 , piece_id=N2AEKOF7MLRQZ2HS7FSXV3U6WNI7YQLWP3W4PTFZLMAWQQREBTKQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.192449502 (486h30m16.743919953s) [noise=true , address=145.79.227.14:4839 , piece_id=RDZDYLJRCVHDKTTCAF3JB5ULPKFMSC2E4HNLB2RPLPY2GTUC6ZDQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.193091886 (486h30m16.744562347s) [noise=true , address=145.79.227.12:4767 , piece_id=5Q27HBFWOOI6PIIIMXELWG3QYSEA5B5LAOMU5K4FICMUMPDZIQAA, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.193463874 (486h30m16.744934335s) [noise=true , address=145.79.227.39:4013 , piece_id=YFEKNDHNI77M7MG76EISOGVU32UOZ536CK4X3Z6SFGOLJV2E7EHQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.195162279 (486h30m16.74663274s) [noise=true , address=145.79.227.20:4438 , piece_id=VZK574G5G2HXO5JMVMMEKGFCOJKOTU5KUIJDDEROFXRYTBTITO3A, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.197119445 (486h30m16.748589896s) [noise=true , address=145.79.227.7:4295 , piece_id=OLB3PUIQ4RYD2GGVF4I32FEGXO2O4X56PLJMOKNTJVBK53KZ3VYQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.197903478 (486h30m16.749373929s) [noise=true , address=145.79.227.32:4836 , piece_id=4KBSAAZLHC7OYLWN7RBDTQZUUTMK4HFGFUBES7K2PDKBXZST3OBA, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:19.999395950 (486h30m18.550866411s) [upload=428397]: Next returned nil segment Nov 29 14:34:19.999415177 (486h30m18.550885648s) [upload=428397]: Waiting for error group managing segments... Nov 29 14:34:20.285725942 (486h30m18.837196404s) [noise=true , address=77.90.31.13:4627 , piece_id=62A76IX5BE7WFBCCEMPSENCE5PWOSEMCUIKEATUPWGSLTCFJ4BGQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:20.363243793 (486h30m18.914714244s) [noise=true , address=45.91.116.5:4297 , piece_id=GD7HY7SUS6XPN6KJDC5AVRP2ECTJSB2MK4YBQXCNUOTUBVKFRGVQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:22.551752465 (486h30m21.103222926s) [noise=true , address=145.79.227.31:4798 , piece_id=NG2JWZTD5NPFNDIVWYSWFHQHEDTURKJOY6YZ7G6LAQRIDI7WD3FQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:22.666545505 (486h30m21.218015967s) [noise=true , address=145.79.227.22:4509 , piece_id=N2AEKOF7MLRQZ2HS7FSXV3U6WNI7YQLWP3W4PTFZLMAWQQREBTKQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:23.006352209 (486h30m21.557822671s) [noise=true , address=145.79.227.14:4839 , piece_id=RDZDYLJRCVHDKTTCAF3JB5ULPKFMSC2E4HNLB2RPLPY2GTUC6ZDQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:23.023700063 (486h30m21.575170525s) [noise=true , address=145.79.227.33:4869 , piece_id=V3JJCDM7EBRRZL2GPDMSCQAEHP2YX2QXDTDZDT5M6FGVLB27G7PQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:23.426873130 (486h30m21.978343581s) [noise=true , address=145.79.227.32:4836 , piece_id=4KBSAAZLHC7OYLWN7RBDTQZUUTMK4HFGFUBES7K2PDKBXZST3OBA, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:23.722460741 (486h30m22.273931202s) [noise=true , address=145.79.227.40:4049 , piece_id=DIBJ3Y4NA3HRAZD7GLO72STJHS5XHA3OTCAMWL4VCHMEILLCL3UA, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:25.698494096 (486h30m24.249964557s) [noise=true , address=145.79.227.39:4013 , piece_id=YFEKNDHNI77M7MG76EISOGVU32UOZ536CK4X3Z6SFGOLJV2E7EHQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:25.822411189 (486h30m24.373881651s) [noise=true , address=145.79.227.2:4114 , piece_id=ZB7V2MM6KI6VTSGSC4ZD4QI6EA2265OP3MVLWD2AYAHFERAUE7CQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:29.722449985 (486h30m28.273920436s) [noise=true , address=145.79.227.12:4767 , piece_id=5Q27HBFWOOI6PIIIMXELWG3QYSEA5B5LAOMU5K4FICMUMPDZIQAA, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:47.238199596 (486h30m45.789670047s) [noise=true , address=145.79.227.20:4438 , piece_id=VZK574G5G2HXO5JMVMMEKGFCOJKOTU5KUIJDDEROFXRYTBTITO3A, seg_pos={0 0}, upload=428397]: Done uploading piece. err: ecclient: upload failed (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, address:145.79.227.20:4438): protocol: write tcp 188.166.225.170:34106->145.79.227.20:4438: use of closed network connection; write tcp 188.166.225.170:34106->145.79.227.20:4438: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34106->145.79.227.20:4438: use of closed network connection Nov 29 14:34:47.238464621 (486h30m45.789935082s) [noise=true , address=145.79.227.21:4473 , piece_id=S2V3HONDHED6NZSIFZOTWU2DQ6FKESGGZ4EVMZX36LI4NT32WDWQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:46118->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:46118->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46118->145.79.227.21:4473: use of closed network connection Nov 29 14:34:47.238703104 (486h30m45.790173565s) [noise=true , address=145.79.227.24:4583 , piece_id=B6GTB3UF3XX64M5U57FFCFAMWNWDJUDPIZDG23KOHXOVCFTGKCLA, seg_pos={0 0}, upload=428397]: Done uploading piece. err: ecclient: upload failed (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, address:145.79.227.24:4583): protocol: write tcp 188.166.225.170:51236->145.79.227.24:4583: use of closed network connection; write tcp 188.166.225.170:51236->145.79.227.24:4583: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51236->145.79.227.24:4583: use of closed network connection Nov 29 14:34:47.238869100 (486h30m45.790339561s) [noise=true , address=145.79.227.7:4295 , piece_id=OLB3PUIQ4RYD2GGVF4I32FEGXO2O4X56PLJMOKNTJVBK53KZ3VYQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:145.79.227.7:4295): protocol: write tcp 188.166.225.170:46306->145.79.227.7:4295: use of closed network connection; write tcp 188.166.225.170:46306->145.79.227.7:4295: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46306->145.79.227.7:4295: use of closed network connection Nov 29 14:34:47.239032171 (486h30m45.790502632s) [noise=true , address=145.79.227.6:4259 , piece_id=KIYGYFEAYWG2NUAQZNWG4ZLMRY6GFJADI5LD635JE5M4DLHTX4TQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:145.79.227.6:4259): protocol: write tcp 188.166.225.170:48184->145.79.227.6:4259: use of closed network connection; write tcp 188.166.225.170:48184->145.79.227.6:4259: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48184->145.79.227.6:4259: use of closed network connection Nov 29 14:34:47.239076555 (486h30m45.790547027s) [seg_pos={0 0}, upload=428397]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 29 14:34:47.481897829 (486h30m46.0333683s) [upload=428397]: Done uploading segments... Nov 29 14:34:47.481943356 (486h30m46.033413817s) [upload=428397]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.974130421s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.............................X..............................] duration: 3.517112028s trace: https://jaeger.dev.storj.io/trace/275e9b01e747ebf4 FAIL (2025-11-29 15:02:08 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.517414964s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..............................X.............................] duration: 3.525645199s trace: https://jaeger.dev.storj.io/trace/275e9b01e747ebf4 FAIL (2025-11-29 15:01:34 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.525673432s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 5.825442803s trace: https://jaeger.dev.storj.io/trace/4add906a25eb3f89 FAIL (2025-11-29 15:01:17 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Nov 29 14:34:17.941236229 (486h30m16.49270669s) [upload=428397]: Got next segment. Inline: false Nov 29 14:34:17.941289010 (486h30m16.492759471s) [upload=428397]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 29 14:34:17.941296063 (486h30m16.492766524s) [upload=428397]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 29 14:34:18.189160800 (486h30m16.740631261s) [seg_pos={0 0}, upload=428397]: Begin upload segment... Nov 29 14:34:18.189218760 (486h30m16.740689221s) [seg_pos={0 0}, upload=428397]: Done begin upload segment. Nov 29 14:34:18.189254097 (486h30m16.740724548s) [noise=true , address=145.79.227.40:4049 , piece_id=DIBJ3Y4NA3HRAZD7GLO72STJHS5XHA3OTCAMWL4VCHMEILLCL3UA, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.189472724 (486h30m16.740943185s) [noise=true , address=77.90.31.13:4627 , piece_id=62A76IX5BE7WFBCCEMPSENCE5PWOSEMCUIKEATUPWGSLTCFJ4BGQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.190037069 (486h30m16.74150753s) [noise=true , address=145.79.227.2:4114 , piece_id=ZB7V2MM6KI6VTSGSC4ZD4QI6EA2265OP3MVLWD2AYAHFERAUE7CQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.190276184 (486h30m16.741746645s) [noise=true , address=145.79.227.21:4473 , piece_id=S2V3HONDHED6NZSIFZOTWU2DQ6FKESGGZ4EVMZX36LI4NT32WDWQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.190487406 (486h30m16.741957867s) [noise=true , address=145.79.227.31:4798 , piece_id=NG2JWZTD5NPFNDIVWYSWFHQHEDTURKJOY6YZ7G6LAQRIDI7WD3FQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.190825751 (486h30m16.742296212s) [noise=true , address=45.91.116.5:4297 , piece_id=GD7HY7SUS6XPN6KJDC5AVRP2ECTJSB2MK4YBQXCNUOTUBVKFRGVQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.190967721 (486h30m16.742438192s) [noise=true , address=145.79.227.24:4583 , piece_id=B6GTB3UF3XX64M5U57FFCFAMWNWDJUDPIZDG23KOHXOVCFTGKCLA, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.191308490 (486h30m16.742778951s) [noise=true , address=145.79.227.33:4869 , piece_id=V3JJCDM7EBRRZL2GPDMSCQAEHP2YX2QXDTDZDT5M6FGVLB27G7PQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.191365077 (486h30m16.742835528s) [noise=true , address=145.79.227.6:4259 , piece_id=KIYGYFEAYWG2NUAQZNWG4ZLMRY6GFJADI5LD635JE5M4DLHTX4TQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.191782502 (486h30m16.743252963s) [noise=true , address=145.79.227.22:4509 , piece_id=N2AEKOF7MLRQZ2HS7FSXV3U6WNI7YQLWP3W4PTFZLMAWQQREBTKQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.192449502 (486h30m16.743919953s) [noise=true , address=145.79.227.14:4839 , piece_id=RDZDYLJRCVHDKTTCAF3JB5ULPKFMSC2E4HNLB2RPLPY2GTUC6ZDQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.193091886 (486h30m16.744562347s) [noise=true , address=145.79.227.12:4767 , piece_id=5Q27HBFWOOI6PIIIMXELWG3QYSEA5B5LAOMU5K4FICMUMPDZIQAA, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.193463874 (486h30m16.744934335s) [noise=true , address=145.79.227.39:4013 , piece_id=YFEKNDHNI77M7MG76EISOGVU32UOZ536CK4X3Z6SFGOLJV2E7EHQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.195162279 (486h30m16.74663274s) [noise=true , address=145.79.227.20:4438 , piece_id=VZK574G5G2HXO5JMVMMEKGFCOJKOTU5KUIJDDEROFXRYTBTITO3A, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.197119445 (486h30m16.748589896s) [noise=true , address=145.79.227.7:4295 , piece_id=OLB3PUIQ4RYD2GGVF4I32FEGXO2O4X56PLJMOKNTJVBK53KZ3VYQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.197903478 (486h30m16.749373929s) [noise=true , address=145.79.227.32:4836 , piece_id=4KBSAAZLHC7OYLWN7RBDTQZUUTMK4HFGFUBES7K2PDKBXZST3OBA, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:19.999395950 (486h30m18.550866411s) [upload=428397]: Next returned nil segment Nov 29 14:34:19.999415177 (486h30m18.550885648s) [upload=428397]: Waiting for error group managing segments... Nov 29 14:34:20.285725942 (486h30m18.837196404s) [noise=true , address=77.90.31.13:4627 , piece_id=62A76IX5BE7WFBCCEMPSENCE5PWOSEMCUIKEATUPWGSLTCFJ4BGQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:20.363243793 (486h30m18.914714244s) [noise=true , address=45.91.116.5:4297 , piece_id=GD7HY7SUS6XPN6KJDC5AVRP2ECTJSB2MK4YBQXCNUOTUBVKFRGVQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:22.551752465 (486h30m21.103222926s) [noise=true , address=145.79.227.31:4798 , piece_id=NG2JWZTD5NPFNDIVWYSWFHQHEDTURKJOY6YZ7G6LAQRIDI7WD3FQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:22.666545505 (486h30m21.218015967s) [noise=true , address=145.79.227.22:4509 , piece_id=N2AEKOF7MLRQZ2HS7FSXV3U6WNI7YQLWP3W4PTFZLMAWQQREBTKQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:23.006352209 (486h30m21.557822671s) [noise=true , address=145.79.227.14:4839 , piece_id=RDZDYLJRCVHDKTTCAF3JB5ULPKFMSC2E4HNLB2RPLPY2GTUC6ZDQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:23.023700063 (486h30m21.575170525s) [noise=true , address=145.79.227.33:4869 , piece_id=V3JJCDM7EBRRZL2GPDMSCQAEHP2YX2QXDTDZDT5M6FGVLB27G7PQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:23.426873130 (486h30m21.978343581s) [noise=true , address=145.79.227.32:4836 , piece_id=4KBSAAZLHC7OYLWN7RBDTQZUUTMK4HFGFUBES7K2PDKBXZST3OBA, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:23.722460741 (486h30m22.273931202s) [noise=true , address=145.79.227.40:4049 , piece_id=DIBJ3Y4NA3HRAZD7GLO72STJHS5XHA3OTCAMWL4VCHMEILLCL3UA, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:25.698494096 (486h30m24.249964557s) [noise=true , address=145.79.227.39:4013 , piece_id=YFEKNDHNI77M7MG76EISOGVU32UOZ536CK4X3Z6SFGOLJV2E7EHQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:25.822411189 (486h30m24.373881651s) [noise=true , address=145.79.227.2:4114 , piece_id=ZB7V2MM6KI6VTSGSC4ZD4QI6EA2265OP3MVLWD2AYAHFERAUE7CQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:29.722449985 (486h30m28.273920436s) [noise=true , address=145.79.227.12:4767 , piece_id=5Q27HBFWOOI6PIIIMXELWG3QYSEA5B5LAOMU5K4FICMUMPDZIQAA, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:47.238199596 (486h30m45.789670047s) [noise=true , address=145.79.227.20:4438 , piece_id=VZK574G5G2HXO5JMVMMEKGFCOJKOTU5KUIJDDEROFXRYTBTITO3A, seg_pos={0 0}, upload=428397]: Done uploading piece. err: ecclient: upload failed (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, address:145.79.227.20:4438): protocol: write tcp 188.166.225.170:34106->145.79.227.20:4438: use of closed network connection; write tcp 188.166.225.170:34106->145.79.227.20:4438: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34106->145.79.227.20:4438: use of closed network connection Nov 29 14:34:47.238464621 (486h30m45.789935082s) [noise=true , address=145.79.227.21:4473 , piece_id=S2V3HONDHED6NZSIFZOTWU2DQ6FKESGGZ4EVMZX36LI4NT32WDWQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:46118->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:46118->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46118->145.79.227.21:4473: use of closed network connection Nov 29 14:34:47.238703104 (486h30m45.790173565s) [noise=true , address=145.79.227.24:4583 , piece_id=B6GTB3UF3XX64M5U57FFCFAMWNWDJUDPIZDG23KOHXOVCFTGKCLA, seg_pos={0 0}, upload=428397]: Done uploading piece. err: ecclient: upload failed (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, address:145.79.227.24:4583): protocol: write tcp 188.166.225.170:51236->145.79.227.24:4583: use of closed network connection; write tcp 188.166.225.170:51236->145.79.227.24:4583: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51236->145.79.227.24:4583: use of closed network connection Nov 29 14:34:47.238869100 (486h30m45.790339561s) [noise=true , address=145.79.227.7:4295 , piece_id=OLB3PUIQ4RYD2GGVF4I32FEGXO2O4X56PLJMOKNTJVBK53KZ3VYQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:145.79.227.7:4295): protocol: write tcp 188.166.225.170:46306->145.79.227.7:4295: use of closed network connection; write tcp 188.166.225.170:46306->145.79.227.7:4295: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46306->145.79.227.7:4295: use of closed network connection Nov 29 14:34:47.239032171 (486h30m45.790502632s) [noise=true , address=145.79.227.6:4259 , piece_id=KIYGYFEAYWG2NUAQZNWG4ZLMRY6GFJADI5LD635JE5M4DLHTX4TQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:145.79.227.6:4259): protocol: write tcp 188.166.225.170:48184->145.79.227.6:4259: use of closed network connection; write tcp 188.166.225.170:48184->145.79.227.6:4259: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48184->145.79.227.6:4259: use of closed network connection Nov 29 14:34:47.239076555 (486h30m45.790547027s) [seg_pos={0 0}, upload=428397]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 29 14:34:47.481897829 (486h30m46.0333683s) [upload=428397]: Done uploading segments... Nov 29 14:34:47.481943356 (486h30m46.033413817s) [upload=428397]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.825474804s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...............................X............................] duration: 3.413173728s trace: https://jaeger.dev.storj.io/trace/275e9b01e747ebf4 FAIL (2025-11-29 15:00:27 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.413204026s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [................................X...........................] duration: 3.444564032s trace: https://jaeger.dev.storj.io/trace/275e9b01e747ebf4 FAIL (2025-11-29 15:00:07 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.444594361s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 6.07494179s trace: https://jaeger.dev.storj.io/trace/4add906a25eb3f89 FAIL (2025-11-29 15:00:04 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Nov 29 14:34:17.941236229 (486h30m16.49270669s) [upload=428397]: Got next segment. Inline: false Nov 29 14:34:17.941289010 (486h30m16.492759471s) [upload=428397]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 29 14:34:17.941296063 (486h30m16.492766524s) [upload=428397]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 29 14:34:18.189160800 (486h30m16.740631261s) [seg_pos={0 0}, upload=428397]: Begin upload segment... Nov 29 14:34:18.189218760 (486h30m16.740689221s) [seg_pos={0 0}, upload=428397]: Done begin upload segment. Nov 29 14:34:18.189254097 (486h30m16.740724548s) [noise=true , address=145.79.227.40:4049 , piece_id=DIBJ3Y4NA3HRAZD7GLO72STJHS5XHA3OTCAMWL4VCHMEILLCL3UA, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.189472724 (486h30m16.740943185s) [noise=true , address=77.90.31.13:4627 , piece_id=62A76IX5BE7WFBCCEMPSENCE5PWOSEMCUIKEATUPWGSLTCFJ4BGQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.190037069 (486h30m16.74150753s) [noise=true , address=145.79.227.2:4114 , piece_id=ZB7V2MM6KI6VTSGSC4ZD4QI6EA2265OP3MVLWD2AYAHFERAUE7CQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.190276184 (486h30m16.741746645s) [noise=true , address=145.79.227.21:4473 , piece_id=S2V3HONDHED6NZSIFZOTWU2DQ6FKESGGZ4EVMZX36LI4NT32WDWQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.190487406 (486h30m16.741957867s) [noise=true , address=145.79.227.31:4798 , piece_id=NG2JWZTD5NPFNDIVWYSWFHQHEDTURKJOY6YZ7G6LAQRIDI7WD3FQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.190825751 (486h30m16.742296212s) [noise=true , address=45.91.116.5:4297 , piece_id=GD7HY7SUS6XPN6KJDC5AVRP2ECTJSB2MK4YBQXCNUOTUBVKFRGVQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.190967721 (486h30m16.742438192s) [noise=true , address=145.79.227.24:4583 , piece_id=B6GTB3UF3XX64M5U57FFCFAMWNWDJUDPIZDG23KOHXOVCFTGKCLA, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.191308490 (486h30m16.742778951s) [noise=true , address=145.79.227.33:4869 , piece_id=V3JJCDM7EBRRZL2GPDMSCQAEHP2YX2QXDTDZDT5M6FGVLB27G7PQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.191365077 (486h30m16.742835528s) [noise=true , address=145.79.227.6:4259 , piece_id=KIYGYFEAYWG2NUAQZNWG4ZLMRY6GFJADI5LD635JE5M4DLHTX4TQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.191782502 (486h30m16.743252963s) [noise=true , address=145.79.227.22:4509 , piece_id=N2AEKOF7MLRQZ2HS7FSXV3U6WNI7YQLWP3W4PTFZLMAWQQREBTKQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.192449502 (486h30m16.743919953s) [noise=true , address=145.79.227.14:4839 , piece_id=RDZDYLJRCVHDKTTCAF3JB5ULPKFMSC2E4HNLB2RPLPY2GTUC6ZDQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.193091886 (486h30m16.744562347s) [noise=true , address=145.79.227.12:4767 , piece_id=5Q27HBFWOOI6PIIIMXELWG3QYSEA5B5LAOMU5K4FICMUMPDZIQAA, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.193463874 (486h30m16.744934335s) [noise=true , address=145.79.227.39:4013 , piece_id=YFEKNDHNI77M7MG76EISOGVU32UOZ536CK4X3Z6SFGOLJV2E7EHQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.195162279 (486h30m16.74663274s) [noise=true , address=145.79.227.20:4438 , piece_id=VZK574G5G2HXO5JMVMMEKGFCOJKOTU5KUIJDDEROFXRYTBTITO3A, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.197119445 (486h30m16.748589896s) [noise=true , address=145.79.227.7:4295 , piece_id=OLB3PUIQ4RYD2GGVF4I32FEGXO2O4X56PLJMOKNTJVBK53KZ3VYQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.197903478 (486h30m16.749373929s) [noise=true , address=145.79.227.32:4836 , piece_id=4KBSAAZLHC7OYLWN7RBDTQZUUTMK4HFGFUBES7K2PDKBXZST3OBA, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:19.999395950 (486h30m18.550866411s) [upload=428397]: Next returned nil segment Nov 29 14:34:19.999415177 (486h30m18.550885648s) [upload=428397]: Waiting for error group managing segments... Nov 29 14:34:20.285725942 (486h30m18.837196404s) [noise=true , address=77.90.31.13:4627 , piece_id=62A76IX5BE7WFBCCEMPSENCE5PWOSEMCUIKEATUPWGSLTCFJ4BGQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:20.363243793 (486h30m18.914714244s) [noise=true , address=45.91.116.5:4297 , piece_id=GD7HY7SUS6XPN6KJDC5AVRP2ECTJSB2MK4YBQXCNUOTUBVKFRGVQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:22.551752465 (486h30m21.103222926s) [noise=true , address=145.79.227.31:4798 , piece_id=NG2JWZTD5NPFNDIVWYSWFHQHEDTURKJOY6YZ7G6LAQRIDI7WD3FQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:22.666545505 (486h30m21.218015967s) [noise=true , address=145.79.227.22:4509 , piece_id=N2AEKOF7MLRQZ2HS7FSXV3U6WNI7YQLWP3W4PTFZLMAWQQREBTKQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:23.006352209 (486h30m21.557822671s) [noise=true , address=145.79.227.14:4839 , piece_id=RDZDYLJRCVHDKTTCAF3JB5ULPKFMSC2E4HNLB2RPLPY2GTUC6ZDQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:23.023700063 (486h30m21.575170525s) [noise=true , address=145.79.227.33:4869 , piece_id=V3JJCDM7EBRRZL2GPDMSCQAEHP2YX2QXDTDZDT5M6FGVLB27G7PQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:23.426873130 (486h30m21.978343581s) [noise=true , address=145.79.227.32:4836 , piece_id=4KBSAAZLHC7OYLWN7RBDTQZUUTMK4HFGFUBES7K2PDKBXZST3OBA, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:23.722460741 (486h30m22.273931202s) [noise=true , address=145.79.227.40:4049 , piece_id=DIBJ3Y4NA3HRAZD7GLO72STJHS5XHA3OTCAMWL4VCHMEILLCL3UA, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:25.698494096 (486h30m24.249964557s) [noise=true , address=145.79.227.39:4013 , piece_id=YFEKNDHNI77M7MG76EISOGVU32UOZ536CK4X3Z6SFGOLJV2E7EHQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:25.822411189 (486h30m24.373881651s) [noise=true , address=145.79.227.2:4114 , piece_id=ZB7V2MM6KI6VTSGSC4ZD4QI6EA2265OP3MVLWD2AYAHFERAUE7CQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:29.722449985 (486h30m28.273920436s) [noise=true , address=145.79.227.12:4767 , piece_id=5Q27HBFWOOI6PIIIMXELWG3QYSEA5B5LAOMU5K4FICMUMPDZIQAA, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:47.238199596 (486h30m45.789670047s) [noise=true , address=145.79.227.20:4438 , piece_id=VZK574G5G2HXO5JMVMMEKGFCOJKOTU5KUIJDDEROFXRYTBTITO3A, seg_pos={0 0}, upload=428397]: Done uploading piece. err: ecclient: upload failed (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, address:145.79.227.20:4438): protocol: write tcp 188.166.225.170:34106->145.79.227.20:4438: use of closed network connection; write tcp 188.166.225.170:34106->145.79.227.20:4438: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34106->145.79.227.20:4438: use of closed network connection Nov 29 14:34:47.238464621 (486h30m45.789935082s) [noise=true , address=145.79.227.21:4473 , piece_id=S2V3HONDHED6NZSIFZOTWU2DQ6FKESGGZ4EVMZX36LI4NT32WDWQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:46118->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:46118->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46118->145.79.227.21:4473: use of closed network connection Nov 29 14:34:47.238703104 (486h30m45.790173565s) [noise=true , address=145.79.227.24:4583 , piece_id=B6GTB3UF3XX64M5U57FFCFAMWNWDJUDPIZDG23KOHXOVCFTGKCLA, seg_pos={0 0}, upload=428397]: Done uploading piece. err: ecclient: upload failed (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, address:145.79.227.24:4583): protocol: write tcp 188.166.225.170:51236->145.79.227.24:4583: use of closed network connection; write tcp 188.166.225.170:51236->145.79.227.24:4583: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51236->145.79.227.24:4583: use of closed network connection Nov 29 14:34:47.238869100 (486h30m45.790339561s) [noise=true , address=145.79.227.7:4295 , piece_id=OLB3PUIQ4RYD2GGVF4I32FEGXO2O4X56PLJMOKNTJVBK53KZ3VYQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:145.79.227.7:4295): protocol: write tcp 188.166.225.170:46306->145.79.227.7:4295: use of closed network connection; write tcp 188.166.225.170:46306->145.79.227.7:4295: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46306->145.79.227.7:4295: use of closed network connection Nov 29 14:34:47.239032171 (486h30m45.790502632s) [noise=true , address=145.79.227.6:4259 , piece_id=KIYGYFEAYWG2NUAQZNWG4ZLMRY6GFJADI5LD635JE5M4DLHTX4TQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:145.79.227.6:4259): protocol: write tcp 188.166.225.170:48184->145.79.227.6:4259: use of closed network connection; write tcp 188.166.225.170:48184->145.79.227.6:4259: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48184->145.79.227.6:4259: use of closed network connection Nov 29 14:34:47.239076555 (486h30m45.790547027s) [seg_pos={0 0}, upload=428397]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 29 14:34:47.481897829 (486h30m46.0333683s) [upload=428397]: Done uploading segments... Nov 29 14:34:47.481943356 (486h30m46.033413817s) [upload=428397]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 6.074977877s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [X................................X..........................] duration: 3.358914196s trace: https://jaeger.dev.storj.io/trace/275e9b01e747ebf4 FAIL (2025-11-29 14:59:51 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.35900649s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 5.756350721s trace: https://jaeger.dev.storj.io/trace/4add906a25eb3f89 FAIL (2025-11-29 14:59:39 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Nov 29 14:34:17.941236229 (486h30m16.49270669s) [upload=428397]: Got next segment. Inline: false Nov 29 14:34:17.941289010 (486h30m16.492759471s) [upload=428397]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 29 14:34:17.941296063 (486h30m16.492766524s) [upload=428397]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 29 14:34:18.189160800 (486h30m16.740631261s) [seg_pos={0 0}, upload=428397]: Begin upload segment... Nov 29 14:34:18.189218760 (486h30m16.740689221s) [seg_pos={0 0}, upload=428397]: Done begin upload segment. Nov 29 14:34:18.189254097 (486h30m16.740724548s) [noise=true , address=145.79.227.40:4049 , piece_id=DIBJ3Y4NA3HRAZD7GLO72STJHS5XHA3OTCAMWL4VCHMEILLCL3UA, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.189472724 (486h30m16.740943185s) [noise=true , address=77.90.31.13:4627 , piece_id=62A76IX5BE7WFBCCEMPSENCE5PWOSEMCUIKEATUPWGSLTCFJ4BGQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.190037069 (486h30m16.74150753s) [noise=true , address=145.79.227.2:4114 , piece_id=ZB7V2MM6KI6VTSGSC4ZD4QI6EA2265OP3MVLWD2AYAHFERAUE7CQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.190276184 (486h30m16.741746645s) [noise=true , address=145.79.227.21:4473 , piece_id=S2V3HONDHED6NZSIFZOTWU2DQ6FKESGGZ4EVMZX36LI4NT32WDWQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.190487406 (486h30m16.741957867s) [noise=true , address=145.79.227.31:4798 , piece_id=NG2JWZTD5NPFNDIVWYSWFHQHEDTURKJOY6YZ7G6LAQRIDI7WD3FQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.190825751 (486h30m16.742296212s) [noise=true , address=45.91.116.5:4297 , piece_id=GD7HY7SUS6XPN6KJDC5AVRP2ECTJSB2MK4YBQXCNUOTUBVKFRGVQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.190967721 (486h30m16.742438192s) [noise=true , address=145.79.227.24:4583 , piece_id=B6GTB3UF3XX64M5U57FFCFAMWNWDJUDPIZDG23KOHXOVCFTGKCLA, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.191308490 (486h30m16.742778951s) [noise=true , address=145.79.227.33:4869 , piece_id=V3JJCDM7EBRRZL2GPDMSCQAEHP2YX2QXDTDZDT5M6FGVLB27G7PQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.191365077 (486h30m16.742835528s) [noise=true , address=145.79.227.6:4259 , piece_id=KIYGYFEAYWG2NUAQZNWG4ZLMRY6GFJADI5LD635JE5M4DLHTX4TQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.191782502 (486h30m16.743252963s) [noise=true , address=145.79.227.22:4509 , piece_id=N2AEKOF7MLRQZ2HS7FSXV3U6WNI7YQLWP3W4PTFZLMAWQQREBTKQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.192449502 (486h30m16.743919953s) [noise=true , address=145.79.227.14:4839 , piece_id=RDZDYLJRCVHDKTTCAF3JB5ULPKFMSC2E4HNLB2RPLPY2GTUC6ZDQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.193091886 (486h30m16.744562347s) [noise=true , address=145.79.227.12:4767 , piece_id=5Q27HBFWOOI6PIIIMXELWG3QYSEA5B5LAOMU5K4FICMUMPDZIQAA, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.193463874 (486h30m16.744934335s) [noise=true , address=145.79.227.39:4013 , piece_id=YFEKNDHNI77M7MG76EISOGVU32UOZ536CK4X3Z6SFGOLJV2E7EHQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.195162279 (486h30m16.74663274s) [noise=true , address=145.79.227.20:4438 , piece_id=VZK574G5G2HXO5JMVMMEKGFCOJKOTU5KUIJDDEROFXRYTBTITO3A, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.197119445 (486h30m16.748589896s) [noise=true , address=145.79.227.7:4295 , piece_id=OLB3PUIQ4RYD2GGVF4I32FEGXO2O4X56PLJMOKNTJVBK53KZ3VYQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.197903478 (486h30m16.749373929s) [noise=true , address=145.79.227.32:4836 , piece_id=4KBSAAZLHC7OYLWN7RBDTQZUUTMK4HFGFUBES7K2PDKBXZST3OBA, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:19.999395950 (486h30m18.550866411s) [upload=428397]: Next returned nil segment Nov 29 14:34:19.999415177 (486h30m18.550885648s) [upload=428397]: Waiting for error group managing segments... Nov 29 14:34:20.285725942 (486h30m18.837196404s) [noise=true , address=77.90.31.13:4627 , piece_id=62A76IX5BE7WFBCCEMPSENCE5PWOSEMCUIKEATUPWGSLTCFJ4BGQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:20.363243793 (486h30m18.914714244s) [noise=true , address=45.91.116.5:4297 , piece_id=GD7HY7SUS6XPN6KJDC5AVRP2ECTJSB2MK4YBQXCNUOTUBVKFRGVQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:22.551752465 (486h30m21.103222926s) [noise=true , address=145.79.227.31:4798 , piece_id=NG2JWZTD5NPFNDIVWYSWFHQHEDTURKJOY6YZ7G6LAQRIDI7WD3FQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:22.666545505 (486h30m21.218015967s) [noise=true , address=145.79.227.22:4509 , piece_id=N2AEKOF7MLRQZ2HS7FSXV3U6WNI7YQLWP3W4PTFZLMAWQQREBTKQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:23.006352209 (486h30m21.557822671s) [noise=true , address=145.79.227.14:4839 , piece_id=RDZDYLJRCVHDKTTCAF3JB5ULPKFMSC2E4HNLB2RPLPY2GTUC6ZDQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:23.023700063 (486h30m21.575170525s) [noise=true , address=145.79.227.33:4869 , piece_id=V3JJCDM7EBRRZL2GPDMSCQAEHP2YX2QXDTDZDT5M6FGVLB27G7PQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:23.426873130 (486h30m21.978343581s) [noise=true , address=145.79.227.32:4836 , piece_id=4KBSAAZLHC7OYLWN7RBDTQZUUTMK4HFGFUBES7K2PDKBXZST3OBA, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:23.722460741 (486h30m22.273931202s) [noise=true , address=145.79.227.40:4049 , piece_id=DIBJ3Y4NA3HRAZD7GLO72STJHS5XHA3OTCAMWL4VCHMEILLCL3UA, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:25.698494096 (486h30m24.249964557s) [noise=true , address=145.79.227.39:4013 , piece_id=YFEKNDHNI77M7MG76EISOGVU32UOZ536CK4X3Z6SFGOLJV2E7EHQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:25.822411189 (486h30m24.373881651s) [noise=true , address=145.79.227.2:4114 , piece_id=ZB7V2MM6KI6VTSGSC4ZD4QI6EA2265OP3MVLWD2AYAHFERAUE7CQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:29.722449985 (486h30m28.273920436s) [noise=true , address=145.79.227.12:4767 , piece_id=5Q27HBFWOOI6PIIIMXELWG3QYSEA5B5LAOMU5K4FICMUMPDZIQAA, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:47.238199596 (486h30m45.789670047s) [noise=true , address=145.79.227.20:4438 , piece_id=VZK574G5G2HXO5JMVMMEKGFCOJKOTU5KUIJDDEROFXRYTBTITO3A, seg_pos={0 0}, upload=428397]: Done uploading piece. err: ecclient: upload failed (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, address:145.79.227.20:4438): protocol: write tcp 188.166.225.170:34106->145.79.227.20:4438: use of closed network connection; write tcp 188.166.225.170:34106->145.79.227.20:4438: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34106->145.79.227.20:4438: use of closed network connection Nov 29 14:34:47.238464621 (486h30m45.789935082s) [noise=true , address=145.79.227.21:4473 , piece_id=S2V3HONDHED6NZSIFZOTWU2DQ6FKESGGZ4EVMZX36LI4NT32WDWQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:46118->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:46118->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46118->145.79.227.21:4473: use of closed network connection Nov 29 14:34:47.238703104 (486h30m45.790173565s) [noise=true , address=145.79.227.24:4583 , piece_id=B6GTB3UF3XX64M5U57FFCFAMWNWDJUDPIZDG23KOHXOVCFTGKCLA, seg_pos={0 0}, upload=428397]: Done uploading piece. err: ecclient: upload failed (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, address:145.79.227.24:4583): protocol: write tcp 188.166.225.170:51236->145.79.227.24:4583: use of closed network connection; write tcp 188.166.225.170:51236->145.79.227.24:4583: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51236->145.79.227.24:4583: use of closed network connection Nov 29 14:34:47.238869100 (486h30m45.790339561s) [noise=true , address=145.79.227.7:4295 , piece_id=OLB3PUIQ4RYD2GGVF4I32FEGXO2O4X56PLJMOKNTJVBK53KZ3VYQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:145.79.227.7:4295): protocol: write tcp 188.166.225.170:46306->145.79.227.7:4295: use of closed network connection; write tcp 188.166.225.170:46306->145.79.227.7:4295: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46306->145.79.227.7:4295: use of closed network connection Nov 29 14:34:47.239032171 (486h30m45.790502632s) [noise=true , address=145.79.227.6:4259 , piece_id=KIYGYFEAYWG2NUAQZNWG4ZLMRY6GFJADI5LD635JE5M4DLHTX4TQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:145.79.227.6:4259): protocol: write tcp 188.166.225.170:48184->145.79.227.6:4259: use of closed network connection; write tcp 188.166.225.170:48184->145.79.227.6:4259: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48184->145.79.227.6:4259: use of closed network connection Nov 29 14:34:47.239076555 (486h30m45.790547027s) [seg_pos={0 0}, upload=428397]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 29 14:34:47.481897829 (486h30m46.0333683s) [upload=428397]: Done uploading segments... Nov 29 14:34:47.481943356 (486h30m46.033413817s) [upload=428397]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.756378104s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.X................................X.........................] duration: 3.426298452s trace: https://jaeger.dev.storj.io/trace/275e9b01e747ebf4 FAIL (2025-11-29 14:58:59 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.426360639s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 6.034708286s trace: https://jaeger.dev.storj.io/trace/4add906a25eb3f89 FAIL (2025-11-29 14:58:34 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Nov 29 14:34:17.941236229 (486h30m16.49270669s) [upload=428397]: Got next segment. Inline: false Nov 29 14:34:17.941289010 (486h30m16.492759471s) [upload=428397]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 29 14:34:17.941296063 (486h30m16.492766524s) [upload=428397]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 29 14:34:18.189160800 (486h30m16.740631261s) [seg_pos={0 0}, upload=428397]: Begin upload segment... Nov 29 14:34:18.189218760 (486h30m16.740689221s) [seg_pos={0 0}, upload=428397]: Done begin upload segment. Nov 29 14:34:18.189254097 (486h30m16.740724548s) [noise=true , address=145.79.227.40:4049 , piece_id=DIBJ3Y4NA3HRAZD7GLO72STJHS5XHA3OTCAMWL4VCHMEILLCL3UA, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.189472724 (486h30m16.740943185s) [noise=true , address=77.90.31.13:4627 , piece_id=62A76IX5BE7WFBCCEMPSENCE5PWOSEMCUIKEATUPWGSLTCFJ4BGQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.190037069 (486h30m16.74150753s) [noise=true , address=145.79.227.2:4114 , piece_id=ZB7V2MM6KI6VTSGSC4ZD4QI6EA2265OP3MVLWD2AYAHFERAUE7CQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.190276184 (486h30m16.741746645s) [noise=true , address=145.79.227.21:4473 , piece_id=S2V3HONDHED6NZSIFZOTWU2DQ6FKESGGZ4EVMZX36LI4NT32WDWQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.190487406 (486h30m16.741957867s) [noise=true , address=145.79.227.31:4798 , piece_id=NG2JWZTD5NPFNDIVWYSWFHQHEDTURKJOY6YZ7G6LAQRIDI7WD3FQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.190825751 (486h30m16.742296212s) [noise=true , address=45.91.116.5:4297 , piece_id=GD7HY7SUS6XPN6KJDC5AVRP2ECTJSB2MK4YBQXCNUOTUBVKFRGVQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.190967721 (486h30m16.742438192s) [noise=true , address=145.79.227.24:4583 , piece_id=B6GTB3UF3XX64M5U57FFCFAMWNWDJUDPIZDG23KOHXOVCFTGKCLA, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.191308490 (486h30m16.742778951s) [noise=true , address=145.79.227.33:4869 , piece_id=V3JJCDM7EBRRZL2GPDMSCQAEHP2YX2QXDTDZDT5M6FGVLB27G7PQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.191365077 (486h30m16.742835528s) [noise=true , address=145.79.227.6:4259 , piece_id=KIYGYFEAYWG2NUAQZNWG4ZLMRY6GFJADI5LD635JE5M4DLHTX4TQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.191782502 (486h30m16.743252963s) [noise=true , address=145.79.227.22:4509 , piece_id=N2AEKOF7MLRQZ2HS7FSXV3U6WNI7YQLWP3W4PTFZLMAWQQREBTKQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.192449502 (486h30m16.743919953s) [noise=true , address=145.79.227.14:4839 , piece_id=RDZDYLJRCVHDKTTCAF3JB5ULPKFMSC2E4HNLB2RPLPY2GTUC6ZDQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.193091886 (486h30m16.744562347s) [noise=true , address=145.79.227.12:4767 , piece_id=5Q27HBFWOOI6PIIIMXELWG3QYSEA5B5LAOMU5K4FICMUMPDZIQAA, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.193463874 (486h30m16.744934335s) [noise=true , address=145.79.227.39:4013 , piece_id=YFEKNDHNI77M7MG76EISOGVU32UOZ536CK4X3Z6SFGOLJV2E7EHQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.195162279 (486h30m16.74663274s) [noise=true , address=145.79.227.20:4438 , piece_id=VZK574G5G2HXO5JMVMMEKGFCOJKOTU5KUIJDDEROFXRYTBTITO3A, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.197119445 (486h30m16.748589896s) [noise=true , address=145.79.227.7:4295 , piece_id=OLB3PUIQ4RYD2GGVF4I32FEGXO2O4X56PLJMOKNTJVBK53KZ3VYQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.197903478 (486h30m16.749373929s) [noise=true , address=145.79.227.32:4836 , piece_id=4KBSAAZLHC7OYLWN7RBDTQZUUTMK4HFGFUBES7K2PDKBXZST3OBA, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:19.999395950 (486h30m18.550866411s) [upload=428397]: Next returned nil segment Nov 29 14:34:19.999415177 (486h30m18.550885648s) [upload=428397]: Waiting for error group managing segments... Nov 29 14:34:20.285725942 (486h30m18.837196404s) [noise=true , address=77.90.31.13:4627 , piece_id=62A76IX5BE7WFBCCEMPSENCE5PWOSEMCUIKEATUPWGSLTCFJ4BGQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:20.363243793 (486h30m18.914714244s) [noise=true , address=45.91.116.5:4297 , piece_id=GD7HY7SUS6XPN6KJDC5AVRP2ECTJSB2MK4YBQXCNUOTUBVKFRGVQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:22.551752465 (486h30m21.103222926s) [noise=true , address=145.79.227.31:4798 , piece_id=NG2JWZTD5NPFNDIVWYSWFHQHEDTURKJOY6YZ7G6LAQRIDI7WD3FQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:22.666545505 (486h30m21.218015967s) [noise=true , address=145.79.227.22:4509 , piece_id=N2AEKOF7MLRQZ2HS7FSXV3U6WNI7YQLWP3W4PTFZLMAWQQREBTKQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:23.006352209 (486h30m21.557822671s) [noise=true , address=145.79.227.14:4839 , piece_id=RDZDYLJRCVHDKTTCAF3JB5ULPKFMSC2E4HNLB2RPLPY2GTUC6ZDQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:23.023700063 (486h30m21.575170525s) [noise=true , address=145.79.227.33:4869 , piece_id=V3JJCDM7EBRRZL2GPDMSCQAEHP2YX2QXDTDZDT5M6FGVLB27G7PQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:23.426873130 (486h30m21.978343581s) [noise=true , address=145.79.227.32:4836 , piece_id=4KBSAAZLHC7OYLWN7RBDTQZUUTMK4HFGFUBES7K2PDKBXZST3OBA, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:23.722460741 (486h30m22.273931202s) [noise=true , address=145.79.227.40:4049 , piece_id=DIBJ3Y4NA3HRAZD7GLO72STJHS5XHA3OTCAMWL4VCHMEILLCL3UA, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:25.698494096 (486h30m24.249964557s) [noise=true , address=145.79.227.39:4013 , piece_id=YFEKNDHNI77M7MG76EISOGVU32UOZ536CK4X3Z6SFGOLJV2E7EHQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:25.822411189 (486h30m24.373881651s) [noise=true , address=145.79.227.2:4114 , piece_id=ZB7V2MM6KI6VTSGSC4ZD4QI6EA2265OP3MVLWD2AYAHFERAUE7CQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:29.722449985 (486h30m28.273920436s) [noise=true , address=145.79.227.12:4767 , piece_id=5Q27HBFWOOI6PIIIMXELWG3QYSEA5B5LAOMU5K4FICMUMPDZIQAA, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:47.238199596 (486h30m45.789670047s) [noise=true , address=145.79.227.20:4438 , piece_id=VZK574G5G2HXO5JMVMMEKGFCOJKOTU5KUIJDDEROFXRYTBTITO3A, seg_pos={0 0}, upload=428397]: Done uploading piece. err: ecclient: upload failed (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, address:145.79.227.20:4438): protocol: write tcp 188.166.225.170:34106->145.79.227.20:4438: use of closed network connection; write tcp 188.166.225.170:34106->145.79.227.20:4438: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34106->145.79.227.20:4438: use of closed network connection Nov 29 14:34:47.238464621 (486h30m45.789935082s) [noise=true , address=145.79.227.21:4473 , piece_id=S2V3HONDHED6NZSIFZOTWU2DQ6FKESGGZ4EVMZX36LI4NT32WDWQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:46118->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:46118->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46118->145.79.227.21:4473: use of closed network connection Nov 29 14:34:47.238703104 (486h30m45.790173565s) [noise=true , address=145.79.227.24:4583 , piece_id=B6GTB3UF3XX64M5U57FFCFAMWNWDJUDPIZDG23KOHXOVCFTGKCLA, seg_pos={0 0}, upload=428397]: Done uploading piece. err: ecclient: upload failed (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, address:145.79.227.24:4583): protocol: write tcp 188.166.225.170:51236->145.79.227.24:4583: use of closed network connection; write tcp 188.166.225.170:51236->145.79.227.24:4583: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51236->145.79.227.24:4583: use of closed network connection Nov 29 14:34:47.238869100 (486h30m45.790339561s) [noise=true , address=145.79.227.7:4295 , piece_id=OLB3PUIQ4RYD2GGVF4I32FEGXO2O4X56PLJMOKNTJVBK53KZ3VYQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:145.79.227.7:4295): protocol: write tcp 188.166.225.170:46306->145.79.227.7:4295: use of closed network connection; write tcp 188.166.225.170:46306->145.79.227.7:4295: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46306->145.79.227.7:4295: use of closed network connection Nov 29 14:34:47.239032171 (486h30m45.790502632s) [noise=true , address=145.79.227.6:4259 , piece_id=KIYGYFEAYWG2NUAQZNWG4ZLMRY6GFJADI5LD635JE5M4DLHTX4TQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:145.79.227.6:4259): protocol: write tcp 188.166.225.170:48184->145.79.227.6:4259: use of closed network connection; write tcp 188.166.225.170:48184->145.79.227.6:4259: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48184->145.79.227.6:4259: use of closed network connection Nov 29 14:34:47.239076555 (486h30m45.790547027s) [seg_pos={0 0}, upload=428397]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 29 14:34:47.481897829 (486h30m46.0333683s) [upload=428397]: Done uploading segments... Nov 29 14:34:47.481943356 (486h30m46.033413817s) [upload=428397]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 6.034738824s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [..X................................X........................] duration: 3.328336833s trace: https://jaeger.dev.storj.io/trace/275e9b01e747ebf4 FAIL (2025-11-29 14:57:47 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.328365958s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 5.786146006s trace: https://jaeger.dev.storj.io/trace/4add906a25eb3f89 FAIL (2025-11-29 14:57:32 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Nov 29 14:34:17.941236229 (486h30m16.49270669s) [upload=428397]: Got next segment. Inline: false Nov 29 14:34:17.941289010 (486h30m16.492759471s) [upload=428397]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 29 14:34:17.941296063 (486h30m16.492766524s) [upload=428397]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 29 14:34:18.189160800 (486h30m16.740631261s) [seg_pos={0 0}, upload=428397]: Begin upload segment... Nov 29 14:34:18.189218760 (486h30m16.740689221s) [seg_pos={0 0}, upload=428397]: Done begin upload segment. Nov 29 14:34:18.189254097 (486h30m16.740724548s) [noise=true , address=145.79.227.40:4049 , piece_id=DIBJ3Y4NA3HRAZD7GLO72STJHS5XHA3OTCAMWL4VCHMEILLCL3UA, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.189472724 (486h30m16.740943185s) [noise=true , address=77.90.31.13:4627 , piece_id=62A76IX5BE7WFBCCEMPSENCE5PWOSEMCUIKEATUPWGSLTCFJ4BGQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.190037069 (486h30m16.74150753s) [noise=true , address=145.79.227.2:4114 , piece_id=ZB7V2MM6KI6VTSGSC4ZD4QI6EA2265OP3MVLWD2AYAHFERAUE7CQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.190276184 (486h30m16.741746645s) [noise=true , address=145.79.227.21:4473 , piece_id=S2V3HONDHED6NZSIFZOTWU2DQ6FKESGGZ4EVMZX36LI4NT32WDWQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.190487406 (486h30m16.741957867s) [noise=true , address=145.79.227.31:4798 , piece_id=NG2JWZTD5NPFNDIVWYSWFHQHEDTURKJOY6YZ7G6LAQRIDI7WD3FQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.190825751 (486h30m16.742296212s) [noise=true , address=45.91.116.5:4297 , piece_id=GD7HY7SUS6XPN6KJDC5AVRP2ECTJSB2MK4YBQXCNUOTUBVKFRGVQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.190967721 (486h30m16.742438192s) [noise=true , address=145.79.227.24:4583 , piece_id=B6GTB3UF3XX64M5U57FFCFAMWNWDJUDPIZDG23KOHXOVCFTGKCLA, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.191308490 (486h30m16.742778951s) [noise=true , address=145.79.227.33:4869 , piece_id=V3JJCDM7EBRRZL2GPDMSCQAEHP2YX2QXDTDZDT5M6FGVLB27G7PQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.191365077 (486h30m16.742835528s) [noise=true , address=145.79.227.6:4259 , piece_id=KIYGYFEAYWG2NUAQZNWG4ZLMRY6GFJADI5LD635JE5M4DLHTX4TQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.191782502 (486h30m16.743252963s) [noise=true , address=145.79.227.22:4509 , piece_id=N2AEKOF7MLRQZ2HS7FSXV3U6WNI7YQLWP3W4PTFZLMAWQQREBTKQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.192449502 (486h30m16.743919953s) [noise=true , address=145.79.227.14:4839 , piece_id=RDZDYLJRCVHDKTTCAF3JB5ULPKFMSC2E4HNLB2RPLPY2GTUC6ZDQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.193091886 (486h30m16.744562347s) [noise=true , address=145.79.227.12:4767 , piece_id=5Q27HBFWOOI6PIIIMXELWG3QYSEA5B5LAOMU5K4FICMUMPDZIQAA, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.193463874 (486h30m16.744934335s) [noise=true , address=145.79.227.39:4013 , piece_id=YFEKNDHNI77M7MG76EISOGVU32UOZ536CK4X3Z6SFGOLJV2E7EHQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.195162279 (486h30m16.74663274s) [noise=true , address=145.79.227.20:4438 , piece_id=VZK574G5G2HXO5JMVMMEKGFCOJKOTU5KUIJDDEROFXRYTBTITO3A, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.197119445 (486h30m16.748589896s) [noise=true , address=145.79.227.7:4295 , piece_id=OLB3PUIQ4RYD2GGVF4I32FEGXO2O4X56PLJMOKNTJVBK53KZ3VYQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.197903478 (486h30m16.749373929s) [noise=true , address=145.79.227.32:4836 , piece_id=4KBSAAZLHC7OYLWN7RBDTQZUUTMK4HFGFUBES7K2PDKBXZST3OBA, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:19.999395950 (486h30m18.550866411s) [upload=428397]: Next returned nil segment Nov 29 14:34:19.999415177 (486h30m18.550885648s) [upload=428397]: Waiting for error group managing segments... Nov 29 14:34:20.285725942 (486h30m18.837196404s) [noise=true , address=77.90.31.13:4627 , piece_id=62A76IX5BE7WFBCCEMPSENCE5PWOSEMCUIKEATUPWGSLTCFJ4BGQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:20.363243793 (486h30m18.914714244s) [noise=true , address=45.91.116.5:4297 , piece_id=GD7HY7SUS6XPN6KJDC5AVRP2ECTJSB2MK4YBQXCNUOTUBVKFRGVQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:22.551752465 (486h30m21.103222926s) [noise=true , address=145.79.227.31:4798 , piece_id=NG2JWZTD5NPFNDIVWYSWFHQHEDTURKJOY6YZ7G6LAQRIDI7WD3FQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:22.666545505 (486h30m21.218015967s) [noise=true , address=145.79.227.22:4509 , piece_id=N2AEKOF7MLRQZ2HS7FSXV3U6WNI7YQLWP3W4PTFZLMAWQQREBTKQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:23.006352209 (486h30m21.557822671s) [noise=true , address=145.79.227.14:4839 , piece_id=RDZDYLJRCVHDKTTCAF3JB5ULPKFMSC2E4HNLB2RPLPY2GTUC6ZDQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:23.023700063 (486h30m21.575170525s) [noise=true , address=145.79.227.33:4869 , piece_id=V3JJCDM7EBRRZL2GPDMSCQAEHP2YX2QXDTDZDT5M6FGVLB27G7PQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:23.426873130 (486h30m21.978343581s) [noise=true , address=145.79.227.32:4836 , piece_id=4KBSAAZLHC7OYLWN7RBDTQZUUTMK4HFGFUBES7K2PDKBXZST3OBA, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:23.722460741 (486h30m22.273931202s) [noise=true , address=145.79.227.40:4049 , piece_id=DIBJ3Y4NA3HRAZD7GLO72STJHS5XHA3OTCAMWL4VCHMEILLCL3UA, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:25.698494096 (486h30m24.249964557s) [noise=true , address=145.79.227.39:4013 , piece_id=YFEKNDHNI77M7MG76EISOGVU32UOZ536CK4X3Z6SFGOLJV2E7EHQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:25.822411189 (486h30m24.373881651s) [noise=true , address=145.79.227.2:4114 , piece_id=ZB7V2MM6KI6VTSGSC4ZD4QI6EA2265OP3MVLWD2AYAHFERAUE7CQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:29.722449985 (486h30m28.273920436s) [noise=true , address=145.79.227.12:4767 , piece_id=5Q27HBFWOOI6PIIIMXELWG3QYSEA5B5LAOMU5K4FICMUMPDZIQAA, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:47.238199596 (486h30m45.789670047s) [noise=true , address=145.79.227.20:4438 , piece_id=VZK574G5G2HXO5JMVMMEKGFCOJKOTU5KUIJDDEROFXRYTBTITO3A, seg_pos={0 0}, upload=428397]: Done uploading piece. err: ecclient: upload failed (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, address:145.79.227.20:4438): protocol: write tcp 188.166.225.170:34106->145.79.227.20:4438: use of closed network connection; write tcp 188.166.225.170:34106->145.79.227.20:4438: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34106->145.79.227.20:4438: use of closed network connection Nov 29 14:34:47.238464621 (486h30m45.789935082s) [noise=true , address=145.79.227.21:4473 , piece_id=S2V3HONDHED6NZSIFZOTWU2DQ6FKESGGZ4EVMZX36LI4NT32WDWQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:46118->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:46118->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46118->145.79.227.21:4473: use of closed network connection Nov 29 14:34:47.238703104 (486h30m45.790173565s) [noise=true , address=145.79.227.24:4583 , piece_id=B6GTB3UF3XX64M5U57FFCFAMWNWDJUDPIZDG23KOHXOVCFTGKCLA, seg_pos={0 0}, upload=428397]: Done uploading piece. err: ecclient: upload failed (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, address:145.79.227.24:4583): protocol: write tcp 188.166.225.170:51236->145.79.227.24:4583: use of closed network connection; write tcp 188.166.225.170:51236->145.79.227.24:4583: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51236->145.79.227.24:4583: use of closed network connection Nov 29 14:34:47.238869100 (486h30m45.790339561s) [noise=true , address=145.79.227.7:4295 , piece_id=OLB3PUIQ4RYD2GGVF4I32FEGXO2O4X56PLJMOKNTJVBK53KZ3VYQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:145.79.227.7:4295): protocol: write tcp 188.166.225.170:46306->145.79.227.7:4295: use of closed network connection; write tcp 188.166.225.170:46306->145.79.227.7:4295: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46306->145.79.227.7:4295: use of closed network connection Nov 29 14:34:47.239032171 (486h30m45.790502632s) [noise=true , address=145.79.227.6:4259 , piece_id=KIYGYFEAYWG2NUAQZNWG4ZLMRY6GFJADI5LD635JE5M4DLHTX4TQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:145.79.227.6:4259): protocol: write tcp 188.166.225.170:48184->145.79.227.6:4259: use of closed network connection; write tcp 188.166.225.170:48184->145.79.227.6:4259: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48184->145.79.227.6:4259: use of closed network connection Nov 29 14:34:47.239076555 (486h30m45.790547027s) [seg_pos={0 0}, upload=428397]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 29 14:34:47.481897829 (486h30m46.0333683s) [upload=428397]: Done uploading segments... Nov 29 14:34:47.481943356 (486h30m46.033413817s) [upload=428397]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.786196942s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 5.726270359s trace: https://jaeger.dev.storj.io/trace/4add906a25eb3f89 FAIL (2025-11-29 14:57:13 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Nov 29 14:34:17.941236229 (486h30m16.49270669s) [upload=428397]: Got next segment. Inline: false Nov 29 14:34:17.941289010 (486h30m16.492759471s) [upload=428397]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 29 14:34:17.941296063 (486h30m16.492766524s) [upload=428397]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 29 14:34:18.189160800 (486h30m16.740631261s) [seg_pos={0 0}, upload=428397]: Begin upload segment... Nov 29 14:34:18.189218760 (486h30m16.740689221s) [seg_pos={0 0}, upload=428397]: Done begin upload segment. Nov 29 14:34:18.189254097 (486h30m16.740724548s) [noise=true , address=145.79.227.40:4049 , piece_id=DIBJ3Y4NA3HRAZD7GLO72STJHS5XHA3OTCAMWL4VCHMEILLCL3UA, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.189472724 (486h30m16.740943185s) [noise=true , address=77.90.31.13:4627 , piece_id=62A76IX5BE7WFBCCEMPSENCE5PWOSEMCUIKEATUPWGSLTCFJ4BGQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.190037069 (486h30m16.74150753s) [noise=true , address=145.79.227.2:4114 , piece_id=ZB7V2MM6KI6VTSGSC4ZD4QI6EA2265OP3MVLWD2AYAHFERAUE7CQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.190276184 (486h30m16.741746645s) [noise=true , address=145.79.227.21:4473 , piece_id=S2V3HONDHED6NZSIFZOTWU2DQ6FKESGGZ4EVMZX36LI4NT32WDWQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.190487406 (486h30m16.741957867s) [noise=true , address=145.79.227.31:4798 , piece_id=NG2JWZTD5NPFNDIVWYSWFHQHEDTURKJOY6YZ7G6LAQRIDI7WD3FQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.190825751 (486h30m16.742296212s) [noise=true , address=45.91.116.5:4297 , piece_id=GD7HY7SUS6XPN6KJDC5AVRP2ECTJSB2MK4YBQXCNUOTUBVKFRGVQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.190967721 (486h30m16.742438192s) [noise=true , address=145.79.227.24:4583 , piece_id=B6GTB3UF3XX64M5U57FFCFAMWNWDJUDPIZDG23KOHXOVCFTGKCLA, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.191308490 (486h30m16.742778951s) [noise=true , address=145.79.227.33:4869 , piece_id=V3JJCDM7EBRRZL2GPDMSCQAEHP2YX2QXDTDZDT5M6FGVLB27G7PQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.191365077 (486h30m16.742835528s) [noise=true , address=145.79.227.6:4259 , piece_id=KIYGYFEAYWG2NUAQZNWG4ZLMRY6GFJADI5LD635JE5M4DLHTX4TQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.191782502 (486h30m16.743252963s) [noise=true , address=145.79.227.22:4509 , piece_id=N2AEKOF7MLRQZ2HS7FSXV3U6WNI7YQLWP3W4PTFZLMAWQQREBTKQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.192449502 (486h30m16.743919953s) [noise=true , address=145.79.227.14:4839 , piece_id=RDZDYLJRCVHDKTTCAF3JB5ULPKFMSC2E4HNLB2RPLPY2GTUC6ZDQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.193091886 (486h30m16.744562347s) [noise=true , address=145.79.227.12:4767 , piece_id=5Q27HBFWOOI6PIIIMXELWG3QYSEA5B5LAOMU5K4FICMUMPDZIQAA, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.193463874 (486h30m16.744934335s) [noise=true , address=145.79.227.39:4013 , piece_id=YFEKNDHNI77M7MG76EISOGVU32UOZ536CK4X3Z6SFGOLJV2E7EHQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.195162279 (486h30m16.74663274s) [noise=true , address=145.79.227.20:4438 , piece_id=VZK574G5G2HXO5JMVMMEKGFCOJKOTU5KUIJDDEROFXRYTBTITO3A, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.197119445 (486h30m16.748589896s) [noise=true , address=145.79.227.7:4295 , piece_id=OLB3PUIQ4RYD2GGVF4I32FEGXO2O4X56PLJMOKNTJVBK53KZ3VYQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.197903478 (486h30m16.749373929s) [noise=true , address=145.79.227.32:4836 , piece_id=4KBSAAZLHC7OYLWN7RBDTQZUUTMK4HFGFUBES7K2PDKBXZST3OBA, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:19.999395950 (486h30m18.550866411s) [upload=428397]: Next returned nil segment Nov 29 14:34:19.999415177 (486h30m18.550885648s) [upload=428397]: Waiting for error group managing segments... Nov 29 14:34:20.285725942 (486h30m18.837196404s) [noise=true , address=77.90.31.13:4627 , piece_id=62A76IX5BE7WFBCCEMPSENCE5PWOSEMCUIKEATUPWGSLTCFJ4BGQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:20.363243793 (486h30m18.914714244s) [noise=true , address=45.91.116.5:4297 , piece_id=GD7HY7SUS6XPN6KJDC5AVRP2ECTJSB2MK4YBQXCNUOTUBVKFRGVQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:22.551752465 (486h30m21.103222926s) [noise=true , address=145.79.227.31:4798 , piece_id=NG2JWZTD5NPFNDIVWYSWFHQHEDTURKJOY6YZ7G6LAQRIDI7WD3FQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:22.666545505 (486h30m21.218015967s) [noise=true , address=145.79.227.22:4509 , piece_id=N2AEKOF7MLRQZ2HS7FSXV3U6WNI7YQLWP3W4PTFZLMAWQQREBTKQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:23.006352209 (486h30m21.557822671s) [noise=true , address=145.79.227.14:4839 , piece_id=RDZDYLJRCVHDKTTCAF3JB5ULPKFMSC2E4HNLB2RPLPY2GTUC6ZDQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:23.023700063 (486h30m21.575170525s) [noise=true , address=145.79.227.33:4869 , piece_id=V3JJCDM7EBRRZL2GPDMSCQAEHP2YX2QXDTDZDT5M6FGVLB27G7PQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:23.426873130 (486h30m21.978343581s) [noise=true , address=145.79.227.32:4836 , piece_id=4KBSAAZLHC7OYLWN7RBDTQZUUTMK4HFGFUBES7K2PDKBXZST3OBA, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:23.722460741 (486h30m22.273931202s) [noise=true , address=145.79.227.40:4049 , piece_id=DIBJ3Y4NA3HRAZD7GLO72STJHS5XHA3OTCAMWL4VCHMEILLCL3UA, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:25.698494096 (486h30m24.249964557s) [noise=true , address=145.79.227.39:4013 , piece_id=YFEKNDHNI77M7MG76EISOGVU32UOZ536CK4X3Z6SFGOLJV2E7EHQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:25.822411189 (486h30m24.373881651s) [noise=true , address=145.79.227.2:4114 , piece_id=ZB7V2MM6KI6VTSGSC4ZD4QI6EA2265OP3MVLWD2AYAHFERAUE7CQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:29.722449985 (486h30m28.273920436s) [noise=true , address=145.79.227.12:4767 , piece_id=5Q27HBFWOOI6PIIIMXELWG3QYSEA5B5LAOMU5K4FICMUMPDZIQAA, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:47.238199596 (486h30m45.789670047s) [noise=true , address=145.79.227.20:4438 , piece_id=VZK574G5G2HXO5JMVMMEKGFCOJKOTU5KUIJDDEROFXRYTBTITO3A, seg_pos={0 0}, upload=428397]: Done uploading piece. err: ecclient: upload failed (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, address:145.79.227.20:4438): protocol: write tcp 188.166.225.170:34106->145.79.227.20:4438: use of closed network connection; write tcp 188.166.225.170:34106->145.79.227.20:4438: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34106->145.79.227.20:4438: use of closed network connection Nov 29 14:34:47.238464621 (486h30m45.789935082s) [noise=true , address=145.79.227.21:4473 , piece_id=S2V3HONDHED6NZSIFZOTWU2DQ6FKESGGZ4EVMZX36LI4NT32WDWQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:46118->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:46118->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46118->145.79.227.21:4473: use of closed network connection Nov 29 14:34:47.238703104 (486h30m45.790173565s) [noise=true , address=145.79.227.24:4583 , piece_id=B6GTB3UF3XX64M5U57FFCFAMWNWDJUDPIZDG23KOHXOVCFTGKCLA, seg_pos={0 0}, upload=428397]: Done uploading piece. err: ecclient: upload failed (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, address:145.79.227.24:4583): protocol: write tcp 188.166.225.170:51236->145.79.227.24:4583: use of closed network connection; write tcp 188.166.225.170:51236->145.79.227.24:4583: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51236->145.79.227.24:4583: use of closed network connection Nov 29 14:34:47.238869100 (486h30m45.790339561s) [noise=true , address=145.79.227.7:4295 , piece_id=OLB3PUIQ4RYD2GGVF4I32FEGXO2O4X56PLJMOKNTJVBK53KZ3VYQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:145.79.227.7:4295): protocol: write tcp 188.166.225.170:46306->145.79.227.7:4295: use of closed network connection; write tcp 188.166.225.170:46306->145.79.227.7:4295: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46306->145.79.227.7:4295: use of closed network connection Nov 29 14:34:47.239032171 (486h30m45.790502632s) [noise=true , address=145.79.227.6:4259 , piece_id=KIYGYFEAYWG2NUAQZNWG4ZLMRY6GFJADI5LD635JE5M4DLHTX4TQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:145.79.227.6:4259): protocol: write tcp 188.166.225.170:48184->145.79.227.6:4259: use of closed network connection; write tcp 188.166.225.170:48184->145.79.227.6:4259: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48184->145.79.227.6:4259: use of closed network connection Nov 29 14:34:47.239076555 (486h30m45.790547027s) [seg_pos={0 0}, upload=428397]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 29 14:34:47.481897829 (486h30m46.0333683s) [upload=428397]: Done uploading segments... Nov 29 14:34:47.481943356 (486h30m46.033413817s) [upload=428397]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.726388655s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 5.962866455s trace: https://jaeger.dev.storj.io/trace/4add906a25eb3f89 FAIL (2025-11-29 14:56:43 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Nov 29 14:34:17.941236229 (486h30m16.49270669s) [upload=428397]: Got next segment. Inline: false Nov 29 14:34:17.941289010 (486h30m16.492759471s) [upload=428397]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 29 14:34:17.941296063 (486h30m16.492766524s) [upload=428397]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 29 14:34:18.189160800 (486h30m16.740631261s) [seg_pos={0 0}, upload=428397]: Begin upload segment... Nov 29 14:34:18.189218760 (486h30m16.740689221s) [seg_pos={0 0}, upload=428397]: Done begin upload segment. Nov 29 14:34:18.189254097 (486h30m16.740724548s) [noise=true , address=145.79.227.40:4049 , piece_id=DIBJ3Y4NA3HRAZD7GLO72STJHS5XHA3OTCAMWL4VCHMEILLCL3UA, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.189472724 (486h30m16.740943185s) [noise=true , address=77.90.31.13:4627 , piece_id=62A76IX5BE7WFBCCEMPSENCE5PWOSEMCUIKEATUPWGSLTCFJ4BGQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.190037069 (486h30m16.74150753s) [noise=true , address=145.79.227.2:4114 , piece_id=ZB7V2MM6KI6VTSGSC4ZD4QI6EA2265OP3MVLWD2AYAHFERAUE7CQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.190276184 (486h30m16.741746645s) [noise=true , address=145.79.227.21:4473 , piece_id=S2V3HONDHED6NZSIFZOTWU2DQ6FKESGGZ4EVMZX36LI4NT32WDWQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.190487406 (486h30m16.741957867s) [noise=true , address=145.79.227.31:4798 , piece_id=NG2JWZTD5NPFNDIVWYSWFHQHEDTURKJOY6YZ7G6LAQRIDI7WD3FQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.190825751 (486h30m16.742296212s) [noise=true , address=45.91.116.5:4297 , piece_id=GD7HY7SUS6XPN6KJDC5AVRP2ECTJSB2MK4YBQXCNUOTUBVKFRGVQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.190967721 (486h30m16.742438192s) [noise=true , address=145.79.227.24:4583 , piece_id=B6GTB3UF3XX64M5U57FFCFAMWNWDJUDPIZDG23KOHXOVCFTGKCLA, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.191308490 (486h30m16.742778951s) [noise=true , address=145.79.227.33:4869 , piece_id=V3JJCDM7EBRRZL2GPDMSCQAEHP2YX2QXDTDZDT5M6FGVLB27G7PQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.191365077 (486h30m16.742835528s) [noise=true , address=145.79.227.6:4259 , piece_id=KIYGYFEAYWG2NUAQZNWG4ZLMRY6GFJADI5LD635JE5M4DLHTX4TQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.191782502 (486h30m16.743252963s) [noise=true , address=145.79.227.22:4509 , piece_id=N2AEKOF7MLRQZ2HS7FSXV3U6WNI7YQLWP3W4PTFZLMAWQQREBTKQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.192449502 (486h30m16.743919953s) [noise=true , address=145.79.227.14:4839 , piece_id=RDZDYLJRCVHDKTTCAF3JB5ULPKFMSC2E4HNLB2RPLPY2GTUC6ZDQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.193091886 (486h30m16.744562347s) [noise=true , address=145.79.227.12:4767 , piece_id=5Q27HBFWOOI6PIIIMXELWG3QYSEA5B5LAOMU5K4FICMUMPDZIQAA, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.193463874 (486h30m16.744934335s) [noise=true , address=145.79.227.39:4013 , piece_id=YFEKNDHNI77M7MG76EISOGVU32UOZ536CK4X3Z6SFGOLJV2E7EHQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.195162279 (486h30m16.74663274s) [noise=true , address=145.79.227.20:4438 , piece_id=VZK574G5G2HXO5JMVMMEKGFCOJKOTU5KUIJDDEROFXRYTBTITO3A, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.197119445 (486h30m16.748589896s) [noise=true , address=145.79.227.7:4295 , piece_id=OLB3PUIQ4RYD2GGVF4I32FEGXO2O4X56PLJMOKNTJVBK53KZ3VYQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.197903478 (486h30m16.749373929s) [noise=true , address=145.79.227.32:4836 , piece_id=4KBSAAZLHC7OYLWN7RBDTQZUUTMK4HFGFUBES7K2PDKBXZST3OBA, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:19.999395950 (486h30m18.550866411s) [upload=428397]: Next returned nil segment Nov 29 14:34:19.999415177 (486h30m18.550885648s) [upload=428397]: Waiting for error group managing segments... Nov 29 14:34:20.285725942 (486h30m18.837196404s) [noise=true , address=77.90.31.13:4627 , piece_id=62A76IX5BE7WFBCCEMPSENCE5PWOSEMCUIKEATUPWGSLTCFJ4BGQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:20.363243793 (486h30m18.914714244s) [noise=true , address=45.91.116.5:4297 , piece_id=GD7HY7SUS6XPN6KJDC5AVRP2ECTJSB2MK4YBQXCNUOTUBVKFRGVQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:22.551752465 (486h30m21.103222926s) [noise=true , address=145.79.227.31:4798 , piece_id=NG2JWZTD5NPFNDIVWYSWFHQHEDTURKJOY6YZ7G6LAQRIDI7WD3FQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:22.666545505 (486h30m21.218015967s) [noise=true , address=145.79.227.22:4509 , piece_id=N2AEKOF7MLRQZ2HS7FSXV3U6WNI7YQLWP3W4PTFZLMAWQQREBTKQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:23.006352209 (486h30m21.557822671s) [noise=true , address=145.79.227.14:4839 , piece_id=RDZDYLJRCVHDKTTCAF3JB5ULPKFMSC2E4HNLB2RPLPY2GTUC6ZDQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:23.023700063 (486h30m21.575170525s) [noise=true , address=145.79.227.33:4869 , piece_id=V3JJCDM7EBRRZL2GPDMSCQAEHP2YX2QXDTDZDT5M6FGVLB27G7PQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:23.426873130 (486h30m21.978343581s) [noise=true , address=145.79.227.32:4836 , piece_id=4KBSAAZLHC7OYLWN7RBDTQZUUTMK4HFGFUBES7K2PDKBXZST3OBA, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:23.722460741 (486h30m22.273931202s) [noise=true , address=145.79.227.40:4049 , piece_id=DIBJ3Y4NA3HRAZD7GLO72STJHS5XHA3OTCAMWL4VCHMEILLCL3UA, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:25.698494096 (486h30m24.249964557s) [noise=true , address=145.79.227.39:4013 , piece_id=YFEKNDHNI77M7MG76EISOGVU32UOZ536CK4X3Z6SFGOLJV2E7EHQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:25.822411189 (486h30m24.373881651s) [noise=true , address=145.79.227.2:4114 , piece_id=ZB7V2MM6KI6VTSGSC4ZD4QI6EA2265OP3MVLWD2AYAHFERAUE7CQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:29.722449985 (486h30m28.273920436s) [noise=true , address=145.79.227.12:4767 , piece_id=5Q27HBFWOOI6PIIIMXELWG3QYSEA5B5LAOMU5K4FICMUMPDZIQAA, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:47.238199596 (486h30m45.789670047s) [noise=true , address=145.79.227.20:4438 , piece_id=VZK574G5G2HXO5JMVMMEKGFCOJKOTU5KUIJDDEROFXRYTBTITO3A, seg_pos={0 0}, upload=428397]: Done uploading piece. err: ecclient: upload failed (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, address:145.79.227.20:4438): protocol: write tcp 188.166.225.170:34106->145.79.227.20:4438: use of closed network connection; write tcp 188.166.225.170:34106->145.79.227.20:4438: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34106->145.79.227.20:4438: use of closed network connection Nov 29 14:34:47.238464621 (486h30m45.789935082s) [noise=true , address=145.79.227.21:4473 , piece_id=S2V3HONDHED6NZSIFZOTWU2DQ6FKESGGZ4EVMZX36LI4NT32WDWQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:46118->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:46118->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46118->145.79.227.21:4473: use of closed network connection Nov 29 14:34:47.238703104 (486h30m45.790173565s) [noise=true , address=145.79.227.24:4583 , piece_id=B6GTB3UF3XX64M5U57FFCFAMWNWDJUDPIZDG23KOHXOVCFTGKCLA, seg_pos={0 0}, upload=428397]: Done uploading piece. err: ecclient: upload failed (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, address:145.79.227.24:4583): protocol: write tcp 188.166.225.170:51236->145.79.227.24:4583: use of closed network connection; write tcp 188.166.225.170:51236->145.79.227.24:4583: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51236->145.79.227.24:4583: use of closed network connection Nov 29 14:34:47.238869100 (486h30m45.790339561s) [noise=true , address=145.79.227.7:4295 , piece_id=OLB3PUIQ4RYD2GGVF4I32FEGXO2O4X56PLJMOKNTJVBK53KZ3VYQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:145.79.227.7:4295): protocol: write tcp 188.166.225.170:46306->145.79.227.7:4295: use of closed network connection; write tcp 188.166.225.170:46306->145.79.227.7:4295: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46306->145.79.227.7:4295: use of closed network connection Nov 29 14:34:47.239032171 (486h30m45.790502632s) [noise=true , address=145.79.227.6:4259 , piece_id=KIYGYFEAYWG2NUAQZNWG4ZLMRY6GFJADI5LD635JE5M4DLHTX4TQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:145.79.227.6:4259): protocol: write tcp 188.166.225.170:48184->145.79.227.6:4259: use of closed network connection; write tcp 188.166.225.170:48184->145.79.227.6:4259: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48184->145.79.227.6:4259: use of closed network connection Nov 29 14:34:47.239076555 (486h30m45.790547027s) [seg_pos={0 0}, upload=428397]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 29 14:34:47.481897829 (486h30m46.0333683s) [upload=428397]: Done uploading segments... Nov 29 14:34:47.481943356 (486h30m46.033413817s) [upload=428397]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.962893455s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [...X................................X.......................] duration: 3.351140057s trace: https://jaeger.dev.storj.io/trace/275e9b01e747ebf4 FAIL (2025-11-29 14:56:41 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.351174242s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 5.729025964s trace: https://jaeger.dev.storj.io/trace/4add906a25eb3f89 FAIL (2025-11-29 14:56:26 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Nov 29 14:34:17.941236229 (486h30m16.49270669s) [upload=428397]: Got next segment. Inline: false Nov 29 14:34:17.941289010 (486h30m16.492759471s) [upload=428397]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 29 14:34:17.941296063 (486h30m16.492766524s) [upload=428397]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 29 14:34:18.189160800 (486h30m16.740631261s) [seg_pos={0 0}, upload=428397]: Begin upload segment... Nov 29 14:34:18.189218760 (486h30m16.740689221s) [seg_pos={0 0}, upload=428397]: Done begin upload segment. Nov 29 14:34:18.189254097 (486h30m16.740724548s) [noise=true , address=145.79.227.40:4049 , piece_id=DIBJ3Y4NA3HRAZD7GLO72STJHS5XHA3OTCAMWL4VCHMEILLCL3UA, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.189472724 (486h30m16.740943185s) [noise=true , address=77.90.31.13:4627 , piece_id=62A76IX5BE7WFBCCEMPSENCE5PWOSEMCUIKEATUPWGSLTCFJ4BGQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.190037069 (486h30m16.74150753s) [noise=true , address=145.79.227.2:4114 , piece_id=ZB7V2MM6KI6VTSGSC4ZD4QI6EA2265OP3MVLWD2AYAHFERAUE7CQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.190276184 (486h30m16.741746645s) [noise=true , address=145.79.227.21:4473 , piece_id=S2V3HONDHED6NZSIFZOTWU2DQ6FKESGGZ4EVMZX36LI4NT32WDWQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.190487406 (486h30m16.741957867s) [noise=true , address=145.79.227.31:4798 , piece_id=NG2JWZTD5NPFNDIVWYSWFHQHEDTURKJOY6YZ7G6LAQRIDI7WD3FQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.190825751 (486h30m16.742296212s) [noise=true , address=45.91.116.5:4297 , piece_id=GD7HY7SUS6XPN6KJDC5AVRP2ECTJSB2MK4YBQXCNUOTUBVKFRGVQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.190967721 (486h30m16.742438192s) [noise=true , address=145.79.227.24:4583 , piece_id=B6GTB3UF3XX64M5U57FFCFAMWNWDJUDPIZDG23KOHXOVCFTGKCLA, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.191308490 (486h30m16.742778951s) [noise=true , address=145.79.227.33:4869 , piece_id=V3JJCDM7EBRRZL2GPDMSCQAEHP2YX2QXDTDZDT5M6FGVLB27G7PQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.191365077 (486h30m16.742835528s) [noise=true , address=145.79.227.6:4259 , piece_id=KIYGYFEAYWG2NUAQZNWG4ZLMRY6GFJADI5LD635JE5M4DLHTX4TQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.191782502 (486h30m16.743252963s) [noise=true , address=145.79.227.22:4509 , piece_id=N2AEKOF7MLRQZ2HS7FSXV3U6WNI7YQLWP3W4PTFZLMAWQQREBTKQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.192449502 (486h30m16.743919953s) [noise=true , address=145.79.227.14:4839 , piece_id=RDZDYLJRCVHDKTTCAF3JB5ULPKFMSC2E4HNLB2RPLPY2GTUC6ZDQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.193091886 (486h30m16.744562347s) [noise=true , address=145.79.227.12:4767 , piece_id=5Q27HBFWOOI6PIIIMXELWG3QYSEA5B5LAOMU5K4FICMUMPDZIQAA, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.193463874 (486h30m16.744934335s) [noise=true , address=145.79.227.39:4013 , piece_id=YFEKNDHNI77M7MG76EISOGVU32UOZ536CK4X3Z6SFGOLJV2E7EHQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.195162279 (486h30m16.74663274s) [noise=true , address=145.79.227.20:4438 , piece_id=VZK574G5G2HXO5JMVMMEKGFCOJKOTU5KUIJDDEROFXRYTBTITO3A, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.197119445 (486h30m16.748589896s) [noise=true , address=145.79.227.7:4295 , piece_id=OLB3PUIQ4RYD2GGVF4I32FEGXO2O4X56PLJMOKNTJVBK53KZ3VYQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.197903478 (486h30m16.749373929s) [noise=true , address=145.79.227.32:4836 , piece_id=4KBSAAZLHC7OYLWN7RBDTQZUUTMK4HFGFUBES7K2PDKBXZST3OBA, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:19.999395950 (486h30m18.550866411s) [upload=428397]: Next returned nil segment Nov 29 14:34:19.999415177 (486h30m18.550885648s) [upload=428397]: Waiting for error group managing segments... Nov 29 14:34:20.285725942 (486h30m18.837196404s) [noise=true , address=77.90.31.13:4627 , piece_id=62A76IX5BE7WFBCCEMPSENCE5PWOSEMCUIKEATUPWGSLTCFJ4BGQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:20.363243793 (486h30m18.914714244s) [noise=true , address=45.91.116.5:4297 , piece_id=GD7HY7SUS6XPN6KJDC5AVRP2ECTJSB2MK4YBQXCNUOTUBVKFRGVQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:22.551752465 (486h30m21.103222926s) [noise=true , address=145.79.227.31:4798 , piece_id=NG2JWZTD5NPFNDIVWYSWFHQHEDTURKJOY6YZ7G6LAQRIDI7WD3FQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:22.666545505 (486h30m21.218015967s) [noise=true , address=145.79.227.22:4509 , piece_id=N2AEKOF7MLRQZ2HS7FSXV3U6WNI7YQLWP3W4PTFZLMAWQQREBTKQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:23.006352209 (486h30m21.557822671s) [noise=true , address=145.79.227.14:4839 , piece_id=RDZDYLJRCVHDKTTCAF3JB5ULPKFMSC2E4HNLB2RPLPY2GTUC6ZDQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:23.023700063 (486h30m21.575170525s) [noise=true , address=145.79.227.33:4869 , piece_id=V3JJCDM7EBRRZL2GPDMSCQAEHP2YX2QXDTDZDT5M6FGVLB27G7PQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:23.426873130 (486h30m21.978343581s) [noise=true , address=145.79.227.32:4836 , piece_id=4KBSAAZLHC7OYLWN7RBDTQZUUTMK4HFGFUBES7K2PDKBXZST3OBA, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:23.722460741 (486h30m22.273931202s) [noise=true , address=145.79.227.40:4049 , piece_id=DIBJ3Y4NA3HRAZD7GLO72STJHS5XHA3OTCAMWL4VCHMEILLCL3UA, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:25.698494096 (486h30m24.249964557s) [noise=true , address=145.79.227.39:4013 , piece_id=YFEKNDHNI77M7MG76EISOGVU32UOZ536CK4X3Z6SFGOLJV2E7EHQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:25.822411189 (486h30m24.373881651s) [noise=true , address=145.79.227.2:4114 , piece_id=ZB7V2MM6KI6VTSGSC4ZD4QI6EA2265OP3MVLWD2AYAHFERAUE7CQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:29.722449985 (486h30m28.273920436s) [noise=true , address=145.79.227.12:4767 , piece_id=5Q27HBFWOOI6PIIIMXELWG3QYSEA5B5LAOMU5K4FICMUMPDZIQAA, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:47.238199596 (486h30m45.789670047s) [noise=true , address=145.79.227.20:4438 , piece_id=VZK574G5G2HXO5JMVMMEKGFCOJKOTU5KUIJDDEROFXRYTBTITO3A, seg_pos={0 0}, upload=428397]: Done uploading piece. err: ecclient: upload failed (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, address:145.79.227.20:4438): protocol: write tcp 188.166.225.170:34106->145.79.227.20:4438: use of closed network connection; write tcp 188.166.225.170:34106->145.79.227.20:4438: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34106->145.79.227.20:4438: use of closed network connection Nov 29 14:34:47.238464621 (486h30m45.789935082s) [noise=true , address=145.79.227.21:4473 , piece_id=S2V3HONDHED6NZSIFZOTWU2DQ6FKESGGZ4EVMZX36LI4NT32WDWQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:46118->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:46118->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46118->145.79.227.21:4473: use of closed network connection Nov 29 14:34:47.238703104 (486h30m45.790173565s) [noise=true , address=145.79.227.24:4583 , piece_id=B6GTB3UF3XX64M5U57FFCFAMWNWDJUDPIZDG23KOHXOVCFTGKCLA, seg_pos={0 0}, upload=428397]: Done uploading piece. err: ecclient: upload failed (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, address:145.79.227.24:4583): protocol: write tcp 188.166.225.170:51236->145.79.227.24:4583: use of closed network connection; write tcp 188.166.225.170:51236->145.79.227.24:4583: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51236->145.79.227.24:4583: use of closed network connection Nov 29 14:34:47.238869100 (486h30m45.790339561s) [noise=true , address=145.79.227.7:4295 , piece_id=OLB3PUIQ4RYD2GGVF4I32FEGXO2O4X56PLJMOKNTJVBK53KZ3VYQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:145.79.227.7:4295): protocol: write tcp 188.166.225.170:46306->145.79.227.7:4295: use of closed network connection; write tcp 188.166.225.170:46306->145.79.227.7:4295: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46306->145.79.227.7:4295: use of closed network connection Nov 29 14:34:47.239032171 (486h30m45.790502632s) [noise=true , address=145.79.227.6:4259 , piece_id=KIYGYFEAYWG2NUAQZNWG4ZLMRY6GFJADI5LD635JE5M4DLHTX4TQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:145.79.227.6:4259): protocol: write tcp 188.166.225.170:48184->145.79.227.6:4259: use of closed network connection; write tcp 188.166.225.170:48184->145.79.227.6:4259: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48184->145.79.227.6:4259: use of closed network connection Nov 29 14:34:47.239076555 (486h30m45.790547027s) [seg_pos={0 0}, upload=428397]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 29 14:34:47.481897829 (486h30m46.0333683s) [upload=428397]: Done uploading segments... Nov 29 14:34:47.481943356 (486h30m46.033413817s) [upload=428397]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.729047334s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 5.730290313s trace: https://jaeger.dev.storj.io/trace/4add906a25eb3f89 FAIL (2025-11-29 14:55:55 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Nov 29 14:34:17.941236229 (486h30m16.49270669s) [upload=428397]: Got next segment. Inline: false Nov 29 14:34:17.941289010 (486h30m16.492759471s) [upload=428397]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 29 14:34:17.941296063 (486h30m16.492766524s) [upload=428397]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 29 14:34:18.189160800 (486h30m16.740631261s) [seg_pos={0 0}, upload=428397]: Begin upload segment... Nov 29 14:34:18.189218760 (486h30m16.740689221s) [seg_pos={0 0}, upload=428397]: Done begin upload segment. Nov 29 14:34:18.189254097 (486h30m16.740724548s) [noise=true , address=145.79.227.40:4049 , piece_id=DIBJ3Y4NA3HRAZD7GLO72STJHS5XHA3OTCAMWL4VCHMEILLCL3UA, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.189472724 (486h30m16.740943185s) [noise=true , address=77.90.31.13:4627 , piece_id=62A76IX5BE7WFBCCEMPSENCE5PWOSEMCUIKEATUPWGSLTCFJ4BGQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.190037069 (486h30m16.74150753s) [noise=true , address=145.79.227.2:4114 , piece_id=ZB7V2MM6KI6VTSGSC4ZD4QI6EA2265OP3MVLWD2AYAHFERAUE7CQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.190276184 (486h30m16.741746645s) [noise=true , address=145.79.227.21:4473 , piece_id=S2V3HONDHED6NZSIFZOTWU2DQ6FKESGGZ4EVMZX36LI4NT32WDWQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.190487406 (486h30m16.741957867s) [noise=true , address=145.79.227.31:4798 , piece_id=NG2JWZTD5NPFNDIVWYSWFHQHEDTURKJOY6YZ7G6LAQRIDI7WD3FQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.190825751 (486h30m16.742296212s) [noise=true , address=45.91.116.5:4297 , piece_id=GD7HY7SUS6XPN6KJDC5AVRP2ECTJSB2MK4YBQXCNUOTUBVKFRGVQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.190967721 (486h30m16.742438192s) [noise=true , address=145.79.227.24:4583 , piece_id=B6GTB3UF3XX64M5U57FFCFAMWNWDJUDPIZDG23KOHXOVCFTGKCLA, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.191308490 (486h30m16.742778951s) [noise=true , address=145.79.227.33:4869 , piece_id=V3JJCDM7EBRRZL2GPDMSCQAEHP2YX2QXDTDZDT5M6FGVLB27G7PQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.191365077 (486h30m16.742835528s) [noise=true , address=145.79.227.6:4259 , piece_id=KIYGYFEAYWG2NUAQZNWG4ZLMRY6GFJADI5LD635JE5M4DLHTX4TQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.191782502 (486h30m16.743252963s) [noise=true , address=145.79.227.22:4509 , piece_id=N2AEKOF7MLRQZ2HS7FSXV3U6WNI7YQLWP3W4PTFZLMAWQQREBTKQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.192449502 (486h30m16.743919953s) [noise=true , address=145.79.227.14:4839 , piece_id=RDZDYLJRCVHDKTTCAF3JB5ULPKFMSC2E4HNLB2RPLPY2GTUC6ZDQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.193091886 (486h30m16.744562347s) [noise=true , address=145.79.227.12:4767 , piece_id=5Q27HBFWOOI6PIIIMXELWG3QYSEA5B5LAOMU5K4FICMUMPDZIQAA, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.193463874 (486h30m16.744934335s) [noise=true , address=145.79.227.39:4013 , piece_id=YFEKNDHNI77M7MG76EISOGVU32UOZ536CK4X3Z6SFGOLJV2E7EHQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.195162279 (486h30m16.74663274s) [noise=true , address=145.79.227.20:4438 , piece_id=VZK574G5G2HXO5JMVMMEKGFCOJKOTU5KUIJDDEROFXRYTBTITO3A, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.197119445 (486h30m16.748589896s) [noise=true , address=145.79.227.7:4295 , piece_id=OLB3PUIQ4RYD2GGVF4I32FEGXO2O4X56PLJMOKNTJVBK53KZ3VYQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.197903478 (486h30m16.749373929s) [noise=true , address=145.79.227.32:4836 , piece_id=4KBSAAZLHC7OYLWN7RBDTQZUUTMK4HFGFUBES7K2PDKBXZST3OBA, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:19.999395950 (486h30m18.550866411s) [upload=428397]: Next returned nil segment Nov 29 14:34:19.999415177 (486h30m18.550885648s) [upload=428397]: Waiting for error group managing segments... Nov 29 14:34:20.285725942 (486h30m18.837196404s) [noise=true , address=77.90.31.13:4627 , piece_id=62A76IX5BE7WFBCCEMPSENCE5PWOSEMCUIKEATUPWGSLTCFJ4BGQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:20.363243793 (486h30m18.914714244s) [noise=true , address=45.91.116.5:4297 , piece_id=GD7HY7SUS6XPN6KJDC5AVRP2ECTJSB2MK4YBQXCNUOTUBVKFRGVQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:22.551752465 (486h30m21.103222926s) [noise=true , address=145.79.227.31:4798 , piece_id=NG2JWZTD5NPFNDIVWYSWFHQHEDTURKJOY6YZ7G6LAQRIDI7WD3FQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:22.666545505 (486h30m21.218015967s) [noise=true , address=145.79.227.22:4509 , piece_id=N2AEKOF7MLRQZ2HS7FSXV3U6WNI7YQLWP3W4PTFZLMAWQQREBTKQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:23.006352209 (486h30m21.557822671s) [noise=true , address=145.79.227.14:4839 , piece_id=RDZDYLJRCVHDKTTCAF3JB5ULPKFMSC2E4HNLB2RPLPY2GTUC6ZDQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:23.023700063 (486h30m21.575170525s) [noise=true , address=145.79.227.33:4869 , piece_id=V3JJCDM7EBRRZL2GPDMSCQAEHP2YX2QXDTDZDT5M6FGVLB27G7PQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:23.426873130 (486h30m21.978343581s) [noise=true , address=145.79.227.32:4836 , piece_id=4KBSAAZLHC7OYLWN7RBDTQZUUTMK4HFGFUBES7K2PDKBXZST3OBA, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:23.722460741 (486h30m22.273931202s) [noise=true , address=145.79.227.40:4049 , piece_id=DIBJ3Y4NA3HRAZD7GLO72STJHS5XHA3OTCAMWL4VCHMEILLCL3UA, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:25.698494096 (486h30m24.249964557s) [noise=true , address=145.79.227.39:4013 , piece_id=YFEKNDHNI77M7MG76EISOGVU32UOZ536CK4X3Z6SFGOLJV2E7EHQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:25.822411189 (486h30m24.373881651s) [noise=true , address=145.79.227.2:4114 , piece_id=ZB7V2MM6KI6VTSGSC4ZD4QI6EA2265OP3MVLWD2AYAHFERAUE7CQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:29.722449985 (486h30m28.273920436s) [noise=true , address=145.79.227.12:4767 , piece_id=5Q27HBFWOOI6PIIIMXELWG3QYSEA5B5LAOMU5K4FICMUMPDZIQAA, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:47.238199596 (486h30m45.789670047s) [noise=true , address=145.79.227.20:4438 , piece_id=VZK574G5G2HXO5JMVMMEKGFCOJKOTU5KUIJDDEROFXRYTBTITO3A, seg_pos={0 0}, upload=428397]: Done uploading piece. err: ecclient: upload failed (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, address:145.79.227.20:4438): protocol: write tcp 188.166.225.170:34106->145.79.227.20:4438: use of closed network connection; write tcp 188.166.225.170:34106->145.79.227.20:4438: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34106->145.79.227.20:4438: use of closed network connection Nov 29 14:34:47.238464621 (486h30m45.789935082s) [noise=true , address=145.79.227.21:4473 , piece_id=S2V3HONDHED6NZSIFZOTWU2DQ6FKESGGZ4EVMZX36LI4NT32WDWQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:46118->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:46118->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46118->145.79.227.21:4473: use of closed network connection Nov 29 14:34:47.238703104 (486h30m45.790173565s) [noise=true , address=145.79.227.24:4583 , piece_id=B6GTB3UF3XX64M5U57FFCFAMWNWDJUDPIZDG23KOHXOVCFTGKCLA, seg_pos={0 0}, upload=428397]: Done uploading piece. err: ecclient: upload failed (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, address:145.79.227.24:4583): protocol: write tcp 188.166.225.170:51236->145.79.227.24:4583: use of closed network connection; write tcp 188.166.225.170:51236->145.79.227.24:4583: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51236->145.79.227.24:4583: use of closed network connection Nov 29 14:34:47.238869100 (486h30m45.790339561s) [noise=true , address=145.79.227.7:4295 , piece_id=OLB3PUIQ4RYD2GGVF4I32FEGXO2O4X56PLJMOKNTJVBK53KZ3VYQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:145.79.227.7:4295): protocol: write tcp 188.166.225.170:46306->145.79.227.7:4295: use of closed network connection; write tcp 188.166.225.170:46306->145.79.227.7:4295: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46306->145.79.227.7:4295: use of closed network connection Nov 29 14:34:47.239032171 (486h30m45.790502632s) [noise=true , address=145.79.227.6:4259 , piece_id=KIYGYFEAYWG2NUAQZNWG4ZLMRY6GFJADI5LD635JE5M4DLHTX4TQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:145.79.227.6:4259): protocol: write tcp 188.166.225.170:48184->145.79.227.6:4259: use of closed network connection; write tcp 188.166.225.170:48184->145.79.227.6:4259: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48184->145.79.227.6:4259: use of closed network connection Nov 29 14:34:47.239076555 (486h30m45.790547027s) [seg_pos={0 0}, upload=428397]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 29 14:34:47.481897829 (486h30m46.0333683s) [upload=428397]: Done uploading segments... Nov 29 14:34:47.481943356 (486h30m46.033413817s) [upload=428397]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.730317455s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [....X................................X......................] duration: 3.514042345s trace: https://jaeger.dev.storj.io/trace/275e9b01e747ebf4 FAIL (2025-11-29 14:55:52 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.514441626s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 6.406303492s trace: https://jaeger.dev.storj.io/trace/61a1d0645153f16 FAIL (2025-11-29 14:55:29 UTC) orbiter warning: context deadline exceeded Duration: 6.406350242s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.....X................................X.....................] duration: 3.404890808s trace: https://jaeger.dev.storj.io/trace/275e9b01e747ebf4 FAIL (2025-11-29 14:55:28 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.404922418s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 5.7384551s trace: https://jaeger.dev.storj.io/trace/4add906a25eb3f89 FAIL (2025-11-29 14:54:48 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Nov 29 14:34:17.941236229 (486h30m16.49270669s) [upload=428397]: Got next segment. Inline: false Nov 29 14:34:17.941289010 (486h30m16.492759471s) [upload=428397]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 29 14:34:17.941296063 (486h30m16.492766524s) [upload=428397]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 29 14:34:18.189160800 (486h30m16.740631261s) [seg_pos={0 0}, upload=428397]: Begin upload segment... Nov 29 14:34:18.189218760 (486h30m16.740689221s) [seg_pos={0 0}, upload=428397]: Done begin upload segment. Nov 29 14:34:18.189254097 (486h30m16.740724548s) [noise=true , address=145.79.227.40:4049 , piece_id=DIBJ3Y4NA3HRAZD7GLO72STJHS5XHA3OTCAMWL4VCHMEILLCL3UA, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.189472724 (486h30m16.740943185s) [noise=true , address=77.90.31.13:4627 , piece_id=62A76IX5BE7WFBCCEMPSENCE5PWOSEMCUIKEATUPWGSLTCFJ4BGQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.190037069 (486h30m16.74150753s) [noise=true , address=145.79.227.2:4114 , piece_id=ZB7V2MM6KI6VTSGSC4ZD4QI6EA2265OP3MVLWD2AYAHFERAUE7CQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.190276184 (486h30m16.741746645s) [noise=true , address=145.79.227.21:4473 , piece_id=S2V3HONDHED6NZSIFZOTWU2DQ6FKESGGZ4EVMZX36LI4NT32WDWQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.190487406 (486h30m16.741957867s) [noise=true , address=145.79.227.31:4798 , piece_id=NG2JWZTD5NPFNDIVWYSWFHQHEDTURKJOY6YZ7G6LAQRIDI7WD3FQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.190825751 (486h30m16.742296212s) [noise=true , address=45.91.116.5:4297 , piece_id=GD7HY7SUS6XPN6KJDC5AVRP2ECTJSB2MK4YBQXCNUOTUBVKFRGVQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.190967721 (486h30m16.742438192s) [noise=true , address=145.79.227.24:4583 , piece_id=B6GTB3UF3XX64M5U57FFCFAMWNWDJUDPIZDG23KOHXOVCFTGKCLA, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.191308490 (486h30m16.742778951s) [noise=true , address=145.79.227.33:4869 , piece_id=V3JJCDM7EBRRZL2GPDMSCQAEHP2YX2QXDTDZDT5M6FGVLB27G7PQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.191365077 (486h30m16.742835528s) [noise=true , address=145.79.227.6:4259 , piece_id=KIYGYFEAYWG2NUAQZNWG4ZLMRY6GFJADI5LD635JE5M4DLHTX4TQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.191782502 (486h30m16.743252963s) [noise=true , address=145.79.227.22:4509 , piece_id=N2AEKOF7MLRQZ2HS7FSXV3U6WNI7YQLWP3W4PTFZLMAWQQREBTKQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.192449502 (486h30m16.743919953s) [noise=true , address=145.79.227.14:4839 , piece_id=RDZDYLJRCVHDKTTCAF3JB5ULPKFMSC2E4HNLB2RPLPY2GTUC6ZDQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.193091886 (486h30m16.744562347s) [noise=true , address=145.79.227.12:4767 , piece_id=5Q27HBFWOOI6PIIIMXELWG3QYSEA5B5LAOMU5K4FICMUMPDZIQAA, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.193463874 (486h30m16.744934335s) [noise=true , address=145.79.227.39:4013 , piece_id=YFEKNDHNI77M7MG76EISOGVU32UOZ536CK4X3Z6SFGOLJV2E7EHQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.195162279 (486h30m16.74663274s) [noise=true , address=145.79.227.20:4438 , piece_id=VZK574G5G2HXO5JMVMMEKGFCOJKOTU5KUIJDDEROFXRYTBTITO3A, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.197119445 (486h30m16.748589896s) [noise=true , address=145.79.227.7:4295 , piece_id=OLB3PUIQ4RYD2GGVF4I32FEGXO2O4X56PLJMOKNTJVBK53KZ3VYQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.197903478 (486h30m16.749373929s) [noise=true , address=145.79.227.32:4836 , piece_id=4KBSAAZLHC7OYLWN7RBDTQZUUTMK4HFGFUBES7K2PDKBXZST3OBA, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:19.999395950 (486h30m18.550866411s) [upload=428397]: Next returned nil segment Nov 29 14:34:19.999415177 (486h30m18.550885648s) [upload=428397]: Waiting for error group managing segments... Nov 29 14:34:20.285725942 (486h30m18.837196404s) [noise=true , address=77.90.31.13:4627 , piece_id=62A76IX5BE7WFBCCEMPSENCE5PWOSEMCUIKEATUPWGSLTCFJ4BGQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:20.363243793 (486h30m18.914714244s) [noise=true , address=45.91.116.5:4297 , piece_id=GD7HY7SUS6XPN6KJDC5AVRP2ECTJSB2MK4YBQXCNUOTUBVKFRGVQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:22.551752465 (486h30m21.103222926s) [noise=true , address=145.79.227.31:4798 , piece_id=NG2JWZTD5NPFNDIVWYSWFHQHEDTURKJOY6YZ7G6LAQRIDI7WD3FQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:22.666545505 (486h30m21.218015967s) [noise=true , address=145.79.227.22:4509 , piece_id=N2AEKOF7MLRQZ2HS7FSXV3U6WNI7YQLWP3W4PTFZLMAWQQREBTKQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:23.006352209 (486h30m21.557822671s) [noise=true , address=145.79.227.14:4839 , piece_id=RDZDYLJRCVHDKTTCAF3JB5ULPKFMSC2E4HNLB2RPLPY2GTUC6ZDQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:23.023700063 (486h30m21.575170525s) [noise=true , address=145.79.227.33:4869 , piece_id=V3JJCDM7EBRRZL2GPDMSCQAEHP2YX2QXDTDZDT5M6FGVLB27G7PQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:23.426873130 (486h30m21.978343581s) [noise=true , address=145.79.227.32:4836 , piece_id=4KBSAAZLHC7OYLWN7RBDTQZUUTMK4HFGFUBES7K2PDKBXZST3OBA, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:23.722460741 (486h30m22.273931202s) [noise=true , address=145.79.227.40:4049 , piece_id=DIBJ3Y4NA3HRAZD7GLO72STJHS5XHA3OTCAMWL4VCHMEILLCL3UA, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:25.698494096 (486h30m24.249964557s) [noise=true , address=145.79.227.39:4013 , piece_id=YFEKNDHNI77M7MG76EISOGVU32UOZ536CK4X3Z6SFGOLJV2E7EHQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:25.822411189 (486h30m24.373881651s) [noise=true , address=145.79.227.2:4114 , piece_id=ZB7V2MM6KI6VTSGSC4ZD4QI6EA2265OP3MVLWD2AYAHFERAUE7CQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:29.722449985 (486h30m28.273920436s) [noise=true , address=145.79.227.12:4767 , piece_id=5Q27HBFWOOI6PIIIMXELWG3QYSEA5B5LAOMU5K4FICMUMPDZIQAA, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:47.238199596 (486h30m45.789670047s) [noise=true , address=145.79.227.20:4438 , piece_id=VZK574G5G2HXO5JMVMMEKGFCOJKOTU5KUIJDDEROFXRYTBTITO3A, seg_pos={0 0}, upload=428397]: Done uploading piece. err: ecclient: upload failed (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, address:145.79.227.20:4438): protocol: write tcp 188.166.225.170:34106->145.79.227.20:4438: use of closed network connection; write tcp 188.166.225.170:34106->145.79.227.20:4438: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34106->145.79.227.20:4438: use of closed network connection Nov 29 14:34:47.238464621 (486h30m45.789935082s) [noise=true , address=145.79.227.21:4473 , piece_id=S2V3HONDHED6NZSIFZOTWU2DQ6FKESGGZ4EVMZX36LI4NT32WDWQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:46118->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:46118->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46118->145.79.227.21:4473: use of closed network connection Nov 29 14:34:47.238703104 (486h30m45.790173565s) [noise=true , address=145.79.227.24:4583 , piece_id=B6GTB3UF3XX64M5U57FFCFAMWNWDJUDPIZDG23KOHXOVCFTGKCLA, seg_pos={0 0}, upload=428397]: Done uploading piece. err: ecclient: upload failed (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, address:145.79.227.24:4583): protocol: write tcp 188.166.225.170:51236->145.79.227.24:4583: use of closed network connection; write tcp 188.166.225.170:51236->145.79.227.24:4583: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51236->145.79.227.24:4583: use of closed network connection Nov 29 14:34:47.238869100 (486h30m45.790339561s) [noise=true , address=145.79.227.7:4295 , piece_id=OLB3PUIQ4RYD2GGVF4I32FEGXO2O4X56PLJMOKNTJVBK53KZ3VYQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:145.79.227.7:4295): protocol: write tcp 188.166.225.170:46306->145.79.227.7:4295: use of closed network connection; write tcp 188.166.225.170:46306->145.79.227.7:4295: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46306->145.79.227.7:4295: use of closed network connection Nov 29 14:34:47.239032171 (486h30m45.790502632s) [noise=true , address=145.79.227.6:4259 , piece_id=KIYGYFEAYWG2NUAQZNWG4ZLMRY6GFJADI5LD635JE5M4DLHTX4TQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:145.79.227.6:4259): protocol: write tcp 188.166.225.170:48184->145.79.227.6:4259: use of closed network connection; write tcp 188.166.225.170:48184->145.79.227.6:4259: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48184->145.79.227.6:4259: use of closed network connection Nov 29 14:34:47.239076555 (486h30m45.790547027s) [seg_pos={0 0}, upload=428397]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 29 14:34:47.481897829 (486h30m46.0333683s) [upload=428397]: Done uploading segments... Nov 29 14:34:47.481943356 (486h30m46.033413817s) [upload=428397]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.738492322s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [......X................................X....................] duration: 3.56067053s trace: https://jaeger.dev.storj.io/trace/275e9b01e747ebf4 FAIL (2025-11-29 14:54:39 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.560721978s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 5.991544083s trace: https://jaeger.dev.storj.io/trace/61a1d0645153f16 FAIL (2025-11-29 14:54:29 UTC) orbiter warning: context deadline exceeded Duration: 5.991573089s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 6.02367645s trace: https://jaeger.dev.storj.io/trace/4add906a25eb3f89 FAIL (2025-11-29 14:54:15 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Nov 29 14:34:17.941236229 (486h30m16.49270669s) [upload=428397]: Got next segment. Inline: false Nov 29 14:34:17.941289010 (486h30m16.492759471s) [upload=428397]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 29 14:34:17.941296063 (486h30m16.492766524s) [upload=428397]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 29 14:34:18.189160800 (486h30m16.740631261s) [seg_pos={0 0}, upload=428397]: Begin upload segment... Nov 29 14:34:18.189218760 (486h30m16.740689221s) [seg_pos={0 0}, upload=428397]: Done begin upload segment. Nov 29 14:34:18.189254097 (486h30m16.740724548s) [noise=true , address=145.79.227.40:4049 , piece_id=DIBJ3Y4NA3HRAZD7GLO72STJHS5XHA3OTCAMWL4VCHMEILLCL3UA, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.189472724 (486h30m16.740943185s) [noise=true , address=77.90.31.13:4627 , piece_id=62A76IX5BE7WFBCCEMPSENCE5PWOSEMCUIKEATUPWGSLTCFJ4BGQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.190037069 (486h30m16.74150753s) [noise=true , address=145.79.227.2:4114 , piece_id=ZB7V2MM6KI6VTSGSC4ZD4QI6EA2265OP3MVLWD2AYAHFERAUE7CQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.190276184 (486h30m16.741746645s) [noise=true , address=145.79.227.21:4473 , piece_id=S2V3HONDHED6NZSIFZOTWU2DQ6FKESGGZ4EVMZX36LI4NT32WDWQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.190487406 (486h30m16.741957867s) [noise=true , address=145.79.227.31:4798 , piece_id=NG2JWZTD5NPFNDIVWYSWFHQHEDTURKJOY6YZ7G6LAQRIDI7WD3FQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.190825751 (486h30m16.742296212s) [noise=true , address=45.91.116.5:4297 , piece_id=GD7HY7SUS6XPN6KJDC5AVRP2ECTJSB2MK4YBQXCNUOTUBVKFRGVQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.190967721 (486h30m16.742438192s) [noise=true , address=145.79.227.24:4583 , piece_id=B6GTB3UF3XX64M5U57FFCFAMWNWDJUDPIZDG23KOHXOVCFTGKCLA, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.191308490 (486h30m16.742778951s) [noise=true , address=145.79.227.33:4869 , piece_id=V3JJCDM7EBRRZL2GPDMSCQAEHP2YX2QXDTDZDT5M6FGVLB27G7PQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.191365077 (486h30m16.742835528s) [noise=true , address=145.79.227.6:4259 , piece_id=KIYGYFEAYWG2NUAQZNWG4ZLMRY6GFJADI5LD635JE5M4DLHTX4TQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.191782502 (486h30m16.743252963s) [noise=true , address=145.79.227.22:4509 , piece_id=N2AEKOF7MLRQZ2HS7FSXV3U6WNI7YQLWP3W4PTFZLMAWQQREBTKQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.192449502 (486h30m16.743919953s) [noise=true , address=145.79.227.14:4839 , piece_id=RDZDYLJRCVHDKTTCAF3JB5ULPKFMSC2E4HNLB2RPLPY2GTUC6ZDQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.193091886 (486h30m16.744562347s) [noise=true , address=145.79.227.12:4767 , piece_id=5Q27HBFWOOI6PIIIMXELWG3QYSEA5B5LAOMU5K4FICMUMPDZIQAA, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.193463874 (486h30m16.744934335s) [noise=true , address=145.79.227.39:4013 , piece_id=YFEKNDHNI77M7MG76EISOGVU32UOZ536CK4X3Z6SFGOLJV2E7EHQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.195162279 (486h30m16.74663274s) [noise=true , address=145.79.227.20:4438 , piece_id=VZK574G5G2HXO5JMVMMEKGFCOJKOTU5KUIJDDEROFXRYTBTITO3A, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.197119445 (486h30m16.748589896s) [noise=true , address=145.79.227.7:4295 , piece_id=OLB3PUIQ4RYD2GGVF4I32FEGXO2O4X56PLJMOKNTJVBK53KZ3VYQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.197903478 (486h30m16.749373929s) [noise=true , address=145.79.227.32:4836 , piece_id=4KBSAAZLHC7OYLWN7RBDTQZUUTMK4HFGFUBES7K2PDKBXZST3OBA, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:19.999395950 (486h30m18.550866411s) [upload=428397]: Next returned nil segment Nov 29 14:34:19.999415177 (486h30m18.550885648s) [upload=428397]: Waiting for error group managing segments... Nov 29 14:34:20.285725942 (486h30m18.837196404s) [noise=true , address=77.90.31.13:4627 , piece_id=62A76IX5BE7WFBCCEMPSENCE5PWOSEMCUIKEATUPWGSLTCFJ4BGQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:20.363243793 (486h30m18.914714244s) [noise=true , address=45.91.116.5:4297 , piece_id=GD7HY7SUS6XPN6KJDC5AVRP2ECTJSB2MK4YBQXCNUOTUBVKFRGVQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:22.551752465 (486h30m21.103222926s) [noise=true , address=145.79.227.31:4798 , piece_id=NG2JWZTD5NPFNDIVWYSWFHQHEDTURKJOY6YZ7G6LAQRIDI7WD3FQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:22.666545505 (486h30m21.218015967s) [noise=true , address=145.79.227.22:4509 , piece_id=N2AEKOF7MLRQZ2HS7FSXV3U6WNI7YQLWP3W4PTFZLMAWQQREBTKQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:23.006352209 (486h30m21.557822671s) [noise=true , address=145.79.227.14:4839 , piece_id=RDZDYLJRCVHDKTTCAF3JB5ULPKFMSC2E4HNLB2RPLPY2GTUC6ZDQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:23.023700063 (486h30m21.575170525s) [noise=true , address=145.79.227.33:4869 , piece_id=V3JJCDM7EBRRZL2GPDMSCQAEHP2YX2QXDTDZDT5M6FGVLB27G7PQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:23.426873130 (486h30m21.978343581s) [noise=true , address=145.79.227.32:4836 , piece_id=4KBSAAZLHC7OYLWN7RBDTQZUUTMK4HFGFUBES7K2PDKBXZST3OBA, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:23.722460741 (486h30m22.273931202s) [noise=true , address=145.79.227.40:4049 , piece_id=DIBJ3Y4NA3HRAZD7GLO72STJHS5XHA3OTCAMWL4VCHMEILLCL3UA, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:25.698494096 (486h30m24.249964557s) [noise=true , address=145.79.227.39:4013 , piece_id=YFEKNDHNI77M7MG76EISOGVU32UOZ536CK4X3Z6SFGOLJV2E7EHQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:25.822411189 (486h30m24.373881651s) [noise=true , address=145.79.227.2:4114 , piece_id=ZB7V2MM6KI6VTSGSC4ZD4QI6EA2265OP3MVLWD2AYAHFERAUE7CQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:29.722449985 (486h30m28.273920436s) [noise=true , address=145.79.227.12:4767 , piece_id=5Q27HBFWOOI6PIIIMXELWG3QYSEA5B5LAOMU5K4FICMUMPDZIQAA, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:47.238199596 (486h30m45.789670047s) [noise=true , address=145.79.227.20:4438 , piece_id=VZK574G5G2HXO5JMVMMEKGFCOJKOTU5KUIJDDEROFXRYTBTITO3A, seg_pos={0 0}, upload=428397]: Done uploading piece. err: ecclient: upload failed (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, address:145.79.227.20:4438): protocol: write tcp 188.166.225.170:34106->145.79.227.20:4438: use of closed network connection; write tcp 188.166.225.170:34106->145.79.227.20:4438: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34106->145.79.227.20:4438: use of closed network connection Nov 29 14:34:47.238464621 (486h30m45.789935082s) [noise=true , address=145.79.227.21:4473 , piece_id=S2V3HONDHED6NZSIFZOTWU2DQ6FKESGGZ4EVMZX36LI4NT32WDWQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:46118->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:46118->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46118->145.79.227.21:4473: use of closed network connection Nov 29 14:34:47.238703104 (486h30m45.790173565s) [noise=true , address=145.79.227.24:4583 , piece_id=B6GTB3UF3XX64M5U57FFCFAMWNWDJUDPIZDG23KOHXOVCFTGKCLA, seg_pos={0 0}, upload=428397]: Done uploading piece. err: ecclient: upload failed (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, address:145.79.227.24:4583): protocol: write tcp 188.166.225.170:51236->145.79.227.24:4583: use of closed network connection; write tcp 188.166.225.170:51236->145.79.227.24:4583: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51236->145.79.227.24:4583: use of closed network connection Nov 29 14:34:47.238869100 (486h30m45.790339561s) [noise=true , address=145.79.227.7:4295 , piece_id=OLB3PUIQ4RYD2GGVF4I32FEGXO2O4X56PLJMOKNTJVBK53KZ3VYQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:145.79.227.7:4295): protocol: write tcp 188.166.225.170:46306->145.79.227.7:4295: use of closed network connection; write tcp 188.166.225.170:46306->145.79.227.7:4295: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46306->145.79.227.7:4295: use of closed network connection Nov 29 14:34:47.239032171 (486h30m45.790502632s) [noise=true , address=145.79.227.6:4259 , piece_id=KIYGYFEAYWG2NUAQZNWG4ZLMRY6GFJADI5LD635JE5M4DLHTX4TQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:145.79.227.6:4259): protocol: write tcp 188.166.225.170:48184->145.79.227.6:4259: use of closed network connection; write tcp 188.166.225.170:48184->145.79.227.6:4259: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48184->145.79.227.6:4259: use of closed network connection Nov 29 14:34:47.239076555 (486h30m45.790547027s) [seg_pos={0 0}, upload=428397]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 29 14:34:47.481897829 (486h30m46.0333683s) [upload=428397]: Done uploading segments... Nov 29 14:34:47.481943356 (486h30m46.033413817s) [upload=428397]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 6.023706226s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 5.845383253s trace: https://jaeger.dev.storj.io/trace/61a1d0645153f16 FAIL (2025-11-29 14:53:56 UTC) orbiter warning: context deadline exceeded Duration: 5.845409833s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.......X................................X...................] duration: 3.683973329s trace: https://jaeger.dev.storj.io/trace/275e9b01e747ebf4 FAIL (2025-11-29 14:53:50 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.684004098s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 5.886687443s trace: https://jaeger.dev.storj.io/trace/4add906a25eb3f89 FAIL (2025-11-29 14:53:49 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Nov 29 14:34:17.941236229 (486h30m16.49270669s) [upload=428397]: Got next segment. Inline: false Nov 29 14:34:17.941289010 (486h30m16.492759471s) [upload=428397]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 29 14:34:17.941296063 (486h30m16.492766524s) [upload=428397]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 29 14:34:18.189160800 (486h30m16.740631261s) [seg_pos={0 0}, upload=428397]: Begin upload segment... Nov 29 14:34:18.189218760 (486h30m16.740689221s) [seg_pos={0 0}, upload=428397]: Done begin upload segment. Nov 29 14:34:18.189254097 (486h30m16.740724548s) [noise=true , address=145.79.227.40:4049 , piece_id=DIBJ3Y4NA3HRAZD7GLO72STJHS5XHA3OTCAMWL4VCHMEILLCL3UA, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.189472724 (486h30m16.740943185s) [noise=true , address=77.90.31.13:4627 , piece_id=62A76IX5BE7WFBCCEMPSENCE5PWOSEMCUIKEATUPWGSLTCFJ4BGQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.190037069 (486h30m16.74150753s) [noise=true , address=145.79.227.2:4114 , piece_id=ZB7V2MM6KI6VTSGSC4ZD4QI6EA2265OP3MVLWD2AYAHFERAUE7CQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.190276184 (486h30m16.741746645s) [noise=true , address=145.79.227.21:4473 , piece_id=S2V3HONDHED6NZSIFZOTWU2DQ6FKESGGZ4EVMZX36LI4NT32WDWQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.190487406 (486h30m16.741957867s) [noise=true , address=145.79.227.31:4798 , piece_id=NG2JWZTD5NPFNDIVWYSWFHQHEDTURKJOY6YZ7G6LAQRIDI7WD3FQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.190825751 (486h30m16.742296212s) [noise=true , address=45.91.116.5:4297 , piece_id=GD7HY7SUS6XPN6KJDC5AVRP2ECTJSB2MK4YBQXCNUOTUBVKFRGVQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.190967721 (486h30m16.742438192s) [noise=true , address=145.79.227.24:4583 , piece_id=B6GTB3UF3XX64M5U57FFCFAMWNWDJUDPIZDG23KOHXOVCFTGKCLA, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.191308490 (486h30m16.742778951s) [noise=true , address=145.79.227.33:4869 , piece_id=V3JJCDM7EBRRZL2GPDMSCQAEHP2YX2QXDTDZDT5M6FGVLB27G7PQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.191365077 (486h30m16.742835528s) [noise=true , address=145.79.227.6:4259 , piece_id=KIYGYFEAYWG2NUAQZNWG4ZLMRY6GFJADI5LD635JE5M4DLHTX4TQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.191782502 (486h30m16.743252963s) [noise=true , address=145.79.227.22:4509 , piece_id=N2AEKOF7MLRQZ2HS7FSXV3U6WNI7YQLWP3W4PTFZLMAWQQREBTKQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.192449502 (486h30m16.743919953s) [noise=true , address=145.79.227.14:4839 , piece_id=RDZDYLJRCVHDKTTCAF3JB5ULPKFMSC2E4HNLB2RPLPY2GTUC6ZDQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.193091886 (486h30m16.744562347s) [noise=true , address=145.79.227.12:4767 , piece_id=5Q27HBFWOOI6PIIIMXELWG3QYSEA5B5LAOMU5K4FICMUMPDZIQAA, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.193463874 (486h30m16.744934335s) [noise=true , address=145.79.227.39:4013 , piece_id=YFEKNDHNI77M7MG76EISOGVU32UOZ536CK4X3Z6SFGOLJV2E7EHQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.195162279 (486h30m16.74663274s) [noise=true , address=145.79.227.20:4438 , piece_id=VZK574G5G2HXO5JMVMMEKGFCOJKOTU5KUIJDDEROFXRYTBTITO3A, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.197119445 (486h30m16.748589896s) [noise=true , address=145.79.227.7:4295 , piece_id=OLB3PUIQ4RYD2GGVF4I32FEGXO2O4X56PLJMOKNTJVBK53KZ3VYQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.197903478 (486h30m16.749373929s) [noise=true , address=145.79.227.32:4836 , piece_id=4KBSAAZLHC7OYLWN7RBDTQZUUTMK4HFGFUBES7K2PDKBXZST3OBA, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:19.999395950 (486h30m18.550866411s) [upload=428397]: Next returned nil segment Nov 29 14:34:19.999415177 (486h30m18.550885648s) [upload=428397]: Waiting for error group managing segments... Nov 29 14:34:20.285725942 (486h30m18.837196404s) [noise=true , address=77.90.31.13:4627 , piece_id=62A76IX5BE7WFBCCEMPSENCE5PWOSEMCUIKEATUPWGSLTCFJ4BGQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:20.363243793 (486h30m18.914714244s) [noise=true , address=45.91.116.5:4297 , piece_id=GD7HY7SUS6XPN6KJDC5AVRP2ECTJSB2MK4YBQXCNUOTUBVKFRGVQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:22.551752465 (486h30m21.103222926s) [noise=true , address=145.79.227.31:4798 , piece_id=NG2JWZTD5NPFNDIVWYSWFHQHEDTURKJOY6YZ7G6LAQRIDI7WD3FQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:22.666545505 (486h30m21.218015967s) [noise=true , address=145.79.227.22:4509 , piece_id=N2AEKOF7MLRQZ2HS7FSXV3U6WNI7YQLWP3W4PTFZLMAWQQREBTKQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:23.006352209 (486h30m21.557822671s) [noise=true , address=145.79.227.14:4839 , piece_id=RDZDYLJRCVHDKTTCAF3JB5ULPKFMSC2E4HNLB2RPLPY2GTUC6ZDQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:23.023700063 (486h30m21.575170525s) [noise=true , address=145.79.227.33:4869 , piece_id=V3JJCDM7EBRRZL2GPDMSCQAEHP2YX2QXDTDZDT5M6FGVLB27G7PQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:23.426873130 (486h30m21.978343581s) [noise=true , address=145.79.227.32:4836 , piece_id=4KBSAAZLHC7OYLWN7RBDTQZUUTMK4HFGFUBES7K2PDKBXZST3OBA, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:23.722460741 (486h30m22.273931202s) [noise=true , address=145.79.227.40:4049 , piece_id=DIBJ3Y4NA3HRAZD7GLO72STJHS5XHA3OTCAMWL4VCHMEILLCL3UA, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:25.698494096 (486h30m24.249964557s) [noise=true , address=145.79.227.39:4013 , piece_id=YFEKNDHNI77M7MG76EISOGVU32UOZ536CK4X3Z6SFGOLJV2E7EHQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:25.822411189 (486h30m24.373881651s) [noise=true , address=145.79.227.2:4114 , piece_id=ZB7V2MM6KI6VTSGSC4ZD4QI6EA2265OP3MVLWD2AYAHFERAUE7CQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:29.722449985 (486h30m28.273920436s) [noise=true , address=145.79.227.12:4767 , piece_id=5Q27HBFWOOI6PIIIMXELWG3QYSEA5B5LAOMU5K4FICMUMPDZIQAA, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:47.238199596 (486h30m45.789670047s) [noise=true , address=145.79.227.20:4438 , piece_id=VZK574G5G2HXO5JMVMMEKGFCOJKOTU5KUIJDDEROFXRYTBTITO3A, seg_pos={0 0}, upload=428397]: Done uploading piece. err: ecclient: upload failed (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, address:145.79.227.20:4438): protocol: write tcp 188.166.225.170:34106->145.79.227.20:4438: use of closed network connection; write tcp 188.166.225.170:34106->145.79.227.20:4438: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34106->145.79.227.20:4438: use of closed network connection Nov 29 14:34:47.238464621 (486h30m45.789935082s) [noise=true , address=145.79.227.21:4473 , piece_id=S2V3HONDHED6NZSIFZOTWU2DQ6FKESGGZ4EVMZX36LI4NT32WDWQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:46118->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:46118->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46118->145.79.227.21:4473: use of closed network connection Nov 29 14:34:47.238703104 (486h30m45.790173565s) [noise=true , address=145.79.227.24:4583 , piece_id=B6GTB3UF3XX64M5U57FFCFAMWNWDJUDPIZDG23KOHXOVCFTGKCLA, seg_pos={0 0}, upload=428397]: Done uploading piece. err: ecclient: upload failed (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, address:145.79.227.24:4583): protocol: write tcp 188.166.225.170:51236->145.79.227.24:4583: use of closed network connection; write tcp 188.166.225.170:51236->145.79.227.24:4583: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51236->145.79.227.24:4583: use of closed network connection Nov 29 14:34:47.238869100 (486h30m45.790339561s) [noise=true , address=145.79.227.7:4295 , piece_id=OLB3PUIQ4RYD2GGVF4I32FEGXO2O4X56PLJMOKNTJVBK53KZ3VYQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:145.79.227.7:4295): protocol: write tcp 188.166.225.170:46306->145.79.227.7:4295: use of closed network connection; write tcp 188.166.225.170:46306->145.79.227.7:4295: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46306->145.79.227.7:4295: use of closed network connection Nov 29 14:34:47.239032171 (486h30m45.790502632s) [noise=true , address=145.79.227.6:4259 , piece_id=KIYGYFEAYWG2NUAQZNWG4ZLMRY6GFJADI5LD635JE5M4DLHTX4TQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:145.79.227.6:4259): protocol: write tcp 188.166.225.170:48184->145.79.227.6:4259: use of closed network connection; write tcp 188.166.225.170:48184->145.79.227.6:4259: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48184->145.79.227.6:4259: use of closed network connection Nov 29 14:34:47.239076555 (486h30m45.790547027s) [seg_pos={0 0}, upload=428397]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 29 14:34:47.481897829 (486h30m46.0333683s) [upload=428397]: Done uploading segments... Nov 29 14:34:47.481943356 (486h30m46.033413817s) [upload=428397]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.886715897s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 5.835418418s trace: https://jaeger.dev.storj.io/trace/4add906a25eb3f89 FAIL (2025-11-29 14:53:24 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Nov 29 14:34:17.941236229 (486h30m16.49270669s) [upload=428397]: Got next segment. Inline: false Nov 29 14:34:17.941289010 (486h30m16.492759471s) [upload=428397]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 29 14:34:17.941296063 (486h30m16.492766524s) [upload=428397]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 29 14:34:18.189160800 (486h30m16.740631261s) [seg_pos={0 0}, upload=428397]: Begin upload segment... Nov 29 14:34:18.189218760 (486h30m16.740689221s) [seg_pos={0 0}, upload=428397]: Done begin upload segment. Nov 29 14:34:18.189254097 (486h30m16.740724548s) [noise=true , address=145.79.227.40:4049 , piece_id=DIBJ3Y4NA3HRAZD7GLO72STJHS5XHA3OTCAMWL4VCHMEILLCL3UA, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.189472724 (486h30m16.740943185s) [noise=true , address=77.90.31.13:4627 , piece_id=62A76IX5BE7WFBCCEMPSENCE5PWOSEMCUIKEATUPWGSLTCFJ4BGQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.190037069 (486h30m16.74150753s) [noise=true , address=145.79.227.2:4114 , piece_id=ZB7V2MM6KI6VTSGSC4ZD4QI6EA2265OP3MVLWD2AYAHFERAUE7CQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.190276184 (486h30m16.741746645s) [noise=true , address=145.79.227.21:4473 , piece_id=S2V3HONDHED6NZSIFZOTWU2DQ6FKESGGZ4EVMZX36LI4NT32WDWQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.190487406 (486h30m16.741957867s) [noise=true , address=145.79.227.31:4798 , piece_id=NG2JWZTD5NPFNDIVWYSWFHQHEDTURKJOY6YZ7G6LAQRIDI7WD3FQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.190825751 (486h30m16.742296212s) [noise=true , address=45.91.116.5:4297 , piece_id=GD7HY7SUS6XPN6KJDC5AVRP2ECTJSB2MK4YBQXCNUOTUBVKFRGVQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.190967721 (486h30m16.742438192s) [noise=true , address=145.79.227.24:4583 , piece_id=B6GTB3UF3XX64M5U57FFCFAMWNWDJUDPIZDG23KOHXOVCFTGKCLA, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.191308490 (486h30m16.742778951s) [noise=true , address=145.79.227.33:4869 , piece_id=V3JJCDM7EBRRZL2GPDMSCQAEHP2YX2QXDTDZDT5M6FGVLB27G7PQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.191365077 (486h30m16.742835528s) [noise=true , address=145.79.227.6:4259 , piece_id=KIYGYFEAYWG2NUAQZNWG4ZLMRY6GFJADI5LD635JE5M4DLHTX4TQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.191782502 (486h30m16.743252963s) [noise=true , address=145.79.227.22:4509 , piece_id=N2AEKOF7MLRQZ2HS7FSXV3U6WNI7YQLWP3W4PTFZLMAWQQREBTKQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.192449502 (486h30m16.743919953s) [noise=true , address=145.79.227.14:4839 , piece_id=RDZDYLJRCVHDKTTCAF3JB5ULPKFMSC2E4HNLB2RPLPY2GTUC6ZDQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.193091886 (486h30m16.744562347s) [noise=true , address=145.79.227.12:4767 , piece_id=5Q27HBFWOOI6PIIIMXELWG3QYSEA5B5LAOMU5K4FICMUMPDZIQAA, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.193463874 (486h30m16.744934335s) [noise=true , address=145.79.227.39:4013 , piece_id=YFEKNDHNI77M7MG76EISOGVU32UOZ536CK4X3Z6SFGOLJV2E7EHQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.195162279 (486h30m16.74663274s) [noise=true , address=145.79.227.20:4438 , piece_id=VZK574G5G2HXO5JMVMMEKGFCOJKOTU5KUIJDDEROFXRYTBTITO3A, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.197119445 (486h30m16.748589896s) [noise=true , address=145.79.227.7:4295 , piece_id=OLB3PUIQ4RYD2GGVF4I32FEGXO2O4X56PLJMOKNTJVBK53KZ3VYQ, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:18.197903478 (486h30m16.749373929s) [noise=true , address=145.79.227.32:4836 , piece_id=4KBSAAZLHC7OYLWN7RBDTQZUUTMK4HFGFUBES7K2PDKBXZST3OBA, seg_pos={0 0}, upload=428397]: Uploading piece... Nov 29 14:34:19.999395950 (486h30m18.550866411s) [upload=428397]: Next returned nil segment Nov 29 14:34:19.999415177 (486h30m18.550885648s) [upload=428397]: Waiting for error group managing segments... Nov 29 14:34:20.285725942 (486h30m18.837196404s) [noise=true , address=77.90.31.13:4627 , piece_id=62A76IX5BE7WFBCCEMPSENCE5PWOSEMCUIKEATUPWGSLTCFJ4BGQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:20.363243793 (486h30m18.914714244s) [noise=true , address=45.91.116.5:4297 , piece_id=GD7HY7SUS6XPN6KJDC5AVRP2ECTJSB2MK4YBQXCNUOTUBVKFRGVQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:22.551752465 (486h30m21.103222926s) [noise=true , address=145.79.227.31:4798 , piece_id=NG2JWZTD5NPFNDIVWYSWFHQHEDTURKJOY6YZ7G6LAQRIDI7WD3FQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:22.666545505 (486h30m21.218015967s) [noise=true , address=145.79.227.22:4509 , piece_id=N2AEKOF7MLRQZ2HS7FSXV3U6WNI7YQLWP3W4PTFZLMAWQQREBTKQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:23.006352209 (486h30m21.557822671s) [noise=true , address=145.79.227.14:4839 , piece_id=RDZDYLJRCVHDKTTCAF3JB5ULPKFMSC2E4HNLB2RPLPY2GTUC6ZDQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:23.023700063 (486h30m21.575170525s) [noise=true , address=145.79.227.33:4869 , piece_id=V3JJCDM7EBRRZL2GPDMSCQAEHP2YX2QXDTDZDT5M6FGVLB27G7PQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:23.426873130 (486h30m21.978343581s) [noise=true , address=145.79.227.32:4836 , piece_id=4KBSAAZLHC7OYLWN7RBDTQZUUTMK4HFGFUBES7K2PDKBXZST3OBA, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:23.722460741 (486h30m22.273931202s) [noise=true , address=145.79.227.40:4049 , piece_id=DIBJ3Y4NA3HRAZD7GLO72STJHS5XHA3OTCAMWL4VCHMEILLCL3UA, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:25.698494096 (486h30m24.249964557s) [noise=true , address=145.79.227.39:4013 , piece_id=YFEKNDHNI77M7MG76EISOGVU32UOZ536CK4X3Z6SFGOLJV2E7EHQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:25.822411189 (486h30m24.373881651s) [noise=true , address=145.79.227.2:4114 , piece_id=ZB7V2MM6KI6VTSGSC4ZD4QI6EA2265OP3MVLWD2AYAHFERAUE7CQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:29.722449985 (486h30m28.273920436s) [noise=true , address=145.79.227.12:4767 , piece_id=5Q27HBFWOOI6PIIIMXELWG3QYSEA5B5LAOMU5K4FICMUMPDZIQAA, seg_pos={0 0}, upload=428397]: Done uploading piece. err: <nil> Nov 29 14:34:47.238199596 (486h30m45.789670047s) [noise=true , address=145.79.227.20:4438 , piece_id=VZK574G5G2HXO5JMVMMEKGFCOJKOTU5KUIJDDEROFXRYTBTITO3A, seg_pos={0 0}, upload=428397]: Done uploading piece. err: ecclient: upload failed (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, address:145.79.227.20:4438): protocol: write tcp 188.166.225.170:34106->145.79.227.20:4438: use of closed network connection; write tcp 188.166.225.170:34106->145.79.227.20:4438: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34106->145.79.227.20:4438: use of closed network connection Nov 29 14:34:47.238464621 (486h30m45.789935082s) [noise=true , address=145.79.227.21:4473 , piece_id=S2V3HONDHED6NZSIFZOTWU2DQ6FKESGGZ4EVMZX36LI4NT32WDWQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:46118->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:46118->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46118->145.79.227.21:4473: use of closed network connection Nov 29 14:34:47.238703104 (486h30m45.790173565s) [noise=true , address=145.79.227.24:4583 , piece_id=B6GTB3UF3XX64M5U57FFCFAMWNWDJUDPIZDG23KOHXOVCFTGKCLA, seg_pos={0 0}, upload=428397]: Done uploading piece. err: ecclient: upload failed (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, address:145.79.227.24:4583): protocol: write tcp 188.166.225.170:51236->145.79.227.24:4583: use of closed network connection; write tcp 188.166.225.170:51236->145.79.227.24:4583: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51236->145.79.227.24:4583: use of closed network connection Nov 29 14:34:47.238869100 (486h30m45.790339561s) [noise=true , address=145.79.227.7:4295 , piece_id=OLB3PUIQ4RYD2GGVF4I32FEGXO2O4X56PLJMOKNTJVBK53KZ3VYQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:145.79.227.7:4295): protocol: write tcp 188.166.225.170:46306->145.79.227.7:4295: use of closed network connection; write tcp 188.166.225.170:46306->145.79.227.7:4295: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46306->145.79.227.7:4295: use of closed network connection Nov 29 14:34:47.239032171 (486h30m45.790502632s) [noise=true , address=145.79.227.6:4259 , piece_id=KIYGYFEAYWG2NUAQZNWG4ZLMRY6GFJADI5LD635JE5M4DLHTX4TQ, seg_pos={0 0}, upload=428397]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:145.79.227.6:4259): protocol: write tcp 188.166.225.170:48184->145.79.227.6:4259: use of closed network connection; write tcp 188.166.225.170:48184->145.79.227.6:4259: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48184->145.79.227.6:4259: use of closed network connection Nov 29 14:34:47.239076555 (486h30m45.790547027s) [seg_pos={0 0}, upload=428397]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 29 14:34:47.481897829 (486h30m46.0333683s) [upload=428397]: Done uploading segments... Nov 29 14:34:47.481943356 (486h30m46.033413817s) [upload=428397]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.835448996s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)