Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [XX..........................................................] duration: 3.64370273s trace: https://jaeger.dev.storj.io/trace/42d763a000d5a90d FAIL (2025-11-15 15:28:13 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.643733128s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [......X.........................................X...........] duration: 6.349218179s trace: https://jaeger.dev.storj.io/trace/46f6435aaf1b852a FAIL (2025-11-15 15:27:56 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 4) ======LOGS====== 6MZEKN7XOL667LYJSD7HHCBJIUGFBEYQFOQ, seg_pos={0 0}, upload=133220]: Uploading piece... Nov 15 15:21:08.018428616 (151h17m6.569899077s) [noise=true , address=45.91.116.11:4118 , piece_id=J36EGQPIQCW7CCGC4SKQBUVK7QC4UNIGBDBPYAB3A7TSYV377PDA, seg_pos={0 0}, upload=133220]: Uploading piece... Nov 15 15:21:08.018661680 (151h17m6.570132141s) [noise=true , address=145.79.227.26:4654 , piece_id=QZPWDI7TKCW3OGH36VQQH7EBQJQU2KGJZJ3Y5VZHVEMEDEOXPARA, seg_pos={0 0}, upload=133220]: Uploading piece... Nov 15 15:21:08.018882460 (151h17m6.570352921s) [noise=true , address=145.79.227.18:4005 , piece_id=ZWBRZGVA7CMHBEGSXAHFQPXLATKVSRWCAK5YQIYVZK7QJO3U5ZEA, seg_pos={0 0}, upload=133220]: Uploading piece... Nov 15 15:21:08.019281930 (151h17m6.570752391s) [noise=true , address=45.91.116.8:4045 , piece_id=FVDTJ3HTQGVPMVLV2656OCTNZT2Z3ZPFNC2C5F6GFHIDOAYLM3BA, seg_pos={0 0}, upload=133220]: Uploading piece... Nov 15 15:21:08.019316877 (151h17m6.570787338s) [noise=true , address=145.79.227.32:4836 , piece_id=2A5BU3NYUBVIEL62CAOPMBQEU7T5BDW5QOXVF3AZQFA2YU7VMLKQ, seg_pos={0 0}, upload=133220]: Uploading piece... Nov 15 15:21:08.019832619 (151h17m6.57130308s) [noise=true , address=45.91.116.5:4299 , piece_id=Z2XDDTQKI67WJ62RDWXNP76OITD3JVWIFPNBD5UYKT5ZP3J53UXA, seg_pos={0 0}, upload=133220]: Uploading piece... Nov 15 15:21:08.021202999 (151h17m6.57267347s) [noise=true , address=145.79.227.9:4365 , piece_id=5VNCHNGBR5XDMYFGRUVQ3SDACVDWY2DTVRUKVALTO7J6O6TVFP5A, seg_pos={0 0}, upload=133220]: Uploading piece... Nov 15 15:21:08.021959260 (151h17m6.573429721s) [noise=true , address=145.79.227.11:4729 , piece_id=FO73QHKPLIJZFWKCZZIJE33HKR2JQAMDVAAXFV7MZCZFL2D63U4A, seg_pos={0 0}, upload=133220]: Uploading piece... Nov 15 15:21:08.035442264 (151h17m6.586912725s) [noise=true , address=145.79.227.23:4546 , piece_id=I7U4GEI2HGXU7UTN7E4APJGRRBFASQ4DCZ32K32BGOENKHCEMPMA, seg_pos={0 0}, upload=133220]: Uploading piece... Nov 15 15:21:08.041195330 (151h17m6.592665811s) [noise=true , address=145.79.227.31:4798 , piece_id=YF6ZJD2LITIDCZAOJQQ2XVZIEGTDVCY7ZENSMWT6QRQU4R3QEGWQ, seg_pos={0 0}, upload=133220]: Uploading piece... Nov 15 15:21:09.871225261 (151h17m8.422695722s) [upload=133220]: Next returned nil segment Nov 15 15:21:09.871243576 (151h17m8.422714037s) [upload=133220]: Waiting for error group managing segments... Nov 15 15:21:10.165867146 (151h17m8.717337638s) [noise=true , address=77.90.31.19:4843 , piece_id=M47NOUZRG6LHAHBF26MZEKN7XOL667LYJSD7HHCBJIUGFBEYQFOQ, seg_pos={0 0}, upload=133220]: Done uploading piece. err: <nil> Nov 15 15:21:10.323410936 (151h17m8.874881397s) [noise=true , address=45.91.116.5:4299 , piece_id=Z2XDDTQKI67WJ62RDWXNP76OITD3JVWIFPNBD5UYKT5ZP3J53UXA, seg_pos={0 0}, upload=133220]: Done uploading piece. err: <nil> Nov 15 15:21:10.404481083 (151h17m8.955951544s) [noise=true , address=45.91.116.11:4118 , piece_id=J36EGQPIQCW7CCGC4SKQBUVK7QC4UNIGBDBPYAB3A7TSYV377PDA, seg_pos={0 0}, upload=133220]: Done uploading piece. err: <nil> Nov 15 15:21:11.440265666 (151h17m9.991736127s) [noise=true , address=45.91.116.8:4045 , piece_id=FVDTJ3HTQGVPMVLV2656OCTNZT2Z3ZPFNC2C5F6GFHIDOAYLM3BA, seg_pos={0 0}, upload=133220]: Done uploading piece. err: <nil> Nov 15 15:21:37.011608695 (151h17m35.563079156s) [noise=true , address=145.79.227.31:4798 , piece_id=YF6ZJD2LITIDCZAOJQQ2XVZIEGTDVCY7ZENSMWT6QRQU4R3QEGWQ, seg_pos={0 0}, upload=133220]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:145.79.227.31:4798): protocol: write tcp 188.166.225.170:45634->145.79.227.31:4798: use of closed network connection; write tcp 188.166.225.170:45634->145.79.227.31:4798: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45634->145.79.227.31:4798: use of closed network connection Nov 15 15:21:37.011777536 (151h17m35.563247997s) [noise=true , address=145.79.227.32:4836 , piece_id=2A5BU3NYUBVIEL62CAOPMBQEU7T5BDW5QOXVF3AZQFA2YU7VMLKQ, seg_pos={0 0}, upload=133220]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: write tcp 188.166.225.170:45832->145.79.227.32:4836: use of closed network connection; write tcp 188.166.225.170:45832->145.79.227.32:4836: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45832->145.79.227.32:4836: use of closed network connection Nov 15 15:21:37.011812132 (151h17m35.563282593s) [noise=true , address=145.79.227.30:4763 , piece_id=5HETJEEXU6DMLLAXICVF4A42YZHKQEVQ7AOKV734F7JRCIMDLTRQ, seg_pos={0 0}, upload=133220]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: read tcp 188.166.225.170:36386->145.79.227.30:4763: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:21:37.011874541 (151h17m35.563345002s) [noise=true , address=145.79.227.11:4729 , piece_id=FO73QHKPLIJZFWKCZZIJE33HKR2JQAMDVAAXFV7MZCZFL2D63U4A, seg_pos={0 0}, upload=133220]: Done uploading piece. err: ecclient: upload failed (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR, address:145.79.227.11:4729): protocol: write tcp 188.166.225.170:38964->145.79.227.11:4729: use of closed network connection; write tcp 188.166.225.170:38964->145.79.227.11:4729: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38964->145.79.227.11:4729: use of closed network connection Nov 15 15:21:37.011995582 (151h17m35.563466043s) [noise=true , address=145.79.227.12:4767 , piece_id=MZHSGDCEHU55EAMMFF6Z2XVRPNWGBKL3WLG2RTYG7QQTMW6UBKXA, seg_pos={0 0}, upload=133220]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:145.79.227.12:4767): protocol: write tcp 188.166.225.170:57008->145.79.227.12:4767: use of closed network connection; write tcp 188.166.225.170:57008->145.79.227.12:4767: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57008->145.79.227.12:4767: use of closed network connection Nov 15 15:21:37.012092546 (151h17m35.563563017s) [noise=true , address=145.79.227.28:4727 , piece_id=C4HI5WNAVBZ6OWWVF5WBMLJ4YDU6WJDLZ5ZXURAFV7WQ5H6F44BA, seg_pos={0 0}, upload=133220]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: read tcp 188.166.225.170:46810->145.79.227.28:4727: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:21:37.012296374 (151h17m35.563766835s) [noise=true , address=145.79.227.27:4689 , piece_id=KPQUJYT543MTIOZA653A74F2VDCGQAJ5QQV64AWMIEMKGRMUYQBQ, seg_pos={0 0}, upload=133220]: Done uploading piece. err: ecclient: upload failed (node:1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, address:145.79.227.27:4689): protocol: read tcp 188.166.225.170:49674->145.79.227.27:4689: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:21:37.012338324 (151h17m35.563808825s) [noise=true , address=145.79.227.9:4365 , piece_id=5VNCHNGBR5XDMYFGRUVQ3SDACVDWY2DTVRUKVALTO7J6O6TVFP5A, seg_pos={0 0}, upload=133220]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: read tcp 188.166.225.170:54022->145.79.227.9:4365: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:21:37.012485495 (151h17m35.563955956s) [noise=true , address=145.79.227.26:4654 , piece_id=QZPWDI7TKCW3OGH36VQQH7EBQJQU2KGJZJ3Y5VZHVEMEDEOXPARA, seg_pos={0 0}, upload=133220]: Done uploading piece. err: ecclient: upload failed (node:1KhBWMerZe9tUmzdVEYkBb6NmJ25dGP26Ea7CLjNJT5gdAeiah, address:145.79.227.26:4654): protocol: read tcp 188.166.225.170:53342->145.79.227.26:4654: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:21:37.012542613 (151h17m35.564013064s) [noise=true , address=145.79.227.18:4005 , piece_id=ZWBRZGVA7CMHBEGSXAHFQPXLATKVSRWCAK5YQIYVZK7QJO3U5ZEA, seg_pos={0 0}, upload=133220]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: read tcp 188.166.225.170:52578->145.79.227.18:4005: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:21:37.012760118 (151h17m35.564230579s) [noise=true , address=145.79.227.23:4546 , piece_id=I7U4GEI2HGXU7UTN7E4APJGRRBFASQ4DCZ32K32BGOENKHCEMPMA, seg_pos={0 0}, upload=133220]: Done uploading piece. err: ecclient: upload failed (node:1ZMTFpqfMjeVWXJuvsGdbGkMtGQn6axUZFoU1N9sK6fnbPcMy5, address:145.79.227.23:4546): protocol: read tcp 188.166.225.170:50436->145.79.227.23:4546: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:21:37.012942445 (151h17m35.564412896s) [noise=true , address=145.79.227.14:4839 , piece_id=E2IGX3AZ5HDJYSUZW426L3CJIJAPTFNU7GXICPRN2CBDLLW4U35Q, seg_pos={0 0}, upload=133220]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: read tcp 188.166.225.170:50180->145.79.227.14:4839: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:21:37.012979546 (151h17m35.564449997s) [seg_pos={0 0}, upload=133220]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil> Nov 15 15:21:37.270684105 (151h17m35.822154567s) [upload=133220]: Done uploading segments... Nov 15 15:21:37.270724753 (151h17m35.822195214s) [upload=133220]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4) Duration: 6.349248368s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [XXX.........................................................] duration: 3.608208078s trace: https://jaeger.dev.storj.io/trace/42d763a000d5a90d FAIL (2025-11-15 15:27:28 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.608238365s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......X.........................................X..........] duration: 5.956724235s trace: https://jaeger.dev.storj.io/trace/46f6435aaf1b852a FAIL (2025-11-15 15:27:26 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 4) ======LOGS====== 6MZEKN7XOL667LYJSD7HHCBJIUGFBEYQFOQ, seg_pos={0 0}, upload=133220]: Uploading piece... Nov 15 15:21:08.018428616 (151h17m6.569899077s) [noise=true , address=45.91.116.11:4118 , piece_id=J36EGQPIQCW7CCGC4SKQBUVK7QC4UNIGBDBPYAB3A7TSYV377PDA, seg_pos={0 0}, upload=133220]: Uploading piece... Nov 15 15:21:08.018661680 (151h17m6.570132141s) [noise=true , address=145.79.227.26:4654 , piece_id=QZPWDI7TKCW3OGH36VQQH7EBQJQU2KGJZJ3Y5VZHVEMEDEOXPARA, seg_pos={0 0}, upload=133220]: Uploading piece... Nov 15 15:21:08.018882460 (151h17m6.570352921s) [noise=true , address=145.79.227.18:4005 , piece_id=ZWBRZGVA7CMHBEGSXAHFQPXLATKVSRWCAK5YQIYVZK7QJO3U5ZEA, seg_pos={0 0}, upload=133220]: Uploading piece... Nov 15 15:21:08.019281930 (151h17m6.570752391s) [noise=true , address=45.91.116.8:4045 , piece_id=FVDTJ3HTQGVPMVLV2656OCTNZT2Z3ZPFNC2C5F6GFHIDOAYLM3BA, seg_pos={0 0}, upload=133220]: Uploading piece... Nov 15 15:21:08.019316877 (151h17m6.570787338s) [noise=true , address=145.79.227.32:4836 , piece_id=2A5BU3NYUBVIEL62CAOPMBQEU7T5BDW5QOXVF3AZQFA2YU7VMLKQ, seg_pos={0 0}, upload=133220]: Uploading piece... Nov 15 15:21:08.019832619 (151h17m6.57130308s) [noise=true , address=45.91.116.5:4299 , piece_id=Z2XDDTQKI67WJ62RDWXNP76OITD3JVWIFPNBD5UYKT5ZP3J53UXA, seg_pos={0 0}, upload=133220]: Uploading piece... Nov 15 15:21:08.021202999 (151h17m6.57267347s) [noise=true , address=145.79.227.9:4365 , piece_id=5VNCHNGBR5XDMYFGRUVQ3SDACVDWY2DTVRUKVALTO7J6O6TVFP5A, seg_pos={0 0}, upload=133220]: Uploading piece... Nov 15 15:21:08.021959260 (151h17m6.573429721s) [noise=true , address=145.79.227.11:4729 , piece_id=FO73QHKPLIJZFWKCZZIJE33HKR2JQAMDVAAXFV7MZCZFL2D63U4A, seg_pos={0 0}, upload=133220]: Uploading piece... Nov 15 15:21:08.035442264 (151h17m6.586912725s) [noise=true , address=145.79.227.23:4546 , piece_id=I7U4GEI2HGXU7UTN7E4APJGRRBFASQ4DCZ32K32BGOENKHCEMPMA, seg_pos={0 0}, upload=133220]: Uploading piece... Nov 15 15:21:08.041195330 (151h17m6.592665811s) [noise=true , address=145.79.227.31:4798 , piece_id=YF6ZJD2LITIDCZAOJQQ2XVZIEGTDVCY7ZENSMWT6QRQU4R3QEGWQ, seg_pos={0 0}, upload=133220]: Uploading piece... Nov 15 15:21:09.871225261 (151h17m8.422695722s) [upload=133220]: Next returned nil segment Nov 15 15:21:09.871243576 (151h17m8.422714037s) [upload=133220]: Waiting for error group managing segments... Nov 15 15:21:10.165867146 (151h17m8.717337638s) [noise=true , address=77.90.31.19:4843 , piece_id=M47NOUZRG6LHAHBF26MZEKN7XOL667LYJSD7HHCBJIUGFBEYQFOQ, seg_pos={0 0}, upload=133220]: Done uploading piece. err: <nil> Nov 15 15:21:10.323410936 (151h17m8.874881397s) [noise=true , address=45.91.116.5:4299 , piece_id=Z2XDDTQKI67WJ62RDWXNP76OITD3JVWIFPNBD5UYKT5ZP3J53UXA, seg_pos={0 0}, upload=133220]: Done uploading piece. err: <nil> Nov 15 15:21:10.404481083 (151h17m8.955951544s) [noise=true , address=45.91.116.11:4118 , piece_id=J36EGQPIQCW7CCGC4SKQBUVK7QC4UNIGBDBPYAB3A7TSYV377PDA, seg_pos={0 0}, upload=133220]: Done uploading piece. err: <nil> Nov 15 15:21:11.440265666 (151h17m9.991736127s) [noise=true , address=45.91.116.8:4045 , piece_id=FVDTJ3HTQGVPMVLV2656OCTNZT2Z3ZPFNC2C5F6GFHIDOAYLM3BA, seg_pos={0 0}, upload=133220]: Done uploading piece. err: <nil> Nov 15 15:21:37.011608695 (151h17m35.563079156s) [noise=true , address=145.79.227.31:4798 , piece_id=YF6ZJD2LITIDCZAOJQQ2XVZIEGTDVCY7ZENSMWT6QRQU4R3QEGWQ, seg_pos={0 0}, upload=133220]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:145.79.227.31:4798): protocol: write tcp 188.166.225.170:45634->145.79.227.31:4798: use of closed network connection; write tcp 188.166.225.170:45634->145.79.227.31:4798: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45634->145.79.227.31:4798: use of closed network connection Nov 15 15:21:37.011777536 (151h17m35.563247997s) [noise=true , address=145.79.227.32:4836 , piece_id=2A5BU3NYUBVIEL62CAOPMBQEU7T5BDW5QOXVF3AZQFA2YU7VMLKQ, seg_pos={0 0}, upload=133220]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: write tcp 188.166.225.170:45832->145.79.227.32:4836: use of closed network connection; write tcp 188.166.225.170:45832->145.79.227.32:4836: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45832->145.79.227.32:4836: use of closed network connection Nov 15 15:21:37.011812132 (151h17m35.563282593s) [noise=true , address=145.79.227.30:4763 , piece_id=5HETJEEXU6DMLLAXICVF4A42YZHKQEVQ7AOKV734F7JRCIMDLTRQ, seg_pos={0 0}, upload=133220]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: read tcp 188.166.225.170:36386->145.79.227.30:4763: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:21:37.011874541 (151h17m35.563345002s) [noise=true , address=145.79.227.11:4729 , piece_id=FO73QHKPLIJZFWKCZZIJE33HKR2JQAMDVAAXFV7MZCZFL2D63U4A, seg_pos={0 0}, upload=133220]: Done uploading piece. err: ecclient: upload failed (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR, address:145.79.227.11:4729): protocol: write tcp 188.166.225.170:38964->145.79.227.11:4729: use of closed network connection; write tcp 188.166.225.170:38964->145.79.227.11:4729: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38964->145.79.227.11:4729: use of closed network connection Nov 15 15:21:37.011995582 (151h17m35.563466043s) [noise=true , address=145.79.227.12:4767 , piece_id=MZHSGDCEHU55EAMMFF6Z2XVRPNWGBKL3WLG2RTYG7QQTMW6UBKXA, seg_pos={0 0}, upload=133220]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:145.79.227.12:4767): protocol: write tcp 188.166.225.170:57008->145.79.227.12:4767: use of closed network connection; write tcp 188.166.225.170:57008->145.79.227.12:4767: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57008->145.79.227.12:4767: use of closed network connection Nov 15 15:21:37.012092546 (151h17m35.563563017s) [noise=true , address=145.79.227.28:4727 , piece_id=C4HI5WNAVBZ6OWWVF5WBMLJ4YDU6WJDLZ5ZXURAFV7WQ5H6F44BA, seg_pos={0 0}, upload=133220]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: read tcp 188.166.225.170:46810->145.79.227.28:4727: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:21:37.012296374 (151h17m35.563766835s) [noise=true , address=145.79.227.27:4689 , piece_id=KPQUJYT543MTIOZA653A74F2VDCGQAJ5QQV64AWMIEMKGRMUYQBQ, seg_pos={0 0}, upload=133220]: Done uploading piece. err: ecclient: upload failed (node:1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, address:145.79.227.27:4689): protocol: read tcp 188.166.225.170:49674->145.79.227.27:4689: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:21:37.012338324 (151h17m35.563808825s) [noise=true , address=145.79.227.9:4365 , piece_id=5VNCHNGBR5XDMYFGRUVQ3SDACVDWY2DTVRUKVALTO7J6O6TVFP5A, seg_pos={0 0}, upload=133220]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: read tcp 188.166.225.170:54022->145.79.227.9:4365: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:21:37.012485495 (151h17m35.563955956s) [noise=true , address=145.79.227.26:4654 , piece_id=QZPWDI7TKCW3OGH36VQQH7EBQJQU2KGJZJ3Y5VZHVEMEDEOXPARA, seg_pos={0 0}, upload=133220]: Done uploading piece. err: ecclient: upload failed (node:1KhBWMerZe9tUmzdVEYkBb6NmJ25dGP26Ea7CLjNJT5gdAeiah, address:145.79.227.26:4654): protocol: read tcp 188.166.225.170:53342->145.79.227.26:4654: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:21:37.012542613 (151h17m35.564013064s) [noise=true , address=145.79.227.18:4005 , piece_id=ZWBRZGVA7CMHBEGSXAHFQPXLATKVSRWCAK5YQIYVZK7QJO3U5ZEA, seg_pos={0 0}, upload=133220]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: read tcp 188.166.225.170:52578->145.79.227.18:4005: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:21:37.012760118 (151h17m35.564230579s) [noise=true , address=145.79.227.23:4546 , piece_id=I7U4GEI2HGXU7UTN7E4APJGRRBFASQ4DCZ32K32BGOENKHCEMPMA, seg_pos={0 0}, upload=133220]: Done uploading piece. err: ecclient: upload failed (node:1ZMTFpqfMjeVWXJuvsGdbGkMtGQn6axUZFoU1N9sK6fnbPcMy5, address:145.79.227.23:4546): protocol: read tcp 188.166.225.170:50436->145.79.227.23:4546: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:21:37.012942445 (151h17m35.564412896s) [noise=true , address=145.79.227.14:4839 , piece_id=E2IGX3AZ5HDJYSUZW426L3CJIJAPTFNU7GXICPRN2CBDLLW4U35Q, seg_pos={0 0}, upload=133220]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: read tcp 188.166.225.170:50180->145.79.227.14:4839: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:21:37.012979546 (151h17m35.564449997s) [seg_pos={0 0}, upload=133220]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil> Nov 15 15:21:37.270684105 (151h17m35.822154567s) [upload=133220]: Done uploading segments... Nov 15 15:21:37.270724753 (151h17m35.822195214s) [upload=133220]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4) Duration: 5.956757376s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [XXXX........................................................] duration: 3.522550839s trace: https://jaeger.dev.storj.io/trace/42d763a000d5a90d FAIL (2025-11-15 15:26:36 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.522634989s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [........X.........................................X.........] duration: 8.688440982s trace: https://jaeger.dev.storj.io/trace/46f6435aaf1b852a FAIL (2025-11-15 15:26:16 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 4) ======LOGS====== 6MZEKN7XOL667LYJSD7HHCBJIUGFBEYQFOQ, seg_pos={0 0}, upload=133220]: Uploading piece... Nov 15 15:21:08.018428616 (151h17m6.569899077s) [noise=true , address=45.91.116.11:4118 , piece_id=J36EGQPIQCW7CCGC4SKQBUVK7QC4UNIGBDBPYAB3A7TSYV377PDA, seg_pos={0 0}, upload=133220]: Uploading piece... Nov 15 15:21:08.018661680 (151h17m6.570132141s) [noise=true , address=145.79.227.26:4654 , piece_id=QZPWDI7TKCW3OGH36VQQH7EBQJQU2KGJZJ3Y5VZHVEMEDEOXPARA, seg_pos={0 0}, upload=133220]: Uploading piece... Nov 15 15:21:08.018882460 (151h17m6.570352921s) [noise=true , address=145.79.227.18:4005 , piece_id=ZWBRZGVA7CMHBEGSXAHFQPXLATKVSRWCAK5YQIYVZK7QJO3U5ZEA, seg_pos={0 0}, upload=133220]: Uploading piece... Nov 15 15:21:08.019281930 (151h17m6.570752391s) [noise=true , address=45.91.116.8:4045 , piece_id=FVDTJ3HTQGVPMVLV2656OCTNZT2Z3ZPFNC2C5F6GFHIDOAYLM3BA, seg_pos={0 0}, upload=133220]: Uploading piece... Nov 15 15:21:08.019316877 (151h17m6.570787338s) [noise=true , address=145.79.227.32:4836 , piece_id=2A5BU3NYUBVIEL62CAOPMBQEU7T5BDW5QOXVF3AZQFA2YU7VMLKQ, seg_pos={0 0}, upload=133220]: Uploading piece... Nov 15 15:21:08.019832619 (151h17m6.57130308s) [noise=true , address=45.91.116.5:4299 , piece_id=Z2XDDTQKI67WJ62RDWXNP76OITD3JVWIFPNBD5UYKT5ZP3J53UXA, seg_pos={0 0}, upload=133220]: Uploading piece... Nov 15 15:21:08.021202999 (151h17m6.57267347s) [noise=true , address=145.79.227.9:4365 , piece_id=5VNCHNGBR5XDMYFGRUVQ3SDACVDWY2DTVRUKVALTO7J6O6TVFP5A, seg_pos={0 0}, upload=133220]: Uploading piece... Nov 15 15:21:08.021959260 (151h17m6.573429721s) [noise=true , address=145.79.227.11:4729 , piece_id=FO73QHKPLIJZFWKCZZIJE33HKR2JQAMDVAAXFV7MZCZFL2D63U4A, seg_pos={0 0}, upload=133220]: Uploading piece... Nov 15 15:21:08.035442264 (151h17m6.586912725s) [noise=true , address=145.79.227.23:4546 , piece_id=I7U4GEI2HGXU7UTN7E4APJGRRBFASQ4DCZ32K32BGOENKHCEMPMA, seg_pos={0 0}, upload=133220]: Uploading piece... Nov 15 15:21:08.041195330 (151h17m6.592665811s) [noise=true , address=145.79.227.31:4798 , piece_id=YF6ZJD2LITIDCZAOJQQ2XVZIEGTDVCY7ZENSMWT6QRQU4R3QEGWQ, seg_pos={0 0}, upload=133220]: Uploading piece... Nov 15 15:21:09.871225261 (151h17m8.422695722s) [upload=133220]: Next returned nil segment Nov 15 15:21:09.871243576 (151h17m8.422714037s) [upload=133220]: Waiting for error group managing segments... Nov 15 15:21:10.165867146 (151h17m8.717337638s) [noise=true , address=77.90.31.19:4843 , piece_id=M47NOUZRG6LHAHBF26MZEKN7XOL667LYJSD7HHCBJIUGFBEYQFOQ, seg_pos={0 0}, upload=133220]: Done uploading piece. err: <nil> Nov 15 15:21:10.323410936 (151h17m8.874881397s) [noise=true , address=45.91.116.5:4299 , piece_id=Z2XDDTQKI67WJ62RDWXNP76OITD3JVWIFPNBD5UYKT5ZP3J53UXA, seg_pos={0 0}, upload=133220]: Done uploading piece. err: <nil> Nov 15 15:21:10.404481083 (151h17m8.955951544s) [noise=true , address=45.91.116.11:4118 , piece_id=J36EGQPIQCW7CCGC4SKQBUVK7QC4UNIGBDBPYAB3A7TSYV377PDA, seg_pos={0 0}, upload=133220]: Done uploading piece. err: <nil> Nov 15 15:21:11.440265666 (151h17m9.991736127s) [noise=true , address=45.91.116.8:4045 , piece_id=FVDTJ3HTQGVPMVLV2656OCTNZT2Z3ZPFNC2C5F6GFHIDOAYLM3BA, seg_pos={0 0}, upload=133220]: Done uploading piece. err: <nil> Nov 15 15:21:37.011608695 (151h17m35.563079156s) [noise=true , address=145.79.227.31:4798 , piece_id=YF6ZJD2LITIDCZAOJQQ2XVZIEGTDVCY7ZENSMWT6QRQU4R3QEGWQ, seg_pos={0 0}, upload=133220]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:145.79.227.31:4798): protocol: write tcp 188.166.225.170:45634->145.79.227.31:4798: use of closed network connection; write tcp 188.166.225.170:45634->145.79.227.31:4798: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45634->145.79.227.31:4798: use of closed network connection Nov 15 15:21:37.011777536 (151h17m35.563247997s) [noise=true , address=145.79.227.32:4836 , piece_id=2A5BU3NYUBVIEL62CAOPMBQEU7T5BDW5QOXVF3AZQFA2YU7VMLKQ, seg_pos={0 0}, upload=133220]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: write tcp 188.166.225.170:45832->145.79.227.32:4836: use of closed network connection; write tcp 188.166.225.170:45832->145.79.227.32:4836: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45832->145.79.227.32:4836: use of closed network connection Nov 15 15:21:37.011812132 (151h17m35.563282593s) [noise=true , address=145.79.227.30:4763 , piece_id=5HETJEEXU6DMLLAXICVF4A42YZHKQEVQ7AOKV734F7JRCIMDLTRQ, seg_pos={0 0}, upload=133220]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: read tcp 188.166.225.170:36386->145.79.227.30:4763: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:21:37.011874541 (151h17m35.563345002s) [noise=true , address=145.79.227.11:4729 , piece_id=FO73QHKPLIJZFWKCZZIJE33HKR2JQAMDVAAXFV7MZCZFL2D63U4A, seg_pos={0 0}, upload=133220]: Done uploading piece. err: ecclient: upload failed (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR, address:145.79.227.11:4729): protocol: write tcp 188.166.225.170:38964->145.79.227.11:4729: use of closed network connection; write tcp 188.166.225.170:38964->145.79.227.11:4729: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38964->145.79.227.11:4729: use of closed network connection Nov 15 15:21:37.011995582 (151h17m35.563466043s) [noise=true , address=145.79.227.12:4767 , piece_id=MZHSGDCEHU55EAMMFF6Z2XVRPNWGBKL3WLG2RTYG7QQTMW6UBKXA, seg_pos={0 0}, upload=133220]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:145.79.227.12:4767): protocol: write tcp 188.166.225.170:57008->145.79.227.12:4767: use of closed network connection; write tcp 188.166.225.170:57008->145.79.227.12:4767: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57008->145.79.227.12:4767: use of closed network connection Nov 15 15:21:37.012092546 (151h17m35.563563017s) [noise=true , address=145.79.227.28:4727 , piece_id=C4HI5WNAVBZ6OWWVF5WBMLJ4YDU6WJDLZ5ZXURAFV7WQ5H6F44BA, seg_pos={0 0}, upload=133220]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: read tcp 188.166.225.170:46810->145.79.227.28:4727: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:21:37.012296374 (151h17m35.563766835s) [noise=true , address=145.79.227.27:4689 , piece_id=KPQUJYT543MTIOZA653A74F2VDCGQAJ5QQV64AWMIEMKGRMUYQBQ, seg_pos={0 0}, upload=133220]: Done uploading piece. err: ecclient: upload failed (node:1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, address:145.79.227.27:4689): protocol: read tcp 188.166.225.170:49674->145.79.227.27:4689: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:21:37.012338324 (151h17m35.563808825s) [noise=true , address=145.79.227.9:4365 , piece_id=5VNCHNGBR5XDMYFGRUVQ3SDACVDWY2DTVRUKVALTO7J6O6TVFP5A, seg_pos={0 0}, upload=133220]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: read tcp 188.166.225.170:54022->145.79.227.9:4365: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:21:37.012485495 (151h17m35.563955956s) [noise=true , address=145.79.227.26:4654 , piece_id=QZPWDI7TKCW3OGH36VQQH7EBQJQU2KGJZJ3Y5VZHVEMEDEOXPARA, seg_pos={0 0}, upload=133220]: Done uploading piece. err: ecclient: upload failed (node:1KhBWMerZe9tUmzdVEYkBb6NmJ25dGP26Ea7CLjNJT5gdAeiah, address:145.79.227.26:4654): protocol: read tcp 188.166.225.170:53342->145.79.227.26:4654: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:21:37.012542613 (151h17m35.564013064s) [noise=true , address=145.79.227.18:4005 , piece_id=ZWBRZGVA7CMHBEGSXAHFQPXLATKVSRWCAK5YQIYVZK7QJO3U5ZEA, seg_pos={0 0}, upload=133220]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: read tcp 188.166.225.170:52578->145.79.227.18:4005: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:21:37.012760118 (151h17m35.564230579s) [noise=true , address=145.79.227.23:4546 , piece_id=I7U4GEI2HGXU7UTN7E4APJGRRBFASQ4DCZ32K32BGOENKHCEMPMA, seg_pos={0 0}, upload=133220]: Done uploading piece. err: ecclient: upload failed (node:1ZMTFpqfMjeVWXJuvsGdbGkMtGQn6axUZFoU1N9sK6fnbPcMy5, address:145.79.227.23:4546): protocol: read tcp 188.166.225.170:50436->145.79.227.23:4546: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:21:37.012942445 (151h17m35.564412896s) [noise=true , address=145.79.227.14:4839 , piece_id=E2IGX3AZ5HDJYSUZW426L3CJIJAPTFNU7GXICPRN2CBDLLW4U35Q, seg_pos={0 0}, upload=133220]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: read tcp 188.166.225.170:50180->145.79.227.14:4839: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:21:37.012979546 (151h17m35.564449997s) [seg_pos={0 0}, upload=133220]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil> Nov 15 15:21:37.270684105 (151h17m35.822154567s) [upload=133220]: Done uploading segments... Nov 15 15:21:37.270724753 (151h17m35.822195214s) [upload=133220]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4) Duration: 8.688488232s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [.XXXX.......................................................] duration: 3.601670158s trace: https://jaeger.dev.storj.io/trace/42d763a000d5a90d FAIL (2025-11-15 15:26:09 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.601689695s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........X.........................................X........] duration: 6.231273165s trace: https://jaeger.dev.storj.io/trace/46f6435aaf1b852a FAIL (2025-11-15 15:25:46 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 4) ======LOGS====== 6MZEKN7XOL667LYJSD7HHCBJIUGFBEYQFOQ, seg_pos={0 0}, upload=133220]: Uploading piece... Nov 15 15:21:08.018428616 (151h17m6.569899077s) [noise=true , address=45.91.116.11:4118 , piece_id=J36EGQPIQCW7CCGC4SKQBUVK7QC4UNIGBDBPYAB3A7TSYV377PDA, seg_pos={0 0}, upload=133220]: Uploading piece... Nov 15 15:21:08.018661680 (151h17m6.570132141s) [noise=true , address=145.79.227.26:4654 , piece_id=QZPWDI7TKCW3OGH36VQQH7EBQJQU2KGJZJ3Y5VZHVEMEDEOXPARA, seg_pos={0 0}, upload=133220]: Uploading piece... Nov 15 15:21:08.018882460 (151h17m6.570352921s) [noise=true , address=145.79.227.18:4005 , piece_id=ZWBRZGVA7CMHBEGSXAHFQPXLATKVSRWCAK5YQIYVZK7QJO3U5ZEA, seg_pos={0 0}, upload=133220]: Uploading piece... Nov 15 15:21:08.019281930 (151h17m6.570752391s) [noise=true , address=45.91.116.8:4045 , piece_id=FVDTJ3HTQGVPMVLV2656OCTNZT2Z3ZPFNC2C5F6GFHIDOAYLM3BA, seg_pos={0 0}, upload=133220]: Uploading piece... Nov 15 15:21:08.019316877 (151h17m6.570787338s) [noise=true , address=145.79.227.32:4836 , piece_id=2A5BU3NYUBVIEL62CAOPMBQEU7T5BDW5QOXVF3AZQFA2YU7VMLKQ, seg_pos={0 0}, upload=133220]: Uploading piece... Nov 15 15:21:08.019832619 (151h17m6.57130308s) [noise=true , address=45.91.116.5:4299 , piece_id=Z2XDDTQKI67WJ62RDWXNP76OITD3JVWIFPNBD5UYKT5ZP3J53UXA, seg_pos={0 0}, upload=133220]: Uploading piece... Nov 15 15:21:08.021202999 (151h17m6.57267347s) [noise=true , address=145.79.227.9:4365 , piece_id=5VNCHNGBR5XDMYFGRUVQ3SDACVDWY2DTVRUKVALTO7J6O6TVFP5A, seg_pos={0 0}, upload=133220]: Uploading piece... Nov 15 15:21:08.021959260 (151h17m6.573429721s) [noise=true , address=145.79.227.11:4729 , piece_id=FO73QHKPLIJZFWKCZZIJE33HKR2JQAMDVAAXFV7MZCZFL2D63U4A, seg_pos={0 0}, upload=133220]: Uploading piece... Nov 15 15:21:08.035442264 (151h17m6.586912725s) [noise=true , address=145.79.227.23:4546 , piece_id=I7U4GEI2HGXU7UTN7E4APJGRRBFASQ4DCZ32K32BGOENKHCEMPMA, seg_pos={0 0}, upload=133220]: Uploading piece... Nov 15 15:21:08.041195330 (151h17m6.592665811s) [noise=true , address=145.79.227.31:4798 , piece_id=YF6ZJD2LITIDCZAOJQQ2XVZIEGTDVCY7ZENSMWT6QRQU4R3QEGWQ, seg_pos={0 0}, upload=133220]: Uploading piece... Nov 15 15:21:09.871225261 (151h17m8.422695722s) [upload=133220]: Next returned nil segment Nov 15 15:21:09.871243576 (151h17m8.422714037s) [upload=133220]: Waiting for error group managing segments... Nov 15 15:21:10.165867146 (151h17m8.717337638s) [noise=true , address=77.90.31.19:4843 , piece_id=M47NOUZRG6LHAHBF26MZEKN7XOL667LYJSD7HHCBJIUGFBEYQFOQ, seg_pos={0 0}, upload=133220]: Done uploading piece. err: <nil> Nov 15 15:21:10.323410936 (151h17m8.874881397s) [noise=true , address=45.91.116.5:4299 , piece_id=Z2XDDTQKI67WJ62RDWXNP76OITD3JVWIFPNBD5UYKT5ZP3J53UXA, seg_pos={0 0}, upload=133220]: Done uploading piece. err: <nil> Nov 15 15:21:10.404481083 (151h17m8.955951544s) [noise=true , address=45.91.116.11:4118 , piece_id=J36EGQPIQCW7CCGC4SKQBUVK7QC4UNIGBDBPYAB3A7TSYV377PDA, seg_pos={0 0}, upload=133220]: Done uploading piece. err: <nil> Nov 15 15:21:11.440265666 (151h17m9.991736127s) [noise=true , address=45.91.116.8:4045 , piece_id=FVDTJ3HTQGVPMVLV2656OCTNZT2Z3ZPFNC2C5F6GFHIDOAYLM3BA, seg_pos={0 0}, upload=133220]: Done uploading piece. err: <nil> Nov 15 15:21:37.011608695 (151h17m35.563079156s) [noise=true , address=145.79.227.31:4798 , piece_id=YF6ZJD2LITIDCZAOJQQ2XVZIEGTDVCY7ZENSMWT6QRQU4R3QEGWQ, seg_pos={0 0}, upload=133220]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:145.79.227.31:4798): protocol: write tcp 188.166.225.170:45634->145.79.227.31:4798: use of closed network connection; write tcp 188.166.225.170:45634->145.79.227.31:4798: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45634->145.79.227.31:4798: use of closed network connection Nov 15 15:21:37.011777536 (151h17m35.563247997s) [noise=true , address=145.79.227.32:4836 , piece_id=2A5BU3NYUBVIEL62CAOPMBQEU7T5BDW5QOXVF3AZQFA2YU7VMLKQ, seg_pos={0 0}, upload=133220]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: write tcp 188.166.225.170:45832->145.79.227.32:4836: use of closed network connection; write tcp 188.166.225.170:45832->145.79.227.32:4836: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45832->145.79.227.32:4836: use of closed network connection Nov 15 15:21:37.011812132 (151h17m35.563282593s) [noise=true , address=145.79.227.30:4763 , piece_id=5HETJEEXU6DMLLAXICVF4A42YZHKQEVQ7AOKV734F7JRCIMDLTRQ, seg_pos={0 0}, upload=133220]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: read tcp 188.166.225.170:36386->145.79.227.30:4763: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:21:37.011874541 (151h17m35.563345002s) [noise=true , address=145.79.227.11:4729 , piece_id=FO73QHKPLIJZFWKCZZIJE33HKR2JQAMDVAAXFV7MZCZFL2D63U4A, seg_pos={0 0}, upload=133220]: Done uploading piece. err: ecclient: upload failed (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR, address:145.79.227.11:4729): protocol: write tcp 188.166.225.170:38964->145.79.227.11:4729: use of closed network connection; write tcp 188.166.225.170:38964->145.79.227.11:4729: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38964->145.79.227.11:4729: use of closed network connection Nov 15 15:21:37.011995582 (151h17m35.563466043s) [noise=true , address=145.79.227.12:4767 , piece_id=MZHSGDCEHU55EAMMFF6Z2XVRPNWGBKL3WLG2RTYG7QQTMW6UBKXA, seg_pos={0 0}, upload=133220]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:145.79.227.12:4767): protocol: write tcp 188.166.225.170:57008->145.79.227.12:4767: use of closed network connection; write tcp 188.166.225.170:57008->145.79.227.12:4767: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57008->145.79.227.12:4767: use of closed network connection Nov 15 15:21:37.012092546 (151h17m35.563563017s) [noise=true , address=145.79.227.28:4727 , piece_id=C4HI5WNAVBZ6OWWVF5WBMLJ4YDU6WJDLZ5ZXURAFV7WQ5H6F44BA, seg_pos={0 0}, upload=133220]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: read tcp 188.166.225.170:46810->145.79.227.28:4727: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:21:37.012296374 (151h17m35.563766835s) [noise=true , address=145.79.227.27:4689 , piece_id=KPQUJYT543MTIOZA653A74F2VDCGQAJ5QQV64AWMIEMKGRMUYQBQ, seg_pos={0 0}, upload=133220]: Done uploading piece. err: ecclient: upload failed (node:1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, address:145.79.227.27:4689): protocol: read tcp 188.166.225.170:49674->145.79.227.27:4689: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:21:37.012338324 (151h17m35.563808825s) [noise=true , address=145.79.227.9:4365 , piece_id=5VNCHNGBR5XDMYFGRUVQ3SDACVDWY2DTVRUKVALTO7J6O6TVFP5A, seg_pos={0 0}, upload=133220]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: read tcp 188.166.225.170:54022->145.79.227.9:4365: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:21:37.012485495 (151h17m35.563955956s) [noise=true , address=145.79.227.26:4654 , piece_id=QZPWDI7TKCW3OGH36VQQH7EBQJQU2KGJZJ3Y5VZHVEMEDEOXPARA, seg_pos={0 0}, upload=133220]: Done uploading piece. err: ecclient: upload failed (node:1KhBWMerZe9tUmzdVEYkBb6NmJ25dGP26Ea7CLjNJT5gdAeiah, address:145.79.227.26:4654): protocol: read tcp 188.166.225.170:53342->145.79.227.26:4654: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:21:37.012542613 (151h17m35.564013064s) [noise=true , address=145.79.227.18:4005 , piece_id=ZWBRZGVA7CMHBEGSXAHFQPXLATKVSRWCAK5YQIYVZK7QJO3U5ZEA, seg_pos={0 0}, upload=133220]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: read tcp 188.166.225.170:52578->145.79.227.18:4005: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:21:37.012760118 (151h17m35.564230579s) [noise=true , address=145.79.227.23:4546 , piece_id=I7U4GEI2HGXU7UTN7E4APJGRRBFASQ4DCZ32K32BGOENKHCEMPMA, seg_pos={0 0}, upload=133220]: Done uploading piece. err: ecclient: upload failed (node:1ZMTFpqfMjeVWXJuvsGdbGkMtGQn6axUZFoU1N9sK6fnbPcMy5, address:145.79.227.23:4546): protocol: read tcp 188.166.225.170:50436->145.79.227.23:4546: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:21:37.012942445 (151h17m35.564412896s) [noise=true , address=145.79.227.14:4839 , piece_id=E2IGX3AZ5HDJYSUZW426L3CJIJAPTFNU7GXICPRN2CBDLLW4U35Q, seg_pos={0 0}, upload=133220]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: read tcp 188.166.225.170:50180->145.79.227.14:4839: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:21:37.012979546 (151h17m35.564449997s) [seg_pos={0 0}, upload=133220]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil> Nov 15 15:21:37.270684105 (151h17m35.822154567s) [upload=133220]: Done uploading segments... Nov 15 15:21:37.270724753 (151h17m35.822195214s) [upload=133220]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4) Duration: 6.231300066s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 5 out of 60 is failed (threshold=6) [X.XXXX......................................................] duration: 3.550084945s trace: https://jaeger.dev.storj.io/trace/42d763a000d5a90d FAIL (2025-11-15 15:25:30 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.550103701s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........X.........................................X.......] duration: 6.133642485s trace: https://jaeger.dev.storj.io/trace/46f6435aaf1b852a FAIL (2025-11-15 15:24:57 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 4) ======LOGS====== 6MZEKN7XOL667LYJSD7HHCBJIUGFBEYQFOQ, seg_pos={0 0}, upload=133220]: Uploading piece... Nov 15 15:21:08.018428616 (151h17m6.569899077s) [noise=true , address=45.91.116.11:4118 , piece_id=J36EGQPIQCW7CCGC4SKQBUVK7QC4UNIGBDBPYAB3A7TSYV377PDA, seg_pos={0 0}, upload=133220]: Uploading piece... Nov 15 15:21:08.018661680 (151h17m6.570132141s) [noise=true , address=145.79.227.26:4654 , piece_id=QZPWDI7TKCW3OGH36VQQH7EBQJQU2KGJZJ3Y5VZHVEMEDEOXPARA, seg_pos={0 0}, upload=133220]: Uploading piece... Nov 15 15:21:08.018882460 (151h17m6.570352921s) [noise=true , address=145.79.227.18:4005 , piece_id=ZWBRZGVA7CMHBEGSXAHFQPXLATKVSRWCAK5YQIYVZK7QJO3U5ZEA, seg_pos={0 0}, upload=133220]: Uploading piece... Nov 15 15:21:08.019281930 (151h17m6.570752391s) [noise=true , address=45.91.116.8:4045 , piece_id=FVDTJ3HTQGVPMVLV2656OCTNZT2Z3ZPFNC2C5F6GFHIDOAYLM3BA, seg_pos={0 0}, upload=133220]: Uploading piece... Nov 15 15:21:08.019316877 (151h17m6.570787338s) [noise=true , address=145.79.227.32:4836 , piece_id=2A5BU3NYUBVIEL62CAOPMBQEU7T5BDW5QOXVF3AZQFA2YU7VMLKQ, seg_pos={0 0}, upload=133220]: Uploading piece... Nov 15 15:21:08.019832619 (151h17m6.57130308s) [noise=true , address=45.91.116.5:4299 , piece_id=Z2XDDTQKI67WJ62RDWXNP76OITD3JVWIFPNBD5UYKT5ZP3J53UXA, seg_pos={0 0}, upload=133220]: Uploading piece... Nov 15 15:21:08.021202999 (151h17m6.57267347s) [noise=true , address=145.79.227.9:4365 , piece_id=5VNCHNGBR5XDMYFGRUVQ3SDACVDWY2DTVRUKVALTO7J6O6TVFP5A, seg_pos={0 0}, upload=133220]: Uploading piece... Nov 15 15:21:08.021959260 (151h17m6.573429721s) [noise=true , address=145.79.227.11:4729 , piece_id=FO73QHKPLIJZFWKCZZIJE33HKR2JQAMDVAAXFV7MZCZFL2D63U4A, seg_pos={0 0}, upload=133220]: Uploading piece... Nov 15 15:21:08.035442264 (151h17m6.586912725s) [noise=true , address=145.79.227.23:4546 , piece_id=I7U4GEI2HGXU7UTN7E4APJGRRBFASQ4DCZ32K32BGOENKHCEMPMA, seg_pos={0 0}, upload=133220]: Uploading piece... Nov 15 15:21:08.041195330 (151h17m6.592665811s) [noise=true , address=145.79.227.31:4798 , piece_id=YF6ZJD2LITIDCZAOJQQ2XVZIEGTDVCY7ZENSMWT6QRQU4R3QEGWQ, seg_pos={0 0}, upload=133220]: Uploading piece... Nov 15 15:21:09.871225261 (151h17m8.422695722s) [upload=133220]: Next returned nil segment Nov 15 15:21:09.871243576 (151h17m8.422714037s) [upload=133220]: Waiting for error group managing segments... Nov 15 15:21:10.165867146 (151h17m8.717337638s) [noise=true , address=77.90.31.19:4843 , piece_id=M47NOUZRG6LHAHBF26MZEKN7XOL667LYJSD7HHCBJIUGFBEYQFOQ, seg_pos={0 0}, upload=133220]: Done uploading piece. err: <nil> Nov 15 15:21:10.323410936 (151h17m8.874881397s) [noise=true , address=45.91.116.5:4299 , piece_id=Z2XDDTQKI67WJ62RDWXNP76OITD3JVWIFPNBD5UYKT5ZP3J53UXA, seg_pos={0 0}, upload=133220]: Done uploading piece. err: <nil> Nov 15 15:21:10.404481083 (151h17m8.955951544s) [noise=true , address=45.91.116.11:4118 , piece_id=J36EGQPIQCW7CCGC4SKQBUVK7QC4UNIGBDBPYAB3A7TSYV377PDA, seg_pos={0 0}, upload=133220]: Done uploading piece. err: <nil> Nov 15 15:21:11.440265666 (151h17m9.991736127s) [noise=true , address=45.91.116.8:4045 , piece_id=FVDTJ3HTQGVPMVLV2656OCTNZT2Z3ZPFNC2C5F6GFHIDOAYLM3BA, seg_pos={0 0}, upload=133220]: Done uploading piece. err: <nil> Nov 15 15:21:37.011608695 (151h17m35.563079156s) [noise=true , address=145.79.227.31:4798 , piece_id=YF6ZJD2LITIDCZAOJQQ2XVZIEGTDVCY7ZENSMWT6QRQU4R3QEGWQ, seg_pos={0 0}, upload=133220]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:145.79.227.31:4798): protocol: write tcp 188.166.225.170:45634->145.79.227.31:4798: use of closed network connection; write tcp 188.166.225.170:45634->145.79.227.31:4798: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45634->145.79.227.31:4798: use of closed network connection Nov 15 15:21:37.011777536 (151h17m35.563247997s) [noise=true , address=145.79.227.32:4836 , piece_id=2A5BU3NYUBVIEL62CAOPMBQEU7T5BDW5QOXVF3AZQFA2YU7VMLKQ, seg_pos={0 0}, upload=133220]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: write tcp 188.166.225.170:45832->145.79.227.32:4836: use of closed network connection; write tcp 188.166.225.170:45832->145.79.227.32:4836: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45832->145.79.227.32:4836: use of closed network connection Nov 15 15:21:37.011812132 (151h17m35.563282593s) [noise=true , address=145.79.227.30:4763 , piece_id=5HETJEEXU6DMLLAXICVF4A42YZHKQEVQ7AOKV734F7JRCIMDLTRQ, seg_pos={0 0}, upload=133220]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: read tcp 188.166.225.170:36386->145.79.227.30:4763: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:21:37.011874541 (151h17m35.563345002s) [noise=true , address=145.79.227.11:4729 , piece_id=FO73QHKPLIJZFWKCZZIJE33HKR2JQAMDVAAXFV7MZCZFL2D63U4A, seg_pos={0 0}, upload=133220]: Done uploading piece. err: ecclient: upload failed (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR, address:145.79.227.11:4729): protocol: write tcp 188.166.225.170:38964->145.79.227.11:4729: use of closed network connection; write tcp 188.166.225.170:38964->145.79.227.11:4729: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38964->145.79.227.11:4729: use of closed network connection Nov 15 15:21:37.011995582 (151h17m35.563466043s) [noise=true , address=145.79.227.12:4767 , piece_id=MZHSGDCEHU55EAMMFF6Z2XVRPNWGBKL3WLG2RTYG7QQTMW6UBKXA, seg_pos={0 0}, upload=133220]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:145.79.227.12:4767): protocol: write tcp 188.166.225.170:57008->145.79.227.12:4767: use of closed network connection; write tcp 188.166.225.170:57008->145.79.227.12:4767: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57008->145.79.227.12:4767: use of closed network connection Nov 15 15:21:37.012092546 (151h17m35.563563017s) [noise=true , address=145.79.227.28:4727 , piece_id=C4HI5WNAVBZ6OWWVF5WBMLJ4YDU6WJDLZ5ZXURAFV7WQ5H6F44BA, seg_pos={0 0}, upload=133220]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: read tcp 188.166.225.170:46810->145.79.227.28:4727: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:21:37.012296374 (151h17m35.563766835s) [noise=true , address=145.79.227.27:4689 , piece_id=KPQUJYT543MTIOZA653A74F2VDCGQAJ5QQV64AWMIEMKGRMUYQBQ, seg_pos={0 0}, upload=133220]: Done uploading piece. err: ecclient: upload failed (node:1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, address:145.79.227.27:4689): protocol: read tcp 188.166.225.170:49674->145.79.227.27:4689: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:21:37.012338324 (151h17m35.563808825s) [noise=true , address=145.79.227.9:4365 , piece_id=5VNCHNGBR5XDMYFGRUVQ3SDACVDWY2DTVRUKVALTO7J6O6TVFP5A, seg_pos={0 0}, upload=133220]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: read tcp 188.166.225.170:54022->145.79.227.9:4365: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:21:37.012485495 (151h17m35.563955956s) [noise=true , address=145.79.227.26:4654 , piece_id=QZPWDI7TKCW3OGH36VQQH7EBQJQU2KGJZJ3Y5VZHVEMEDEOXPARA, seg_pos={0 0}, upload=133220]: Done uploading piece. err: ecclient: upload failed (node:1KhBWMerZe9tUmzdVEYkBb6NmJ25dGP26Ea7CLjNJT5gdAeiah, address:145.79.227.26:4654): protocol: read tcp 188.166.225.170:53342->145.79.227.26:4654: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:21:37.012542613 (151h17m35.564013064s) [noise=true , address=145.79.227.18:4005 , piece_id=ZWBRZGVA7CMHBEGSXAHFQPXLATKVSRWCAK5YQIYVZK7QJO3U5ZEA, seg_pos={0 0}, upload=133220]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: read tcp 188.166.225.170:52578->145.79.227.18:4005: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:21:37.012760118 (151h17m35.564230579s) [noise=true , address=145.79.227.23:4546 , piece_id=I7U4GEI2HGXU7UTN7E4APJGRRBFASQ4DCZ32K32BGOENKHCEMPMA, seg_pos={0 0}, upload=133220]: Done uploading piece. err: ecclient: upload failed (node:1ZMTFpqfMjeVWXJuvsGdbGkMtGQn6axUZFoU1N9sK6fnbPcMy5, address:145.79.227.23:4546): protocol: read tcp 188.166.225.170:50436->145.79.227.23:4546: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:21:37.012942445 (151h17m35.564412896s) [noise=true , address=145.79.227.14:4839 , piece_id=E2IGX3AZ5HDJYSUZW426L3CJIJAPTFNU7GXICPRN2CBDLLW4U35Q, seg_pos={0 0}, upload=133220]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: read tcp 188.166.225.170:50180->145.79.227.14:4839: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:21:37.012979546 (151h17m35.564449997s) [seg_pos={0 0}, upload=133220]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil> Nov 15 15:21:37.270684105 (151h17m35.822154567s) [upload=133220]: Done uploading segments... Nov 15 15:21:37.270724753 (151h17m35.822195214s) [upload=133220]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4) Duration: 6.133685658s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [XX.XXXX.....................................................] duration: 3.52816863s trace: https://jaeger.dev.storj.io/trace/42d763a000d5a90d FAIL (2025-11-15 15:24:47 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.528201472s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...........X.........................................X......] duration: 6.226054707s trace: https://jaeger.dev.storj.io/trace/46f6435aaf1b852a FAIL (2025-11-15 15:24:38 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 4) ======LOGS====== 6MZEKN7XOL667LYJSD7HHCBJIUGFBEYQFOQ, seg_pos={0 0}, upload=133220]: Uploading piece... Nov 15 15:21:08.018428616 (151h17m6.569899077s) [noise=true , address=45.91.116.11:4118 , piece_id=J36EGQPIQCW7CCGC4SKQBUVK7QC4UNIGBDBPYAB3A7TSYV377PDA, seg_pos={0 0}, upload=133220]: Uploading piece... Nov 15 15:21:08.018661680 (151h17m6.570132141s) [noise=true , address=145.79.227.26:4654 , piece_id=QZPWDI7TKCW3OGH36VQQH7EBQJQU2KGJZJ3Y5VZHVEMEDEOXPARA, seg_pos={0 0}, upload=133220]: Uploading piece... Nov 15 15:21:08.018882460 (151h17m6.570352921s) [noise=true , address=145.79.227.18:4005 , piece_id=ZWBRZGVA7CMHBEGSXAHFQPXLATKVSRWCAK5YQIYVZK7QJO3U5ZEA, seg_pos={0 0}, upload=133220]: Uploading piece... Nov 15 15:21:08.019281930 (151h17m6.570752391s) [noise=true , address=45.91.116.8:4045 , piece_id=FVDTJ3HTQGVPMVLV2656OCTNZT2Z3ZPFNC2C5F6GFHIDOAYLM3BA, seg_pos={0 0}, upload=133220]: Uploading piece... Nov 15 15:21:08.019316877 (151h17m6.570787338s) [noise=true , address=145.79.227.32:4836 , piece_id=2A5BU3NYUBVIEL62CAOPMBQEU7T5BDW5QOXVF3AZQFA2YU7VMLKQ, seg_pos={0 0}, upload=133220]: Uploading piece... Nov 15 15:21:08.019832619 (151h17m6.57130308s) [noise=true , address=45.91.116.5:4299 , piece_id=Z2XDDTQKI67WJ62RDWXNP76OITD3JVWIFPNBD5UYKT5ZP3J53UXA, seg_pos={0 0}, upload=133220]: Uploading piece... Nov 15 15:21:08.021202999 (151h17m6.57267347s) [noise=true , address=145.79.227.9:4365 , piece_id=5VNCHNGBR5XDMYFGRUVQ3SDACVDWY2DTVRUKVALTO7J6O6TVFP5A, seg_pos={0 0}, upload=133220]: Uploading piece... Nov 15 15:21:08.021959260 (151h17m6.573429721s) [noise=true , address=145.79.227.11:4729 , piece_id=FO73QHKPLIJZFWKCZZIJE33HKR2JQAMDVAAXFV7MZCZFL2D63U4A, seg_pos={0 0}, upload=133220]: Uploading piece... Nov 15 15:21:08.035442264 (151h17m6.586912725s) [noise=true , address=145.79.227.23:4546 , piece_id=I7U4GEI2HGXU7UTN7E4APJGRRBFASQ4DCZ32K32BGOENKHCEMPMA, seg_pos={0 0}, upload=133220]: Uploading piece... Nov 15 15:21:08.041195330 (151h17m6.592665811s) [noise=true , address=145.79.227.31:4798 , piece_id=YF6ZJD2LITIDCZAOJQQ2XVZIEGTDVCY7ZENSMWT6QRQU4R3QEGWQ, seg_pos={0 0}, upload=133220]: Uploading piece... Nov 15 15:21:09.871225261 (151h17m8.422695722s) [upload=133220]: Next returned nil segment Nov 15 15:21:09.871243576 (151h17m8.422714037s) [upload=133220]: Waiting for error group managing segments... Nov 15 15:21:10.165867146 (151h17m8.717337638s) [noise=true , address=77.90.31.19:4843 , piece_id=M47NOUZRG6LHAHBF26MZEKN7XOL667LYJSD7HHCBJIUGFBEYQFOQ, seg_pos={0 0}, upload=133220]: Done uploading piece. err: <nil> Nov 15 15:21:10.323410936 (151h17m8.874881397s) [noise=true , address=45.91.116.5:4299 , piece_id=Z2XDDTQKI67WJ62RDWXNP76OITD3JVWIFPNBD5UYKT5ZP3J53UXA, seg_pos={0 0}, upload=133220]: Done uploading piece. err: <nil> Nov 15 15:21:10.404481083 (151h17m8.955951544s) [noise=true , address=45.91.116.11:4118 , piece_id=J36EGQPIQCW7CCGC4SKQBUVK7QC4UNIGBDBPYAB3A7TSYV377PDA, seg_pos={0 0}, upload=133220]: Done uploading piece. err: <nil> Nov 15 15:21:11.440265666 (151h17m9.991736127s) [noise=true , address=45.91.116.8:4045 , piece_id=FVDTJ3HTQGVPMVLV2656OCTNZT2Z3ZPFNC2C5F6GFHIDOAYLM3BA, seg_pos={0 0}, upload=133220]: Done uploading piece. err: <nil> Nov 15 15:21:37.011608695 (151h17m35.563079156s) [noise=true , address=145.79.227.31:4798 , piece_id=YF6ZJD2LITIDCZAOJQQ2XVZIEGTDVCY7ZENSMWT6QRQU4R3QEGWQ, seg_pos={0 0}, upload=133220]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:145.79.227.31:4798): protocol: write tcp 188.166.225.170:45634->145.79.227.31:4798: use of closed network connection; write tcp 188.166.225.170:45634->145.79.227.31:4798: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45634->145.79.227.31:4798: use of closed network connection Nov 15 15:21:37.011777536 (151h17m35.563247997s) [noise=true , address=145.79.227.32:4836 , piece_id=2A5BU3NYUBVIEL62CAOPMBQEU7T5BDW5QOXVF3AZQFA2YU7VMLKQ, seg_pos={0 0}, upload=133220]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: write tcp 188.166.225.170:45832->145.79.227.32:4836: use of closed network connection; write tcp 188.166.225.170:45832->145.79.227.32:4836: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45832->145.79.227.32:4836: use of closed network connection Nov 15 15:21:37.011812132 (151h17m35.563282593s) [noise=true , address=145.79.227.30:4763 , piece_id=5HETJEEXU6DMLLAXICVF4A42YZHKQEVQ7AOKV734F7JRCIMDLTRQ, seg_pos={0 0}, upload=133220]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: read tcp 188.166.225.170:36386->145.79.227.30:4763: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:21:37.011874541 (151h17m35.563345002s) [noise=true , address=145.79.227.11:4729 , piece_id=FO73QHKPLIJZFWKCZZIJE33HKR2JQAMDVAAXFV7MZCZFL2D63U4A, seg_pos={0 0}, upload=133220]: Done uploading piece. err: ecclient: upload failed (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR, address:145.79.227.11:4729): protocol: write tcp 188.166.225.170:38964->145.79.227.11:4729: use of closed network connection; write tcp 188.166.225.170:38964->145.79.227.11:4729: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38964->145.79.227.11:4729: use of closed network connection Nov 15 15:21:37.011995582 (151h17m35.563466043s) [noise=true , address=145.79.227.12:4767 , piece_id=MZHSGDCEHU55EAMMFF6Z2XVRPNWGBKL3WLG2RTYG7QQTMW6UBKXA, seg_pos={0 0}, upload=133220]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:145.79.227.12:4767): protocol: write tcp 188.166.225.170:57008->145.79.227.12:4767: use of closed network connection; write tcp 188.166.225.170:57008->145.79.227.12:4767: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57008->145.79.227.12:4767: use of closed network connection Nov 15 15:21:37.012092546 (151h17m35.563563017s) [noise=true , address=145.79.227.28:4727 , piece_id=C4HI5WNAVBZ6OWWVF5WBMLJ4YDU6WJDLZ5ZXURAFV7WQ5H6F44BA, seg_pos={0 0}, upload=133220]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: read tcp 188.166.225.170:46810->145.79.227.28:4727: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:21:37.012296374 (151h17m35.563766835s) [noise=true , address=145.79.227.27:4689 , piece_id=KPQUJYT543MTIOZA653A74F2VDCGQAJ5QQV64AWMIEMKGRMUYQBQ, seg_pos={0 0}, upload=133220]: Done uploading piece. err: ecclient: upload failed (node:1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, address:145.79.227.27:4689): protocol: read tcp 188.166.225.170:49674->145.79.227.27:4689: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:21:37.012338324 (151h17m35.563808825s) [noise=true , address=145.79.227.9:4365 , piece_id=5VNCHNGBR5XDMYFGRUVQ3SDACVDWY2DTVRUKVALTO7J6O6TVFP5A, seg_pos={0 0}, upload=133220]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: read tcp 188.166.225.170:54022->145.79.227.9:4365: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:21:37.012485495 (151h17m35.563955956s) [noise=true , address=145.79.227.26:4654 , piece_id=QZPWDI7TKCW3OGH36VQQH7EBQJQU2KGJZJ3Y5VZHVEMEDEOXPARA, seg_pos={0 0}, upload=133220]: Done uploading piece. err: ecclient: upload failed (node:1KhBWMerZe9tUmzdVEYkBb6NmJ25dGP26Ea7CLjNJT5gdAeiah, address:145.79.227.26:4654): protocol: read tcp 188.166.225.170:53342->145.79.227.26:4654: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:21:37.012542613 (151h17m35.564013064s) [noise=true , address=145.79.227.18:4005 , piece_id=ZWBRZGVA7CMHBEGSXAHFQPXLATKVSRWCAK5YQIYVZK7QJO3U5ZEA, seg_pos={0 0}, upload=133220]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: read tcp 188.166.225.170:52578->145.79.227.18:4005: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:21:37.012760118 (151h17m35.564230579s) [noise=true , address=145.79.227.23:4546 , piece_id=I7U4GEI2HGXU7UTN7E4APJGRRBFASQ4DCZ32K32BGOENKHCEMPMA, seg_pos={0 0}, upload=133220]: Done uploading piece. err: ecclient: upload failed (node:1ZMTFpqfMjeVWXJuvsGdbGkMtGQn6axUZFoU1N9sK6fnbPcMy5, address:145.79.227.23:4546): protocol: read tcp 188.166.225.170:50436->145.79.227.23:4546: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:21:37.012942445 (151h17m35.564412896s) [noise=true , address=145.79.227.14:4839 , piece_id=E2IGX3AZ5HDJYSUZW426L3CJIJAPTFNU7GXICPRN2CBDLLW4U35Q, seg_pos={0 0}, upload=133220]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: read tcp 188.166.225.170:50180->145.79.227.14:4839: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:21:37.012979546 (151h17m35.564449997s) [seg_pos={0 0}, upload=133220]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil> Nov 15 15:21:37.270684105 (151h17m35.822154567s) [upload=133220]: Done uploading segments... Nov 15 15:21:37.270724753 (151h17m35.822195214s) [upload=133220]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4) Duration: 6.226079263s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [............X.........................................X.....] duration: 5.999076499s trace: https://jaeger.dev.storj.io/trace/46f6435aaf1b852a FAIL (2025-11-15 15:23:55 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 4) ======LOGS====== 6MZEKN7XOL667LYJSD7HHCBJIUGFBEYQFOQ, seg_pos={0 0}, upload=133220]: Uploading piece... Nov 15 15:21:08.018428616 (151h17m6.569899077s) [noise=true , address=45.91.116.11:4118 , piece_id=J36EGQPIQCW7CCGC4SKQBUVK7QC4UNIGBDBPYAB3A7TSYV377PDA, seg_pos={0 0}, upload=133220]: Uploading piece... Nov 15 15:21:08.018661680 (151h17m6.570132141s) [noise=true , address=145.79.227.26:4654 , piece_id=QZPWDI7TKCW3OGH36VQQH7EBQJQU2KGJZJ3Y5VZHVEMEDEOXPARA, seg_pos={0 0}, upload=133220]: Uploading piece... Nov 15 15:21:08.018882460 (151h17m6.570352921s) [noise=true , address=145.79.227.18:4005 , piece_id=ZWBRZGVA7CMHBEGSXAHFQPXLATKVSRWCAK5YQIYVZK7QJO3U5ZEA, seg_pos={0 0}, upload=133220]: Uploading piece... Nov 15 15:21:08.019281930 (151h17m6.570752391s) [noise=true , address=45.91.116.8:4045 , piece_id=FVDTJ3HTQGVPMVLV2656OCTNZT2Z3ZPFNC2C5F6GFHIDOAYLM3BA, seg_pos={0 0}, upload=133220]: Uploading piece... Nov 15 15:21:08.019316877 (151h17m6.570787338s) [noise=true , address=145.79.227.32:4836 , piece_id=2A5BU3NYUBVIEL62CAOPMBQEU7T5BDW5QOXVF3AZQFA2YU7VMLKQ, seg_pos={0 0}, upload=133220]: Uploading piece... Nov 15 15:21:08.019832619 (151h17m6.57130308s) [noise=true , address=45.91.116.5:4299 , piece_id=Z2XDDTQKI67WJ62RDWXNP76OITD3JVWIFPNBD5UYKT5ZP3J53UXA, seg_pos={0 0}, upload=133220]: Uploading piece... Nov 15 15:21:08.021202999 (151h17m6.57267347s) [noise=true , address=145.79.227.9:4365 , piece_id=5VNCHNGBR5XDMYFGRUVQ3SDACVDWY2DTVRUKVALTO7J6O6TVFP5A, seg_pos={0 0}, upload=133220]: Uploading piece... Nov 15 15:21:08.021959260 (151h17m6.573429721s) [noise=true , address=145.79.227.11:4729 , piece_id=FO73QHKPLIJZFWKCZZIJE33HKR2JQAMDVAAXFV7MZCZFL2D63U4A, seg_pos={0 0}, upload=133220]: Uploading piece... Nov 15 15:21:08.035442264 (151h17m6.586912725s) [noise=true , address=145.79.227.23:4546 , piece_id=I7U4GEI2HGXU7UTN7E4APJGRRBFASQ4DCZ32K32BGOENKHCEMPMA, seg_pos={0 0}, upload=133220]: Uploading piece... Nov 15 15:21:08.041195330 (151h17m6.592665811s) [noise=true , address=145.79.227.31:4798 , piece_id=YF6ZJD2LITIDCZAOJQQ2XVZIEGTDVCY7ZENSMWT6QRQU4R3QEGWQ, seg_pos={0 0}, upload=133220]: Uploading piece... Nov 15 15:21:09.871225261 (151h17m8.422695722s) [upload=133220]: Next returned nil segment Nov 15 15:21:09.871243576 (151h17m8.422714037s) [upload=133220]: Waiting for error group managing segments... Nov 15 15:21:10.165867146 (151h17m8.717337638s) [noise=true , address=77.90.31.19:4843 , piece_id=M47NOUZRG6LHAHBF26MZEKN7XOL667LYJSD7HHCBJIUGFBEYQFOQ, seg_pos={0 0}, upload=133220]: Done uploading piece. err: <nil> Nov 15 15:21:10.323410936 (151h17m8.874881397s) [noise=true , address=45.91.116.5:4299 , piece_id=Z2XDDTQKI67WJ62RDWXNP76OITD3JVWIFPNBD5UYKT5ZP3J53UXA, seg_pos={0 0}, upload=133220]: Done uploading piece. err: <nil> Nov 15 15:21:10.404481083 (151h17m8.955951544s) [noise=true , address=45.91.116.11:4118 , piece_id=J36EGQPIQCW7CCGC4SKQBUVK7QC4UNIGBDBPYAB3A7TSYV377PDA, seg_pos={0 0}, upload=133220]: Done uploading piece. err: <nil> Nov 15 15:21:11.440265666 (151h17m9.991736127s) [noise=true , address=45.91.116.8:4045 , piece_id=FVDTJ3HTQGVPMVLV2656OCTNZT2Z3ZPFNC2C5F6GFHIDOAYLM3BA, seg_pos={0 0}, upload=133220]: Done uploading piece. err: <nil> Nov 15 15:21:37.011608695 (151h17m35.563079156s) [noise=true , address=145.79.227.31:4798 , piece_id=YF6ZJD2LITIDCZAOJQQ2XVZIEGTDVCY7ZENSMWT6QRQU4R3QEGWQ, seg_pos={0 0}, upload=133220]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:145.79.227.31:4798): protocol: write tcp 188.166.225.170:45634->145.79.227.31:4798: use of closed network connection; write tcp 188.166.225.170:45634->145.79.227.31:4798: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45634->145.79.227.31:4798: use of closed network connection Nov 15 15:21:37.011777536 (151h17m35.563247997s) [noise=true , address=145.79.227.32:4836 , piece_id=2A5BU3NYUBVIEL62CAOPMBQEU7T5BDW5QOXVF3AZQFA2YU7VMLKQ, seg_pos={0 0}, upload=133220]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: write tcp 188.166.225.170:45832->145.79.227.32:4836: use of closed network connection; write tcp 188.166.225.170:45832->145.79.227.32:4836: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45832->145.79.227.32:4836: use of closed network connection Nov 15 15:21:37.011812132 (151h17m35.563282593s) [noise=true , address=145.79.227.30:4763 , piece_id=5HETJEEXU6DMLLAXICVF4A42YZHKQEVQ7AOKV734F7JRCIMDLTRQ, seg_pos={0 0}, upload=133220]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: read tcp 188.166.225.170:36386->145.79.227.30:4763: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:21:37.011874541 (151h17m35.563345002s) [noise=true , address=145.79.227.11:4729 , piece_id=FO73QHKPLIJZFWKCZZIJE33HKR2JQAMDVAAXFV7MZCZFL2D63U4A, seg_pos={0 0}, upload=133220]: Done uploading piece. err: ecclient: upload failed (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR, address:145.79.227.11:4729): protocol: write tcp 188.166.225.170:38964->145.79.227.11:4729: use of closed network connection; write tcp 188.166.225.170:38964->145.79.227.11:4729: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38964->145.79.227.11:4729: use of closed network connection Nov 15 15:21:37.011995582 (151h17m35.563466043s) [noise=true , address=145.79.227.12:4767 , piece_id=MZHSGDCEHU55EAMMFF6Z2XVRPNWGBKL3WLG2RTYG7QQTMW6UBKXA, seg_pos={0 0}, upload=133220]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:145.79.227.12:4767): protocol: write tcp 188.166.225.170:57008->145.79.227.12:4767: use of closed network connection; write tcp 188.166.225.170:57008->145.79.227.12:4767: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57008->145.79.227.12:4767: use of closed network connection Nov 15 15:21:37.012092546 (151h17m35.563563017s) [noise=true , address=145.79.227.28:4727 , piece_id=C4HI5WNAVBZ6OWWVF5WBMLJ4YDU6WJDLZ5ZXURAFV7WQ5H6F44BA, seg_pos={0 0}, upload=133220]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: read tcp 188.166.225.170:46810->145.79.227.28:4727: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:21:37.012296374 (151h17m35.563766835s) [noise=true , address=145.79.227.27:4689 , piece_id=KPQUJYT543MTIOZA653A74F2VDCGQAJ5QQV64AWMIEMKGRMUYQBQ, seg_pos={0 0}, upload=133220]: Done uploading piece. err: ecclient: upload failed (node:1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, address:145.79.227.27:4689): protocol: read tcp 188.166.225.170:49674->145.79.227.27:4689: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:21:37.012338324 (151h17m35.563808825s) [noise=true , address=145.79.227.9:4365 , piece_id=5VNCHNGBR5XDMYFGRUVQ3SDACVDWY2DTVRUKVALTO7J6O6TVFP5A, seg_pos={0 0}, upload=133220]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: read tcp 188.166.225.170:54022->145.79.227.9:4365: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:21:37.012485495 (151h17m35.563955956s) [noise=true , address=145.79.227.26:4654 , piece_id=QZPWDI7TKCW3OGH36VQQH7EBQJQU2KGJZJ3Y5VZHVEMEDEOXPARA, seg_pos={0 0}, upload=133220]: Done uploading piece. err: ecclient: upload failed (node:1KhBWMerZe9tUmzdVEYkBb6NmJ25dGP26Ea7CLjNJT5gdAeiah, address:145.79.227.26:4654): protocol: read tcp 188.166.225.170:53342->145.79.227.26:4654: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:21:37.012542613 (151h17m35.564013064s) [noise=true , address=145.79.227.18:4005 , piece_id=ZWBRZGVA7CMHBEGSXAHFQPXLATKVSRWCAK5YQIYVZK7QJO3U5ZEA, seg_pos={0 0}, upload=133220]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: read tcp 188.166.225.170:52578->145.79.227.18:4005: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:21:37.012760118 (151h17m35.564230579s) [noise=true , address=145.79.227.23:4546 , piece_id=I7U4GEI2HGXU7UTN7E4APJGRRBFASQ4DCZ32K32BGOENKHCEMPMA, seg_pos={0 0}, upload=133220]: Done uploading piece. err: ecclient: upload failed (node:1ZMTFpqfMjeVWXJuvsGdbGkMtGQn6axUZFoU1N9sK6fnbPcMy5, address:145.79.227.23:4546): protocol: read tcp 188.166.225.170:50436->145.79.227.23:4546: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:21:37.012942445 (151h17m35.564412896s) [noise=true , address=145.79.227.14:4839 , piece_id=E2IGX3AZ5HDJYSUZW426L3CJIJAPTFNU7GXICPRN2CBDLLW4U35Q, seg_pos={0 0}, upload=133220]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: read tcp 188.166.225.170:50180->145.79.227.14:4839: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:21:37.012979546 (151h17m35.564449997s) [seg_pos={0 0}, upload=133220]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil> Nov 15 15:21:37.270684105 (151h17m35.822154567s) [upload=133220]: Done uploading segments... Nov 15 15:21:37.270724753 (151h17m35.822195214s) [upload=133220]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4) Duration: 5.999105305s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 7 out of 60 is failed (threshold=6) [XXX.XXXX....................................................] duration: 5.000606613s trace: https://jaeger.dev.storj.io/trace/42d763a000d5a90d FAIL (2025-11-15 15:23:54 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000624416s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 7 out of 60 is failed (threshold=6) [.XXX.XXXX...................................................] duration: 4.377757945s trace: https://jaeger.dev.storj.io/trace/42d763a000d5a90d FAIL (2025-11-15 15:23:37 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.377785738s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.............X.........................................X....] duration: 6.043460877s trace: https://jaeger.dev.storj.io/trace/46f6435aaf1b852a FAIL (2025-11-15 15:23:26 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 4) ======LOGS====== 6MZEKN7XOL667LYJSD7HHCBJIUGFBEYQFOQ, seg_pos={0 0}, upload=133220]: Uploading piece... Nov 15 15:21:08.018428616 (151h17m6.569899077s) [noise=true , address=45.91.116.11:4118 , piece_id=J36EGQPIQCW7CCGC4SKQBUVK7QC4UNIGBDBPYAB3A7TSYV377PDA, seg_pos={0 0}, upload=133220]: Uploading piece... Nov 15 15:21:08.018661680 (151h17m6.570132141s) [noise=true , address=145.79.227.26:4654 , piece_id=QZPWDI7TKCW3OGH36VQQH7EBQJQU2KGJZJ3Y5VZHVEMEDEOXPARA, seg_pos={0 0}, upload=133220]: Uploading piece... Nov 15 15:21:08.018882460 (151h17m6.570352921s) [noise=true , address=145.79.227.18:4005 , piece_id=ZWBRZGVA7CMHBEGSXAHFQPXLATKVSRWCAK5YQIYVZK7QJO3U5ZEA, seg_pos={0 0}, upload=133220]: Uploading piece... Nov 15 15:21:08.019281930 (151h17m6.570752391s) [noise=true , address=45.91.116.8:4045 , piece_id=FVDTJ3HTQGVPMVLV2656OCTNZT2Z3ZPFNC2C5F6GFHIDOAYLM3BA, seg_pos={0 0}, upload=133220]: Uploading piece... Nov 15 15:21:08.019316877 (151h17m6.570787338s) [noise=true , address=145.79.227.32:4836 , piece_id=2A5BU3NYUBVIEL62CAOPMBQEU7T5BDW5QOXVF3AZQFA2YU7VMLKQ, seg_pos={0 0}, upload=133220]: Uploading piece... Nov 15 15:21:08.019832619 (151h17m6.57130308s) [noise=true , address=45.91.116.5:4299 , piece_id=Z2XDDTQKI67WJ62RDWXNP76OITD3JVWIFPNBD5UYKT5ZP3J53UXA, seg_pos={0 0}, upload=133220]: Uploading piece... Nov 15 15:21:08.021202999 (151h17m6.57267347s) [noise=true , address=145.79.227.9:4365 , piece_id=5VNCHNGBR5XDMYFGRUVQ3SDACVDWY2DTVRUKVALTO7J6O6TVFP5A, seg_pos={0 0}, upload=133220]: Uploading piece... Nov 15 15:21:08.021959260 (151h17m6.573429721s) [noise=true , address=145.79.227.11:4729 , piece_id=FO73QHKPLIJZFWKCZZIJE33HKR2JQAMDVAAXFV7MZCZFL2D63U4A, seg_pos={0 0}, upload=133220]: Uploading piece... Nov 15 15:21:08.035442264 (151h17m6.586912725s) [noise=true , address=145.79.227.23:4546 , piece_id=I7U4GEI2HGXU7UTN7E4APJGRRBFASQ4DCZ32K32BGOENKHCEMPMA, seg_pos={0 0}, upload=133220]: Uploading piece... Nov 15 15:21:08.041195330 (151h17m6.592665811s) [noise=true , address=145.79.227.31:4798 , piece_id=YF6ZJD2LITIDCZAOJQQ2XVZIEGTDVCY7ZENSMWT6QRQU4R3QEGWQ, seg_pos={0 0}, upload=133220]: Uploading piece... Nov 15 15:21:09.871225261 (151h17m8.422695722s) [upload=133220]: Next returned nil segment Nov 15 15:21:09.871243576 (151h17m8.422714037s) [upload=133220]: Waiting for error group managing segments... Nov 15 15:21:10.165867146 (151h17m8.717337638s) [noise=true , address=77.90.31.19:4843 , piece_id=M47NOUZRG6LHAHBF26MZEKN7XOL667LYJSD7HHCBJIUGFBEYQFOQ, seg_pos={0 0}, upload=133220]: Done uploading piece. err: <nil> Nov 15 15:21:10.323410936 (151h17m8.874881397s) [noise=true , address=45.91.116.5:4299 , piece_id=Z2XDDTQKI67WJ62RDWXNP76OITD3JVWIFPNBD5UYKT5ZP3J53UXA, seg_pos={0 0}, upload=133220]: Done uploading piece. err: <nil> Nov 15 15:21:10.404481083 (151h17m8.955951544s) [noise=true , address=45.91.116.11:4118 , piece_id=J36EGQPIQCW7CCGC4SKQBUVK7QC4UNIGBDBPYAB3A7TSYV377PDA, seg_pos={0 0}, upload=133220]: Done uploading piece. err: <nil> Nov 15 15:21:11.440265666 (151h17m9.991736127s) [noise=true , address=45.91.116.8:4045 , piece_id=FVDTJ3HTQGVPMVLV2656OCTNZT2Z3ZPFNC2C5F6GFHIDOAYLM3BA, seg_pos={0 0}, upload=133220]: Done uploading piece. err: <nil> Nov 15 15:21:37.011608695 (151h17m35.563079156s) [noise=true , address=145.79.227.31:4798 , piece_id=YF6ZJD2LITIDCZAOJQQ2XVZIEGTDVCY7ZENSMWT6QRQU4R3QEGWQ, seg_pos={0 0}, upload=133220]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:145.79.227.31:4798): protocol: write tcp 188.166.225.170:45634->145.79.227.31:4798: use of closed network connection; write tcp 188.166.225.170:45634->145.79.227.31:4798: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45634->145.79.227.31:4798: use of closed network connection Nov 15 15:21:37.011777536 (151h17m35.563247997s) [noise=true , address=145.79.227.32:4836 , piece_id=2A5BU3NYUBVIEL62CAOPMBQEU7T5BDW5QOXVF3AZQFA2YU7VMLKQ, seg_pos={0 0}, upload=133220]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: write tcp 188.166.225.170:45832->145.79.227.32:4836: use of closed network connection; write tcp 188.166.225.170:45832->145.79.227.32:4836: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45832->145.79.227.32:4836: use of closed network connection Nov 15 15:21:37.011812132 (151h17m35.563282593s) [noise=true , address=145.79.227.30:4763 , piece_id=5HETJEEXU6DMLLAXICVF4A42YZHKQEVQ7AOKV734F7JRCIMDLTRQ, seg_pos={0 0}, upload=133220]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: read tcp 188.166.225.170:36386->145.79.227.30:4763: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:21:37.011874541 (151h17m35.563345002s) [noise=true , address=145.79.227.11:4729 , piece_id=FO73QHKPLIJZFWKCZZIJE33HKR2JQAMDVAAXFV7MZCZFL2D63U4A, seg_pos={0 0}, upload=133220]: Done uploading piece. err: ecclient: upload failed (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR, address:145.79.227.11:4729): protocol: write tcp 188.166.225.170:38964->145.79.227.11:4729: use of closed network connection; write tcp 188.166.225.170:38964->145.79.227.11:4729: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38964->145.79.227.11:4729: use of closed network connection Nov 15 15:21:37.011995582 (151h17m35.563466043s) [noise=true , address=145.79.227.12:4767 , piece_id=MZHSGDCEHU55EAMMFF6Z2XVRPNWGBKL3WLG2RTYG7QQTMW6UBKXA, seg_pos={0 0}, upload=133220]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:145.79.227.12:4767): protocol: write tcp 188.166.225.170:57008->145.79.227.12:4767: use of closed network connection; write tcp 188.166.225.170:57008->145.79.227.12:4767: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57008->145.79.227.12:4767: use of closed network connection Nov 15 15:21:37.012092546 (151h17m35.563563017s) [noise=true , address=145.79.227.28:4727 , piece_id=C4HI5WNAVBZ6OWWVF5WBMLJ4YDU6WJDLZ5ZXURAFV7WQ5H6F44BA, seg_pos={0 0}, upload=133220]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: read tcp 188.166.225.170:46810->145.79.227.28:4727: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:21:37.012296374 (151h17m35.563766835s) [noise=true , address=145.79.227.27:4689 , piece_id=KPQUJYT543MTIOZA653A74F2VDCGQAJ5QQV64AWMIEMKGRMUYQBQ, seg_pos={0 0}, upload=133220]: Done uploading piece. err: ecclient: upload failed (node:1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, address:145.79.227.27:4689): protocol: read tcp 188.166.225.170:49674->145.79.227.27:4689: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:21:37.012338324 (151h17m35.563808825s) [noise=true , address=145.79.227.9:4365 , piece_id=5VNCHNGBR5XDMYFGRUVQ3SDACVDWY2DTVRUKVALTO7J6O6TVFP5A, seg_pos={0 0}, upload=133220]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: read tcp 188.166.225.170:54022->145.79.227.9:4365: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:21:37.012485495 (151h17m35.563955956s) [noise=true , address=145.79.227.26:4654 , piece_id=QZPWDI7TKCW3OGH36VQQH7EBQJQU2KGJZJ3Y5VZHVEMEDEOXPARA, seg_pos={0 0}, upload=133220]: Done uploading piece. err: ecclient: upload failed (node:1KhBWMerZe9tUmzdVEYkBb6NmJ25dGP26Ea7CLjNJT5gdAeiah, address:145.79.227.26:4654): protocol: read tcp 188.166.225.170:53342->145.79.227.26:4654: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:21:37.012542613 (151h17m35.564013064s) [noise=true , address=145.79.227.18:4005 , piece_id=ZWBRZGVA7CMHBEGSXAHFQPXLATKVSRWCAK5YQIYVZK7QJO3U5ZEA, seg_pos={0 0}, upload=133220]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: read tcp 188.166.225.170:52578->145.79.227.18:4005: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:21:37.012760118 (151h17m35.564230579s) [noise=true , address=145.79.227.23:4546 , piece_id=I7U4GEI2HGXU7UTN7E4APJGRRBFASQ4DCZ32K32BGOENKHCEMPMA, seg_pos={0 0}, upload=133220]: Done uploading piece. err: ecclient: upload failed (node:1ZMTFpqfMjeVWXJuvsGdbGkMtGQn6axUZFoU1N9sK6fnbPcMy5, address:145.79.227.23:4546): protocol: read tcp 188.166.225.170:50436->145.79.227.23:4546: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:21:37.012942445 (151h17m35.564412896s) [noise=true , address=145.79.227.14:4839 , piece_id=E2IGX3AZ5HDJYSUZW426L3CJIJAPTFNU7GXICPRN2CBDLLW4U35Q, seg_pos={0 0}, upload=133220]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: read tcp 188.166.225.170:50180->145.79.227.14:4839: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:21:37.012979546 (151h17m35.564449997s) [seg_pos={0 0}, upload=133220]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil> Nov 15 15:21:37.270684105 (151h17m35.822154567s) [upload=133220]: Done uploading segments... Nov 15 15:21:37.270724753 (151h17m35.822195214s) [upload=133220]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4) Duration: 6.043493769s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 8 out of 60 is failed (threshold=6) [X.XXX.XXXX..................................................] duration: 3.492516336s trace: https://jaeger.dev.storj.io/trace/42d763a000d5a90d FAIL (2025-11-15 15:22:58 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.492545632s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 8 out of 60 is failed (threshold=6) [.X.XXX.XXXX.................................................] duration: 4.83560604s trace: https://jaeger.dev.storj.io/trace/42d763a000d5a90d FAIL (2025-11-15 15:22:42 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.835671363s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..............X.........................................X...] duration: 6.018358889s trace: https://jaeger.dev.storj.io/trace/46f6435aaf1b852a FAIL (2025-11-15 15:22:38 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 4) ======LOGS====== 6MZEKN7XOL667LYJSD7HHCBJIUGFBEYQFOQ, seg_pos={0 0}, upload=133220]: Uploading piece... Nov 15 15:21:08.018428616 (151h17m6.569899077s) [noise=true , address=45.91.116.11:4118 , piece_id=J36EGQPIQCW7CCGC4SKQBUVK7QC4UNIGBDBPYAB3A7TSYV377PDA, seg_pos={0 0}, upload=133220]: Uploading piece... Nov 15 15:21:08.018661680 (151h17m6.570132141s) [noise=true , address=145.79.227.26:4654 , piece_id=QZPWDI7TKCW3OGH36VQQH7EBQJQU2KGJZJ3Y5VZHVEMEDEOXPARA, seg_pos={0 0}, upload=133220]: Uploading piece... Nov 15 15:21:08.018882460 (151h17m6.570352921s) [noise=true , address=145.79.227.18:4005 , piece_id=ZWBRZGVA7CMHBEGSXAHFQPXLATKVSRWCAK5YQIYVZK7QJO3U5ZEA, seg_pos={0 0}, upload=133220]: Uploading piece... Nov 15 15:21:08.019281930 (151h17m6.570752391s) [noise=true , address=45.91.116.8:4045 , piece_id=FVDTJ3HTQGVPMVLV2656OCTNZT2Z3ZPFNC2C5F6GFHIDOAYLM3BA, seg_pos={0 0}, upload=133220]: Uploading piece... Nov 15 15:21:08.019316877 (151h17m6.570787338s) [noise=true , address=145.79.227.32:4836 , piece_id=2A5BU3NYUBVIEL62CAOPMBQEU7T5BDW5QOXVF3AZQFA2YU7VMLKQ, seg_pos={0 0}, upload=133220]: Uploading piece... Nov 15 15:21:08.019832619 (151h17m6.57130308s) [noise=true , address=45.91.116.5:4299 , piece_id=Z2XDDTQKI67WJ62RDWXNP76OITD3JVWIFPNBD5UYKT5ZP3J53UXA, seg_pos={0 0}, upload=133220]: Uploading piece... Nov 15 15:21:08.021202999 (151h17m6.57267347s) [noise=true , address=145.79.227.9:4365 , piece_id=5VNCHNGBR5XDMYFGRUVQ3SDACVDWY2DTVRUKVALTO7J6O6TVFP5A, seg_pos={0 0}, upload=133220]: Uploading piece... Nov 15 15:21:08.021959260 (151h17m6.573429721s) [noise=true , address=145.79.227.11:4729 , piece_id=FO73QHKPLIJZFWKCZZIJE33HKR2JQAMDVAAXFV7MZCZFL2D63U4A, seg_pos={0 0}, upload=133220]: Uploading piece... Nov 15 15:21:08.035442264 (151h17m6.586912725s) [noise=true , address=145.79.227.23:4546 , piece_id=I7U4GEI2HGXU7UTN7E4APJGRRBFASQ4DCZ32K32BGOENKHCEMPMA, seg_pos={0 0}, upload=133220]: Uploading piece... Nov 15 15:21:08.041195330 (151h17m6.592665811s) [noise=true , address=145.79.227.31:4798 , piece_id=YF6ZJD2LITIDCZAOJQQ2XVZIEGTDVCY7ZENSMWT6QRQU4R3QEGWQ, seg_pos={0 0}, upload=133220]: Uploading piece... Nov 15 15:21:09.871225261 (151h17m8.422695722s) [upload=133220]: Next returned nil segment Nov 15 15:21:09.871243576 (151h17m8.422714037s) [upload=133220]: Waiting for error group managing segments... Nov 15 15:21:10.165867146 (151h17m8.717337638s) [noise=true , address=77.90.31.19:4843 , piece_id=M47NOUZRG6LHAHBF26MZEKN7XOL667LYJSD7HHCBJIUGFBEYQFOQ, seg_pos={0 0}, upload=133220]: Done uploading piece. err: <nil> Nov 15 15:21:10.323410936 (151h17m8.874881397s) [noise=true , address=45.91.116.5:4299 , piece_id=Z2XDDTQKI67WJ62RDWXNP76OITD3JVWIFPNBD5UYKT5ZP3J53UXA, seg_pos={0 0}, upload=133220]: Done uploading piece. err: <nil> Nov 15 15:21:10.404481083 (151h17m8.955951544s) [noise=true , address=45.91.116.11:4118 , piece_id=J36EGQPIQCW7CCGC4SKQBUVK7QC4UNIGBDBPYAB3A7TSYV377PDA, seg_pos={0 0}, upload=133220]: Done uploading piece. err: <nil> Nov 15 15:21:11.440265666 (151h17m9.991736127s) [noise=true , address=45.91.116.8:4045 , piece_id=FVDTJ3HTQGVPMVLV2656OCTNZT2Z3ZPFNC2C5F6GFHIDOAYLM3BA, seg_pos={0 0}, upload=133220]: Done uploading piece. err: <nil> Nov 15 15:21:37.011608695 (151h17m35.563079156s) [noise=true , address=145.79.227.31:4798 , piece_id=YF6ZJD2LITIDCZAOJQQ2XVZIEGTDVCY7ZENSMWT6QRQU4R3QEGWQ, seg_pos={0 0}, upload=133220]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:145.79.227.31:4798): protocol: write tcp 188.166.225.170:45634->145.79.227.31:4798: use of closed network connection; write tcp 188.166.225.170:45634->145.79.227.31:4798: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45634->145.79.227.31:4798: use of closed network connection Nov 15 15:21:37.011777536 (151h17m35.563247997s) [noise=true , address=145.79.227.32:4836 , piece_id=2A5BU3NYUBVIEL62CAOPMBQEU7T5BDW5QOXVF3AZQFA2YU7VMLKQ, seg_pos={0 0}, upload=133220]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: write tcp 188.166.225.170:45832->145.79.227.32:4836: use of closed network connection; write tcp 188.166.225.170:45832->145.79.227.32:4836: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45832->145.79.227.32:4836: use of closed network connection Nov 15 15:21:37.011812132 (151h17m35.563282593s) [noise=true , address=145.79.227.30:4763 , piece_id=5HETJEEXU6DMLLAXICVF4A42YZHKQEVQ7AOKV734F7JRCIMDLTRQ, seg_pos={0 0}, upload=133220]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: read tcp 188.166.225.170:36386->145.79.227.30:4763: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:21:37.011874541 (151h17m35.563345002s) [noise=true , address=145.79.227.11:4729 , piece_id=FO73QHKPLIJZFWKCZZIJE33HKR2JQAMDVAAXFV7MZCZFL2D63U4A, seg_pos={0 0}, upload=133220]: Done uploading piece. err: ecclient: upload failed (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR, address:145.79.227.11:4729): protocol: write tcp 188.166.225.170:38964->145.79.227.11:4729: use of closed network connection; write tcp 188.166.225.170:38964->145.79.227.11:4729: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38964->145.79.227.11:4729: use of closed network connection Nov 15 15:21:37.011995582 (151h17m35.563466043s) [noise=true , address=145.79.227.12:4767 , piece_id=MZHSGDCEHU55EAMMFF6Z2XVRPNWGBKL3WLG2RTYG7QQTMW6UBKXA, seg_pos={0 0}, upload=133220]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:145.79.227.12:4767): protocol: write tcp 188.166.225.170:57008->145.79.227.12:4767: use of closed network connection; write tcp 188.166.225.170:57008->145.79.227.12:4767: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57008->145.79.227.12:4767: use of closed network connection Nov 15 15:21:37.012092546 (151h17m35.563563017s) [noise=true , address=145.79.227.28:4727 , piece_id=C4HI5WNAVBZ6OWWVF5WBMLJ4YDU6WJDLZ5ZXURAFV7WQ5H6F44BA, seg_pos={0 0}, upload=133220]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: read tcp 188.166.225.170:46810->145.79.227.28:4727: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:21:37.012296374 (151h17m35.563766835s) [noise=true , address=145.79.227.27:4689 , piece_id=KPQUJYT543MTIOZA653A74F2VDCGQAJ5QQV64AWMIEMKGRMUYQBQ, seg_pos={0 0}, upload=133220]: Done uploading piece. err: ecclient: upload failed (node:1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, address:145.79.227.27:4689): protocol: read tcp 188.166.225.170:49674->145.79.227.27:4689: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:21:37.012338324 (151h17m35.563808825s) [noise=true , address=145.79.227.9:4365 , piece_id=5VNCHNGBR5XDMYFGRUVQ3SDACVDWY2DTVRUKVALTO7J6O6TVFP5A, seg_pos={0 0}, upload=133220]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: read tcp 188.166.225.170:54022->145.79.227.9:4365: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:21:37.012485495 (151h17m35.563955956s) [noise=true , address=145.79.227.26:4654 , piece_id=QZPWDI7TKCW3OGH36VQQH7EBQJQU2KGJZJ3Y5VZHVEMEDEOXPARA, seg_pos={0 0}, upload=133220]: Done uploading piece. err: ecclient: upload failed (node:1KhBWMerZe9tUmzdVEYkBb6NmJ25dGP26Ea7CLjNJT5gdAeiah, address:145.79.227.26:4654): protocol: read tcp 188.166.225.170:53342->145.79.227.26:4654: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:21:37.012542613 (151h17m35.564013064s) [noise=true , address=145.79.227.18:4005 , piece_id=ZWBRZGVA7CMHBEGSXAHFQPXLATKVSRWCAK5YQIYVZK7QJO3U5ZEA, seg_pos={0 0}, upload=133220]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: read tcp 188.166.225.170:52578->145.79.227.18:4005: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:21:37.012760118 (151h17m35.564230579s) [noise=true , address=145.79.227.23:4546 , piece_id=I7U4GEI2HGXU7UTN7E4APJGRRBFASQ4DCZ32K32BGOENKHCEMPMA, seg_pos={0 0}, upload=133220]: Done uploading piece. err: ecclient: upload failed (node:1ZMTFpqfMjeVWXJuvsGdbGkMtGQn6axUZFoU1N9sK6fnbPcMy5, address:145.79.227.23:4546): protocol: read tcp 188.166.225.170:50436->145.79.227.23:4546: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:21:37.012942445 (151h17m35.564412896s) [noise=true , address=145.79.227.14:4839 , piece_id=E2IGX3AZ5HDJYSUZW426L3CJIJAPTFNU7GXICPRN2CBDLLW4U35Q, seg_pos={0 0}, upload=133220]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: read tcp 188.166.225.170:50180->145.79.227.14:4839: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:21:37.012979546 (151h17m35.564449997s) [seg_pos={0 0}, upload=133220]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil> Nov 15 15:21:37.270684105 (151h17m35.822154567s) [upload=133220]: Done uploading segments... Nov 15 15:21:37.270724753 (151h17m35.822195214s) [upload=133220]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4) Duration: 6.018395808s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 8 out of 60 is failed (threshold=6) [..X.XXX.XXXX................................................] duration: 3.627305778s trace: https://jaeger.dev.storj.io/trace/42d763a000d5a90d FAIL (2025-11-15 15:22:21 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.627470821s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...............X.........................................X..] duration: 6.077996339s trace: https://jaeger.dev.storj.io/trace/46f6435aaf1b852a FAIL (2025-11-15 15:22:20 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 4) ======LOGS====== 6MZEKN7XOL667LYJSD7HHCBJIUGFBEYQFOQ, seg_pos={0 0}, upload=133220]: Uploading piece... Nov 15 15:21:08.018428616 (151h17m6.569899077s) [noise=true , address=45.91.116.11:4118 , piece_id=J36EGQPIQCW7CCGC4SKQBUVK7QC4UNIGBDBPYAB3A7TSYV377PDA, seg_pos={0 0}, upload=133220]: Uploading piece... Nov 15 15:21:08.018661680 (151h17m6.570132141s) [noise=true , address=145.79.227.26:4654 , piece_id=QZPWDI7TKCW3OGH36VQQH7EBQJQU2KGJZJ3Y5VZHVEMEDEOXPARA, seg_pos={0 0}, upload=133220]: Uploading piece... Nov 15 15:21:08.018882460 (151h17m6.570352921s) [noise=true , address=145.79.227.18:4005 , piece_id=ZWBRZGVA7CMHBEGSXAHFQPXLATKVSRWCAK5YQIYVZK7QJO3U5ZEA, seg_pos={0 0}, upload=133220]: Uploading piece... Nov 15 15:21:08.019281930 (151h17m6.570752391s) [noise=true , address=45.91.116.8:4045 , piece_id=FVDTJ3HTQGVPMVLV2656OCTNZT2Z3ZPFNC2C5F6GFHIDOAYLM3BA, seg_pos={0 0}, upload=133220]: Uploading piece... Nov 15 15:21:08.019316877 (151h17m6.570787338s) [noise=true , address=145.79.227.32:4836 , piece_id=2A5BU3NYUBVIEL62CAOPMBQEU7T5BDW5QOXVF3AZQFA2YU7VMLKQ, seg_pos={0 0}, upload=133220]: Uploading piece... Nov 15 15:21:08.019832619 (151h17m6.57130308s) [noise=true , address=45.91.116.5:4299 , piece_id=Z2XDDTQKI67WJ62RDWXNP76OITD3JVWIFPNBD5UYKT5ZP3J53UXA, seg_pos={0 0}, upload=133220]: Uploading piece... Nov 15 15:21:08.021202999 (151h17m6.57267347s) [noise=true , address=145.79.227.9:4365 , piece_id=5VNCHNGBR5XDMYFGRUVQ3SDACVDWY2DTVRUKVALTO7J6O6TVFP5A, seg_pos={0 0}, upload=133220]: Uploading piece... Nov 15 15:21:08.021959260 (151h17m6.573429721s) [noise=true , address=145.79.227.11:4729 , piece_id=FO73QHKPLIJZFWKCZZIJE33HKR2JQAMDVAAXFV7MZCZFL2D63U4A, seg_pos={0 0}, upload=133220]: Uploading piece... Nov 15 15:21:08.035442264 (151h17m6.586912725s) [noise=true , address=145.79.227.23:4546 , piece_id=I7U4GEI2HGXU7UTN7E4APJGRRBFASQ4DCZ32K32BGOENKHCEMPMA, seg_pos={0 0}, upload=133220]: Uploading piece... Nov 15 15:21:08.041195330 (151h17m6.592665811s) [noise=true , address=145.79.227.31:4798 , piece_id=YF6ZJD2LITIDCZAOJQQ2XVZIEGTDVCY7ZENSMWT6QRQU4R3QEGWQ, seg_pos={0 0}, upload=133220]: Uploading piece... Nov 15 15:21:09.871225261 (151h17m8.422695722s) [upload=133220]: Next returned nil segment Nov 15 15:21:09.871243576 (151h17m8.422714037s) [upload=133220]: Waiting for error group managing segments... Nov 15 15:21:10.165867146 (151h17m8.717337638s) [noise=true , address=77.90.31.19:4843 , piece_id=M47NOUZRG6LHAHBF26MZEKN7XOL667LYJSD7HHCBJIUGFBEYQFOQ, seg_pos={0 0}, upload=133220]: Done uploading piece. err: <nil> Nov 15 15:21:10.323410936 (151h17m8.874881397s) [noise=true , address=45.91.116.5:4299 , piece_id=Z2XDDTQKI67WJ62RDWXNP76OITD3JVWIFPNBD5UYKT5ZP3J53UXA, seg_pos={0 0}, upload=133220]: Done uploading piece. err: <nil> Nov 15 15:21:10.404481083 (151h17m8.955951544s) [noise=true , address=45.91.116.11:4118 , piece_id=J36EGQPIQCW7CCGC4SKQBUVK7QC4UNIGBDBPYAB3A7TSYV377PDA, seg_pos={0 0}, upload=133220]: Done uploading piece. err: <nil> Nov 15 15:21:11.440265666 (151h17m9.991736127s) [noise=true , address=45.91.116.8:4045 , piece_id=FVDTJ3HTQGVPMVLV2656OCTNZT2Z3ZPFNC2C5F6GFHIDOAYLM3BA, seg_pos={0 0}, upload=133220]: Done uploading piece. err: <nil> Nov 15 15:21:37.011608695 (151h17m35.563079156s) [noise=true , address=145.79.227.31:4798 , piece_id=YF6ZJD2LITIDCZAOJQQ2XVZIEGTDVCY7ZENSMWT6QRQU4R3QEGWQ, seg_pos={0 0}, upload=133220]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:145.79.227.31:4798): protocol: write tcp 188.166.225.170:45634->145.79.227.31:4798: use of closed network connection; write tcp 188.166.225.170:45634->145.79.227.31:4798: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45634->145.79.227.31:4798: use of closed network connection Nov 15 15:21:37.011777536 (151h17m35.563247997s) [noise=true , address=145.79.227.32:4836 , piece_id=2A5BU3NYUBVIEL62CAOPMBQEU7T5BDW5QOXVF3AZQFA2YU7VMLKQ, seg_pos={0 0}, upload=133220]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: write tcp 188.166.225.170:45832->145.79.227.32:4836: use of closed network connection; write tcp 188.166.225.170:45832->145.79.227.32:4836: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45832->145.79.227.32:4836: use of closed network connection Nov 15 15:21:37.011812132 (151h17m35.563282593s) [noise=true , address=145.79.227.30:4763 , piece_id=5HETJEEXU6DMLLAXICVF4A42YZHKQEVQ7AOKV734F7JRCIMDLTRQ, seg_pos={0 0}, upload=133220]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: read tcp 188.166.225.170:36386->145.79.227.30:4763: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:21:37.011874541 (151h17m35.563345002s) [noise=true , address=145.79.227.11:4729 , piece_id=FO73QHKPLIJZFWKCZZIJE33HKR2JQAMDVAAXFV7MZCZFL2D63U4A, seg_pos={0 0}, upload=133220]: Done uploading piece. err: ecclient: upload failed (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR, address:145.79.227.11:4729): protocol: write tcp 188.166.225.170:38964->145.79.227.11:4729: use of closed network connection; write tcp 188.166.225.170:38964->145.79.227.11:4729: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38964->145.79.227.11:4729: use of closed network connection Nov 15 15:21:37.011995582 (151h17m35.563466043s) [noise=true , address=145.79.227.12:4767 , piece_id=MZHSGDCEHU55EAMMFF6Z2XVRPNWGBKL3WLG2RTYG7QQTMW6UBKXA, seg_pos={0 0}, upload=133220]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:145.79.227.12:4767): protocol: write tcp 188.166.225.170:57008->145.79.227.12:4767: use of closed network connection; write tcp 188.166.225.170:57008->145.79.227.12:4767: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57008->145.79.227.12:4767: use of closed network connection Nov 15 15:21:37.012092546 (151h17m35.563563017s) [noise=true , address=145.79.227.28:4727 , piece_id=C4HI5WNAVBZ6OWWVF5WBMLJ4YDU6WJDLZ5ZXURAFV7WQ5H6F44BA, seg_pos={0 0}, upload=133220]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: read tcp 188.166.225.170:46810->145.79.227.28:4727: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:21:37.012296374 (151h17m35.563766835s) [noise=true , address=145.79.227.27:4689 , piece_id=KPQUJYT543MTIOZA653A74F2VDCGQAJ5QQV64AWMIEMKGRMUYQBQ, seg_pos={0 0}, upload=133220]: Done uploading piece. err: ecclient: upload failed (node:1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, address:145.79.227.27:4689): protocol: read tcp 188.166.225.170:49674->145.79.227.27:4689: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:21:37.012338324 (151h17m35.563808825s) [noise=true , address=145.79.227.9:4365 , piece_id=5VNCHNGBR5XDMYFGRUVQ3SDACVDWY2DTVRUKVALTO7J6O6TVFP5A, seg_pos={0 0}, upload=133220]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: read tcp 188.166.225.170:54022->145.79.227.9:4365: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:21:37.012485495 (151h17m35.563955956s) [noise=true , address=145.79.227.26:4654 , piece_id=QZPWDI7TKCW3OGH36VQQH7EBQJQU2KGJZJ3Y5VZHVEMEDEOXPARA, seg_pos={0 0}, upload=133220]: Done uploading piece. err: ecclient: upload failed (node:1KhBWMerZe9tUmzdVEYkBb6NmJ25dGP26Ea7CLjNJT5gdAeiah, address:145.79.227.26:4654): protocol: read tcp 188.166.225.170:53342->145.79.227.26:4654: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:21:37.012542613 (151h17m35.564013064s) [noise=true , address=145.79.227.18:4005 , piece_id=ZWBRZGVA7CMHBEGSXAHFQPXLATKVSRWCAK5YQIYVZK7QJO3U5ZEA, seg_pos={0 0}, upload=133220]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: read tcp 188.166.225.170:52578->145.79.227.18:4005: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:21:37.012760118 (151h17m35.564230579s) [noise=true , address=145.79.227.23:4546 , piece_id=I7U4GEI2HGXU7UTN7E4APJGRRBFASQ4DCZ32K32BGOENKHCEMPMA, seg_pos={0 0}, upload=133220]: Done uploading piece. err: ecclient: upload failed (node:1ZMTFpqfMjeVWXJuvsGdbGkMtGQn6axUZFoU1N9sK6fnbPcMy5, address:145.79.227.23:4546): protocol: read tcp 188.166.225.170:50436->145.79.227.23:4546: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:21:37.012942445 (151h17m35.564412896s) [noise=true , address=145.79.227.14:4839 , piece_id=E2IGX3AZ5HDJYSUZW426L3CJIJAPTFNU7GXICPRN2CBDLLW4U35Q, seg_pos={0 0}, upload=133220]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: read tcp 188.166.225.170:50180->145.79.227.14:4839: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:21:37.012979546 (151h17m35.564449997s) [seg_pos={0 0}, upload=133220]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil> Nov 15 15:21:37.270684105 (151h17m35.822154567s) [upload=133220]: Done uploading segments... Nov 15 15:21:37.270724753 (151h17m35.822195214s) [upload=133220]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4) Duration: 6.078020455s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [................X.........................................X.] duration: 29.094750707s trace: https://jaeger.dev.storj.io/trace/46f6435aaf1b852a FAIL (2025-11-15 15:22:14 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 4) ======LOGS====== 6MZEKN7XOL667LYJSD7HHCBJIUGFBEYQFOQ, seg_pos={0 0}, upload=133220]: Uploading piece... Nov 15 15:21:08.018428616 (151h17m6.569899077s) [noise=true , address=45.91.116.11:4118 , piece_id=J36EGQPIQCW7CCGC4SKQBUVK7QC4UNIGBDBPYAB3A7TSYV377PDA, seg_pos={0 0}, upload=133220]: Uploading piece... Nov 15 15:21:08.018661680 (151h17m6.570132141s) [noise=true , address=145.79.227.26:4654 , piece_id=QZPWDI7TKCW3OGH36VQQH7EBQJQU2KGJZJ3Y5VZHVEMEDEOXPARA, seg_pos={0 0}, upload=133220]: Uploading piece... Nov 15 15:21:08.018882460 (151h17m6.570352921s) [noise=true , address=145.79.227.18:4005 , piece_id=ZWBRZGVA7CMHBEGSXAHFQPXLATKVSRWCAK5YQIYVZK7QJO3U5ZEA, seg_pos={0 0}, upload=133220]: Uploading piece... Nov 15 15:21:08.019281930 (151h17m6.570752391s) [noise=true , address=45.91.116.8:4045 , piece_id=FVDTJ3HTQGVPMVLV2656OCTNZT2Z3ZPFNC2C5F6GFHIDOAYLM3BA, seg_pos={0 0}, upload=133220]: Uploading piece... Nov 15 15:21:08.019316877 (151h17m6.570787338s) [noise=true , address=145.79.227.32:4836 , piece_id=2A5BU3NYUBVIEL62CAOPMBQEU7T5BDW5QOXVF3AZQFA2YU7VMLKQ, seg_pos={0 0}, upload=133220]: Uploading piece... Nov 15 15:21:08.019832619 (151h17m6.57130308s) [noise=true , address=45.91.116.5:4299 , piece_id=Z2XDDTQKI67WJ62RDWXNP76OITD3JVWIFPNBD5UYKT5ZP3J53UXA, seg_pos={0 0}, upload=133220]: Uploading piece... Nov 15 15:21:08.021202999 (151h17m6.57267347s) [noise=true , address=145.79.227.9:4365 , piece_id=5VNCHNGBR5XDMYFGRUVQ3SDACVDWY2DTVRUKVALTO7J6O6TVFP5A, seg_pos={0 0}, upload=133220]: Uploading piece... Nov 15 15:21:08.021959260 (151h17m6.573429721s) [noise=true , address=145.79.227.11:4729 , piece_id=FO73QHKPLIJZFWKCZZIJE33HKR2JQAMDVAAXFV7MZCZFL2D63U4A, seg_pos={0 0}, upload=133220]: Uploading piece... Nov 15 15:21:08.035442264 (151h17m6.586912725s) [noise=true , address=145.79.227.23:4546 , piece_id=I7U4GEI2HGXU7UTN7E4APJGRRBFASQ4DCZ32K32BGOENKHCEMPMA, seg_pos={0 0}, upload=133220]: Uploading piece... Nov 15 15:21:08.041195330 (151h17m6.592665811s) [noise=true , address=145.79.227.31:4798 , piece_id=YF6ZJD2LITIDCZAOJQQ2XVZIEGTDVCY7ZENSMWT6QRQU4R3QEGWQ, seg_pos={0 0}, upload=133220]: Uploading piece... Nov 15 15:21:09.871225261 (151h17m8.422695722s) [upload=133220]: Next returned nil segment Nov 15 15:21:09.871243576 (151h17m8.422714037s) [upload=133220]: Waiting for error group managing segments... Nov 15 15:21:10.165867146 (151h17m8.717337638s) [noise=true , address=77.90.31.19:4843 , piece_id=M47NOUZRG6LHAHBF26MZEKN7XOL667LYJSD7HHCBJIUGFBEYQFOQ, seg_pos={0 0}, upload=133220]: Done uploading piece. err: <nil> Nov 15 15:21:10.323410936 (151h17m8.874881397s) [noise=true , address=45.91.116.5:4299 , piece_id=Z2XDDTQKI67WJ62RDWXNP76OITD3JVWIFPNBD5UYKT5ZP3J53UXA, seg_pos={0 0}, upload=133220]: Done uploading piece. err: <nil> Nov 15 15:21:10.404481083 (151h17m8.955951544s) [noise=true , address=45.91.116.11:4118 , piece_id=J36EGQPIQCW7CCGC4SKQBUVK7QC4UNIGBDBPYAB3A7TSYV377PDA, seg_pos={0 0}, upload=133220]: Done uploading piece. err: <nil> Nov 15 15:21:11.440265666 (151h17m9.991736127s) [noise=true , address=45.91.116.8:4045 , piece_id=FVDTJ3HTQGVPMVLV2656OCTNZT2Z3ZPFNC2C5F6GFHIDOAYLM3BA, seg_pos={0 0}, upload=133220]: Done uploading piece. err: <nil> Nov 15 15:21:37.011608695 (151h17m35.563079156s) [noise=true , address=145.79.227.31:4798 , piece_id=YF6ZJD2LITIDCZAOJQQ2XVZIEGTDVCY7ZENSMWT6QRQU4R3QEGWQ, seg_pos={0 0}, upload=133220]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:145.79.227.31:4798): protocol: write tcp 188.166.225.170:45634->145.79.227.31:4798: use of closed network connection; write tcp 188.166.225.170:45634->145.79.227.31:4798: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45634->145.79.227.31:4798: use of closed network connection Nov 15 15:21:37.011777536 (151h17m35.563247997s) [noise=true , address=145.79.227.32:4836 , piece_id=2A5BU3NYUBVIEL62CAOPMBQEU7T5BDW5QOXVF3AZQFA2YU7VMLKQ, seg_pos={0 0}, upload=133220]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: write tcp 188.166.225.170:45832->145.79.227.32:4836: use of closed network connection; write tcp 188.166.225.170:45832->145.79.227.32:4836: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45832->145.79.227.32:4836: use of closed network connection Nov 15 15:21:37.011812132 (151h17m35.563282593s) [noise=true , address=145.79.227.30:4763 , piece_id=5HETJEEXU6DMLLAXICVF4A42YZHKQEVQ7AOKV734F7JRCIMDLTRQ, seg_pos={0 0}, upload=133220]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: read tcp 188.166.225.170:36386->145.79.227.30:4763: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:21:37.011874541 (151h17m35.563345002s) [noise=true , address=145.79.227.11:4729 , piece_id=FO73QHKPLIJZFWKCZZIJE33HKR2JQAMDVAAXFV7MZCZFL2D63U4A, seg_pos={0 0}, upload=133220]: Done uploading piece. err: ecclient: upload failed (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR, address:145.79.227.11:4729): protocol: write tcp 188.166.225.170:38964->145.79.227.11:4729: use of closed network connection; write tcp 188.166.225.170:38964->145.79.227.11:4729: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38964->145.79.227.11:4729: use of closed network connection Nov 15 15:21:37.011995582 (151h17m35.563466043s) [noise=true , address=145.79.227.12:4767 , piece_id=MZHSGDCEHU55EAMMFF6Z2XVRPNWGBKL3WLG2RTYG7QQTMW6UBKXA, seg_pos={0 0}, upload=133220]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:145.79.227.12:4767): protocol: write tcp 188.166.225.170:57008->145.79.227.12:4767: use of closed network connection; write tcp 188.166.225.170:57008->145.79.227.12:4767: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57008->145.79.227.12:4767: use of closed network connection Nov 15 15:21:37.012092546 (151h17m35.563563017s) [noise=true , address=145.79.227.28:4727 , piece_id=C4HI5WNAVBZ6OWWVF5WBMLJ4YDU6WJDLZ5ZXURAFV7WQ5H6F44BA, seg_pos={0 0}, upload=133220]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: read tcp 188.166.225.170:46810->145.79.227.28:4727: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:21:37.012296374 (151h17m35.563766835s) [noise=true , address=145.79.227.27:4689 , piece_id=KPQUJYT543MTIOZA653A74F2VDCGQAJ5QQV64AWMIEMKGRMUYQBQ, seg_pos={0 0}, upload=133220]: Done uploading piece. err: ecclient: upload failed (node:1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, address:145.79.227.27:4689): protocol: read tcp 188.166.225.170:49674->145.79.227.27:4689: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:21:37.012338324 (151h17m35.563808825s) [noise=true , address=145.79.227.9:4365 , piece_id=5VNCHNGBR5XDMYFGRUVQ3SDACVDWY2DTVRUKVALTO7J6O6TVFP5A, seg_pos={0 0}, upload=133220]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: read tcp 188.166.225.170:54022->145.79.227.9:4365: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:21:37.012485495 (151h17m35.563955956s) [noise=true , address=145.79.227.26:4654 , piece_id=QZPWDI7TKCW3OGH36VQQH7EBQJQU2KGJZJ3Y5VZHVEMEDEOXPARA, seg_pos={0 0}, upload=133220]: Done uploading piece. err: ecclient: upload failed (node:1KhBWMerZe9tUmzdVEYkBb6NmJ25dGP26Ea7CLjNJT5gdAeiah, address:145.79.227.26:4654): protocol: read tcp 188.166.225.170:53342->145.79.227.26:4654: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:21:37.012542613 (151h17m35.564013064s) [noise=true , address=145.79.227.18:4005 , piece_id=ZWBRZGVA7CMHBEGSXAHFQPXLATKVSRWCAK5YQIYVZK7QJO3U5ZEA, seg_pos={0 0}, upload=133220]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: read tcp 188.166.225.170:52578->145.79.227.18:4005: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:21:37.012760118 (151h17m35.564230579s) [noise=true , address=145.79.227.23:4546 , piece_id=I7U4GEI2HGXU7UTN7E4APJGRRBFASQ4DCZ32K32BGOENKHCEMPMA, seg_pos={0 0}, upload=133220]: Done uploading piece. err: ecclient: upload failed (node:1ZMTFpqfMjeVWXJuvsGdbGkMtGQn6axUZFoU1N9sK6fnbPcMy5, address:145.79.227.23:4546): protocol: read tcp 188.166.225.170:50436->145.79.227.23:4546: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:21:37.012942445 (151h17m35.564412896s) [noise=true , address=145.79.227.14:4839 , piece_id=E2IGX3AZ5HDJYSUZW426L3CJIJAPTFNU7GXICPRN2CBDLLW4U35Q, seg_pos={0 0}, upload=133220]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: read tcp 188.166.225.170:50180->145.79.227.14:4839: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:21:37.012979546 (151h17m35.564449997s) [seg_pos={0 0}, upload=133220]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil> Nov 15 15:21:37.270684105 (151h17m35.822154567s) [upload=133220]: Done uploading segments... Nov 15 15:21:37.270724753 (151h17m35.822195214s) [upload=133220]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4) Duration: 29.094815691s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.................X.........................................X] duration: 30.260274028s trace: https://jaeger.dev.storj.io/trace/46f6435aaf1b852a FAIL (2025-11-15 15:21:37 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 4) ======LOGS====== 6MZEKN7XOL667LYJSD7HHCBJIUGFBEYQFOQ, seg_pos={0 0}, upload=133220]: Uploading piece... Nov 15 15:21:08.018428616 (151h17m6.569899077s) [noise=true , address=45.91.116.11:4118 , piece_id=J36EGQPIQCW7CCGC4SKQBUVK7QC4UNIGBDBPYAB3A7TSYV377PDA, seg_pos={0 0}, upload=133220]: Uploading piece... Nov 15 15:21:08.018661680 (151h17m6.570132141s) [noise=true , address=145.79.227.26:4654 , piece_id=QZPWDI7TKCW3OGH36VQQH7EBQJQU2KGJZJ3Y5VZHVEMEDEOXPARA, seg_pos={0 0}, upload=133220]: Uploading piece... Nov 15 15:21:08.018882460 (151h17m6.570352921s) [noise=true , address=145.79.227.18:4005 , piece_id=ZWBRZGVA7CMHBEGSXAHFQPXLATKVSRWCAK5YQIYVZK7QJO3U5ZEA, seg_pos={0 0}, upload=133220]: Uploading piece... Nov 15 15:21:08.019281930 (151h17m6.570752391s) [noise=true , address=45.91.116.8:4045 , piece_id=FVDTJ3HTQGVPMVLV2656OCTNZT2Z3ZPFNC2C5F6GFHIDOAYLM3BA, seg_pos={0 0}, upload=133220]: Uploading piece... Nov 15 15:21:08.019316877 (151h17m6.570787338s) [noise=true , address=145.79.227.32:4836 , piece_id=2A5BU3NYUBVIEL62CAOPMBQEU7T5BDW5QOXVF3AZQFA2YU7VMLKQ, seg_pos={0 0}, upload=133220]: Uploading piece... Nov 15 15:21:08.019832619 (151h17m6.57130308s) [noise=true , address=45.91.116.5:4299 , piece_id=Z2XDDTQKI67WJ62RDWXNP76OITD3JVWIFPNBD5UYKT5ZP3J53UXA, seg_pos={0 0}, upload=133220]: Uploading piece... Nov 15 15:21:08.021202999 (151h17m6.57267347s) [noise=true , address=145.79.227.9:4365 , piece_id=5VNCHNGBR5XDMYFGRUVQ3SDACVDWY2DTVRUKVALTO7J6O6TVFP5A, seg_pos={0 0}, upload=133220]: Uploading piece... Nov 15 15:21:08.021959260 (151h17m6.573429721s) [noise=true , address=145.79.227.11:4729 , piece_id=FO73QHKPLIJZFWKCZZIJE33HKR2JQAMDVAAXFV7MZCZFL2D63U4A, seg_pos={0 0}, upload=133220]: Uploading piece... Nov 15 15:21:08.035442264 (151h17m6.586912725s) [noise=true , address=145.79.227.23:4546 , piece_id=I7U4GEI2HGXU7UTN7E4APJGRRBFASQ4DCZ32K32BGOENKHCEMPMA, seg_pos={0 0}, upload=133220]: Uploading piece... Nov 15 15:21:08.041195330 (151h17m6.592665811s) [noise=true , address=145.79.227.31:4798 , piece_id=YF6ZJD2LITIDCZAOJQQ2XVZIEGTDVCY7ZENSMWT6QRQU4R3QEGWQ, seg_pos={0 0}, upload=133220]: Uploading piece... Nov 15 15:21:09.871225261 (151h17m8.422695722s) [upload=133220]: Next returned nil segment Nov 15 15:21:09.871243576 (151h17m8.422714037s) [upload=133220]: Waiting for error group managing segments... Nov 15 15:21:10.165867146 (151h17m8.717337638s) [noise=true , address=77.90.31.19:4843 , piece_id=M47NOUZRG6LHAHBF26MZEKN7XOL667LYJSD7HHCBJIUGFBEYQFOQ, seg_pos={0 0}, upload=133220]: Done uploading piece. err: <nil> Nov 15 15:21:10.323410936 (151h17m8.874881397s) [noise=true , address=45.91.116.5:4299 , piece_id=Z2XDDTQKI67WJ62RDWXNP76OITD3JVWIFPNBD5UYKT5ZP3J53UXA, seg_pos={0 0}, upload=133220]: Done uploading piece. err: <nil> Nov 15 15:21:10.404481083 (151h17m8.955951544s) [noise=true , address=45.91.116.11:4118 , piece_id=J36EGQPIQCW7CCGC4SKQBUVK7QC4UNIGBDBPYAB3A7TSYV377PDA, seg_pos={0 0}, upload=133220]: Done uploading piece. err: <nil> Nov 15 15:21:11.440265666 (151h17m9.991736127s) [noise=true , address=45.91.116.8:4045 , piece_id=FVDTJ3HTQGVPMVLV2656OCTNZT2Z3ZPFNC2C5F6GFHIDOAYLM3BA, seg_pos={0 0}, upload=133220]: Done uploading piece. err: <nil> Nov 15 15:21:37.011608695 (151h17m35.563079156s) [noise=true , address=145.79.227.31:4798 , piece_id=YF6ZJD2LITIDCZAOJQQ2XVZIEGTDVCY7ZENSMWT6QRQU4R3QEGWQ, seg_pos={0 0}, upload=133220]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:145.79.227.31:4798): protocol: write tcp 188.166.225.170:45634->145.79.227.31:4798: use of closed network connection; write tcp 188.166.225.170:45634->145.79.227.31:4798: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45634->145.79.227.31:4798: use of closed network connection Nov 15 15:21:37.011777536 (151h17m35.563247997s) [noise=true , address=145.79.227.32:4836 , piece_id=2A5BU3NYUBVIEL62CAOPMBQEU7T5BDW5QOXVF3AZQFA2YU7VMLKQ, seg_pos={0 0}, upload=133220]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: write tcp 188.166.225.170:45832->145.79.227.32:4836: use of closed network connection; write tcp 188.166.225.170:45832->145.79.227.32:4836: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45832->145.79.227.32:4836: use of closed network connection Nov 15 15:21:37.011812132 (151h17m35.563282593s) [noise=true , address=145.79.227.30:4763 , piece_id=5HETJEEXU6DMLLAXICVF4A42YZHKQEVQ7AOKV734F7JRCIMDLTRQ, seg_pos={0 0}, upload=133220]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: read tcp 188.166.225.170:36386->145.79.227.30:4763: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:21:37.011874541 (151h17m35.563345002s) [noise=true , address=145.79.227.11:4729 , piece_id=FO73QHKPLIJZFWKCZZIJE33HKR2JQAMDVAAXFV7MZCZFL2D63U4A, seg_pos={0 0}, upload=133220]: Done uploading piece. err: ecclient: upload failed (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR, address:145.79.227.11:4729): protocol: write tcp 188.166.225.170:38964->145.79.227.11:4729: use of closed network connection; write tcp 188.166.225.170:38964->145.79.227.11:4729: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38964->145.79.227.11:4729: use of closed network connection Nov 15 15:21:37.011995582 (151h17m35.563466043s) [noise=true , address=145.79.227.12:4767 , piece_id=MZHSGDCEHU55EAMMFF6Z2XVRPNWGBKL3WLG2RTYG7QQTMW6UBKXA, seg_pos={0 0}, upload=133220]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:145.79.227.12:4767): protocol: write tcp 188.166.225.170:57008->145.79.227.12:4767: use of closed network connection; write tcp 188.166.225.170:57008->145.79.227.12:4767: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57008->145.79.227.12:4767: use of closed network connection Nov 15 15:21:37.012092546 (151h17m35.563563017s) [noise=true , address=145.79.227.28:4727 , piece_id=C4HI5WNAVBZ6OWWVF5WBMLJ4YDU6WJDLZ5ZXURAFV7WQ5H6F44BA, seg_pos={0 0}, upload=133220]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: read tcp 188.166.225.170:46810->145.79.227.28:4727: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:21:37.012296374 (151h17m35.563766835s) [noise=true , address=145.79.227.27:4689 , piece_id=KPQUJYT543MTIOZA653A74F2VDCGQAJ5QQV64AWMIEMKGRMUYQBQ, seg_pos={0 0}, upload=133220]: Done uploading piece. err: ecclient: upload failed (node:1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, address:145.79.227.27:4689): protocol: read tcp 188.166.225.170:49674->145.79.227.27:4689: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:21:37.012338324 (151h17m35.563808825s) [noise=true , address=145.79.227.9:4365 , piece_id=5VNCHNGBR5XDMYFGRUVQ3SDACVDWY2DTVRUKVALTO7J6O6TVFP5A, seg_pos={0 0}, upload=133220]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: read tcp 188.166.225.170:54022->145.79.227.9:4365: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:21:37.012485495 (151h17m35.563955956s) [noise=true , address=145.79.227.26:4654 , piece_id=QZPWDI7TKCW3OGH36VQQH7EBQJQU2KGJZJ3Y5VZHVEMEDEOXPARA, seg_pos={0 0}, upload=133220]: Done uploading piece. err: ecclient: upload failed (node:1KhBWMerZe9tUmzdVEYkBb6NmJ25dGP26Ea7CLjNJT5gdAeiah, address:145.79.227.26:4654): protocol: read tcp 188.166.225.170:53342->145.79.227.26:4654: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:21:37.012542613 (151h17m35.564013064s) [noise=true , address=145.79.227.18:4005 , piece_id=ZWBRZGVA7CMHBEGSXAHFQPXLATKVSRWCAK5YQIYVZK7QJO3U5ZEA, seg_pos={0 0}, upload=133220]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: read tcp 188.166.225.170:52578->145.79.227.18:4005: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:21:37.012760118 (151h17m35.564230579s) [noise=true , address=145.79.227.23:4546 , piece_id=I7U4GEI2HGXU7UTN7E4APJGRRBFASQ4DCZ32K32BGOENKHCEMPMA, seg_pos={0 0}, upload=133220]: Done uploading piece. err: ecclient: upload failed (node:1ZMTFpqfMjeVWXJuvsGdbGkMtGQn6axUZFoU1N9sK6fnbPcMy5, address:145.79.227.23:4546): protocol: read tcp 188.166.225.170:50436->145.79.227.23:4546: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:21:37.012942445 (151h17m35.564412896s) [noise=true , address=145.79.227.14:4839 , piece_id=E2IGX3AZ5HDJYSUZW426L3CJIJAPTFNU7GXICPRN2CBDLLW4U35Q, seg_pos={0 0}, upload=133220]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: read tcp 188.166.225.170:50180->145.79.227.14:4839: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 15:21:37.012979546 (151h17m35.564449997s) [seg_pos={0 0}, upload=133220]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil> Nov 15 15:21:37.270684105 (151h17m35.822154567s) [upload=133220]: Done uploading segments... Nov 15 15:21:37.270724753 (151h17m35.822195214s) [upload=133220]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4) Duration: 30.26030564s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 8 out of 60 is failed (threshold=6) [...X.XXX.XXXX...............................................] duration: 4.739273073s trace: https://jaeger.dev.storj.io/trace/42d763a000d5a90d FAIL (2025-11-15 15:21:35 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.739301286s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 9 out of 60 is failed (threshold=6) [X...X.XXX.XXXX..............................................] duration: 3.454047869s trace: https://jaeger.dev.storj.io/trace/42d763a000d5a90d FAIL (2025-11-15 15:20:25 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.454079048s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 6.283358999s trace: https://jaeger.dev.storj.io/trace/495e264ce0048970 FAIL (2025-11-15 15:20:01 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 5) ======LOGS====== T4H6SYA, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.348693684 (150h46m7.900164155s) [noise=true , address=145.79.227.13:4803 , piece_id=H2K3B22G6YTEGGS2Y76DCJJHIQIKGXZMDDPQ4MMCUPJGQIRWO5HA, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.349289590 (150h46m7.900760051s) [noise=true , address=77.90.31.15:4699 , piece_id=AH4N55LZLPUJGQOPDLXM6I65SUUKOHUCPXPYE2I7XKI5KHYNJPWA, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.349845298 (150h46m7.901315759s) [noise=true , address=145.79.227.27:4689 , piece_id=5GEUPACS2UDKXX6RZAL7AOUVXIF3HQRAYZUCAY54GFANZ3ODTYBA, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.350334239 (150h46m7.9018047s) [noise=true , address=45.91.116.7:4011 , piece_id=CSTREAXVJWKIACVOGMDI7QXPAU7JK624TZMQIOB7A2UDDJ3YBOTA, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.350770229 (150h46m7.90224069s) [noise=true , address=145.79.227.22:4509 , piece_id=AUB7HF5665Y26RSULL2WH6PUOQJUXSQT66OFMCM5YZWWRPPI6SRA, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.351317160 (150h46m7.902787632s) [noise=true , address=45.91.116.12:4226 , piece_id=T64PPRSXNN5ZPKAY4YXS5WWAXCNUU4G7Y7TXJXLE4VBNULK3OGMA, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.353921029 (150h46m7.9053915s) [noise=true , address=45.91.116.5:4299 , piece_id=5FETUEJKO5PEHNGKOTPFAAXSPRY7WU5WPSEULBVJ2DPDINPN3V3A, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.354632563 (150h46m7.906103024s) [noise=true , address=145.79.227.8:4329 , piece_id=7IRP7Y6YSPNBN7DJJOMPPB7T2PDQ43G55MFAYHAWIT3Y4ULXSXZA, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.355164717 (150h46m7.906635178s) [noise=true , address=145.79.227.24:4581 , piece_id=E63A2HTXTWEQCYTOZA47S24YKVDKQ3BHAVZUSKUDALZWVIY6NMOQ, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.355607720 (150h46m7.907078181s) [noise=true , address=145.79.227.16:4910 , piece_id=7BH46URBTAPGFQ2JKIXUUG4HMX6ERQS34BHPGBDQBMSNCVPVTXZQ, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.356053489 (150h46m7.90752395s) [noise=true , address=145.79.227.9:4367 , piece_id=EFWN62D4D4U2FX7D53NGUNY3UVJBDZTMD7DC47N5IEW6SVXLIF7A, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.357082539 (150h46m7.90855301s) [noise=true , address=145.79.227.23:4545 , piece_id=HJFCVMDY2CHYOIS4NYNFEXWHVQ3JGIQ4WAMKT4BC4MTNAMJWVGEA, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.360976825 (150h46m7.912447286s) [noise=true , address=77.90.31.17:4772 , piece_id=ZTLV4QEMHCQJUBZ7CZWQK27Z6IKXYICU76OSMZPETELOYRXTK5LQ, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:11.203310499 (150h46m9.754780951s) [upload=132752]: Next returned nil segment Nov 15 14:50:11.203370323 (150h46m9.754840784s) [upload=132752]: Waiting for error group managing segments... Nov 15 14:50:11.496843592 (150h46m10.048314053s) [noise=true , address=45.91.116.5:4299 , piece_id=5FETUEJKO5PEHNGKOTPFAAXSPRY7WU5WPSEULBVJ2DPDINPN3V3A, seg_pos={0 0}, upload=132752]: Done uploading piece. err: <nil> Nov 15 14:50:11.503251574 (150h46m10.054722025s) [noise=true , address=45.91.116.7:4011 , piece_id=CSTREAXVJWKIACVOGMDI7QXPAU7JK624TZMQIOB7A2UDDJ3YBOTA, seg_pos={0 0}, upload=132752]: Done uploading piece. err: <nil> Nov 15 14:50:11.519837998 (150h46m10.071308449s) [noise=true , address=45.91.116.12:4226 , piece_id=T64PPRSXNN5ZPKAY4YXS5WWAXCNUU4G7Y7TXJXLE4VBNULK3OGMA, seg_pos={0 0}, upload=132752]: Done uploading piece. err: <nil> Nov 15 14:50:11.587364703 (150h46m10.138835164s) [noise=true , address=77.90.31.17:4772 , piece_id=ZTLV4QEMHCQJUBZ7CZWQK27Z6IKXYICU76OSMZPETELOYRXTK5LQ, seg_pos={0 0}, upload=132752]: Done uploading piece. err: <nil> Nov 15 14:50:12.990960591 (150h46m11.542431052s) [noise=true , address=77.90.31.15:4699 , piece_id=AH4N55LZLPUJGQOPDLXM6I65SUUKOHUCPXPYE2I7XKI5KHYNJPWA, seg_pos={0 0}, upload=132752]: Done uploading piece. err: <nil> Nov 15 14:50:38.328113117 (150h46m36.879583578s) [noise=true , address=145.79.227.10:4401 , piece_id=SZWIK2OAO3YVLHLAU5FU6KHXODXVXDANAJP5WEKRSLUXDT4H6SYA, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: write tcp 188.166.225.170:34644->145.79.227.10:4401: use of closed network connection; write tcp 188.166.225.170:34644->145.79.227.10:4401: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34644->145.79.227.10:4401: use of closed network connection Nov 15 14:50:38.328591017 (150h46m36.880061478s) [noise=true , address=145.79.227.24:4581 , piece_id=E63A2HTXTWEQCYTOZA47S24YKVDKQ3BHAVZUSKUDALZWVIY6NMOQ, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:123ib1A8nmVmR272odjhVsUgADYbaMRMiPahSLoZ5eamaV6Qbb1, address:145.79.227.24:4581): protocol: read tcp 188.166.225.170:50846->145.79.227.24:4581: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 14:50:38.328758827 (150h46m36.880229318s) [noise=true , address=145.79.227.8:4330 , piece_id=MG7OE5OYG64CQUEGAUZNL654VHQQV347RSY2OKZAY5MXDXOSCMRQ, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW, address:145.79.227.8:4330): protocol: read tcp 188.166.225.170:43188->145.79.227.8:4330: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 14:50:38.328953126 (150h46m36.880423587s) [noise=true , address=145.79.227.32:4834 , piece_id=CJN674FREG73VEEP362QDU7HAAIDOCCYFGSL6ZE7QPDC3NHPWK2A, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: read tcp 188.166.225.170:38312->145.79.227.32:4834: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 14:50:38.329418293 (150h46m36.880888754s) [noise=true , address=145.79.227.9:4367 , piece_id=EFWN62D4D4U2FX7D53NGUNY3UVJBDZTMD7DC47N5IEW6SVXLIF7A, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6, address:145.79.227.9:4367): protocol: write tcp 188.166.225.170:36134->145.79.227.9:4367: use of closed network connection; write tcp 188.166.225.170:36134->145.79.227.9:4367: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36134->145.79.227.9:4367: use of closed network connection Nov 15 14:50:38.329617381 (150h46m36.881087842s) [noise=true , address=145.79.227.13:4803 , piece_id=H2K3B22G6YTEGGS2Y76DCJJHIQIKGXZMDDPQ4MMCUPJGQIRWO5HA, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:145.79.227.13:4803): protocol: read tcp 188.166.225.170:50394->145.79.227.13:4803: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 14:50:38.329698186 (150h46m36.881168657s) [noise=true , address=145.79.227.22:4509 , piece_id=AUB7HF5665Y26RSULL2WH6PUOQJUXSQT66OFMCM5YZWWRPPI6SRA, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:1229E58YirF37sZU6hfZGrDNFSxQDYsWSo4x3mG9BoYsNsLNtjy, address:145.79.227.22:4509): protocol: rpc: dial tcp 145.79.227.22:4509: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 14:50:38.329926219 (150h46m36.88139668s) [noise=true , address=145.79.227.8:4329 , piece_id=7IRP7Y6YSPNBN7DJJOMPPB7T2PDQ43G55MFAYHAWIT3Y4ULXSXZA, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: read tcp 188.166.225.170:44482->145.79.227.8:4329: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 14:50:38.330195863 (150h46m36.881666334s) [noise=true , address=145.79.227.16:4910 , piece_id=7BH46URBTAPGFQ2JKIXUUG4HMX6ERQS34BHPGBDQBMSNCVPVTXZQ, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: read tcp 188.166.225.170:58432->145.79.227.16:4910: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 14:50:38.330267870 (150h46m36.881738331s) [noise=true , address=145.79.227.23:4545 , piece_id=HJFCVMDY2CHYOIS4NYNFEXWHVQ3JGIQ4WAMKT4BC4MTNAMJWVGEA, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:12P5cnEza1Xxwg4qo1nau11zYHsHxqEAwhFsnt2UE1RhtTrJmra, address:145.79.227.23:4545): protocol: read tcp 188.166.225.170:42348->145.79.227.23:4545: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 14:50:38.330623417 (150h46m36.882093898s) [noise=true , address=145.79.227.27:4689 , piece_id=5GEUPACS2UDKXX6RZAL7AOUVXIF3HQRAYZUCAY54GFANZ3ODTYBA, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, address:145.79.227.27:4689): protocol: read tcp 188.166.225.170:52596->145.79.227.27:4689: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 14:50:38.330679213 (150h46m36.882149675s) [seg_pos={0 0}, upload=132752]: Done waiting for segment. successful: 5 optimal: 13 errs: <nil> Nov 15 14:50:38.589562836 (150h46m37.141033297s) [upload=132752]: Done uploading segments... Nov 15 14:50:38.589597782 (150h46m37.141068243s) [upload=132752]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 5) Duration: 6.283407963s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 10 out of 60 is failed (threshold=6) [XX...X.XXX.XXXX.............................................] duration: 4.319453876s trace: https://jaeger.dev.storj.io/trace/42d763a000d5a90d FAIL (2025-11-15 15:19:54 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.319488523s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 6.334167889s trace: https://jaeger.dev.storj.io/trace/495e264ce0048970 FAIL (2025-11-15 15:19:31 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 5) ======LOGS====== T4H6SYA, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.348693684 (150h46m7.900164155s) [noise=true , address=145.79.227.13:4803 , piece_id=H2K3B22G6YTEGGS2Y76DCJJHIQIKGXZMDDPQ4MMCUPJGQIRWO5HA, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.349289590 (150h46m7.900760051s) [noise=true , address=77.90.31.15:4699 , piece_id=AH4N55LZLPUJGQOPDLXM6I65SUUKOHUCPXPYE2I7XKI5KHYNJPWA, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.349845298 (150h46m7.901315759s) [noise=true , address=145.79.227.27:4689 , piece_id=5GEUPACS2UDKXX6RZAL7AOUVXIF3HQRAYZUCAY54GFANZ3ODTYBA, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.350334239 (150h46m7.9018047s) [noise=true , address=45.91.116.7:4011 , piece_id=CSTREAXVJWKIACVOGMDI7QXPAU7JK624TZMQIOB7A2UDDJ3YBOTA, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.350770229 (150h46m7.90224069s) [noise=true , address=145.79.227.22:4509 , piece_id=AUB7HF5665Y26RSULL2WH6PUOQJUXSQT66OFMCM5YZWWRPPI6SRA, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.351317160 (150h46m7.902787632s) [noise=true , address=45.91.116.12:4226 , piece_id=T64PPRSXNN5ZPKAY4YXS5WWAXCNUU4G7Y7TXJXLE4VBNULK3OGMA, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.353921029 (150h46m7.9053915s) [noise=true , address=45.91.116.5:4299 , piece_id=5FETUEJKO5PEHNGKOTPFAAXSPRY7WU5WPSEULBVJ2DPDINPN3V3A, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.354632563 (150h46m7.906103024s) [noise=true , address=145.79.227.8:4329 , piece_id=7IRP7Y6YSPNBN7DJJOMPPB7T2PDQ43G55MFAYHAWIT3Y4ULXSXZA, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.355164717 (150h46m7.906635178s) [noise=true , address=145.79.227.24:4581 , piece_id=E63A2HTXTWEQCYTOZA47S24YKVDKQ3BHAVZUSKUDALZWVIY6NMOQ, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.355607720 (150h46m7.907078181s) [noise=true , address=145.79.227.16:4910 , piece_id=7BH46URBTAPGFQ2JKIXUUG4HMX6ERQS34BHPGBDQBMSNCVPVTXZQ, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.356053489 (150h46m7.90752395s) [noise=true , address=145.79.227.9:4367 , piece_id=EFWN62D4D4U2FX7D53NGUNY3UVJBDZTMD7DC47N5IEW6SVXLIF7A, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.357082539 (150h46m7.90855301s) [noise=true , address=145.79.227.23:4545 , piece_id=HJFCVMDY2CHYOIS4NYNFEXWHVQ3JGIQ4WAMKT4BC4MTNAMJWVGEA, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.360976825 (150h46m7.912447286s) [noise=true , address=77.90.31.17:4772 , piece_id=ZTLV4QEMHCQJUBZ7CZWQK27Z6IKXYICU76OSMZPETELOYRXTK5LQ, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:11.203310499 (150h46m9.754780951s) [upload=132752]: Next returned nil segment Nov 15 14:50:11.203370323 (150h46m9.754840784s) [upload=132752]: Waiting for error group managing segments... Nov 15 14:50:11.496843592 (150h46m10.048314053s) [noise=true , address=45.91.116.5:4299 , piece_id=5FETUEJKO5PEHNGKOTPFAAXSPRY7WU5WPSEULBVJ2DPDINPN3V3A, seg_pos={0 0}, upload=132752]: Done uploading piece. err: <nil> Nov 15 14:50:11.503251574 (150h46m10.054722025s) [noise=true , address=45.91.116.7:4011 , piece_id=CSTREAXVJWKIACVOGMDI7QXPAU7JK624TZMQIOB7A2UDDJ3YBOTA, seg_pos={0 0}, upload=132752]: Done uploading piece. err: <nil> Nov 15 14:50:11.519837998 (150h46m10.071308449s) [noise=true , address=45.91.116.12:4226 , piece_id=T64PPRSXNN5ZPKAY4YXS5WWAXCNUU4G7Y7TXJXLE4VBNULK3OGMA, seg_pos={0 0}, upload=132752]: Done uploading piece. err: <nil> Nov 15 14:50:11.587364703 (150h46m10.138835164s) [noise=true , address=77.90.31.17:4772 , piece_id=ZTLV4QEMHCQJUBZ7CZWQK27Z6IKXYICU76OSMZPETELOYRXTK5LQ, seg_pos={0 0}, upload=132752]: Done uploading piece. err: <nil> Nov 15 14:50:12.990960591 (150h46m11.542431052s) [noise=true , address=77.90.31.15:4699 , piece_id=AH4N55LZLPUJGQOPDLXM6I65SUUKOHUCPXPYE2I7XKI5KHYNJPWA, seg_pos={0 0}, upload=132752]: Done uploading piece. err: <nil> Nov 15 14:50:38.328113117 (150h46m36.879583578s) [noise=true , address=145.79.227.10:4401 , piece_id=SZWIK2OAO3YVLHLAU5FU6KHXODXVXDANAJP5WEKRSLUXDT4H6SYA, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: write tcp 188.166.225.170:34644->145.79.227.10:4401: use of closed network connection; write tcp 188.166.225.170:34644->145.79.227.10:4401: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34644->145.79.227.10:4401: use of closed network connection Nov 15 14:50:38.328591017 (150h46m36.880061478s) [noise=true , address=145.79.227.24:4581 , piece_id=E63A2HTXTWEQCYTOZA47S24YKVDKQ3BHAVZUSKUDALZWVIY6NMOQ, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:123ib1A8nmVmR272odjhVsUgADYbaMRMiPahSLoZ5eamaV6Qbb1, address:145.79.227.24:4581): protocol: read tcp 188.166.225.170:50846->145.79.227.24:4581: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 14:50:38.328758827 (150h46m36.880229318s) [noise=true , address=145.79.227.8:4330 , piece_id=MG7OE5OYG64CQUEGAUZNL654VHQQV347RSY2OKZAY5MXDXOSCMRQ, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW, address:145.79.227.8:4330): protocol: read tcp 188.166.225.170:43188->145.79.227.8:4330: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 14:50:38.328953126 (150h46m36.880423587s) [noise=true , address=145.79.227.32:4834 , piece_id=CJN674FREG73VEEP362QDU7HAAIDOCCYFGSL6ZE7QPDC3NHPWK2A, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: read tcp 188.166.225.170:38312->145.79.227.32:4834: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 14:50:38.329418293 (150h46m36.880888754s) [noise=true , address=145.79.227.9:4367 , piece_id=EFWN62D4D4U2FX7D53NGUNY3UVJBDZTMD7DC47N5IEW6SVXLIF7A, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6, address:145.79.227.9:4367): protocol: write tcp 188.166.225.170:36134->145.79.227.9:4367: use of closed network connection; write tcp 188.166.225.170:36134->145.79.227.9:4367: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36134->145.79.227.9:4367: use of closed network connection Nov 15 14:50:38.329617381 (150h46m36.881087842s) [noise=true , address=145.79.227.13:4803 , piece_id=H2K3B22G6YTEGGS2Y76DCJJHIQIKGXZMDDPQ4MMCUPJGQIRWO5HA, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:145.79.227.13:4803): protocol: read tcp 188.166.225.170:50394->145.79.227.13:4803: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 14:50:38.329698186 (150h46m36.881168657s) [noise=true , address=145.79.227.22:4509 , piece_id=AUB7HF5665Y26RSULL2WH6PUOQJUXSQT66OFMCM5YZWWRPPI6SRA, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:1229E58YirF37sZU6hfZGrDNFSxQDYsWSo4x3mG9BoYsNsLNtjy, address:145.79.227.22:4509): protocol: rpc: dial tcp 145.79.227.22:4509: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 14:50:38.329926219 (150h46m36.88139668s) [noise=true , address=145.79.227.8:4329 , piece_id=7IRP7Y6YSPNBN7DJJOMPPB7T2PDQ43G55MFAYHAWIT3Y4ULXSXZA, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: read tcp 188.166.225.170:44482->145.79.227.8:4329: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 14:50:38.330195863 (150h46m36.881666334s) [noise=true , address=145.79.227.16:4910 , piece_id=7BH46URBTAPGFQ2JKIXUUG4HMX6ERQS34BHPGBDQBMSNCVPVTXZQ, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: read tcp 188.166.225.170:58432->145.79.227.16:4910: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 14:50:38.330267870 (150h46m36.881738331s) [noise=true , address=145.79.227.23:4545 , piece_id=HJFCVMDY2CHYOIS4NYNFEXWHVQ3JGIQ4WAMKT4BC4MTNAMJWVGEA, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:12P5cnEza1Xxwg4qo1nau11zYHsHxqEAwhFsnt2UE1RhtTrJmra, address:145.79.227.23:4545): protocol: read tcp 188.166.225.170:42348->145.79.227.23:4545: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 14:50:38.330623417 (150h46m36.882093898s) [noise=true , address=145.79.227.27:4689 , piece_id=5GEUPACS2UDKXX6RZAL7AOUVXIF3HQRAYZUCAY54GFANZ3ODTYBA, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, address:145.79.227.27:4689): protocol: read tcp 188.166.225.170:52596->145.79.227.27:4689: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 14:50:38.330679213 (150h46m36.882149675s) [seg_pos={0 0}, upload=132752]: Done waiting for segment. successful: 5 optimal: 13 errs: <nil> Nov 15 14:50:38.589562836 (150h46m37.141033297s) [upload=132752]: Done uploading segments... Nov 15 14:50:38.589597782 (150h46m37.141068243s) [upload=132752]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 5) Duration: 6.334188699s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 11 out of 60 is failed (threshold=6) [XXX...X.XXX.XXXX............................................] duration: 3.566423254s trace: https://jaeger.dev.storj.io/trace/42d763a000d5a90d FAIL (2025-11-15 15:18:56 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.566493989s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 6.06881146s trace: https://jaeger.dev.storj.io/trace/495e264ce0048970 FAIL (2025-11-15 15:18:52 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 5) ======LOGS====== T4H6SYA, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.348693684 (150h46m7.900164155s) [noise=true , address=145.79.227.13:4803 , piece_id=H2K3B22G6YTEGGS2Y76DCJJHIQIKGXZMDDPQ4MMCUPJGQIRWO5HA, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.349289590 (150h46m7.900760051s) [noise=true , address=77.90.31.15:4699 , piece_id=AH4N55LZLPUJGQOPDLXM6I65SUUKOHUCPXPYE2I7XKI5KHYNJPWA, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.349845298 (150h46m7.901315759s) [noise=true , address=145.79.227.27:4689 , piece_id=5GEUPACS2UDKXX6RZAL7AOUVXIF3HQRAYZUCAY54GFANZ3ODTYBA, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.350334239 (150h46m7.9018047s) [noise=true , address=45.91.116.7:4011 , piece_id=CSTREAXVJWKIACVOGMDI7QXPAU7JK624TZMQIOB7A2UDDJ3YBOTA, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.350770229 (150h46m7.90224069s) [noise=true , address=145.79.227.22:4509 , piece_id=AUB7HF5665Y26RSULL2WH6PUOQJUXSQT66OFMCM5YZWWRPPI6SRA, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.351317160 (150h46m7.902787632s) [noise=true , address=45.91.116.12:4226 , piece_id=T64PPRSXNN5ZPKAY4YXS5WWAXCNUU4G7Y7TXJXLE4VBNULK3OGMA, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.353921029 (150h46m7.9053915s) [noise=true , address=45.91.116.5:4299 , piece_id=5FETUEJKO5PEHNGKOTPFAAXSPRY7WU5WPSEULBVJ2DPDINPN3V3A, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.354632563 (150h46m7.906103024s) [noise=true , address=145.79.227.8:4329 , piece_id=7IRP7Y6YSPNBN7DJJOMPPB7T2PDQ43G55MFAYHAWIT3Y4ULXSXZA, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.355164717 (150h46m7.906635178s) [noise=true , address=145.79.227.24:4581 , piece_id=E63A2HTXTWEQCYTOZA47S24YKVDKQ3BHAVZUSKUDALZWVIY6NMOQ, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.355607720 (150h46m7.907078181s) [noise=true , address=145.79.227.16:4910 , piece_id=7BH46URBTAPGFQ2JKIXUUG4HMX6ERQS34BHPGBDQBMSNCVPVTXZQ, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.356053489 (150h46m7.90752395s) [noise=true , address=145.79.227.9:4367 , piece_id=EFWN62D4D4U2FX7D53NGUNY3UVJBDZTMD7DC47N5IEW6SVXLIF7A, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.357082539 (150h46m7.90855301s) [noise=true , address=145.79.227.23:4545 , piece_id=HJFCVMDY2CHYOIS4NYNFEXWHVQ3JGIQ4WAMKT4BC4MTNAMJWVGEA, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.360976825 (150h46m7.912447286s) [noise=true , address=77.90.31.17:4772 , piece_id=ZTLV4QEMHCQJUBZ7CZWQK27Z6IKXYICU76OSMZPETELOYRXTK5LQ, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:11.203310499 (150h46m9.754780951s) [upload=132752]: Next returned nil segment Nov 15 14:50:11.203370323 (150h46m9.754840784s) [upload=132752]: Waiting for error group managing segments... Nov 15 14:50:11.496843592 (150h46m10.048314053s) [noise=true , address=45.91.116.5:4299 , piece_id=5FETUEJKO5PEHNGKOTPFAAXSPRY7WU5WPSEULBVJ2DPDINPN3V3A, seg_pos={0 0}, upload=132752]: Done uploading piece. err: <nil> Nov 15 14:50:11.503251574 (150h46m10.054722025s) [noise=true , address=45.91.116.7:4011 , piece_id=CSTREAXVJWKIACVOGMDI7QXPAU7JK624TZMQIOB7A2UDDJ3YBOTA, seg_pos={0 0}, upload=132752]: Done uploading piece. err: <nil> Nov 15 14:50:11.519837998 (150h46m10.071308449s) [noise=true , address=45.91.116.12:4226 , piece_id=T64PPRSXNN5ZPKAY4YXS5WWAXCNUU4G7Y7TXJXLE4VBNULK3OGMA, seg_pos={0 0}, upload=132752]: Done uploading piece. err: <nil> Nov 15 14:50:11.587364703 (150h46m10.138835164s) [noise=true , address=77.90.31.17:4772 , piece_id=ZTLV4QEMHCQJUBZ7CZWQK27Z6IKXYICU76OSMZPETELOYRXTK5LQ, seg_pos={0 0}, upload=132752]: Done uploading piece. err: <nil> Nov 15 14:50:12.990960591 (150h46m11.542431052s) [noise=true , address=77.90.31.15:4699 , piece_id=AH4N55LZLPUJGQOPDLXM6I65SUUKOHUCPXPYE2I7XKI5KHYNJPWA, seg_pos={0 0}, upload=132752]: Done uploading piece. err: <nil> Nov 15 14:50:38.328113117 (150h46m36.879583578s) [noise=true , address=145.79.227.10:4401 , piece_id=SZWIK2OAO3YVLHLAU5FU6KHXODXVXDANAJP5WEKRSLUXDT4H6SYA, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: write tcp 188.166.225.170:34644->145.79.227.10:4401: use of closed network connection; write tcp 188.166.225.170:34644->145.79.227.10:4401: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34644->145.79.227.10:4401: use of closed network connection Nov 15 14:50:38.328591017 (150h46m36.880061478s) [noise=true , address=145.79.227.24:4581 , piece_id=E63A2HTXTWEQCYTOZA47S24YKVDKQ3BHAVZUSKUDALZWVIY6NMOQ, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:123ib1A8nmVmR272odjhVsUgADYbaMRMiPahSLoZ5eamaV6Qbb1, address:145.79.227.24:4581): protocol: read tcp 188.166.225.170:50846->145.79.227.24:4581: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 14:50:38.328758827 (150h46m36.880229318s) [noise=true , address=145.79.227.8:4330 , piece_id=MG7OE5OYG64CQUEGAUZNL654VHQQV347RSY2OKZAY5MXDXOSCMRQ, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW, address:145.79.227.8:4330): protocol: read tcp 188.166.225.170:43188->145.79.227.8:4330: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 14:50:38.328953126 (150h46m36.880423587s) [noise=true , address=145.79.227.32:4834 , piece_id=CJN674FREG73VEEP362QDU7HAAIDOCCYFGSL6ZE7QPDC3NHPWK2A, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: read tcp 188.166.225.170:38312->145.79.227.32:4834: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 14:50:38.329418293 (150h46m36.880888754s) [noise=true , address=145.79.227.9:4367 , piece_id=EFWN62D4D4U2FX7D53NGUNY3UVJBDZTMD7DC47N5IEW6SVXLIF7A, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6, address:145.79.227.9:4367): protocol: write tcp 188.166.225.170:36134->145.79.227.9:4367: use of closed network connection; write tcp 188.166.225.170:36134->145.79.227.9:4367: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36134->145.79.227.9:4367: use of closed network connection Nov 15 14:50:38.329617381 (150h46m36.881087842s) [noise=true , address=145.79.227.13:4803 , piece_id=H2K3B22G6YTEGGS2Y76DCJJHIQIKGXZMDDPQ4MMCUPJGQIRWO5HA, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:145.79.227.13:4803): protocol: read tcp 188.166.225.170:50394->145.79.227.13:4803: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 14:50:38.329698186 (150h46m36.881168657s) [noise=true , address=145.79.227.22:4509 , piece_id=AUB7HF5665Y26RSULL2WH6PUOQJUXSQT66OFMCM5YZWWRPPI6SRA, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:1229E58YirF37sZU6hfZGrDNFSxQDYsWSo4x3mG9BoYsNsLNtjy, address:145.79.227.22:4509): protocol: rpc: dial tcp 145.79.227.22:4509: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 14:50:38.329926219 (150h46m36.88139668s) [noise=true , address=145.79.227.8:4329 , piece_id=7IRP7Y6YSPNBN7DJJOMPPB7T2PDQ43G55MFAYHAWIT3Y4ULXSXZA, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: read tcp 188.166.225.170:44482->145.79.227.8:4329: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 14:50:38.330195863 (150h46m36.881666334s) [noise=true , address=145.79.227.16:4910 , piece_id=7BH46URBTAPGFQ2JKIXUUG4HMX6ERQS34BHPGBDQBMSNCVPVTXZQ, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: read tcp 188.166.225.170:58432->145.79.227.16:4910: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 14:50:38.330267870 (150h46m36.881738331s) [noise=true , address=145.79.227.23:4545 , piece_id=HJFCVMDY2CHYOIS4NYNFEXWHVQ3JGIQ4WAMKT4BC4MTNAMJWVGEA, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:12P5cnEza1Xxwg4qo1nau11zYHsHxqEAwhFsnt2UE1RhtTrJmra, address:145.79.227.23:4545): protocol: read tcp 188.166.225.170:42348->145.79.227.23:4545: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 14:50:38.330623417 (150h46m36.882093898s) [noise=true , address=145.79.227.27:4689 , piece_id=5GEUPACS2UDKXX6RZAL7AOUVXIF3HQRAYZUCAY54GFANZ3ODTYBA, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, address:145.79.227.27:4689): protocol: read tcp 188.166.225.170:52596->145.79.227.27:4689: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 14:50:38.330679213 (150h46m36.882149675s) [seg_pos={0 0}, upload=132752]: Done waiting for segment. successful: 5 optimal: 13 errs: <nil> Nov 15 14:50:38.589562836 (150h46m37.141033297s) [upload=132752]: Done uploading segments... Nov 15 14:50:38.589597782 (150h46m37.141068243s) [upload=132752]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 5) Duration: 6.068842127s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 12 out of 60 is failed (threshold=6) [XXXX...X.XXX.XXXX...........................................] duration: 3.477416596s trace: https://jaeger.dev.storj.io/trace/42d763a000d5a90d FAIL (2025-11-15 15:17:48 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.477444469s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 6.02611191s trace: https://jaeger.dev.storj.io/trace/495e264ce0048970 FAIL (2025-11-15 15:17:42 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 5) ======LOGS====== T4H6SYA, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.348693684 (150h46m7.900164155s) [noise=true , address=145.79.227.13:4803 , piece_id=H2K3B22G6YTEGGS2Y76DCJJHIQIKGXZMDDPQ4MMCUPJGQIRWO5HA, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.349289590 (150h46m7.900760051s) [noise=true , address=77.90.31.15:4699 , piece_id=AH4N55LZLPUJGQOPDLXM6I65SUUKOHUCPXPYE2I7XKI5KHYNJPWA, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.349845298 (150h46m7.901315759s) [noise=true , address=145.79.227.27:4689 , piece_id=5GEUPACS2UDKXX6RZAL7AOUVXIF3HQRAYZUCAY54GFANZ3ODTYBA, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.350334239 (150h46m7.9018047s) [noise=true , address=45.91.116.7:4011 , piece_id=CSTREAXVJWKIACVOGMDI7QXPAU7JK624TZMQIOB7A2UDDJ3YBOTA, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.350770229 (150h46m7.90224069s) [noise=true , address=145.79.227.22:4509 , piece_id=AUB7HF5665Y26RSULL2WH6PUOQJUXSQT66OFMCM5YZWWRPPI6SRA, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.351317160 (150h46m7.902787632s) [noise=true , address=45.91.116.12:4226 , piece_id=T64PPRSXNN5ZPKAY4YXS5WWAXCNUU4G7Y7TXJXLE4VBNULK3OGMA, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.353921029 (150h46m7.9053915s) [noise=true , address=45.91.116.5:4299 , piece_id=5FETUEJKO5PEHNGKOTPFAAXSPRY7WU5WPSEULBVJ2DPDINPN3V3A, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.354632563 (150h46m7.906103024s) [noise=true , address=145.79.227.8:4329 , piece_id=7IRP7Y6YSPNBN7DJJOMPPB7T2PDQ43G55MFAYHAWIT3Y4ULXSXZA, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.355164717 (150h46m7.906635178s) [noise=true , address=145.79.227.24:4581 , piece_id=E63A2HTXTWEQCYTOZA47S24YKVDKQ3BHAVZUSKUDALZWVIY6NMOQ, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.355607720 (150h46m7.907078181s) [noise=true , address=145.79.227.16:4910 , piece_id=7BH46URBTAPGFQ2JKIXUUG4HMX6ERQS34BHPGBDQBMSNCVPVTXZQ, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.356053489 (150h46m7.90752395s) [noise=true , address=145.79.227.9:4367 , piece_id=EFWN62D4D4U2FX7D53NGUNY3UVJBDZTMD7DC47N5IEW6SVXLIF7A, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.357082539 (150h46m7.90855301s) [noise=true , address=145.79.227.23:4545 , piece_id=HJFCVMDY2CHYOIS4NYNFEXWHVQ3JGIQ4WAMKT4BC4MTNAMJWVGEA, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.360976825 (150h46m7.912447286s) [noise=true , address=77.90.31.17:4772 , piece_id=ZTLV4QEMHCQJUBZ7CZWQK27Z6IKXYICU76OSMZPETELOYRXTK5LQ, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:11.203310499 (150h46m9.754780951s) [upload=132752]: Next returned nil segment Nov 15 14:50:11.203370323 (150h46m9.754840784s) [upload=132752]: Waiting for error group managing segments... Nov 15 14:50:11.496843592 (150h46m10.048314053s) [noise=true , address=45.91.116.5:4299 , piece_id=5FETUEJKO5PEHNGKOTPFAAXSPRY7WU5WPSEULBVJ2DPDINPN3V3A, seg_pos={0 0}, upload=132752]: Done uploading piece. err: <nil> Nov 15 14:50:11.503251574 (150h46m10.054722025s) [noise=true , address=45.91.116.7:4011 , piece_id=CSTREAXVJWKIACVOGMDI7QXPAU7JK624TZMQIOB7A2UDDJ3YBOTA, seg_pos={0 0}, upload=132752]: Done uploading piece. err: <nil> Nov 15 14:50:11.519837998 (150h46m10.071308449s) [noise=true , address=45.91.116.12:4226 , piece_id=T64PPRSXNN5ZPKAY4YXS5WWAXCNUU4G7Y7TXJXLE4VBNULK3OGMA, seg_pos={0 0}, upload=132752]: Done uploading piece. err: <nil> Nov 15 14:50:11.587364703 (150h46m10.138835164s) [noise=true , address=77.90.31.17:4772 , piece_id=ZTLV4QEMHCQJUBZ7CZWQK27Z6IKXYICU76OSMZPETELOYRXTK5LQ, seg_pos={0 0}, upload=132752]: Done uploading piece. err: <nil> Nov 15 14:50:12.990960591 (150h46m11.542431052s) [noise=true , address=77.90.31.15:4699 , piece_id=AH4N55LZLPUJGQOPDLXM6I65SUUKOHUCPXPYE2I7XKI5KHYNJPWA, seg_pos={0 0}, upload=132752]: Done uploading piece. err: <nil> Nov 15 14:50:38.328113117 (150h46m36.879583578s) [noise=true , address=145.79.227.10:4401 , piece_id=SZWIK2OAO3YVLHLAU5FU6KHXODXVXDANAJP5WEKRSLUXDT4H6SYA, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: write tcp 188.166.225.170:34644->145.79.227.10:4401: use of closed network connection; write tcp 188.166.225.170:34644->145.79.227.10:4401: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34644->145.79.227.10:4401: use of closed network connection Nov 15 14:50:38.328591017 (150h46m36.880061478s) [noise=true , address=145.79.227.24:4581 , piece_id=E63A2HTXTWEQCYTOZA47S24YKVDKQ3BHAVZUSKUDALZWVIY6NMOQ, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:123ib1A8nmVmR272odjhVsUgADYbaMRMiPahSLoZ5eamaV6Qbb1, address:145.79.227.24:4581): protocol: read tcp 188.166.225.170:50846->145.79.227.24:4581: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 14:50:38.328758827 (150h46m36.880229318s) [noise=true , address=145.79.227.8:4330 , piece_id=MG7OE5OYG64CQUEGAUZNL654VHQQV347RSY2OKZAY5MXDXOSCMRQ, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW, address:145.79.227.8:4330): protocol: read tcp 188.166.225.170:43188->145.79.227.8:4330: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 14:50:38.328953126 (150h46m36.880423587s) [noise=true , address=145.79.227.32:4834 , piece_id=CJN674FREG73VEEP362QDU7HAAIDOCCYFGSL6ZE7QPDC3NHPWK2A, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: read tcp 188.166.225.170:38312->145.79.227.32:4834: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 14:50:38.329418293 (150h46m36.880888754s) [noise=true , address=145.79.227.9:4367 , piece_id=EFWN62D4D4U2FX7D53NGUNY3UVJBDZTMD7DC47N5IEW6SVXLIF7A, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6, address:145.79.227.9:4367): protocol: write tcp 188.166.225.170:36134->145.79.227.9:4367: use of closed network connection; write tcp 188.166.225.170:36134->145.79.227.9:4367: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36134->145.79.227.9:4367: use of closed network connection Nov 15 14:50:38.329617381 (150h46m36.881087842s) [noise=true , address=145.79.227.13:4803 , piece_id=H2K3B22G6YTEGGS2Y76DCJJHIQIKGXZMDDPQ4MMCUPJGQIRWO5HA, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:145.79.227.13:4803): protocol: read tcp 188.166.225.170:50394->145.79.227.13:4803: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 14:50:38.329698186 (150h46m36.881168657s) [noise=true , address=145.79.227.22:4509 , piece_id=AUB7HF5665Y26RSULL2WH6PUOQJUXSQT66OFMCM5YZWWRPPI6SRA, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:1229E58YirF37sZU6hfZGrDNFSxQDYsWSo4x3mG9BoYsNsLNtjy, address:145.79.227.22:4509): protocol: rpc: dial tcp 145.79.227.22:4509: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 14:50:38.329926219 (150h46m36.88139668s) [noise=true , address=145.79.227.8:4329 , piece_id=7IRP7Y6YSPNBN7DJJOMPPB7T2PDQ43G55MFAYHAWIT3Y4ULXSXZA, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: read tcp 188.166.225.170:44482->145.79.227.8:4329: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 14:50:38.330195863 (150h46m36.881666334s) [noise=true , address=145.79.227.16:4910 , piece_id=7BH46URBTAPGFQ2JKIXUUG4HMX6ERQS34BHPGBDQBMSNCVPVTXZQ, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: read tcp 188.166.225.170:58432->145.79.227.16:4910: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 14:50:38.330267870 (150h46m36.881738331s) [noise=true , address=145.79.227.23:4545 , piece_id=HJFCVMDY2CHYOIS4NYNFEXWHVQ3JGIQ4WAMKT4BC4MTNAMJWVGEA, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:12P5cnEza1Xxwg4qo1nau11zYHsHxqEAwhFsnt2UE1RhtTrJmra, address:145.79.227.23:4545): protocol: read tcp 188.166.225.170:42348->145.79.227.23:4545: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 14:50:38.330623417 (150h46m36.882093898s) [noise=true , address=145.79.227.27:4689 , piece_id=5GEUPACS2UDKXX6RZAL7AOUVXIF3HQRAYZUCAY54GFANZ3ODTYBA, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, address:145.79.227.27:4689): protocol: read tcp 188.166.225.170:52596->145.79.227.27:4689: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 14:50:38.330679213 (150h46m36.882149675s) [seg_pos={0 0}, upload=132752]: Done waiting for segment. successful: 5 optimal: 13 errs: <nil> Nov 15 14:50:38.589562836 (150h46m37.141033297s) [upload=132752]: Done uploading segments... Nov 15 14:50:38.589597782 (150h46m37.141068243s) [upload=132752]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 5) Duration: 6.026135425s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 11.742590208s trace: https://jaeger.dev.storj.io/trace/495e264ce0048970 FAIL (2025-11-15 15:16:58 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 5) ======LOGS====== T4H6SYA, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.348693684 (150h46m7.900164155s) [noise=true , address=145.79.227.13:4803 , piece_id=H2K3B22G6YTEGGS2Y76DCJJHIQIKGXZMDDPQ4MMCUPJGQIRWO5HA, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.349289590 (150h46m7.900760051s) [noise=true , address=77.90.31.15:4699 , piece_id=AH4N55LZLPUJGQOPDLXM6I65SUUKOHUCPXPYE2I7XKI5KHYNJPWA, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.349845298 (150h46m7.901315759s) [noise=true , address=145.79.227.27:4689 , piece_id=5GEUPACS2UDKXX6RZAL7AOUVXIF3HQRAYZUCAY54GFANZ3ODTYBA, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.350334239 (150h46m7.9018047s) [noise=true , address=45.91.116.7:4011 , piece_id=CSTREAXVJWKIACVOGMDI7QXPAU7JK624TZMQIOB7A2UDDJ3YBOTA, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.350770229 (150h46m7.90224069s) [noise=true , address=145.79.227.22:4509 , piece_id=AUB7HF5665Y26RSULL2WH6PUOQJUXSQT66OFMCM5YZWWRPPI6SRA, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.351317160 (150h46m7.902787632s) [noise=true , address=45.91.116.12:4226 , piece_id=T64PPRSXNN5ZPKAY4YXS5WWAXCNUU4G7Y7TXJXLE4VBNULK3OGMA, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.353921029 (150h46m7.9053915s) [noise=true , address=45.91.116.5:4299 , piece_id=5FETUEJKO5PEHNGKOTPFAAXSPRY7WU5WPSEULBVJ2DPDINPN3V3A, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.354632563 (150h46m7.906103024s) [noise=true , address=145.79.227.8:4329 , piece_id=7IRP7Y6YSPNBN7DJJOMPPB7T2PDQ43G55MFAYHAWIT3Y4ULXSXZA, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.355164717 (150h46m7.906635178s) [noise=true , address=145.79.227.24:4581 , piece_id=E63A2HTXTWEQCYTOZA47S24YKVDKQ3BHAVZUSKUDALZWVIY6NMOQ, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.355607720 (150h46m7.907078181s) [noise=true , address=145.79.227.16:4910 , piece_id=7BH46URBTAPGFQ2JKIXUUG4HMX6ERQS34BHPGBDQBMSNCVPVTXZQ, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.356053489 (150h46m7.90752395s) [noise=true , address=145.79.227.9:4367 , piece_id=EFWN62D4D4U2FX7D53NGUNY3UVJBDZTMD7DC47N5IEW6SVXLIF7A, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.357082539 (150h46m7.90855301s) [noise=true , address=145.79.227.23:4545 , piece_id=HJFCVMDY2CHYOIS4NYNFEXWHVQ3JGIQ4WAMKT4BC4MTNAMJWVGEA, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.360976825 (150h46m7.912447286s) [noise=true , address=77.90.31.17:4772 , piece_id=ZTLV4QEMHCQJUBZ7CZWQK27Z6IKXYICU76OSMZPETELOYRXTK5LQ, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:11.203310499 (150h46m9.754780951s) [upload=132752]: Next returned nil segment Nov 15 14:50:11.203370323 (150h46m9.754840784s) [upload=132752]: Waiting for error group managing segments... Nov 15 14:50:11.496843592 (150h46m10.048314053s) [noise=true , address=45.91.116.5:4299 , piece_id=5FETUEJKO5PEHNGKOTPFAAXSPRY7WU5WPSEULBVJ2DPDINPN3V3A, seg_pos={0 0}, upload=132752]: Done uploading piece. err: <nil> Nov 15 14:50:11.503251574 (150h46m10.054722025s) [noise=true , address=45.91.116.7:4011 , piece_id=CSTREAXVJWKIACVOGMDI7QXPAU7JK624TZMQIOB7A2UDDJ3YBOTA, seg_pos={0 0}, upload=132752]: Done uploading piece. err: <nil> Nov 15 14:50:11.519837998 (150h46m10.071308449s) [noise=true , address=45.91.116.12:4226 , piece_id=T64PPRSXNN5ZPKAY4YXS5WWAXCNUU4G7Y7TXJXLE4VBNULK3OGMA, seg_pos={0 0}, upload=132752]: Done uploading piece. err: <nil> Nov 15 14:50:11.587364703 (150h46m10.138835164s) [noise=true , address=77.90.31.17:4772 , piece_id=ZTLV4QEMHCQJUBZ7CZWQK27Z6IKXYICU76OSMZPETELOYRXTK5LQ, seg_pos={0 0}, upload=132752]: Done uploading piece. err: <nil> Nov 15 14:50:12.990960591 (150h46m11.542431052s) [noise=true , address=77.90.31.15:4699 , piece_id=AH4N55LZLPUJGQOPDLXM6I65SUUKOHUCPXPYE2I7XKI5KHYNJPWA, seg_pos={0 0}, upload=132752]: Done uploading piece. err: <nil> Nov 15 14:50:38.328113117 (150h46m36.879583578s) [noise=true , address=145.79.227.10:4401 , piece_id=SZWIK2OAO3YVLHLAU5FU6KHXODXVXDANAJP5WEKRSLUXDT4H6SYA, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: write tcp 188.166.225.170:34644->145.79.227.10:4401: use of closed network connection; write tcp 188.166.225.170:34644->145.79.227.10:4401: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34644->145.79.227.10:4401: use of closed network connection Nov 15 14:50:38.328591017 (150h46m36.880061478s) [noise=true , address=145.79.227.24:4581 , piece_id=E63A2HTXTWEQCYTOZA47S24YKVDKQ3BHAVZUSKUDALZWVIY6NMOQ, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:123ib1A8nmVmR272odjhVsUgADYbaMRMiPahSLoZ5eamaV6Qbb1, address:145.79.227.24:4581): protocol: read tcp 188.166.225.170:50846->145.79.227.24:4581: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 14:50:38.328758827 (150h46m36.880229318s) [noise=true , address=145.79.227.8:4330 , piece_id=MG7OE5OYG64CQUEGAUZNL654VHQQV347RSY2OKZAY5MXDXOSCMRQ, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW, address:145.79.227.8:4330): protocol: read tcp 188.166.225.170:43188->145.79.227.8:4330: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 14:50:38.328953126 (150h46m36.880423587s) [noise=true , address=145.79.227.32:4834 , piece_id=CJN674FREG73VEEP362QDU7HAAIDOCCYFGSL6ZE7QPDC3NHPWK2A, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: read tcp 188.166.225.170:38312->145.79.227.32:4834: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 14:50:38.329418293 (150h46m36.880888754s) [noise=true , address=145.79.227.9:4367 , piece_id=EFWN62D4D4U2FX7D53NGUNY3UVJBDZTMD7DC47N5IEW6SVXLIF7A, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6, address:145.79.227.9:4367): protocol: write tcp 188.166.225.170:36134->145.79.227.9:4367: use of closed network connection; write tcp 188.166.225.170:36134->145.79.227.9:4367: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36134->145.79.227.9:4367: use of closed network connection Nov 15 14:50:38.329617381 (150h46m36.881087842s) [noise=true , address=145.79.227.13:4803 , piece_id=H2K3B22G6YTEGGS2Y76DCJJHIQIKGXZMDDPQ4MMCUPJGQIRWO5HA, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:145.79.227.13:4803): protocol: read tcp 188.166.225.170:50394->145.79.227.13:4803: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 14:50:38.329698186 (150h46m36.881168657s) [noise=true , address=145.79.227.22:4509 , piece_id=AUB7HF5665Y26RSULL2WH6PUOQJUXSQT66OFMCM5YZWWRPPI6SRA, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:1229E58YirF37sZU6hfZGrDNFSxQDYsWSo4x3mG9BoYsNsLNtjy, address:145.79.227.22:4509): protocol: rpc: dial tcp 145.79.227.22:4509: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 14:50:38.329926219 (150h46m36.88139668s) [noise=true , address=145.79.227.8:4329 , piece_id=7IRP7Y6YSPNBN7DJJOMPPB7T2PDQ43G55MFAYHAWIT3Y4ULXSXZA, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: read tcp 188.166.225.170:44482->145.79.227.8:4329: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 14:50:38.330195863 (150h46m36.881666334s) [noise=true , address=145.79.227.16:4910 , piece_id=7BH46URBTAPGFQ2JKIXUUG4HMX6ERQS34BHPGBDQBMSNCVPVTXZQ, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: read tcp 188.166.225.170:58432->145.79.227.16:4910: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 14:50:38.330267870 (150h46m36.881738331s) [noise=true , address=145.79.227.23:4545 , piece_id=HJFCVMDY2CHYOIS4NYNFEXWHVQ3JGIQ4WAMKT4BC4MTNAMJWVGEA, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:12P5cnEza1Xxwg4qo1nau11zYHsHxqEAwhFsnt2UE1RhtTrJmra, address:145.79.227.23:4545): protocol: read tcp 188.166.225.170:42348->145.79.227.23:4545: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 14:50:38.330623417 (150h46m36.882093898s) [noise=true , address=145.79.227.27:4689 , piece_id=5GEUPACS2UDKXX6RZAL7AOUVXIF3HQRAYZUCAY54GFANZ3ODTYBA, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, address:145.79.227.27:4689): protocol: read tcp 188.166.225.170:52596->145.79.227.27:4689: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 14:50:38.330679213 (150h46m36.882149675s) [seg_pos={0 0}, upload=132752]: Done waiting for segment. successful: 5 optimal: 13 errs: <nil> Nov 15 14:50:38.589562836 (150h46m37.141033297s) [upload=132752]: Done uploading segments... Nov 15 14:50:38.589597782 (150h46m37.141068243s) [upload=132752]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 5) Duration: 11.742689136s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 12 out of 60 is failed (threshold=6) [.XXXX...X.XXX.XXXX..........................................] duration: 3.53907166s trace: https://jaeger.dev.storj.io/trace/42d763a000d5a90d FAIL (2025-11-15 15:16:45 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.53910291s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 6.265482695s trace: https://jaeger.dev.storj.io/trace/495e264ce0048970 FAIL (2025-11-15 15:16:05 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 5) ======LOGS====== T4H6SYA, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.348693684 (150h46m7.900164155s) [noise=true , address=145.79.227.13:4803 , piece_id=H2K3B22G6YTEGGS2Y76DCJJHIQIKGXZMDDPQ4MMCUPJGQIRWO5HA, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.349289590 (150h46m7.900760051s) [noise=true , address=77.90.31.15:4699 , piece_id=AH4N55LZLPUJGQOPDLXM6I65SUUKOHUCPXPYE2I7XKI5KHYNJPWA, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.349845298 (150h46m7.901315759s) [noise=true , address=145.79.227.27:4689 , piece_id=5GEUPACS2UDKXX6RZAL7AOUVXIF3HQRAYZUCAY54GFANZ3ODTYBA, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.350334239 (150h46m7.9018047s) [noise=true , address=45.91.116.7:4011 , piece_id=CSTREAXVJWKIACVOGMDI7QXPAU7JK624TZMQIOB7A2UDDJ3YBOTA, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.350770229 (150h46m7.90224069s) [noise=true , address=145.79.227.22:4509 , piece_id=AUB7HF5665Y26RSULL2WH6PUOQJUXSQT66OFMCM5YZWWRPPI6SRA, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.351317160 (150h46m7.902787632s) [noise=true , address=45.91.116.12:4226 , piece_id=T64PPRSXNN5ZPKAY4YXS5WWAXCNUU4G7Y7TXJXLE4VBNULK3OGMA, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.353921029 (150h46m7.9053915s) [noise=true , address=45.91.116.5:4299 , piece_id=5FETUEJKO5PEHNGKOTPFAAXSPRY7WU5WPSEULBVJ2DPDINPN3V3A, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.354632563 (150h46m7.906103024s) [noise=true , address=145.79.227.8:4329 , piece_id=7IRP7Y6YSPNBN7DJJOMPPB7T2PDQ43G55MFAYHAWIT3Y4ULXSXZA, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.355164717 (150h46m7.906635178s) [noise=true , address=145.79.227.24:4581 , piece_id=E63A2HTXTWEQCYTOZA47S24YKVDKQ3BHAVZUSKUDALZWVIY6NMOQ, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.355607720 (150h46m7.907078181s) [noise=true , address=145.79.227.16:4910 , piece_id=7BH46URBTAPGFQ2JKIXUUG4HMX6ERQS34BHPGBDQBMSNCVPVTXZQ, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.356053489 (150h46m7.90752395s) [noise=true , address=145.79.227.9:4367 , piece_id=EFWN62D4D4U2FX7D53NGUNY3UVJBDZTMD7DC47N5IEW6SVXLIF7A, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.357082539 (150h46m7.90855301s) [noise=true , address=145.79.227.23:4545 , piece_id=HJFCVMDY2CHYOIS4NYNFEXWHVQ3JGIQ4WAMKT4BC4MTNAMJWVGEA, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.360976825 (150h46m7.912447286s) [noise=true , address=77.90.31.17:4772 , piece_id=ZTLV4QEMHCQJUBZ7CZWQK27Z6IKXYICU76OSMZPETELOYRXTK5LQ, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:11.203310499 (150h46m9.754780951s) [upload=132752]: Next returned nil segment Nov 15 14:50:11.203370323 (150h46m9.754840784s) [upload=132752]: Waiting for error group managing segments... Nov 15 14:50:11.496843592 (150h46m10.048314053s) [noise=true , address=45.91.116.5:4299 , piece_id=5FETUEJKO5PEHNGKOTPFAAXSPRY7WU5WPSEULBVJ2DPDINPN3V3A, seg_pos={0 0}, upload=132752]: Done uploading piece. err: <nil> Nov 15 14:50:11.503251574 (150h46m10.054722025s) [noise=true , address=45.91.116.7:4011 , piece_id=CSTREAXVJWKIACVOGMDI7QXPAU7JK624TZMQIOB7A2UDDJ3YBOTA, seg_pos={0 0}, upload=132752]: Done uploading piece. err: <nil> Nov 15 14:50:11.519837998 (150h46m10.071308449s) [noise=true , address=45.91.116.12:4226 , piece_id=T64PPRSXNN5ZPKAY4YXS5WWAXCNUU4G7Y7TXJXLE4VBNULK3OGMA, seg_pos={0 0}, upload=132752]: Done uploading piece. err: <nil> Nov 15 14:50:11.587364703 (150h46m10.138835164s) [noise=true , address=77.90.31.17:4772 , piece_id=ZTLV4QEMHCQJUBZ7CZWQK27Z6IKXYICU76OSMZPETELOYRXTK5LQ, seg_pos={0 0}, upload=132752]: Done uploading piece. err: <nil> Nov 15 14:50:12.990960591 (150h46m11.542431052s) [noise=true , address=77.90.31.15:4699 , piece_id=AH4N55LZLPUJGQOPDLXM6I65SUUKOHUCPXPYE2I7XKI5KHYNJPWA, seg_pos={0 0}, upload=132752]: Done uploading piece. err: <nil> Nov 15 14:50:38.328113117 (150h46m36.879583578s) [noise=true , address=145.79.227.10:4401 , piece_id=SZWIK2OAO3YVLHLAU5FU6KHXODXVXDANAJP5WEKRSLUXDT4H6SYA, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: write tcp 188.166.225.170:34644->145.79.227.10:4401: use of closed network connection; write tcp 188.166.225.170:34644->145.79.227.10:4401: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34644->145.79.227.10:4401: use of closed network connection Nov 15 14:50:38.328591017 (150h46m36.880061478s) [noise=true , address=145.79.227.24:4581 , piece_id=E63A2HTXTWEQCYTOZA47S24YKVDKQ3BHAVZUSKUDALZWVIY6NMOQ, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:123ib1A8nmVmR272odjhVsUgADYbaMRMiPahSLoZ5eamaV6Qbb1, address:145.79.227.24:4581): protocol: read tcp 188.166.225.170:50846->145.79.227.24:4581: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 14:50:38.328758827 (150h46m36.880229318s) [noise=true , address=145.79.227.8:4330 , piece_id=MG7OE5OYG64CQUEGAUZNL654VHQQV347RSY2OKZAY5MXDXOSCMRQ, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW, address:145.79.227.8:4330): protocol: read tcp 188.166.225.170:43188->145.79.227.8:4330: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 14:50:38.328953126 (150h46m36.880423587s) [noise=true , address=145.79.227.32:4834 , piece_id=CJN674FREG73VEEP362QDU7HAAIDOCCYFGSL6ZE7QPDC3NHPWK2A, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: read tcp 188.166.225.170:38312->145.79.227.32:4834: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 14:50:38.329418293 (150h46m36.880888754s) [noise=true , address=145.79.227.9:4367 , piece_id=EFWN62D4D4U2FX7D53NGUNY3UVJBDZTMD7DC47N5IEW6SVXLIF7A, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6, address:145.79.227.9:4367): protocol: write tcp 188.166.225.170:36134->145.79.227.9:4367: use of closed network connection; write tcp 188.166.225.170:36134->145.79.227.9:4367: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36134->145.79.227.9:4367: use of closed network connection Nov 15 14:50:38.329617381 (150h46m36.881087842s) [noise=true , address=145.79.227.13:4803 , piece_id=H2K3B22G6YTEGGS2Y76DCJJHIQIKGXZMDDPQ4MMCUPJGQIRWO5HA, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:145.79.227.13:4803): protocol: read tcp 188.166.225.170:50394->145.79.227.13:4803: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 14:50:38.329698186 (150h46m36.881168657s) [noise=true , address=145.79.227.22:4509 , piece_id=AUB7HF5665Y26RSULL2WH6PUOQJUXSQT66OFMCM5YZWWRPPI6SRA, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:1229E58YirF37sZU6hfZGrDNFSxQDYsWSo4x3mG9BoYsNsLNtjy, address:145.79.227.22:4509): protocol: rpc: dial tcp 145.79.227.22:4509: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 14:50:38.329926219 (150h46m36.88139668s) [noise=true , address=145.79.227.8:4329 , piece_id=7IRP7Y6YSPNBN7DJJOMPPB7T2PDQ43G55MFAYHAWIT3Y4ULXSXZA, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: read tcp 188.166.225.170:44482->145.79.227.8:4329: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 14:50:38.330195863 (150h46m36.881666334s) [noise=true , address=145.79.227.16:4910 , piece_id=7BH46URBTAPGFQ2JKIXUUG4HMX6ERQS34BHPGBDQBMSNCVPVTXZQ, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: read tcp 188.166.225.170:58432->145.79.227.16:4910: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 14:50:38.330267870 (150h46m36.881738331s) [noise=true , address=145.79.227.23:4545 , piece_id=HJFCVMDY2CHYOIS4NYNFEXWHVQ3JGIQ4WAMKT4BC4MTNAMJWVGEA, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:12P5cnEza1Xxwg4qo1nau11zYHsHxqEAwhFsnt2UE1RhtTrJmra, address:145.79.227.23:4545): protocol: read tcp 188.166.225.170:42348->145.79.227.23:4545: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 14:50:38.330623417 (150h46m36.882093898s) [noise=true , address=145.79.227.27:4689 , piece_id=5GEUPACS2UDKXX6RZAL7AOUVXIF3HQRAYZUCAY54GFANZ3ODTYBA, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, address:145.79.227.27:4689): protocol: read tcp 188.166.225.170:52596->145.79.227.27:4689: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 14:50:38.330679213 (150h46m36.882149675s) [seg_pos={0 0}, upload=132752]: Done waiting for segment. successful: 5 optimal: 13 errs: <nil> Nov 15 14:50:38.589562836 (150h46m37.141033297s) [upload=132752]: Done uploading segments... Nov 15 14:50:38.589597782 (150h46m37.141068243s) [upload=132752]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 5) Duration: 6.265514965s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 12 out of 60 is failed (threshold=6) [..XXXX...X.XXX.XXXX.........................................] duration: 3.543821927s trace: https://jaeger.dev.storj.io/trace/42d763a000d5a90d FAIL (2025-11-15 15:15:32 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.543921065s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 6.205219253s trace: https://jaeger.dev.storj.io/trace/495e264ce0048970 FAIL (2025-11-15 15:15:22 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 5) ======LOGS====== T4H6SYA, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.348693684 (150h46m7.900164155s) [noise=true , address=145.79.227.13:4803 , piece_id=H2K3B22G6YTEGGS2Y76DCJJHIQIKGXZMDDPQ4MMCUPJGQIRWO5HA, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.349289590 (150h46m7.900760051s) [noise=true , address=77.90.31.15:4699 , piece_id=AH4N55LZLPUJGQOPDLXM6I65SUUKOHUCPXPYE2I7XKI5KHYNJPWA, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.349845298 (150h46m7.901315759s) [noise=true , address=145.79.227.27:4689 , piece_id=5GEUPACS2UDKXX6RZAL7AOUVXIF3HQRAYZUCAY54GFANZ3ODTYBA, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.350334239 (150h46m7.9018047s) [noise=true , address=45.91.116.7:4011 , piece_id=CSTREAXVJWKIACVOGMDI7QXPAU7JK624TZMQIOB7A2UDDJ3YBOTA, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.350770229 (150h46m7.90224069s) [noise=true , address=145.79.227.22:4509 , piece_id=AUB7HF5665Y26RSULL2WH6PUOQJUXSQT66OFMCM5YZWWRPPI6SRA, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.351317160 (150h46m7.902787632s) [noise=true , address=45.91.116.12:4226 , piece_id=T64PPRSXNN5ZPKAY4YXS5WWAXCNUU4G7Y7TXJXLE4VBNULK3OGMA, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.353921029 (150h46m7.9053915s) [noise=true , address=45.91.116.5:4299 , piece_id=5FETUEJKO5PEHNGKOTPFAAXSPRY7WU5WPSEULBVJ2DPDINPN3V3A, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.354632563 (150h46m7.906103024s) [noise=true , address=145.79.227.8:4329 , piece_id=7IRP7Y6YSPNBN7DJJOMPPB7T2PDQ43G55MFAYHAWIT3Y4ULXSXZA, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.355164717 (150h46m7.906635178s) [noise=true , address=145.79.227.24:4581 , piece_id=E63A2HTXTWEQCYTOZA47S24YKVDKQ3BHAVZUSKUDALZWVIY6NMOQ, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.355607720 (150h46m7.907078181s) [noise=true , address=145.79.227.16:4910 , piece_id=7BH46URBTAPGFQ2JKIXUUG4HMX6ERQS34BHPGBDQBMSNCVPVTXZQ, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.356053489 (150h46m7.90752395s) [noise=true , address=145.79.227.9:4367 , piece_id=EFWN62D4D4U2FX7D53NGUNY3UVJBDZTMD7DC47N5IEW6SVXLIF7A, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.357082539 (150h46m7.90855301s) [noise=true , address=145.79.227.23:4545 , piece_id=HJFCVMDY2CHYOIS4NYNFEXWHVQ3JGIQ4WAMKT4BC4MTNAMJWVGEA, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.360976825 (150h46m7.912447286s) [noise=true , address=77.90.31.17:4772 , piece_id=ZTLV4QEMHCQJUBZ7CZWQK27Z6IKXYICU76OSMZPETELOYRXTK5LQ, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:11.203310499 (150h46m9.754780951s) [upload=132752]: Next returned nil segment Nov 15 14:50:11.203370323 (150h46m9.754840784s) [upload=132752]: Waiting for error group managing segments... Nov 15 14:50:11.496843592 (150h46m10.048314053s) [noise=true , address=45.91.116.5:4299 , piece_id=5FETUEJKO5PEHNGKOTPFAAXSPRY7WU5WPSEULBVJ2DPDINPN3V3A, seg_pos={0 0}, upload=132752]: Done uploading piece. err: <nil> Nov 15 14:50:11.503251574 (150h46m10.054722025s) [noise=true , address=45.91.116.7:4011 , piece_id=CSTREAXVJWKIACVOGMDI7QXPAU7JK624TZMQIOB7A2UDDJ3YBOTA, seg_pos={0 0}, upload=132752]: Done uploading piece. err: <nil> Nov 15 14:50:11.519837998 (150h46m10.071308449s) [noise=true , address=45.91.116.12:4226 , piece_id=T64PPRSXNN5ZPKAY4YXS5WWAXCNUU4G7Y7TXJXLE4VBNULK3OGMA, seg_pos={0 0}, upload=132752]: Done uploading piece. err: <nil> Nov 15 14:50:11.587364703 (150h46m10.138835164s) [noise=true , address=77.90.31.17:4772 , piece_id=ZTLV4QEMHCQJUBZ7CZWQK27Z6IKXYICU76OSMZPETELOYRXTK5LQ, seg_pos={0 0}, upload=132752]: Done uploading piece. err: <nil> Nov 15 14:50:12.990960591 (150h46m11.542431052s) [noise=true , address=77.90.31.15:4699 , piece_id=AH4N55LZLPUJGQOPDLXM6I65SUUKOHUCPXPYE2I7XKI5KHYNJPWA, seg_pos={0 0}, upload=132752]: Done uploading piece. err: <nil> Nov 15 14:50:38.328113117 (150h46m36.879583578s) [noise=true , address=145.79.227.10:4401 , piece_id=SZWIK2OAO3YVLHLAU5FU6KHXODXVXDANAJP5WEKRSLUXDT4H6SYA, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: write tcp 188.166.225.170:34644->145.79.227.10:4401: use of closed network connection; write tcp 188.166.225.170:34644->145.79.227.10:4401: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34644->145.79.227.10:4401: use of closed network connection Nov 15 14:50:38.328591017 (150h46m36.880061478s) [noise=true , address=145.79.227.24:4581 , piece_id=E63A2HTXTWEQCYTOZA47S24YKVDKQ3BHAVZUSKUDALZWVIY6NMOQ, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:123ib1A8nmVmR272odjhVsUgADYbaMRMiPahSLoZ5eamaV6Qbb1, address:145.79.227.24:4581): protocol: read tcp 188.166.225.170:50846->145.79.227.24:4581: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 14:50:38.328758827 (150h46m36.880229318s) [noise=true , address=145.79.227.8:4330 , piece_id=MG7OE5OYG64CQUEGAUZNL654VHQQV347RSY2OKZAY5MXDXOSCMRQ, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW, address:145.79.227.8:4330): protocol: read tcp 188.166.225.170:43188->145.79.227.8:4330: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 14:50:38.328953126 (150h46m36.880423587s) [noise=true , address=145.79.227.32:4834 , piece_id=CJN674FREG73VEEP362QDU7HAAIDOCCYFGSL6ZE7QPDC3NHPWK2A, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: read tcp 188.166.225.170:38312->145.79.227.32:4834: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 14:50:38.329418293 (150h46m36.880888754s) [noise=true , address=145.79.227.9:4367 , piece_id=EFWN62D4D4U2FX7D53NGUNY3UVJBDZTMD7DC47N5IEW6SVXLIF7A, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6, address:145.79.227.9:4367): protocol: write tcp 188.166.225.170:36134->145.79.227.9:4367: use of closed network connection; write tcp 188.166.225.170:36134->145.79.227.9:4367: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36134->145.79.227.9:4367: use of closed network connection Nov 15 14:50:38.329617381 (150h46m36.881087842s) [noise=true , address=145.79.227.13:4803 , piece_id=H2K3B22G6YTEGGS2Y76DCJJHIQIKGXZMDDPQ4MMCUPJGQIRWO5HA, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:145.79.227.13:4803): protocol: read tcp 188.166.225.170:50394->145.79.227.13:4803: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 14:50:38.329698186 (150h46m36.881168657s) [noise=true , address=145.79.227.22:4509 , piece_id=AUB7HF5665Y26RSULL2WH6PUOQJUXSQT66OFMCM5YZWWRPPI6SRA, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:1229E58YirF37sZU6hfZGrDNFSxQDYsWSo4x3mG9BoYsNsLNtjy, address:145.79.227.22:4509): protocol: rpc: dial tcp 145.79.227.22:4509: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 14:50:38.329926219 (150h46m36.88139668s) [noise=true , address=145.79.227.8:4329 , piece_id=7IRP7Y6YSPNBN7DJJOMPPB7T2PDQ43G55MFAYHAWIT3Y4ULXSXZA, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: read tcp 188.166.225.170:44482->145.79.227.8:4329: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 14:50:38.330195863 (150h46m36.881666334s) [noise=true , address=145.79.227.16:4910 , piece_id=7BH46URBTAPGFQ2JKIXUUG4HMX6ERQS34BHPGBDQBMSNCVPVTXZQ, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: read tcp 188.166.225.170:58432->145.79.227.16:4910: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 14:50:38.330267870 (150h46m36.881738331s) [noise=true , address=145.79.227.23:4545 , piece_id=HJFCVMDY2CHYOIS4NYNFEXWHVQ3JGIQ4WAMKT4BC4MTNAMJWVGEA, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:12P5cnEza1Xxwg4qo1nau11zYHsHxqEAwhFsnt2UE1RhtTrJmra, address:145.79.227.23:4545): protocol: read tcp 188.166.225.170:42348->145.79.227.23:4545: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 14:50:38.330623417 (150h46m36.882093898s) [noise=true , address=145.79.227.27:4689 , piece_id=5GEUPACS2UDKXX6RZAL7AOUVXIF3HQRAYZUCAY54GFANZ3ODTYBA, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, address:145.79.227.27:4689): protocol: read tcp 188.166.225.170:52596->145.79.227.27:4689: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 14:50:38.330679213 (150h46m36.882149675s) [seg_pos={0 0}, upload=132752]: Done waiting for segment. successful: 5 optimal: 13 errs: <nil> Nov 15 14:50:38.589562836 (150h46m37.141033297s) [upload=132752]: Done uploading segments... Nov 15 14:50:38.589597782 (150h46m37.141068243s) [upload=132752]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 5) Duration: 6.205253829s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 12 out of 60 is failed (threshold=6) [...XXXX...X.XXX.XXXX........................................] duration: 3.58855032s trace: https://jaeger.dev.storj.io/trace/42d763a000d5a90d FAIL (2025-11-15 15:15:15 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.588584044s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 6.086058089s trace: https://jaeger.dev.storj.io/trace/495e264ce0048970 FAIL (2025-11-15 15:15:03 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 5) ======LOGS====== T4H6SYA, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.348693684 (150h46m7.900164155s) [noise=true , address=145.79.227.13:4803 , piece_id=H2K3B22G6YTEGGS2Y76DCJJHIQIKGXZMDDPQ4MMCUPJGQIRWO5HA, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.349289590 (150h46m7.900760051s) [noise=true , address=77.90.31.15:4699 , piece_id=AH4N55LZLPUJGQOPDLXM6I65SUUKOHUCPXPYE2I7XKI5KHYNJPWA, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.349845298 (150h46m7.901315759s) [noise=true , address=145.79.227.27:4689 , piece_id=5GEUPACS2UDKXX6RZAL7AOUVXIF3HQRAYZUCAY54GFANZ3ODTYBA, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.350334239 (150h46m7.9018047s) [noise=true , address=45.91.116.7:4011 , piece_id=CSTREAXVJWKIACVOGMDI7QXPAU7JK624TZMQIOB7A2UDDJ3YBOTA, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.350770229 (150h46m7.90224069s) [noise=true , address=145.79.227.22:4509 , piece_id=AUB7HF5665Y26RSULL2WH6PUOQJUXSQT66OFMCM5YZWWRPPI6SRA, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.351317160 (150h46m7.902787632s) [noise=true , address=45.91.116.12:4226 , piece_id=T64PPRSXNN5ZPKAY4YXS5WWAXCNUU4G7Y7TXJXLE4VBNULK3OGMA, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.353921029 (150h46m7.9053915s) [noise=true , address=45.91.116.5:4299 , piece_id=5FETUEJKO5PEHNGKOTPFAAXSPRY7WU5WPSEULBVJ2DPDINPN3V3A, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.354632563 (150h46m7.906103024s) [noise=true , address=145.79.227.8:4329 , piece_id=7IRP7Y6YSPNBN7DJJOMPPB7T2PDQ43G55MFAYHAWIT3Y4ULXSXZA, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.355164717 (150h46m7.906635178s) [noise=true , address=145.79.227.24:4581 , piece_id=E63A2HTXTWEQCYTOZA47S24YKVDKQ3BHAVZUSKUDALZWVIY6NMOQ, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.355607720 (150h46m7.907078181s) [noise=true , address=145.79.227.16:4910 , piece_id=7BH46URBTAPGFQ2JKIXUUG4HMX6ERQS34BHPGBDQBMSNCVPVTXZQ, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.356053489 (150h46m7.90752395s) [noise=true , address=145.79.227.9:4367 , piece_id=EFWN62D4D4U2FX7D53NGUNY3UVJBDZTMD7DC47N5IEW6SVXLIF7A, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.357082539 (150h46m7.90855301s) [noise=true , address=145.79.227.23:4545 , piece_id=HJFCVMDY2CHYOIS4NYNFEXWHVQ3JGIQ4WAMKT4BC4MTNAMJWVGEA, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.360976825 (150h46m7.912447286s) [noise=true , address=77.90.31.17:4772 , piece_id=ZTLV4QEMHCQJUBZ7CZWQK27Z6IKXYICU76OSMZPETELOYRXTK5LQ, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:11.203310499 (150h46m9.754780951s) [upload=132752]: Next returned nil segment Nov 15 14:50:11.203370323 (150h46m9.754840784s) [upload=132752]: Waiting for error group managing segments... Nov 15 14:50:11.496843592 (150h46m10.048314053s) [noise=true , address=45.91.116.5:4299 , piece_id=5FETUEJKO5PEHNGKOTPFAAXSPRY7WU5WPSEULBVJ2DPDINPN3V3A, seg_pos={0 0}, upload=132752]: Done uploading piece. err: <nil> Nov 15 14:50:11.503251574 (150h46m10.054722025s) [noise=true , address=45.91.116.7:4011 , piece_id=CSTREAXVJWKIACVOGMDI7QXPAU7JK624TZMQIOB7A2UDDJ3YBOTA, seg_pos={0 0}, upload=132752]: Done uploading piece. err: <nil> Nov 15 14:50:11.519837998 (150h46m10.071308449s) [noise=true , address=45.91.116.12:4226 , piece_id=T64PPRSXNN5ZPKAY4YXS5WWAXCNUU4G7Y7TXJXLE4VBNULK3OGMA, seg_pos={0 0}, upload=132752]: Done uploading piece. err: <nil> Nov 15 14:50:11.587364703 (150h46m10.138835164s) [noise=true , address=77.90.31.17:4772 , piece_id=ZTLV4QEMHCQJUBZ7CZWQK27Z6IKXYICU76OSMZPETELOYRXTK5LQ, seg_pos={0 0}, upload=132752]: Done uploading piece. err: <nil> Nov 15 14:50:12.990960591 (150h46m11.542431052s) [noise=true , address=77.90.31.15:4699 , piece_id=AH4N55LZLPUJGQOPDLXM6I65SUUKOHUCPXPYE2I7XKI5KHYNJPWA, seg_pos={0 0}, upload=132752]: Done uploading piece. err: <nil> Nov 15 14:50:38.328113117 (150h46m36.879583578s) [noise=true , address=145.79.227.10:4401 , piece_id=SZWIK2OAO3YVLHLAU5FU6KHXODXVXDANAJP5WEKRSLUXDT4H6SYA, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: write tcp 188.166.225.170:34644->145.79.227.10:4401: use of closed network connection; write tcp 188.166.225.170:34644->145.79.227.10:4401: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34644->145.79.227.10:4401: use of closed network connection Nov 15 14:50:38.328591017 (150h46m36.880061478s) [noise=true , address=145.79.227.24:4581 , piece_id=E63A2HTXTWEQCYTOZA47S24YKVDKQ3BHAVZUSKUDALZWVIY6NMOQ, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:123ib1A8nmVmR272odjhVsUgADYbaMRMiPahSLoZ5eamaV6Qbb1, address:145.79.227.24:4581): protocol: read tcp 188.166.225.170:50846->145.79.227.24:4581: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 14:50:38.328758827 (150h46m36.880229318s) [noise=true , address=145.79.227.8:4330 , piece_id=MG7OE5OYG64CQUEGAUZNL654VHQQV347RSY2OKZAY5MXDXOSCMRQ, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW, address:145.79.227.8:4330): protocol: read tcp 188.166.225.170:43188->145.79.227.8:4330: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 14:50:38.328953126 (150h46m36.880423587s) [noise=true , address=145.79.227.32:4834 , piece_id=CJN674FREG73VEEP362QDU7HAAIDOCCYFGSL6ZE7QPDC3NHPWK2A, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: read tcp 188.166.225.170:38312->145.79.227.32:4834: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 14:50:38.329418293 (150h46m36.880888754s) [noise=true , address=145.79.227.9:4367 , piece_id=EFWN62D4D4U2FX7D53NGUNY3UVJBDZTMD7DC47N5IEW6SVXLIF7A, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6, address:145.79.227.9:4367): protocol: write tcp 188.166.225.170:36134->145.79.227.9:4367: use of closed network connection; write tcp 188.166.225.170:36134->145.79.227.9:4367: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36134->145.79.227.9:4367: use of closed network connection Nov 15 14:50:38.329617381 (150h46m36.881087842s) [noise=true , address=145.79.227.13:4803 , piece_id=H2K3B22G6YTEGGS2Y76DCJJHIQIKGXZMDDPQ4MMCUPJGQIRWO5HA, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:145.79.227.13:4803): protocol: read tcp 188.166.225.170:50394->145.79.227.13:4803: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 14:50:38.329698186 (150h46m36.881168657s) [noise=true , address=145.79.227.22:4509 , piece_id=AUB7HF5665Y26RSULL2WH6PUOQJUXSQT66OFMCM5YZWWRPPI6SRA, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:1229E58YirF37sZU6hfZGrDNFSxQDYsWSo4x3mG9BoYsNsLNtjy, address:145.79.227.22:4509): protocol: rpc: dial tcp 145.79.227.22:4509: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 14:50:38.329926219 (150h46m36.88139668s) [noise=true , address=145.79.227.8:4329 , piece_id=7IRP7Y6YSPNBN7DJJOMPPB7T2PDQ43G55MFAYHAWIT3Y4ULXSXZA, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: read tcp 188.166.225.170:44482->145.79.227.8:4329: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 14:50:38.330195863 (150h46m36.881666334s) [noise=true , address=145.79.227.16:4910 , piece_id=7BH46URBTAPGFQ2JKIXUUG4HMX6ERQS34BHPGBDQBMSNCVPVTXZQ, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: read tcp 188.166.225.170:58432->145.79.227.16:4910: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 14:50:38.330267870 (150h46m36.881738331s) [noise=true , address=145.79.227.23:4545 , piece_id=HJFCVMDY2CHYOIS4NYNFEXWHVQ3JGIQ4WAMKT4BC4MTNAMJWVGEA, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:12P5cnEza1Xxwg4qo1nau11zYHsHxqEAwhFsnt2UE1RhtTrJmra, address:145.79.227.23:4545): protocol: read tcp 188.166.225.170:42348->145.79.227.23:4545: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 14:50:38.330623417 (150h46m36.882093898s) [noise=true , address=145.79.227.27:4689 , piece_id=5GEUPACS2UDKXX6RZAL7AOUVXIF3HQRAYZUCAY54GFANZ3ODTYBA, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, address:145.79.227.27:4689): protocol: read tcp 188.166.225.170:52596->145.79.227.27:4689: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 14:50:38.330679213 (150h46m36.882149675s) [seg_pos={0 0}, upload=132752]: Done waiting for segment. successful: 5 optimal: 13 errs: <nil> Nov 15 14:50:38.589562836 (150h46m37.141033297s) [upload=132752]: Done uploading segments... Nov 15 14:50:38.589597782 (150h46m37.141068243s) [upload=132752]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 5) Duration: 6.086079963s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 12 out of 60 is failed (threshold=6) [....XXXX...X.XXX.XXXX.......................................] duration: 3.589102542s trace: https://jaeger.dev.storj.io/trace/42d763a000d5a90d FAIL (2025-11-15 15:14:26 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.589133661s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 6.24704091s trace: https://jaeger.dev.storj.io/trace/495e264ce0048970 FAIL (2025-11-15 15:14:10 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 5) ======LOGS====== T4H6SYA, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.348693684 (150h46m7.900164155s) [noise=true , address=145.79.227.13:4803 , piece_id=H2K3B22G6YTEGGS2Y76DCJJHIQIKGXZMDDPQ4MMCUPJGQIRWO5HA, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.349289590 (150h46m7.900760051s) [noise=true , address=77.90.31.15:4699 , piece_id=AH4N55LZLPUJGQOPDLXM6I65SUUKOHUCPXPYE2I7XKI5KHYNJPWA, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.349845298 (150h46m7.901315759s) [noise=true , address=145.79.227.27:4689 , piece_id=5GEUPACS2UDKXX6RZAL7AOUVXIF3HQRAYZUCAY54GFANZ3ODTYBA, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.350334239 (150h46m7.9018047s) [noise=true , address=45.91.116.7:4011 , piece_id=CSTREAXVJWKIACVOGMDI7QXPAU7JK624TZMQIOB7A2UDDJ3YBOTA, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.350770229 (150h46m7.90224069s) [noise=true , address=145.79.227.22:4509 , piece_id=AUB7HF5665Y26RSULL2WH6PUOQJUXSQT66OFMCM5YZWWRPPI6SRA, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.351317160 (150h46m7.902787632s) [noise=true , address=45.91.116.12:4226 , piece_id=T64PPRSXNN5ZPKAY4YXS5WWAXCNUU4G7Y7TXJXLE4VBNULK3OGMA, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.353921029 (150h46m7.9053915s) [noise=true , address=45.91.116.5:4299 , piece_id=5FETUEJKO5PEHNGKOTPFAAXSPRY7WU5WPSEULBVJ2DPDINPN3V3A, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.354632563 (150h46m7.906103024s) [noise=true , address=145.79.227.8:4329 , piece_id=7IRP7Y6YSPNBN7DJJOMPPB7T2PDQ43G55MFAYHAWIT3Y4ULXSXZA, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.355164717 (150h46m7.906635178s) [noise=true , address=145.79.227.24:4581 , piece_id=E63A2HTXTWEQCYTOZA47S24YKVDKQ3BHAVZUSKUDALZWVIY6NMOQ, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.355607720 (150h46m7.907078181s) [noise=true , address=145.79.227.16:4910 , piece_id=7BH46URBTAPGFQ2JKIXUUG4HMX6ERQS34BHPGBDQBMSNCVPVTXZQ, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.356053489 (150h46m7.90752395s) [noise=true , address=145.79.227.9:4367 , piece_id=EFWN62D4D4U2FX7D53NGUNY3UVJBDZTMD7DC47N5IEW6SVXLIF7A, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.357082539 (150h46m7.90855301s) [noise=true , address=145.79.227.23:4545 , piece_id=HJFCVMDY2CHYOIS4NYNFEXWHVQ3JGIQ4WAMKT4BC4MTNAMJWVGEA, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.360976825 (150h46m7.912447286s) [noise=true , address=77.90.31.17:4772 , piece_id=ZTLV4QEMHCQJUBZ7CZWQK27Z6IKXYICU76OSMZPETELOYRXTK5LQ, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:11.203310499 (150h46m9.754780951s) [upload=132752]: Next returned nil segment Nov 15 14:50:11.203370323 (150h46m9.754840784s) [upload=132752]: Waiting for error group managing segments... Nov 15 14:50:11.496843592 (150h46m10.048314053s) [noise=true , address=45.91.116.5:4299 , piece_id=5FETUEJKO5PEHNGKOTPFAAXSPRY7WU5WPSEULBVJ2DPDINPN3V3A, seg_pos={0 0}, upload=132752]: Done uploading piece. err: <nil> Nov 15 14:50:11.503251574 (150h46m10.054722025s) [noise=true , address=45.91.116.7:4011 , piece_id=CSTREAXVJWKIACVOGMDI7QXPAU7JK624TZMQIOB7A2UDDJ3YBOTA, seg_pos={0 0}, upload=132752]: Done uploading piece. err: <nil> Nov 15 14:50:11.519837998 (150h46m10.071308449s) [noise=true , address=45.91.116.12:4226 , piece_id=T64PPRSXNN5ZPKAY4YXS5WWAXCNUU4G7Y7TXJXLE4VBNULK3OGMA, seg_pos={0 0}, upload=132752]: Done uploading piece. err: <nil> Nov 15 14:50:11.587364703 (150h46m10.138835164s) [noise=true , address=77.90.31.17:4772 , piece_id=ZTLV4QEMHCQJUBZ7CZWQK27Z6IKXYICU76OSMZPETELOYRXTK5LQ, seg_pos={0 0}, upload=132752]: Done uploading piece. err: <nil> Nov 15 14:50:12.990960591 (150h46m11.542431052s) [noise=true , address=77.90.31.15:4699 , piece_id=AH4N55LZLPUJGQOPDLXM6I65SUUKOHUCPXPYE2I7XKI5KHYNJPWA, seg_pos={0 0}, upload=132752]: Done uploading piece. err: <nil> Nov 15 14:50:38.328113117 (150h46m36.879583578s) [noise=true , address=145.79.227.10:4401 , piece_id=SZWIK2OAO3YVLHLAU5FU6KHXODXVXDANAJP5WEKRSLUXDT4H6SYA, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: write tcp 188.166.225.170:34644->145.79.227.10:4401: use of closed network connection; write tcp 188.166.225.170:34644->145.79.227.10:4401: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34644->145.79.227.10:4401: use of closed network connection Nov 15 14:50:38.328591017 (150h46m36.880061478s) [noise=true , address=145.79.227.24:4581 , piece_id=E63A2HTXTWEQCYTOZA47S24YKVDKQ3BHAVZUSKUDALZWVIY6NMOQ, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:123ib1A8nmVmR272odjhVsUgADYbaMRMiPahSLoZ5eamaV6Qbb1, address:145.79.227.24:4581): protocol: read tcp 188.166.225.170:50846->145.79.227.24:4581: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 14:50:38.328758827 (150h46m36.880229318s) [noise=true , address=145.79.227.8:4330 , piece_id=MG7OE5OYG64CQUEGAUZNL654VHQQV347RSY2OKZAY5MXDXOSCMRQ, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW, address:145.79.227.8:4330): protocol: read tcp 188.166.225.170:43188->145.79.227.8:4330: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 14:50:38.328953126 (150h46m36.880423587s) [noise=true , address=145.79.227.32:4834 , piece_id=CJN674FREG73VEEP362QDU7HAAIDOCCYFGSL6ZE7QPDC3NHPWK2A, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: read tcp 188.166.225.170:38312->145.79.227.32:4834: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 14:50:38.329418293 (150h46m36.880888754s) [noise=true , address=145.79.227.9:4367 , piece_id=EFWN62D4D4U2FX7D53NGUNY3UVJBDZTMD7DC47N5IEW6SVXLIF7A, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6, address:145.79.227.9:4367): protocol: write tcp 188.166.225.170:36134->145.79.227.9:4367: use of closed network connection; write tcp 188.166.225.170:36134->145.79.227.9:4367: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36134->145.79.227.9:4367: use of closed network connection Nov 15 14:50:38.329617381 (150h46m36.881087842s) [noise=true , address=145.79.227.13:4803 , piece_id=H2K3B22G6YTEGGS2Y76DCJJHIQIKGXZMDDPQ4MMCUPJGQIRWO5HA, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:145.79.227.13:4803): protocol: read tcp 188.166.225.170:50394->145.79.227.13:4803: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 14:50:38.329698186 (150h46m36.881168657s) [noise=true , address=145.79.227.22:4509 , piece_id=AUB7HF5665Y26RSULL2WH6PUOQJUXSQT66OFMCM5YZWWRPPI6SRA, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:1229E58YirF37sZU6hfZGrDNFSxQDYsWSo4x3mG9BoYsNsLNtjy, address:145.79.227.22:4509): protocol: rpc: dial tcp 145.79.227.22:4509: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 14:50:38.329926219 (150h46m36.88139668s) [noise=true , address=145.79.227.8:4329 , piece_id=7IRP7Y6YSPNBN7DJJOMPPB7T2PDQ43G55MFAYHAWIT3Y4ULXSXZA, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: read tcp 188.166.225.170:44482->145.79.227.8:4329: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 14:50:38.330195863 (150h46m36.881666334s) [noise=true , address=145.79.227.16:4910 , piece_id=7BH46URBTAPGFQ2JKIXUUG4HMX6ERQS34BHPGBDQBMSNCVPVTXZQ, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: read tcp 188.166.225.170:58432->145.79.227.16:4910: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 14:50:38.330267870 (150h46m36.881738331s) [noise=true , address=145.79.227.23:4545 , piece_id=HJFCVMDY2CHYOIS4NYNFEXWHVQ3JGIQ4WAMKT4BC4MTNAMJWVGEA, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:12P5cnEza1Xxwg4qo1nau11zYHsHxqEAwhFsnt2UE1RhtTrJmra, address:145.79.227.23:4545): protocol: read tcp 188.166.225.170:42348->145.79.227.23:4545: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 14:50:38.330623417 (150h46m36.882093898s) [noise=true , address=145.79.227.27:4689 , piece_id=5GEUPACS2UDKXX6RZAL7AOUVXIF3HQRAYZUCAY54GFANZ3ODTYBA, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, address:145.79.227.27:4689): protocol: read tcp 188.166.225.170:52596->145.79.227.27:4689: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 14:50:38.330679213 (150h46m36.882149675s) [seg_pos={0 0}, upload=132752]: Done waiting for segment. successful: 5 optimal: 13 errs: <nil> Nov 15 14:50:38.589562836 (150h46m37.141033297s) [upload=132752]: Done uploading segments... Nov 15 14:50:38.589597782 (150h46m37.141068243s) [upload=132752]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 5) Duration: 6.247072802s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 16.702914573s trace: https://jaeger.dev.storj.io/trace/33f29fa14cfef10f FAIL (2025-11-15 15:13:43 UTC) orbiter warning: context deadline exceeded Duration: 16.702939199s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 13 out of 60 is failed (threshold=6) [X....XXXX...X.XXX.XXXX......................................] duration: 3.636475282s trace: https://jaeger.dev.storj.io/trace/42d763a000d5a90d FAIL (2025-11-15 15:13:30 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.636509476s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 6.522282374s trace: https://jaeger.dev.storj.io/trace/495e264ce0048970 FAIL (2025-11-15 15:13:10 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 5) ======LOGS====== T4H6SYA, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.348693684 (150h46m7.900164155s) [noise=true , address=145.79.227.13:4803 , piece_id=H2K3B22G6YTEGGS2Y76DCJJHIQIKGXZMDDPQ4MMCUPJGQIRWO5HA, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.349289590 (150h46m7.900760051s) [noise=true , address=77.90.31.15:4699 , piece_id=AH4N55LZLPUJGQOPDLXM6I65SUUKOHUCPXPYE2I7XKI5KHYNJPWA, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.349845298 (150h46m7.901315759s) [noise=true , address=145.79.227.27:4689 , piece_id=5GEUPACS2UDKXX6RZAL7AOUVXIF3HQRAYZUCAY54GFANZ3ODTYBA, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.350334239 (150h46m7.9018047s) [noise=true , address=45.91.116.7:4011 , piece_id=CSTREAXVJWKIACVOGMDI7QXPAU7JK624TZMQIOB7A2UDDJ3YBOTA, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.350770229 (150h46m7.90224069s) [noise=true , address=145.79.227.22:4509 , piece_id=AUB7HF5665Y26RSULL2WH6PUOQJUXSQT66OFMCM5YZWWRPPI6SRA, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.351317160 (150h46m7.902787632s) [noise=true , address=45.91.116.12:4226 , piece_id=T64PPRSXNN5ZPKAY4YXS5WWAXCNUU4G7Y7TXJXLE4VBNULK3OGMA, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.353921029 (150h46m7.9053915s) [noise=true , address=45.91.116.5:4299 , piece_id=5FETUEJKO5PEHNGKOTPFAAXSPRY7WU5WPSEULBVJ2DPDINPN3V3A, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.354632563 (150h46m7.906103024s) [noise=true , address=145.79.227.8:4329 , piece_id=7IRP7Y6YSPNBN7DJJOMPPB7T2PDQ43G55MFAYHAWIT3Y4ULXSXZA, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.355164717 (150h46m7.906635178s) [noise=true , address=145.79.227.24:4581 , piece_id=E63A2HTXTWEQCYTOZA47S24YKVDKQ3BHAVZUSKUDALZWVIY6NMOQ, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.355607720 (150h46m7.907078181s) [noise=true , address=145.79.227.16:4910 , piece_id=7BH46URBTAPGFQ2JKIXUUG4HMX6ERQS34BHPGBDQBMSNCVPVTXZQ, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.356053489 (150h46m7.90752395s) [noise=true , address=145.79.227.9:4367 , piece_id=EFWN62D4D4U2FX7D53NGUNY3UVJBDZTMD7DC47N5IEW6SVXLIF7A, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.357082539 (150h46m7.90855301s) [noise=true , address=145.79.227.23:4545 , piece_id=HJFCVMDY2CHYOIS4NYNFEXWHVQ3JGIQ4WAMKT4BC4MTNAMJWVGEA, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.360976825 (150h46m7.912447286s) [noise=true , address=77.90.31.17:4772 , piece_id=ZTLV4QEMHCQJUBZ7CZWQK27Z6IKXYICU76OSMZPETELOYRXTK5LQ, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:11.203310499 (150h46m9.754780951s) [upload=132752]: Next returned nil segment Nov 15 14:50:11.203370323 (150h46m9.754840784s) [upload=132752]: Waiting for error group managing segments... Nov 15 14:50:11.496843592 (150h46m10.048314053s) [noise=true , address=45.91.116.5:4299 , piece_id=5FETUEJKO5PEHNGKOTPFAAXSPRY7WU5WPSEULBVJ2DPDINPN3V3A, seg_pos={0 0}, upload=132752]: Done uploading piece. err: <nil> Nov 15 14:50:11.503251574 (150h46m10.054722025s) [noise=true , address=45.91.116.7:4011 , piece_id=CSTREAXVJWKIACVOGMDI7QXPAU7JK624TZMQIOB7A2UDDJ3YBOTA, seg_pos={0 0}, upload=132752]: Done uploading piece. err: <nil> Nov 15 14:50:11.519837998 (150h46m10.071308449s) [noise=true , address=45.91.116.12:4226 , piece_id=T64PPRSXNN5ZPKAY4YXS5WWAXCNUU4G7Y7TXJXLE4VBNULK3OGMA, seg_pos={0 0}, upload=132752]: Done uploading piece. err: <nil> Nov 15 14:50:11.587364703 (150h46m10.138835164s) [noise=true , address=77.90.31.17:4772 , piece_id=ZTLV4QEMHCQJUBZ7CZWQK27Z6IKXYICU76OSMZPETELOYRXTK5LQ, seg_pos={0 0}, upload=132752]: Done uploading piece. err: <nil> Nov 15 14:50:12.990960591 (150h46m11.542431052s) [noise=true , address=77.90.31.15:4699 , piece_id=AH4N55LZLPUJGQOPDLXM6I65SUUKOHUCPXPYE2I7XKI5KHYNJPWA, seg_pos={0 0}, upload=132752]: Done uploading piece. err: <nil> Nov 15 14:50:38.328113117 (150h46m36.879583578s) [noise=true , address=145.79.227.10:4401 , piece_id=SZWIK2OAO3YVLHLAU5FU6KHXODXVXDANAJP5WEKRSLUXDT4H6SYA, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: write tcp 188.166.225.170:34644->145.79.227.10:4401: use of closed network connection; write tcp 188.166.225.170:34644->145.79.227.10:4401: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34644->145.79.227.10:4401: use of closed network connection Nov 15 14:50:38.328591017 (150h46m36.880061478s) [noise=true , address=145.79.227.24:4581 , piece_id=E63A2HTXTWEQCYTOZA47S24YKVDKQ3BHAVZUSKUDALZWVIY6NMOQ, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:123ib1A8nmVmR272odjhVsUgADYbaMRMiPahSLoZ5eamaV6Qbb1, address:145.79.227.24:4581): protocol: read tcp 188.166.225.170:50846->145.79.227.24:4581: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 14:50:38.328758827 (150h46m36.880229318s) [noise=true , address=145.79.227.8:4330 , piece_id=MG7OE5OYG64CQUEGAUZNL654VHQQV347RSY2OKZAY5MXDXOSCMRQ, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW, address:145.79.227.8:4330): protocol: read tcp 188.166.225.170:43188->145.79.227.8:4330: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 14:50:38.328953126 (150h46m36.880423587s) [noise=true , address=145.79.227.32:4834 , piece_id=CJN674FREG73VEEP362QDU7HAAIDOCCYFGSL6ZE7QPDC3NHPWK2A, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: read tcp 188.166.225.170:38312->145.79.227.32:4834: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 14:50:38.329418293 (150h46m36.880888754s) [noise=true , address=145.79.227.9:4367 , piece_id=EFWN62D4D4U2FX7D53NGUNY3UVJBDZTMD7DC47N5IEW6SVXLIF7A, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6, address:145.79.227.9:4367): protocol: write tcp 188.166.225.170:36134->145.79.227.9:4367: use of closed network connection; write tcp 188.166.225.170:36134->145.79.227.9:4367: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36134->145.79.227.9:4367: use of closed network connection Nov 15 14:50:38.329617381 (150h46m36.881087842s) [noise=true , address=145.79.227.13:4803 , piece_id=H2K3B22G6YTEGGS2Y76DCJJHIQIKGXZMDDPQ4MMCUPJGQIRWO5HA, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:145.79.227.13:4803): protocol: read tcp 188.166.225.170:50394->145.79.227.13:4803: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 14:50:38.329698186 (150h46m36.881168657s) [noise=true , address=145.79.227.22:4509 , piece_id=AUB7HF5665Y26RSULL2WH6PUOQJUXSQT66OFMCM5YZWWRPPI6SRA, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:1229E58YirF37sZU6hfZGrDNFSxQDYsWSo4x3mG9BoYsNsLNtjy, address:145.79.227.22:4509): protocol: rpc: dial tcp 145.79.227.22:4509: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 14:50:38.329926219 (150h46m36.88139668s) [noise=true , address=145.79.227.8:4329 , piece_id=7IRP7Y6YSPNBN7DJJOMPPB7T2PDQ43G55MFAYHAWIT3Y4ULXSXZA, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: read tcp 188.166.225.170:44482->145.79.227.8:4329: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 14:50:38.330195863 (150h46m36.881666334s) [noise=true , address=145.79.227.16:4910 , piece_id=7BH46URBTAPGFQ2JKIXUUG4HMX6ERQS34BHPGBDQBMSNCVPVTXZQ, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: read tcp 188.166.225.170:58432->145.79.227.16:4910: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 14:50:38.330267870 (150h46m36.881738331s) [noise=true , address=145.79.227.23:4545 , piece_id=HJFCVMDY2CHYOIS4NYNFEXWHVQ3JGIQ4WAMKT4BC4MTNAMJWVGEA, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:12P5cnEza1Xxwg4qo1nau11zYHsHxqEAwhFsnt2UE1RhtTrJmra, address:145.79.227.23:4545): protocol: read tcp 188.166.225.170:42348->145.79.227.23:4545: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 14:50:38.330623417 (150h46m36.882093898s) [noise=true , address=145.79.227.27:4689 , piece_id=5GEUPACS2UDKXX6RZAL7AOUVXIF3HQRAYZUCAY54GFANZ3ODTYBA, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, address:145.79.227.27:4689): protocol: read tcp 188.166.225.170:52596->145.79.227.27:4689: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 14:50:38.330679213 (150h46m36.882149675s) [seg_pos={0 0}, upload=132752]: Done waiting for segment. successful: 5 optimal: 13 errs: <nil> Nov 15 14:50:38.589562836 (150h46m37.141033297s) [upload=132752]: Done uploading segments... Nov 15 14:50:38.589597782 (150h46m37.141068243s) [upload=132752]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 5) Duration: 6.522326849s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [XX..........................................................] duration: 13.14323068s trace: https://jaeger.dev.storj.io/trace/33f29fa14cfef10f FAIL (2025-11-15 15:12:33 UTC) orbiter warning: context deadline exceeded Duration: 13.143261048s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 13 out of 60 is failed (threshold=6) [.X....XXXX...X.XXX.XXXX.....................................] duration: 3.438827423s trace: https://jaeger.dev.storj.io/trace/42d763a000d5a90d FAIL (2025-11-15 15:12:31 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.438854505s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 6.214521507s trace: https://jaeger.dev.storj.io/trace/495e264ce0048970 FAIL (2025-11-15 15:12:01 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 5) ======LOGS====== T4H6SYA, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.348693684 (150h46m7.900164155s) [noise=true , address=145.79.227.13:4803 , piece_id=H2K3B22G6YTEGGS2Y76DCJJHIQIKGXZMDDPQ4MMCUPJGQIRWO5HA, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.349289590 (150h46m7.900760051s) [noise=true , address=77.90.31.15:4699 , piece_id=AH4N55LZLPUJGQOPDLXM6I65SUUKOHUCPXPYE2I7XKI5KHYNJPWA, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.349845298 (150h46m7.901315759s) [noise=true , address=145.79.227.27:4689 , piece_id=5GEUPACS2UDKXX6RZAL7AOUVXIF3HQRAYZUCAY54GFANZ3ODTYBA, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.350334239 (150h46m7.9018047s) [noise=true , address=45.91.116.7:4011 , piece_id=CSTREAXVJWKIACVOGMDI7QXPAU7JK624TZMQIOB7A2UDDJ3YBOTA, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.350770229 (150h46m7.90224069s) [noise=true , address=145.79.227.22:4509 , piece_id=AUB7HF5665Y26RSULL2WH6PUOQJUXSQT66OFMCM5YZWWRPPI6SRA, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.351317160 (150h46m7.902787632s) [noise=true , address=45.91.116.12:4226 , piece_id=T64PPRSXNN5ZPKAY4YXS5WWAXCNUU4G7Y7TXJXLE4VBNULK3OGMA, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.353921029 (150h46m7.9053915s) [noise=true , address=45.91.116.5:4299 , piece_id=5FETUEJKO5PEHNGKOTPFAAXSPRY7WU5WPSEULBVJ2DPDINPN3V3A, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.354632563 (150h46m7.906103024s) [noise=true , address=145.79.227.8:4329 , piece_id=7IRP7Y6YSPNBN7DJJOMPPB7T2PDQ43G55MFAYHAWIT3Y4ULXSXZA, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.355164717 (150h46m7.906635178s) [noise=true , address=145.79.227.24:4581 , piece_id=E63A2HTXTWEQCYTOZA47S24YKVDKQ3BHAVZUSKUDALZWVIY6NMOQ, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.355607720 (150h46m7.907078181s) [noise=true , address=145.79.227.16:4910 , piece_id=7BH46URBTAPGFQ2JKIXUUG4HMX6ERQS34BHPGBDQBMSNCVPVTXZQ, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.356053489 (150h46m7.90752395s) [noise=true , address=145.79.227.9:4367 , piece_id=EFWN62D4D4U2FX7D53NGUNY3UVJBDZTMD7DC47N5IEW6SVXLIF7A, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.357082539 (150h46m7.90855301s) [noise=true , address=145.79.227.23:4545 , piece_id=HJFCVMDY2CHYOIS4NYNFEXWHVQ3JGIQ4WAMKT4BC4MTNAMJWVGEA, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:09.360976825 (150h46m7.912447286s) [noise=true , address=77.90.31.17:4772 , piece_id=ZTLV4QEMHCQJUBZ7CZWQK27Z6IKXYICU76OSMZPETELOYRXTK5LQ, seg_pos={0 0}, upload=132752]: Uploading piece... Nov 15 14:50:11.203310499 (150h46m9.754780951s) [upload=132752]: Next returned nil segment Nov 15 14:50:11.203370323 (150h46m9.754840784s) [upload=132752]: Waiting for error group managing segments... Nov 15 14:50:11.496843592 (150h46m10.048314053s) [noise=true , address=45.91.116.5:4299 , piece_id=5FETUEJKO5PEHNGKOTPFAAXSPRY7WU5WPSEULBVJ2DPDINPN3V3A, seg_pos={0 0}, upload=132752]: Done uploading piece. err: <nil> Nov 15 14:50:11.503251574 (150h46m10.054722025s) [noise=true , address=45.91.116.7:4011 , piece_id=CSTREAXVJWKIACVOGMDI7QXPAU7JK624TZMQIOB7A2UDDJ3YBOTA, seg_pos={0 0}, upload=132752]: Done uploading piece. err: <nil> Nov 15 14:50:11.519837998 (150h46m10.071308449s) [noise=true , address=45.91.116.12:4226 , piece_id=T64PPRSXNN5ZPKAY4YXS5WWAXCNUU4G7Y7TXJXLE4VBNULK3OGMA, seg_pos={0 0}, upload=132752]: Done uploading piece. err: <nil> Nov 15 14:50:11.587364703 (150h46m10.138835164s) [noise=true , address=77.90.31.17:4772 , piece_id=ZTLV4QEMHCQJUBZ7CZWQK27Z6IKXYICU76OSMZPETELOYRXTK5LQ, seg_pos={0 0}, upload=132752]: Done uploading piece. err: <nil> Nov 15 14:50:12.990960591 (150h46m11.542431052s) [noise=true , address=77.90.31.15:4699 , piece_id=AH4N55LZLPUJGQOPDLXM6I65SUUKOHUCPXPYE2I7XKI5KHYNJPWA, seg_pos={0 0}, upload=132752]: Done uploading piece. err: <nil> Nov 15 14:50:38.328113117 (150h46m36.879583578s) [noise=true , address=145.79.227.10:4401 , piece_id=SZWIK2OAO3YVLHLAU5FU6KHXODXVXDANAJP5WEKRSLUXDT4H6SYA, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: write tcp 188.166.225.170:34644->145.79.227.10:4401: use of closed network connection; write tcp 188.166.225.170:34644->145.79.227.10:4401: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34644->145.79.227.10:4401: use of closed network connection Nov 15 14:50:38.328591017 (150h46m36.880061478s) [noise=true , address=145.79.227.24:4581 , piece_id=E63A2HTXTWEQCYTOZA47S24YKVDKQ3BHAVZUSKUDALZWVIY6NMOQ, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:123ib1A8nmVmR272odjhVsUgADYbaMRMiPahSLoZ5eamaV6Qbb1, address:145.79.227.24:4581): protocol: read tcp 188.166.225.170:50846->145.79.227.24:4581: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 14:50:38.328758827 (150h46m36.880229318s) [noise=true , address=145.79.227.8:4330 , piece_id=MG7OE5OYG64CQUEGAUZNL654VHQQV347RSY2OKZAY5MXDXOSCMRQ, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW, address:145.79.227.8:4330): protocol: read tcp 188.166.225.170:43188->145.79.227.8:4330: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 14:50:38.328953126 (150h46m36.880423587s) [noise=true , address=145.79.227.32:4834 , piece_id=CJN674FREG73VEEP362QDU7HAAIDOCCYFGSL6ZE7QPDC3NHPWK2A, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: read tcp 188.166.225.170:38312->145.79.227.32:4834: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 14:50:38.329418293 (150h46m36.880888754s) [noise=true , address=145.79.227.9:4367 , piece_id=EFWN62D4D4U2FX7D53NGUNY3UVJBDZTMD7DC47N5IEW6SVXLIF7A, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6, address:145.79.227.9:4367): protocol: write tcp 188.166.225.170:36134->145.79.227.9:4367: use of closed network connection; write tcp 188.166.225.170:36134->145.79.227.9:4367: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36134->145.79.227.9:4367: use of closed network connection Nov 15 14:50:38.329617381 (150h46m36.881087842s) [noise=true , address=145.79.227.13:4803 , piece_id=H2K3B22G6YTEGGS2Y76DCJJHIQIKGXZMDDPQ4MMCUPJGQIRWO5HA, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:145.79.227.13:4803): protocol: read tcp 188.166.225.170:50394->145.79.227.13:4803: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 14:50:38.329698186 (150h46m36.881168657s) [noise=true , address=145.79.227.22:4509 , piece_id=AUB7HF5665Y26RSULL2WH6PUOQJUXSQT66OFMCM5YZWWRPPI6SRA, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:1229E58YirF37sZU6hfZGrDNFSxQDYsWSo4x3mG9BoYsNsLNtjy, address:145.79.227.22:4509): protocol: rpc: dial tcp 145.79.227.22:4509: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 14:50:38.329926219 (150h46m36.88139668s) [noise=true , address=145.79.227.8:4329 , piece_id=7IRP7Y6YSPNBN7DJJOMPPB7T2PDQ43G55MFAYHAWIT3Y4ULXSXZA, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: read tcp 188.166.225.170:44482->145.79.227.8:4329: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 14:50:38.330195863 (150h46m36.881666334s) [noise=true , address=145.79.227.16:4910 , piece_id=7BH46URBTAPGFQ2JKIXUUG4HMX6ERQS34BHPGBDQBMSNCVPVTXZQ, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: read tcp 188.166.225.170:58432->145.79.227.16:4910: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 14:50:38.330267870 (150h46m36.881738331s) [noise=true , address=145.79.227.23:4545 , piece_id=HJFCVMDY2CHYOIS4NYNFEXWHVQ3JGIQ4WAMKT4BC4MTNAMJWVGEA, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:12P5cnEza1Xxwg4qo1nau11zYHsHxqEAwhFsnt2UE1RhtTrJmra, address:145.79.227.23:4545): protocol: read tcp 188.166.225.170:42348->145.79.227.23:4545: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 14:50:38.330623417 (150h46m36.882093898s) [noise=true , address=145.79.227.27:4689 , piece_id=5GEUPACS2UDKXX6RZAL7AOUVXIF3HQRAYZUCAY54GFANZ3ODTYBA, seg_pos={0 0}, upload=132752]: Done uploading piece. err: ecclient: upload failed (node:1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, address:145.79.227.27:4689): protocol: read tcp 188.166.225.170:52596->145.79.227.27:4689: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 15 14:50:38.330679213 (150h46m36.882149675s) [seg_pos={0 0}, upload=132752]: Done waiting for segment. successful: 5 optimal: 13 errs: <nil> Nov 15 14:50:38.589562836 (150h46m37.141033297s) [upload=132752]: Done uploading segments... Nov 15 14:50:38.589597782 (150h46m37.141068243s) [upload=132752]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 5) Duration: 6.214555953s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.XX.........................................................] duration: 15.375999332s trace: https://jaeger.dev.storj.io/trace/33f29fa14cfef10f FAIL (2025-11-15 15:11:56 UTC) orbiter warning: context deadline exceeded Duration: 15.376028738s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 14 out of 60 is failed (threshold=6) [X.X....XXXX...X.XXX.XXXX....................................] duration: 3.531817467s trace: https://jaeger.dev.storj.io/trace/42d763a000d5a90d FAIL (2025-11-15 15:11:52 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.531849207s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 15 out of 60 is failed (threshold=6) [XX.X....XXXX...X.XXX.XXXX...................................] duration: 3.607878906s trace: https://jaeger.dev.storj.io/trace/42d763a000d5a90d FAIL (2025-11-15 15:11:14 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.608060733s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)