Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 6.005901256s FAIL (2025-09-22 05:49:07 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 5h37m59.543525066s) [seg_pos={0 0}, upload=330171]: Begin upload segment... Sep 22 05:06:23.504018422 (375h37m59.543591552s) [seg_pos={0 0}, upload=330171]: Done begin upload segment. Sep 22 05:06:23.504059370 (375h37m59.543632501s) [noise=true , address=45.91.116.9:4292 , piece_id=3QGXKEK5C77XDD34Z77C3K2LYEMGJT2SADTL6RGW2YCYQFV5CLUA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.504681175 (375h37m59.544254355s) [noise=true , address=145.79.227.39:4039 , piece_id=MZRBH2VHR7CCJK4OCCXINOQTNBP7PUP236H7DB2DFL7ZFDBNI2IQ, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.505278141 (375h37m59.544851301s) [noise=true , address=145.79.227.33:4881 , piece_id=AIKNNVT5PMYFYKRMUFN5DQMY7TWEUN3FUC64IQQYKAMV723TKKZA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.505743808 (375h37m59.545316948s) [noise=true , address=145.79.227.46:4601 , piece_id=EVWU7M2P2W4FQCNDEQ2OO3SR4KHPTXQRM7GEI6Q6DHYMDHK4PP4Q, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.505904454 (375h37m59.545477584s) [noise=true , address=145.79.227.24:4594 , piece_id=V55Y7DFR7QJWBBQRKSTUWYNEZSCAIWAY2XHAOZOF2H7JDFSX6IBA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.506859502 (375h37m59.546432622s) [noise=true , address=145.79.227.10:4420 , piece_id=ANGUQNIL2D4K2YR7KVJMY37WW7ILRVXSZOB7FRRAKJPD3AGGPGOA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.506958631 (375h37m59.546531761s) [noise=true , address=145.79.227.16:4933 , piece_id=WOBRP7MRD6VVMMXEHKNL4XDSPKQBHVINQD3I4N2SQU2MOSC2ZUCQ, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.507277077 (375h37m59.546850217s) [noise=true , address=145.79.227.7:4308 , piece_id=QEUKJMQDWYMBVKSVNNVDK53P6CZ4X47IINRSWCHAFG7APSFOOAFQ, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.508336856 (375h37m59.547909986s) [noise=true , address=77.90.31.19:4847 , piece_id=6GLTB4OVJTM67VRKRWATCIWE7WVK35IQMQIVOEZMZM6I5DFNRIBA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.513150782 (375h37m59.552723912s) [noise=true , address=145.79.227.12:4770 , piece_id=X5KVN5TA67L4TTD7LVG6N4I54BKNESFAHXRPVIWGDFCEPSHRKXPA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.513934875 (375h37m59.553508006s) [noise=true , address=45.91.116.14:4214 , piece_id=DOY7FC4SJKGAFP3LHSJXKWCRSYI6I6IR2R2FZ47QJNDM4SCAUXQQ, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.514566187 (375h37m59.554139318s) [noise=true , address=145.79.227.16:4939 , piece_id=DLS7J35PB7NXNMAPTVHBZ3VN3U725CSVZZKEYY37HZWQPWCCJO7A, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.515189424 (375h37m59.554762544s) [noise=true , address=145.79.227.25:4637 , piece_id=3SMC4AHCMKJEDXJOJVZ3FJEKW23VZDSWW6MDCENAYJ7HKOPWNHBQ, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.515805377 (375h37m59.555378508s) [noise=true , address=145.79.227.6:4292 , piece_id=BYUJPZSYFTYD4C4F5EBAW6BSFXTILW6EASOIRHZQI7ARCSG63F2Q, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.516494880 (375h37m59.55606801s) [noise=true , address=77.90.31.16:4764 , piece_id=PSAP3HPVNEURWWNC7K2K2G4NEUENKWDUR7OJY6UBGRTCSBYNHRWQ, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.526689722 (375h37m59.566262863s) [noise=true , address=145.79.227.31:4818 , piece_id=KAJZKTKOHNN5U4CTYJ7CLO32ONU6VSEJ2QBLPBI5WWZZ37YOI5NA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.531527154 (375h37m59.571100294s) [noise=true , address=145.79.227.49:4338 , piece_id=NBR62GETN7QBXC7QJCUB5DZHKVMC5AKQQBIIGDRXNECNZFULEDDA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.537921861 (375h37m59.577494991s) [noise=true , address=145.79.227.5:4247 , piece_id=QGN2S5CET2J6LGW6YPQYDWEVOGEMHWRSRMJG2IUQ36KA23OG2R3A, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:24.564378058 (375h38m0.603951188s) [noise=true , address=145.79.227.5:4247 , piece_id=QGN2S5CET2J6LGW6YPQYDWEVOGEMHWRSRMJG2IUQ36KA23OG2R3A, seg_pos={0 0}, upload=330171]: Done uploading piece. err: ecclient: upload failed (node:1262xxX6aUPTYgKdvRxAyc77PmPBWJtq5cXRJFoELxHbfn18DLE, address:145.79.227.5:4247): protocol: rpc: dial tcp 145.79.227.5:4247: connect: no route to host; conn closed: op Sep 22 05:06:24.986206552 (375h38m1.025779682s) [upload=330171]: Next returned nil segment Sep 22 05:06:24.986226830 (375h38m1.02579995s) [upload=330171]: Waiting for error group managing segments... Sep 22 05:06:25.383578176 (375h38m1.423151306s) [noise=true , address=145.79.227.24:4594 , piece_id=V55Y7DFR7QJWBBQRKSTUWYNEZSCAIWAY2XHAOZOF2H7JDFSX6IBA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.383806571 (375h38m1.423379702s) [noise=true , address=145.79.227.33:4881 , piece_id=AIKNNVT5PMYFYKRMUFN5DQMY7TWEUN3FUC64IQQYKAMV723TKKZA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.398811493 (375h38m1.438384634s) [noise=true , address=145.79.227.46:4601 , piece_id=EVWU7M2P2W4FQCNDEQ2OO3SR4KHPTXQRM7GEI6Q6DHYMDHK4PP4Q, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.399530783 (375h38m1.439103913s) [noise=true , address=145.79.227.39:4039 , piece_id=MZRBH2VHR7CCJK4OCCXINOQTNBP7PUP236H7DB2DFL7ZFDBNI2IQ, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.406508421 (375h38m1.446081551s) [noise=true , address=145.79.227.49:4338 , piece_id=NBR62GETN7QBXC7QJCUB5DZHKVMC5AKQQBIIGDRXNECNZFULEDDA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.415736332 (375h38m1.455309462s) [noise=true , address=145.79.227.25:4637 , piece_id=3SMC4AHCMKJEDXJOJVZ3FJEKW23VZDSWW6MDCENAYJ7HKOPWNHBQ, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.437405044 (375h38m1.476978184s) [noise=true , address=145.79.227.31:4818 , piece_id=KAJZKTKOHNN5U4CTYJ7CLO32ONU6VSEJ2QBLPBI5WWZZ37YOI5NA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.465513107 (375h38m1.505086237s) [noise=true , address=77.90.31.19:4847 , piece_id=6GLTB4OVJTM67VRKRWATCIWE7WVK35IQMQIVOEZMZM6I5DFNRIBA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.501349827 (375h38m1.540922957s) [noise=true , address=45.91.116.9:4292 , piece_id=3QGXKEK5C77XDD34Z77C3K2LYEMGJT2SADTL6RGW2YCYQFV5CLUA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.579007182 (375h38m1.618580302s) [noise=true , address=77.90.31.16:4764 , piece_id=PSAP3HPVNEURWWNC7K2K2G4NEUENKWDUR7OJY6UBGRTCSBYNHRWQ, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.586807715 (375h38m1.626380845s) [noise=true , address=45.91.116.14:4214 , piece_id=DOY7FC4SJKGAFP3LHSJXKWCRSYI6I6IR2R2FZ47QJNDM4SCAUXQQ, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:07:22.245417769 (375h38m58.284990899s) [noise=true , address=145.79.227.16:4933 , piece_id=WOBRP7MRD6VVMMXEHKNL4XDSPKQBHVINQD3I4N2SQU2MOSC2ZUCQ, seg_pos={0 0}, upload=330171]: Done uploading piece. err: ecclient: upload failed (node:19m4inaXvTdwf2rKip3eFELv6JAgFvXJ8KqjPZYbb48i8Ktnwe, address:145.79.227.16:4933): protocol: rpc: dial tcp 145.79.227.16:4933: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:07:22.245544310 (375h38m58.28511744s) [noise=true , address=145.79.227.12:4770 , piece_id=X5KVN5TA67L4TTD7LVG6N4I54BKNESFAHXRPVIWGDFCEPSHRKXPA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: ecclient: upload failed (node:1Ug6GpYToYDXiHmiJP4GoE7eJV3kqRveix2S5SLDcKQJtEbNPz, address:145.79.227.12:4770): protocol: rpc: dial tcp 145.79.227.12:4770: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:07:22.245567825 (375h38m58.285140955s) [noise=true , address=145.79.227.16:4939 , piece_id=DLS7J35PB7NXNMAPTVHBZ3VN3U725CSVZZKEYY37HZWQPWCCJO7A, seg_pos={0 0}, upload=330171]: Done uploading piece. err: ecclient: upload failed (node:1mRmp3HhD95PruS4AF8soRsSXNYUeb4mJE7y52FvoeN9D3Gu4F, address:145.79.227.16:4939): protocol: rpc: dial tcp 145.79.227.16:4939: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:07:22.245634181 (375h38m58.285207312s) [noise=true , address=145.79.227.7:4308 , piece_id=QEUKJMQDWYMBVKSVNNVDK53P6CZ4X47IINRSWCHAFG7APSFOOAFQ, seg_pos={0 0}, upload=330171]: Done uploading piece. err: ecclient: upload failed (node:1PgA5SZkBvK673LsV7p2sXpbi1r79esNNZspPZTzaitqfbjtyY, address:145.79.227.7:4308): protocol: rpc: dial tcp 145.79.227.7:4308: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:07:22.245657335 (375h38m58.285230455s) [noise=true , address=145.79.227.10:4420 , piece_id=ANGUQNIL2D4K2YR7KVJMY37WW7ILRVXSZOB7FRRAKJPD3AGGPGOA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: ecclient: upload failed (node:12hJrjeMUMYmFw3yxRtQQKRh9ciPfdu5FXo8NGrsc6HBqgfJzzp, address:145.79.227.10:4420): protocol: rpc: dial tcp 145.79.227.10:4420: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:07:22.245675269 (375h38m58.28524839s) [noise=true , address=145.79.227.6:4292 , piece_id=BYUJPZSYFTYD4C4F5EBAW6BSFXTILW6EASOIRHZQI7ARCSG63F2Q, seg_pos={0 0}, upload=330171]: Done uploading piece. err: ecclient: upload failed (node:12UpWmMhEmk5veKZvzsQ3MCxCMnfEKtFgCugBTsYfy9Ejm7G4xD, address:145.79.227.6:4292): protocol: rpc: dial tcp 145.79.227.6:4292: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:07:22.245712370 (375h38m58.2852855s) [seg_pos={0 0}, upload=330171]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Sep 22 05:07:22.500217377 (375h38m58.539790507s) [upload=330171]: Done uploading segments... Sep 22 05:07:22.500253296 (375h38m58.539826426s) [upload=330171]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 6.005933299s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [....X..X...X.X......X.........X.X..X....X..XX...............] duration: 6.116879246s FAIL (2025-09-22 05:48:42 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.116896318s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 6.028570218s FAIL (2025-09-22 05:48:41 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 5h37m59.543525066s) [seg_pos={0 0}, upload=330171]: Begin upload segment... Sep 22 05:06:23.504018422 (375h37m59.543591552s) [seg_pos={0 0}, upload=330171]: Done begin upload segment. Sep 22 05:06:23.504059370 (375h37m59.543632501s) [noise=true , address=45.91.116.9:4292 , piece_id=3QGXKEK5C77XDD34Z77C3K2LYEMGJT2SADTL6RGW2YCYQFV5CLUA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.504681175 (375h37m59.544254355s) [noise=true , address=145.79.227.39:4039 , piece_id=MZRBH2VHR7CCJK4OCCXINOQTNBP7PUP236H7DB2DFL7ZFDBNI2IQ, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.505278141 (375h37m59.544851301s) [noise=true , address=145.79.227.33:4881 , piece_id=AIKNNVT5PMYFYKRMUFN5DQMY7TWEUN3FUC64IQQYKAMV723TKKZA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.505743808 (375h37m59.545316948s) [noise=true , address=145.79.227.46:4601 , piece_id=EVWU7M2P2W4FQCNDEQ2OO3SR4KHPTXQRM7GEI6Q6DHYMDHK4PP4Q, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.505904454 (375h37m59.545477584s) [noise=true , address=145.79.227.24:4594 , piece_id=V55Y7DFR7QJWBBQRKSTUWYNEZSCAIWAY2XHAOZOF2H7JDFSX6IBA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.506859502 (375h37m59.546432622s) [noise=true , address=145.79.227.10:4420 , piece_id=ANGUQNIL2D4K2YR7KVJMY37WW7ILRVXSZOB7FRRAKJPD3AGGPGOA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.506958631 (375h37m59.546531761s) [noise=true , address=145.79.227.16:4933 , piece_id=WOBRP7MRD6VVMMXEHKNL4XDSPKQBHVINQD3I4N2SQU2MOSC2ZUCQ, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.507277077 (375h37m59.546850217s) [noise=true , address=145.79.227.7:4308 , piece_id=QEUKJMQDWYMBVKSVNNVDK53P6CZ4X47IINRSWCHAFG7APSFOOAFQ, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.508336856 (375h37m59.547909986s) [noise=true , address=77.90.31.19:4847 , piece_id=6GLTB4OVJTM67VRKRWATCIWE7WVK35IQMQIVOEZMZM6I5DFNRIBA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.513150782 (375h37m59.552723912s) [noise=true , address=145.79.227.12:4770 , piece_id=X5KVN5TA67L4TTD7LVG6N4I54BKNESFAHXRPVIWGDFCEPSHRKXPA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.513934875 (375h37m59.553508006s) [noise=true , address=45.91.116.14:4214 , piece_id=DOY7FC4SJKGAFP3LHSJXKWCRSYI6I6IR2R2FZ47QJNDM4SCAUXQQ, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.514566187 (375h37m59.554139318s) [noise=true , address=145.79.227.16:4939 , piece_id=DLS7J35PB7NXNMAPTVHBZ3VN3U725CSVZZKEYY37HZWQPWCCJO7A, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.515189424 (375h37m59.554762544s) [noise=true , address=145.79.227.25:4637 , piece_id=3SMC4AHCMKJEDXJOJVZ3FJEKW23VZDSWW6MDCENAYJ7HKOPWNHBQ, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.515805377 (375h37m59.555378508s) [noise=true , address=145.79.227.6:4292 , piece_id=BYUJPZSYFTYD4C4F5EBAW6BSFXTILW6EASOIRHZQI7ARCSG63F2Q, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.516494880 (375h37m59.55606801s) [noise=true , address=77.90.31.16:4764 , piece_id=PSAP3HPVNEURWWNC7K2K2G4NEUENKWDUR7OJY6UBGRTCSBYNHRWQ, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.526689722 (375h37m59.566262863s) [noise=true , address=145.79.227.31:4818 , piece_id=KAJZKTKOHNN5U4CTYJ7CLO32ONU6VSEJ2QBLPBI5WWZZ37YOI5NA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.531527154 (375h37m59.571100294s) [noise=true , address=145.79.227.49:4338 , piece_id=NBR62GETN7QBXC7QJCUB5DZHKVMC5AKQQBIIGDRXNECNZFULEDDA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.537921861 (375h37m59.577494991s) [noise=true , address=145.79.227.5:4247 , piece_id=QGN2S5CET2J6LGW6YPQYDWEVOGEMHWRSRMJG2IUQ36KA23OG2R3A, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:24.564378058 (375h38m0.603951188s) [noise=true , address=145.79.227.5:4247 , piece_id=QGN2S5CET2J6LGW6YPQYDWEVOGEMHWRSRMJG2IUQ36KA23OG2R3A, seg_pos={0 0}, upload=330171]: Done uploading piece. err: ecclient: upload failed (node:1262xxX6aUPTYgKdvRxAyc77PmPBWJtq5cXRJFoELxHbfn18DLE, address:145.79.227.5:4247): protocol: rpc: dial tcp 145.79.227.5:4247: connect: no route to host; conn closed: op Sep 22 05:06:24.986206552 (375h38m1.025779682s) [upload=330171]: Next returned nil segment Sep 22 05:06:24.986226830 (375h38m1.02579995s) [upload=330171]: Waiting for error group managing segments... Sep 22 05:06:25.383578176 (375h38m1.423151306s) [noise=true , address=145.79.227.24:4594 , piece_id=V55Y7DFR7QJWBBQRKSTUWYNEZSCAIWAY2XHAOZOF2H7JDFSX6IBA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.383806571 (375h38m1.423379702s) [noise=true , address=145.79.227.33:4881 , piece_id=AIKNNVT5PMYFYKRMUFN5DQMY7TWEUN3FUC64IQQYKAMV723TKKZA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.398811493 (375h38m1.438384634s) [noise=true , address=145.79.227.46:4601 , piece_id=EVWU7M2P2W4FQCNDEQ2OO3SR4KHPTXQRM7GEI6Q6DHYMDHK4PP4Q, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.399530783 (375h38m1.439103913s) [noise=true , address=145.79.227.39:4039 , piece_id=MZRBH2VHR7CCJK4OCCXINOQTNBP7PUP236H7DB2DFL7ZFDBNI2IQ, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.406508421 (375h38m1.446081551s) [noise=true , address=145.79.227.49:4338 , piece_id=NBR62GETN7QBXC7QJCUB5DZHKVMC5AKQQBIIGDRXNECNZFULEDDA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.415736332 (375h38m1.455309462s) [noise=true , address=145.79.227.25:4637 , piece_id=3SMC4AHCMKJEDXJOJVZ3FJEKW23VZDSWW6MDCENAYJ7HKOPWNHBQ, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.437405044 (375h38m1.476978184s) [noise=true , address=145.79.227.31:4818 , piece_id=KAJZKTKOHNN5U4CTYJ7CLO32ONU6VSEJ2QBLPBI5WWZZ37YOI5NA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.465513107 (375h38m1.505086237s) [noise=true , address=77.90.31.19:4847 , piece_id=6GLTB4OVJTM67VRKRWATCIWE7WVK35IQMQIVOEZMZM6I5DFNRIBA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.501349827 (375h38m1.540922957s) [noise=true , address=45.91.116.9:4292 , piece_id=3QGXKEK5C77XDD34Z77C3K2LYEMGJT2SADTL6RGW2YCYQFV5CLUA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.579007182 (375h38m1.618580302s) [noise=true , address=77.90.31.16:4764 , piece_id=PSAP3HPVNEURWWNC7K2K2G4NEUENKWDUR7OJY6UBGRTCSBYNHRWQ, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.586807715 (375h38m1.626380845s) [noise=true , address=45.91.116.14:4214 , piece_id=DOY7FC4SJKGAFP3LHSJXKWCRSYI6I6IR2R2FZ47QJNDM4SCAUXQQ, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:07:22.245417769 (375h38m58.284990899s) [noise=true , address=145.79.227.16:4933 , piece_id=WOBRP7MRD6VVMMXEHKNL4XDSPKQBHVINQD3I4N2SQU2MOSC2ZUCQ, seg_pos={0 0}, upload=330171]: Done uploading piece. err: ecclient: upload failed (node:19m4inaXvTdwf2rKip3eFELv6JAgFvXJ8KqjPZYbb48i8Ktnwe, address:145.79.227.16:4933): protocol: rpc: dial tcp 145.79.227.16:4933: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:07:22.245544310 (375h38m58.28511744s) [noise=true , address=145.79.227.12:4770 , piece_id=X5KVN5TA67L4TTD7LVG6N4I54BKNESFAHXRPVIWGDFCEPSHRKXPA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: ecclient: upload failed (node:1Ug6GpYToYDXiHmiJP4GoE7eJV3kqRveix2S5SLDcKQJtEbNPz, address:145.79.227.12:4770): protocol: rpc: dial tcp 145.79.227.12:4770: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:07:22.245567825 (375h38m58.285140955s) [noise=true , address=145.79.227.16:4939 , piece_id=DLS7J35PB7NXNMAPTVHBZ3VN3U725CSVZZKEYY37HZWQPWCCJO7A, seg_pos={0 0}, upload=330171]: Done uploading piece. err: ecclient: upload failed (node:1mRmp3HhD95PruS4AF8soRsSXNYUeb4mJE7y52FvoeN9D3Gu4F, address:145.79.227.16:4939): protocol: rpc: dial tcp 145.79.227.16:4939: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:07:22.245634181 (375h38m58.285207312s) [noise=true , address=145.79.227.7:4308 , piece_id=QEUKJMQDWYMBVKSVNNVDK53P6CZ4X47IINRSWCHAFG7APSFOOAFQ, seg_pos={0 0}, upload=330171]: Done uploading piece. err: ecclient: upload failed (node:1PgA5SZkBvK673LsV7p2sXpbi1r79esNNZspPZTzaitqfbjtyY, address:145.79.227.7:4308): protocol: rpc: dial tcp 145.79.227.7:4308: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:07:22.245657335 (375h38m58.285230455s) [noise=true , address=145.79.227.10:4420 , piece_id=ANGUQNIL2D4K2YR7KVJMY37WW7ILRVXSZOB7FRRAKJPD3AGGPGOA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: ecclient: upload failed (node:12hJrjeMUMYmFw3yxRtQQKRh9ciPfdu5FXo8NGrsc6HBqgfJzzp, address:145.79.227.10:4420): protocol: rpc: dial tcp 145.79.227.10:4420: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:07:22.245675269 (375h38m58.28524839s) [noise=true , address=145.79.227.6:4292 , piece_id=BYUJPZSYFTYD4C4F5EBAW6BSFXTILW6EASOIRHZQI7ARCSG63F2Q, seg_pos={0 0}, upload=330171]: Done uploading piece. err: ecclient: upload failed (node:12UpWmMhEmk5veKZvzsQ3MCxCMnfEKtFgCugBTsYfy9Ejm7G4xD, address:145.79.227.6:4292): protocol: rpc: dial tcp 145.79.227.6:4292: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:07:22.245712370 (375h38m58.2852855s) [seg_pos={0 0}, upload=330171]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Sep 22 05:07:22.500217377 (375h38m58.539790507s) [upload=330171]: Done uploading segments... Sep 22 05:07:22.500253296 (375h38m58.539826426s) [upload=330171]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 6.028603661s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.X.XX.....X.X....................X..........................] duration: 6.216419335s FAIL (2025-09-22 05:48:36 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== 3.337157461s) [upload=330511]: Starting upload Sep 22 05:29:47.297668972 (376h1m23.337242102s) [upload=330511]: Done starting upload Sep 22 05:29:47.297899231 (376h1m23.337472361s) [upload=330511]: Uploading segments... Sep 22 05:29:47.298002678 (376h1m23.337575818s) [upload=330511]: Got next segment. Inline: false Sep 22 05:29:47.298059145 (376h1m23.337632275s) [upload=330511]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 05:29:47.298066109 (376h1m23.337639229s) [upload=330511]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 05:29:47.552911815 (376h1m23.592484945s) [seg_pos={0 0}, upload=330511]: Begin upload segment... Sep 22 05:29:47.552975406 (376h1m23.592548536s) [seg_pos={0 0}, upload=330511]: Done begin upload segment. Sep 22 05:29:47.553013599 (376h1m23.592586719s) [noise=true , address=45.91.116.11:4117 , piece_id=NIFPLA2N3L6XDWTIFUM3YYRGBMFQJKWFVCGYXD2GBFYVYDXW6OEQ, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.553528389 (376h1m23.593101539s) [noise=true , address=145.79.227.6:4258 , piece_id=D7HMSQH7ZLLKNTUZGJLVHIW2KEQZDL3HSS6VO2BSNRZJAC3USFAQ, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.553925034 (376h1m23.593498154s) [noise=true , address=45.91.116.9:4261 , piece_id=REZZWYP25KZOP4UBJWJPHGCDUWCW3EBOJG2DOLCDYZY5B43KBMSQ, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.553961283 (376h1m23.593534413s) [noise=true , address=77.90.31.14:4663 , piece_id=LYGO7UTA55Q5Q7APPDJZNNWZ4S2XHAT4LS4V27VF27A3435DVJQA, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.554326960 (376h1m23.59390009s) [noise=true , address=77.90.31.16:4735 , piece_id=K5KYKKY3BQV6UVJK4G3DYC24DFFIEFQX7KGGFLS4P35KJRFHA6UQ, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.554351497 (376h1m23.593924627s) [noise=true , address=145.79.227.16:4912 , piece_id=AG2FUP2KFIJ7W55H2VRYTJYWQ57MAXCTEMTDWSR4FUJAYE3HFKBQ, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.554588828 (376h1m23.594161948s) [noise=true , address=145.79.227.12:4767 , piece_id=OUVFSM62B7UZ4U4WGFFBOHCPHXEXR6A6H2FQL3DTG5L3DEPUVYVQ, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.554724957 (376h1m23.594298087s) [noise=true , address=77.90.31.18:4807 , piece_id=JFOMLF75MIXEDP5CKDHH7J3PWGWESB7BIKK3SCAIOVNX74DJDC3A, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.554875986 (376h1m23.594449116s) [noise=true , address=145.79.227.3:4151 , piece_id=O6AU3AOBKJRIUMEAWPHMWWDDNRTA52WL72JNUTK5JFLVG2TNV5IA, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.555093950 (376h1m23.59466708s) [noise=true , address=145.79.227.17:4972 , piece_id=Q3KQQ57XWDUCXNHP33STZ7DN6VQ6FVYZTDPLINEPZASFEVY5XDJA, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.555135429 (376h1m23.594708559s) [noise=true , address=45.91.116.6:4973 , piece_id=WPWUFOFQHQZFRPTR33GTDST2NYCVPXKLDVCJQFPQ4A7PI44LJKOQ, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.555464797 (376h1m23.595037917s) [noise=true , address=45.91.116.14:4189 , piece_id=6VJ6XCCEV2CZSARNRF3ATZ4KCUI75QKWIWQWJL5QH4XTQG4MCLUQ, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.555726194 (376h1m23.595299314s) [noise=true , address=77.90.31.15:4699 , piece_id=6XZN64DOILMELBYDU4D72ER33XDKMRIJRSAXX5ITFYPG46YEBITQ, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.556009483 (376h1m23.595582614s) [noise=true , address=145.79.227.19:4077 , piece_id=EDBYIPZUOPFRRIZBZJJEP2U7NZFCUHRLDZESZ7J5CQZFKADNLE6Q, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.556364609 (376h1m23.59593773s) [noise=true , address=145.79.227.15:4875 , piece_id=ZEWO52263YKTTRYDVF4PV52JTPEDJR3XGVRG3BFE5QQKDQSCE2LQ, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.556611299 (376h1m23.596184429s) [noise=true , address=145.79.227.5:4222 , piece_id=CZNVNWHETTFQS6JR2YLE65LNQU4WX74DAROY5I3KAYHG4QDLDOKA, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:49.308510933 (376h1m25.348084063s) [upload=330511]: Next returned nil segment Sep 22 05:29:49.308533666 (376h1m25.348106796s) [upload=330511]: Waiting for error group managing segments... Sep 22 05:29:49.602590138 (376h1m25.642163268s) [noise=true , address=45.91.116.11:4117 , piece_id=NIFPLA2N3L6XDWTIFUM3YYRGBMFQJKWFVCGYXD2GBFYVYDXW6OEQ, seg_pos={0 0}, upload=330511]: Done uploading piece. err: <nil> Sep 22 05:29:49.633269677 (376h1m25.672842807s) [noise=true , address=77.90.31.15:4699 , piece_id=6XZN64DOILMELBYDU4D72ER33XDKMRIJRSAXX5ITFYPG46YEBITQ, seg_pos={0 0}, upload=330511]: Done uploading piece. err: <nil> Sep 22 05:29:49.637382749 (376h1m25.676955879s) [noise=true , address=45.91.116.6:4973 , piece_id=WPWUFOFQHQZFRPTR33GTDST2NYCVPXKLDVCJQFPQ4A7PI44LJKOQ, seg_pos={0 0}, upload=330511]: Done uploading piece. err: <nil> Sep 22 05:29:49.648543782 (376h1m25.688116902s) [noise=true , address=77.90.31.16:4735 , piece_id=K5KYKKY3BQV6UVJK4G3DYC24DFFIEFQX7KGGFLS4P35KJRFHA6UQ, seg_pos={0 0}, upload=330511]: Done uploading piece. err: <nil> Sep 22 05:29:49.653934647 (376h1m25.693507767s) [noise=true , address=45.91.116.14:4189 , piece_id=6VJ6XCCEV2CZSARNRF3ATZ4KCUI75QKWIWQWJL5QH4XTQG4MCLUQ, seg_pos={0 0}, upload=330511]: Done uploading piece. err: <nil> Sep 22 05:29:49.669096699 (376h1m25.708669829s) [noise=true , address=45.91.116.9:4261 , piece_id=REZZWYP25KZOP4UBJWJPHGCDUWCW3EBOJG2DOLCDYZY5B43KBMSQ, seg_pos={0 0}, upload=330511]: Done uploading piece. err: <nil> Sep 22 05:29:49.673672402 (376h1m25.713245532s) [noise=true , address=77.90.31.14:4663 , piece_id=LYGO7UTA55Q5Q7APPDJZNNWZ4S2XHAT4LS4V27VF27A3435DVJQA, seg_pos={0 0}, upload=330511]: Done uploading piece. err: <nil> Sep 22 05:29:49.741841689 (376h1m25.781414809s) [noise=true , address=77.90.31.18:4807 , piece_id=JFOMLF75MIXEDP5CKDHH7J3PWGWESB7BIKK3SCAIOVNX74DJDC3A, seg_pos={0 0}, upload=330511]: Done uploading piece. err: <nil> Sep 22 05:29:58.021015047 (376h1m34.060588167s) [noise=true , address=145.79.227.5:4222 , piece_id=CZNVNWHETTFQS6JR2YLE65LNQU4WX74DAROY5I3KAYHG4QDLDOKA, seg_pos={0 0}, upload=330511]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:145.79.227.5:4222): protocol: rpc: dial tcp 145.79.227.5:4222: connect: no route to host; conn closed: op; piecestore: piecestore close: manager closed: conn closed: op Sep 22 05:30:05.028605724 (376h1m41.068178854s) [noise=true , address=145.79.227.3:4151 , piece_id=O6AU3AOBKJRIUMEAWPHMWWDDNRTA52WL72JNUTK5JFLVG2TNV5IA, seg_pos={0 0}, upload=330511]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:145.79.227.3:4151): protocol: rpc: dial tcp 145.79.227.3:4151: connect: no route to host; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:30:16.586571260 (376h1m52.62614439s) [noise=true , address=145.79.227.12:4767 , piece_id=OUVFSM62B7UZ4U4WGFFBOHCPHXEXR6A6H2FQL3DTG5L3DEPUVYVQ, seg_pos={0 0}, upload=330511]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:145.79.227.12:4767): protocol: rpc: dial tcp 145.79.227.12:4767: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:30:16.586679978 (376h1m52.626253118s) [noise=true , address=145.79.227.6:4258 , piece_id=D7HMSQH7ZLLKNTUZGJLVHIW2KEQZDL3HSS6VO2BSNRZJAC3USFAQ, seg_pos={0 0}, upload=330511]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:145.79.227.6:4258): protocol: rpc: dial tcp 145.79.227.6:4258: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:30:16.586708171 (376h1m52.626281301s) [noise=true , address=145.79.227.16:4912 , piece_id=AG2FUP2KFIJ7W55H2VRYTJYWQ57MAXCTEMTDWSR4FUJAYE3HFKBQ, seg_pos={0 0}, upload=330511]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:145.79.227.16:4912): protocol: rpc: dial tcp 145.79.227.16:4912: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:30:16.586749971 (376h1m52.626323101s) [noise=true , address=145.79.227.15:4875 , piece_id=ZEWO52263YKTTRYDVF4PV52JTPEDJR3XGVRG3BFE5QQKDQSCE2LQ, seg_pos={0 0}, upload=330511]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: rpc: dial tcp 145.79.227.15:4875: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:30:16.586795768 (376h1m52.626368888s) [noise=true , address=145.79.227.17:4972 , piece_id=Q3KQQ57XWDUCXNHP33STZ7DN6VQ6FVYZTDPLINEPZASFEVY5XDJA, seg_pos={0 0}, upload=330511]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:145.79.227.17:4972): protocol: rpc: dial tcp 145.79.227.17:4972: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:30:16.586806088 (376h1m52.626379218s) [noise=true , address=145.79.227.19:4077 , piece_id=EDBYIPZUOPFRRIZBZJJEP2U7NZFCUHRLDZESZ7J5CQZFKADNLE6Q, seg_pos={0 0}, upload=330511]: Done uploading piece. err: ecclient: upload failed (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk, address:145.79.227.19:4077): protocol: rpc: dial tcp 145.79.227.19:4077: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:30:16.586831426 (376h1m52.626404556s) [seg_pos={0 0}, upload=330511]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Sep 22 05:30:16.834944071 (376h1m52.874517201s) [upload=330511]: Done uploading segments... Sep 22 05:30:16.834981573 (376h1m52.874554703s) [upload=330511]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 6.216457989s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [X...X.......................................................] duration: 5.784526658s FAIL (2025-09-22 05:48:22 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 5h37m59.543525066s) [seg_pos={0 0}, upload=330171]: Begin upload segment... Sep 22 05:06:23.504018422 (375h37m59.543591552s) [seg_pos={0 0}, upload=330171]: Done begin upload segment. Sep 22 05:06:23.504059370 (375h37m59.543632501s) [noise=true , address=45.91.116.9:4292 , piece_id=3QGXKEK5C77XDD34Z77C3K2LYEMGJT2SADTL6RGW2YCYQFV5CLUA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.504681175 (375h37m59.544254355s) [noise=true , address=145.79.227.39:4039 , piece_id=MZRBH2VHR7CCJK4OCCXINOQTNBP7PUP236H7DB2DFL7ZFDBNI2IQ, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.505278141 (375h37m59.544851301s) [noise=true , address=145.79.227.33:4881 , piece_id=AIKNNVT5PMYFYKRMUFN5DQMY7TWEUN3FUC64IQQYKAMV723TKKZA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.505743808 (375h37m59.545316948s) [noise=true , address=145.79.227.46:4601 , piece_id=EVWU7M2P2W4FQCNDEQ2OO3SR4KHPTXQRM7GEI6Q6DHYMDHK4PP4Q, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.505904454 (375h37m59.545477584s) [noise=true , address=145.79.227.24:4594 , piece_id=V55Y7DFR7QJWBBQRKSTUWYNEZSCAIWAY2XHAOZOF2H7JDFSX6IBA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.506859502 (375h37m59.546432622s) [noise=true , address=145.79.227.10:4420 , piece_id=ANGUQNIL2D4K2YR7KVJMY37WW7ILRVXSZOB7FRRAKJPD3AGGPGOA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.506958631 (375h37m59.546531761s) [noise=true , address=145.79.227.16:4933 , piece_id=WOBRP7MRD6VVMMXEHKNL4XDSPKQBHVINQD3I4N2SQU2MOSC2ZUCQ, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.507277077 (375h37m59.546850217s) [noise=true , address=145.79.227.7:4308 , piece_id=QEUKJMQDWYMBVKSVNNVDK53P6CZ4X47IINRSWCHAFG7APSFOOAFQ, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.508336856 (375h37m59.547909986s) [noise=true , address=77.90.31.19:4847 , piece_id=6GLTB4OVJTM67VRKRWATCIWE7WVK35IQMQIVOEZMZM6I5DFNRIBA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.513150782 (375h37m59.552723912s) [noise=true , address=145.79.227.12:4770 , piece_id=X5KVN5TA67L4TTD7LVG6N4I54BKNESFAHXRPVIWGDFCEPSHRKXPA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.513934875 (375h37m59.553508006s) [noise=true , address=45.91.116.14:4214 , piece_id=DOY7FC4SJKGAFP3LHSJXKWCRSYI6I6IR2R2FZ47QJNDM4SCAUXQQ, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.514566187 (375h37m59.554139318s) [noise=true , address=145.79.227.16:4939 , piece_id=DLS7J35PB7NXNMAPTVHBZ3VN3U725CSVZZKEYY37HZWQPWCCJO7A, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.515189424 (375h37m59.554762544s) [noise=true , address=145.79.227.25:4637 , piece_id=3SMC4AHCMKJEDXJOJVZ3FJEKW23VZDSWW6MDCENAYJ7HKOPWNHBQ, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.515805377 (375h37m59.555378508s) [noise=true , address=145.79.227.6:4292 , piece_id=BYUJPZSYFTYD4C4F5EBAW6BSFXTILW6EASOIRHZQI7ARCSG63F2Q, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.516494880 (375h37m59.55606801s) [noise=true , address=77.90.31.16:4764 , piece_id=PSAP3HPVNEURWWNC7K2K2G4NEUENKWDUR7OJY6UBGRTCSBYNHRWQ, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.526689722 (375h37m59.566262863s) [noise=true , address=145.79.227.31:4818 , piece_id=KAJZKTKOHNN5U4CTYJ7CLO32ONU6VSEJ2QBLPBI5WWZZ37YOI5NA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.531527154 (375h37m59.571100294s) [noise=true , address=145.79.227.49:4338 , piece_id=NBR62GETN7QBXC7QJCUB5DZHKVMC5AKQQBIIGDRXNECNZFULEDDA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.537921861 (375h37m59.577494991s) [noise=true , address=145.79.227.5:4247 , piece_id=QGN2S5CET2J6LGW6YPQYDWEVOGEMHWRSRMJG2IUQ36KA23OG2R3A, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:24.564378058 (375h38m0.603951188s) [noise=true , address=145.79.227.5:4247 , piece_id=QGN2S5CET2J6LGW6YPQYDWEVOGEMHWRSRMJG2IUQ36KA23OG2R3A, seg_pos={0 0}, upload=330171]: Done uploading piece. err: ecclient: upload failed (node:1262xxX6aUPTYgKdvRxAyc77PmPBWJtq5cXRJFoELxHbfn18DLE, address:145.79.227.5:4247): protocol: rpc: dial tcp 145.79.227.5:4247: connect: no route to host; conn closed: op Sep 22 05:06:24.986206552 (375h38m1.025779682s) [upload=330171]: Next returned nil segment Sep 22 05:06:24.986226830 (375h38m1.02579995s) [upload=330171]: Waiting for error group managing segments... Sep 22 05:06:25.383578176 (375h38m1.423151306s) [noise=true , address=145.79.227.24:4594 , piece_id=V55Y7DFR7QJWBBQRKSTUWYNEZSCAIWAY2XHAOZOF2H7JDFSX6IBA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.383806571 (375h38m1.423379702s) [noise=true , address=145.79.227.33:4881 , piece_id=AIKNNVT5PMYFYKRMUFN5DQMY7TWEUN3FUC64IQQYKAMV723TKKZA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.398811493 (375h38m1.438384634s) [noise=true , address=145.79.227.46:4601 , piece_id=EVWU7M2P2W4FQCNDEQ2OO3SR4KHPTXQRM7GEI6Q6DHYMDHK4PP4Q, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.399530783 (375h38m1.439103913s) [noise=true , address=145.79.227.39:4039 , piece_id=MZRBH2VHR7CCJK4OCCXINOQTNBP7PUP236H7DB2DFL7ZFDBNI2IQ, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.406508421 (375h38m1.446081551s) [noise=true , address=145.79.227.49:4338 , piece_id=NBR62GETN7QBXC7QJCUB5DZHKVMC5AKQQBIIGDRXNECNZFULEDDA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.415736332 (375h38m1.455309462s) [noise=true , address=145.79.227.25:4637 , piece_id=3SMC4AHCMKJEDXJOJVZ3FJEKW23VZDSWW6MDCENAYJ7HKOPWNHBQ, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.437405044 (375h38m1.476978184s) [noise=true , address=145.79.227.31:4818 , piece_id=KAJZKTKOHNN5U4CTYJ7CLO32ONU6VSEJ2QBLPBI5WWZZ37YOI5NA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.465513107 (375h38m1.505086237s) [noise=true , address=77.90.31.19:4847 , piece_id=6GLTB4OVJTM67VRKRWATCIWE7WVK35IQMQIVOEZMZM6I5DFNRIBA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.501349827 (375h38m1.540922957s) [noise=true , address=45.91.116.9:4292 , piece_id=3QGXKEK5C77XDD34Z77C3K2LYEMGJT2SADTL6RGW2YCYQFV5CLUA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.579007182 (375h38m1.618580302s) [noise=true , address=77.90.31.16:4764 , piece_id=PSAP3HPVNEURWWNC7K2K2G4NEUENKWDUR7OJY6UBGRTCSBYNHRWQ, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.586807715 (375h38m1.626380845s) [noise=true , address=45.91.116.14:4214 , piece_id=DOY7FC4SJKGAFP3LHSJXKWCRSYI6I6IR2R2FZ47QJNDM4SCAUXQQ, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:07:22.245417769 (375h38m58.284990899s) [noise=true , address=145.79.227.16:4933 , piece_id=WOBRP7MRD6VVMMXEHKNL4XDSPKQBHVINQD3I4N2SQU2MOSC2ZUCQ, seg_pos={0 0}, upload=330171]: Done uploading piece. err: ecclient: upload failed (node:19m4inaXvTdwf2rKip3eFELv6JAgFvXJ8KqjPZYbb48i8Ktnwe, address:145.79.227.16:4933): protocol: rpc: dial tcp 145.79.227.16:4933: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:07:22.245544310 (375h38m58.28511744s) [noise=true , address=145.79.227.12:4770 , piece_id=X5KVN5TA67L4TTD7LVG6N4I54BKNESFAHXRPVIWGDFCEPSHRKXPA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: ecclient: upload failed (node:1Ug6GpYToYDXiHmiJP4GoE7eJV3kqRveix2S5SLDcKQJtEbNPz, address:145.79.227.12:4770): protocol: rpc: dial tcp 145.79.227.12:4770: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:07:22.245567825 (375h38m58.285140955s) [noise=true , address=145.79.227.16:4939 , piece_id=DLS7J35PB7NXNMAPTVHBZ3VN3U725CSVZZKEYY37HZWQPWCCJO7A, seg_pos={0 0}, upload=330171]: Done uploading piece. err: ecclient: upload failed (node:1mRmp3HhD95PruS4AF8soRsSXNYUeb4mJE7y52FvoeN9D3Gu4F, address:145.79.227.16:4939): protocol: rpc: dial tcp 145.79.227.16:4939: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:07:22.245634181 (375h38m58.285207312s) [noise=true , address=145.79.227.7:4308 , piece_id=QEUKJMQDWYMBVKSVNNVDK53P6CZ4X47IINRSWCHAFG7APSFOOAFQ, seg_pos={0 0}, upload=330171]: Done uploading piece. err: ecclient: upload failed (node:1PgA5SZkBvK673LsV7p2sXpbi1r79esNNZspPZTzaitqfbjtyY, address:145.79.227.7:4308): protocol: rpc: dial tcp 145.79.227.7:4308: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:07:22.245657335 (375h38m58.285230455s) [noise=true , address=145.79.227.10:4420 , piece_id=ANGUQNIL2D4K2YR7KVJMY37WW7ILRVXSZOB7FRRAKJPD3AGGPGOA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: ecclient: upload failed (node:12hJrjeMUMYmFw3yxRtQQKRh9ciPfdu5FXo8NGrsc6HBqgfJzzp, address:145.79.227.10:4420): protocol: rpc: dial tcp 145.79.227.10:4420: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:07:22.245675269 (375h38m58.28524839s) [noise=true , address=145.79.227.6:4292 , piece_id=BYUJPZSYFTYD4C4F5EBAW6BSFXTILW6EASOIRHZQI7ARCSG63F2Q, seg_pos={0 0}, upload=330171]: Done uploading piece. err: ecclient: upload failed (node:12UpWmMhEmk5veKZvzsQ3MCxCMnfEKtFgCugBTsYfy9Ejm7G4xD, address:145.79.227.6:4292): protocol: rpc: dial tcp 145.79.227.6:4292: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:07:22.245712370 (375h38m58.2852855s) [seg_pos={0 0}, upload=330171]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Sep 22 05:07:22.500217377 (375h38m58.539790507s) [upload=330171]: Done uploading segments... Sep 22 05:07:22.500253296 (375h38m58.539826426s) [upload=330171]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 5.784556435s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [X.X.XX.....X.X....................X.........................] duration: 5.990658739s FAIL (2025-09-22 05:48:17 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== 3.337157461s) [upload=330511]: Starting upload Sep 22 05:29:47.297668972 (376h1m23.337242102s) [upload=330511]: Done starting upload Sep 22 05:29:47.297899231 (376h1m23.337472361s) [upload=330511]: Uploading segments... Sep 22 05:29:47.298002678 (376h1m23.337575818s) [upload=330511]: Got next segment. Inline: false Sep 22 05:29:47.298059145 (376h1m23.337632275s) [upload=330511]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 05:29:47.298066109 (376h1m23.337639229s) [upload=330511]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 05:29:47.552911815 (376h1m23.592484945s) [seg_pos={0 0}, upload=330511]: Begin upload segment... Sep 22 05:29:47.552975406 (376h1m23.592548536s) [seg_pos={0 0}, upload=330511]: Done begin upload segment. Sep 22 05:29:47.553013599 (376h1m23.592586719s) [noise=true , address=45.91.116.11:4117 , piece_id=NIFPLA2N3L6XDWTIFUM3YYRGBMFQJKWFVCGYXD2GBFYVYDXW6OEQ, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.553528389 (376h1m23.593101539s) [noise=true , address=145.79.227.6:4258 , piece_id=D7HMSQH7ZLLKNTUZGJLVHIW2KEQZDL3HSS6VO2BSNRZJAC3USFAQ, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.553925034 (376h1m23.593498154s) [noise=true , address=45.91.116.9:4261 , piece_id=REZZWYP25KZOP4UBJWJPHGCDUWCW3EBOJG2DOLCDYZY5B43KBMSQ, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.553961283 (376h1m23.593534413s) [noise=true , address=77.90.31.14:4663 , piece_id=LYGO7UTA55Q5Q7APPDJZNNWZ4S2XHAT4LS4V27VF27A3435DVJQA, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.554326960 (376h1m23.59390009s) [noise=true , address=77.90.31.16:4735 , piece_id=K5KYKKY3BQV6UVJK4G3DYC24DFFIEFQX7KGGFLS4P35KJRFHA6UQ, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.554351497 (376h1m23.593924627s) [noise=true , address=145.79.227.16:4912 , piece_id=AG2FUP2KFIJ7W55H2VRYTJYWQ57MAXCTEMTDWSR4FUJAYE3HFKBQ, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.554588828 (376h1m23.594161948s) [noise=true , address=145.79.227.12:4767 , piece_id=OUVFSM62B7UZ4U4WGFFBOHCPHXEXR6A6H2FQL3DTG5L3DEPUVYVQ, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.554724957 (376h1m23.594298087s) [noise=true , address=77.90.31.18:4807 , piece_id=JFOMLF75MIXEDP5CKDHH7J3PWGWESB7BIKK3SCAIOVNX74DJDC3A, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.554875986 (376h1m23.594449116s) [noise=true , address=145.79.227.3:4151 , piece_id=O6AU3AOBKJRIUMEAWPHMWWDDNRTA52WL72JNUTK5JFLVG2TNV5IA, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.555093950 (376h1m23.59466708s) [noise=true , address=145.79.227.17:4972 , piece_id=Q3KQQ57XWDUCXNHP33STZ7DN6VQ6FVYZTDPLINEPZASFEVY5XDJA, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.555135429 (376h1m23.594708559s) [noise=true , address=45.91.116.6:4973 , piece_id=WPWUFOFQHQZFRPTR33GTDST2NYCVPXKLDVCJQFPQ4A7PI44LJKOQ, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.555464797 (376h1m23.595037917s) [noise=true , address=45.91.116.14:4189 , piece_id=6VJ6XCCEV2CZSARNRF3ATZ4KCUI75QKWIWQWJL5QH4XTQG4MCLUQ, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.555726194 (376h1m23.595299314s) [noise=true , address=77.90.31.15:4699 , piece_id=6XZN64DOILMELBYDU4D72ER33XDKMRIJRSAXX5ITFYPG46YEBITQ, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.556009483 (376h1m23.595582614s) [noise=true , address=145.79.227.19:4077 , piece_id=EDBYIPZUOPFRRIZBZJJEP2U7NZFCUHRLDZESZ7J5CQZFKADNLE6Q, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.556364609 (376h1m23.59593773s) [noise=true , address=145.79.227.15:4875 , piece_id=ZEWO52263YKTTRYDVF4PV52JTPEDJR3XGVRG3BFE5QQKDQSCE2LQ, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.556611299 (376h1m23.596184429s) [noise=true , address=145.79.227.5:4222 , piece_id=CZNVNWHETTFQS6JR2YLE65LNQU4WX74DAROY5I3KAYHG4QDLDOKA, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:49.308510933 (376h1m25.348084063s) [upload=330511]: Next returned nil segment Sep 22 05:29:49.308533666 (376h1m25.348106796s) [upload=330511]: Waiting for error group managing segments... Sep 22 05:29:49.602590138 (376h1m25.642163268s) [noise=true , address=45.91.116.11:4117 , piece_id=NIFPLA2N3L6XDWTIFUM3YYRGBMFQJKWFVCGYXD2GBFYVYDXW6OEQ, seg_pos={0 0}, upload=330511]: Done uploading piece. err: <nil> Sep 22 05:29:49.633269677 (376h1m25.672842807s) [noise=true , address=77.90.31.15:4699 , piece_id=6XZN64DOILMELBYDU4D72ER33XDKMRIJRSAXX5ITFYPG46YEBITQ, seg_pos={0 0}, upload=330511]: Done uploading piece. err: <nil> Sep 22 05:29:49.637382749 (376h1m25.676955879s) [noise=true , address=45.91.116.6:4973 , piece_id=WPWUFOFQHQZFRPTR33GTDST2NYCVPXKLDVCJQFPQ4A7PI44LJKOQ, seg_pos={0 0}, upload=330511]: Done uploading piece. err: <nil> Sep 22 05:29:49.648543782 (376h1m25.688116902s) [noise=true , address=77.90.31.16:4735 , piece_id=K5KYKKY3BQV6UVJK4G3DYC24DFFIEFQX7KGGFLS4P35KJRFHA6UQ, seg_pos={0 0}, upload=330511]: Done uploading piece. err: <nil> Sep 22 05:29:49.653934647 (376h1m25.693507767s) [noise=true , address=45.91.116.14:4189 , piece_id=6VJ6XCCEV2CZSARNRF3ATZ4KCUI75QKWIWQWJL5QH4XTQG4MCLUQ, seg_pos={0 0}, upload=330511]: Done uploading piece. err: <nil> Sep 22 05:29:49.669096699 (376h1m25.708669829s) [noise=true , address=45.91.116.9:4261 , piece_id=REZZWYP25KZOP4UBJWJPHGCDUWCW3EBOJG2DOLCDYZY5B43KBMSQ, seg_pos={0 0}, upload=330511]: Done uploading piece. err: <nil> Sep 22 05:29:49.673672402 (376h1m25.713245532s) [noise=true , address=77.90.31.14:4663 , piece_id=LYGO7UTA55Q5Q7APPDJZNNWZ4S2XHAT4LS4V27VF27A3435DVJQA, seg_pos={0 0}, upload=330511]: Done uploading piece. err: <nil> Sep 22 05:29:49.741841689 (376h1m25.781414809s) [noise=true , address=77.90.31.18:4807 , piece_id=JFOMLF75MIXEDP5CKDHH7J3PWGWESB7BIKK3SCAIOVNX74DJDC3A, seg_pos={0 0}, upload=330511]: Done uploading piece. err: <nil> Sep 22 05:29:58.021015047 (376h1m34.060588167s) [noise=true , address=145.79.227.5:4222 , piece_id=CZNVNWHETTFQS6JR2YLE65LNQU4WX74DAROY5I3KAYHG4QDLDOKA, seg_pos={0 0}, upload=330511]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:145.79.227.5:4222): protocol: rpc: dial tcp 145.79.227.5:4222: connect: no route to host; conn closed: op; piecestore: piecestore close: manager closed: conn closed: op Sep 22 05:30:05.028605724 (376h1m41.068178854s) [noise=true , address=145.79.227.3:4151 , piece_id=O6AU3AOBKJRIUMEAWPHMWWDDNRTA52WL72JNUTK5JFLVG2TNV5IA, seg_pos={0 0}, upload=330511]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:145.79.227.3:4151): protocol: rpc: dial tcp 145.79.227.3:4151: connect: no route to host; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:30:16.586571260 (376h1m52.62614439s) [noise=true , address=145.79.227.12:4767 , piece_id=OUVFSM62B7UZ4U4WGFFBOHCPHXEXR6A6H2FQL3DTG5L3DEPUVYVQ, seg_pos={0 0}, upload=330511]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:145.79.227.12:4767): protocol: rpc: dial tcp 145.79.227.12:4767: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:30:16.586679978 (376h1m52.626253118s) [noise=true , address=145.79.227.6:4258 , piece_id=D7HMSQH7ZLLKNTUZGJLVHIW2KEQZDL3HSS6VO2BSNRZJAC3USFAQ, seg_pos={0 0}, upload=330511]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:145.79.227.6:4258): protocol: rpc: dial tcp 145.79.227.6:4258: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:30:16.586708171 (376h1m52.626281301s) [noise=true , address=145.79.227.16:4912 , piece_id=AG2FUP2KFIJ7W55H2VRYTJYWQ57MAXCTEMTDWSR4FUJAYE3HFKBQ, seg_pos={0 0}, upload=330511]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:145.79.227.16:4912): protocol: rpc: dial tcp 145.79.227.16:4912: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:30:16.586749971 (376h1m52.626323101s) [noise=true , address=145.79.227.15:4875 , piece_id=ZEWO52263YKTTRYDVF4PV52JTPEDJR3XGVRG3BFE5QQKDQSCE2LQ, seg_pos={0 0}, upload=330511]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: rpc: dial tcp 145.79.227.15:4875: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:30:16.586795768 (376h1m52.626368888s) [noise=true , address=145.79.227.17:4972 , piece_id=Q3KQQ57XWDUCXNHP33STZ7DN6VQ6FVYZTDPLINEPZASFEVY5XDJA, seg_pos={0 0}, upload=330511]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:145.79.227.17:4972): protocol: rpc: dial tcp 145.79.227.17:4972: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:30:16.586806088 (376h1m52.626379218s) [noise=true , address=145.79.227.19:4077 , piece_id=EDBYIPZUOPFRRIZBZJJEP2U7NZFCUHRLDZESZ7J5CQZFKADNLE6Q, seg_pos={0 0}, upload=330511]: Done uploading piece. err: ecclient: upload failed (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk, address:145.79.227.19:4077): protocol: rpc: dial tcp 145.79.227.19:4077: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:30:16.586831426 (376h1m52.626404556s) [seg_pos={0 0}, upload=330511]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Sep 22 05:30:16.834944071 (376h1m52.874517201s) [upload=330511]: Done uploading segments... Sep 22 05:30:16.834981573 (376h1m52.874554703s) [upload=330511]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 5.990696119s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [XX.X.XX.....X.X....................X........................] duration: 5.939775537s FAIL (2025-09-22 05:47:57 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== 3.337157461s) [upload=330511]: Starting upload Sep 22 05:29:47.297668972 (376h1m23.337242102s) [upload=330511]: Done starting upload Sep 22 05:29:47.297899231 (376h1m23.337472361s) [upload=330511]: Uploading segments... Sep 22 05:29:47.298002678 (376h1m23.337575818s) [upload=330511]: Got next segment. Inline: false Sep 22 05:29:47.298059145 (376h1m23.337632275s) [upload=330511]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 05:29:47.298066109 (376h1m23.337639229s) [upload=330511]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 05:29:47.552911815 (376h1m23.592484945s) [seg_pos={0 0}, upload=330511]: Begin upload segment... Sep 22 05:29:47.552975406 (376h1m23.592548536s) [seg_pos={0 0}, upload=330511]: Done begin upload segment. Sep 22 05:29:47.553013599 (376h1m23.592586719s) [noise=true , address=45.91.116.11:4117 , piece_id=NIFPLA2N3L6XDWTIFUM3YYRGBMFQJKWFVCGYXD2GBFYVYDXW6OEQ, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.553528389 (376h1m23.593101539s) [noise=true , address=145.79.227.6:4258 , piece_id=D7HMSQH7ZLLKNTUZGJLVHIW2KEQZDL3HSS6VO2BSNRZJAC3USFAQ, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.553925034 (376h1m23.593498154s) [noise=true , address=45.91.116.9:4261 , piece_id=REZZWYP25KZOP4UBJWJPHGCDUWCW3EBOJG2DOLCDYZY5B43KBMSQ, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.553961283 (376h1m23.593534413s) [noise=true , address=77.90.31.14:4663 , piece_id=LYGO7UTA55Q5Q7APPDJZNNWZ4S2XHAT4LS4V27VF27A3435DVJQA, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.554326960 (376h1m23.59390009s) [noise=true , address=77.90.31.16:4735 , piece_id=K5KYKKY3BQV6UVJK4G3DYC24DFFIEFQX7KGGFLS4P35KJRFHA6UQ, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.554351497 (376h1m23.593924627s) [noise=true , address=145.79.227.16:4912 , piece_id=AG2FUP2KFIJ7W55H2VRYTJYWQ57MAXCTEMTDWSR4FUJAYE3HFKBQ, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.554588828 (376h1m23.594161948s) [noise=true , address=145.79.227.12:4767 , piece_id=OUVFSM62B7UZ4U4WGFFBOHCPHXEXR6A6H2FQL3DTG5L3DEPUVYVQ, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.554724957 (376h1m23.594298087s) [noise=true , address=77.90.31.18:4807 , piece_id=JFOMLF75MIXEDP5CKDHH7J3PWGWESB7BIKK3SCAIOVNX74DJDC3A, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.554875986 (376h1m23.594449116s) [noise=true , address=145.79.227.3:4151 , piece_id=O6AU3AOBKJRIUMEAWPHMWWDDNRTA52WL72JNUTK5JFLVG2TNV5IA, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.555093950 (376h1m23.59466708s) [noise=true , address=145.79.227.17:4972 , piece_id=Q3KQQ57XWDUCXNHP33STZ7DN6VQ6FVYZTDPLINEPZASFEVY5XDJA, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.555135429 (376h1m23.594708559s) [noise=true , address=45.91.116.6:4973 , piece_id=WPWUFOFQHQZFRPTR33GTDST2NYCVPXKLDVCJQFPQ4A7PI44LJKOQ, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.555464797 (376h1m23.595037917s) [noise=true , address=45.91.116.14:4189 , piece_id=6VJ6XCCEV2CZSARNRF3ATZ4KCUI75QKWIWQWJL5QH4XTQG4MCLUQ, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.555726194 (376h1m23.595299314s) [noise=true , address=77.90.31.15:4699 , piece_id=6XZN64DOILMELBYDU4D72ER33XDKMRIJRSAXX5ITFYPG46YEBITQ, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.556009483 (376h1m23.595582614s) [noise=true , address=145.79.227.19:4077 , piece_id=EDBYIPZUOPFRRIZBZJJEP2U7NZFCUHRLDZESZ7J5CQZFKADNLE6Q, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.556364609 (376h1m23.59593773s) [noise=true , address=145.79.227.15:4875 , piece_id=ZEWO52263YKTTRYDVF4PV52JTPEDJR3XGVRG3BFE5QQKDQSCE2LQ, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.556611299 (376h1m23.596184429s) [noise=true , address=145.79.227.5:4222 , piece_id=CZNVNWHETTFQS6JR2YLE65LNQU4WX74DAROY5I3KAYHG4QDLDOKA, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:49.308510933 (376h1m25.348084063s) [upload=330511]: Next returned nil segment Sep 22 05:29:49.308533666 (376h1m25.348106796s) [upload=330511]: Waiting for error group managing segments... Sep 22 05:29:49.602590138 (376h1m25.642163268s) [noise=true , address=45.91.116.11:4117 , piece_id=NIFPLA2N3L6XDWTIFUM3YYRGBMFQJKWFVCGYXD2GBFYVYDXW6OEQ, seg_pos={0 0}, upload=330511]: Done uploading piece. err: <nil> Sep 22 05:29:49.633269677 (376h1m25.672842807s) [noise=true , address=77.90.31.15:4699 , piece_id=6XZN64DOILMELBYDU4D72ER33XDKMRIJRSAXX5ITFYPG46YEBITQ, seg_pos={0 0}, upload=330511]: Done uploading piece. err: <nil> Sep 22 05:29:49.637382749 (376h1m25.676955879s) [noise=true , address=45.91.116.6:4973 , piece_id=WPWUFOFQHQZFRPTR33GTDST2NYCVPXKLDVCJQFPQ4A7PI44LJKOQ, seg_pos={0 0}, upload=330511]: Done uploading piece. err: <nil> Sep 22 05:29:49.648543782 (376h1m25.688116902s) [noise=true , address=77.90.31.16:4735 , piece_id=K5KYKKY3BQV6UVJK4G3DYC24DFFIEFQX7KGGFLS4P35KJRFHA6UQ, seg_pos={0 0}, upload=330511]: Done uploading piece. err: <nil> Sep 22 05:29:49.653934647 (376h1m25.693507767s) [noise=true , address=45.91.116.14:4189 , piece_id=6VJ6XCCEV2CZSARNRF3ATZ4KCUI75QKWIWQWJL5QH4XTQG4MCLUQ, seg_pos={0 0}, upload=330511]: Done uploading piece. err: <nil> Sep 22 05:29:49.669096699 (376h1m25.708669829s) [noise=true , address=45.91.116.9:4261 , piece_id=REZZWYP25KZOP4UBJWJPHGCDUWCW3EBOJG2DOLCDYZY5B43KBMSQ, seg_pos={0 0}, upload=330511]: Done uploading piece. err: <nil> Sep 22 05:29:49.673672402 (376h1m25.713245532s) [noise=true , address=77.90.31.14:4663 , piece_id=LYGO7UTA55Q5Q7APPDJZNNWZ4S2XHAT4LS4V27VF27A3435DVJQA, seg_pos={0 0}, upload=330511]: Done uploading piece. err: <nil> Sep 22 05:29:49.741841689 (376h1m25.781414809s) [noise=true , address=77.90.31.18:4807 , piece_id=JFOMLF75MIXEDP5CKDHH7J3PWGWESB7BIKK3SCAIOVNX74DJDC3A, seg_pos={0 0}, upload=330511]: Done uploading piece. err: <nil> Sep 22 05:29:58.021015047 (376h1m34.060588167s) [noise=true , address=145.79.227.5:4222 , piece_id=CZNVNWHETTFQS6JR2YLE65LNQU4WX74DAROY5I3KAYHG4QDLDOKA, seg_pos={0 0}, upload=330511]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:145.79.227.5:4222): protocol: rpc: dial tcp 145.79.227.5:4222: connect: no route to host; conn closed: op; piecestore: piecestore close: manager closed: conn closed: op Sep 22 05:30:05.028605724 (376h1m41.068178854s) [noise=true , address=145.79.227.3:4151 , piece_id=O6AU3AOBKJRIUMEAWPHMWWDDNRTA52WL72JNUTK5JFLVG2TNV5IA, seg_pos={0 0}, upload=330511]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:145.79.227.3:4151): protocol: rpc: dial tcp 145.79.227.3:4151: connect: no route to host; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:30:16.586571260 (376h1m52.62614439s) [noise=true , address=145.79.227.12:4767 , piece_id=OUVFSM62B7UZ4U4WGFFBOHCPHXEXR6A6H2FQL3DTG5L3DEPUVYVQ, seg_pos={0 0}, upload=330511]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:145.79.227.12:4767): protocol: rpc: dial tcp 145.79.227.12:4767: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:30:16.586679978 (376h1m52.626253118s) [noise=true , address=145.79.227.6:4258 , piece_id=D7HMSQH7ZLLKNTUZGJLVHIW2KEQZDL3HSS6VO2BSNRZJAC3USFAQ, seg_pos={0 0}, upload=330511]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:145.79.227.6:4258): protocol: rpc: dial tcp 145.79.227.6:4258: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:30:16.586708171 (376h1m52.626281301s) [noise=true , address=145.79.227.16:4912 , piece_id=AG2FUP2KFIJ7W55H2VRYTJYWQ57MAXCTEMTDWSR4FUJAYE3HFKBQ, seg_pos={0 0}, upload=330511]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:145.79.227.16:4912): protocol: rpc: dial tcp 145.79.227.16:4912: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:30:16.586749971 (376h1m52.626323101s) [noise=true , address=145.79.227.15:4875 , piece_id=ZEWO52263YKTTRYDVF4PV52JTPEDJR3XGVRG3BFE5QQKDQSCE2LQ, seg_pos={0 0}, upload=330511]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: rpc: dial tcp 145.79.227.15:4875: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:30:16.586795768 (376h1m52.626368888s) [noise=true , address=145.79.227.17:4972 , piece_id=Q3KQQ57XWDUCXNHP33STZ7DN6VQ6FVYZTDPLINEPZASFEVY5XDJA, seg_pos={0 0}, upload=330511]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:145.79.227.17:4972): protocol: rpc: dial tcp 145.79.227.17:4972: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:30:16.586806088 (376h1m52.626379218s) [noise=true , address=145.79.227.19:4077 , piece_id=EDBYIPZUOPFRRIZBZJJEP2U7NZFCUHRLDZESZ7J5CQZFKADNLE6Q, seg_pos={0 0}, upload=330511]: Done uploading piece. err: ecclient: upload failed (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk, address:145.79.227.19:4077): protocol: rpc: dial tcp 145.79.227.19:4077: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:30:16.586831426 (376h1m52.626404556s) [seg_pos={0 0}, upload=330511]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Sep 22 05:30:16.834944071 (376h1m52.874517201s) [upload=330511]: Done uploading segments... Sep 22 05:30:16.834981573 (376h1m52.874554703s) [upload=330511]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 5.93980883s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.X...X......................................................] duration: 5.784022442s FAIL (2025-09-22 05:47:57 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 5h37m59.543525066s) [seg_pos={0 0}, upload=330171]: Begin upload segment... Sep 22 05:06:23.504018422 (375h37m59.543591552s) [seg_pos={0 0}, upload=330171]: Done begin upload segment. Sep 22 05:06:23.504059370 (375h37m59.543632501s) [noise=true , address=45.91.116.9:4292 , piece_id=3QGXKEK5C77XDD34Z77C3K2LYEMGJT2SADTL6RGW2YCYQFV5CLUA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.504681175 (375h37m59.544254355s) [noise=true , address=145.79.227.39:4039 , piece_id=MZRBH2VHR7CCJK4OCCXINOQTNBP7PUP236H7DB2DFL7ZFDBNI2IQ, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.505278141 (375h37m59.544851301s) [noise=true , address=145.79.227.33:4881 , piece_id=AIKNNVT5PMYFYKRMUFN5DQMY7TWEUN3FUC64IQQYKAMV723TKKZA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.505743808 (375h37m59.545316948s) [noise=true , address=145.79.227.46:4601 , piece_id=EVWU7M2P2W4FQCNDEQ2OO3SR4KHPTXQRM7GEI6Q6DHYMDHK4PP4Q, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.505904454 (375h37m59.545477584s) [noise=true , address=145.79.227.24:4594 , piece_id=V55Y7DFR7QJWBBQRKSTUWYNEZSCAIWAY2XHAOZOF2H7JDFSX6IBA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.506859502 (375h37m59.546432622s) [noise=true , address=145.79.227.10:4420 , piece_id=ANGUQNIL2D4K2YR7KVJMY37WW7ILRVXSZOB7FRRAKJPD3AGGPGOA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.506958631 (375h37m59.546531761s) [noise=true , address=145.79.227.16:4933 , piece_id=WOBRP7MRD6VVMMXEHKNL4XDSPKQBHVINQD3I4N2SQU2MOSC2ZUCQ, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.507277077 (375h37m59.546850217s) [noise=true , address=145.79.227.7:4308 , piece_id=QEUKJMQDWYMBVKSVNNVDK53P6CZ4X47IINRSWCHAFG7APSFOOAFQ, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.508336856 (375h37m59.547909986s) [noise=true , address=77.90.31.19:4847 , piece_id=6GLTB4OVJTM67VRKRWATCIWE7WVK35IQMQIVOEZMZM6I5DFNRIBA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.513150782 (375h37m59.552723912s) [noise=true , address=145.79.227.12:4770 , piece_id=X5KVN5TA67L4TTD7LVG6N4I54BKNESFAHXRPVIWGDFCEPSHRKXPA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.513934875 (375h37m59.553508006s) [noise=true , address=45.91.116.14:4214 , piece_id=DOY7FC4SJKGAFP3LHSJXKWCRSYI6I6IR2R2FZ47QJNDM4SCAUXQQ, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.514566187 (375h37m59.554139318s) [noise=true , address=145.79.227.16:4939 , piece_id=DLS7J35PB7NXNMAPTVHBZ3VN3U725CSVZZKEYY37HZWQPWCCJO7A, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.515189424 (375h37m59.554762544s) [noise=true , address=145.79.227.25:4637 , piece_id=3SMC4AHCMKJEDXJOJVZ3FJEKW23VZDSWW6MDCENAYJ7HKOPWNHBQ, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.515805377 (375h37m59.555378508s) [noise=true , address=145.79.227.6:4292 , piece_id=BYUJPZSYFTYD4C4F5EBAW6BSFXTILW6EASOIRHZQI7ARCSG63F2Q, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.516494880 (375h37m59.55606801s) [noise=true , address=77.90.31.16:4764 , piece_id=PSAP3HPVNEURWWNC7K2K2G4NEUENKWDUR7OJY6UBGRTCSBYNHRWQ, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.526689722 (375h37m59.566262863s) [noise=true , address=145.79.227.31:4818 , piece_id=KAJZKTKOHNN5U4CTYJ7CLO32ONU6VSEJ2QBLPBI5WWZZ37YOI5NA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.531527154 (375h37m59.571100294s) [noise=true , address=145.79.227.49:4338 , piece_id=NBR62GETN7QBXC7QJCUB5DZHKVMC5AKQQBIIGDRXNECNZFULEDDA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.537921861 (375h37m59.577494991s) [noise=true , address=145.79.227.5:4247 , piece_id=QGN2S5CET2J6LGW6YPQYDWEVOGEMHWRSRMJG2IUQ36KA23OG2R3A, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:24.564378058 (375h38m0.603951188s) [noise=true , address=145.79.227.5:4247 , piece_id=QGN2S5CET2J6LGW6YPQYDWEVOGEMHWRSRMJG2IUQ36KA23OG2R3A, seg_pos={0 0}, upload=330171]: Done uploading piece. err: ecclient: upload failed (node:1262xxX6aUPTYgKdvRxAyc77PmPBWJtq5cXRJFoELxHbfn18DLE, address:145.79.227.5:4247): protocol: rpc: dial tcp 145.79.227.5:4247: connect: no route to host; conn closed: op Sep 22 05:06:24.986206552 (375h38m1.025779682s) [upload=330171]: Next returned nil segment Sep 22 05:06:24.986226830 (375h38m1.02579995s) [upload=330171]: Waiting for error group managing segments... Sep 22 05:06:25.383578176 (375h38m1.423151306s) [noise=true , address=145.79.227.24:4594 , piece_id=V55Y7DFR7QJWBBQRKSTUWYNEZSCAIWAY2XHAOZOF2H7JDFSX6IBA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.383806571 (375h38m1.423379702s) [noise=true , address=145.79.227.33:4881 , piece_id=AIKNNVT5PMYFYKRMUFN5DQMY7TWEUN3FUC64IQQYKAMV723TKKZA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.398811493 (375h38m1.438384634s) [noise=true , address=145.79.227.46:4601 , piece_id=EVWU7M2P2W4FQCNDEQ2OO3SR4KHPTXQRM7GEI6Q6DHYMDHK4PP4Q, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.399530783 (375h38m1.439103913s) [noise=true , address=145.79.227.39:4039 , piece_id=MZRBH2VHR7CCJK4OCCXINOQTNBP7PUP236H7DB2DFL7ZFDBNI2IQ, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.406508421 (375h38m1.446081551s) [noise=true , address=145.79.227.49:4338 , piece_id=NBR62GETN7QBXC7QJCUB5DZHKVMC5AKQQBIIGDRXNECNZFULEDDA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.415736332 (375h38m1.455309462s) [noise=true , address=145.79.227.25:4637 , piece_id=3SMC4AHCMKJEDXJOJVZ3FJEKW23VZDSWW6MDCENAYJ7HKOPWNHBQ, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.437405044 (375h38m1.476978184s) [noise=true , address=145.79.227.31:4818 , piece_id=KAJZKTKOHNN5U4CTYJ7CLO32ONU6VSEJ2QBLPBI5WWZZ37YOI5NA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.465513107 (375h38m1.505086237s) [noise=true , address=77.90.31.19:4847 , piece_id=6GLTB4OVJTM67VRKRWATCIWE7WVK35IQMQIVOEZMZM6I5DFNRIBA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.501349827 (375h38m1.540922957s) [noise=true , address=45.91.116.9:4292 , piece_id=3QGXKEK5C77XDD34Z77C3K2LYEMGJT2SADTL6RGW2YCYQFV5CLUA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.579007182 (375h38m1.618580302s) [noise=true , address=77.90.31.16:4764 , piece_id=PSAP3HPVNEURWWNC7K2K2G4NEUENKWDUR7OJY6UBGRTCSBYNHRWQ, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.586807715 (375h38m1.626380845s) [noise=true , address=45.91.116.14:4214 , piece_id=DOY7FC4SJKGAFP3LHSJXKWCRSYI6I6IR2R2FZ47QJNDM4SCAUXQQ, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:07:22.245417769 (375h38m58.284990899s) [noise=true , address=145.79.227.16:4933 , piece_id=WOBRP7MRD6VVMMXEHKNL4XDSPKQBHVINQD3I4N2SQU2MOSC2ZUCQ, seg_pos={0 0}, upload=330171]: Done uploading piece. err: ecclient: upload failed (node:19m4inaXvTdwf2rKip3eFELv6JAgFvXJ8KqjPZYbb48i8Ktnwe, address:145.79.227.16:4933): protocol: rpc: dial tcp 145.79.227.16:4933: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:07:22.245544310 (375h38m58.28511744s) [noise=true , address=145.79.227.12:4770 , piece_id=X5KVN5TA67L4TTD7LVG6N4I54BKNESFAHXRPVIWGDFCEPSHRKXPA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: ecclient: upload failed (node:1Ug6GpYToYDXiHmiJP4GoE7eJV3kqRveix2S5SLDcKQJtEbNPz, address:145.79.227.12:4770): protocol: rpc: dial tcp 145.79.227.12:4770: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:07:22.245567825 (375h38m58.285140955s) [noise=true , address=145.79.227.16:4939 , piece_id=DLS7J35PB7NXNMAPTVHBZ3VN3U725CSVZZKEYY37HZWQPWCCJO7A, seg_pos={0 0}, upload=330171]: Done uploading piece. err: ecclient: upload failed (node:1mRmp3HhD95PruS4AF8soRsSXNYUeb4mJE7y52FvoeN9D3Gu4F, address:145.79.227.16:4939): protocol: rpc: dial tcp 145.79.227.16:4939: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:07:22.245634181 (375h38m58.285207312s) [noise=true , address=145.79.227.7:4308 , piece_id=QEUKJMQDWYMBVKSVNNVDK53P6CZ4X47IINRSWCHAFG7APSFOOAFQ, seg_pos={0 0}, upload=330171]: Done uploading piece. err: ecclient: upload failed (node:1PgA5SZkBvK673LsV7p2sXpbi1r79esNNZspPZTzaitqfbjtyY, address:145.79.227.7:4308): protocol: rpc: dial tcp 145.79.227.7:4308: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:07:22.245657335 (375h38m58.285230455s) [noise=true , address=145.79.227.10:4420 , piece_id=ANGUQNIL2D4K2YR7KVJMY37WW7ILRVXSZOB7FRRAKJPD3AGGPGOA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: ecclient: upload failed (node:12hJrjeMUMYmFw3yxRtQQKRh9ciPfdu5FXo8NGrsc6HBqgfJzzp, address:145.79.227.10:4420): protocol: rpc: dial tcp 145.79.227.10:4420: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:07:22.245675269 (375h38m58.28524839s) [noise=true , address=145.79.227.6:4292 , piece_id=BYUJPZSYFTYD4C4F5EBAW6BSFXTILW6EASOIRHZQI7ARCSG63F2Q, seg_pos={0 0}, upload=330171]: Done uploading piece. err: ecclient: upload failed (node:12UpWmMhEmk5veKZvzsQ3MCxCMnfEKtFgCugBTsYfy9Ejm7G4xD, address:145.79.227.6:4292): protocol: rpc: dial tcp 145.79.227.6:4292: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:07:22.245712370 (375h38m58.2852855s) [seg_pos={0 0}, upload=330171]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Sep 22 05:07:22.500217377 (375h38m58.539790507s) [upload=330171]: Done uploading segments... Sep 22 05:07:22.500253296 (375h38m58.539826426s) [upload=330171]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 5.784052809s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [X....X..X...X.X......X.........X.X..X....X..XX..............] duration: 5.961057562s FAIL (2025-09-22 05:47:49 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.961082598s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [XX....X..X...X.X......X.........X.X..X....X..XX.............] duration: 6.038573667s FAIL (2025-09-22 05:47:27 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.038616719s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [XXX.X.XX.....X.X....................X.......................] duration: 5.90009775s FAIL (2025-09-22 05:47:05 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== 3.337157461s) [upload=330511]: Starting upload Sep 22 05:29:47.297668972 (376h1m23.337242102s) [upload=330511]: Done starting upload Sep 22 05:29:47.297899231 (376h1m23.337472361s) [upload=330511]: Uploading segments... Sep 22 05:29:47.298002678 (376h1m23.337575818s) [upload=330511]: Got next segment. Inline: false Sep 22 05:29:47.298059145 (376h1m23.337632275s) [upload=330511]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 05:29:47.298066109 (376h1m23.337639229s) [upload=330511]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 05:29:47.552911815 (376h1m23.592484945s) [seg_pos={0 0}, upload=330511]: Begin upload segment... Sep 22 05:29:47.552975406 (376h1m23.592548536s) [seg_pos={0 0}, upload=330511]: Done begin upload segment. Sep 22 05:29:47.553013599 (376h1m23.592586719s) [noise=true , address=45.91.116.11:4117 , piece_id=NIFPLA2N3L6XDWTIFUM3YYRGBMFQJKWFVCGYXD2GBFYVYDXW6OEQ, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.553528389 (376h1m23.593101539s) [noise=true , address=145.79.227.6:4258 , piece_id=D7HMSQH7ZLLKNTUZGJLVHIW2KEQZDL3HSS6VO2BSNRZJAC3USFAQ, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.553925034 (376h1m23.593498154s) [noise=true , address=45.91.116.9:4261 , piece_id=REZZWYP25KZOP4UBJWJPHGCDUWCW3EBOJG2DOLCDYZY5B43KBMSQ, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.553961283 (376h1m23.593534413s) [noise=true , address=77.90.31.14:4663 , piece_id=LYGO7UTA55Q5Q7APPDJZNNWZ4S2XHAT4LS4V27VF27A3435DVJQA, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.554326960 (376h1m23.59390009s) [noise=true , address=77.90.31.16:4735 , piece_id=K5KYKKY3BQV6UVJK4G3DYC24DFFIEFQX7KGGFLS4P35KJRFHA6UQ, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.554351497 (376h1m23.593924627s) [noise=true , address=145.79.227.16:4912 , piece_id=AG2FUP2KFIJ7W55H2VRYTJYWQ57MAXCTEMTDWSR4FUJAYE3HFKBQ, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.554588828 (376h1m23.594161948s) [noise=true , address=145.79.227.12:4767 , piece_id=OUVFSM62B7UZ4U4WGFFBOHCPHXEXR6A6H2FQL3DTG5L3DEPUVYVQ, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.554724957 (376h1m23.594298087s) [noise=true , address=77.90.31.18:4807 , piece_id=JFOMLF75MIXEDP5CKDHH7J3PWGWESB7BIKK3SCAIOVNX74DJDC3A, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.554875986 (376h1m23.594449116s) [noise=true , address=145.79.227.3:4151 , piece_id=O6AU3AOBKJRIUMEAWPHMWWDDNRTA52WL72JNUTK5JFLVG2TNV5IA, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.555093950 (376h1m23.59466708s) [noise=true , address=145.79.227.17:4972 , piece_id=Q3KQQ57XWDUCXNHP33STZ7DN6VQ6FVYZTDPLINEPZASFEVY5XDJA, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.555135429 (376h1m23.594708559s) [noise=true , address=45.91.116.6:4973 , piece_id=WPWUFOFQHQZFRPTR33GTDST2NYCVPXKLDVCJQFPQ4A7PI44LJKOQ, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.555464797 (376h1m23.595037917s) [noise=true , address=45.91.116.14:4189 , piece_id=6VJ6XCCEV2CZSARNRF3ATZ4KCUI75QKWIWQWJL5QH4XTQG4MCLUQ, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.555726194 (376h1m23.595299314s) [noise=true , address=77.90.31.15:4699 , piece_id=6XZN64DOILMELBYDU4D72ER33XDKMRIJRSAXX5ITFYPG46YEBITQ, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.556009483 (376h1m23.595582614s) [noise=true , address=145.79.227.19:4077 , piece_id=EDBYIPZUOPFRRIZBZJJEP2U7NZFCUHRLDZESZ7J5CQZFKADNLE6Q, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.556364609 (376h1m23.59593773s) [noise=true , address=145.79.227.15:4875 , piece_id=ZEWO52263YKTTRYDVF4PV52JTPEDJR3XGVRG3BFE5QQKDQSCE2LQ, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.556611299 (376h1m23.596184429s) [noise=true , address=145.79.227.5:4222 , piece_id=CZNVNWHETTFQS6JR2YLE65LNQU4WX74DAROY5I3KAYHG4QDLDOKA, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:49.308510933 (376h1m25.348084063s) [upload=330511]: Next returned nil segment Sep 22 05:29:49.308533666 (376h1m25.348106796s) [upload=330511]: Waiting for error group managing segments... Sep 22 05:29:49.602590138 (376h1m25.642163268s) [noise=true , address=45.91.116.11:4117 , piece_id=NIFPLA2N3L6XDWTIFUM3YYRGBMFQJKWFVCGYXD2GBFYVYDXW6OEQ, seg_pos={0 0}, upload=330511]: Done uploading piece. err: <nil> Sep 22 05:29:49.633269677 (376h1m25.672842807s) [noise=true , address=77.90.31.15:4699 , piece_id=6XZN64DOILMELBYDU4D72ER33XDKMRIJRSAXX5ITFYPG46YEBITQ, seg_pos={0 0}, upload=330511]: Done uploading piece. err: <nil> Sep 22 05:29:49.637382749 (376h1m25.676955879s) [noise=true , address=45.91.116.6:4973 , piece_id=WPWUFOFQHQZFRPTR33GTDST2NYCVPXKLDVCJQFPQ4A7PI44LJKOQ, seg_pos={0 0}, upload=330511]: Done uploading piece. err: <nil> Sep 22 05:29:49.648543782 (376h1m25.688116902s) [noise=true , address=77.90.31.16:4735 , piece_id=K5KYKKY3BQV6UVJK4G3DYC24DFFIEFQX7KGGFLS4P35KJRFHA6UQ, seg_pos={0 0}, upload=330511]: Done uploading piece. err: <nil> Sep 22 05:29:49.653934647 (376h1m25.693507767s) [noise=true , address=45.91.116.14:4189 , piece_id=6VJ6XCCEV2CZSARNRF3ATZ4KCUI75QKWIWQWJL5QH4XTQG4MCLUQ, seg_pos={0 0}, upload=330511]: Done uploading piece. err: <nil> Sep 22 05:29:49.669096699 (376h1m25.708669829s) [noise=true , address=45.91.116.9:4261 , piece_id=REZZWYP25KZOP4UBJWJPHGCDUWCW3EBOJG2DOLCDYZY5B43KBMSQ, seg_pos={0 0}, upload=330511]: Done uploading piece. err: <nil> Sep 22 05:29:49.673672402 (376h1m25.713245532s) [noise=true , address=77.90.31.14:4663 , piece_id=LYGO7UTA55Q5Q7APPDJZNNWZ4S2XHAT4LS4V27VF27A3435DVJQA, seg_pos={0 0}, upload=330511]: Done uploading piece. err: <nil> Sep 22 05:29:49.741841689 (376h1m25.781414809s) [noise=true , address=77.90.31.18:4807 , piece_id=JFOMLF75MIXEDP5CKDHH7J3PWGWESB7BIKK3SCAIOVNX74DJDC3A, seg_pos={0 0}, upload=330511]: Done uploading piece. err: <nil> Sep 22 05:29:58.021015047 (376h1m34.060588167s) [noise=true , address=145.79.227.5:4222 , piece_id=CZNVNWHETTFQS6JR2YLE65LNQU4WX74DAROY5I3KAYHG4QDLDOKA, seg_pos={0 0}, upload=330511]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:145.79.227.5:4222): protocol: rpc: dial tcp 145.79.227.5:4222: connect: no route to host; conn closed: op; piecestore: piecestore close: manager closed: conn closed: op Sep 22 05:30:05.028605724 (376h1m41.068178854s) [noise=true , address=145.79.227.3:4151 , piece_id=O6AU3AOBKJRIUMEAWPHMWWDDNRTA52WL72JNUTK5JFLVG2TNV5IA, seg_pos={0 0}, upload=330511]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:145.79.227.3:4151): protocol: rpc: dial tcp 145.79.227.3:4151: connect: no route to host; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:30:16.586571260 (376h1m52.62614439s) [noise=true , address=145.79.227.12:4767 , piece_id=OUVFSM62B7UZ4U4WGFFBOHCPHXEXR6A6H2FQL3DTG5L3DEPUVYVQ, seg_pos={0 0}, upload=330511]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:145.79.227.12:4767): protocol: rpc: dial tcp 145.79.227.12:4767: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:30:16.586679978 (376h1m52.626253118s) [noise=true , address=145.79.227.6:4258 , piece_id=D7HMSQH7ZLLKNTUZGJLVHIW2KEQZDL3HSS6VO2BSNRZJAC3USFAQ, seg_pos={0 0}, upload=330511]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:145.79.227.6:4258): protocol: rpc: dial tcp 145.79.227.6:4258: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:30:16.586708171 (376h1m52.626281301s) [noise=true , address=145.79.227.16:4912 , piece_id=AG2FUP2KFIJ7W55H2VRYTJYWQ57MAXCTEMTDWSR4FUJAYE3HFKBQ, seg_pos={0 0}, upload=330511]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:145.79.227.16:4912): protocol: rpc: dial tcp 145.79.227.16:4912: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:30:16.586749971 (376h1m52.626323101s) [noise=true , address=145.79.227.15:4875 , piece_id=ZEWO52263YKTTRYDVF4PV52JTPEDJR3XGVRG3BFE5QQKDQSCE2LQ, seg_pos={0 0}, upload=330511]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: rpc: dial tcp 145.79.227.15:4875: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:30:16.586795768 (376h1m52.626368888s) [noise=true , address=145.79.227.17:4972 , piece_id=Q3KQQ57XWDUCXNHP33STZ7DN6VQ6FVYZTDPLINEPZASFEVY5XDJA, seg_pos={0 0}, upload=330511]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:145.79.227.17:4972): protocol: rpc: dial tcp 145.79.227.17:4972: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:30:16.586806088 (376h1m52.626379218s) [noise=true , address=145.79.227.19:4077 , piece_id=EDBYIPZUOPFRRIZBZJJEP2U7NZFCUHRLDZESZ7J5CQZFKADNLE6Q, seg_pos={0 0}, upload=330511]: Done uploading piece. err: ecclient: upload failed (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk, address:145.79.227.19:4077): protocol: rpc: dial tcp 145.79.227.19:4077: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:30:16.586831426 (376h1m52.626404556s) [seg_pos={0 0}, upload=330511]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Sep 22 05:30:16.834944071 (376h1m52.874517201s) [upload=330511]: Done uploading segments... Sep 22 05:30:16.834981573 (376h1m52.874554703s) [upload=330511]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 5.900126435s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [..X...X.....................................................] duration: 5.815538303s FAIL (2025-09-22 05:46:54 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 5h37m59.543525066s) [seg_pos={0 0}, upload=330171]: Begin upload segment... Sep 22 05:06:23.504018422 (375h37m59.543591552s) [seg_pos={0 0}, upload=330171]: Done begin upload segment. Sep 22 05:06:23.504059370 (375h37m59.543632501s) [noise=true , address=45.91.116.9:4292 , piece_id=3QGXKEK5C77XDD34Z77C3K2LYEMGJT2SADTL6RGW2YCYQFV5CLUA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.504681175 (375h37m59.544254355s) [noise=true , address=145.79.227.39:4039 , piece_id=MZRBH2VHR7CCJK4OCCXINOQTNBP7PUP236H7DB2DFL7ZFDBNI2IQ, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.505278141 (375h37m59.544851301s) [noise=true , address=145.79.227.33:4881 , piece_id=AIKNNVT5PMYFYKRMUFN5DQMY7TWEUN3FUC64IQQYKAMV723TKKZA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.505743808 (375h37m59.545316948s) [noise=true , address=145.79.227.46:4601 , piece_id=EVWU7M2P2W4FQCNDEQ2OO3SR4KHPTXQRM7GEI6Q6DHYMDHK4PP4Q, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.505904454 (375h37m59.545477584s) [noise=true , address=145.79.227.24:4594 , piece_id=V55Y7DFR7QJWBBQRKSTUWYNEZSCAIWAY2XHAOZOF2H7JDFSX6IBA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.506859502 (375h37m59.546432622s) [noise=true , address=145.79.227.10:4420 , piece_id=ANGUQNIL2D4K2YR7KVJMY37WW7ILRVXSZOB7FRRAKJPD3AGGPGOA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.506958631 (375h37m59.546531761s) [noise=true , address=145.79.227.16:4933 , piece_id=WOBRP7MRD6VVMMXEHKNL4XDSPKQBHVINQD3I4N2SQU2MOSC2ZUCQ, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.507277077 (375h37m59.546850217s) [noise=true , address=145.79.227.7:4308 , piece_id=QEUKJMQDWYMBVKSVNNVDK53P6CZ4X47IINRSWCHAFG7APSFOOAFQ, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.508336856 (375h37m59.547909986s) [noise=true , address=77.90.31.19:4847 , piece_id=6GLTB4OVJTM67VRKRWATCIWE7WVK35IQMQIVOEZMZM6I5DFNRIBA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.513150782 (375h37m59.552723912s) [noise=true , address=145.79.227.12:4770 , piece_id=X5KVN5TA67L4TTD7LVG6N4I54BKNESFAHXRPVIWGDFCEPSHRKXPA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.513934875 (375h37m59.553508006s) [noise=true , address=45.91.116.14:4214 , piece_id=DOY7FC4SJKGAFP3LHSJXKWCRSYI6I6IR2R2FZ47QJNDM4SCAUXQQ, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.514566187 (375h37m59.554139318s) [noise=true , address=145.79.227.16:4939 , piece_id=DLS7J35PB7NXNMAPTVHBZ3VN3U725CSVZZKEYY37HZWQPWCCJO7A, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.515189424 (375h37m59.554762544s) [noise=true , address=145.79.227.25:4637 , piece_id=3SMC4AHCMKJEDXJOJVZ3FJEKW23VZDSWW6MDCENAYJ7HKOPWNHBQ, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.515805377 (375h37m59.555378508s) [noise=true , address=145.79.227.6:4292 , piece_id=BYUJPZSYFTYD4C4F5EBAW6BSFXTILW6EASOIRHZQI7ARCSG63F2Q, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.516494880 (375h37m59.55606801s) [noise=true , address=77.90.31.16:4764 , piece_id=PSAP3HPVNEURWWNC7K2K2G4NEUENKWDUR7OJY6UBGRTCSBYNHRWQ, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.526689722 (375h37m59.566262863s) [noise=true , address=145.79.227.31:4818 , piece_id=KAJZKTKOHNN5U4CTYJ7CLO32ONU6VSEJ2QBLPBI5WWZZ37YOI5NA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.531527154 (375h37m59.571100294s) [noise=true , address=145.79.227.49:4338 , piece_id=NBR62GETN7QBXC7QJCUB5DZHKVMC5AKQQBIIGDRXNECNZFULEDDA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.537921861 (375h37m59.577494991s) [noise=true , address=145.79.227.5:4247 , piece_id=QGN2S5CET2J6LGW6YPQYDWEVOGEMHWRSRMJG2IUQ36KA23OG2R3A, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:24.564378058 (375h38m0.603951188s) [noise=true , address=145.79.227.5:4247 , piece_id=QGN2S5CET2J6LGW6YPQYDWEVOGEMHWRSRMJG2IUQ36KA23OG2R3A, seg_pos={0 0}, upload=330171]: Done uploading piece. err: ecclient: upload failed (node:1262xxX6aUPTYgKdvRxAyc77PmPBWJtq5cXRJFoELxHbfn18DLE, address:145.79.227.5:4247): protocol: rpc: dial tcp 145.79.227.5:4247: connect: no route to host; conn closed: op Sep 22 05:06:24.986206552 (375h38m1.025779682s) [upload=330171]: Next returned nil segment Sep 22 05:06:24.986226830 (375h38m1.02579995s) [upload=330171]: Waiting for error group managing segments... Sep 22 05:06:25.383578176 (375h38m1.423151306s) [noise=true , address=145.79.227.24:4594 , piece_id=V55Y7DFR7QJWBBQRKSTUWYNEZSCAIWAY2XHAOZOF2H7JDFSX6IBA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.383806571 (375h38m1.423379702s) [noise=true , address=145.79.227.33:4881 , piece_id=AIKNNVT5PMYFYKRMUFN5DQMY7TWEUN3FUC64IQQYKAMV723TKKZA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.398811493 (375h38m1.438384634s) [noise=true , address=145.79.227.46:4601 , piece_id=EVWU7M2P2W4FQCNDEQ2OO3SR4KHPTXQRM7GEI6Q6DHYMDHK4PP4Q, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.399530783 (375h38m1.439103913s) [noise=true , address=145.79.227.39:4039 , piece_id=MZRBH2VHR7CCJK4OCCXINOQTNBP7PUP236H7DB2DFL7ZFDBNI2IQ, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.406508421 (375h38m1.446081551s) [noise=true , address=145.79.227.49:4338 , piece_id=NBR62GETN7QBXC7QJCUB5DZHKVMC5AKQQBIIGDRXNECNZFULEDDA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.415736332 (375h38m1.455309462s) [noise=true , address=145.79.227.25:4637 , piece_id=3SMC4AHCMKJEDXJOJVZ3FJEKW23VZDSWW6MDCENAYJ7HKOPWNHBQ, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.437405044 (375h38m1.476978184s) [noise=true , address=145.79.227.31:4818 , piece_id=KAJZKTKOHNN5U4CTYJ7CLO32ONU6VSEJ2QBLPBI5WWZZ37YOI5NA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.465513107 (375h38m1.505086237s) [noise=true , address=77.90.31.19:4847 , piece_id=6GLTB4OVJTM67VRKRWATCIWE7WVK35IQMQIVOEZMZM6I5DFNRIBA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.501349827 (375h38m1.540922957s) [noise=true , address=45.91.116.9:4292 , piece_id=3QGXKEK5C77XDD34Z77C3K2LYEMGJT2SADTL6RGW2YCYQFV5CLUA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.579007182 (375h38m1.618580302s) [noise=true , address=77.90.31.16:4764 , piece_id=PSAP3HPVNEURWWNC7K2K2G4NEUENKWDUR7OJY6UBGRTCSBYNHRWQ, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.586807715 (375h38m1.626380845s) [noise=true , address=45.91.116.14:4214 , piece_id=DOY7FC4SJKGAFP3LHSJXKWCRSYI6I6IR2R2FZ47QJNDM4SCAUXQQ, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:07:22.245417769 (375h38m58.284990899s) [noise=true , address=145.79.227.16:4933 , piece_id=WOBRP7MRD6VVMMXEHKNL4XDSPKQBHVINQD3I4N2SQU2MOSC2ZUCQ, seg_pos={0 0}, upload=330171]: Done uploading piece. err: ecclient: upload failed (node:19m4inaXvTdwf2rKip3eFELv6JAgFvXJ8KqjPZYbb48i8Ktnwe, address:145.79.227.16:4933): protocol: rpc: dial tcp 145.79.227.16:4933: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:07:22.245544310 (375h38m58.28511744s) [noise=true , address=145.79.227.12:4770 , piece_id=X5KVN5TA67L4TTD7LVG6N4I54BKNESFAHXRPVIWGDFCEPSHRKXPA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: ecclient: upload failed (node:1Ug6GpYToYDXiHmiJP4GoE7eJV3kqRveix2S5SLDcKQJtEbNPz, address:145.79.227.12:4770): protocol: rpc: dial tcp 145.79.227.12:4770: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:07:22.245567825 (375h38m58.285140955s) [noise=true , address=145.79.227.16:4939 , piece_id=DLS7J35PB7NXNMAPTVHBZ3VN3U725CSVZZKEYY37HZWQPWCCJO7A, seg_pos={0 0}, upload=330171]: Done uploading piece. err: ecclient: upload failed (node:1mRmp3HhD95PruS4AF8soRsSXNYUeb4mJE7y52FvoeN9D3Gu4F, address:145.79.227.16:4939): protocol: rpc: dial tcp 145.79.227.16:4939: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:07:22.245634181 (375h38m58.285207312s) [noise=true , address=145.79.227.7:4308 , piece_id=QEUKJMQDWYMBVKSVNNVDK53P6CZ4X47IINRSWCHAFG7APSFOOAFQ, seg_pos={0 0}, upload=330171]: Done uploading piece. err: ecclient: upload failed (node:1PgA5SZkBvK673LsV7p2sXpbi1r79esNNZspPZTzaitqfbjtyY, address:145.79.227.7:4308): protocol: rpc: dial tcp 145.79.227.7:4308: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:07:22.245657335 (375h38m58.285230455s) [noise=true , address=145.79.227.10:4420 , piece_id=ANGUQNIL2D4K2YR7KVJMY37WW7ILRVXSZOB7FRRAKJPD3AGGPGOA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: ecclient: upload failed (node:12hJrjeMUMYmFw3yxRtQQKRh9ciPfdu5FXo8NGrsc6HBqgfJzzp, address:145.79.227.10:4420): protocol: rpc: dial tcp 145.79.227.10:4420: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:07:22.245675269 (375h38m58.28524839s) [noise=true , address=145.79.227.6:4292 , piece_id=BYUJPZSYFTYD4C4F5EBAW6BSFXTILW6EASOIRHZQI7ARCSG63F2Q, seg_pos={0 0}, upload=330171]: Done uploading piece. err: ecclient: upload failed (node:12UpWmMhEmk5veKZvzsQ3MCxCMnfEKtFgCugBTsYfy9Ejm7G4xD, address:145.79.227.6:4292): protocol: rpc: dial tcp 145.79.227.6:4292: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:07:22.245712370 (375h38m58.2852855s) [seg_pos={0 0}, upload=330171]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Sep 22 05:07:22.500217377 (375h38m58.539790507s) [upload=330171]: Done uploading segments... Sep 22 05:07:22.500253296 (375h38m58.539826426s) [upload=330171]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 5.81556825s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [...X...X....................................................] duration: 5.95473608s FAIL (2025-09-22 05:46:24 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 5h37m59.543525066s) [seg_pos={0 0}, upload=330171]: Begin upload segment... Sep 22 05:06:23.504018422 (375h37m59.543591552s) [seg_pos={0 0}, upload=330171]: Done begin upload segment. Sep 22 05:06:23.504059370 (375h37m59.543632501s) [noise=true , address=45.91.116.9:4292 , piece_id=3QGXKEK5C77XDD34Z77C3K2LYEMGJT2SADTL6RGW2YCYQFV5CLUA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.504681175 (375h37m59.544254355s) [noise=true , address=145.79.227.39:4039 , piece_id=MZRBH2VHR7CCJK4OCCXINOQTNBP7PUP236H7DB2DFL7ZFDBNI2IQ, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.505278141 (375h37m59.544851301s) [noise=true , address=145.79.227.33:4881 , piece_id=AIKNNVT5PMYFYKRMUFN5DQMY7TWEUN3FUC64IQQYKAMV723TKKZA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.505743808 (375h37m59.545316948s) [noise=true , address=145.79.227.46:4601 , piece_id=EVWU7M2P2W4FQCNDEQ2OO3SR4KHPTXQRM7GEI6Q6DHYMDHK4PP4Q, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.505904454 (375h37m59.545477584s) [noise=true , address=145.79.227.24:4594 , piece_id=V55Y7DFR7QJWBBQRKSTUWYNEZSCAIWAY2XHAOZOF2H7JDFSX6IBA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.506859502 (375h37m59.546432622s) [noise=true , address=145.79.227.10:4420 , piece_id=ANGUQNIL2D4K2YR7KVJMY37WW7ILRVXSZOB7FRRAKJPD3AGGPGOA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.506958631 (375h37m59.546531761s) [noise=true , address=145.79.227.16:4933 , piece_id=WOBRP7MRD6VVMMXEHKNL4XDSPKQBHVINQD3I4N2SQU2MOSC2ZUCQ, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.507277077 (375h37m59.546850217s) [noise=true , address=145.79.227.7:4308 , piece_id=QEUKJMQDWYMBVKSVNNVDK53P6CZ4X47IINRSWCHAFG7APSFOOAFQ, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.508336856 (375h37m59.547909986s) [noise=true , address=77.90.31.19:4847 , piece_id=6GLTB4OVJTM67VRKRWATCIWE7WVK35IQMQIVOEZMZM6I5DFNRIBA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.513150782 (375h37m59.552723912s) [noise=true , address=145.79.227.12:4770 , piece_id=X5KVN5TA67L4TTD7LVG6N4I54BKNESFAHXRPVIWGDFCEPSHRKXPA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.513934875 (375h37m59.553508006s) [noise=true , address=45.91.116.14:4214 , piece_id=DOY7FC4SJKGAFP3LHSJXKWCRSYI6I6IR2R2FZ47QJNDM4SCAUXQQ, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.514566187 (375h37m59.554139318s) [noise=true , address=145.79.227.16:4939 , piece_id=DLS7J35PB7NXNMAPTVHBZ3VN3U725CSVZZKEYY37HZWQPWCCJO7A, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.515189424 (375h37m59.554762544s) [noise=true , address=145.79.227.25:4637 , piece_id=3SMC4AHCMKJEDXJOJVZ3FJEKW23VZDSWW6MDCENAYJ7HKOPWNHBQ, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.515805377 (375h37m59.555378508s) [noise=true , address=145.79.227.6:4292 , piece_id=BYUJPZSYFTYD4C4F5EBAW6BSFXTILW6EASOIRHZQI7ARCSG63F2Q, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.516494880 (375h37m59.55606801s) [noise=true , address=77.90.31.16:4764 , piece_id=PSAP3HPVNEURWWNC7K2K2G4NEUENKWDUR7OJY6UBGRTCSBYNHRWQ, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.526689722 (375h37m59.566262863s) [noise=true , address=145.79.227.31:4818 , piece_id=KAJZKTKOHNN5U4CTYJ7CLO32ONU6VSEJ2QBLPBI5WWZZ37YOI5NA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.531527154 (375h37m59.571100294s) [noise=true , address=145.79.227.49:4338 , piece_id=NBR62GETN7QBXC7QJCUB5DZHKVMC5AKQQBIIGDRXNECNZFULEDDA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.537921861 (375h37m59.577494991s) [noise=true , address=145.79.227.5:4247 , piece_id=QGN2S5CET2J6LGW6YPQYDWEVOGEMHWRSRMJG2IUQ36KA23OG2R3A, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:24.564378058 (375h38m0.603951188s) [noise=true , address=145.79.227.5:4247 , piece_id=QGN2S5CET2J6LGW6YPQYDWEVOGEMHWRSRMJG2IUQ36KA23OG2R3A, seg_pos={0 0}, upload=330171]: Done uploading piece. err: ecclient: upload failed (node:1262xxX6aUPTYgKdvRxAyc77PmPBWJtq5cXRJFoELxHbfn18DLE, address:145.79.227.5:4247): protocol: rpc: dial tcp 145.79.227.5:4247: connect: no route to host; conn closed: op Sep 22 05:06:24.986206552 (375h38m1.025779682s) [upload=330171]: Next returned nil segment Sep 22 05:06:24.986226830 (375h38m1.02579995s) [upload=330171]: Waiting for error group managing segments... Sep 22 05:06:25.383578176 (375h38m1.423151306s) [noise=true , address=145.79.227.24:4594 , piece_id=V55Y7DFR7QJWBBQRKSTUWYNEZSCAIWAY2XHAOZOF2H7JDFSX6IBA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.383806571 (375h38m1.423379702s) [noise=true , address=145.79.227.33:4881 , piece_id=AIKNNVT5PMYFYKRMUFN5DQMY7TWEUN3FUC64IQQYKAMV723TKKZA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.398811493 (375h38m1.438384634s) [noise=true , address=145.79.227.46:4601 , piece_id=EVWU7M2P2W4FQCNDEQ2OO3SR4KHPTXQRM7GEI6Q6DHYMDHK4PP4Q, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.399530783 (375h38m1.439103913s) [noise=true , address=145.79.227.39:4039 , piece_id=MZRBH2VHR7CCJK4OCCXINOQTNBP7PUP236H7DB2DFL7ZFDBNI2IQ, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.406508421 (375h38m1.446081551s) [noise=true , address=145.79.227.49:4338 , piece_id=NBR62GETN7QBXC7QJCUB5DZHKVMC5AKQQBIIGDRXNECNZFULEDDA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.415736332 (375h38m1.455309462s) [noise=true , address=145.79.227.25:4637 , piece_id=3SMC4AHCMKJEDXJOJVZ3FJEKW23VZDSWW6MDCENAYJ7HKOPWNHBQ, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.437405044 (375h38m1.476978184s) [noise=true , address=145.79.227.31:4818 , piece_id=KAJZKTKOHNN5U4CTYJ7CLO32ONU6VSEJ2QBLPBI5WWZZ37YOI5NA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.465513107 (375h38m1.505086237s) [noise=true , address=77.90.31.19:4847 , piece_id=6GLTB4OVJTM67VRKRWATCIWE7WVK35IQMQIVOEZMZM6I5DFNRIBA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.501349827 (375h38m1.540922957s) [noise=true , address=45.91.116.9:4292 , piece_id=3QGXKEK5C77XDD34Z77C3K2LYEMGJT2SADTL6RGW2YCYQFV5CLUA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.579007182 (375h38m1.618580302s) [noise=true , address=77.90.31.16:4764 , piece_id=PSAP3HPVNEURWWNC7K2K2G4NEUENKWDUR7OJY6UBGRTCSBYNHRWQ, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.586807715 (375h38m1.626380845s) [noise=true , address=45.91.116.14:4214 , piece_id=DOY7FC4SJKGAFP3LHSJXKWCRSYI6I6IR2R2FZ47QJNDM4SCAUXQQ, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:07:22.245417769 (375h38m58.284990899s) [noise=true , address=145.79.227.16:4933 , piece_id=WOBRP7MRD6VVMMXEHKNL4XDSPKQBHVINQD3I4N2SQU2MOSC2ZUCQ, seg_pos={0 0}, upload=330171]: Done uploading piece. err: ecclient: upload failed (node:19m4inaXvTdwf2rKip3eFELv6JAgFvXJ8KqjPZYbb48i8Ktnwe, address:145.79.227.16:4933): protocol: rpc: dial tcp 145.79.227.16:4933: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:07:22.245544310 (375h38m58.28511744s) [noise=true , address=145.79.227.12:4770 , piece_id=X5KVN5TA67L4TTD7LVG6N4I54BKNESFAHXRPVIWGDFCEPSHRKXPA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: ecclient: upload failed (node:1Ug6GpYToYDXiHmiJP4GoE7eJV3kqRveix2S5SLDcKQJtEbNPz, address:145.79.227.12:4770): protocol: rpc: dial tcp 145.79.227.12:4770: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:07:22.245567825 (375h38m58.285140955s) [noise=true , address=145.79.227.16:4939 , piece_id=DLS7J35PB7NXNMAPTVHBZ3VN3U725CSVZZKEYY37HZWQPWCCJO7A, seg_pos={0 0}, upload=330171]: Done uploading piece. err: ecclient: upload failed (node:1mRmp3HhD95PruS4AF8soRsSXNYUeb4mJE7y52FvoeN9D3Gu4F, address:145.79.227.16:4939): protocol: rpc: dial tcp 145.79.227.16:4939: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:07:22.245634181 (375h38m58.285207312s) [noise=true , address=145.79.227.7:4308 , piece_id=QEUKJMQDWYMBVKSVNNVDK53P6CZ4X47IINRSWCHAFG7APSFOOAFQ, seg_pos={0 0}, upload=330171]: Done uploading piece. err: ecclient: upload failed (node:1PgA5SZkBvK673LsV7p2sXpbi1r79esNNZspPZTzaitqfbjtyY, address:145.79.227.7:4308): protocol: rpc: dial tcp 145.79.227.7:4308: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:07:22.245657335 (375h38m58.285230455s) [noise=true , address=145.79.227.10:4420 , piece_id=ANGUQNIL2D4K2YR7KVJMY37WW7ILRVXSZOB7FRRAKJPD3AGGPGOA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: ecclient: upload failed (node:12hJrjeMUMYmFw3yxRtQQKRh9ciPfdu5FXo8NGrsc6HBqgfJzzp, address:145.79.227.10:4420): protocol: rpc: dial tcp 145.79.227.10:4420: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:07:22.245675269 (375h38m58.28524839s) [noise=true , address=145.79.227.6:4292 , piece_id=BYUJPZSYFTYD4C4F5EBAW6BSFXTILW6EASOIRHZQI7ARCSG63F2Q, seg_pos={0 0}, upload=330171]: Done uploading piece. err: ecclient: upload failed (node:12UpWmMhEmk5veKZvzsQ3MCxCMnfEKtFgCugBTsYfy9Ejm7G4xD, address:145.79.227.6:4292): protocol: rpc: dial tcp 145.79.227.6:4292: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:07:22.245712370 (375h38m58.2852855s) [seg_pos={0 0}, upload=330171]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Sep 22 05:07:22.500217377 (375h38m58.539790507s) [upload=330171]: Done uploading segments... Sep 22 05:07:22.500253296 (375h38m58.539826426s) [upload=330171]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 5.954769304s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [.XX....X..X...X.X......X.........X.X..X....X..XX............] duration: 6.361360311s FAIL (2025-09-22 05:46:19 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.361380519s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [XXXX.X.XX.....X.X....................X......................] duration: 5.929617062s FAIL (2025-09-22 05:46:18 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== 3.337157461s) [upload=330511]: Starting upload Sep 22 05:29:47.297668972 (376h1m23.337242102s) [upload=330511]: Done starting upload Sep 22 05:29:47.297899231 (376h1m23.337472361s) [upload=330511]: Uploading segments... Sep 22 05:29:47.298002678 (376h1m23.337575818s) [upload=330511]: Got next segment. Inline: false Sep 22 05:29:47.298059145 (376h1m23.337632275s) [upload=330511]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 05:29:47.298066109 (376h1m23.337639229s) [upload=330511]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 05:29:47.552911815 (376h1m23.592484945s) [seg_pos={0 0}, upload=330511]: Begin upload segment... Sep 22 05:29:47.552975406 (376h1m23.592548536s) [seg_pos={0 0}, upload=330511]: Done begin upload segment. Sep 22 05:29:47.553013599 (376h1m23.592586719s) [noise=true , address=45.91.116.11:4117 , piece_id=NIFPLA2N3L6XDWTIFUM3YYRGBMFQJKWFVCGYXD2GBFYVYDXW6OEQ, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.553528389 (376h1m23.593101539s) [noise=true , address=145.79.227.6:4258 , piece_id=D7HMSQH7ZLLKNTUZGJLVHIW2KEQZDL3HSS6VO2BSNRZJAC3USFAQ, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.553925034 (376h1m23.593498154s) [noise=true , address=45.91.116.9:4261 , piece_id=REZZWYP25KZOP4UBJWJPHGCDUWCW3EBOJG2DOLCDYZY5B43KBMSQ, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.553961283 (376h1m23.593534413s) [noise=true , address=77.90.31.14:4663 , piece_id=LYGO7UTA55Q5Q7APPDJZNNWZ4S2XHAT4LS4V27VF27A3435DVJQA, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.554326960 (376h1m23.59390009s) [noise=true , address=77.90.31.16:4735 , piece_id=K5KYKKY3BQV6UVJK4G3DYC24DFFIEFQX7KGGFLS4P35KJRFHA6UQ, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.554351497 (376h1m23.593924627s) [noise=true , address=145.79.227.16:4912 , piece_id=AG2FUP2KFIJ7W55H2VRYTJYWQ57MAXCTEMTDWSR4FUJAYE3HFKBQ, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.554588828 (376h1m23.594161948s) [noise=true , address=145.79.227.12:4767 , piece_id=OUVFSM62B7UZ4U4WGFFBOHCPHXEXR6A6H2FQL3DTG5L3DEPUVYVQ, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.554724957 (376h1m23.594298087s) [noise=true , address=77.90.31.18:4807 , piece_id=JFOMLF75MIXEDP5CKDHH7J3PWGWESB7BIKK3SCAIOVNX74DJDC3A, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.554875986 (376h1m23.594449116s) [noise=true , address=145.79.227.3:4151 , piece_id=O6AU3AOBKJRIUMEAWPHMWWDDNRTA52WL72JNUTK5JFLVG2TNV5IA, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.555093950 (376h1m23.59466708s) [noise=true , address=145.79.227.17:4972 , piece_id=Q3KQQ57XWDUCXNHP33STZ7DN6VQ6FVYZTDPLINEPZASFEVY5XDJA, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.555135429 (376h1m23.594708559s) [noise=true , address=45.91.116.6:4973 , piece_id=WPWUFOFQHQZFRPTR33GTDST2NYCVPXKLDVCJQFPQ4A7PI44LJKOQ, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.555464797 (376h1m23.595037917s) [noise=true , address=45.91.116.14:4189 , piece_id=6VJ6XCCEV2CZSARNRF3ATZ4KCUI75QKWIWQWJL5QH4XTQG4MCLUQ, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.555726194 (376h1m23.595299314s) [noise=true , address=77.90.31.15:4699 , piece_id=6XZN64DOILMELBYDU4D72ER33XDKMRIJRSAXX5ITFYPG46YEBITQ, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.556009483 (376h1m23.595582614s) [noise=true , address=145.79.227.19:4077 , piece_id=EDBYIPZUOPFRRIZBZJJEP2U7NZFCUHRLDZESZ7J5CQZFKADNLE6Q, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.556364609 (376h1m23.59593773s) [noise=true , address=145.79.227.15:4875 , piece_id=ZEWO52263YKTTRYDVF4PV52JTPEDJR3XGVRG3BFE5QQKDQSCE2LQ, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.556611299 (376h1m23.596184429s) [noise=true , address=145.79.227.5:4222 , piece_id=CZNVNWHETTFQS6JR2YLE65LNQU4WX74DAROY5I3KAYHG4QDLDOKA, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:49.308510933 (376h1m25.348084063s) [upload=330511]: Next returned nil segment Sep 22 05:29:49.308533666 (376h1m25.348106796s) [upload=330511]: Waiting for error group managing segments... Sep 22 05:29:49.602590138 (376h1m25.642163268s) [noise=true , address=45.91.116.11:4117 , piece_id=NIFPLA2N3L6XDWTIFUM3YYRGBMFQJKWFVCGYXD2GBFYVYDXW6OEQ, seg_pos={0 0}, upload=330511]: Done uploading piece. err: <nil> Sep 22 05:29:49.633269677 (376h1m25.672842807s) [noise=true , address=77.90.31.15:4699 , piece_id=6XZN64DOILMELBYDU4D72ER33XDKMRIJRSAXX5ITFYPG46YEBITQ, seg_pos={0 0}, upload=330511]: Done uploading piece. err: <nil> Sep 22 05:29:49.637382749 (376h1m25.676955879s) [noise=true , address=45.91.116.6:4973 , piece_id=WPWUFOFQHQZFRPTR33GTDST2NYCVPXKLDVCJQFPQ4A7PI44LJKOQ, seg_pos={0 0}, upload=330511]: Done uploading piece. err: <nil> Sep 22 05:29:49.648543782 (376h1m25.688116902s) [noise=true , address=77.90.31.16:4735 , piece_id=K5KYKKY3BQV6UVJK4G3DYC24DFFIEFQX7KGGFLS4P35KJRFHA6UQ, seg_pos={0 0}, upload=330511]: Done uploading piece. err: <nil> Sep 22 05:29:49.653934647 (376h1m25.693507767s) [noise=true , address=45.91.116.14:4189 , piece_id=6VJ6XCCEV2CZSARNRF3ATZ4KCUI75QKWIWQWJL5QH4XTQG4MCLUQ, seg_pos={0 0}, upload=330511]: Done uploading piece. err: <nil> Sep 22 05:29:49.669096699 (376h1m25.708669829s) [noise=true , address=45.91.116.9:4261 , piece_id=REZZWYP25KZOP4UBJWJPHGCDUWCW3EBOJG2DOLCDYZY5B43KBMSQ, seg_pos={0 0}, upload=330511]: Done uploading piece. err: <nil> Sep 22 05:29:49.673672402 (376h1m25.713245532s) [noise=true , address=77.90.31.14:4663 , piece_id=LYGO7UTA55Q5Q7APPDJZNNWZ4S2XHAT4LS4V27VF27A3435DVJQA, seg_pos={0 0}, upload=330511]: Done uploading piece. err: <nil> Sep 22 05:29:49.741841689 (376h1m25.781414809s) [noise=true , address=77.90.31.18:4807 , piece_id=JFOMLF75MIXEDP5CKDHH7J3PWGWESB7BIKK3SCAIOVNX74DJDC3A, seg_pos={0 0}, upload=330511]: Done uploading piece. err: <nil> Sep 22 05:29:58.021015047 (376h1m34.060588167s) [noise=true , address=145.79.227.5:4222 , piece_id=CZNVNWHETTFQS6JR2YLE65LNQU4WX74DAROY5I3KAYHG4QDLDOKA, seg_pos={0 0}, upload=330511]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:145.79.227.5:4222): protocol: rpc: dial tcp 145.79.227.5:4222: connect: no route to host; conn closed: op; piecestore: piecestore close: manager closed: conn closed: op Sep 22 05:30:05.028605724 (376h1m41.068178854s) [noise=true , address=145.79.227.3:4151 , piece_id=O6AU3AOBKJRIUMEAWPHMWWDDNRTA52WL72JNUTK5JFLVG2TNV5IA, seg_pos={0 0}, upload=330511]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:145.79.227.3:4151): protocol: rpc: dial tcp 145.79.227.3:4151: connect: no route to host; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:30:16.586571260 (376h1m52.62614439s) [noise=true , address=145.79.227.12:4767 , piece_id=OUVFSM62B7UZ4U4WGFFBOHCPHXEXR6A6H2FQL3DTG5L3DEPUVYVQ, seg_pos={0 0}, upload=330511]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:145.79.227.12:4767): protocol: rpc: dial tcp 145.79.227.12:4767: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:30:16.586679978 (376h1m52.626253118s) [noise=true , address=145.79.227.6:4258 , piece_id=D7HMSQH7ZLLKNTUZGJLVHIW2KEQZDL3HSS6VO2BSNRZJAC3USFAQ, seg_pos={0 0}, upload=330511]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:145.79.227.6:4258): protocol: rpc: dial tcp 145.79.227.6:4258: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:30:16.586708171 (376h1m52.626281301s) [noise=true , address=145.79.227.16:4912 , piece_id=AG2FUP2KFIJ7W55H2VRYTJYWQ57MAXCTEMTDWSR4FUJAYE3HFKBQ, seg_pos={0 0}, upload=330511]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:145.79.227.16:4912): protocol: rpc: dial tcp 145.79.227.16:4912: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:30:16.586749971 (376h1m52.626323101s) [noise=true , address=145.79.227.15:4875 , piece_id=ZEWO52263YKTTRYDVF4PV52JTPEDJR3XGVRG3BFE5QQKDQSCE2LQ, seg_pos={0 0}, upload=330511]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: rpc: dial tcp 145.79.227.15:4875: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:30:16.586795768 (376h1m52.626368888s) [noise=true , address=145.79.227.17:4972 , piece_id=Q3KQQ57XWDUCXNHP33STZ7DN6VQ6FVYZTDPLINEPZASFEVY5XDJA, seg_pos={0 0}, upload=330511]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:145.79.227.17:4972): protocol: rpc: dial tcp 145.79.227.17:4972: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:30:16.586806088 (376h1m52.626379218s) [noise=true , address=145.79.227.19:4077 , piece_id=EDBYIPZUOPFRRIZBZJJEP2U7NZFCUHRLDZESZ7J5CQZFKADNLE6Q, seg_pos={0 0}, upload=330511]: Done uploading piece. err: ecclient: upload failed (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk, address:145.79.227.19:4077): protocol: rpc: dial tcp 145.79.227.19:4077: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:30:16.586831426 (376h1m52.626404556s) [seg_pos={0 0}, upload=330511]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Sep 22 05:30:16.834944071 (376h1m52.874517201s) [upload=330511]: Done uploading segments... Sep 22 05:30:16.834981573 (376h1m52.874554703s) [upload=330511]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 5.929653912s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [....X...X...................................................] duration: 6.19718797s FAIL (2025-09-22 05:46:01 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 5h37m59.543525066s) [seg_pos={0 0}, upload=330171]: Begin upload segment... Sep 22 05:06:23.504018422 (375h37m59.543591552s) [seg_pos={0 0}, upload=330171]: Done begin upload segment. Sep 22 05:06:23.504059370 (375h37m59.543632501s) [noise=true , address=45.91.116.9:4292 , piece_id=3QGXKEK5C77XDD34Z77C3K2LYEMGJT2SADTL6RGW2YCYQFV5CLUA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.504681175 (375h37m59.544254355s) [noise=true , address=145.79.227.39:4039 , piece_id=MZRBH2VHR7CCJK4OCCXINOQTNBP7PUP236H7DB2DFL7ZFDBNI2IQ, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.505278141 (375h37m59.544851301s) [noise=true , address=145.79.227.33:4881 , piece_id=AIKNNVT5PMYFYKRMUFN5DQMY7TWEUN3FUC64IQQYKAMV723TKKZA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.505743808 (375h37m59.545316948s) [noise=true , address=145.79.227.46:4601 , piece_id=EVWU7M2P2W4FQCNDEQ2OO3SR4KHPTXQRM7GEI6Q6DHYMDHK4PP4Q, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.505904454 (375h37m59.545477584s) [noise=true , address=145.79.227.24:4594 , piece_id=V55Y7DFR7QJWBBQRKSTUWYNEZSCAIWAY2XHAOZOF2H7JDFSX6IBA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.506859502 (375h37m59.546432622s) [noise=true , address=145.79.227.10:4420 , piece_id=ANGUQNIL2D4K2YR7KVJMY37WW7ILRVXSZOB7FRRAKJPD3AGGPGOA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.506958631 (375h37m59.546531761s) [noise=true , address=145.79.227.16:4933 , piece_id=WOBRP7MRD6VVMMXEHKNL4XDSPKQBHVINQD3I4N2SQU2MOSC2ZUCQ, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.507277077 (375h37m59.546850217s) [noise=true , address=145.79.227.7:4308 , piece_id=QEUKJMQDWYMBVKSVNNVDK53P6CZ4X47IINRSWCHAFG7APSFOOAFQ, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.508336856 (375h37m59.547909986s) [noise=true , address=77.90.31.19:4847 , piece_id=6GLTB4OVJTM67VRKRWATCIWE7WVK35IQMQIVOEZMZM6I5DFNRIBA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.513150782 (375h37m59.552723912s) [noise=true , address=145.79.227.12:4770 , piece_id=X5KVN5TA67L4TTD7LVG6N4I54BKNESFAHXRPVIWGDFCEPSHRKXPA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.513934875 (375h37m59.553508006s) [noise=true , address=45.91.116.14:4214 , piece_id=DOY7FC4SJKGAFP3LHSJXKWCRSYI6I6IR2R2FZ47QJNDM4SCAUXQQ, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.514566187 (375h37m59.554139318s) [noise=true , address=145.79.227.16:4939 , piece_id=DLS7J35PB7NXNMAPTVHBZ3VN3U725CSVZZKEYY37HZWQPWCCJO7A, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.515189424 (375h37m59.554762544s) [noise=true , address=145.79.227.25:4637 , piece_id=3SMC4AHCMKJEDXJOJVZ3FJEKW23VZDSWW6MDCENAYJ7HKOPWNHBQ, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.515805377 (375h37m59.555378508s) [noise=true , address=145.79.227.6:4292 , piece_id=BYUJPZSYFTYD4C4F5EBAW6BSFXTILW6EASOIRHZQI7ARCSG63F2Q, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.516494880 (375h37m59.55606801s) [noise=true , address=77.90.31.16:4764 , piece_id=PSAP3HPVNEURWWNC7K2K2G4NEUENKWDUR7OJY6UBGRTCSBYNHRWQ, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.526689722 (375h37m59.566262863s) [noise=true , address=145.79.227.31:4818 , piece_id=KAJZKTKOHNN5U4CTYJ7CLO32ONU6VSEJ2QBLPBI5WWZZ37YOI5NA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.531527154 (375h37m59.571100294s) [noise=true , address=145.79.227.49:4338 , piece_id=NBR62GETN7QBXC7QJCUB5DZHKVMC5AKQQBIIGDRXNECNZFULEDDA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.537921861 (375h37m59.577494991s) [noise=true , address=145.79.227.5:4247 , piece_id=QGN2S5CET2J6LGW6YPQYDWEVOGEMHWRSRMJG2IUQ36KA23OG2R3A, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:24.564378058 (375h38m0.603951188s) [noise=true , address=145.79.227.5:4247 , piece_id=QGN2S5CET2J6LGW6YPQYDWEVOGEMHWRSRMJG2IUQ36KA23OG2R3A, seg_pos={0 0}, upload=330171]: Done uploading piece. err: ecclient: upload failed (node:1262xxX6aUPTYgKdvRxAyc77PmPBWJtq5cXRJFoELxHbfn18DLE, address:145.79.227.5:4247): protocol: rpc: dial tcp 145.79.227.5:4247: connect: no route to host; conn closed: op Sep 22 05:06:24.986206552 (375h38m1.025779682s) [upload=330171]: Next returned nil segment Sep 22 05:06:24.986226830 (375h38m1.02579995s) [upload=330171]: Waiting for error group managing segments... Sep 22 05:06:25.383578176 (375h38m1.423151306s) [noise=true , address=145.79.227.24:4594 , piece_id=V55Y7DFR7QJWBBQRKSTUWYNEZSCAIWAY2XHAOZOF2H7JDFSX6IBA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.383806571 (375h38m1.423379702s) [noise=true , address=145.79.227.33:4881 , piece_id=AIKNNVT5PMYFYKRMUFN5DQMY7TWEUN3FUC64IQQYKAMV723TKKZA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.398811493 (375h38m1.438384634s) [noise=true , address=145.79.227.46:4601 , piece_id=EVWU7M2P2W4FQCNDEQ2OO3SR4KHPTXQRM7GEI6Q6DHYMDHK4PP4Q, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.399530783 (375h38m1.439103913s) [noise=true , address=145.79.227.39:4039 , piece_id=MZRBH2VHR7CCJK4OCCXINOQTNBP7PUP236H7DB2DFL7ZFDBNI2IQ, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.406508421 (375h38m1.446081551s) [noise=true , address=145.79.227.49:4338 , piece_id=NBR62GETN7QBXC7QJCUB5DZHKVMC5AKQQBIIGDRXNECNZFULEDDA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.415736332 (375h38m1.455309462s) [noise=true , address=145.79.227.25:4637 , piece_id=3SMC4AHCMKJEDXJOJVZ3FJEKW23VZDSWW6MDCENAYJ7HKOPWNHBQ, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.437405044 (375h38m1.476978184s) [noise=true , address=145.79.227.31:4818 , piece_id=KAJZKTKOHNN5U4CTYJ7CLO32ONU6VSEJ2QBLPBI5WWZZ37YOI5NA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.465513107 (375h38m1.505086237s) [noise=true , address=77.90.31.19:4847 , piece_id=6GLTB4OVJTM67VRKRWATCIWE7WVK35IQMQIVOEZMZM6I5DFNRIBA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.501349827 (375h38m1.540922957s) [noise=true , address=45.91.116.9:4292 , piece_id=3QGXKEK5C77XDD34Z77C3K2LYEMGJT2SADTL6RGW2YCYQFV5CLUA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.579007182 (375h38m1.618580302s) [noise=true , address=77.90.31.16:4764 , piece_id=PSAP3HPVNEURWWNC7K2K2G4NEUENKWDUR7OJY6UBGRTCSBYNHRWQ, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.586807715 (375h38m1.626380845s) [noise=true , address=45.91.116.14:4214 , piece_id=DOY7FC4SJKGAFP3LHSJXKWCRSYI6I6IR2R2FZ47QJNDM4SCAUXQQ, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:07:22.245417769 (375h38m58.284990899s) [noise=true , address=145.79.227.16:4933 , piece_id=WOBRP7MRD6VVMMXEHKNL4XDSPKQBHVINQD3I4N2SQU2MOSC2ZUCQ, seg_pos={0 0}, upload=330171]: Done uploading piece. err: ecclient: upload failed (node:19m4inaXvTdwf2rKip3eFELv6JAgFvXJ8KqjPZYbb48i8Ktnwe, address:145.79.227.16:4933): protocol: rpc: dial tcp 145.79.227.16:4933: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:07:22.245544310 (375h38m58.28511744s) [noise=true , address=145.79.227.12:4770 , piece_id=X5KVN5TA67L4TTD7LVG6N4I54BKNESFAHXRPVIWGDFCEPSHRKXPA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: ecclient: upload failed (node:1Ug6GpYToYDXiHmiJP4GoE7eJV3kqRveix2S5SLDcKQJtEbNPz, address:145.79.227.12:4770): protocol: rpc: dial tcp 145.79.227.12:4770: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:07:22.245567825 (375h38m58.285140955s) [noise=true , address=145.79.227.16:4939 , piece_id=DLS7J35PB7NXNMAPTVHBZ3VN3U725CSVZZKEYY37HZWQPWCCJO7A, seg_pos={0 0}, upload=330171]: Done uploading piece. err: ecclient: upload failed (node:1mRmp3HhD95PruS4AF8soRsSXNYUeb4mJE7y52FvoeN9D3Gu4F, address:145.79.227.16:4939): protocol: rpc: dial tcp 145.79.227.16:4939: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:07:22.245634181 (375h38m58.285207312s) [noise=true , address=145.79.227.7:4308 , piece_id=QEUKJMQDWYMBVKSVNNVDK53P6CZ4X47IINRSWCHAFG7APSFOOAFQ, seg_pos={0 0}, upload=330171]: Done uploading piece. err: ecclient: upload failed (node:1PgA5SZkBvK673LsV7p2sXpbi1r79esNNZspPZTzaitqfbjtyY, address:145.79.227.7:4308): protocol: rpc: dial tcp 145.79.227.7:4308: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:07:22.245657335 (375h38m58.285230455s) [noise=true , address=145.79.227.10:4420 , piece_id=ANGUQNIL2D4K2YR7KVJMY37WW7ILRVXSZOB7FRRAKJPD3AGGPGOA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: ecclient: upload failed (node:12hJrjeMUMYmFw3yxRtQQKRh9ciPfdu5FXo8NGrsc6HBqgfJzzp, address:145.79.227.10:4420): protocol: rpc: dial tcp 145.79.227.10:4420: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:07:22.245675269 (375h38m58.28524839s) [noise=true , address=145.79.227.6:4292 , piece_id=BYUJPZSYFTYD4C4F5EBAW6BSFXTILW6EASOIRHZQI7ARCSG63F2Q, seg_pos={0 0}, upload=330171]: Done uploading piece. err: ecclient: upload failed (node:12UpWmMhEmk5veKZvzsQ3MCxCMnfEKtFgCugBTsYfy9Ejm7G4xD, address:145.79.227.6:4292): protocol: rpc: dial tcp 145.79.227.6:4292: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:07:22.245712370 (375h38m58.2852855s) [seg_pos={0 0}, upload=330171]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Sep 22 05:07:22.500217377 (375h38m58.539790507s) [upload=330171]: Done uploading segments... Sep 22 05:07:22.500253296 (375h38m58.539826426s) [upload=330171]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 6.197223127s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [..XX....X..X...X.X......X.........X.X..X....X..XX...........] duration: 6.24333805s FAIL (2025-09-22 05:45:42 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.243376021s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [.XXXX.X.XX.....X.X....................X.....................] duration: 5.9592815s FAIL (2025-09-22 05:45:23 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== 3.337157461s) [upload=330511]: Starting upload Sep 22 05:29:47.297668972 (376h1m23.337242102s) [upload=330511]: Done starting upload Sep 22 05:29:47.297899231 (376h1m23.337472361s) [upload=330511]: Uploading segments... Sep 22 05:29:47.298002678 (376h1m23.337575818s) [upload=330511]: Got next segment. Inline: false Sep 22 05:29:47.298059145 (376h1m23.337632275s) [upload=330511]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 05:29:47.298066109 (376h1m23.337639229s) [upload=330511]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 05:29:47.552911815 (376h1m23.592484945s) [seg_pos={0 0}, upload=330511]: Begin upload segment... Sep 22 05:29:47.552975406 (376h1m23.592548536s) [seg_pos={0 0}, upload=330511]: Done begin upload segment. Sep 22 05:29:47.553013599 (376h1m23.592586719s) [noise=true , address=45.91.116.11:4117 , piece_id=NIFPLA2N3L6XDWTIFUM3YYRGBMFQJKWFVCGYXD2GBFYVYDXW6OEQ, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.553528389 (376h1m23.593101539s) [noise=true , address=145.79.227.6:4258 , piece_id=D7HMSQH7ZLLKNTUZGJLVHIW2KEQZDL3HSS6VO2BSNRZJAC3USFAQ, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.553925034 (376h1m23.593498154s) [noise=true , address=45.91.116.9:4261 , piece_id=REZZWYP25KZOP4UBJWJPHGCDUWCW3EBOJG2DOLCDYZY5B43KBMSQ, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.553961283 (376h1m23.593534413s) [noise=true , address=77.90.31.14:4663 , piece_id=LYGO7UTA55Q5Q7APPDJZNNWZ4S2XHAT4LS4V27VF27A3435DVJQA, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.554326960 (376h1m23.59390009s) [noise=true , address=77.90.31.16:4735 , piece_id=K5KYKKY3BQV6UVJK4G3DYC24DFFIEFQX7KGGFLS4P35KJRFHA6UQ, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.554351497 (376h1m23.593924627s) [noise=true , address=145.79.227.16:4912 , piece_id=AG2FUP2KFIJ7W55H2VRYTJYWQ57MAXCTEMTDWSR4FUJAYE3HFKBQ, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.554588828 (376h1m23.594161948s) [noise=true , address=145.79.227.12:4767 , piece_id=OUVFSM62B7UZ4U4WGFFBOHCPHXEXR6A6H2FQL3DTG5L3DEPUVYVQ, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.554724957 (376h1m23.594298087s) [noise=true , address=77.90.31.18:4807 , piece_id=JFOMLF75MIXEDP5CKDHH7J3PWGWESB7BIKK3SCAIOVNX74DJDC3A, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.554875986 (376h1m23.594449116s) [noise=true , address=145.79.227.3:4151 , piece_id=O6AU3AOBKJRIUMEAWPHMWWDDNRTA52WL72JNUTK5JFLVG2TNV5IA, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.555093950 (376h1m23.59466708s) [noise=true , address=145.79.227.17:4972 , piece_id=Q3KQQ57XWDUCXNHP33STZ7DN6VQ6FVYZTDPLINEPZASFEVY5XDJA, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.555135429 (376h1m23.594708559s) [noise=true , address=45.91.116.6:4973 , piece_id=WPWUFOFQHQZFRPTR33GTDST2NYCVPXKLDVCJQFPQ4A7PI44LJKOQ, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.555464797 (376h1m23.595037917s) [noise=true , address=45.91.116.14:4189 , piece_id=6VJ6XCCEV2CZSARNRF3ATZ4KCUI75QKWIWQWJL5QH4XTQG4MCLUQ, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.555726194 (376h1m23.595299314s) [noise=true , address=77.90.31.15:4699 , piece_id=6XZN64DOILMELBYDU4D72ER33XDKMRIJRSAXX5ITFYPG46YEBITQ, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.556009483 (376h1m23.595582614s) [noise=true , address=145.79.227.19:4077 , piece_id=EDBYIPZUOPFRRIZBZJJEP2U7NZFCUHRLDZESZ7J5CQZFKADNLE6Q, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.556364609 (376h1m23.59593773s) [noise=true , address=145.79.227.15:4875 , piece_id=ZEWO52263YKTTRYDVF4PV52JTPEDJR3XGVRG3BFE5QQKDQSCE2LQ, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.556611299 (376h1m23.596184429s) [noise=true , address=145.79.227.5:4222 , piece_id=CZNVNWHETTFQS6JR2YLE65LNQU4WX74DAROY5I3KAYHG4QDLDOKA, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:49.308510933 (376h1m25.348084063s) [upload=330511]: Next returned nil segment Sep 22 05:29:49.308533666 (376h1m25.348106796s) [upload=330511]: Waiting for error group managing segments... Sep 22 05:29:49.602590138 (376h1m25.642163268s) [noise=true , address=45.91.116.11:4117 , piece_id=NIFPLA2N3L6XDWTIFUM3YYRGBMFQJKWFVCGYXD2GBFYVYDXW6OEQ, seg_pos={0 0}, upload=330511]: Done uploading piece. err: <nil> Sep 22 05:29:49.633269677 (376h1m25.672842807s) [noise=true , address=77.90.31.15:4699 , piece_id=6XZN64DOILMELBYDU4D72ER33XDKMRIJRSAXX5ITFYPG46YEBITQ, seg_pos={0 0}, upload=330511]: Done uploading piece. err: <nil> Sep 22 05:29:49.637382749 (376h1m25.676955879s) [noise=true , address=45.91.116.6:4973 , piece_id=WPWUFOFQHQZFRPTR33GTDST2NYCVPXKLDVCJQFPQ4A7PI44LJKOQ, seg_pos={0 0}, upload=330511]: Done uploading piece. err: <nil> Sep 22 05:29:49.648543782 (376h1m25.688116902s) [noise=true , address=77.90.31.16:4735 , piece_id=K5KYKKY3BQV6UVJK4G3DYC24DFFIEFQX7KGGFLS4P35KJRFHA6UQ, seg_pos={0 0}, upload=330511]: Done uploading piece. err: <nil> Sep 22 05:29:49.653934647 (376h1m25.693507767s) [noise=true , address=45.91.116.14:4189 , piece_id=6VJ6XCCEV2CZSARNRF3ATZ4KCUI75QKWIWQWJL5QH4XTQG4MCLUQ, seg_pos={0 0}, upload=330511]: Done uploading piece. err: <nil> Sep 22 05:29:49.669096699 (376h1m25.708669829s) [noise=true , address=45.91.116.9:4261 , piece_id=REZZWYP25KZOP4UBJWJPHGCDUWCW3EBOJG2DOLCDYZY5B43KBMSQ, seg_pos={0 0}, upload=330511]: Done uploading piece. err: <nil> Sep 22 05:29:49.673672402 (376h1m25.713245532s) [noise=true , address=77.90.31.14:4663 , piece_id=LYGO7UTA55Q5Q7APPDJZNNWZ4S2XHAT4LS4V27VF27A3435DVJQA, seg_pos={0 0}, upload=330511]: Done uploading piece. err: <nil> Sep 22 05:29:49.741841689 (376h1m25.781414809s) [noise=true , address=77.90.31.18:4807 , piece_id=JFOMLF75MIXEDP5CKDHH7J3PWGWESB7BIKK3SCAIOVNX74DJDC3A, seg_pos={0 0}, upload=330511]: Done uploading piece. err: <nil> Sep 22 05:29:58.021015047 (376h1m34.060588167s) [noise=true , address=145.79.227.5:4222 , piece_id=CZNVNWHETTFQS6JR2YLE65LNQU4WX74DAROY5I3KAYHG4QDLDOKA, seg_pos={0 0}, upload=330511]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:145.79.227.5:4222): protocol: rpc: dial tcp 145.79.227.5:4222: connect: no route to host; conn closed: op; piecestore: piecestore close: manager closed: conn closed: op Sep 22 05:30:05.028605724 (376h1m41.068178854s) [noise=true , address=145.79.227.3:4151 , piece_id=O6AU3AOBKJRIUMEAWPHMWWDDNRTA52WL72JNUTK5JFLVG2TNV5IA, seg_pos={0 0}, upload=330511]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:145.79.227.3:4151): protocol: rpc: dial tcp 145.79.227.3:4151: connect: no route to host; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:30:16.586571260 (376h1m52.62614439s) [noise=true , address=145.79.227.12:4767 , piece_id=OUVFSM62B7UZ4U4WGFFBOHCPHXEXR6A6H2FQL3DTG5L3DEPUVYVQ, seg_pos={0 0}, upload=330511]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:145.79.227.12:4767): protocol: rpc: dial tcp 145.79.227.12:4767: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:30:16.586679978 (376h1m52.626253118s) [noise=true , address=145.79.227.6:4258 , piece_id=D7HMSQH7ZLLKNTUZGJLVHIW2KEQZDL3HSS6VO2BSNRZJAC3USFAQ, seg_pos={0 0}, upload=330511]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:145.79.227.6:4258): protocol: rpc: dial tcp 145.79.227.6:4258: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:30:16.586708171 (376h1m52.626281301s) [noise=true , address=145.79.227.16:4912 , piece_id=AG2FUP2KFIJ7W55H2VRYTJYWQ57MAXCTEMTDWSR4FUJAYE3HFKBQ, seg_pos={0 0}, upload=330511]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:145.79.227.16:4912): protocol: rpc: dial tcp 145.79.227.16:4912: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:30:16.586749971 (376h1m52.626323101s) [noise=true , address=145.79.227.15:4875 , piece_id=ZEWO52263YKTTRYDVF4PV52JTPEDJR3XGVRG3BFE5QQKDQSCE2LQ, seg_pos={0 0}, upload=330511]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: rpc: dial tcp 145.79.227.15:4875: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:30:16.586795768 (376h1m52.626368888s) [noise=true , address=145.79.227.17:4972 , piece_id=Q3KQQ57XWDUCXNHP33STZ7DN6VQ6FVYZTDPLINEPZASFEVY5XDJA, seg_pos={0 0}, upload=330511]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:145.79.227.17:4972): protocol: rpc: dial tcp 145.79.227.17:4972: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:30:16.586806088 (376h1m52.626379218s) [noise=true , address=145.79.227.19:4077 , piece_id=EDBYIPZUOPFRRIZBZJJEP2U7NZFCUHRLDZESZ7J5CQZFKADNLE6Q, seg_pos={0 0}, upload=330511]: Done uploading piece. err: ecclient: upload failed (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk, address:145.79.227.19:4077): protocol: rpc: dial tcp 145.79.227.19:4077: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:30:16.586831426 (376h1m52.626404556s) [seg_pos={0 0}, upload=330511]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Sep 22 05:30:16.834944071 (376h1m52.874517201s) [upload=330511]: Done uploading segments... Sep 22 05:30:16.834981573 (376h1m52.874554703s) [upload=330511]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 5.959312077s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....X...X..................................................] duration: 5.813426534s FAIL (2025-09-22 05:45:17 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 5h37m59.543525066s) [seg_pos={0 0}, upload=330171]: Begin upload segment... Sep 22 05:06:23.504018422 (375h37m59.543591552s) [seg_pos={0 0}, upload=330171]: Done begin upload segment. Sep 22 05:06:23.504059370 (375h37m59.543632501s) [noise=true , address=45.91.116.9:4292 , piece_id=3QGXKEK5C77XDD34Z77C3K2LYEMGJT2SADTL6RGW2YCYQFV5CLUA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.504681175 (375h37m59.544254355s) [noise=true , address=145.79.227.39:4039 , piece_id=MZRBH2VHR7CCJK4OCCXINOQTNBP7PUP236H7DB2DFL7ZFDBNI2IQ, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.505278141 (375h37m59.544851301s) [noise=true , address=145.79.227.33:4881 , piece_id=AIKNNVT5PMYFYKRMUFN5DQMY7TWEUN3FUC64IQQYKAMV723TKKZA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.505743808 (375h37m59.545316948s) [noise=true , address=145.79.227.46:4601 , piece_id=EVWU7M2P2W4FQCNDEQ2OO3SR4KHPTXQRM7GEI6Q6DHYMDHK4PP4Q, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.505904454 (375h37m59.545477584s) [noise=true , address=145.79.227.24:4594 , piece_id=V55Y7DFR7QJWBBQRKSTUWYNEZSCAIWAY2XHAOZOF2H7JDFSX6IBA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.506859502 (375h37m59.546432622s) [noise=true , address=145.79.227.10:4420 , piece_id=ANGUQNIL2D4K2YR7KVJMY37WW7ILRVXSZOB7FRRAKJPD3AGGPGOA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.506958631 (375h37m59.546531761s) [noise=true , address=145.79.227.16:4933 , piece_id=WOBRP7MRD6VVMMXEHKNL4XDSPKQBHVINQD3I4N2SQU2MOSC2ZUCQ, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.507277077 (375h37m59.546850217s) [noise=true , address=145.79.227.7:4308 , piece_id=QEUKJMQDWYMBVKSVNNVDK53P6CZ4X47IINRSWCHAFG7APSFOOAFQ, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.508336856 (375h37m59.547909986s) [noise=true , address=77.90.31.19:4847 , piece_id=6GLTB4OVJTM67VRKRWATCIWE7WVK35IQMQIVOEZMZM6I5DFNRIBA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.513150782 (375h37m59.552723912s) [noise=true , address=145.79.227.12:4770 , piece_id=X5KVN5TA67L4TTD7LVG6N4I54BKNESFAHXRPVIWGDFCEPSHRKXPA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.513934875 (375h37m59.553508006s) [noise=true , address=45.91.116.14:4214 , piece_id=DOY7FC4SJKGAFP3LHSJXKWCRSYI6I6IR2R2FZ47QJNDM4SCAUXQQ, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.514566187 (375h37m59.554139318s) [noise=true , address=145.79.227.16:4939 , piece_id=DLS7J35PB7NXNMAPTVHBZ3VN3U725CSVZZKEYY37HZWQPWCCJO7A, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.515189424 (375h37m59.554762544s) [noise=true , address=145.79.227.25:4637 , piece_id=3SMC4AHCMKJEDXJOJVZ3FJEKW23VZDSWW6MDCENAYJ7HKOPWNHBQ, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.515805377 (375h37m59.555378508s) [noise=true , address=145.79.227.6:4292 , piece_id=BYUJPZSYFTYD4C4F5EBAW6BSFXTILW6EASOIRHZQI7ARCSG63F2Q, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.516494880 (375h37m59.55606801s) [noise=true , address=77.90.31.16:4764 , piece_id=PSAP3HPVNEURWWNC7K2K2G4NEUENKWDUR7OJY6UBGRTCSBYNHRWQ, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.526689722 (375h37m59.566262863s) [noise=true , address=145.79.227.31:4818 , piece_id=KAJZKTKOHNN5U4CTYJ7CLO32ONU6VSEJ2QBLPBI5WWZZ37YOI5NA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.531527154 (375h37m59.571100294s) [noise=true , address=145.79.227.49:4338 , piece_id=NBR62GETN7QBXC7QJCUB5DZHKVMC5AKQQBIIGDRXNECNZFULEDDA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.537921861 (375h37m59.577494991s) [noise=true , address=145.79.227.5:4247 , piece_id=QGN2S5CET2J6LGW6YPQYDWEVOGEMHWRSRMJG2IUQ36KA23OG2R3A, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:24.564378058 (375h38m0.603951188s) [noise=true , address=145.79.227.5:4247 , piece_id=QGN2S5CET2J6LGW6YPQYDWEVOGEMHWRSRMJG2IUQ36KA23OG2R3A, seg_pos={0 0}, upload=330171]: Done uploading piece. err: ecclient: upload failed (node:1262xxX6aUPTYgKdvRxAyc77PmPBWJtq5cXRJFoELxHbfn18DLE, address:145.79.227.5:4247): protocol: rpc: dial tcp 145.79.227.5:4247: connect: no route to host; conn closed: op Sep 22 05:06:24.986206552 (375h38m1.025779682s) [upload=330171]: Next returned nil segment Sep 22 05:06:24.986226830 (375h38m1.02579995s) [upload=330171]: Waiting for error group managing segments... Sep 22 05:06:25.383578176 (375h38m1.423151306s) [noise=true , address=145.79.227.24:4594 , piece_id=V55Y7DFR7QJWBBQRKSTUWYNEZSCAIWAY2XHAOZOF2H7JDFSX6IBA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.383806571 (375h38m1.423379702s) [noise=true , address=145.79.227.33:4881 , piece_id=AIKNNVT5PMYFYKRMUFN5DQMY7TWEUN3FUC64IQQYKAMV723TKKZA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.398811493 (375h38m1.438384634s) [noise=true , address=145.79.227.46:4601 , piece_id=EVWU7M2P2W4FQCNDEQ2OO3SR4KHPTXQRM7GEI6Q6DHYMDHK4PP4Q, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.399530783 (375h38m1.439103913s) [noise=true , address=145.79.227.39:4039 , piece_id=MZRBH2VHR7CCJK4OCCXINOQTNBP7PUP236H7DB2DFL7ZFDBNI2IQ, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.406508421 (375h38m1.446081551s) [noise=true , address=145.79.227.49:4338 , piece_id=NBR62GETN7QBXC7QJCUB5DZHKVMC5AKQQBIIGDRXNECNZFULEDDA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.415736332 (375h38m1.455309462s) [noise=true , address=145.79.227.25:4637 , piece_id=3SMC4AHCMKJEDXJOJVZ3FJEKW23VZDSWW6MDCENAYJ7HKOPWNHBQ, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.437405044 (375h38m1.476978184s) [noise=true , address=145.79.227.31:4818 , piece_id=KAJZKTKOHNN5U4CTYJ7CLO32ONU6VSEJ2QBLPBI5WWZZ37YOI5NA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.465513107 (375h38m1.505086237s) [noise=true , address=77.90.31.19:4847 , piece_id=6GLTB4OVJTM67VRKRWATCIWE7WVK35IQMQIVOEZMZM6I5DFNRIBA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.501349827 (375h38m1.540922957s) [noise=true , address=45.91.116.9:4292 , piece_id=3QGXKEK5C77XDD34Z77C3K2LYEMGJT2SADTL6RGW2YCYQFV5CLUA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.579007182 (375h38m1.618580302s) [noise=true , address=77.90.31.16:4764 , piece_id=PSAP3HPVNEURWWNC7K2K2G4NEUENKWDUR7OJY6UBGRTCSBYNHRWQ, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.586807715 (375h38m1.626380845s) [noise=true , address=45.91.116.14:4214 , piece_id=DOY7FC4SJKGAFP3LHSJXKWCRSYI6I6IR2R2FZ47QJNDM4SCAUXQQ, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:07:22.245417769 (375h38m58.284990899s) [noise=true , address=145.79.227.16:4933 , piece_id=WOBRP7MRD6VVMMXEHKNL4XDSPKQBHVINQD3I4N2SQU2MOSC2ZUCQ, seg_pos={0 0}, upload=330171]: Done uploading piece. err: ecclient: upload failed (node:19m4inaXvTdwf2rKip3eFELv6JAgFvXJ8KqjPZYbb48i8Ktnwe, address:145.79.227.16:4933): protocol: rpc: dial tcp 145.79.227.16:4933: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:07:22.245544310 (375h38m58.28511744s) [noise=true , address=145.79.227.12:4770 , piece_id=X5KVN5TA67L4TTD7LVG6N4I54BKNESFAHXRPVIWGDFCEPSHRKXPA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: ecclient: upload failed (node:1Ug6GpYToYDXiHmiJP4GoE7eJV3kqRveix2S5SLDcKQJtEbNPz, address:145.79.227.12:4770): protocol: rpc: dial tcp 145.79.227.12:4770: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:07:22.245567825 (375h38m58.285140955s) [noise=true , address=145.79.227.16:4939 , piece_id=DLS7J35PB7NXNMAPTVHBZ3VN3U725CSVZZKEYY37HZWQPWCCJO7A, seg_pos={0 0}, upload=330171]: Done uploading piece. err: ecclient: upload failed (node:1mRmp3HhD95PruS4AF8soRsSXNYUeb4mJE7y52FvoeN9D3Gu4F, address:145.79.227.16:4939): protocol: rpc: dial tcp 145.79.227.16:4939: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:07:22.245634181 (375h38m58.285207312s) [noise=true , address=145.79.227.7:4308 , piece_id=QEUKJMQDWYMBVKSVNNVDK53P6CZ4X47IINRSWCHAFG7APSFOOAFQ, seg_pos={0 0}, upload=330171]: Done uploading piece. err: ecclient: upload failed (node:1PgA5SZkBvK673LsV7p2sXpbi1r79esNNZspPZTzaitqfbjtyY, address:145.79.227.7:4308): protocol: rpc: dial tcp 145.79.227.7:4308: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:07:22.245657335 (375h38m58.285230455s) [noise=true , address=145.79.227.10:4420 , piece_id=ANGUQNIL2D4K2YR7KVJMY37WW7ILRVXSZOB7FRRAKJPD3AGGPGOA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: ecclient: upload failed (node:12hJrjeMUMYmFw3yxRtQQKRh9ciPfdu5FXo8NGrsc6HBqgfJzzp, address:145.79.227.10:4420): protocol: rpc: dial tcp 145.79.227.10:4420: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:07:22.245675269 (375h38m58.28524839s) [noise=true , address=145.79.227.6:4292 , piece_id=BYUJPZSYFTYD4C4F5EBAW6BSFXTILW6EASOIRHZQI7ARCSG63F2Q, seg_pos={0 0}, upload=330171]: Done uploading piece. err: ecclient: upload failed (node:12UpWmMhEmk5veKZvzsQ3MCxCMnfEKtFgCugBTsYfy9Ejm7G4xD, address:145.79.227.6:4292): protocol: rpc: dial tcp 145.79.227.6:4292: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:07:22.245712370 (375h38m58.2852855s) [seg_pos={0 0}, upload=330171]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Sep 22 05:07:22.500217377 (375h38m58.539790507s) [upload=330171]: Done uploading segments... Sep 22 05:07:22.500253296 (375h38m58.539826426s) [upload=330171]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 5.813470519s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [...XX....X..X...X.X......X.........X.X..X....X..XX..........] duration: 6.129323606s FAIL (2025-09-22 05:45:09 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.129353782s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [..XXXX.X.XX.....X.X....................X....................] duration: 5.915841655s FAIL (2025-09-22 05:44:44 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== 3.337157461s) [upload=330511]: Starting upload Sep 22 05:29:47.297668972 (376h1m23.337242102s) [upload=330511]: Done starting upload Sep 22 05:29:47.297899231 (376h1m23.337472361s) [upload=330511]: Uploading segments... Sep 22 05:29:47.298002678 (376h1m23.337575818s) [upload=330511]: Got next segment. Inline: false Sep 22 05:29:47.298059145 (376h1m23.337632275s) [upload=330511]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 05:29:47.298066109 (376h1m23.337639229s) [upload=330511]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 05:29:47.552911815 (376h1m23.592484945s) [seg_pos={0 0}, upload=330511]: Begin upload segment... Sep 22 05:29:47.552975406 (376h1m23.592548536s) [seg_pos={0 0}, upload=330511]: Done begin upload segment. Sep 22 05:29:47.553013599 (376h1m23.592586719s) [noise=true , address=45.91.116.11:4117 , piece_id=NIFPLA2N3L6XDWTIFUM3YYRGBMFQJKWFVCGYXD2GBFYVYDXW6OEQ, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.553528389 (376h1m23.593101539s) [noise=true , address=145.79.227.6:4258 , piece_id=D7HMSQH7ZLLKNTUZGJLVHIW2KEQZDL3HSS6VO2BSNRZJAC3USFAQ, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.553925034 (376h1m23.593498154s) [noise=true , address=45.91.116.9:4261 , piece_id=REZZWYP25KZOP4UBJWJPHGCDUWCW3EBOJG2DOLCDYZY5B43KBMSQ, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.553961283 (376h1m23.593534413s) [noise=true , address=77.90.31.14:4663 , piece_id=LYGO7UTA55Q5Q7APPDJZNNWZ4S2XHAT4LS4V27VF27A3435DVJQA, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.554326960 (376h1m23.59390009s) [noise=true , address=77.90.31.16:4735 , piece_id=K5KYKKY3BQV6UVJK4G3DYC24DFFIEFQX7KGGFLS4P35KJRFHA6UQ, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.554351497 (376h1m23.593924627s) [noise=true , address=145.79.227.16:4912 , piece_id=AG2FUP2KFIJ7W55H2VRYTJYWQ57MAXCTEMTDWSR4FUJAYE3HFKBQ, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.554588828 (376h1m23.594161948s) [noise=true , address=145.79.227.12:4767 , piece_id=OUVFSM62B7UZ4U4WGFFBOHCPHXEXR6A6H2FQL3DTG5L3DEPUVYVQ, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.554724957 (376h1m23.594298087s) [noise=true , address=77.90.31.18:4807 , piece_id=JFOMLF75MIXEDP5CKDHH7J3PWGWESB7BIKK3SCAIOVNX74DJDC3A, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.554875986 (376h1m23.594449116s) [noise=true , address=145.79.227.3:4151 , piece_id=O6AU3AOBKJRIUMEAWPHMWWDDNRTA52WL72JNUTK5JFLVG2TNV5IA, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.555093950 (376h1m23.59466708s) [noise=true , address=145.79.227.17:4972 , piece_id=Q3KQQ57XWDUCXNHP33STZ7DN6VQ6FVYZTDPLINEPZASFEVY5XDJA, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.555135429 (376h1m23.594708559s) [noise=true , address=45.91.116.6:4973 , piece_id=WPWUFOFQHQZFRPTR33GTDST2NYCVPXKLDVCJQFPQ4A7PI44LJKOQ, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.555464797 (376h1m23.595037917s) [noise=true , address=45.91.116.14:4189 , piece_id=6VJ6XCCEV2CZSARNRF3ATZ4KCUI75QKWIWQWJL5QH4XTQG4MCLUQ, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.555726194 (376h1m23.595299314s) [noise=true , address=77.90.31.15:4699 , piece_id=6XZN64DOILMELBYDU4D72ER33XDKMRIJRSAXX5ITFYPG46YEBITQ, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.556009483 (376h1m23.595582614s) [noise=true , address=145.79.227.19:4077 , piece_id=EDBYIPZUOPFRRIZBZJJEP2U7NZFCUHRLDZESZ7J5CQZFKADNLE6Q, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.556364609 (376h1m23.59593773s) [noise=true , address=145.79.227.15:4875 , piece_id=ZEWO52263YKTTRYDVF4PV52JTPEDJR3XGVRG3BFE5QQKDQSCE2LQ, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.556611299 (376h1m23.596184429s) [noise=true , address=145.79.227.5:4222 , piece_id=CZNVNWHETTFQS6JR2YLE65LNQU4WX74DAROY5I3KAYHG4QDLDOKA, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:49.308510933 (376h1m25.348084063s) [upload=330511]: Next returned nil segment Sep 22 05:29:49.308533666 (376h1m25.348106796s) [upload=330511]: Waiting for error group managing segments... Sep 22 05:29:49.602590138 (376h1m25.642163268s) [noise=true , address=45.91.116.11:4117 , piece_id=NIFPLA2N3L6XDWTIFUM3YYRGBMFQJKWFVCGYXD2GBFYVYDXW6OEQ, seg_pos={0 0}, upload=330511]: Done uploading piece. err: <nil> Sep 22 05:29:49.633269677 (376h1m25.672842807s) [noise=true , address=77.90.31.15:4699 , piece_id=6XZN64DOILMELBYDU4D72ER33XDKMRIJRSAXX5ITFYPG46YEBITQ, seg_pos={0 0}, upload=330511]: Done uploading piece. err: <nil> Sep 22 05:29:49.637382749 (376h1m25.676955879s) [noise=true , address=45.91.116.6:4973 , piece_id=WPWUFOFQHQZFRPTR33GTDST2NYCVPXKLDVCJQFPQ4A7PI44LJKOQ, seg_pos={0 0}, upload=330511]: Done uploading piece. err: <nil> Sep 22 05:29:49.648543782 (376h1m25.688116902s) [noise=true , address=77.90.31.16:4735 , piece_id=K5KYKKY3BQV6UVJK4G3DYC24DFFIEFQX7KGGFLS4P35KJRFHA6UQ, seg_pos={0 0}, upload=330511]: Done uploading piece. err: <nil> Sep 22 05:29:49.653934647 (376h1m25.693507767s) [noise=true , address=45.91.116.14:4189 , piece_id=6VJ6XCCEV2CZSARNRF3ATZ4KCUI75QKWIWQWJL5QH4XTQG4MCLUQ, seg_pos={0 0}, upload=330511]: Done uploading piece. err: <nil> Sep 22 05:29:49.669096699 (376h1m25.708669829s) [noise=true , address=45.91.116.9:4261 , piece_id=REZZWYP25KZOP4UBJWJPHGCDUWCW3EBOJG2DOLCDYZY5B43KBMSQ, seg_pos={0 0}, upload=330511]: Done uploading piece. err: <nil> Sep 22 05:29:49.673672402 (376h1m25.713245532s) [noise=true , address=77.90.31.14:4663 , piece_id=LYGO7UTA55Q5Q7APPDJZNNWZ4S2XHAT4LS4V27VF27A3435DVJQA, seg_pos={0 0}, upload=330511]: Done uploading piece. err: <nil> Sep 22 05:29:49.741841689 (376h1m25.781414809s) [noise=true , address=77.90.31.18:4807 , piece_id=JFOMLF75MIXEDP5CKDHH7J3PWGWESB7BIKK3SCAIOVNX74DJDC3A, seg_pos={0 0}, upload=330511]: Done uploading piece. err: <nil> Sep 22 05:29:58.021015047 (376h1m34.060588167s) [noise=true , address=145.79.227.5:4222 , piece_id=CZNVNWHETTFQS6JR2YLE65LNQU4WX74DAROY5I3KAYHG4QDLDOKA, seg_pos={0 0}, upload=330511]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:145.79.227.5:4222): protocol: rpc: dial tcp 145.79.227.5:4222: connect: no route to host; conn closed: op; piecestore: piecestore close: manager closed: conn closed: op Sep 22 05:30:05.028605724 (376h1m41.068178854s) [noise=true , address=145.79.227.3:4151 , piece_id=O6AU3AOBKJRIUMEAWPHMWWDDNRTA52WL72JNUTK5JFLVG2TNV5IA, seg_pos={0 0}, upload=330511]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:145.79.227.3:4151): protocol: rpc: dial tcp 145.79.227.3:4151: connect: no route to host; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:30:16.586571260 (376h1m52.62614439s) [noise=true , address=145.79.227.12:4767 , piece_id=OUVFSM62B7UZ4U4WGFFBOHCPHXEXR6A6H2FQL3DTG5L3DEPUVYVQ, seg_pos={0 0}, upload=330511]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:145.79.227.12:4767): protocol: rpc: dial tcp 145.79.227.12:4767: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:30:16.586679978 (376h1m52.626253118s) [noise=true , address=145.79.227.6:4258 , piece_id=D7HMSQH7ZLLKNTUZGJLVHIW2KEQZDL3HSS6VO2BSNRZJAC3USFAQ, seg_pos={0 0}, upload=330511]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:145.79.227.6:4258): protocol: rpc: dial tcp 145.79.227.6:4258: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:30:16.586708171 (376h1m52.626281301s) [noise=true , address=145.79.227.16:4912 , piece_id=AG2FUP2KFIJ7W55H2VRYTJYWQ57MAXCTEMTDWSR4FUJAYE3HFKBQ, seg_pos={0 0}, upload=330511]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:145.79.227.16:4912): protocol: rpc: dial tcp 145.79.227.16:4912: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:30:16.586749971 (376h1m52.626323101s) [noise=true , address=145.79.227.15:4875 , piece_id=ZEWO52263YKTTRYDVF4PV52JTPEDJR3XGVRG3BFE5QQKDQSCE2LQ, seg_pos={0 0}, upload=330511]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: rpc: dial tcp 145.79.227.15:4875: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:30:16.586795768 (376h1m52.626368888s) [noise=true , address=145.79.227.17:4972 , piece_id=Q3KQQ57XWDUCXNHP33STZ7DN6VQ6FVYZTDPLINEPZASFEVY5XDJA, seg_pos={0 0}, upload=330511]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:145.79.227.17:4972): protocol: rpc: dial tcp 145.79.227.17:4972: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:30:16.586806088 (376h1m52.626379218s) [noise=true , address=145.79.227.19:4077 , piece_id=EDBYIPZUOPFRRIZBZJJEP2U7NZFCUHRLDZESZ7J5CQZFKADNLE6Q, seg_pos={0 0}, upload=330511]: Done uploading piece. err: ecclient: upload failed (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk, address:145.79.227.19:4077): protocol: rpc: dial tcp 145.79.227.19:4077: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:30:16.586831426 (376h1m52.626404556s) [seg_pos={0 0}, upload=330511]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Sep 22 05:30:16.834944071 (376h1m52.874517201s) [upload=330511]: Done uploading segments... Sep 22 05:30:16.834981573 (376h1m52.874554703s) [upload=330511]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 5.915874497s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [......X...X.................................................] duration: 5.915584357s FAIL (2025-09-22 05:44:22 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 5h37m59.543525066s) [seg_pos={0 0}, upload=330171]: Begin upload segment... Sep 22 05:06:23.504018422 (375h37m59.543591552s) [seg_pos={0 0}, upload=330171]: Done begin upload segment. Sep 22 05:06:23.504059370 (375h37m59.543632501s) [noise=true , address=45.91.116.9:4292 , piece_id=3QGXKEK5C77XDD34Z77C3K2LYEMGJT2SADTL6RGW2YCYQFV5CLUA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.504681175 (375h37m59.544254355s) [noise=true , address=145.79.227.39:4039 , piece_id=MZRBH2VHR7CCJK4OCCXINOQTNBP7PUP236H7DB2DFL7ZFDBNI2IQ, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.505278141 (375h37m59.544851301s) [noise=true , address=145.79.227.33:4881 , piece_id=AIKNNVT5PMYFYKRMUFN5DQMY7TWEUN3FUC64IQQYKAMV723TKKZA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.505743808 (375h37m59.545316948s) [noise=true , address=145.79.227.46:4601 , piece_id=EVWU7M2P2W4FQCNDEQ2OO3SR4KHPTXQRM7GEI6Q6DHYMDHK4PP4Q, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.505904454 (375h37m59.545477584s) [noise=true , address=145.79.227.24:4594 , piece_id=V55Y7DFR7QJWBBQRKSTUWYNEZSCAIWAY2XHAOZOF2H7JDFSX6IBA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.506859502 (375h37m59.546432622s) [noise=true , address=145.79.227.10:4420 , piece_id=ANGUQNIL2D4K2YR7KVJMY37WW7ILRVXSZOB7FRRAKJPD3AGGPGOA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.506958631 (375h37m59.546531761s) [noise=true , address=145.79.227.16:4933 , piece_id=WOBRP7MRD6VVMMXEHKNL4XDSPKQBHVINQD3I4N2SQU2MOSC2ZUCQ, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.507277077 (375h37m59.546850217s) [noise=true , address=145.79.227.7:4308 , piece_id=QEUKJMQDWYMBVKSVNNVDK53P6CZ4X47IINRSWCHAFG7APSFOOAFQ, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.508336856 (375h37m59.547909986s) [noise=true , address=77.90.31.19:4847 , piece_id=6GLTB4OVJTM67VRKRWATCIWE7WVK35IQMQIVOEZMZM6I5DFNRIBA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.513150782 (375h37m59.552723912s) [noise=true , address=145.79.227.12:4770 , piece_id=X5KVN5TA67L4TTD7LVG6N4I54BKNESFAHXRPVIWGDFCEPSHRKXPA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.513934875 (375h37m59.553508006s) [noise=true , address=45.91.116.14:4214 , piece_id=DOY7FC4SJKGAFP3LHSJXKWCRSYI6I6IR2R2FZ47QJNDM4SCAUXQQ, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.514566187 (375h37m59.554139318s) [noise=true , address=145.79.227.16:4939 , piece_id=DLS7J35PB7NXNMAPTVHBZ3VN3U725CSVZZKEYY37HZWQPWCCJO7A, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.515189424 (375h37m59.554762544s) [noise=true , address=145.79.227.25:4637 , piece_id=3SMC4AHCMKJEDXJOJVZ3FJEKW23VZDSWW6MDCENAYJ7HKOPWNHBQ, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.515805377 (375h37m59.555378508s) [noise=true , address=145.79.227.6:4292 , piece_id=BYUJPZSYFTYD4C4F5EBAW6BSFXTILW6EASOIRHZQI7ARCSG63F2Q, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.516494880 (375h37m59.55606801s) [noise=true , address=77.90.31.16:4764 , piece_id=PSAP3HPVNEURWWNC7K2K2G4NEUENKWDUR7OJY6UBGRTCSBYNHRWQ, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.526689722 (375h37m59.566262863s) [noise=true , address=145.79.227.31:4818 , piece_id=KAJZKTKOHNN5U4CTYJ7CLO32ONU6VSEJ2QBLPBI5WWZZ37YOI5NA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.531527154 (375h37m59.571100294s) [noise=true , address=145.79.227.49:4338 , piece_id=NBR62GETN7QBXC7QJCUB5DZHKVMC5AKQQBIIGDRXNECNZFULEDDA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.537921861 (375h37m59.577494991s) [noise=true , address=145.79.227.5:4247 , piece_id=QGN2S5CET2J6LGW6YPQYDWEVOGEMHWRSRMJG2IUQ36KA23OG2R3A, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:24.564378058 (375h38m0.603951188s) [noise=true , address=145.79.227.5:4247 , piece_id=QGN2S5CET2J6LGW6YPQYDWEVOGEMHWRSRMJG2IUQ36KA23OG2R3A, seg_pos={0 0}, upload=330171]: Done uploading piece. err: ecclient: upload failed (node:1262xxX6aUPTYgKdvRxAyc77PmPBWJtq5cXRJFoELxHbfn18DLE, address:145.79.227.5:4247): protocol: rpc: dial tcp 145.79.227.5:4247: connect: no route to host; conn closed: op Sep 22 05:06:24.986206552 (375h38m1.025779682s) [upload=330171]: Next returned nil segment Sep 22 05:06:24.986226830 (375h38m1.02579995s) [upload=330171]: Waiting for error group managing segments... Sep 22 05:06:25.383578176 (375h38m1.423151306s) [noise=true , address=145.79.227.24:4594 , piece_id=V55Y7DFR7QJWBBQRKSTUWYNEZSCAIWAY2XHAOZOF2H7JDFSX6IBA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.383806571 (375h38m1.423379702s) [noise=true , address=145.79.227.33:4881 , piece_id=AIKNNVT5PMYFYKRMUFN5DQMY7TWEUN3FUC64IQQYKAMV723TKKZA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.398811493 (375h38m1.438384634s) [noise=true , address=145.79.227.46:4601 , piece_id=EVWU7M2P2W4FQCNDEQ2OO3SR4KHPTXQRM7GEI6Q6DHYMDHK4PP4Q, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.399530783 (375h38m1.439103913s) [noise=true , address=145.79.227.39:4039 , piece_id=MZRBH2VHR7CCJK4OCCXINOQTNBP7PUP236H7DB2DFL7ZFDBNI2IQ, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.406508421 (375h38m1.446081551s) [noise=true , address=145.79.227.49:4338 , piece_id=NBR62GETN7QBXC7QJCUB5DZHKVMC5AKQQBIIGDRXNECNZFULEDDA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.415736332 (375h38m1.455309462s) [noise=true , address=145.79.227.25:4637 , piece_id=3SMC4AHCMKJEDXJOJVZ3FJEKW23VZDSWW6MDCENAYJ7HKOPWNHBQ, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.437405044 (375h38m1.476978184s) [noise=true , address=145.79.227.31:4818 , piece_id=KAJZKTKOHNN5U4CTYJ7CLO32ONU6VSEJ2QBLPBI5WWZZ37YOI5NA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.465513107 (375h38m1.505086237s) [noise=true , address=77.90.31.19:4847 , piece_id=6GLTB4OVJTM67VRKRWATCIWE7WVK35IQMQIVOEZMZM6I5DFNRIBA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.501349827 (375h38m1.540922957s) [noise=true , address=45.91.116.9:4292 , piece_id=3QGXKEK5C77XDD34Z77C3K2LYEMGJT2SADTL6RGW2YCYQFV5CLUA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.579007182 (375h38m1.618580302s) [noise=true , address=77.90.31.16:4764 , piece_id=PSAP3HPVNEURWWNC7K2K2G4NEUENKWDUR7OJY6UBGRTCSBYNHRWQ, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.586807715 (375h38m1.626380845s) [noise=true , address=45.91.116.14:4214 , piece_id=DOY7FC4SJKGAFP3LHSJXKWCRSYI6I6IR2R2FZ47QJNDM4SCAUXQQ, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:07:22.245417769 (375h38m58.284990899s) [noise=true , address=145.79.227.16:4933 , piece_id=WOBRP7MRD6VVMMXEHKNL4XDSPKQBHVINQD3I4N2SQU2MOSC2ZUCQ, seg_pos={0 0}, upload=330171]: Done uploading piece. err: ecclient: upload failed (node:19m4inaXvTdwf2rKip3eFELv6JAgFvXJ8KqjPZYbb48i8Ktnwe, address:145.79.227.16:4933): protocol: rpc: dial tcp 145.79.227.16:4933: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:07:22.245544310 (375h38m58.28511744s) [noise=true , address=145.79.227.12:4770 , piece_id=X5KVN5TA67L4TTD7LVG6N4I54BKNESFAHXRPVIWGDFCEPSHRKXPA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: ecclient: upload failed (node:1Ug6GpYToYDXiHmiJP4GoE7eJV3kqRveix2S5SLDcKQJtEbNPz, address:145.79.227.12:4770): protocol: rpc: dial tcp 145.79.227.12:4770: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:07:22.245567825 (375h38m58.285140955s) [noise=true , address=145.79.227.16:4939 , piece_id=DLS7J35PB7NXNMAPTVHBZ3VN3U725CSVZZKEYY37HZWQPWCCJO7A, seg_pos={0 0}, upload=330171]: Done uploading piece. err: ecclient: upload failed (node:1mRmp3HhD95PruS4AF8soRsSXNYUeb4mJE7y52FvoeN9D3Gu4F, address:145.79.227.16:4939): protocol: rpc: dial tcp 145.79.227.16:4939: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:07:22.245634181 (375h38m58.285207312s) [noise=true , address=145.79.227.7:4308 , piece_id=QEUKJMQDWYMBVKSVNNVDK53P6CZ4X47IINRSWCHAFG7APSFOOAFQ, seg_pos={0 0}, upload=330171]: Done uploading piece. err: ecclient: upload failed (node:1PgA5SZkBvK673LsV7p2sXpbi1r79esNNZspPZTzaitqfbjtyY, address:145.79.227.7:4308): protocol: rpc: dial tcp 145.79.227.7:4308: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:07:22.245657335 (375h38m58.285230455s) [noise=true , address=145.79.227.10:4420 , piece_id=ANGUQNIL2D4K2YR7KVJMY37WW7ILRVXSZOB7FRRAKJPD3AGGPGOA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: ecclient: upload failed (node:12hJrjeMUMYmFw3yxRtQQKRh9ciPfdu5FXo8NGrsc6HBqgfJzzp, address:145.79.227.10:4420): protocol: rpc: dial tcp 145.79.227.10:4420: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:07:22.245675269 (375h38m58.28524839s) [noise=true , address=145.79.227.6:4292 , piece_id=BYUJPZSYFTYD4C4F5EBAW6BSFXTILW6EASOIRHZQI7ARCSG63F2Q, seg_pos={0 0}, upload=330171]: Done uploading piece. err: ecclient: upload failed (node:12UpWmMhEmk5veKZvzsQ3MCxCMnfEKtFgCugBTsYfy9Ejm7G4xD, address:145.79.227.6:4292): protocol: rpc: dial tcp 145.79.227.6:4292: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:07:22.245712370 (375h38m58.2852855s) [seg_pos={0 0}, upload=330171]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Sep 22 05:07:22.500217377 (375h38m58.539790507s) [upload=330171]: Done uploading segments... Sep 22 05:07:22.500253296 (375h38m58.539826426s) [upload=330171]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 5.915621187s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [....XX....X..X...X.X......X.........X.X..X....X..XX.........] duration: 6.083491627s FAIL (2025-09-22 05:44:22 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.083519369s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [.....XX....X..X...X.X......X.........X.X..X....X..XX........] duration: 6.063863132s FAIL (2025-09-22 05:43:57 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.063888341s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [...XXXX.X.XX.....X.X....................X...................] duration: 5.979699899s FAIL (2025-09-22 05:43:34 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== 3.337157461s) [upload=330511]: Starting upload Sep 22 05:29:47.297668972 (376h1m23.337242102s) [upload=330511]: Done starting upload Sep 22 05:29:47.297899231 (376h1m23.337472361s) [upload=330511]: Uploading segments... Sep 22 05:29:47.298002678 (376h1m23.337575818s) [upload=330511]: Got next segment. Inline: false Sep 22 05:29:47.298059145 (376h1m23.337632275s) [upload=330511]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 05:29:47.298066109 (376h1m23.337639229s) [upload=330511]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 05:29:47.552911815 (376h1m23.592484945s) [seg_pos={0 0}, upload=330511]: Begin upload segment... Sep 22 05:29:47.552975406 (376h1m23.592548536s) [seg_pos={0 0}, upload=330511]: Done begin upload segment. Sep 22 05:29:47.553013599 (376h1m23.592586719s) [noise=true , address=45.91.116.11:4117 , piece_id=NIFPLA2N3L6XDWTIFUM3YYRGBMFQJKWFVCGYXD2GBFYVYDXW6OEQ, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.553528389 (376h1m23.593101539s) [noise=true , address=145.79.227.6:4258 , piece_id=D7HMSQH7ZLLKNTUZGJLVHIW2KEQZDL3HSS6VO2BSNRZJAC3USFAQ, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.553925034 (376h1m23.593498154s) [noise=true , address=45.91.116.9:4261 , piece_id=REZZWYP25KZOP4UBJWJPHGCDUWCW3EBOJG2DOLCDYZY5B43KBMSQ, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.553961283 (376h1m23.593534413s) [noise=true , address=77.90.31.14:4663 , piece_id=LYGO7UTA55Q5Q7APPDJZNNWZ4S2XHAT4LS4V27VF27A3435DVJQA, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.554326960 (376h1m23.59390009s) [noise=true , address=77.90.31.16:4735 , piece_id=K5KYKKY3BQV6UVJK4G3DYC24DFFIEFQX7KGGFLS4P35KJRFHA6UQ, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.554351497 (376h1m23.593924627s) [noise=true , address=145.79.227.16:4912 , piece_id=AG2FUP2KFIJ7W55H2VRYTJYWQ57MAXCTEMTDWSR4FUJAYE3HFKBQ, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.554588828 (376h1m23.594161948s) [noise=true , address=145.79.227.12:4767 , piece_id=OUVFSM62B7UZ4U4WGFFBOHCPHXEXR6A6H2FQL3DTG5L3DEPUVYVQ, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.554724957 (376h1m23.594298087s) [noise=true , address=77.90.31.18:4807 , piece_id=JFOMLF75MIXEDP5CKDHH7J3PWGWESB7BIKK3SCAIOVNX74DJDC3A, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.554875986 (376h1m23.594449116s) [noise=true , address=145.79.227.3:4151 , piece_id=O6AU3AOBKJRIUMEAWPHMWWDDNRTA52WL72JNUTK5JFLVG2TNV5IA, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.555093950 (376h1m23.59466708s) [noise=true , address=145.79.227.17:4972 , piece_id=Q3KQQ57XWDUCXNHP33STZ7DN6VQ6FVYZTDPLINEPZASFEVY5XDJA, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.555135429 (376h1m23.594708559s) [noise=true , address=45.91.116.6:4973 , piece_id=WPWUFOFQHQZFRPTR33GTDST2NYCVPXKLDVCJQFPQ4A7PI44LJKOQ, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.555464797 (376h1m23.595037917s) [noise=true , address=45.91.116.14:4189 , piece_id=6VJ6XCCEV2CZSARNRF3ATZ4KCUI75QKWIWQWJL5QH4XTQG4MCLUQ, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.555726194 (376h1m23.595299314s) [noise=true , address=77.90.31.15:4699 , piece_id=6XZN64DOILMELBYDU4D72ER33XDKMRIJRSAXX5ITFYPG46YEBITQ, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.556009483 (376h1m23.595582614s) [noise=true , address=145.79.227.19:4077 , piece_id=EDBYIPZUOPFRRIZBZJJEP2U7NZFCUHRLDZESZ7J5CQZFKADNLE6Q, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.556364609 (376h1m23.59593773s) [noise=true , address=145.79.227.15:4875 , piece_id=ZEWO52263YKTTRYDVF4PV52JTPEDJR3XGVRG3BFE5QQKDQSCE2LQ, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.556611299 (376h1m23.596184429s) [noise=true , address=145.79.227.5:4222 , piece_id=CZNVNWHETTFQS6JR2YLE65LNQU4WX74DAROY5I3KAYHG4QDLDOKA, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:49.308510933 (376h1m25.348084063s) [upload=330511]: Next returned nil segment Sep 22 05:29:49.308533666 (376h1m25.348106796s) [upload=330511]: Waiting for error group managing segments... Sep 22 05:29:49.602590138 (376h1m25.642163268s) [noise=true , address=45.91.116.11:4117 , piece_id=NIFPLA2N3L6XDWTIFUM3YYRGBMFQJKWFVCGYXD2GBFYVYDXW6OEQ, seg_pos={0 0}, upload=330511]: Done uploading piece. err: <nil> Sep 22 05:29:49.633269677 (376h1m25.672842807s) [noise=true , address=77.90.31.15:4699 , piece_id=6XZN64DOILMELBYDU4D72ER33XDKMRIJRSAXX5ITFYPG46YEBITQ, seg_pos={0 0}, upload=330511]: Done uploading piece. err: <nil> Sep 22 05:29:49.637382749 (376h1m25.676955879s) [noise=true , address=45.91.116.6:4973 , piece_id=WPWUFOFQHQZFRPTR33GTDST2NYCVPXKLDVCJQFPQ4A7PI44LJKOQ, seg_pos={0 0}, upload=330511]: Done uploading piece. err: <nil> Sep 22 05:29:49.648543782 (376h1m25.688116902s) [noise=true , address=77.90.31.16:4735 , piece_id=K5KYKKY3BQV6UVJK4G3DYC24DFFIEFQX7KGGFLS4P35KJRFHA6UQ, seg_pos={0 0}, upload=330511]: Done uploading piece. err: <nil> Sep 22 05:29:49.653934647 (376h1m25.693507767s) [noise=true , address=45.91.116.14:4189 , piece_id=6VJ6XCCEV2CZSARNRF3ATZ4KCUI75QKWIWQWJL5QH4XTQG4MCLUQ, seg_pos={0 0}, upload=330511]: Done uploading piece. err: <nil> Sep 22 05:29:49.669096699 (376h1m25.708669829s) [noise=true , address=45.91.116.9:4261 , piece_id=REZZWYP25KZOP4UBJWJPHGCDUWCW3EBOJG2DOLCDYZY5B43KBMSQ, seg_pos={0 0}, upload=330511]: Done uploading piece. err: <nil> Sep 22 05:29:49.673672402 (376h1m25.713245532s) [noise=true , address=77.90.31.14:4663 , piece_id=LYGO7UTA55Q5Q7APPDJZNNWZ4S2XHAT4LS4V27VF27A3435DVJQA, seg_pos={0 0}, upload=330511]: Done uploading piece. err: <nil> Sep 22 05:29:49.741841689 (376h1m25.781414809s) [noise=true , address=77.90.31.18:4807 , piece_id=JFOMLF75MIXEDP5CKDHH7J3PWGWESB7BIKK3SCAIOVNX74DJDC3A, seg_pos={0 0}, upload=330511]: Done uploading piece. err: <nil> Sep 22 05:29:58.021015047 (376h1m34.060588167s) [noise=true , address=145.79.227.5:4222 , piece_id=CZNVNWHETTFQS6JR2YLE65LNQU4WX74DAROY5I3KAYHG4QDLDOKA, seg_pos={0 0}, upload=330511]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:145.79.227.5:4222): protocol: rpc: dial tcp 145.79.227.5:4222: connect: no route to host; conn closed: op; piecestore: piecestore close: manager closed: conn closed: op Sep 22 05:30:05.028605724 (376h1m41.068178854s) [noise=true , address=145.79.227.3:4151 , piece_id=O6AU3AOBKJRIUMEAWPHMWWDDNRTA52WL72JNUTK5JFLVG2TNV5IA, seg_pos={0 0}, upload=330511]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:145.79.227.3:4151): protocol: rpc: dial tcp 145.79.227.3:4151: connect: no route to host; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:30:16.586571260 (376h1m52.62614439s) [noise=true , address=145.79.227.12:4767 , piece_id=OUVFSM62B7UZ4U4WGFFBOHCPHXEXR6A6H2FQL3DTG5L3DEPUVYVQ, seg_pos={0 0}, upload=330511]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:145.79.227.12:4767): protocol: rpc: dial tcp 145.79.227.12:4767: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:30:16.586679978 (376h1m52.626253118s) [noise=true , address=145.79.227.6:4258 , piece_id=D7HMSQH7ZLLKNTUZGJLVHIW2KEQZDL3HSS6VO2BSNRZJAC3USFAQ, seg_pos={0 0}, upload=330511]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:145.79.227.6:4258): protocol: rpc: dial tcp 145.79.227.6:4258: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:30:16.586708171 (376h1m52.626281301s) [noise=true , address=145.79.227.16:4912 , piece_id=AG2FUP2KFIJ7W55H2VRYTJYWQ57MAXCTEMTDWSR4FUJAYE3HFKBQ, seg_pos={0 0}, upload=330511]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:145.79.227.16:4912): protocol: rpc: dial tcp 145.79.227.16:4912: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:30:16.586749971 (376h1m52.626323101s) [noise=true , address=145.79.227.15:4875 , piece_id=ZEWO52263YKTTRYDVF4PV52JTPEDJR3XGVRG3BFE5QQKDQSCE2LQ, seg_pos={0 0}, upload=330511]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: rpc: dial tcp 145.79.227.15:4875: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:30:16.586795768 (376h1m52.626368888s) [noise=true , address=145.79.227.17:4972 , piece_id=Q3KQQ57XWDUCXNHP33STZ7DN6VQ6FVYZTDPLINEPZASFEVY5XDJA, seg_pos={0 0}, upload=330511]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:145.79.227.17:4972): protocol: rpc: dial tcp 145.79.227.17:4972: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:30:16.586806088 (376h1m52.626379218s) [noise=true , address=145.79.227.19:4077 , piece_id=EDBYIPZUOPFRRIZBZJJEP2U7NZFCUHRLDZESZ7J5CQZFKADNLE6Q, seg_pos={0 0}, upload=330511]: Done uploading piece. err: ecclient: upload failed (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk, address:145.79.227.19:4077): protocol: rpc: dial tcp 145.79.227.19:4077: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:30:16.586831426 (376h1m52.626404556s) [seg_pos={0 0}, upload=330511]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Sep 22 05:30:16.834944071 (376h1m52.874517201s) [upload=330511]: Done uploading segments... Sep 22 05:30:16.834981573 (376h1m52.874554703s) [upload=330511]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 5.979758681s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......X...X................................................] duration: 5.869157856s FAIL (2025-09-22 05:43:18 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 5h37m59.543525066s) [seg_pos={0 0}, upload=330171]: Begin upload segment... Sep 22 05:06:23.504018422 (375h37m59.543591552s) [seg_pos={0 0}, upload=330171]: Done begin upload segment. Sep 22 05:06:23.504059370 (375h37m59.543632501s) [noise=true , address=45.91.116.9:4292 , piece_id=3QGXKEK5C77XDD34Z77C3K2LYEMGJT2SADTL6RGW2YCYQFV5CLUA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.504681175 (375h37m59.544254355s) [noise=true , address=145.79.227.39:4039 , piece_id=MZRBH2VHR7CCJK4OCCXINOQTNBP7PUP236H7DB2DFL7ZFDBNI2IQ, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.505278141 (375h37m59.544851301s) [noise=true , address=145.79.227.33:4881 , piece_id=AIKNNVT5PMYFYKRMUFN5DQMY7TWEUN3FUC64IQQYKAMV723TKKZA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.505743808 (375h37m59.545316948s) [noise=true , address=145.79.227.46:4601 , piece_id=EVWU7M2P2W4FQCNDEQ2OO3SR4KHPTXQRM7GEI6Q6DHYMDHK4PP4Q, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.505904454 (375h37m59.545477584s) [noise=true , address=145.79.227.24:4594 , piece_id=V55Y7DFR7QJWBBQRKSTUWYNEZSCAIWAY2XHAOZOF2H7JDFSX6IBA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.506859502 (375h37m59.546432622s) [noise=true , address=145.79.227.10:4420 , piece_id=ANGUQNIL2D4K2YR7KVJMY37WW7ILRVXSZOB7FRRAKJPD3AGGPGOA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.506958631 (375h37m59.546531761s) [noise=true , address=145.79.227.16:4933 , piece_id=WOBRP7MRD6VVMMXEHKNL4XDSPKQBHVINQD3I4N2SQU2MOSC2ZUCQ, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.507277077 (375h37m59.546850217s) [noise=true , address=145.79.227.7:4308 , piece_id=QEUKJMQDWYMBVKSVNNVDK53P6CZ4X47IINRSWCHAFG7APSFOOAFQ, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.508336856 (375h37m59.547909986s) [noise=true , address=77.90.31.19:4847 , piece_id=6GLTB4OVJTM67VRKRWATCIWE7WVK35IQMQIVOEZMZM6I5DFNRIBA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.513150782 (375h37m59.552723912s) [noise=true , address=145.79.227.12:4770 , piece_id=X5KVN5TA67L4TTD7LVG6N4I54BKNESFAHXRPVIWGDFCEPSHRKXPA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.513934875 (375h37m59.553508006s) [noise=true , address=45.91.116.14:4214 , piece_id=DOY7FC4SJKGAFP3LHSJXKWCRSYI6I6IR2R2FZ47QJNDM4SCAUXQQ, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.514566187 (375h37m59.554139318s) [noise=true , address=145.79.227.16:4939 , piece_id=DLS7J35PB7NXNMAPTVHBZ3VN3U725CSVZZKEYY37HZWQPWCCJO7A, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.515189424 (375h37m59.554762544s) [noise=true , address=145.79.227.25:4637 , piece_id=3SMC4AHCMKJEDXJOJVZ3FJEKW23VZDSWW6MDCENAYJ7HKOPWNHBQ, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.515805377 (375h37m59.555378508s) [noise=true , address=145.79.227.6:4292 , piece_id=BYUJPZSYFTYD4C4F5EBAW6BSFXTILW6EASOIRHZQI7ARCSG63F2Q, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.516494880 (375h37m59.55606801s) [noise=true , address=77.90.31.16:4764 , piece_id=PSAP3HPVNEURWWNC7K2K2G4NEUENKWDUR7OJY6UBGRTCSBYNHRWQ, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.526689722 (375h37m59.566262863s) [noise=true , address=145.79.227.31:4818 , piece_id=KAJZKTKOHNN5U4CTYJ7CLO32ONU6VSEJ2QBLPBI5WWZZ37YOI5NA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.531527154 (375h37m59.571100294s) [noise=true , address=145.79.227.49:4338 , piece_id=NBR62GETN7QBXC7QJCUB5DZHKVMC5AKQQBIIGDRXNECNZFULEDDA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.537921861 (375h37m59.577494991s) [noise=true , address=145.79.227.5:4247 , piece_id=QGN2S5CET2J6LGW6YPQYDWEVOGEMHWRSRMJG2IUQ36KA23OG2R3A, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:24.564378058 (375h38m0.603951188s) [noise=true , address=145.79.227.5:4247 , piece_id=QGN2S5CET2J6LGW6YPQYDWEVOGEMHWRSRMJG2IUQ36KA23OG2R3A, seg_pos={0 0}, upload=330171]: Done uploading piece. err: ecclient: upload failed (node:1262xxX6aUPTYgKdvRxAyc77PmPBWJtq5cXRJFoELxHbfn18DLE, address:145.79.227.5:4247): protocol: rpc: dial tcp 145.79.227.5:4247: connect: no route to host; conn closed: op Sep 22 05:06:24.986206552 (375h38m1.025779682s) [upload=330171]: Next returned nil segment Sep 22 05:06:24.986226830 (375h38m1.02579995s) [upload=330171]: Waiting for error group managing segments... Sep 22 05:06:25.383578176 (375h38m1.423151306s) [noise=true , address=145.79.227.24:4594 , piece_id=V55Y7DFR7QJWBBQRKSTUWYNEZSCAIWAY2XHAOZOF2H7JDFSX6IBA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.383806571 (375h38m1.423379702s) [noise=true , address=145.79.227.33:4881 , piece_id=AIKNNVT5PMYFYKRMUFN5DQMY7TWEUN3FUC64IQQYKAMV723TKKZA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.398811493 (375h38m1.438384634s) [noise=true , address=145.79.227.46:4601 , piece_id=EVWU7M2P2W4FQCNDEQ2OO3SR4KHPTXQRM7GEI6Q6DHYMDHK4PP4Q, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.399530783 (375h38m1.439103913s) [noise=true , address=145.79.227.39:4039 , piece_id=MZRBH2VHR7CCJK4OCCXINOQTNBP7PUP236H7DB2DFL7ZFDBNI2IQ, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.406508421 (375h38m1.446081551s) [noise=true , address=145.79.227.49:4338 , piece_id=NBR62GETN7QBXC7QJCUB5DZHKVMC5AKQQBIIGDRXNECNZFULEDDA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.415736332 (375h38m1.455309462s) [noise=true , address=145.79.227.25:4637 , piece_id=3SMC4AHCMKJEDXJOJVZ3FJEKW23VZDSWW6MDCENAYJ7HKOPWNHBQ, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.437405044 (375h38m1.476978184s) [noise=true , address=145.79.227.31:4818 , piece_id=KAJZKTKOHNN5U4CTYJ7CLO32ONU6VSEJ2QBLPBI5WWZZ37YOI5NA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.465513107 (375h38m1.505086237s) [noise=true , address=77.90.31.19:4847 , piece_id=6GLTB4OVJTM67VRKRWATCIWE7WVK35IQMQIVOEZMZM6I5DFNRIBA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.501349827 (375h38m1.540922957s) [noise=true , address=45.91.116.9:4292 , piece_id=3QGXKEK5C77XDD34Z77C3K2LYEMGJT2SADTL6RGW2YCYQFV5CLUA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.579007182 (375h38m1.618580302s) [noise=true , address=77.90.31.16:4764 , piece_id=PSAP3HPVNEURWWNC7K2K2G4NEUENKWDUR7OJY6UBGRTCSBYNHRWQ, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.586807715 (375h38m1.626380845s) [noise=true , address=45.91.116.14:4214 , piece_id=DOY7FC4SJKGAFP3LHSJXKWCRSYI6I6IR2R2FZ47QJNDM4SCAUXQQ, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:07:22.245417769 (375h38m58.284990899s) [noise=true , address=145.79.227.16:4933 , piece_id=WOBRP7MRD6VVMMXEHKNL4XDSPKQBHVINQD3I4N2SQU2MOSC2ZUCQ, seg_pos={0 0}, upload=330171]: Done uploading piece. err: ecclient: upload failed (node:19m4inaXvTdwf2rKip3eFELv6JAgFvXJ8KqjPZYbb48i8Ktnwe, address:145.79.227.16:4933): protocol: rpc: dial tcp 145.79.227.16:4933: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:07:22.245544310 (375h38m58.28511744s) [noise=true , address=145.79.227.12:4770 , piece_id=X5KVN5TA67L4TTD7LVG6N4I54BKNESFAHXRPVIWGDFCEPSHRKXPA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: ecclient: upload failed (node:1Ug6GpYToYDXiHmiJP4GoE7eJV3kqRveix2S5SLDcKQJtEbNPz, address:145.79.227.12:4770): protocol: rpc: dial tcp 145.79.227.12:4770: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:07:22.245567825 (375h38m58.285140955s) [noise=true , address=145.79.227.16:4939 , piece_id=DLS7J35PB7NXNMAPTVHBZ3VN3U725CSVZZKEYY37HZWQPWCCJO7A, seg_pos={0 0}, upload=330171]: Done uploading piece. err: ecclient: upload failed (node:1mRmp3HhD95PruS4AF8soRsSXNYUeb4mJE7y52FvoeN9D3Gu4F, address:145.79.227.16:4939): protocol: rpc: dial tcp 145.79.227.16:4939: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:07:22.245634181 (375h38m58.285207312s) [noise=true , address=145.79.227.7:4308 , piece_id=QEUKJMQDWYMBVKSVNNVDK53P6CZ4X47IINRSWCHAFG7APSFOOAFQ, seg_pos={0 0}, upload=330171]: Done uploading piece. err: ecclient: upload failed (node:1PgA5SZkBvK673LsV7p2sXpbi1r79esNNZspPZTzaitqfbjtyY, address:145.79.227.7:4308): protocol: rpc: dial tcp 145.79.227.7:4308: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:07:22.245657335 (375h38m58.285230455s) [noise=true , address=145.79.227.10:4420 , piece_id=ANGUQNIL2D4K2YR7KVJMY37WW7ILRVXSZOB7FRRAKJPD3AGGPGOA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: ecclient: upload failed (node:12hJrjeMUMYmFw3yxRtQQKRh9ciPfdu5FXo8NGrsc6HBqgfJzzp, address:145.79.227.10:4420): protocol: rpc: dial tcp 145.79.227.10:4420: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:07:22.245675269 (375h38m58.28524839s) [noise=true , address=145.79.227.6:4292 , piece_id=BYUJPZSYFTYD4C4F5EBAW6BSFXTILW6EASOIRHZQI7ARCSG63F2Q, seg_pos={0 0}, upload=330171]: Done uploading piece. err: ecclient: upload failed (node:12UpWmMhEmk5veKZvzsQ3MCxCMnfEKtFgCugBTsYfy9Ejm7G4xD, address:145.79.227.6:4292): protocol: rpc: dial tcp 145.79.227.6:4292: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:07:22.245712370 (375h38m58.2852855s) [seg_pos={0 0}, upload=330171]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Sep 22 05:07:22.500217377 (375h38m58.539790507s) [upload=330171]: Done uploading segments... Sep 22 05:07:22.500253296 (375h38m58.539826426s) [upload=330171]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 5.869188223s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [......XX....X..X...X.X......X.........X.X..X....X..XX.......] duration: 6.186795501s FAIL (2025-09-22 05:43:06 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.186841078s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [....XXXX.X.XX.....X.X....................X..................] duration: 5.962473308s FAIL (2025-09-22 05:42:49 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== 3.337157461s) [upload=330511]: Starting upload Sep 22 05:29:47.297668972 (376h1m23.337242102s) [upload=330511]: Done starting upload Sep 22 05:29:47.297899231 (376h1m23.337472361s) [upload=330511]: Uploading segments... Sep 22 05:29:47.298002678 (376h1m23.337575818s) [upload=330511]: Got next segment. Inline: false Sep 22 05:29:47.298059145 (376h1m23.337632275s) [upload=330511]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 05:29:47.298066109 (376h1m23.337639229s) [upload=330511]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 05:29:47.552911815 (376h1m23.592484945s) [seg_pos={0 0}, upload=330511]: Begin upload segment... Sep 22 05:29:47.552975406 (376h1m23.592548536s) [seg_pos={0 0}, upload=330511]: Done begin upload segment. Sep 22 05:29:47.553013599 (376h1m23.592586719s) [noise=true , address=45.91.116.11:4117 , piece_id=NIFPLA2N3L6XDWTIFUM3YYRGBMFQJKWFVCGYXD2GBFYVYDXW6OEQ, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.553528389 (376h1m23.593101539s) [noise=true , address=145.79.227.6:4258 , piece_id=D7HMSQH7ZLLKNTUZGJLVHIW2KEQZDL3HSS6VO2BSNRZJAC3USFAQ, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.553925034 (376h1m23.593498154s) [noise=true , address=45.91.116.9:4261 , piece_id=REZZWYP25KZOP4UBJWJPHGCDUWCW3EBOJG2DOLCDYZY5B43KBMSQ, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.553961283 (376h1m23.593534413s) [noise=true , address=77.90.31.14:4663 , piece_id=LYGO7UTA55Q5Q7APPDJZNNWZ4S2XHAT4LS4V27VF27A3435DVJQA, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.554326960 (376h1m23.59390009s) [noise=true , address=77.90.31.16:4735 , piece_id=K5KYKKY3BQV6UVJK4G3DYC24DFFIEFQX7KGGFLS4P35KJRFHA6UQ, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.554351497 (376h1m23.593924627s) [noise=true , address=145.79.227.16:4912 , piece_id=AG2FUP2KFIJ7W55H2VRYTJYWQ57MAXCTEMTDWSR4FUJAYE3HFKBQ, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.554588828 (376h1m23.594161948s) [noise=true , address=145.79.227.12:4767 , piece_id=OUVFSM62B7UZ4U4WGFFBOHCPHXEXR6A6H2FQL3DTG5L3DEPUVYVQ, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.554724957 (376h1m23.594298087s) [noise=true , address=77.90.31.18:4807 , piece_id=JFOMLF75MIXEDP5CKDHH7J3PWGWESB7BIKK3SCAIOVNX74DJDC3A, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.554875986 (376h1m23.594449116s) [noise=true , address=145.79.227.3:4151 , piece_id=O6AU3AOBKJRIUMEAWPHMWWDDNRTA52WL72JNUTK5JFLVG2TNV5IA, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.555093950 (376h1m23.59466708s) [noise=true , address=145.79.227.17:4972 , piece_id=Q3KQQ57XWDUCXNHP33STZ7DN6VQ6FVYZTDPLINEPZASFEVY5XDJA, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.555135429 (376h1m23.594708559s) [noise=true , address=45.91.116.6:4973 , piece_id=WPWUFOFQHQZFRPTR33GTDST2NYCVPXKLDVCJQFPQ4A7PI44LJKOQ, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.555464797 (376h1m23.595037917s) [noise=true , address=45.91.116.14:4189 , piece_id=6VJ6XCCEV2CZSARNRF3ATZ4KCUI75QKWIWQWJL5QH4XTQG4MCLUQ, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.555726194 (376h1m23.595299314s) [noise=true , address=77.90.31.15:4699 , piece_id=6XZN64DOILMELBYDU4D72ER33XDKMRIJRSAXX5ITFYPG46YEBITQ, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.556009483 (376h1m23.595582614s) [noise=true , address=145.79.227.19:4077 , piece_id=EDBYIPZUOPFRRIZBZJJEP2U7NZFCUHRLDZESZ7J5CQZFKADNLE6Q, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.556364609 (376h1m23.59593773s) [noise=true , address=145.79.227.15:4875 , piece_id=ZEWO52263YKTTRYDVF4PV52JTPEDJR3XGVRG3BFE5QQKDQSCE2LQ, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.556611299 (376h1m23.596184429s) [noise=true , address=145.79.227.5:4222 , piece_id=CZNVNWHETTFQS6JR2YLE65LNQU4WX74DAROY5I3KAYHG4QDLDOKA, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:49.308510933 (376h1m25.348084063s) [upload=330511]: Next returned nil segment Sep 22 05:29:49.308533666 (376h1m25.348106796s) [upload=330511]: Waiting for error group managing segments... Sep 22 05:29:49.602590138 (376h1m25.642163268s) [noise=true , address=45.91.116.11:4117 , piece_id=NIFPLA2N3L6XDWTIFUM3YYRGBMFQJKWFVCGYXD2GBFYVYDXW6OEQ, seg_pos={0 0}, upload=330511]: Done uploading piece. err: <nil> Sep 22 05:29:49.633269677 (376h1m25.672842807s) [noise=true , address=77.90.31.15:4699 , piece_id=6XZN64DOILMELBYDU4D72ER33XDKMRIJRSAXX5ITFYPG46YEBITQ, seg_pos={0 0}, upload=330511]: Done uploading piece. err: <nil> Sep 22 05:29:49.637382749 (376h1m25.676955879s) [noise=true , address=45.91.116.6:4973 , piece_id=WPWUFOFQHQZFRPTR33GTDST2NYCVPXKLDVCJQFPQ4A7PI44LJKOQ, seg_pos={0 0}, upload=330511]: Done uploading piece. err: <nil> Sep 22 05:29:49.648543782 (376h1m25.688116902s) [noise=true , address=77.90.31.16:4735 , piece_id=K5KYKKY3BQV6UVJK4G3DYC24DFFIEFQX7KGGFLS4P35KJRFHA6UQ, seg_pos={0 0}, upload=330511]: Done uploading piece. err: <nil> Sep 22 05:29:49.653934647 (376h1m25.693507767s) [noise=true , address=45.91.116.14:4189 , piece_id=6VJ6XCCEV2CZSARNRF3ATZ4KCUI75QKWIWQWJL5QH4XTQG4MCLUQ, seg_pos={0 0}, upload=330511]: Done uploading piece. err: <nil> Sep 22 05:29:49.669096699 (376h1m25.708669829s) [noise=true , address=45.91.116.9:4261 , piece_id=REZZWYP25KZOP4UBJWJPHGCDUWCW3EBOJG2DOLCDYZY5B43KBMSQ, seg_pos={0 0}, upload=330511]: Done uploading piece. err: <nil> Sep 22 05:29:49.673672402 (376h1m25.713245532s) [noise=true , address=77.90.31.14:4663 , piece_id=LYGO7UTA55Q5Q7APPDJZNNWZ4S2XHAT4LS4V27VF27A3435DVJQA, seg_pos={0 0}, upload=330511]: Done uploading piece. err: <nil> Sep 22 05:29:49.741841689 (376h1m25.781414809s) [noise=true , address=77.90.31.18:4807 , piece_id=JFOMLF75MIXEDP5CKDHH7J3PWGWESB7BIKK3SCAIOVNX74DJDC3A, seg_pos={0 0}, upload=330511]: Done uploading piece. err: <nil> Sep 22 05:29:58.021015047 (376h1m34.060588167s) [noise=true , address=145.79.227.5:4222 , piece_id=CZNVNWHETTFQS6JR2YLE65LNQU4WX74DAROY5I3KAYHG4QDLDOKA, seg_pos={0 0}, upload=330511]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:145.79.227.5:4222): protocol: rpc: dial tcp 145.79.227.5:4222: connect: no route to host; conn closed: op; piecestore: piecestore close: manager closed: conn closed: op Sep 22 05:30:05.028605724 (376h1m41.068178854s) [noise=true , address=145.79.227.3:4151 , piece_id=O6AU3AOBKJRIUMEAWPHMWWDDNRTA52WL72JNUTK5JFLVG2TNV5IA, seg_pos={0 0}, upload=330511]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:145.79.227.3:4151): protocol: rpc: dial tcp 145.79.227.3:4151: connect: no route to host; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:30:16.586571260 (376h1m52.62614439s) [noise=true , address=145.79.227.12:4767 , piece_id=OUVFSM62B7UZ4U4WGFFBOHCPHXEXR6A6H2FQL3DTG5L3DEPUVYVQ, seg_pos={0 0}, upload=330511]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:145.79.227.12:4767): protocol: rpc: dial tcp 145.79.227.12:4767: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:30:16.586679978 (376h1m52.626253118s) [noise=true , address=145.79.227.6:4258 , piece_id=D7HMSQH7ZLLKNTUZGJLVHIW2KEQZDL3HSS6VO2BSNRZJAC3USFAQ, seg_pos={0 0}, upload=330511]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:145.79.227.6:4258): protocol: rpc: dial tcp 145.79.227.6:4258: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:30:16.586708171 (376h1m52.626281301s) [noise=true , address=145.79.227.16:4912 , piece_id=AG2FUP2KFIJ7W55H2VRYTJYWQ57MAXCTEMTDWSR4FUJAYE3HFKBQ, seg_pos={0 0}, upload=330511]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:145.79.227.16:4912): protocol: rpc: dial tcp 145.79.227.16:4912: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:30:16.586749971 (376h1m52.626323101s) [noise=true , address=145.79.227.15:4875 , piece_id=ZEWO52263YKTTRYDVF4PV52JTPEDJR3XGVRG3BFE5QQKDQSCE2LQ, seg_pos={0 0}, upload=330511]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: rpc: dial tcp 145.79.227.15:4875: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:30:16.586795768 (376h1m52.626368888s) [noise=true , address=145.79.227.17:4972 , piece_id=Q3KQQ57XWDUCXNHP33STZ7DN6VQ6FVYZTDPLINEPZASFEVY5XDJA, seg_pos={0 0}, upload=330511]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:145.79.227.17:4972): protocol: rpc: dial tcp 145.79.227.17:4972: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:30:16.586806088 (376h1m52.626379218s) [noise=true , address=145.79.227.19:4077 , piece_id=EDBYIPZUOPFRRIZBZJJEP2U7NZFCUHRLDZESZ7J5CQZFKADNLE6Q, seg_pos={0 0}, upload=330511]: Done uploading piece. err: ecclient: upload failed (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk, address:145.79.227.19:4077): protocol: rpc: dial tcp 145.79.227.19:4077: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:30:16.586831426 (376h1m52.626404556s) [seg_pos={0 0}, upload=330511]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Sep 22 05:30:16.834944071 (376h1m52.874517201s) [upload=330511]: Done uploading segments... Sep 22 05:30:16.834981573 (376h1m52.874554703s) [upload=330511]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 5.96250694s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [.......XX....X..X...X.X......X.........X.X..X....X..XX......] duration: 5.986322373s FAIL (2025-09-22 05:42:39 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.986352018s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [........X...X...............................................] duration: 6.078279008s FAIL (2025-09-22 05:42:35 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 5h37m59.543525066s) [seg_pos={0 0}, upload=330171]: Begin upload segment... Sep 22 05:06:23.504018422 (375h37m59.543591552s) [seg_pos={0 0}, upload=330171]: Done begin upload segment. Sep 22 05:06:23.504059370 (375h37m59.543632501s) [noise=true , address=45.91.116.9:4292 , piece_id=3QGXKEK5C77XDD34Z77C3K2LYEMGJT2SADTL6RGW2YCYQFV5CLUA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.504681175 (375h37m59.544254355s) [noise=true , address=145.79.227.39:4039 , piece_id=MZRBH2VHR7CCJK4OCCXINOQTNBP7PUP236H7DB2DFL7ZFDBNI2IQ, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.505278141 (375h37m59.544851301s) [noise=true , address=145.79.227.33:4881 , piece_id=AIKNNVT5PMYFYKRMUFN5DQMY7TWEUN3FUC64IQQYKAMV723TKKZA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.505743808 (375h37m59.545316948s) [noise=true , address=145.79.227.46:4601 , piece_id=EVWU7M2P2W4FQCNDEQ2OO3SR4KHPTXQRM7GEI6Q6DHYMDHK4PP4Q, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.505904454 (375h37m59.545477584s) [noise=true , address=145.79.227.24:4594 , piece_id=V55Y7DFR7QJWBBQRKSTUWYNEZSCAIWAY2XHAOZOF2H7JDFSX6IBA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.506859502 (375h37m59.546432622s) [noise=true , address=145.79.227.10:4420 , piece_id=ANGUQNIL2D4K2YR7KVJMY37WW7ILRVXSZOB7FRRAKJPD3AGGPGOA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.506958631 (375h37m59.546531761s) [noise=true , address=145.79.227.16:4933 , piece_id=WOBRP7MRD6VVMMXEHKNL4XDSPKQBHVINQD3I4N2SQU2MOSC2ZUCQ, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.507277077 (375h37m59.546850217s) [noise=true , address=145.79.227.7:4308 , piece_id=QEUKJMQDWYMBVKSVNNVDK53P6CZ4X47IINRSWCHAFG7APSFOOAFQ, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.508336856 (375h37m59.547909986s) [noise=true , address=77.90.31.19:4847 , piece_id=6GLTB4OVJTM67VRKRWATCIWE7WVK35IQMQIVOEZMZM6I5DFNRIBA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.513150782 (375h37m59.552723912s) [noise=true , address=145.79.227.12:4770 , piece_id=X5KVN5TA67L4TTD7LVG6N4I54BKNESFAHXRPVIWGDFCEPSHRKXPA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.513934875 (375h37m59.553508006s) [noise=true , address=45.91.116.14:4214 , piece_id=DOY7FC4SJKGAFP3LHSJXKWCRSYI6I6IR2R2FZ47QJNDM4SCAUXQQ, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.514566187 (375h37m59.554139318s) [noise=true , address=145.79.227.16:4939 , piece_id=DLS7J35PB7NXNMAPTVHBZ3VN3U725CSVZZKEYY37HZWQPWCCJO7A, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.515189424 (375h37m59.554762544s) [noise=true , address=145.79.227.25:4637 , piece_id=3SMC4AHCMKJEDXJOJVZ3FJEKW23VZDSWW6MDCENAYJ7HKOPWNHBQ, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.515805377 (375h37m59.555378508s) [noise=true , address=145.79.227.6:4292 , piece_id=BYUJPZSYFTYD4C4F5EBAW6BSFXTILW6EASOIRHZQI7ARCSG63F2Q, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.516494880 (375h37m59.55606801s) [noise=true , address=77.90.31.16:4764 , piece_id=PSAP3HPVNEURWWNC7K2K2G4NEUENKWDUR7OJY6UBGRTCSBYNHRWQ, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.526689722 (375h37m59.566262863s) [noise=true , address=145.79.227.31:4818 , piece_id=KAJZKTKOHNN5U4CTYJ7CLO32ONU6VSEJ2QBLPBI5WWZZ37YOI5NA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.531527154 (375h37m59.571100294s) [noise=true , address=145.79.227.49:4338 , piece_id=NBR62GETN7QBXC7QJCUB5DZHKVMC5AKQQBIIGDRXNECNZFULEDDA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.537921861 (375h37m59.577494991s) [noise=true , address=145.79.227.5:4247 , piece_id=QGN2S5CET2J6LGW6YPQYDWEVOGEMHWRSRMJG2IUQ36KA23OG2R3A, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:24.564378058 (375h38m0.603951188s) [noise=true , address=145.79.227.5:4247 , piece_id=QGN2S5CET2J6LGW6YPQYDWEVOGEMHWRSRMJG2IUQ36KA23OG2R3A, seg_pos={0 0}, upload=330171]: Done uploading piece. err: ecclient: upload failed (node:1262xxX6aUPTYgKdvRxAyc77PmPBWJtq5cXRJFoELxHbfn18DLE, address:145.79.227.5:4247): protocol: rpc: dial tcp 145.79.227.5:4247: connect: no route to host; conn closed: op Sep 22 05:06:24.986206552 (375h38m1.025779682s) [upload=330171]: Next returned nil segment Sep 22 05:06:24.986226830 (375h38m1.02579995s) [upload=330171]: Waiting for error group managing segments... Sep 22 05:06:25.383578176 (375h38m1.423151306s) [noise=true , address=145.79.227.24:4594 , piece_id=V55Y7DFR7QJWBBQRKSTUWYNEZSCAIWAY2XHAOZOF2H7JDFSX6IBA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.383806571 (375h38m1.423379702s) [noise=true , address=145.79.227.33:4881 , piece_id=AIKNNVT5PMYFYKRMUFN5DQMY7TWEUN3FUC64IQQYKAMV723TKKZA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.398811493 (375h38m1.438384634s) [noise=true , address=145.79.227.46:4601 , piece_id=EVWU7M2P2W4FQCNDEQ2OO3SR4KHPTXQRM7GEI6Q6DHYMDHK4PP4Q, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.399530783 (375h38m1.439103913s) [noise=true , address=145.79.227.39:4039 , piece_id=MZRBH2VHR7CCJK4OCCXINOQTNBP7PUP236H7DB2DFL7ZFDBNI2IQ, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.406508421 (375h38m1.446081551s) [noise=true , address=145.79.227.49:4338 , piece_id=NBR62GETN7QBXC7QJCUB5DZHKVMC5AKQQBIIGDRXNECNZFULEDDA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.415736332 (375h38m1.455309462s) [noise=true , address=145.79.227.25:4637 , piece_id=3SMC4AHCMKJEDXJOJVZ3FJEKW23VZDSWW6MDCENAYJ7HKOPWNHBQ, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.437405044 (375h38m1.476978184s) [noise=true , address=145.79.227.31:4818 , piece_id=KAJZKTKOHNN5U4CTYJ7CLO32ONU6VSEJ2QBLPBI5WWZZ37YOI5NA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.465513107 (375h38m1.505086237s) [noise=true , address=77.90.31.19:4847 , piece_id=6GLTB4OVJTM67VRKRWATCIWE7WVK35IQMQIVOEZMZM6I5DFNRIBA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.501349827 (375h38m1.540922957s) [noise=true , address=45.91.116.9:4292 , piece_id=3QGXKEK5C77XDD34Z77C3K2LYEMGJT2SADTL6RGW2YCYQFV5CLUA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.579007182 (375h38m1.618580302s) [noise=true , address=77.90.31.16:4764 , piece_id=PSAP3HPVNEURWWNC7K2K2G4NEUENKWDUR7OJY6UBGRTCSBYNHRWQ, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.586807715 (375h38m1.626380845s) [noise=true , address=45.91.116.14:4214 , piece_id=DOY7FC4SJKGAFP3LHSJXKWCRSYI6I6IR2R2FZ47QJNDM4SCAUXQQ, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:07:22.245417769 (375h38m58.284990899s) [noise=true , address=145.79.227.16:4933 , piece_id=WOBRP7MRD6VVMMXEHKNL4XDSPKQBHVINQD3I4N2SQU2MOSC2ZUCQ, seg_pos={0 0}, upload=330171]: Done uploading piece. err: ecclient: upload failed (node:19m4inaXvTdwf2rKip3eFELv6JAgFvXJ8KqjPZYbb48i8Ktnwe, address:145.79.227.16:4933): protocol: rpc: dial tcp 145.79.227.16:4933: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:07:22.245544310 (375h38m58.28511744s) [noise=true , address=145.79.227.12:4770 , piece_id=X5KVN5TA67L4TTD7LVG6N4I54BKNESFAHXRPVIWGDFCEPSHRKXPA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: ecclient: upload failed (node:1Ug6GpYToYDXiHmiJP4GoE7eJV3kqRveix2S5SLDcKQJtEbNPz, address:145.79.227.12:4770): protocol: rpc: dial tcp 145.79.227.12:4770: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:07:22.245567825 (375h38m58.285140955s) [noise=true , address=145.79.227.16:4939 , piece_id=DLS7J35PB7NXNMAPTVHBZ3VN3U725CSVZZKEYY37HZWQPWCCJO7A, seg_pos={0 0}, upload=330171]: Done uploading piece. err: ecclient: upload failed (node:1mRmp3HhD95PruS4AF8soRsSXNYUeb4mJE7y52FvoeN9D3Gu4F, address:145.79.227.16:4939): protocol: rpc: dial tcp 145.79.227.16:4939: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:07:22.245634181 (375h38m58.285207312s) [noise=true , address=145.79.227.7:4308 , piece_id=QEUKJMQDWYMBVKSVNNVDK53P6CZ4X47IINRSWCHAFG7APSFOOAFQ, seg_pos={0 0}, upload=330171]: Done uploading piece. err: ecclient: upload failed (node:1PgA5SZkBvK673LsV7p2sXpbi1r79esNNZspPZTzaitqfbjtyY, address:145.79.227.7:4308): protocol: rpc: dial tcp 145.79.227.7:4308: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:07:22.245657335 (375h38m58.285230455s) [noise=true , address=145.79.227.10:4420 , piece_id=ANGUQNIL2D4K2YR7KVJMY37WW7ILRVXSZOB7FRRAKJPD3AGGPGOA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: ecclient: upload failed (node:12hJrjeMUMYmFw3yxRtQQKRh9ciPfdu5FXo8NGrsc6HBqgfJzzp, address:145.79.227.10:4420): protocol: rpc: dial tcp 145.79.227.10:4420: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:07:22.245675269 (375h38m58.28524839s) [noise=true , address=145.79.227.6:4292 , piece_id=BYUJPZSYFTYD4C4F5EBAW6BSFXTILW6EASOIRHZQI7ARCSG63F2Q, seg_pos={0 0}, upload=330171]: Done uploading piece. err: ecclient: upload failed (node:12UpWmMhEmk5veKZvzsQ3MCxCMnfEKtFgCugBTsYfy9Ejm7G4xD, address:145.79.227.6:4292): protocol: rpc: dial tcp 145.79.227.6:4292: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:07:22.245712370 (375h38m58.2852855s) [seg_pos={0 0}, upload=330171]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Sep 22 05:07:22.500217377 (375h38m58.539790507s) [upload=330171]: Done uploading segments... Sep 22 05:07:22.500253296 (375h38m58.539826426s) [upload=330171]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 6.078313583s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [.....XXXX.X.XX.....X.X....................X.................] duration: 6.022250406s FAIL (2025-09-22 05:42:22 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== 3.337157461s) [upload=330511]: Starting upload Sep 22 05:29:47.297668972 (376h1m23.337242102s) [upload=330511]: Done starting upload Sep 22 05:29:47.297899231 (376h1m23.337472361s) [upload=330511]: Uploading segments... Sep 22 05:29:47.298002678 (376h1m23.337575818s) [upload=330511]: Got next segment. Inline: false Sep 22 05:29:47.298059145 (376h1m23.337632275s) [upload=330511]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 05:29:47.298066109 (376h1m23.337639229s) [upload=330511]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 05:29:47.552911815 (376h1m23.592484945s) [seg_pos={0 0}, upload=330511]: Begin upload segment... Sep 22 05:29:47.552975406 (376h1m23.592548536s) [seg_pos={0 0}, upload=330511]: Done begin upload segment. Sep 22 05:29:47.553013599 (376h1m23.592586719s) [noise=true , address=45.91.116.11:4117 , piece_id=NIFPLA2N3L6XDWTIFUM3YYRGBMFQJKWFVCGYXD2GBFYVYDXW6OEQ, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.553528389 (376h1m23.593101539s) [noise=true , address=145.79.227.6:4258 , piece_id=D7HMSQH7ZLLKNTUZGJLVHIW2KEQZDL3HSS6VO2BSNRZJAC3USFAQ, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.553925034 (376h1m23.593498154s) [noise=true , address=45.91.116.9:4261 , piece_id=REZZWYP25KZOP4UBJWJPHGCDUWCW3EBOJG2DOLCDYZY5B43KBMSQ, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.553961283 (376h1m23.593534413s) [noise=true , address=77.90.31.14:4663 , piece_id=LYGO7UTA55Q5Q7APPDJZNNWZ4S2XHAT4LS4V27VF27A3435DVJQA, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.554326960 (376h1m23.59390009s) [noise=true , address=77.90.31.16:4735 , piece_id=K5KYKKY3BQV6UVJK4G3DYC24DFFIEFQX7KGGFLS4P35KJRFHA6UQ, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.554351497 (376h1m23.593924627s) [noise=true , address=145.79.227.16:4912 , piece_id=AG2FUP2KFIJ7W55H2VRYTJYWQ57MAXCTEMTDWSR4FUJAYE3HFKBQ, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.554588828 (376h1m23.594161948s) [noise=true , address=145.79.227.12:4767 , piece_id=OUVFSM62B7UZ4U4WGFFBOHCPHXEXR6A6H2FQL3DTG5L3DEPUVYVQ, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.554724957 (376h1m23.594298087s) [noise=true , address=77.90.31.18:4807 , piece_id=JFOMLF75MIXEDP5CKDHH7J3PWGWESB7BIKK3SCAIOVNX74DJDC3A, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.554875986 (376h1m23.594449116s) [noise=true , address=145.79.227.3:4151 , piece_id=O6AU3AOBKJRIUMEAWPHMWWDDNRTA52WL72JNUTK5JFLVG2TNV5IA, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.555093950 (376h1m23.59466708s) [noise=true , address=145.79.227.17:4972 , piece_id=Q3KQQ57XWDUCXNHP33STZ7DN6VQ6FVYZTDPLINEPZASFEVY5XDJA, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.555135429 (376h1m23.594708559s) [noise=true , address=45.91.116.6:4973 , piece_id=WPWUFOFQHQZFRPTR33GTDST2NYCVPXKLDVCJQFPQ4A7PI44LJKOQ, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.555464797 (376h1m23.595037917s) [noise=true , address=45.91.116.14:4189 , piece_id=6VJ6XCCEV2CZSARNRF3ATZ4KCUI75QKWIWQWJL5QH4XTQG4MCLUQ, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.555726194 (376h1m23.595299314s) [noise=true , address=77.90.31.15:4699 , piece_id=6XZN64DOILMELBYDU4D72ER33XDKMRIJRSAXX5ITFYPG46YEBITQ, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.556009483 (376h1m23.595582614s) [noise=true , address=145.79.227.19:4077 , piece_id=EDBYIPZUOPFRRIZBZJJEP2U7NZFCUHRLDZESZ7J5CQZFKADNLE6Q, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.556364609 (376h1m23.59593773s) [noise=true , address=145.79.227.15:4875 , piece_id=ZEWO52263YKTTRYDVF4PV52JTPEDJR3XGVRG3BFE5QQKDQSCE2LQ, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.556611299 (376h1m23.596184429s) [noise=true , address=145.79.227.5:4222 , piece_id=CZNVNWHETTFQS6JR2YLE65LNQU4WX74DAROY5I3KAYHG4QDLDOKA, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:49.308510933 (376h1m25.348084063s) [upload=330511]: Next returned nil segment Sep 22 05:29:49.308533666 (376h1m25.348106796s) [upload=330511]: Waiting for error group managing segments... Sep 22 05:29:49.602590138 (376h1m25.642163268s) [noise=true , address=45.91.116.11:4117 , piece_id=NIFPLA2N3L6XDWTIFUM3YYRGBMFQJKWFVCGYXD2GBFYVYDXW6OEQ, seg_pos={0 0}, upload=330511]: Done uploading piece. err: <nil> Sep 22 05:29:49.633269677 (376h1m25.672842807s) [noise=true , address=77.90.31.15:4699 , piece_id=6XZN64DOILMELBYDU4D72ER33XDKMRIJRSAXX5ITFYPG46YEBITQ, seg_pos={0 0}, upload=330511]: Done uploading piece. err: <nil> Sep 22 05:29:49.637382749 (376h1m25.676955879s) [noise=true , address=45.91.116.6:4973 , piece_id=WPWUFOFQHQZFRPTR33GTDST2NYCVPXKLDVCJQFPQ4A7PI44LJKOQ, seg_pos={0 0}, upload=330511]: Done uploading piece. err: <nil> Sep 22 05:29:49.648543782 (376h1m25.688116902s) [noise=true , address=77.90.31.16:4735 , piece_id=K5KYKKY3BQV6UVJK4G3DYC24DFFIEFQX7KGGFLS4P35KJRFHA6UQ, seg_pos={0 0}, upload=330511]: Done uploading piece. err: <nil> Sep 22 05:29:49.653934647 (376h1m25.693507767s) [noise=true , address=45.91.116.14:4189 , piece_id=6VJ6XCCEV2CZSARNRF3ATZ4KCUI75QKWIWQWJL5QH4XTQG4MCLUQ, seg_pos={0 0}, upload=330511]: Done uploading piece. err: <nil> Sep 22 05:29:49.669096699 (376h1m25.708669829s) [noise=true , address=45.91.116.9:4261 , piece_id=REZZWYP25KZOP4UBJWJPHGCDUWCW3EBOJG2DOLCDYZY5B43KBMSQ, seg_pos={0 0}, upload=330511]: Done uploading piece. err: <nil> Sep 22 05:29:49.673672402 (376h1m25.713245532s) [noise=true , address=77.90.31.14:4663 , piece_id=LYGO7UTA55Q5Q7APPDJZNNWZ4S2XHAT4LS4V27VF27A3435DVJQA, seg_pos={0 0}, upload=330511]: Done uploading piece. err: <nil> Sep 22 05:29:49.741841689 (376h1m25.781414809s) [noise=true , address=77.90.31.18:4807 , piece_id=JFOMLF75MIXEDP5CKDHH7J3PWGWESB7BIKK3SCAIOVNX74DJDC3A, seg_pos={0 0}, upload=330511]: Done uploading piece. err: <nil> Sep 22 05:29:58.021015047 (376h1m34.060588167s) [noise=true , address=145.79.227.5:4222 , piece_id=CZNVNWHETTFQS6JR2YLE65LNQU4WX74DAROY5I3KAYHG4QDLDOKA, seg_pos={0 0}, upload=330511]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:145.79.227.5:4222): protocol: rpc: dial tcp 145.79.227.5:4222: connect: no route to host; conn closed: op; piecestore: piecestore close: manager closed: conn closed: op Sep 22 05:30:05.028605724 (376h1m41.068178854s) [noise=true , address=145.79.227.3:4151 , piece_id=O6AU3AOBKJRIUMEAWPHMWWDDNRTA52WL72JNUTK5JFLVG2TNV5IA, seg_pos={0 0}, upload=330511]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:145.79.227.3:4151): protocol: rpc: dial tcp 145.79.227.3:4151: connect: no route to host; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:30:16.586571260 (376h1m52.62614439s) [noise=true , address=145.79.227.12:4767 , piece_id=OUVFSM62B7UZ4U4WGFFBOHCPHXEXR6A6H2FQL3DTG5L3DEPUVYVQ, seg_pos={0 0}, upload=330511]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:145.79.227.12:4767): protocol: rpc: dial tcp 145.79.227.12:4767: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:30:16.586679978 (376h1m52.626253118s) [noise=true , address=145.79.227.6:4258 , piece_id=D7HMSQH7ZLLKNTUZGJLVHIW2KEQZDL3HSS6VO2BSNRZJAC3USFAQ, seg_pos={0 0}, upload=330511]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:145.79.227.6:4258): protocol: rpc: dial tcp 145.79.227.6:4258: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:30:16.586708171 (376h1m52.626281301s) [noise=true , address=145.79.227.16:4912 , piece_id=AG2FUP2KFIJ7W55H2VRYTJYWQ57MAXCTEMTDWSR4FUJAYE3HFKBQ, seg_pos={0 0}, upload=330511]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:145.79.227.16:4912): protocol: rpc: dial tcp 145.79.227.16:4912: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:30:16.586749971 (376h1m52.626323101s) [noise=true , address=145.79.227.15:4875 , piece_id=ZEWO52263YKTTRYDVF4PV52JTPEDJR3XGVRG3BFE5QQKDQSCE2LQ, seg_pos={0 0}, upload=330511]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: rpc: dial tcp 145.79.227.15:4875: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:30:16.586795768 (376h1m52.626368888s) [noise=true , address=145.79.227.17:4972 , piece_id=Q3KQQ57XWDUCXNHP33STZ7DN6VQ6FVYZTDPLINEPZASFEVY5XDJA, seg_pos={0 0}, upload=330511]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:145.79.227.17:4972): protocol: rpc: dial tcp 145.79.227.17:4972: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:30:16.586806088 (376h1m52.626379218s) [noise=true , address=145.79.227.19:4077 , piece_id=EDBYIPZUOPFRRIZBZJJEP2U7NZFCUHRLDZESZ7J5CQZFKADNLE6Q, seg_pos={0 0}, upload=330511]: Done uploading piece. err: ecclient: upload failed (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk, address:145.79.227.19:4077): protocol: rpc: dial tcp 145.79.227.19:4077: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:30:16.586831426 (376h1m52.626404556s) [seg_pos={0 0}, upload=330511]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Sep 22 05:30:16.834944071 (376h1m52.874517201s) [upload=330511]: Done uploading segments... Sep 22 05:30:16.834981573 (376h1m52.874554703s) [upload=330511]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 6.022279842s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........X...X..............................................] duration: 5.899352003s FAIL (2025-09-22 05:42:08 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 5h37m59.543525066s) [seg_pos={0 0}, upload=330171]: Begin upload segment... Sep 22 05:06:23.504018422 (375h37m59.543591552s) [seg_pos={0 0}, upload=330171]: Done begin upload segment. Sep 22 05:06:23.504059370 (375h37m59.543632501s) [noise=true , address=45.91.116.9:4292 , piece_id=3QGXKEK5C77XDD34Z77C3K2LYEMGJT2SADTL6RGW2YCYQFV5CLUA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.504681175 (375h37m59.544254355s) [noise=true , address=145.79.227.39:4039 , piece_id=MZRBH2VHR7CCJK4OCCXINOQTNBP7PUP236H7DB2DFL7ZFDBNI2IQ, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.505278141 (375h37m59.544851301s) [noise=true , address=145.79.227.33:4881 , piece_id=AIKNNVT5PMYFYKRMUFN5DQMY7TWEUN3FUC64IQQYKAMV723TKKZA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.505743808 (375h37m59.545316948s) [noise=true , address=145.79.227.46:4601 , piece_id=EVWU7M2P2W4FQCNDEQ2OO3SR4KHPTXQRM7GEI6Q6DHYMDHK4PP4Q, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.505904454 (375h37m59.545477584s) [noise=true , address=145.79.227.24:4594 , piece_id=V55Y7DFR7QJWBBQRKSTUWYNEZSCAIWAY2XHAOZOF2H7JDFSX6IBA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.506859502 (375h37m59.546432622s) [noise=true , address=145.79.227.10:4420 , piece_id=ANGUQNIL2D4K2YR7KVJMY37WW7ILRVXSZOB7FRRAKJPD3AGGPGOA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.506958631 (375h37m59.546531761s) [noise=true , address=145.79.227.16:4933 , piece_id=WOBRP7MRD6VVMMXEHKNL4XDSPKQBHVINQD3I4N2SQU2MOSC2ZUCQ, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.507277077 (375h37m59.546850217s) [noise=true , address=145.79.227.7:4308 , piece_id=QEUKJMQDWYMBVKSVNNVDK53P6CZ4X47IINRSWCHAFG7APSFOOAFQ, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.508336856 (375h37m59.547909986s) [noise=true , address=77.90.31.19:4847 , piece_id=6GLTB4OVJTM67VRKRWATCIWE7WVK35IQMQIVOEZMZM6I5DFNRIBA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.513150782 (375h37m59.552723912s) [noise=true , address=145.79.227.12:4770 , piece_id=X5KVN5TA67L4TTD7LVG6N4I54BKNESFAHXRPVIWGDFCEPSHRKXPA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.513934875 (375h37m59.553508006s) [noise=true , address=45.91.116.14:4214 , piece_id=DOY7FC4SJKGAFP3LHSJXKWCRSYI6I6IR2R2FZ47QJNDM4SCAUXQQ, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.514566187 (375h37m59.554139318s) [noise=true , address=145.79.227.16:4939 , piece_id=DLS7J35PB7NXNMAPTVHBZ3VN3U725CSVZZKEYY37HZWQPWCCJO7A, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.515189424 (375h37m59.554762544s) [noise=true , address=145.79.227.25:4637 , piece_id=3SMC4AHCMKJEDXJOJVZ3FJEKW23VZDSWW6MDCENAYJ7HKOPWNHBQ, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.515805377 (375h37m59.555378508s) [noise=true , address=145.79.227.6:4292 , piece_id=BYUJPZSYFTYD4C4F5EBAW6BSFXTILW6EASOIRHZQI7ARCSG63F2Q, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.516494880 (375h37m59.55606801s) [noise=true , address=77.90.31.16:4764 , piece_id=PSAP3HPVNEURWWNC7K2K2G4NEUENKWDUR7OJY6UBGRTCSBYNHRWQ, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.526689722 (375h37m59.566262863s) [noise=true , address=145.79.227.31:4818 , piece_id=KAJZKTKOHNN5U4CTYJ7CLO32ONU6VSEJ2QBLPBI5WWZZ37YOI5NA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.531527154 (375h37m59.571100294s) [noise=true , address=145.79.227.49:4338 , piece_id=NBR62GETN7QBXC7QJCUB5DZHKVMC5AKQQBIIGDRXNECNZFULEDDA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.537921861 (375h37m59.577494991s) [noise=true , address=145.79.227.5:4247 , piece_id=QGN2S5CET2J6LGW6YPQYDWEVOGEMHWRSRMJG2IUQ36KA23OG2R3A, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:24.564378058 (375h38m0.603951188s) [noise=true , address=145.79.227.5:4247 , piece_id=QGN2S5CET2J6LGW6YPQYDWEVOGEMHWRSRMJG2IUQ36KA23OG2R3A, seg_pos={0 0}, upload=330171]: Done uploading piece. err: ecclient: upload failed (node:1262xxX6aUPTYgKdvRxAyc77PmPBWJtq5cXRJFoELxHbfn18DLE, address:145.79.227.5:4247): protocol: rpc: dial tcp 145.79.227.5:4247: connect: no route to host; conn closed: op Sep 22 05:06:24.986206552 (375h38m1.025779682s) [upload=330171]: Next returned nil segment Sep 22 05:06:24.986226830 (375h38m1.02579995s) [upload=330171]: Waiting for error group managing segments... Sep 22 05:06:25.383578176 (375h38m1.423151306s) [noise=true , address=145.79.227.24:4594 , piece_id=V55Y7DFR7QJWBBQRKSTUWYNEZSCAIWAY2XHAOZOF2H7JDFSX6IBA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.383806571 (375h38m1.423379702s) [noise=true , address=145.79.227.33:4881 , piece_id=AIKNNVT5PMYFYKRMUFN5DQMY7TWEUN3FUC64IQQYKAMV723TKKZA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.398811493 (375h38m1.438384634s) [noise=true , address=145.79.227.46:4601 , piece_id=EVWU7M2P2W4FQCNDEQ2OO3SR4KHPTXQRM7GEI6Q6DHYMDHK4PP4Q, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.399530783 (375h38m1.439103913s) [noise=true , address=145.79.227.39:4039 , piece_id=MZRBH2VHR7CCJK4OCCXINOQTNBP7PUP236H7DB2DFL7ZFDBNI2IQ, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.406508421 (375h38m1.446081551s) [noise=true , address=145.79.227.49:4338 , piece_id=NBR62GETN7QBXC7QJCUB5DZHKVMC5AKQQBIIGDRXNECNZFULEDDA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.415736332 (375h38m1.455309462s) [noise=true , address=145.79.227.25:4637 , piece_id=3SMC4AHCMKJEDXJOJVZ3FJEKW23VZDSWW6MDCENAYJ7HKOPWNHBQ, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.437405044 (375h38m1.476978184s) [noise=true , address=145.79.227.31:4818 , piece_id=KAJZKTKOHNN5U4CTYJ7CLO32ONU6VSEJ2QBLPBI5WWZZ37YOI5NA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.465513107 (375h38m1.505086237s) [noise=true , address=77.90.31.19:4847 , piece_id=6GLTB4OVJTM67VRKRWATCIWE7WVK35IQMQIVOEZMZM6I5DFNRIBA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.501349827 (375h38m1.540922957s) [noise=true , address=45.91.116.9:4292 , piece_id=3QGXKEK5C77XDD34Z77C3K2LYEMGJT2SADTL6RGW2YCYQFV5CLUA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.579007182 (375h38m1.618580302s) [noise=true , address=77.90.31.16:4764 , piece_id=PSAP3HPVNEURWWNC7K2K2G4NEUENKWDUR7OJY6UBGRTCSBYNHRWQ, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.586807715 (375h38m1.626380845s) [noise=true , address=45.91.116.14:4214 , piece_id=DOY7FC4SJKGAFP3LHSJXKWCRSYI6I6IR2R2FZ47QJNDM4SCAUXQQ, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:07:22.245417769 (375h38m58.284990899s) [noise=true , address=145.79.227.16:4933 , piece_id=WOBRP7MRD6VVMMXEHKNL4XDSPKQBHVINQD3I4N2SQU2MOSC2ZUCQ, seg_pos={0 0}, upload=330171]: Done uploading piece. err: ecclient: upload failed (node:19m4inaXvTdwf2rKip3eFELv6JAgFvXJ8KqjPZYbb48i8Ktnwe, address:145.79.227.16:4933): protocol: rpc: dial tcp 145.79.227.16:4933: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:07:22.245544310 (375h38m58.28511744s) [noise=true , address=145.79.227.12:4770 , piece_id=X5KVN5TA67L4TTD7LVG6N4I54BKNESFAHXRPVIWGDFCEPSHRKXPA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: ecclient: upload failed (node:1Ug6GpYToYDXiHmiJP4GoE7eJV3kqRveix2S5SLDcKQJtEbNPz, address:145.79.227.12:4770): protocol: rpc: dial tcp 145.79.227.12:4770: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:07:22.245567825 (375h38m58.285140955s) [noise=true , address=145.79.227.16:4939 , piece_id=DLS7J35PB7NXNMAPTVHBZ3VN3U725CSVZZKEYY37HZWQPWCCJO7A, seg_pos={0 0}, upload=330171]: Done uploading piece. err: ecclient: upload failed (node:1mRmp3HhD95PruS4AF8soRsSXNYUeb4mJE7y52FvoeN9D3Gu4F, address:145.79.227.16:4939): protocol: rpc: dial tcp 145.79.227.16:4939: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:07:22.245634181 (375h38m58.285207312s) [noise=true , address=145.79.227.7:4308 , piece_id=QEUKJMQDWYMBVKSVNNVDK53P6CZ4X47IINRSWCHAFG7APSFOOAFQ, seg_pos={0 0}, upload=330171]: Done uploading piece. err: ecclient: upload failed (node:1PgA5SZkBvK673LsV7p2sXpbi1r79esNNZspPZTzaitqfbjtyY, address:145.79.227.7:4308): protocol: rpc: dial tcp 145.79.227.7:4308: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:07:22.245657335 (375h38m58.285230455s) [noise=true , address=145.79.227.10:4420 , piece_id=ANGUQNIL2D4K2YR7KVJMY37WW7ILRVXSZOB7FRRAKJPD3AGGPGOA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: ecclient: upload failed (node:12hJrjeMUMYmFw3yxRtQQKRh9ciPfdu5FXo8NGrsc6HBqgfJzzp, address:145.79.227.10:4420): protocol: rpc: dial tcp 145.79.227.10:4420: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:07:22.245675269 (375h38m58.28524839s) [noise=true , address=145.79.227.6:4292 , piece_id=BYUJPZSYFTYD4C4F5EBAW6BSFXTILW6EASOIRHZQI7ARCSG63F2Q, seg_pos={0 0}, upload=330171]: Done uploading piece. err: ecclient: upload failed (node:12UpWmMhEmk5veKZvzsQ3MCxCMnfEKtFgCugBTsYfy9Ejm7G4xD, address:145.79.227.6:4292): protocol: rpc: dial tcp 145.79.227.6:4292: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:07:22.245712370 (375h38m58.2852855s) [seg_pos={0 0}, upload=330171]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Sep 22 05:07:22.500217377 (375h38m58.539790507s) [upload=330171]: Done uploading segments... Sep 22 05:07:22.500253296 (375h38m58.539826426s) [upload=330171]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 5.899470719s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [......XXXX.X.XX.....X.X....................X................] duration: 6.025905648s FAIL (2025-09-22 05:41:57 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== 3.337157461s) [upload=330511]: Starting upload Sep 22 05:29:47.297668972 (376h1m23.337242102s) [upload=330511]: Done starting upload Sep 22 05:29:47.297899231 (376h1m23.337472361s) [upload=330511]: Uploading segments... Sep 22 05:29:47.298002678 (376h1m23.337575818s) [upload=330511]: Got next segment. Inline: false Sep 22 05:29:47.298059145 (376h1m23.337632275s) [upload=330511]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 05:29:47.298066109 (376h1m23.337639229s) [upload=330511]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 05:29:47.552911815 (376h1m23.592484945s) [seg_pos={0 0}, upload=330511]: Begin upload segment... Sep 22 05:29:47.552975406 (376h1m23.592548536s) [seg_pos={0 0}, upload=330511]: Done begin upload segment. Sep 22 05:29:47.553013599 (376h1m23.592586719s) [noise=true , address=45.91.116.11:4117 , piece_id=NIFPLA2N3L6XDWTIFUM3YYRGBMFQJKWFVCGYXD2GBFYVYDXW6OEQ, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.553528389 (376h1m23.593101539s) [noise=true , address=145.79.227.6:4258 , piece_id=D7HMSQH7ZLLKNTUZGJLVHIW2KEQZDL3HSS6VO2BSNRZJAC3USFAQ, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.553925034 (376h1m23.593498154s) [noise=true , address=45.91.116.9:4261 , piece_id=REZZWYP25KZOP4UBJWJPHGCDUWCW3EBOJG2DOLCDYZY5B43KBMSQ, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.553961283 (376h1m23.593534413s) [noise=true , address=77.90.31.14:4663 , piece_id=LYGO7UTA55Q5Q7APPDJZNNWZ4S2XHAT4LS4V27VF27A3435DVJQA, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.554326960 (376h1m23.59390009s) [noise=true , address=77.90.31.16:4735 , piece_id=K5KYKKY3BQV6UVJK4G3DYC24DFFIEFQX7KGGFLS4P35KJRFHA6UQ, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.554351497 (376h1m23.593924627s) [noise=true , address=145.79.227.16:4912 , piece_id=AG2FUP2KFIJ7W55H2VRYTJYWQ57MAXCTEMTDWSR4FUJAYE3HFKBQ, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.554588828 (376h1m23.594161948s) [noise=true , address=145.79.227.12:4767 , piece_id=OUVFSM62B7UZ4U4WGFFBOHCPHXEXR6A6H2FQL3DTG5L3DEPUVYVQ, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.554724957 (376h1m23.594298087s) [noise=true , address=77.90.31.18:4807 , piece_id=JFOMLF75MIXEDP5CKDHH7J3PWGWESB7BIKK3SCAIOVNX74DJDC3A, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.554875986 (376h1m23.594449116s) [noise=true , address=145.79.227.3:4151 , piece_id=O6AU3AOBKJRIUMEAWPHMWWDDNRTA52WL72JNUTK5JFLVG2TNV5IA, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.555093950 (376h1m23.59466708s) [noise=true , address=145.79.227.17:4972 , piece_id=Q3KQQ57XWDUCXNHP33STZ7DN6VQ6FVYZTDPLINEPZASFEVY5XDJA, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.555135429 (376h1m23.594708559s) [noise=true , address=45.91.116.6:4973 , piece_id=WPWUFOFQHQZFRPTR33GTDST2NYCVPXKLDVCJQFPQ4A7PI44LJKOQ, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.555464797 (376h1m23.595037917s) [noise=true , address=45.91.116.14:4189 , piece_id=6VJ6XCCEV2CZSARNRF3ATZ4KCUI75QKWIWQWJL5QH4XTQG4MCLUQ, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.555726194 (376h1m23.595299314s) [noise=true , address=77.90.31.15:4699 , piece_id=6XZN64DOILMELBYDU4D72ER33XDKMRIJRSAXX5ITFYPG46YEBITQ, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.556009483 (376h1m23.595582614s) [noise=true , address=145.79.227.19:4077 , piece_id=EDBYIPZUOPFRRIZBZJJEP2U7NZFCUHRLDZESZ7J5CQZFKADNLE6Q, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.556364609 (376h1m23.59593773s) [noise=true , address=145.79.227.15:4875 , piece_id=ZEWO52263YKTTRYDVF4PV52JTPEDJR3XGVRG3BFE5QQKDQSCE2LQ, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.556611299 (376h1m23.596184429s) [noise=true , address=145.79.227.5:4222 , piece_id=CZNVNWHETTFQS6JR2YLE65LNQU4WX74DAROY5I3KAYHG4QDLDOKA, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:49.308510933 (376h1m25.348084063s) [upload=330511]: Next returned nil segment Sep 22 05:29:49.308533666 (376h1m25.348106796s) [upload=330511]: Waiting for error group managing segments... Sep 22 05:29:49.602590138 (376h1m25.642163268s) [noise=true , address=45.91.116.11:4117 , piece_id=NIFPLA2N3L6XDWTIFUM3YYRGBMFQJKWFVCGYXD2GBFYVYDXW6OEQ, seg_pos={0 0}, upload=330511]: Done uploading piece. err: <nil> Sep 22 05:29:49.633269677 (376h1m25.672842807s) [noise=true , address=77.90.31.15:4699 , piece_id=6XZN64DOILMELBYDU4D72ER33XDKMRIJRSAXX5ITFYPG46YEBITQ, seg_pos={0 0}, upload=330511]: Done uploading piece. err: <nil> Sep 22 05:29:49.637382749 (376h1m25.676955879s) [noise=true , address=45.91.116.6:4973 , piece_id=WPWUFOFQHQZFRPTR33GTDST2NYCVPXKLDVCJQFPQ4A7PI44LJKOQ, seg_pos={0 0}, upload=330511]: Done uploading piece. err: <nil> Sep 22 05:29:49.648543782 (376h1m25.688116902s) [noise=true , address=77.90.31.16:4735 , piece_id=K5KYKKY3BQV6UVJK4G3DYC24DFFIEFQX7KGGFLS4P35KJRFHA6UQ, seg_pos={0 0}, upload=330511]: Done uploading piece. err: <nil> Sep 22 05:29:49.653934647 (376h1m25.693507767s) [noise=true , address=45.91.116.14:4189 , piece_id=6VJ6XCCEV2CZSARNRF3ATZ4KCUI75QKWIWQWJL5QH4XTQG4MCLUQ, seg_pos={0 0}, upload=330511]: Done uploading piece. err: <nil> Sep 22 05:29:49.669096699 (376h1m25.708669829s) [noise=true , address=45.91.116.9:4261 , piece_id=REZZWYP25KZOP4UBJWJPHGCDUWCW3EBOJG2DOLCDYZY5B43KBMSQ, seg_pos={0 0}, upload=330511]: Done uploading piece. err: <nil> Sep 22 05:29:49.673672402 (376h1m25.713245532s) [noise=true , address=77.90.31.14:4663 , piece_id=LYGO7UTA55Q5Q7APPDJZNNWZ4S2XHAT4LS4V27VF27A3435DVJQA, seg_pos={0 0}, upload=330511]: Done uploading piece. err: <nil> Sep 22 05:29:49.741841689 (376h1m25.781414809s) [noise=true , address=77.90.31.18:4807 , piece_id=JFOMLF75MIXEDP5CKDHH7J3PWGWESB7BIKK3SCAIOVNX74DJDC3A, seg_pos={0 0}, upload=330511]: Done uploading piece. err: <nil> Sep 22 05:29:58.021015047 (376h1m34.060588167s) [noise=true , address=145.79.227.5:4222 , piece_id=CZNVNWHETTFQS6JR2YLE65LNQU4WX74DAROY5I3KAYHG4QDLDOKA, seg_pos={0 0}, upload=330511]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:145.79.227.5:4222): protocol: rpc: dial tcp 145.79.227.5:4222: connect: no route to host; conn closed: op; piecestore: piecestore close: manager closed: conn closed: op Sep 22 05:30:05.028605724 (376h1m41.068178854s) [noise=true , address=145.79.227.3:4151 , piece_id=O6AU3AOBKJRIUMEAWPHMWWDDNRTA52WL72JNUTK5JFLVG2TNV5IA, seg_pos={0 0}, upload=330511]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:145.79.227.3:4151): protocol: rpc: dial tcp 145.79.227.3:4151: connect: no route to host; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:30:16.586571260 (376h1m52.62614439s) [noise=true , address=145.79.227.12:4767 , piece_id=OUVFSM62B7UZ4U4WGFFBOHCPHXEXR6A6H2FQL3DTG5L3DEPUVYVQ, seg_pos={0 0}, upload=330511]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:145.79.227.12:4767): protocol: rpc: dial tcp 145.79.227.12:4767: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:30:16.586679978 (376h1m52.626253118s) [noise=true , address=145.79.227.6:4258 , piece_id=D7HMSQH7ZLLKNTUZGJLVHIW2KEQZDL3HSS6VO2BSNRZJAC3USFAQ, seg_pos={0 0}, upload=330511]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:145.79.227.6:4258): protocol: rpc: dial tcp 145.79.227.6:4258: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:30:16.586708171 (376h1m52.626281301s) [noise=true , address=145.79.227.16:4912 , piece_id=AG2FUP2KFIJ7W55H2VRYTJYWQ57MAXCTEMTDWSR4FUJAYE3HFKBQ, seg_pos={0 0}, upload=330511]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:145.79.227.16:4912): protocol: rpc: dial tcp 145.79.227.16:4912: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:30:16.586749971 (376h1m52.626323101s) [noise=true , address=145.79.227.15:4875 , piece_id=ZEWO52263YKTTRYDVF4PV52JTPEDJR3XGVRG3BFE5QQKDQSCE2LQ, seg_pos={0 0}, upload=330511]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: rpc: dial tcp 145.79.227.15:4875: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:30:16.586795768 (376h1m52.626368888s) [noise=true , address=145.79.227.17:4972 , piece_id=Q3KQQ57XWDUCXNHP33STZ7DN6VQ6FVYZTDPLINEPZASFEVY5XDJA, seg_pos={0 0}, upload=330511]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:145.79.227.17:4972): protocol: rpc: dial tcp 145.79.227.17:4972: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:30:16.586806088 (376h1m52.626379218s) [noise=true , address=145.79.227.19:4077 , piece_id=EDBYIPZUOPFRRIZBZJJEP2U7NZFCUHRLDZESZ7J5CQZFKADNLE6Q, seg_pos={0 0}, upload=330511]: Done uploading piece. err: ecclient: upload failed (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk, address:145.79.227.19:4077): protocol: rpc: dial tcp 145.79.227.19:4077: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:30:16.586831426 (376h1m52.626404556s) [seg_pos={0 0}, upload=330511]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Sep 22 05:30:16.834944071 (376h1m52.874517201s) [upload=330511]: Done uploading segments... Sep 22 05:30:16.834981573 (376h1m52.874554703s) [upload=330511]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 6.025936879s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [........XX....X..X...X.X......X.........X.X..X....X..XX.....] duration: 6.218440504s FAIL (2025-09-22 05:41:51 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.218466324s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........X...X.............................................] duration: 5.606728017s FAIL (2025-09-22 05:41:49 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 5h37m59.543525066s) [seg_pos={0 0}, upload=330171]: Begin upload segment... Sep 22 05:06:23.504018422 (375h37m59.543591552s) [seg_pos={0 0}, upload=330171]: Done begin upload segment. Sep 22 05:06:23.504059370 (375h37m59.543632501s) [noise=true , address=45.91.116.9:4292 , piece_id=3QGXKEK5C77XDD34Z77C3K2LYEMGJT2SADTL6RGW2YCYQFV5CLUA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.504681175 (375h37m59.544254355s) [noise=true , address=145.79.227.39:4039 , piece_id=MZRBH2VHR7CCJK4OCCXINOQTNBP7PUP236H7DB2DFL7ZFDBNI2IQ, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.505278141 (375h37m59.544851301s) [noise=true , address=145.79.227.33:4881 , piece_id=AIKNNVT5PMYFYKRMUFN5DQMY7TWEUN3FUC64IQQYKAMV723TKKZA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.505743808 (375h37m59.545316948s) [noise=true , address=145.79.227.46:4601 , piece_id=EVWU7M2P2W4FQCNDEQ2OO3SR4KHPTXQRM7GEI6Q6DHYMDHK4PP4Q, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.505904454 (375h37m59.545477584s) [noise=true , address=145.79.227.24:4594 , piece_id=V55Y7DFR7QJWBBQRKSTUWYNEZSCAIWAY2XHAOZOF2H7JDFSX6IBA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.506859502 (375h37m59.546432622s) [noise=true , address=145.79.227.10:4420 , piece_id=ANGUQNIL2D4K2YR7KVJMY37WW7ILRVXSZOB7FRRAKJPD3AGGPGOA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.506958631 (375h37m59.546531761s) [noise=true , address=145.79.227.16:4933 , piece_id=WOBRP7MRD6VVMMXEHKNL4XDSPKQBHVINQD3I4N2SQU2MOSC2ZUCQ, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.507277077 (375h37m59.546850217s) [noise=true , address=145.79.227.7:4308 , piece_id=QEUKJMQDWYMBVKSVNNVDK53P6CZ4X47IINRSWCHAFG7APSFOOAFQ, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.508336856 (375h37m59.547909986s) [noise=true , address=77.90.31.19:4847 , piece_id=6GLTB4OVJTM67VRKRWATCIWE7WVK35IQMQIVOEZMZM6I5DFNRIBA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.513150782 (375h37m59.552723912s) [noise=true , address=145.79.227.12:4770 , piece_id=X5KVN5TA67L4TTD7LVG6N4I54BKNESFAHXRPVIWGDFCEPSHRKXPA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.513934875 (375h37m59.553508006s) [noise=true , address=45.91.116.14:4214 , piece_id=DOY7FC4SJKGAFP3LHSJXKWCRSYI6I6IR2R2FZ47QJNDM4SCAUXQQ, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.514566187 (375h37m59.554139318s) [noise=true , address=145.79.227.16:4939 , piece_id=DLS7J35PB7NXNMAPTVHBZ3VN3U725CSVZZKEYY37HZWQPWCCJO7A, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.515189424 (375h37m59.554762544s) [noise=true , address=145.79.227.25:4637 , piece_id=3SMC4AHCMKJEDXJOJVZ3FJEKW23VZDSWW6MDCENAYJ7HKOPWNHBQ, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.515805377 (375h37m59.555378508s) [noise=true , address=145.79.227.6:4292 , piece_id=BYUJPZSYFTYD4C4F5EBAW6BSFXTILW6EASOIRHZQI7ARCSG63F2Q, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.516494880 (375h37m59.55606801s) [noise=true , address=77.90.31.16:4764 , piece_id=PSAP3HPVNEURWWNC7K2K2G4NEUENKWDUR7OJY6UBGRTCSBYNHRWQ, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.526689722 (375h37m59.566262863s) [noise=true , address=145.79.227.31:4818 , piece_id=KAJZKTKOHNN5U4CTYJ7CLO32ONU6VSEJ2QBLPBI5WWZZ37YOI5NA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.531527154 (375h37m59.571100294s) [noise=true , address=145.79.227.49:4338 , piece_id=NBR62GETN7QBXC7QJCUB5DZHKVMC5AKQQBIIGDRXNECNZFULEDDA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.537921861 (375h37m59.577494991s) [noise=true , address=145.79.227.5:4247 , piece_id=QGN2S5CET2J6LGW6YPQYDWEVOGEMHWRSRMJG2IUQ36KA23OG2R3A, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:24.564378058 (375h38m0.603951188s) [noise=true , address=145.79.227.5:4247 , piece_id=QGN2S5CET2J6LGW6YPQYDWEVOGEMHWRSRMJG2IUQ36KA23OG2R3A, seg_pos={0 0}, upload=330171]: Done uploading piece. err: ecclient: upload failed (node:1262xxX6aUPTYgKdvRxAyc77PmPBWJtq5cXRJFoELxHbfn18DLE, address:145.79.227.5:4247): protocol: rpc: dial tcp 145.79.227.5:4247: connect: no route to host; conn closed: op Sep 22 05:06:24.986206552 (375h38m1.025779682s) [upload=330171]: Next returned nil segment Sep 22 05:06:24.986226830 (375h38m1.02579995s) [upload=330171]: Waiting for error group managing segments... Sep 22 05:06:25.383578176 (375h38m1.423151306s) [noise=true , address=145.79.227.24:4594 , piece_id=V55Y7DFR7QJWBBQRKSTUWYNEZSCAIWAY2XHAOZOF2H7JDFSX6IBA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.383806571 (375h38m1.423379702s) [noise=true , address=145.79.227.33:4881 , piece_id=AIKNNVT5PMYFYKRMUFN5DQMY7TWEUN3FUC64IQQYKAMV723TKKZA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.398811493 (375h38m1.438384634s) [noise=true , address=145.79.227.46:4601 , piece_id=EVWU7M2P2W4FQCNDEQ2OO3SR4KHPTXQRM7GEI6Q6DHYMDHK4PP4Q, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.399530783 (375h38m1.439103913s) [noise=true , address=145.79.227.39:4039 , piece_id=MZRBH2VHR7CCJK4OCCXINOQTNBP7PUP236H7DB2DFL7ZFDBNI2IQ, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.406508421 (375h38m1.446081551s) [noise=true , address=145.79.227.49:4338 , piece_id=NBR62GETN7QBXC7QJCUB5DZHKVMC5AKQQBIIGDRXNECNZFULEDDA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.415736332 (375h38m1.455309462s) [noise=true , address=145.79.227.25:4637 , piece_id=3SMC4AHCMKJEDXJOJVZ3FJEKW23VZDSWW6MDCENAYJ7HKOPWNHBQ, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.437405044 (375h38m1.476978184s) [noise=true , address=145.79.227.31:4818 , piece_id=KAJZKTKOHNN5U4CTYJ7CLO32ONU6VSEJ2QBLPBI5WWZZ37YOI5NA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.465513107 (375h38m1.505086237s) [noise=true , address=77.90.31.19:4847 , piece_id=6GLTB4OVJTM67VRKRWATCIWE7WVK35IQMQIVOEZMZM6I5DFNRIBA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.501349827 (375h38m1.540922957s) [noise=true , address=45.91.116.9:4292 , piece_id=3QGXKEK5C77XDD34Z77C3K2LYEMGJT2SADTL6RGW2YCYQFV5CLUA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.579007182 (375h38m1.618580302s) [noise=true , address=77.90.31.16:4764 , piece_id=PSAP3HPVNEURWWNC7K2K2G4NEUENKWDUR7OJY6UBGRTCSBYNHRWQ, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.586807715 (375h38m1.626380845s) [noise=true , address=45.91.116.14:4214 , piece_id=DOY7FC4SJKGAFP3LHSJXKWCRSYI6I6IR2R2FZ47QJNDM4SCAUXQQ, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:07:22.245417769 (375h38m58.284990899s) [noise=true , address=145.79.227.16:4933 , piece_id=WOBRP7MRD6VVMMXEHKNL4XDSPKQBHVINQD3I4N2SQU2MOSC2ZUCQ, seg_pos={0 0}, upload=330171]: Done uploading piece. err: ecclient: upload failed (node:19m4inaXvTdwf2rKip3eFELv6JAgFvXJ8KqjPZYbb48i8Ktnwe, address:145.79.227.16:4933): protocol: rpc: dial tcp 145.79.227.16:4933: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:07:22.245544310 (375h38m58.28511744s) [noise=true , address=145.79.227.12:4770 , piece_id=X5KVN5TA67L4TTD7LVG6N4I54BKNESFAHXRPVIWGDFCEPSHRKXPA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: ecclient: upload failed (node:1Ug6GpYToYDXiHmiJP4GoE7eJV3kqRveix2S5SLDcKQJtEbNPz, address:145.79.227.12:4770): protocol: rpc: dial tcp 145.79.227.12:4770: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:07:22.245567825 (375h38m58.285140955s) [noise=true , address=145.79.227.16:4939 , piece_id=DLS7J35PB7NXNMAPTVHBZ3VN3U725CSVZZKEYY37HZWQPWCCJO7A, seg_pos={0 0}, upload=330171]: Done uploading piece. err: ecclient: upload failed (node:1mRmp3HhD95PruS4AF8soRsSXNYUeb4mJE7y52FvoeN9D3Gu4F, address:145.79.227.16:4939): protocol: rpc: dial tcp 145.79.227.16:4939: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:07:22.245634181 (375h38m58.285207312s) [noise=true , address=145.79.227.7:4308 , piece_id=QEUKJMQDWYMBVKSVNNVDK53P6CZ4X47IINRSWCHAFG7APSFOOAFQ, seg_pos={0 0}, upload=330171]: Done uploading piece. err: ecclient: upload failed (node:1PgA5SZkBvK673LsV7p2sXpbi1r79esNNZspPZTzaitqfbjtyY, address:145.79.227.7:4308): protocol: rpc: dial tcp 145.79.227.7:4308: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:07:22.245657335 (375h38m58.285230455s) [noise=true , address=145.79.227.10:4420 , piece_id=ANGUQNIL2D4K2YR7KVJMY37WW7ILRVXSZOB7FRRAKJPD3AGGPGOA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: ecclient: upload failed (node:12hJrjeMUMYmFw3yxRtQQKRh9ciPfdu5FXo8NGrsc6HBqgfJzzp, address:145.79.227.10:4420): protocol: rpc: dial tcp 145.79.227.10:4420: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:07:22.245675269 (375h38m58.28524839s) [noise=true , address=145.79.227.6:4292 , piece_id=BYUJPZSYFTYD4C4F5EBAW6BSFXTILW6EASOIRHZQI7ARCSG63F2Q, seg_pos={0 0}, upload=330171]: Done uploading piece. err: ecclient: upload failed (node:12UpWmMhEmk5veKZvzsQ3MCxCMnfEKtFgCugBTsYfy9Ejm7G4xD, address:145.79.227.6:4292): protocol: rpc: dial tcp 145.79.227.6:4292: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:07:22.245712370 (375h38m58.2852855s) [seg_pos={0 0}, upload=330171]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Sep 22 05:07:22.500217377 (375h38m58.539790507s) [upload=330171]: Done uploading segments... Sep 22 05:07:22.500253296 (375h38m58.539826426s) [upload=330171]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 5.606759558s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [...........X...X............................................] duration: 5.956367679s FAIL (2025-09-22 05:41:20 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 5h37m59.543525066s) [seg_pos={0 0}, upload=330171]: Begin upload segment... Sep 22 05:06:23.504018422 (375h37m59.543591552s) [seg_pos={0 0}, upload=330171]: Done begin upload segment. Sep 22 05:06:23.504059370 (375h37m59.543632501s) [noise=true , address=45.91.116.9:4292 , piece_id=3QGXKEK5C77XDD34Z77C3K2LYEMGJT2SADTL6RGW2YCYQFV5CLUA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.504681175 (375h37m59.544254355s) [noise=true , address=145.79.227.39:4039 , piece_id=MZRBH2VHR7CCJK4OCCXINOQTNBP7PUP236H7DB2DFL7ZFDBNI2IQ, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.505278141 (375h37m59.544851301s) [noise=true , address=145.79.227.33:4881 , piece_id=AIKNNVT5PMYFYKRMUFN5DQMY7TWEUN3FUC64IQQYKAMV723TKKZA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.505743808 (375h37m59.545316948s) [noise=true , address=145.79.227.46:4601 , piece_id=EVWU7M2P2W4FQCNDEQ2OO3SR4KHPTXQRM7GEI6Q6DHYMDHK4PP4Q, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.505904454 (375h37m59.545477584s) [noise=true , address=145.79.227.24:4594 , piece_id=V55Y7DFR7QJWBBQRKSTUWYNEZSCAIWAY2XHAOZOF2H7JDFSX6IBA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.506859502 (375h37m59.546432622s) [noise=true , address=145.79.227.10:4420 , piece_id=ANGUQNIL2D4K2YR7KVJMY37WW7ILRVXSZOB7FRRAKJPD3AGGPGOA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.506958631 (375h37m59.546531761s) [noise=true , address=145.79.227.16:4933 , piece_id=WOBRP7MRD6VVMMXEHKNL4XDSPKQBHVINQD3I4N2SQU2MOSC2ZUCQ, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.507277077 (375h37m59.546850217s) [noise=true , address=145.79.227.7:4308 , piece_id=QEUKJMQDWYMBVKSVNNVDK53P6CZ4X47IINRSWCHAFG7APSFOOAFQ, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.508336856 (375h37m59.547909986s) [noise=true , address=77.90.31.19:4847 , piece_id=6GLTB4OVJTM67VRKRWATCIWE7WVK35IQMQIVOEZMZM6I5DFNRIBA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.513150782 (375h37m59.552723912s) [noise=true , address=145.79.227.12:4770 , piece_id=X5KVN5TA67L4TTD7LVG6N4I54BKNESFAHXRPVIWGDFCEPSHRKXPA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.513934875 (375h37m59.553508006s) [noise=true , address=45.91.116.14:4214 , piece_id=DOY7FC4SJKGAFP3LHSJXKWCRSYI6I6IR2R2FZ47QJNDM4SCAUXQQ, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.514566187 (375h37m59.554139318s) [noise=true , address=145.79.227.16:4939 , piece_id=DLS7J35PB7NXNMAPTVHBZ3VN3U725CSVZZKEYY37HZWQPWCCJO7A, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.515189424 (375h37m59.554762544s) [noise=true , address=145.79.227.25:4637 , piece_id=3SMC4AHCMKJEDXJOJVZ3FJEKW23VZDSWW6MDCENAYJ7HKOPWNHBQ, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.515805377 (375h37m59.555378508s) [noise=true , address=145.79.227.6:4292 , piece_id=BYUJPZSYFTYD4C4F5EBAW6BSFXTILW6EASOIRHZQI7ARCSG63F2Q, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.516494880 (375h37m59.55606801s) [noise=true , address=77.90.31.16:4764 , piece_id=PSAP3HPVNEURWWNC7K2K2G4NEUENKWDUR7OJY6UBGRTCSBYNHRWQ, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.526689722 (375h37m59.566262863s) [noise=true , address=145.79.227.31:4818 , piece_id=KAJZKTKOHNN5U4CTYJ7CLO32ONU6VSEJ2QBLPBI5WWZZ37YOI5NA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.531527154 (375h37m59.571100294s) [noise=true , address=145.79.227.49:4338 , piece_id=NBR62GETN7QBXC7QJCUB5DZHKVMC5AKQQBIIGDRXNECNZFULEDDA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.537921861 (375h37m59.577494991s) [noise=true , address=145.79.227.5:4247 , piece_id=QGN2S5CET2J6LGW6YPQYDWEVOGEMHWRSRMJG2IUQ36KA23OG2R3A, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:24.564378058 (375h38m0.603951188s) [noise=true , address=145.79.227.5:4247 , piece_id=QGN2S5CET2J6LGW6YPQYDWEVOGEMHWRSRMJG2IUQ36KA23OG2R3A, seg_pos={0 0}, upload=330171]: Done uploading piece. err: ecclient: upload failed (node:1262xxX6aUPTYgKdvRxAyc77PmPBWJtq5cXRJFoELxHbfn18DLE, address:145.79.227.5:4247): protocol: rpc: dial tcp 145.79.227.5:4247: connect: no route to host; conn closed: op Sep 22 05:06:24.986206552 (375h38m1.025779682s) [upload=330171]: Next returned nil segment Sep 22 05:06:24.986226830 (375h38m1.02579995s) [upload=330171]: Waiting for error group managing segments... Sep 22 05:06:25.383578176 (375h38m1.423151306s) [noise=true , address=145.79.227.24:4594 , piece_id=V55Y7DFR7QJWBBQRKSTUWYNEZSCAIWAY2XHAOZOF2H7JDFSX6IBA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.383806571 (375h38m1.423379702s) [noise=true , address=145.79.227.33:4881 , piece_id=AIKNNVT5PMYFYKRMUFN5DQMY7TWEUN3FUC64IQQYKAMV723TKKZA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.398811493 (375h38m1.438384634s) [noise=true , address=145.79.227.46:4601 , piece_id=EVWU7M2P2W4FQCNDEQ2OO3SR4KHPTXQRM7GEI6Q6DHYMDHK4PP4Q, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.399530783 (375h38m1.439103913s) [noise=true , address=145.79.227.39:4039 , piece_id=MZRBH2VHR7CCJK4OCCXINOQTNBP7PUP236H7DB2DFL7ZFDBNI2IQ, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.406508421 (375h38m1.446081551s) [noise=true , address=145.79.227.49:4338 , piece_id=NBR62GETN7QBXC7QJCUB5DZHKVMC5AKQQBIIGDRXNECNZFULEDDA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.415736332 (375h38m1.455309462s) [noise=true , address=145.79.227.25:4637 , piece_id=3SMC4AHCMKJEDXJOJVZ3FJEKW23VZDSWW6MDCENAYJ7HKOPWNHBQ, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.437405044 (375h38m1.476978184s) [noise=true , address=145.79.227.31:4818 , piece_id=KAJZKTKOHNN5U4CTYJ7CLO32ONU6VSEJ2QBLPBI5WWZZ37YOI5NA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.465513107 (375h38m1.505086237s) [noise=true , address=77.90.31.19:4847 , piece_id=6GLTB4OVJTM67VRKRWATCIWE7WVK35IQMQIVOEZMZM6I5DFNRIBA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.501349827 (375h38m1.540922957s) [noise=true , address=45.91.116.9:4292 , piece_id=3QGXKEK5C77XDD34Z77C3K2LYEMGJT2SADTL6RGW2YCYQFV5CLUA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.579007182 (375h38m1.618580302s) [noise=true , address=77.90.31.16:4764 , piece_id=PSAP3HPVNEURWWNC7K2K2G4NEUENKWDUR7OJY6UBGRTCSBYNHRWQ, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.586807715 (375h38m1.626380845s) [noise=true , address=45.91.116.14:4214 , piece_id=DOY7FC4SJKGAFP3LHSJXKWCRSYI6I6IR2R2FZ47QJNDM4SCAUXQQ, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:07:22.245417769 (375h38m58.284990899s) [noise=true , address=145.79.227.16:4933 , piece_id=WOBRP7MRD6VVMMXEHKNL4XDSPKQBHVINQD3I4N2SQU2MOSC2ZUCQ, seg_pos={0 0}, upload=330171]: Done uploading piece. err: ecclient: upload failed (node:19m4inaXvTdwf2rKip3eFELv6JAgFvXJ8KqjPZYbb48i8Ktnwe, address:145.79.227.16:4933): protocol: rpc: dial tcp 145.79.227.16:4933: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:07:22.245544310 (375h38m58.28511744s) [noise=true , address=145.79.227.12:4770 , piece_id=X5KVN5TA67L4TTD7LVG6N4I54BKNESFAHXRPVIWGDFCEPSHRKXPA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: ecclient: upload failed (node:1Ug6GpYToYDXiHmiJP4GoE7eJV3kqRveix2S5SLDcKQJtEbNPz, address:145.79.227.12:4770): protocol: rpc: dial tcp 145.79.227.12:4770: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:07:22.245567825 (375h38m58.285140955s) [noise=true , address=145.79.227.16:4939 , piece_id=DLS7J35PB7NXNMAPTVHBZ3VN3U725CSVZZKEYY37HZWQPWCCJO7A, seg_pos={0 0}, upload=330171]: Done uploading piece. err: ecclient: upload failed (node:1mRmp3HhD95PruS4AF8soRsSXNYUeb4mJE7y52FvoeN9D3Gu4F, address:145.79.227.16:4939): protocol: rpc: dial tcp 145.79.227.16:4939: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:07:22.245634181 (375h38m58.285207312s) [noise=true , address=145.79.227.7:4308 , piece_id=QEUKJMQDWYMBVKSVNNVDK53P6CZ4X47IINRSWCHAFG7APSFOOAFQ, seg_pos={0 0}, upload=330171]: Done uploading piece. err: ecclient: upload failed (node:1PgA5SZkBvK673LsV7p2sXpbi1r79esNNZspPZTzaitqfbjtyY, address:145.79.227.7:4308): protocol: rpc: dial tcp 145.79.227.7:4308: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:07:22.245657335 (375h38m58.285230455s) [noise=true , address=145.79.227.10:4420 , piece_id=ANGUQNIL2D4K2YR7KVJMY37WW7ILRVXSZOB7FRRAKJPD3AGGPGOA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: ecclient: upload failed (node:12hJrjeMUMYmFw3yxRtQQKRh9ciPfdu5FXo8NGrsc6HBqgfJzzp, address:145.79.227.10:4420): protocol: rpc: dial tcp 145.79.227.10:4420: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:07:22.245675269 (375h38m58.28524839s) [noise=true , address=145.79.227.6:4292 , piece_id=BYUJPZSYFTYD4C4F5EBAW6BSFXTILW6EASOIRHZQI7ARCSG63F2Q, seg_pos={0 0}, upload=330171]: Done uploading piece. err: ecclient: upload failed (node:12UpWmMhEmk5veKZvzsQ3MCxCMnfEKtFgCugBTsYfy9Ejm7G4xD, address:145.79.227.6:4292): protocol: rpc: dial tcp 145.79.227.6:4292: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:07:22.245712370 (375h38m58.2852855s) [seg_pos={0 0}, upload=330171]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Sep 22 05:07:22.500217377 (375h38m58.539790507s) [upload=330171]: Done uploading segments... Sep 22 05:07:22.500253296 (375h38m58.539826426s) [upload=330171]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 5.956428406s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [.......XXXX.X.XX.....X.X....................X...............] duration: 6.171379454s FAIL (2025-09-22 05:41:09 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== 3.337157461s) [upload=330511]: Starting upload Sep 22 05:29:47.297668972 (376h1m23.337242102s) [upload=330511]: Done starting upload Sep 22 05:29:47.297899231 (376h1m23.337472361s) [upload=330511]: Uploading segments... Sep 22 05:29:47.298002678 (376h1m23.337575818s) [upload=330511]: Got next segment. Inline: false Sep 22 05:29:47.298059145 (376h1m23.337632275s) [upload=330511]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 05:29:47.298066109 (376h1m23.337639229s) [upload=330511]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 05:29:47.552911815 (376h1m23.592484945s) [seg_pos={0 0}, upload=330511]: Begin upload segment... Sep 22 05:29:47.552975406 (376h1m23.592548536s) [seg_pos={0 0}, upload=330511]: Done begin upload segment. Sep 22 05:29:47.553013599 (376h1m23.592586719s) [noise=true , address=45.91.116.11:4117 , piece_id=NIFPLA2N3L6XDWTIFUM3YYRGBMFQJKWFVCGYXD2GBFYVYDXW6OEQ, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.553528389 (376h1m23.593101539s) [noise=true , address=145.79.227.6:4258 , piece_id=D7HMSQH7ZLLKNTUZGJLVHIW2KEQZDL3HSS6VO2BSNRZJAC3USFAQ, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.553925034 (376h1m23.593498154s) [noise=true , address=45.91.116.9:4261 , piece_id=REZZWYP25KZOP4UBJWJPHGCDUWCW3EBOJG2DOLCDYZY5B43KBMSQ, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.553961283 (376h1m23.593534413s) [noise=true , address=77.90.31.14:4663 , piece_id=LYGO7UTA55Q5Q7APPDJZNNWZ4S2XHAT4LS4V27VF27A3435DVJQA, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.554326960 (376h1m23.59390009s) [noise=true , address=77.90.31.16:4735 , piece_id=K5KYKKY3BQV6UVJK4G3DYC24DFFIEFQX7KGGFLS4P35KJRFHA6UQ, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.554351497 (376h1m23.593924627s) [noise=true , address=145.79.227.16:4912 , piece_id=AG2FUP2KFIJ7W55H2VRYTJYWQ57MAXCTEMTDWSR4FUJAYE3HFKBQ, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.554588828 (376h1m23.594161948s) [noise=true , address=145.79.227.12:4767 , piece_id=OUVFSM62B7UZ4U4WGFFBOHCPHXEXR6A6H2FQL3DTG5L3DEPUVYVQ, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.554724957 (376h1m23.594298087s) [noise=true , address=77.90.31.18:4807 , piece_id=JFOMLF75MIXEDP5CKDHH7J3PWGWESB7BIKK3SCAIOVNX74DJDC3A, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.554875986 (376h1m23.594449116s) [noise=true , address=145.79.227.3:4151 , piece_id=O6AU3AOBKJRIUMEAWPHMWWDDNRTA52WL72JNUTK5JFLVG2TNV5IA, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.555093950 (376h1m23.59466708s) [noise=true , address=145.79.227.17:4972 , piece_id=Q3KQQ57XWDUCXNHP33STZ7DN6VQ6FVYZTDPLINEPZASFEVY5XDJA, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.555135429 (376h1m23.594708559s) [noise=true , address=45.91.116.6:4973 , piece_id=WPWUFOFQHQZFRPTR33GTDST2NYCVPXKLDVCJQFPQ4A7PI44LJKOQ, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.555464797 (376h1m23.595037917s) [noise=true , address=45.91.116.14:4189 , piece_id=6VJ6XCCEV2CZSARNRF3ATZ4KCUI75QKWIWQWJL5QH4XTQG4MCLUQ, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.555726194 (376h1m23.595299314s) [noise=true , address=77.90.31.15:4699 , piece_id=6XZN64DOILMELBYDU4D72ER33XDKMRIJRSAXX5ITFYPG46YEBITQ, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.556009483 (376h1m23.595582614s) [noise=true , address=145.79.227.19:4077 , piece_id=EDBYIPZUOPFRRIZBZJJEP2U7NZFCUHRLDZESZ7J5CQZFKADNLE6Q, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.556364609 (376h1m23.59593773s) [noise=true , address=145.79.227.15:4875 , piece_id=ZEWO52263YKTTRYDVF4PV52JTPEDJR3XGVRG3BFE5QQKDQSCE2LQ, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.556611299 (376h1m23.596184429s) [noise=true , address=145.79.227.5:4222 , piece_id=CZNVNWHETTFQS6JR2YLE65LNQU4WX74DAROY5I3KAYHG4QDLDOKA, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:49.308510933 (376h1m25.348084063s) [upload=330511]: Next returned nil segment Sep 22 05:29:49.308533666 (376h1m25.348106796s) [upload=330511]: Waiting for error group managing segments... Sep 22 05:29:49.602590138 (376h1m25.642163268s) [noise=true , address=45.91.116.11:4117 , piece_id=NIFPLA2N3L6XDWTIFUM3YYRGBMFQJKWFVCGYXD2GBFYVYDXW6OEQ, seg_pos={0 0}, upload=330511]: Done uploading piece. err: <nil> Sep 22 05:29:49.633269677 (376h1m25.672842807s) [noise=true , address=77.90.31.15:4699 , piece_id=6XZN64DOILMELBYDU4D72ER33XDKMRIJRSAXX5ITFYPG46YEBITQ, seg_pos={0 0}, upload=330511]: Done uploading piece. err: <nil> Sep 22 05:29:49.637382749 (376h1m25.676955879s) [noise=true , address=45.91.116.6:4973 , piece_id=WPWUFOFQHQZFRPTR33GTDST2NYCVPXKLDVCJQFPQ4A7PI44LJKOQ, seg_pos={0 0}, upload=330511]: Done uploading piece. err: <nil> Sep 22 05:29:49.648543782 (376h1m25.688116902s) [noise=true , address=77.90.31.16:4735 , piece_id=K5KYKKY3BQV6UVJK4G3DYC24DFFIEFQX7KGGFLS4P35KJRFHA6UQ, seg_pos={0 0}, upload=330511]: Done uploading piece. err: <nil> Sep 22 05:29:49.653934647 (376h1m25.693507767s) [noise=true , address=45.91.116.14:4189 , piece_id=6VJ6XCCEV2CZSARNRF3ATZ4KCUI75QKWIWQWJL5QH4XTQG4MCLUQ, seg_pos={0 0}, upload=330511]: Done uploading piece. err: <nil> Sep 22 05:29:49.669096699 (376h1m25.708669829s) [noise=true , address=45.91.116.9:4261 , piece_id=REZZWYP25KZOP4UBJWJPHGCDUWCW3EBOJG2DOLCDYZY5B43KBMSQ, seg_pos={0 0}, upload=330511]: Done uploading piece. err: <nil> Sep 22 05:29:49.673672402 (376h1m25.713245532s) [noise=true , address=77.90.31.14:4663 , piece_id=LYGO7UTA55Q5Q7APPDJZNNWZ4S2XHAT4LS4V27VF27A3435DVJQA, seg_pos={0 0}, upload=330511]: Done uploading piece. err: <nil> Sep 22 05:29:49.741841689 (376h1m25.781414809s) [noise=true , address=77.90.31.18:4807 , piece_id=JFOMLF75MIXEDP5CKDHH7J3PWGWESB7BIKK3SCAIOVNX74DJDC3A, seg_pos={0 0}, upload=330511]: Done uploading piece. err: <nil> Sep 22 05:29:58.021015047 (376h1m34.060588167s) [noise=true , address=145.79.227.5:4222 , piece_id=CZNVNWHETTFQS6JR2YLE65LNQU4WX74DAROY5I3KAYHG4QDLDOKA, seg_pos={0 0}, upload=330511]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:145.79.227.5:4222): protocol: rpc: dial tcp 145.79.227.5:4222: connect: no route to host; conn closed: op; piecestore: piecestore close: manager closed: conn closed: op Sep 22 05:30:05.028605724 (376h1m41.068178854s) [noise=true , address=145.79.227.3:4151 , piece_id=O6AU3AOBKJRIUMEAWPHMWWDDNRTA52WL72JNUTK5JFLVG2TNV5IA, seg_pos={0 0}, upload=330511]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:145.79.227.3:4151): protocol: rpc: dial tcp 145.79.227.3:4151: connect: no route to host; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:30:16.586571260 (376h1m52.62614439s) [noise=true , address=145.79.227.12:4767 , piece_id=OUVFSM62B7UZ4U4WGFFBOHCPHXEXR6A6H2FQL3DTG5L3DEPUVYVQ, seg_pos={0 0}, upload=330511]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:145.79.227.12:4767): protocol: rpc: dial tcp 145.79.227.12:4767: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:30:16.586679978 (376h1m52.626253118s) [noise=true , address=145.79.227.6:4258 , piece_id=D7HMSQH7ZLLKNTUZGJLVHIW2KEQZDL3HSS6VO2BSNRZJAC3USFAQ, seg_pos={0 0}, upload=330511]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:145.79.227.6:4258): protocol: rpc: dial tcp 145.79.227.6:4258: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:30:16.586708171 (376h1m52.626281301s) [noise=true , address=145.79.227.16:4912 , piece_id=AG2FUP2KFIJ7W55H2VRYTJYWQ57MAXCTEMTDWSR4FUJAYE3HFKBQ, seg_pos={0 0}, upload=330511]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:145.79.227.16:4912): protocol: rpc: dial tcp 145.79.227.16:4912: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:30:16.586749971 (376h1m52.626323101s) [noise=true , address=145.79.227.15:4875 , piece_id=ZEWO52263YKTTRYDVF4PV52JTPEDJR3XGVRG3BFE5QQKDQSCE2LQ, seg_pos={0 0}, upload=330511]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: rpc: dial tcp 145.79.227.15:4875: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:30:16.586795768 (376h1m52.626368888s) [noise=true , address=145.79.227.17:4972 , piece_id=Q3KQQ57XWDUCXNHP33STZ7DN6VQ6FVYZTDPLINEPZASFEVY5XDJA, seg_pos={0 0}, upload=330511]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:145.79.227.17:4972): protocol: rpc: dial tcp 145.79.227.17:4972: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:30:16.586806088 (376h1m52.626379218s) [noise=true , address=145.79.227.19:4077 , piece_id=EDBYIPZUOPFRRIZBZJJEP2U7NZFCUHRLDZESZ7J5CQZFKADNLE6Q, seg_pos={0 0}, upload=330511]: Done uploading piece. err: ecclient: upload failed (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk, address:145.79.227.19:4077): protocol: rpc: dial tcp 145.79.227.19:4077: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:30:16.586831426 (376h1m52.626404556s) [seg_pos={0 0}, upload=330511]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Sep 22 05:30:16.834944071 (376h1m52.874517201s) [upload=330511]: Done uploading segments... Sep 22 05:30:16.834981573 (376h1m52.874554703s) [upload=330511]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 6.171409832s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [............X...X...........................................] duration: 6.117134241s FAIL (2025-09-22 05:41:03 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 5h37m59.543525066s) [seg_pos={0 0}, upload=330171]: Begin upload segment... Sep 22 05:06:23.504018422 (375h37m59.543591552s) [seg_pos={0 0}, upload=330171]: Done begin upload segment. Sep 22 05:06:23.504059370 (375h37m59.543632501s) [noise=true , address=45.91.116.9:4292 , piece_id=3QGXKEK5C77XDD34Z77C3K2LYEMGJT2SADTL6RGW2YCYQFV5CLUA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.504681175 (375h37m59.544254355s) [noise=true , address=145.79.227.39:4039 , piece_id=MZRBH2VHR7CCJK4OCCXINOQTNBP7PUP236H7DB2DFL7ZFDBNI2IQ, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.505278141 (375h37m59.544851301s) [noise=true , address=145.79.227.33:4881 , piece_id=AIKNNVT5PMYFYKRMUFN5DQMY7TWEUN3FUC64IQQYKAMV723TKKZA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.505743808 (375h37m59.545316948s) [noise=true , address=145.79.227.46:4601 , piece_id=EVWU7M2P2W4FQCNDEQ2OO3SR4KHPTXQRM7GEI6Q6DHYMDHK4PP4Q, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.505904454 (375h37m59.545477584s) [noise=true , address=145.79.227.24:4594 , piece_id=V55Y7DFR7QJWBBQRKSTUWYNEZSCAIWAY2XHAOZOF2H7JDFSX6IBA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.506859502 (375h37m59.546432622s) [noise=true , address=145.79.227.10:4420 , piece_id=ANGUQNIL2D4K2YR7KVJMY37WW7ILRVXSZOB7FRRAKJPD3AGGPGOA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.506958631 (375h37m59.546531761s) [noise=true , address=145.79.227.16:4933 , piece_id=WOBRP7MRD6VVMMXEHKNL4XDSPKQBHVINQD3I4N2SQU2MOSC2ZUCQ, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.507277077 (375h37m59.546850217s) [noise=true , address=145.79.227.7:4308 , piece_id=QEUKJMQDWYMBVKSVNNVDK53P6CZ4X47IINRSWCHAFG7APSFOOAFQ, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.508336856 (375h37m59.547909986s) [noise=true , address=77.90.31.19:4847 , piece_id=6GLTB4OVJTM67VRKRWATCIWE7WVK35IQMQIVOEZMZM6I5DFNRIBA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.513150782 (375h37m59.552723912s) [noise=true , address=145.79.227.12:4770 , piece_id=X5KVN5TA67L4TTD7LVG6N4I54BKNESFAHXRPVIWGDFCEPSHRKXPA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.513934875 (375h37m59.553508006s) [noise=true , address=45.91.116.14:4214 , piece_id=DOY7FC4SJKGAFP3LHSJXKWCRSYI6I6IR2R2FZ47QJNDM4SCAUXQQ, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.514566187 (375h37m59.554139318s) [noise=true , address=145.79.227.16:4939 , piece_id=DLS7J35PB7NXNMAPTVHBZ3VN3U725CSVZZKEYY37HZWQPWCCJO7A, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.515189424 (375h37m59.554762544s) [noise=true , address=145.79.227.25:4637 , piece_id=3SMC4AHCMKJEDXJOJVZ3FJEKW23VZDSWW6MDCENAYJ7HKOPWNHBQ, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.515805377 (375h37m59.555378508s) [noise=true , address=145.79.227.6:4292 , piece_id=BYUJPZSYFTYD4C4F5EBAW6BSFXTILW6EASOIRHZQI7ARCSG63F2Q, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.516494880 (375h37m59.55606801s) [noise=true , address=77.90.31.16:4764 , piece_id=PSAP3HPVNEURWWNC7K2K2G4NEUENKWDUR7OJY6UBGRTCSBYNHRWQ, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.526689722 (375h37m59.566262863s) [noise=true , address=145.79.227.31:4818 , piece_id=KAJZKTKOHNN5U4CTYJ7CLO32ONU6VSEJ2QBLPBI5WWZZ37YOI5NA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.531527154 (375h37m59.571100294s) [noise=true , address=145.79.227.49:4338 , piece_id=NBR62GETN7QBXC7QJCUB5DZHKVMC5AKQQBIIGDRXNECNZFULEDDA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.537921861 (375h37m59.577494991s) [noise=true , address=145.79.227.5:4247 , piece_id=QGN2S5CET2J6LGW6YPQYDWEVOGEMHWRSRMJG2IUQ36KA23OG2R3A, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:24.564378058 (375h38m0.603951188s) [noise=true , address=145.79.227.5:4247 , piece_id=QGN2S5CET2J6LGW6YPQYDWEVOGEMHWRSRMJG2IUQ36KA23OG2R3A, seg_pos={0 0}, upload=330171]: Done uploading piece. err: ecclient: upload failed (node:1262xxX6aUPTYgKdvRxAyc77PmPBWJtq5cXRJFoELxHbfn18DLE, address:145.79.227.5:4247): protocol: rpc: dial tcp 145.79.227.5:4247: connect: no route to host; conn closed: op Sep 22 05:06:24.986206552 (375h38m1.025779682s) [upload=330171]: Next returned nil segment Sep 22 05:06:24.986226830 (375h38m1.02579995s) [upload=330171]: Waiting for error group managing segments... Sep 22 05:06:25.383578176 (375h38m1.423151306s) [noise=true , address=145.79.227.24:4594 , piece_id=V55Y7DFR7QJWBBQRKSTUWYNEZSCAIWAY2XHAOZOF2H7JDFSX6IBA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.383806571 (375h38m1.423379702s) [noise=true , address=145.79.227.33:4881 , piece_id=AIKNNVT5PMYFYKRMUFN5DQMY7TWEUN3FUC64IQQYKAMV723TKKZA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.398811493 (375h38m1.438384634s) [noise=true , address=145.79.227.46:4601 , piece_id=EVWU7M2P2W4FQCNDEQ2OO3SR4KHPTXQRM7GEI6Q6DHYMDHK4PP4Q, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.399530783 (375h38m1.439103913s) [noise=true , address=145.79.227.39:4039 , piece_id=MZRBH2VHR7CCJK4OCCXINOQTNBP7PUP236H7DB2DFL7ZFDBNI2IQ, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.406508421 (375h38m1.446081551s) [noise=true , address=145.79.227.49:4338 , piece_id=NBR62GETN7QBXC7QJCUB5DZHKVMC5AKQQBIIGDRXNECNZFULEDDA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.415736332 (375h38m1.455309462s) [noise=true , address=145.79.227.25:4637 , piece_id=3SMC4AHCMKJEDXJOJVZ3FJEKW23VZDSWW6MDCENAYJ7HKOPWNHBQ, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.437405044 (375h38m1.476978184s) [noise=true , address=145.79.227.31:4818 , piece_id=KAJZKTKOHNN5U4CTYJ7CLO32ONU6VSEJ2QBLPBI5WWZZ37YOI5NA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.465513107 (375h38m1.505086237s) [noise=true , address=77.90.31.19:4847 , piece_id=6GLTB4OVJTM67VRKRWATCIWE7WVK35IQMQIVOEZMZM6I5DFNRIBA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.501349827 (375h38m1.540922957s) [noise=true , address=45.91.116.9:4292 , piece_id=3QGXKEK5C77XDD34Z77C3K2LYEMGJT2SADTL6RGW2YCYQFV5CLUA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.579007182 (375h38m1.618580302s) [noise=true , address=77.90.31.16:4764 , piece_id=PSAP3HPVNEURWWNC7K2K2G4NEUENKWDUR7OJY6UBGRTCSBYNHRWQ, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.586807715 (375h38m1.626380845s) [noise=true , address=45.91.116.14:4214 , piece_id=DOY7FC4SJKGAFP3LHSJXKWCRSYI6I6IR2R2FZ47QJNDM4SCAUXQQ, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:07:22.245417769 (375h38m58.284990899s) [noise=true , address=145.79.227.16:4933 , piece_id=WOBRP7MRD6VVMMXEHKNL4XDSPKQBHVINQD3I4N2SQU2MOSC2ZUCQ, seg_pos={0 0}, upload=330171]: Done uploading piece. err: ecclient: upload failed (node:19m4inaXvTdwf2rKip3eFELv6JAgFvXJ8KqjPZYbb48i8Ktnwe, address:145.79.227.16:4933): protocol: rpc: dial tcp 145.79.227.16:4933: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:07:22.245544310 (375h38m58.28511744s) [noise=true , address=145.79.227.12:4770 , piece_id=X5KVN5TA67L4TTD7LVG6N4I54BKNESFAHXRPVIWGDFCEPSHRKXPA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: ecclient: upload failed (node:1Ug6GpYToYDXiHmiJP4GoE7eJV3kqRveix2S5SLDcKQJtEbNPz, address:145.79.227.12:4770): protocol: rpc: dial tcp 145.79.227.12:4770: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:07:22.245567825 (375h38m58.285140955s) [noise=true , address=145.79.227.16:4939 , piece_id=DLS7J35PB7NXNMAPTVHBZ3VN3U725CSVZZKEYY37HZWQPWCCJO7A, seg_pos={0 0}, upload=330171]: Done uploading piece. err: ecclient: upload failed (node:1mRmp3HhD95PruS4AF8soRsSXNYUeb4mJE7y52FvoeN9D3Gu4F, address:145.79.227.16:4939): protocol: rpc: dial tcp 145.79.227.16:4939: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:07:22.245634181 (375h38m58.285207312s) [noise=true , address=145.79.227.7:4308 , piece_id=QEUKJMQDWYMBVKSVNNVDK53P6CZ4X47IINRSWCHAFG7APSFOOAFQ, seg_pos={0 0}, upload=330171]: Done uploading piece. err: ecclient: upload failed (node:1PgA5SZkBvK673LsV7p2sXpbi1r79esNNZspPZTzaitqfbjtyY, address:145.79.227.7:4308): protocol: rpc: dial tcp 145.79.227.7:4308: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:07:22.245657335 (375h38m58.285230455s) [noise=true , address=145.79.227.10:4420 , piece_id=ANGUQNIL2D4K2YR7KVJMY37WW7ILRVXSZOB7FRRAKJPD3AGGPGOA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: ecclient: upload failed (node:12hJrjeMUMYmFw3yxRtQQKRh9ciPfdu5FXo8NGrsc6HBqgfJzzp, address:145.79.227.10:4420): protocol: rpc: dial tcp 145.79.227.10:4420: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:07:22.245675269 (375h38m58.28524839s) [noise=true , address=145.79.227.6:4292 , piece_id=BYUJPZSYFTYD4C4F5EBAW6BSFXTILW6EASOIRHZQI7ARCSG63F2Q, seg_pos={0 0}, upload=330171]: Done uploading piece. err: ecclient: upload failed (node:12UpWmMhEmk5veKZvzsQ3MCxCMnfEKtFgCugBTsYfy9Ejm7G4xD, address:145.79.227.6:4292): protocol: rpc: dial tcp 145.79.227.6:4292: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:07:22.245712370 (375h38m58.2852855s) [seg_pos={0 0}, upload=330171]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Sep 22 05:07:22.500217377 (375h38m58.539790507s) [upload=330171]: Done uploading segments... Sep 22 05:07:22.500253296 (375h38m58.539826426s) [upload=330171]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 6.117173737s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [.........XX....X..X...X.X......X.........X.X..X....X..XX....] duration: 6.038526529s FAIL (2025-09-22 05:40:39 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.038544994s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [........XXXX.X.XX.....X.X....................X..............] duration: 6.279458582s FAIL (2025-09-22 05:40:36 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== 3.337157461s) [upload=330511]: Starting upload Sep 22 05:29:47.297668972 (376h1m23.337242102s) [upload=330511]: Done starting upload Sep 22 05:29:47.297899231 (376h1m23.337472361s) [upload=330511]: Uploading segments... Sep 22 05:29:47.298002678 (376h1m23.337575818s) [upload=330511]: Got next segment. Inline: false Sep 22 05:29:47.298059145 (376h1m23.337632275s) [upload=330511]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 05:29:47.298066109 (376h1m23.337639229s) [upload=330511]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 05:29:47.552911815 (376h1m23.592484945s) [seg_pos={0 0}, upload=330511]: Begin upload segment... Sep 22 05:29:47.552975406 (376h1m23.592548536s) [seg_pos={0 0}, upload=330511]: Done begin upload segment. Sep 22 05:29:47.553013599 (376h1m23.592586719s) [noise=true , address=45.91.116.11:4117 , piece_id=NIFPLA2N3L6XDWTIFUM3YYRGBMFQJKWFVCGYXD2GBFYVYDXW6OEQ, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.553528389 (376h1m23.593101539s) [noise=true , address=145.79.227.6:4258 , piece_id=D7HMSQH7ZLLKNTUZGJLVHIW2KEQZDL3HSS6VO2BSNRZJAC3USFAQ, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.553925034 (376h1m23.593498154s) [noise=true , address=45.91.116.9:4261 , piece_id=REZZWYP25KZOP4UBJWJPHGCDUWCW3EBOJG2DOLCDYZY5B43KBMSQ, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.553961283 (376h1m23.593534413s) [noise=true , address=77.90.31.14:4663 , piece_id=LYGO7UTA55Q5Q7APPDJZNNWZ4S2XHAT4LS4V27VF27A3435DVJQA, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.554326960 (376h1m23.59390009s) [noise=true , address=77.90.31.16:4735 , piece_id=K5KYKKY3BQV6UVJK4G3DYC24DFFIEFQX7KGGFLS4P35KJRFHA6UQ, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.554351497 (376h1m23.593924627s) [noise=true , address=145.79.227.16:4912 , piece_id=AG2FUP2KFIJ7W55H2VRYTJYWQ57MAXCTEMTDWSR4FUJAYE3HFKBQ, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.554588828 (376h1m23.594161948s) [noise=true , address=145.79.227.12:4767 , piece_id=OUVFSM62B7UZ4U4WGFFBOHCPHXEXR6A6H2FQL3DTG5L3DEPUVYVQ, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.554724957 (376h1m23.594298087s) [noise=true , address=77.90.31.18:4807 , piece_id=JFOMLF75MIXEDP5CKDHH7J3PWGWESB7BIKK3SCAIOVNX74DJDC3A, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.554875986 (376h1m23.594449116s) [noise=true , address=145.79.227.3:4151 , piece_id=O6AU3AOBKJRIUMEAWPHMWWDDNRTA52WL72JNUTK5JFLVG2TNV5IA, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.555093950 (376h1m23.59466708s) [noise=true , address=145.79.227.17:4972 , piece_id=Q3KQQ57XWDUCXNHP33STZ7DN6VQ6FVYZTDPLINEPZASFEVY5XDJA, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.555135429 (376h1m23.594708559s) [noise=true , address=45.91.116.6:4973 , piece_id=WPWUFOFQHQZFRPTR33GTDST2NYCVPXKLDVCJQFPQ4A7PI44LJKOQ, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.555464797 (376h1m23.595037917s) [noise=true , address=45.91.116.14:4189 , piece_id=6VJ6XCCEV2CZSARNRF3ATZ4KCUI75QKWIWQWJL5QH4XTQG4MCLUQ, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.555726194 (376h1m23.595299314s) [noise=true , address=77.90.31.15:4699 , piece_id=6XZN64DOILMELBYDU4D72ER33XDKMRIJRSAXX5ITFYPG46YEBITQ, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.556009483 (376h1m23.595582614s) [noise=true , address=145.79.227.19:4077 , piece_id=EDBYIPZUOPFRRIZBZJJEP2U7NZFCUHRLDZESZ7J5CQZFKADNLE6Q, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.556364609 (376h1m23.59593773s) [noise=true , address=145.79.227.15:4875 , piece_id=ZEWO52263YKTTRYDVF4PV52JTPEDJR3XGVRG3BFE5QQKDQSCE2LQ, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.556611299 (376h1m23.596184429s) [noise=true , address=145.79.227.5:4222 , piece_id=CZNVNWHETTFQS6JR2YLE65LNQU4WX74DAROY5I3KAYHG4QDLDOKA, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:49.308510933 (376h1m25.348084063s) [upload=330511]: Next returned nil segment Sep 22 05:29:49.308533666 (376h1m25.348106796s) [upload=330511]: Waiting for error group managing segments... Sep 22 05:29:49.602590138 (376h1m25.642163268s) [noise=true , address=45.91.116.11:4117 , piece_id=NIFPLA2N3L6XDWTIFUM3YYRGBMFQJKWFVCGYXD2GBFYVYDXW6OEQ, seg_pos={0 0}, upload=330511]: Done uploading piece. err: <nil> Sep 22 05:29:49.633269677 (376h1m25.672842807s) [noise=true , address=77.90.31.15:4699 , piece_id=6XZN64DOILMELBYDU4D72ER33XDKMRIJRSAXX5ITFYPG46YEBITQ, seg_pos={0 0}, upload=330511]: Done uploading piece. err: <nil> Sep 22 05:29:49.637382749 (376h1m25.676955879s) [noise=true , address=45.91.116.6:4973 , piece_id=WPWUFOFQHQZFRPTR33GTDST2NYCVPXKLDVCJQFPQ4A7PI44LJKOQ, seg_pos={0 0}, upload=330511]: Done uploading piece. err: <nil> Sep 22 05:29:49.648543782 (376h1m25.688116902s) [noise=true , address=77.90.31.16:4735 , piece_id=K5KYKKY3BQV6UVJK4G3DYC24DFFIEFQX7KGGFLS4P35KJRFHA6UQ, seg_pos={0 0}, upload=330511]: Done uploading piece. err: <nil> Sep 22 05:29:49.653934647 (376h1m25.693507767s) [noise=true , address=45.91.116.14:4189 , piece_id=6VJ6XCCEV2CZSARNRF3ATZ4KCUI75QKWIWQWJL5QH4XTQG4MCLUQ, seg_pos={0 0}, upload=330511]: Done uploading piece. err: <nil> Sep 22 05:29:49.669096699 (376h1m25.708669829s) [noise=true , address=45.91.116.9:4261 , piece_id=REZZWYP25KZOP4UBJWJPHGCDUWCW3EBOJG2DOLCDYZY5B43KBMSQ, seg_pos={0 0}, upload=330511]: Done uploading piece. err: <nil> Sep 22 05:29:49.673672402 (376h1m25.713245532s) [noise=true , address=77.90.31.14:4663 , piece_id=LYGO7UTA55Q5Q7APPDJZNNWZ4S2XHAT4LS4V27VF27A3435DVJQA, seg_pos={0 0}, upload=330511]: Done uploading piece. err: <nil> Sep 22 05:29:49.741841689 (376h1m25.781414809s) [noise=true , address=77.90.31.18:4807 , piece_id=JFOMLF75MIXEDP5CKDHH7J3PWGWESB7BIKK3SCAIOVNX74DJDC3A, seg_pos={0 0}, upload=330511]: Done uploading piece. err: <nil> Sep 22 05:29:58.021015047 (376h1m34.060588167s) [noise=true , address=145.79.227.5:4222 , piece_id=CZNVNWHETTFQS6JR2YLE65LNQU4WX74DAROY5I3KAYHG4QDLDOKA, seg_pos={0 0}, upload=330511]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:145.79.227.5:4222): protocol: rpc: dial tcp 145.79.227.5:4222: connect: no route to host; conn closed: op; piecestore: piecestore close: manager closed: conn closed: op Sep 22 05:30:05.028605724 (376h1m41.068178854s) [noise=true , address=145.79.227.3:4151 , piece_id=O6AU3AOBKJRIUMEAWPHMWWDDNRTA52WL72JNUTK5JFLVG2TNV5IA, seg_pos={0 0}, upload=330511]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:145.79.227.3:4151): protocol: rpc: dial tcp 145.79.227.3:4151: connect: no route to host; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:30:16.586571260 (376h1m52.62614439s) [noise=true , address=145.79.227.12:4767 , piece_id=OUVFSM62B7UZ4U4WGFFBOHCPHXEXR6A6H2FQL3DTG5L3DEPUVYVQ, seg_pos={0 0}, upload=330511]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:145.79.227.12:4767): protocol: rpc: dial tcp 145.79.227.12:4767: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:30:16.586679978 (376h1m52.626253118s) [noise=true , address=145.79.227.6:4258 , piece_id=D7HMSQH7ZLLKNTUZGJLVHIW2KEQZDL3HSS6VO2BSNRZJAC3USFAQ, seg_pos={0 0}, upload=330511]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:145.79.227.6:4258): protocol: rpc: dial tcp 145.79.227.6:4258: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:30:16.586708171 (376h1m52.626281301s) [noise=true , address=145.79.227.16:4912 , piece_id=AG2FUP2KFIJ7W55H2VRYTJYWQ57MAXCTEMTDWSR4FUJAYE3HFKBQ, seg_pos={0 0}, upload=330511]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:145.79.227.16:4912): protocol: rpc: dial tcp 145.79.227.16:4912: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:30:16.586749971 (376h1m52.626323101s) [noise=true , address=145.79.227.15:4875 , piece_id=ZEWO52263YKTTRYDVF4PV52JTPEDJR3XGVRG3BFE5QQKDQSCE2LQ, seg_pos={0 0}, upload=330511]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: rpc: dial tcp 145.79.227.15:4875: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:30:16.586795768 (376h1m52.626368888s) [noise=true , address=145.79.227.17:4972 , piece_id=Q3KQQ57XWDUCXNHP33STZ7DN6VQ6FVYZTDPLINEPZASFEVY5XDJA, seg_pos={0 0}, upload=330511]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:145.79.227.17:4972): protocol: rpc: dial tcp 145.79.227.17:4972: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:30:16.586806088 (376h1m52.626379218s) [noise=true , address=145.79.227.19:4077 , piece_id=EDBYIPZUOPFRRIZBZJJEP2U7NZFCUHRLDZESZ7J5CQZFKADNLE6Q, seg_pos={0 0}, upload=330511]: Done uploading piece. err: ecclient: upload failed (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk, address:145.79.227.19:4077): protocol: rpc: dial tcp 145.79.227.19:4077: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:30:16.586831426 (376h1m52.626404556s) [seg_pos={0 0}, upload=330511]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Sep 22 05:30:16.834944071 (376h1m52.874517201s) [upload=330511]: Done uploading segments... Sep 22 05:30:16.834981573 (376h1m52.874554703s) [upload=330511]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 6.279491014s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [.........XXXX.X.XX.....X.X....................X.............] duration: 6.032675066s FAIL (2025-09-22 05:40:11 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== 3.337157461s) [upload=330511]: Starting upload Sep 22 05:29:47.297668972 (376h1m23.337242102s) [upload=330511]: Done starting upload Sep 22 05:29:47.297899231 (376h1m23.337472361s) [upload=330511]: Uploading segments... Sep 22 05:29:47.298002678 (376h1m23.337575818s) [upload=330511]: Got next segment. Inline: false Sep 22 05:29:47.298059145 (376h1m23.337632275s) [upload=330511]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 05:29:47.298066109 (376h1m23.337639229s) [upload=330511]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 05:29:47.552911815 (376h1m23.592484945s) [seg_pos={0 0}, upload=330511]: Begin upload segment... Sep 22 05:29:47.552975406 (376h1m23.592548536s) [seg_pos={0 0}, upload=330511]: Done begin upload segment. Sep 22 05:29:47.553013599 (376h1m23.592586719s) [noise=true , address=45.91.116.11:4117 , piece_id=NIFPLA2N3L6XDWTIFUM3YYRGBMFQJKWFVCGYXD2GBFYVYDXW6OEQ, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.553528389 (376h1m23.593101539s) [noise=true , address=145.79.227.6:4258 , piece_id=D7HMSQH7ZLLKNTUZGJLVHIW2KEQZDL3HSS6VO2BSNRZJAC3USFAQ, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.553925034 (376h1m23.593498154s) [noise=true , address=45.91.116.9:4261 , piece_id=REZZWYP25KZOP4UBJWJPHGCDUWCW3EBOJG2DOLCDYZY5B43KBMSQ, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.553961283 (376h1m23.593534413s) [noise=true , address=77.90.31.14:4663 , piece_id=LYGO7UTA55Q5Q7APPDJZNNWZ4S2XHAT4LS4V27VF27A3435DVJQA, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.554326960 (376h1m23.59390009s) [noise=true , address=77.90.31.16:4735 , piece_id=K5KYKKY3BQV6UVJK4G3DYC24DFFIEFQX7KGGFLS4P35KJRFHA6UQ, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.554351497 (376h1m23.593924627s) [noise=true , address=145.79.227.16:4912 , piece_id=AG2FUP2KFIJ7W55H2VRYTJYWQ57MAXCTEMTDWSR4FUJAYE3HFKBQ, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.554588828 (376h1m23.594161948s) [noise=true , address=145.79.227.12:4767 , piece_id=OUVFSM62B7UZ4U4WGFFBOHCPHXEXR6A6H2FQL3DTG5L3DEPUVYVQ, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.554724957 (376h1m23.594298087s) [noise=true , address=77.90.31.18:4807 , piece_id=JFOMLF75MIXEDP5CKDHH7J3PWGWESB7BIKK3SCAIOVNX74DJDC3A, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.554875986 (376h1m23.594449116s) [noise=true , address=145.79.227.3:4151 , piece_id=O6AU3AOBKJRIUMEAWPHMWWDDNRTA52WL72JNUTK5JFLVG2TNV5IA, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.555093950 (376h1m23.59466708s) [noise=true , address=145.79.227.17:4972 , piece_id=Q3KQQ57XWDUCXNHP33STZ7DN6VQ6FVYZTDPLINEPZASFEVY5XDJA, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.555135429 (376h1m23.594708559s) [noise=true , address=45.91.116.6:4973 , piece_id=WPWUFOFQHQZFRPTR33GTDST2NYCVPXKLDVCJQFPQ4A7PI44LJKOQ, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.555464797 (376h1m23.595037917s) [noise=true , address=45.91.116.14:4189 , piece_id=6VJ6XCCEV2CZSARNRF3ATZ4KCUI75QKWIWQWJL5QH4XTQG4MCLUQ, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.555726194 (376h1m23.595299314s) [noise=true , address=77.90.31.15:4699 , piece_id=6XZN64DOILMELBYDU4D72ER33XDKMRIJRSAXX5ITFYPG46YEBITQ, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.556009483 (376h1m23.595582614s) [noise=true , address=145.79.227.19:4077 , piece_id=EDBYIPZUOPFRRIZBZJJEP2U7NZFCUHRLDZESZ7J5CQZFKADNLE6Q, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.556364609 (376h1m23.59593773s) [noise=true , address=145.79.227.15:4875 , piece_id=ZEWO52263YKTTRYDVF4PV52JTPEDJR3XGVRG3BFE5QQKDQSCE2LQ, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.556611299 (376h1m23.596184429s) [noise=true , address=145.79.227.5:4222 , piece_id=CZNVNWHETTFQS6JR2YLE65LNQU4WX74DAROY5I3KAYHG4QDLDOKA, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:49.308510933 (376h1m25.348084063s) [upload=330511]: Next returned nil segment Sep 22 05:29:49.308533666 (376h1m25.348106796s) [upload=330511]: Waiting for error group managing segments... Sep 22 05:29:49.602590138 (376h1m25.642163268s) [noise=true , address=45.91.116.11:4117 , piece_id=NIFPLA2N3L6XDWTIFUM3YYRGBMFQJKWFVCGYXD2GBFYVYDXW6OEQ, seg_pos={0 0}, upload=330511]: Done uploading piece. err: <nil> Sep 22 05:29:49.633269677 (376h1m25.672842807s) [noise=true , address=77.90.31.15:4699 , piece_id=6XZN64DOILMELBYDU4D72ER33XDKMRIJRSAXX5ITFYPG46YEBITQ, seg_pos={0 0}, upload=330511]: Done uploading piece. err: <nil> Sep 22 05:29:49.637382749 (376h1m25.676955879s) [noise=true , address=45.91.116.6:4973 , piece_id=WPWUFOFQHQZFRPTR33GTDST2NYCVPXKLDVCJQFPQ4A7PI44LJKOQ, seg_pos={0 0}, upload=330511]: Done uploading piece. err: <nil> Sep 22 05:29:49.648543782 (376h1m25.688116902s) [noise=true , address=77.90.31.16:4735 , piece_id=K5KYKKY3BQV6UVJK4G3DYC24DFFIEFQX7KGGFLS4P35KJRFHA6UQ, seg_pos={0 0}, upload=330511]: Done uploading piece. err: <nil> Sep 22 05:29:49.653934647 (376h1m25.693507767s) [noise=true , address=45.91.116.14:4189 , piece_id=6VJ6XCCEV2CZSARNRF3ATZ4KCUI75QKWIWQWJL5QH4XTQG4MCLUQ, seg_pos={0 0}, upload=330511]: Done uploading piece. err: <nil> Sep 22 05:29:49.669096699 (376h1m25.708669829s) [noise=true , address=45.91.116.9:4261 , piece_id=REZZWYP25KZOP4UBJWJPHGCDUWCW3EBOJG2DOLCDYZY5B43KBMSQ, seg_pos={0 0}, upload=330511]: Done uploading piece. err: <nil> Sep 22 05:29:49.673672402 (376h1m25.713245532s) [noise=true , address=77.90.31.14:4663 , piece_id=LYGO7UTA55Q5Q7APPDJZNNWZ4S2XHAT4LS4V27VF27A3435DVJQA, seg_pos={0 0}, upload=330511]: Done uploading piece. err: <nil> Sep 22 05:29:49.741841689 (376h1m25.781414809s) [noise=true , address=77.90.31.18:4807 , piece_id=JFOMLF75MIXEDP5CKDHH7J3PWGWESB7BIKK3SCAIOVNX74DJDC3A, seg_pos={0 0}, upload=330511]: Done uploading piece. err: <nil> Sep 22 05:29:58.021015047 (376h1m34.060588167s) [noise=true , address=145.79.227.5:4222 , piece_id=CZNVNWHETTFQS6JR2YLE65LNQU4WX74DAROY5I3KAYHG4QDLDOKA, seg_pos={0 0}, upload=330511]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:145.79.227.5:4222): protocol: rpc: dial tcp 145.79.227.5:4222: connect: no route to host; conn closed: op; piecestore: piecestore close: manager closed: conn closed: op Sep 22 05:30:05.028605724 (376h1m41.068178854s) [noise=true , address=145.79.227.3:4151 , piece_id=O6AU3AOBKJRIUMEAWPHMWWDDNRTA52WL72JNUTK5JFLVG2TNV5IA, seg_pos={0 0}, upload=330511]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:145.79.227.3:4151): protocol: rpc: dial tcp 145.79.227.3:4151: connect: no route to host; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:30:16.586571260 (376h1m52.62614439s) [noise=true , address=145.79.227.12:4767 , piece_id=OUVFSM62B7UZ4U4WGFFBOHCPHXEXR6A6H2FQL3DTG5L3DEPUVYVQ, seg_pos={0 0}, upload=330511]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:145.79.227.12:4767): protocol: rpc: dial tcp 145.79.227.12:4767: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:30:16.586679978 (376h1m52.626253118s) [noise=true , address=145.79.227.6:4258 , piece_id=D7HMSQH7ZLLKNTUZGJLVHIW2KEQZDL3HSS6VO2BSNRZJAC3USFAQ, seg_pos={0 0}, upload=330511]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:145.79.227.6:4258): protocol: rpc: dial tcp 145.79.227.6:4258: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:30:16.586708171 (376h1m52.626281301s) [noise=true , address=145.79.227.16:4912 , piece_id=AG2FUP2KFIJ7W55H2VRYTJYWQ57MAXCTEMTDWSR4FUJAYE3HFKBQ, seg_pos={0 0}, upload=330511]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:145.79.227.16:4912): protocol: rpc: dial tcp 145.79.227.16:4912: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:30:16.586749971 (376h1m52.626323101s) [noise=true , address=145.79.227.15:4875 , piece_id=ZEWO52263YKTTRYDVF4PV52JTPEDJR3XGVRG3BFE5QQKDQSCE2LQ, seg_pos={0 0}, upload=330511]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: rpc: dial tcp 145.79.227.15:4875: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:30:16.586795768 (376h1m52.626368888s) [noise=true , address=145.79.227.17:4972 , piece_id=Q3KQQ57XWDUCXNHP33STZ7DN6VQ6FVYZTDPLINEPZASFEVY5XDJA, seg_pos={0 0}, upload=330511]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:145.79.227.17:4972): protocol: rpc: dial tcp 145.79.227.17:4972: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:30:16.586806088 (376h1m52.626379218s) [noise=true , address=145.79.227.19:4077 , piece_id=EDBYIPZUOPFRRIZBZJJEP2U7NZFCUHRLDZESZ7J5CQZFKADNLE6Q, seg_pos={0 0}, upload=330511]: Done uploading piece. err: ecclient: upload failed (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk, address:145.79.227.19:4077): protocol: rpc: dial tcp 145.79.227.19:4077: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:30:16.586831426 (376h1m52.626404556s) [seg_pos={0 0}, upload=330511]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Sep 22 05:30:16.834944071 (376h1m52.874517201s) [upload=330511]: Done uploading segments... Sep 22 05:30:16.834981573 (376h1m52.874554703s) [upload=330511]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 6.032717809s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [..........XX....X..X...X.X......X.........X.X..X....X..XX...] duration: 6.053512908s FAIL (2025-09-22 05:40:09 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.053534158s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.............X...X..........................................] duration: 6.020406729s FAIL (2025-09-22 05:40:04 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 5h37m59.543525066s) [seg_pos={0 0}, upload=330171]: Begin upload segment... Sep 22 05:06:23.504018422 (375h37m59.543591552s) [seg_pos={0 0}, upload=330171]: Done begin upload segment. Sep 22 05:06:23.504059370 (375h37m59.543632501s) [noise=true , address=45.91.116.9:4292 , piece_id=3QGXKEK5C77XDD34Z77C3K2LYEMGJT2SADTL6RGW2YCYQFV5CLUA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.504681175 (375h37m59.544254355s) [noise=true , address=145.79.227.39:4039 , piece_id=MZRBH2VHR7CCJK4OCCXINOQTNBP7PUP236H7DB2DFL7ZFDBNI2IQ, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.505278141 (375h37m59.544851301s) [noise=true , address=145.79.227.33:4881 , piece_id=AIKNNVT5PMYFYKRMUFN5DQMY7TWEUN3FUC64IQQYKAMV723TKKZA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.505743808 (375h37m59.545316948s) [noise=true , address=145.79.227.46:4601 , piece_id=EVWU7M2P2W4FQCNDEQ2OO3SR4KHPTXQRM7GEI6Q6DHYMDHK4PP4Q, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.505904454 (375h37m59.545477584s) [noise=true , address=145.79.227.24:4594 , piece_id=V55Y7DFR7QJWBBQRKSTUWYNEZSCAIWAY2XHAOZOF2H7JDFSX6IBA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.506859502 (375h37m59.546432622s) [noise=true , address=145.79.227.10:4420 , piece_id=ANGUQNIL2D4K2YR7KVJMY37WW7ILRVXSZOB7FRRAKJPD3AGGPGOA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.506958631 (375h37m59.546531761s) [noise=true , address=145.79.227.16:4933 , piece_id=WOBRP7MRD6VVMMXEHKNL4XDSPKQBHVINQD3I4N2SQU2MOSC2ZUCQ, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.507277077 (375h37m59.546850217s) [noise=true , address=145.79.227.7:4308 , piece_id=QEUKJMQDWYMBVKSVNNVDK53P6CZ4X47IINRSWCHAFG7APSFOOAFQ, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.508336856 (375h37m59.547909986s) [noise=true , address=77.90.31.19:4847 , piece_id=6GLTB4OVJTM67VRKRWATCIWE7WVK35IQMQIVOEZMZM6I5DFNRIBA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.513150782 (375h37m59.552723912s) [noise=true , address=145.79.227.12:4770 , piece_id=X5KVN5TA67L4TTD7LVG6N4I54BKNESFAHXRPVIWGDFCEPSHRKXPA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.513934875 (375h37m59.553508006s) [noise=true , address=45.91.116.14:4214 , piece_id=DOY7FC4SJKGAFP3LHSJXKWCRSYI6I6IR2R2FZ47QJNDM4SCAUXQQ, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.514566187 (375h37m59.554139318s) [noise=true , address=145.79.227.16:4939 , piece_id=DLS7J35PB7NXNMAPTVHBZ3VN3U725CSVZZKEYY37HZWQPWCCJO7A, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.515189424 (375h37m59.554762544s) [noise=true , address=145.79.227.25:4637 , piece_id=3SMC4AHCMKJEDXJOJVZ3FJEKW23VZDSWW6MDCENAYJ7HKOPWNHBQ, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.515805377 (375h37m59.555378508s) [noise=true , address=145.79.227.6:4292 , piece_id=BYUJPZSYFTYD4C4F5EBAW6BSFXTILW6EASOIRHZQI7ARCSG63F2Q, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.516494880 (375h37m59.55606801s) [noise=true , address=77.90.31.16:4764 , piece_id=PSAP3HPVNEURWWNC7K2K2G4NEUENKWDUR7OJY6UBGRTCSBYNHRWQ, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.526689722 (375h37m59.566262863s) [noise=true , address=145.79.227.31:4818 , piece_id=KAJZKTKOHNN5U4CTYJ7CLO32ONU6VSEJ2QBLPBI5WWZZ37YOI5NA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.531527154 (375h37m59.571100294s) [noise=true , address=145.79.227.49:4338 , piece_id=NBR62GETN7QBXC7QJCUB5DZHKVMC5AKQQBIIGDRXNECNZFULEDDA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.537921861 (375h37m59.577494991s) [noise=true , address=145.79.227.5:4247 , piece_id=QGN2S5CET2J6LGW6YPQYDWEVOGEMHWRSRMJG2IUQ36KA23OG2R3A, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:24.564378058 (375h38m0.603951188s) [noise=true , address=145.79.227.5:4247 , piece_id=QGN2S5CET2J6LGW6YPQYDWEVOGEMHWRSRMJG2IUQ36KA23OG2R3A, seg_pos={0 0}, upload=330171]: Done uploading piece. err: ecclient: upload failed (node:1262xxX6aUPTYgKdvRxAyc77PmPBWJtq5cXRJFoELxHbfn18DLE, address:145.79.227.5:4247): protocol: rpc: dial tcp 145.79.227.5:4247: connect: no route to host; conn closed: op Sep 22 05:06:24.986206552 (375h38m1.025779682s) [upload=330171]: Next returned nil segment Sep 22 05:06:24.986226830 (375h38m1.02579995s) [upload=330171]: Waiting for error group managing segments... Sep 22 05:06:25.383578176 (375h38m1.423151306s) [noise=true , address=145.79.227.24:4594 , piece_id=V55Y7DFR7QJWBBQRKSTUWYNEZSCAIWAY2XHAOZOF2H7JDFSX6IBA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.383806571 (375h38m1.423379702s) [noise=true , address=145.79.227.33:4881 , piece_id=AIKNNVT5PMYFYKRMUFN5DQMY7TWEUN3FUC64IQQYKAMV723TKKZA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.398811493 (375h38m1.438384634s) [noise=true , address=145.79.227.46:4601 , piece_id=EVWU7M2P2W4FQCNDEQ2OO3SR4KHPTXQRM7GEI6Q6DHYMDHK4PP4Q, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.399530783 (375h38m1.439103913s) [noise=true , address=145.79.227.39:4039 , piece_id=MZRBH2VHR7CCJK4OCCXINOQTNBP7PUP236H7DB2DFL7ZFDBNI2IQ, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.406508421 (375h38m1.446081551s) [noise=true , address=145.79.227.49:4338 , piece_id=NBR62GETN7QBXC7QJCUB5DZHKVMC5AKQQBIIGDRXNECNZFULEDDA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.415736332 (375h38m1.455309462s) [noise=true , address=145.79.227.25:4637 , piece_id=3SMC4AHCMKJEDXJOJVZ3FJEKW23VZDSWW6MDCENAYJ7HKOPWNHBQ, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.437405044 (375h38m1.476978184s) [noise=true , address=145.79.227.31:4818 , piece_id=KAJZKTKOHNN5U4CTYJ7CLO32ONU6VSEJ2QBLPBI5WWZZ37YOI5NA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.465513107 (375h38m1.505086237s) [noise=true , address=77.90.31.19:4847 , piece_id=6GLTB4OVJTM67VRKRWATCIWE7WVK35IQMQIVOEZMZM6I5DFNRIBA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.501349827 (375h38m1.540922957s) [noise=true , address=45.91.116.9:4292 , piece_id=3QGXKEK5C77XDD34Z77C3K2LYEMGJT2SADTL6RGW2YCYQFV5CLUA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.579007182 (375h38m1.618580302s) [noise=true , address=77.90.31.16:4764 , piece_id=PSAP3HPVNEURWWNC7K2K2G4NEUENKWDUR7OJY6UBGRTCSBYNHRWQ, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.586807715 (375h38m1.626380845s) [noise=true , address=45.91.116.14:4214 , piece_id=DOY7FC4SJKGAFP3LHSJXKWCRSYI6I6IR2R2FZ47QJNDM4SCAUXQQ, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:07:22.245417769 (375h38m58.284990899s) [noise=true , address=145.79.227.16:4933 , piece_id=WOBRP7MRD6VVMMXEHKNL4XDSPKQBHVINQD3I4N2SQU2MOSC2ZUCQ, seg_pos={0 0}, upload=330171]: Done uploading piece. err: ecclient: upload failed (node:19m4inaXvTdwf2rKip3eFELv6JAgFvXJ8KqjPZYbb48i8Ktnwe, address:145.79.227.16:4933): protocol: rpc: dial tcp 145.79.227.16:4933: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:07:22.245544310 (375h38m58.28511744s) [noise=true , address=145.79.227.12:4770 , piece_id=X5KVN5TA67L4TTD7LVG6N4I54BKNESFAHXRPVIWGDFCEPSHRKXPA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: ecclient: upload failed (node:1Ug6GpYToYDXiHmiJP4GoE7eJV3kqRveix2S5SLDcKQJtEbNPz, address:145.79.227.12:4770): protocol: rpc: dial tcp 145.79.227.12:4770: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:07:22.245567825 (375h38m58.285140955s) [noise=true , address=145.79.227.16:4939 , piece_id=DLS7J35PB7NXNMAPTVHBZ3VN3U725CSVZZKEYY37HZWQPWCCJO7A, seg_pos={0 0}, upload=330171]: Done uploading piece. err: ecclient: upload failed (node:1mRmp3HhD95PruS4AF8soRsSXNYUeb4mJE7y52FvoeN9D3Gu4F, address:145.79.227.16:4939): protocol: rpc: dial tcp 145.79.227.16:4939: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:07:22.245634181 (375h38m58.285207312s) [noise=true , address=145.79.227.7:4308 , piece_id=QEUKJMQDWYMBVKSVNNVDK53P6CZ4X47IINRSWCHAFG7APSFOOAFQ, seg_pos={0 0}, upload=330171]: Done uploading piece. err: ecclient: upload failed (node:1PgA5SZkBvK673LsV7p2sXpbi1r79esNNZspPZTzaitqfbjtyY, address:145.79.227.7:4308): protocol: rpc: dial tcp 145.79.227.7:4308: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:07:22.245657335 (375h38m58.285230455s) [noise=true , address=145.79.227.10:4420 , piece_id=ANGUQNIL2D4K2YR7KVJMY37WW7ILRVXSZOB7FRRAKJPD3AGGPGOA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: ecclient: upload failed (node:12hJrjeMUMYmFw3yxRtQQKRh9ciPfdu5FXo8NGrsc6HBqgfJzzp, address:145.79.227.10:4420): protocol: rpc: dial tcp 145.79.227.10:4420: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:07:22.245675269 (375h38m58.28524839s) [noise=true , address=145.79.227.6:4292 , piece_id=BYUJPZSYFTYD4C4F5EBAW6BSFXTILW6EASOIRHZQI7ARCSG63F2Q, seg_pos={0 0}, upload=330171]: Done uploading piece. err: ecclient: upload failed (node:12UpWmMhEmk5veKZvzsQ3MCxCMnfEKtFgCugBTsYfy9Ejm7G4xD, address:145.79.227.6:4292): protocol: rpc: dial tcp 145.79.227.6:4292: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:07:22.245712370 (375h38m58.2852855s) [seg_pos={0 0}, upload=330171]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Sep 22 05:07:22.500217377 (375h38m58.539790507s) [upload=330171]: Done uploading segments... Sep 22 05:07:22.500253296 (375h38m58.539826426s) [upload=330171]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 6.020447438s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [..........XXXX.X.XX.....X.X....................X............] duration: 6.04498722s FAIL (2025-09-22 05:39:53 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== 3.337157461s) [upload=330511]: Starting upload Sep 22 05:29:47.297668972 (376h1m23.337242102s) [upload=330511]: Done starting upload Sep 22 05:29:47.297899231 (376h1m23.337472361s) [upload=330511]: Uploading segments... Sep 22 05:29:47.298002678 (376h1m23.337575818s) [upload=330511]: Got next segment. Inline: false Sep 22 05:29:47.298059145 (376h1m23.337632275s) [upload=330511]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 05:29:47.298066109 (376h1m23.337639229s) [upload=330511]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 05:29:47.552911815 (376h1m23.592484945s) [seg_pos={0 0}, upload=330511]: Begin upload segment... Sep 22 05:29:47.552975406 (376h1m23.592548536s) [seg_pos={0 0}, upload=330511]: Done begin upload segment. Sep 22 05:29:47.553013599 (376h1m23.592586719s) [noise=true , address=45.91.116.11:4117 , piece_id=NIFPLA2N3L6XDWTIFUM3YYRGBMFQJKWFVCGYXD2GBFYVYDXW6OEQ, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.553528389 (376h1m23.593101539s) [noise=true , address=145.79.227.6:4258 , piece_id=D7HMSQH7ZLLKNTUZGJLVHIW2KEQZDL3HSS6VO2BSNRZJAC3USFAQ, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.553925034 (376h1m23.593498154s) [noise=true , address=45.91.116.9:4261 , piece_id=REZZWYP25KZOP4UBJWJPHGCDUWCW3EBOJG2DOLCDYZY5B43KBMSQ, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.553961283 (376h1m23.593534413s) [noise=true , address=77.90.31.14:4663 , piece_id=LYGO7UTA55Q5Q7APPDJZNNWZ4S2XHAT4LS4V27VF27A3435DVJQA, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.554326960 (376h1m23.59390009s) [noise=true , address=77.90.31.16:4735 , piece_id=K5KYKKY3BQV6UVJK4G3DYC24DFFIEFQX7KGGFLS4P35KJRFHA6UQ, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.554351497 (376h1m23.593924627s) [noise=true , address=145.79.227.16:4912 , piece_id=AG2FUP2KFIJ7W55H2VRYTJYWQ57MAXCTEMTDWSR4FUJAYE3HFKBQ, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.554588828 (376h1m23.594161948s) [noise=true , address=145.79.227.12:4767 , piece_id=OUVFSM62B7UZ4U4WGFFBOHCPHXEXR6A6H2FQL3DTG5L3DEPUVYVQ, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.554724957 (376h1m23.594298087s) [noise=true , address=77.90.31.18:4807 , piece_id=JFOMLF75MIXEDP5CKDHH7J3PWGWESB7BIKK3SCAIOVNX74DJDC3A, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.554875986 (376h1m23.594449116s) [noise=true , address=145.79.227.3:4151 , piece_id=O6AU3AOBKJRIUMEAWPHMWWDDNRTA52WL72JNUTK5JFLVG2TNV5IA, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.555093950 (376h1m23.59466708s) [noise=true , address=145.79.227.17:4972 , piece_id=Q3KQQ57XWDUCXNHP33STZ7DN6VQ6FVYZTDPLINEPZASFEVY5XDJA, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.555135429 (376h1m23.594708559s) [noise=true , address=45.91.116.6:4973 , piece_id=WPWUFOFQHQZFRPTR33GTDST2NYCVPXKLDVCJQFPQ4A7PI44LJKOQ, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.555464797 (376h1m23.595037917s) [noise=true , address=45.91.116.14:4189 , piece_id=6VJ6XCCEV2CZSARNRF3ATZ4KCUI75QKWIWQWJL5QH4XTQG4MCLUQ, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.555726194 (376h1m23.595299314s) [noise=true , address=77.90.31.15:4699 , piece_id=6XZN64DOILMELBYDU4D72ER33XDKMRIJRSAXX5ITFYPG46YEBITQ, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.556009483 (376h1m23.595582614s) [noise=true , address=145.79.227.19:4077 , piece_id=EDBYIPZUOPFRRIZBZJJEP2U7NZFCUHRLDZESZ7J5CQZFKADNLE6Q, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.556364609 (376h1m23.59593773s) [noise=true , address=145.79.227.15:4875 , piece_id=ZEWO52263YKTTRYDVF4PV52JTPEDJR3XGVRG3BFE5QQKDQSCE2LQ, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.556611299 (376h1m23.596184429s) [noise=true , address=145.79.227.5:4222 , piece_id=CZNVNWHETTFQS6JR2YLE65LNQU4WX74DAROY5I3KAYHG4QDLDOKA, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:49.308510933 (376h1m25.348084063s) [upload=330511]: Next returned nil segment Sep 22 05:29:49.308533666 (376h1m25.348106796s) [upload=330511]: Waiting for error group managing segments... Sep 22 05:29:49.602590138 (376h1m25.642163268s) [noise=true , address=45.91.116.11:4117 , piece_id=NIFPLA2N3L6XDWTIFUM3YYRGBMFQJKWFVCGYXD2GBFYVYDXW6OEQ, seg_pos={0 0}, upload=330511]: Done uploading piece. err: <nil> Sep 22 05:29:49.633269677 (376h1m25.672842807s) [noise=true , address=77.90.31.15:4699 , piece_id=6XZN64DOILMELBYDU4D72ER33XDKMRIJRSAXX5ITFYPG46YEBITQ, seg_pos={0 0}, upload=330511]: Done uploading piece. err: <nil> Sep 22 05:29:49.637382749 (376h1m25.676955879s) [noise=true , address=45.91.116.6:4973 , piece_id=WPWUFOFQHQZFRPTR33GTDST2NYCVPXKLDVCJQFPQ4A7PI44LJKOQ, seg_pos={0 0}, upload=330511]: Done uploading piece. err: <nil> Sep 22 05:29:49.648543782 (376h1m25.688116902s) [noise=true , address=77.90.31.16:4735 , piece_id=K5KYKKY3BQV6UVJK4G3DYC24DFFIEFQX7KGGFLS4P35KJRFHA6UQ, seg_pos={0 0}, upload=330511]: Done uploading piece. err: <nil> Sep 22 05:29:49.653934647 (376h1m25.693507767s) [noise=true , address=45.91.116.14:4189 , piece_id=6VJ6XCCEV2CZSARNRF3ATZ4KCUI75QKWIWQWJL5QH4XTQG4MCLUQ, seg_pos={0 0}, upload=330511]: Done uploading piece. err: <nil> Sep 22 05:29:49.669096699 (376h1m25.708669829s) [noise=true , address=45.91.116.9:4261 , piece_id=REZZWYP25KZOP4UBJWJPHGCDUWCW3EBOJG2DOLCDYZY5B43KBMSQ, seg_pos={0 0}, upload=330511]: Done uploading piece. err: <nil> Sep 22 05:29:49.673672402 (376h1m25.713245532s) [noise=true , address=77.90.31.14:4663 , piece_id=LYGO7UTA55Q5Q7APPDJZNNWZ4S2XHAT4LS4V27VF27A3435DVJQA, seg_pos={0 0}, upload=330511]: Done uploading piece. err: <nil> Sep 22 05:29:49.741841689 (376h1m25.781414809s) [noise=true , address=77.90.31.18:4807 , piece_id=JFOMLF75MIXEDP5CKDHH7J3PWGWESB7BIKK3SCAIOVNX74DJDC3A, seg_pos={0 0}, upload=330511]: Done uploading piece. err: <nil> Sep 22 05:29:58.021015047 (376h1m34.060588167s) [noise=true , address=145.79.227.5:4222 , piece_id=CZNVNWHETTFQS6JR2YLE65LNQU4WX74DAROY5I3KAYHG4QDLDOKA, seg_pos={0 0}, upload=330511]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:145.79.227.5:4222): protocol: rpc: dial tcp 145.79.227.5:4222: connect: no route to host; conn closed: op; piecestore: piecestore close: manager closed: conn closed: op Sep 22 05:30:05.028605724 (376h1m41.068178854s) [noise=true , address=145.79.227.3:4151 , piece_id=O6AU3AOBKJRIUMEAWPHMWWDDNRTA52WL72JNUTK5JFLVG2TNV5IA, seg_pos={0 0}, upload=330511]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:145.79.227.3:4151): protocol: rpc: dial tcp 145.79.227.3:4151: connect: no route to host; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:30:16.586571260 (376h1m52.62614439s) [noise=true , address=145.79.227.12:4767 , piece_id=OUVFSM62B7UZ4U4WGFFBOHCPHXEXR6A6H2FQL3DTG5L3DEPUVYVQ, seg_pos={0 0}, upload=330511]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:145.79.227.12:4767): protocol: rpc: dial tcp 145.79.227.12:4767: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:30:16.586679978 (376h1m52.626253118s) [noise=true , address=145.79.227.6:4258 , piece_id=D7HMSQH7ZLLKNTUZGJLVHIW2KEQZDL3HSS6VO2BSNRZJAC3USFAQ, seg_pos={0 0}, upload=330511]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:145.79.227.6:4258): protocol: rpc: dial tcp 145.79.227.6:4258: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:30:16.586708171 (376h1m52.626281301s) [noise=true , address=145.79.227.16:4912 , piece_id=AG2FUP2KFIJ7W55H2VRYTJYWQ57MAXCTEMTDWSR4FUJAYE3HFKBQ, seg_pos={0 0}, upload=330511]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:145.79.227.16:4912): protocol: rpc: dial tcp 145.79.227.16:4912: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:30:16.586749971 (376h1m52.626323101s) [noise=true , address=145.79.227.15:4875 , piece_id=ZEWO52263YKTTRYDVF4PV52JTPEDJR3XGVRG3BFE5QQKDQSCE2LQ, seg_pos={0 0}, upload=330511]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: rpc: dial tcp 145.79.227.15:4875: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:30:16.586795768 (376h1m52.626368888s) [noise=true , address=145.79.227.17:4972 , piece_id=Q3KQQ57XWDUCXNHP33STZ7DN6VQ6FVYZTDPLINEPZASFEVY5XDJA, seg_pos={0 0}, upload=330511]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:145.79.227.17:4972): protocol: rpc: dial tcp 145.79.227.17:4972: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:30:16.586806088 (376h1m52.626379218s) [noise=true , address=145.79.227.19:4077 , piece_id=EDBYIPZUOPFRRIZBZJJEP2U7NZFCUHRLDZESZ7J5CQZFKADNLE6Q, seg_pos={0 0}, upload=330511]: Done uploading piece. err: ecclient: upload failed (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk, address:145.79.227.19:4077): protocol: rpc: dial tcp 145.79.227.19:4077: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:30:16.586831426 (376h1m52.626404556s) [seg_pos={0 0}, upload=330511]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Sep 22 05:30:16.834944071 (376h1m52.874517201s) [upload=330511]: Done uploading segments... Sep 22 05:30:16.834981573 (376h1m52.874554703s) [upload=330511]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 6.045099213s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [..............X...X.........................................] duration: 5.950027373s FAIL (2025-09-22 05:39:30 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 5h37m59.543525066s) [seg_pos={0 0}, upload=330171]: Begin upload segment... Sep 22 05:06:23.504018422 (375h37m59.543591552s) [seg_pos={0 0}, upload=330171]: Done begin upload segment. Sep 22 05:06:23.504059370 (375h37m59.543632501s) [noise=true , address=45.91.116.9:4292 , piece_id=3QGXKEK5C77XDD34Z77C3K2LYEMGJT2SADTL6RGW2YCYQFV5CLUA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.504681175 (375h37m59.544254355s) [noise=true , address=145.79.227.39:4039 , piece_id=MZRBH2VHR7CCJK4OCCXINOQTNBP7PUP236H7DB2DFL7ZFDBNI2IQ, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.505278141 (375h37m59.544851301s) [noise=true , address=145.79.227.33:4881 , piece_id=AIKNNVT5PMYFYKRMUFN5DQMY7TWEUN3FUC64IQQYKAMV723TKKZA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.505743808 (375h37m59.545316948s) [noise=true , address=145.79.227.46:4601 , piece_id=EVWU7M2P2W4FQCNDEQ2OO3SR4KHPTXQRM7GEI6Q6DHYMDHK4PP4Q, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.505904454 (375h37m59.545477584s) [noise=true , address=145.79.227.24:4594 , piece_id=V55Y7DFR7QJWBBQRKSTUWYNEZSCAIWAY2XHAOZOF2H7JDFSX6IBA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.506859502 (375h37m59.546432622s) [noise=true , address=145.79.227.10:4420 , piece_id=ANGUQNIL2D4K2YR7KVJMY37WW7ILRVXSZOB7FRRAKJPD3AGGPGOA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.506958631 (375h37m59.546531761s) [noise=true , address=145.79.227.16:4933 , piece_id=WOBRP7MRD6VVMMXEHKNL4XDSPKQBHVINQD3I4N2SQU2MOSC2ZUCQ, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.507277077 (375h37m59.546850217s) [noise=true , address=145.79.227.7:4308 , piece_id=QEUKJMQDWYMBVKSVNNVDK53P6CZ4X47IINRSWCHAFG7APSFOOAFQ, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.508336856 (375h37m59.547909986s) [noise=true , address=77.90.31.19:4847 , piece_id=6GLTB4OVJTM67VRKRWATCIWE7WVK35IQMQIVOEZMZM6I5DFNRIBA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.513150782 (375h37m59.552723912s) [noise=true , address=145.79.227.12:4770 , piece_id=X5KVN5TA67L4TTD7LVG6N4I54BKNESFAHXRPVIWGDFCEPSHRKXPA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.513934875 (375h37m59.553508006s) [noise=true , address=45.91.116.14:4214 , piece_id=DOY7FC4SJKGAFP3LHSJXKWCRSYI6I6IR2R2FZ47QJNDM4SCAUXQQ, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.514566187 (375h37m59.554139318s) [noise=true , address=145.79.227.16:4939 , piece_id=DLS7J35PB7NXNMAPTVHBZ3VN3U725CSVZZKEYY37HZWQPWCCJO7A, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.515189424 (375h37m59.554762544s) [noise=true , address=145.79.227.25:4637 , piece_id=3SMC4AHCMKJEDXJOJVZ3FJEKW23VZDSWW6MDCENAYJ7HKOPWNHBQ, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.515805377 (375h37m59.555378508s) [noise=true , address=145.79.227.6:4292 , piece_id=BYUJPZSYFTYD4C4F5EBAW6BSFXTILW6EASOIRHZQI7ARCSG63F2Q, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.516494880 (375h37m59.55606801s) [noise=true , address=77.90.31.16:4764 , piece_id=PSAP3HPVNEURWWNC7K2K2G4NEUENKWDUR7OJY6UBGRTCSBYNHRWQ, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.526689722 (375h37m59.566262863s) [noise=true , address=145.79.227.31:4818 , piece_id=KAJZKTKOHNN5U4CTYJ7CLO32ONU6VSEJ2QBLPBI5WWZZ37YOI5NA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.531527154 (375h37m59.571100294s) [noise=true , address=145.79.227.49:4338 , piece_id=NBR62GETN7QBXC7QJCUB5DZHKVMC5AKQQBIIGDRXNECNZFULEDDA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.537921861 (375h37m59.577494991s) [noise=true , address=145.79.227.5:4247 , piece_id=QGN2S5CET2J6LGW6YPQYDWEVOGEMHWRSRMJG2IUQ36KA23OG2R3A, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:24.564378058 (375h38m0.603951188s) [noise=true , address=145.79.227.5:4247 , piece_id=QGN2S5CET2J6LGW6YPQYDWEVOGEMHWRSRMJG2IUQ36KA23OG2R3A, seg_pos={0 0}, upload=330171]: Done uploading piece. err: ecclient: upload failed (node:1262xxX6aUPTYgKdvRxAyc77PmPBWJtq5cXRJFoELxHbfn18DLE, address:145.79.227.5:4247): protocol: rpc: dial tcp 145.79.227.5:4247: connect: no route to host; conn closed: op Sep 22 05:06:24.986206552 (375h38m1.025779682s) [upload=330171]: Next returned nil segment Sep 22 05:06:24.986226830 (375h38m1.02579995s) [upload=330171]: Waiting for error group managing segments... Sep 22 05:06:25.383578176 (375h38m1.423151306s) [noise=true , address=145.79.227.24:4594 , piece_id=V55Y7DFR7QJWBBQRKSTUWYNEZSCAIWAY2XHAOZOF2H7JDFSX6IBA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.383806571 (375h38m1.423379702s) [noise=true , address=145.79.227.33:4881 , piece_id=AIKNNVT5PMYFYKRMUFN5DQMY7TWEUN3FUC64IQQYKAMV723TKKZA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.398811493 (375h38m1.438384634s) [noise=true , address=145.79.227.46:4601 , piece_id=EVWU7M2P2W4FQCNDEQ2OO3SR4KHPTXQRM7GEI6Q6DHYMDHK4PP4Q, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.399530783 (375h38m1.439103913s) [noise=true , address=145.79.227.39:4039 , piece_id=MZRBH2VHR7CCJK4OCCXINOQTNBP7PUP236H7DB2DFL7ZFDBNI2IQ, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.406508421 (375h38m1.446081551s) [noise=true , address=145.79.227.49:4338 , piece_id=NBR62GETN7QBXC7QJCUB5DZHKVMC5AKQQBIIGDRXNECNZFULEDDA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.415736332 (375h38m1.455309462s) [noise=true , address=145.79.227.25:4637 , piece_id=3SMC4AHCMKJEDXJOJVZ3FJEKW23VZDSWW6MDCENAYJ7HKOPWNHBQ, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.437405044 (375h38m1.476978184s) [noise=true , address=145.79.227.31:4818 , piece_id=KAJZKTKOHNN5U4CTYJ7CLO32ONU6VSEJ2QBLPBI5WWZZ37YOI5NA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.465513107 (375h38m1.505086237s) [noise=true , address=77.90.31.19:4847 , piece_id=6GLTB4OVJTM67VRKRWATCIWE7WVK35IQMQIVOEZMZM6I5DFNRIBA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.501349827 (375h38m1.540922957s) [noise=true , address=45.91.116.9:4292 , piece_id=3QGXKEK5C77XDD34Z77C3K2LYEMGJT2SADTL6RGW2YCYQFV5CLUA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.579007182 (375h38m1.618580302s) [noise=true , address=77.90.31.16:4764 , piece_id=PSAP3HPVNEURWWNC7K2K2G4NEUENKWDUR7OJY6UBGRTCSBYNHRWQ, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.586807715 (375h38m1.626380845s) [noise=true , address=45.91.116.14:4214 , piece_id=DOY7FC4SJKGAFP3LHSJXKWCRSYI6I6IR2R2FZ47QJNDM4SCAUXQQ, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:07:22.245417769 (375h38m58.284990899s) [noise=true , address=145.79.227.16:4933 , piece_id=WOBRP7MRD6VVMMXEHKNL4XDSPKQBHVINQD3I4N2SQU2MOSC2ZUCQ, seg_pos={0 0}, upload=330171]: Done uploading piece. err: ecclient: upload failed (node:19m4inaXvTdwf2rKip3eFELv6JAgFvXJ8KqjPZYbb48i8Ktnwe, address:145.79.227.16:4933): protocol: rpc: dial tcp 145.79.227.16:4933: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:07:22.245544310 (375h38m58.28511744s) [noise=true , address=145.79.227.12:4770 , piece_id=X5KVN5TA67L4TTD7LVG6N4I54BKNESFAHXRPVIWGDFCEPSHRKXPA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: ecclient: upload failed (node:1Ug6GpYToYDXiHmiJP4GoE7eJV3kqRveix2S5SLDcKQJtEbNPz, address:145.79.227.12:4770): protocol: rpc: dial tcp 145.79.227.12:4770: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:07:22.245567825 (375h38m58.285140955s) [noise=true , address=145.79.227.16:4939 , piece_id=DLS7J35PB7NXNMAPTVHBZ3VN3U725CSVZZKEYY37HZWQPWCCJO7A, seg_pos={0 0}, upload=330171]: Done uploading piece. err: ecclient: upload failed (node:1mRmp3HhD95PruS4AF8soRsSXNYUeb4mJE7y52FvoeN9D3Gu4F, address:145.79.227.16:4939): protocol: rpc: dial tcp 145.79.227.16:4939: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:07:22.245634181 (375h38m58.285207312s) [noise=true , address=145.79.227.7:4308 , piece_id=QEUKJMQDWYMBVKSVNNVDK53P6CZ4X47IINRSWCHAFG7APSFOOAFQ, seg_pos={0 0}, upload=330171]: Done uploading piece. err: ecclient: upload failed (node:1PgA5SZkBvK673LsV7p2sXpbi1r79esNNZspPZTzaitqfbjtyY, address:145.79.227.7:4308): protocol: rpc: dial tcp 145.79.227.7:4308: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:07:22.245657335 (375h38m58.285230455s) [noise=true , address=145.79.227.10:4420 , piece_id=ANGUQNIL2D4K2YR7KVJMY37WW7ILRVXSZOB7FRRAKJPD3AGGPGOA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: ecclient: upload failed (node:12hJrjeMUMYmFw3yxRtQQKRh9ciPfdu5FXo8NGrsc6HBqgfJzzp, address:145.79.227.10:4420): protocol: rpc: dial tcp 145.79.227.10:4420: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:07:22.245675269 (375h38m58.28524839s) [noise=true , address=145.79.227.6:4292 , piece_id=BYUJPZSYFTYD4C4F5EBAW6BSFXTILW6EASOIRHZQI7ARCSG63F2Q, seg_pos={0 0}, upload=330171]: Done uploading piece. err: ecclient: upload failed (node:12UpWmMhEmk5veKZvzsQ3MCxCMnfEKtFgCugBTsYfy9Ejm7G4xD, address:145.79.227.6:4292): protocol: rpc: dial tcp 145.79.227.6:4292: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:07:22.245712370 (375h38m58.2852855s) [seg_pos={0 0}, upload=330171]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Sep 22 05:07:22.500217377 (375h38m58.539790507s) [upload=330171]: Done uploading segments... Sep 22 05:07:22.500253296 (375h38m58.539826426s) [upload=330171]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 5.950082038s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [...........XXXX.X.XX.....X.X....................X...........] duration: 5.962650872s FAIL (2025-09-22 05:39:15 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== 3.337157461s) [upload=330511]: Starting upload Sep 22 05:29:47.297668972 (376h1m23.337242102s) [upload=330511]: Done starting upload Sep 22 05:29:47.297899231 (376h1m23.337472361s) [upload=330511]: Uploading segments... Sep 22 05:29:47.298002678 (376h1m23.337575818s) [upload=330511]: Got next segment. Inline: false Sep 22 05:29:47.298059145 (376h1m23.337632275s) [upload=330511]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 05:29:47.298066109 (376h1m23.337639229s) [upload=330511]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 05:29:47.552911815 (376h1m23.592484945s) [seg_pos={0 0}, upload=330511]: Begin upload segment... Sep 22 05:29:47.552975406 (376h1m23.592548536s) [seg_pos={0 0}, upload=330511]: Done begin upload segment. Sep 22 05:29:47.553013599 (376h1m23.592586719s) [noise=true , address=45.91.116.11:4117 , piece_id=NIFPLA2N3L6XDWTIFUM3YYRGBMFQJKWFVCGYXD2GBFYVYDXW6OEQ, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.553528389 (376h1m23.593101539s) [noise=true , address=145.79.227.6:4258 , piece_id=D7HMSQH7ZLLKNTUZGJLVHIW2KEQZDL3HSS6VO2BSNRZJAC3USFAQ, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.553925034 (376h1m23.593498154s) [noise=true , address=45.91.116.9:4261 , piece_id=REZZWYP25KZOP4UBJWJPHGCDUWCW3EBOJG2DOLCDYZY5B43KBMSQ, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.553961283 (376h1m23.593534413s) [noise=true , address=77.90.31.14:4663 , piece_id=LYGO7UTA55Q5Q7APPDJZNNWZ4S2XHAT4LS4V27VF27A3435DVJQA, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.554326960 (376h1m23.59390009s) [noise=true , address=77.90.31.16:4735 , piece_id=K5KYKKY3BQV6UVJK4G3DYC24DFFIEFQX7KGGFLS4P35KJRFHA6UQ, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.554351497 (376h1m23.593924627s) [noise=true , address=145.79.227.16:4912 , piece_id=AG2FUP2KFIJ7W55H2VRYTJYWQ57MAXCTEMTDWSR4FUJAYE3HFKBQ, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.554588828 (376h1m23.594161948s) [noise=true , address=145.79.227.12:4767 , piece_id=OUVFSM62B7UZ4U4WGFFBOHCPHXEXR6A6H2FQL3DTG5L3DEPUVYVQ, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.554724957 (376h1m23.594298087s) [noise=true , address=77.90.31.18:4807 , piece_id=JFOMLF75MIXEDP5CKDHH7J3PWGWESB7BIKK3SCAIOVNX74DJDC3A, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.554875986 (376h1m23.594449116s) [noise=true , address=145.79.227.3:4151 , piece_id=O6AU3AOBKJRIUMEAWPHMWWDDNRTA52WL72JNUTK5JFLVG2TNV5IA, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.555093950 (376h1m23.59466708s) [noise=true , address=145.79.227.17:4972 , piece_id=Q3KQQ57XWDUCXNHP33STZ7DN6VQ6FVYZTDPLINEPZASFEVY5XDJA, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.555135429 (376h1m23.594708559s) [noise=true , address=45.91.116.6:4973 , piece_id=WPWUFOFQHQZFRPTR33GTDST2NYCVPXKLDVCJQFPQ4A7PI44LJKOQ, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.555464797 (376h1m23.595037917s) [noise=true , address=45.91.116.14:4189 , piece_id=6VJ6XCCEV2CZSARNRF3ATZ4KCUI75QKWIWQWJL5QH4XTQG4MCLUQ, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.555726194 (376h1m23.595299314s) [noise=true , address=77.90.31.15:4699 , piece_id=6XZN64DOILMELBYDU4D72ER33XDKMRIJRSAXX5ITFYPG46YEBITQ, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.556009483 (376h1m23.595582614s) [noise=true , address=145.79.227.19:4077 , piece_id=EDBYIPZUOPFRRIZBZJJEP2U7NZFCUHRLDZESZ7J5CQZFKADNLE6Q, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.556364609 (376h1m23.59593773s) [noise=true , address=145.79.227.15:4875 , piece_id=ZEWO52263YKTTRYDVF4PV52JTPEDJR3XGVRG3BFE5QQKDQSCE2LQ, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.556611299 (376h1m23.596184429s) [noise=true , address=145.79.227.5:4222 , piece_id=CZNVNWHETTFQS6JR2YLE65LNQU4WX74DAROY5I3KAYHG4QDLDOKA, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:49.308510933 (376h1m25.348084063s) [upload=330511]: Next returned nil segment Sep 22 05:29:49.308533666 (376h1m25.348106796s) [upload=330511]: Waiting for error group managing segments... Sep 22 05:29:49.602590138 (376h1m25.642163268s) [noise=true , address=45.91.116.11:4117 , piece_id=NIFPLA2N3L6XDWTIFUM3YYRGBMFQJKWFVCGYXD2GBFYVYDXW6OEQ, seg_pos={0 0}, upload=330511]: Done uploading piece. err: <nil> Sep 22 05:29:49.633269677 (376h1m25.672842807s) [noise=true , address=77.90.31.15:4699 , piece_id=6XZN64DOILMELBYDU4D72ER33XDKMRIJRSAXX5ITFYPG46YEBITQ, seg_pos={0 0}, upload=330511]: Done uploading piece. err: <nil> Sep 22 05:29:49.637382749 (376h1m25.676955879s) [noise=true , address=45.91.116.6:4973 , piece_id=WPWUFOFQHQZFRPTR33GTDST2NYCVPXKLDVCJQFPQ4A7PI44LJKOQ, seg_pos={0 0}, upload=330511]: Done uploading piece. err: <nil> Sep 22 05:29:49.648543782 (376h1m25.688116902s) [noise=true , address=77.90.31.16:4735 , piece_id=K5KYKKY3BQV6UVJK4G3DYC24DFFIEFQX7KGGFLS4P35KJRFHA6UQ, seg_pos={0 0}, upload=330511]: Done uploading piece. err: <nil> Sep 22 05:29:49.653934647 (376h1m25.693507767s) [noise=true , address=45.91.116.14:4189 , piece_id=6VJ6XCCEV2CZSARNRF3ATZ4KCUI75QKWIWQWJL5QH4XTQG4MCLUQ, seg_pos={0 0}, upload=330511]: Done uploading piece. err: <nil> Sep 22 05:29:49.669096699 (376h1m25.708669829s) [noise=true , address=45.91.116.9:4261 , piece_id=REZZWYP25KZOP4UBJWJPHGCDUWCW3EBOJG2DOLCDYZY5B43KBMSQ, seg_pos={0 0}, upload=330511]: Done uploading piece. err: <nil> Sep 22 05:29:49.673672402 (376h1m25.713245532s) [noise=true , address=77.90.31.14:4663 , piece_id=LYGO7UTA55Q5Q7APPDJZNNWZ4S2XHAT4LS4V27VF27A3435DVJQA, seg_pos={0 0}, upload=330511]: Done uploading piece. err: <nil> Sep 22 05:29:49.741841689 (376h1m25.781414809s) [noise=true , address=77.90.31.18:4807 , piece_id=JFOMLF75MIXEDP5CKDHH7J3PWGWESB7BIKK3SCAIOVNX74DJDC3A, seg_pos={0 0}, upload=330511]: Done uploading piece. err: <nil> Sep 22 05:29:58.021015047 (376h1m34.060588167s) [noise=true , address=145.79.227.5:4222 , piece_id=CZNVNWHETTFQS6JR2YLE65LNQU4WX74DAROY5I3KAYHG4QDLDOKA, seg_pos={0 0}, upload=330511]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:145.79.227.5:4222): protocol: rpc: dial tcp 145.79.227.5:4222: connect: no route to host; conn closed: op; piecestore: piecestore close: manager closed: conn closed: op Sep 22 05:30:05.028605724 (376h1m41.068178854s) [noise=true , address=145.79.227.3:4151 , piece_id=O6AU3AOBKJRIUMEAWPHMWWDDNRTA52WL72JNUTK5JFLVG2TNV5IA, seg_pos={0 0}, upload=330511]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:145.79.227.3:4151): protocol: rpc: dial tcp 145.79.227.3:4151: connect: no route to host; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:30:16.586571260 (376h1m52.62614439s) [noise=true , address=145.79.227.12:4767 , piece_id=OUVFSM62B7UZ4U4WGFFBOHCPHXEXR6A6H2FQL3DTG5L3DEPUVYVQ, seg_pos={0 0}, upload=330511]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:145.79.227.12:4767): protocol: rpc: dial tcp 145.79.227.12:4767: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:30:16.586679978 (376h1m52.626253118s) [noise=true , address=145.79.227.6:4258 , piece_id=D7HMSQH7ZLLKNTUZGJLVHIW2KEQZDL3HSS6VO2BSNRZJAC3USFAQ, seg_pos={0 0}, upload=330511]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:145.79.227.6:4258): protocol: rpc: dial tcp 145.79.227.6:4258: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:30:16.586708171 (376h1m52.626281301s) [noise=true , address=145.79.227.16:4912 , piece_id=AG2FUP2KFIJ7W55H2VRYTJYWQ57MAXCTEMTDWSR4FUJAYE3HFKBQ, seg_pos={0 0}, upload=330511]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:145.79.227.16:4912): protocol: rpc: dial tcp 145.79.227.16:4912: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:30:16.586749971 (376h1m52.626323101s) [noise=true , address=145.79.227.15:4875 , piece_id=ZEWO52263YKTTRYDVF4PV52JTPEDJR3XGVRG3BFE5QQKDQSCE2LQ, seg_pos={0 0}, upload=330511]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: rpc: dial tcp 145.79.227.15:4875: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:30:16.586795768 (376h1m52.626368888s) [noise=true , address=145.79.227.17:4972 , piece_id=Q3KQQ57XWDUCXNHP33STZ7DN6VQ6FVYZTDPLINEPZASFEVY5XDJA, seg_pos={0 0}, upload=330511]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:145.79.227.17:4972): protocol: rpc: dial tcp 145.79.227.17:4972: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:30:16.586806088 (376h1m52.626379218s) [noise=true , address=145.79.227.19:4077 , piece_id=EDBYIPZUOPFRRIZBZJJEP2U7NZFCUHRLDZESZ7J5CQZFKADNLE6Q, seg_pos={0 0}, upload=330511]: Done uploading piece. err: ecclient: upload failed (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk, address:145.79.227.19:4077): protocol: rpc: dial tcp 145.79.227.19:4077: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:30:16.586831426 (376h1m52.626404556s) [seg_pos={0 0}, upload=330511]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Sep 22 05:30:16.834944071 (376h1m52.874517201s) [upload=330511]: Done uploading segments... Sep 22 05:30:16.834981573 (376h1m52.874554703s) [upload=330511]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 5.962694665s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [...........XX....X..X...X.X......X.........X.X..X....X..XX..] duration: 6.017013522s FAIL (2025-09-22 05:39:14 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.017040001s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [............XX....X..X...X.X......X.........X.X..X....X..XX.] duration: 6.190840254s FAIL (2025-09-22 05:38:33 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.190885751s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [...............X...X........................................] duration: 5.800656134s FAIL (2025-09-22 05:38:23 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 5h37m59.543525066s) [seg_pos={0 0}, upload=330171]: Begin upload segment... Sep 22 05:06:23.504018422 (375h37m59.543591552s) [seg_pos={0 0}, upload=330171]: Done begin upload segment. Sep 22 05:06:23.504059370 (375h37m59.543632501s) [noise=true , address=45.91.116.9:4292 , piece_id=3QGXKEK5C77XDD34Z77C3K2LYEMGJT2SADTL6RGW2YCYQFV5CLUA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.504681175 (375h37m59.544254355s) [noise=true , address=145.79.227.39:4039 , piece_id=MZRBH2VHR7CCJK4OCCXINOQTNBP7PUP236H7DB2DFL7ZFDBNI2IQ, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.505278141 (375h37m59.544851301s) [noise=true , address=145.79.227.33:4881 , piece_id=AIKNNVT5PMYFYKRMUFN5DQMY7TWEUN3FUC64IQQYKAMV723TKKZA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.505743808 (375h37m59.545316948s) [noise=true , address=145.79.227.46:4601 , piece_id=EVWU7M2P2W4FQCNDEQ2OO3SR4KHPTXQRM7GEI6Q6DHYMDHK4PP4Q, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.505904454 (375h37m59.545477584s) [noise=true , address=145.79.227.24:4594 , piece_id=V55Y7DFR7QJWBBQRKSTUWYNEZSCAIWAY2XHAOZOF2H7JDFSX6IBA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.506859502 (375h37m59.546432622s) [noise=true , address=145.79.227.10:4420 , piece_id=ANGUQNIL2D4K2YR7KVJMY37WW7ILRVXSZOB7FRRAKJPD3AGGPGOA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.506958631 (375h37m59.546531761s) [noise=true , address=145.79.227.16:4933 , piece_id=WOBRP7MRD6VVMMXEHKNL4XDSPKQBHVINQD3I4N2SQU2MOSC2ZUCQ, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.507277077 (375h37m59.546850217s) [noise=true , address=145.79.227.7:4308 , piece_id=QEUKJMQDWYMBVKSVNNVDK53P6CZ4X47IINRSWCHAFG7APSFOOAFQ, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.508336856 (375h37m59.547909986s) [noise=true , address=77.90.31.19:4847 , piece_id=6GLTB4OVJTM67VRKRWATCIWE7WVK35IQMQIVOEZMZM6I5DFNRIBA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.513150782 (375h37m59.552723912s) [noise=true , address=145.79.227.12:4770 , piece_id=X5KVN5TA67L4TTD7LVG6N4I54BKNESFAHXRPVIWGDFCEPSHRKXPA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.513934875 (375h37m59.553508006s) [noise=true , address=45.91.116.14:4214 , piece_id=DOY7FC4SJKGAFP3LHSJXKWCRSYI6I6IR2R2FZ47QJNDM4SCAUXQQ, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.514566187 (375h37m59.554139318s) [noise=true , address=145.79.227.16:4939 , piece_id=DLS7J35PB7NXNMAPTVHBZ3VN3U725CSVZZKEYY37HZWQPWCCJO7A, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.515189424 (375h37m59.554762544s) [noise=true , address=145.79.227.25:4637 , piece_id=3SMC4AHCMKJEDXJOJVZ3FJEKW23VZDSWW6MDCENAYJ7HKOPWNHBQ, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.515805377 (375h37m59.555378508s) [noise=true , address=145.79.227.6:4292 , piece_id=BYUJPZSYFTYD4C4F5EBAW6BSFXTILW6EASOIRHZQI7ARCSG63F2Q, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.516494880 (375h37m59.55606801s) [noise=true , address=77.90.31.16:4764 , piece_id=PSAP3HPVNEURWWNC7K2K2G4NEUENKWDUR7OJY6UBGRTCSBYNHRWQ, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.526689722 (375h37m59.566262863s) [noise=true , address=145.79.227.31:4818 , piece_id=KAJZKTKOHNN5U4CTYJ7CLO32ONU6VSEJ2QBLPBI5WWZZ37YOI5NA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.531527154 (375h37m59.571100294s) [noise=true , address=145.79.227.49:4338 , piece_id=NBR62GETN7QBXC7QJCUB5DZHKVMC5AKQQBIIGDRXNECNZFULEDDA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.537921861 (375h37m59.577494991s) [noise=true , address=145.79.227.5:4247 , piece_id=QGN2S5CET2J6LGW6YPQYDWEVOGEMHWRSRMJG2IUQ36KA23OG2R3A, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:24.564378058 (375h38m0.603951188s) [noise=true , address=145.79.227.5:4247 , piece_id=QGN2S5CET2J6LGW6YPQYDWEVOGEMHWRSRMJG2IUQ36KA23OG2R3A, seg_pos={0 0}, upload=330171]: Done uploading piece. err: ecclient: upload failed (node:1262xxX6aUPTYgKdvRxAyc77PmPBWJtq5cXRJFoELxHbfn18DLE, address:145.79.227.5:4247): protocol: rpc: dial tcp 145.79.227.5:4247: connect: no route to host; conn closed: op Sep 22 05:06:24.986206552 (375h38m1.025779682s) [upload=330171]: Next returned nil segment Sep 22 05:06:24.986226830 (375h38m1.02579995s) [upload=330171]: Waiting for error group managing segments... Sep 22 05:06:25.383578176 (375h38m1.423151306s) [noise=true , address=145.79.227.24:4594 , piece_id=V55Y7DFR7QJWBBQRKSTUWYNEZSCAIWAY2XHAOZOF2H7JDFSX6IBA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.383806571 (375h38m1.423379702s) [noise=true , address=145.79.227.33:4881 , piece_id=AIKNNVT5PMYFYKRMUFN5DQMY7TWEUN3FUC64IQQYKAMV723TKKZA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.398811493 (375h38m1.438384634s) [noise=true , address=145.79.227.46:4601 , piece_id=EVWU7M2P2W4FQCNDEQ2OO3SR4KHPTXQRM7GEI6Q6DHYMDHK4PP4Q, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.399530783 (375h38m1.439103913s) [noise=true , address=145.79.227.39:4039 , piece_id=MZRBH2VHR7CCJK4OCCXINOQTNBP7PUP236H7DB2DFL7ZFDBNI2IQ, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.406508421 (375h38m1.446081551s) [noise=true , address=145.79.227.49:4338 , piece_id=NBR62GETN7QBXC7QJCUB5DZHKVMC5AKQQBIIGDRXNECNZFULEDDA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.415736332 (375h38m1.455309462s) [noise=true , address=145.79.227.25:4637 , piece_id=3SMC4AHCMKJEDXJOJVZ3FJEKW23VZDSWW6MDCENAYJ7HKOPWNHBQ, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.437405044 (375h38m1.476978184s) [noise=true , address=145.79.227.31:4818 , piece_id=KAJZKTKOHNN5U4CTYJ7CLO32ONU6VSEJ2QBLPBI5WWZZ37YOI5NA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.465513107 (375h38m1.505086237s) [noise=true , address=77.90.31.19:4847 , piece_id=6GLTB4OVJTM67VRKRWATCIWE7WVK35IQMQIVOEZMZM6I5DFNRIBA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.501349827 (375h38m1.540922957s) [noise=true , address=45.91.116.9:4292 , piece_id=3QGXKEK5C77XDD34Z77C3K2LYEMGJT2SADTL6RGW2YCYQFV5CLUA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.579007182 (375h38m1.618580302s) [noise=true , address=77.90.31.16:4764 , piece_id=PSAP3HPVNEURWWNC7K2K2G4NEUENKWDUR7OJY6UBGRTCSBYNHRWQ, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.586807715 (375h38m1.626380845s) [noise=true , address=45.91.116.14:4214 , piece_id=DOY7FC4SJKGAFP3LHSJXKWCRSYI6I6IR2R2FZ47QJNDM4SCAUXQQ, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:07:22.245417769 (375h38m58.284990899s) [noise=true , address=145.79.227.16:4933 , piece_id=WOBRP7MRD6VVMMXEHKNL4XDSPKQBHVINQD3I4N2SQU2MOSC2ZUCQ, seg_pos={0 0}, upload=330171]: Done uploading piece. err: ecclient: upload failed (node:19m4inaXvTdwf2rKip3eFELv6JAgFvXJ8KqjPZYbb48i8Ktnwe, address:145.79.227.16:4933): protocol: rpc: dial tcp 145.79.227.16:4933: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:07:22.245544310 (375h38m58.28511744s) [noise=true , address=145.79.227.12:4770 , piece_id=X5KVN5TA67L4TTD7LVG6N4I54BKNESFAHXRPVIWGDFCEPSHRKXPA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: ecclient: upload failed (node:1Ug6GpYToYDXiHmiJP4GoE7eJV3kqRveix2S5SLDcKQJtEbNPz, address:145.79.227.12:4770): protocol: rpc: dial tcp 145.79.227.12:4770: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:07:22.245567825 (375h38m58.285140955s) [noise=true , address=145.79.227.16:4939 , piece_id=DLS7J35PB7NXNMAPTVHBZ3VN3U725CSVZZKEYY37HZWQPWCCJO7A, seg_pos={0 0}, upload=330171]: Done uploading piece. err: ecclient: upload failed (node:1mRmp3HhD95PruS4AF8soRsSXNYUeb4mJE7y52FvoeN9D3Gu4F, address:145.79.227.16:4939): protocol: rpc: dial tcp 145.79.227.16:4939: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:07:22.245634181 (375h38m58.285207312s) [noise=true , address=145.79.227.7:4308 , piece_id=QEUKJMQDWYMBVKSVNNVDK53P6CZ4X47IINRSWCHAFG7APSFOOAFQ, seg_pos={0 0}, upload=330171]: Done uploading piece. err: ecclient: upload failed (node:1PgA5SZkBvK673LsV7p2sXpbi1r79esNNZspPZTzaitqfbjtyY, address:145.79.227.7:4308): protocol: rpc: dial tcp 145.79.227.7:4308: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:07:22.245657335 (375h38m58.285230455s) [noise=true , address=145.79.227.10:4420 , piece_id=ANGUQNIL2D4K2YR7KVJMY37WW7ILRVXSZOB7FRRAKJPD3AGGPGOA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: ecclient: upload failed (node:12hJrjeMUMYmFw3yxRtQQKRh9ciPfdu5FXo8NGrsc6HBqgfJzzp, address:145.79.227.10:4420): protocol: rpc: dial tcp 145.79.227.10:4420: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:07:22.245675269 (375h38m58.28524839s) [noise=true , address=145.79.227.6:4292 , piece_id=BYUJPZSYFTYD4C4F5EBAW6BSFXTILW6EASOIRHZQI7ARCSG63F2Q, seg_pos={0 0}, upload=330171]: Done uploading piece. err: ecclient: upload failed (node:12UpWmMhEmk5veKZvzsQ3MCxCMnfEKtFgCugBTsYfy9Ejm7G4xD, address:145.79.227.6:4292): protocol: rpc: dial tcp 145.79.227.6:4292: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:07:22.245712370 (375h38m58.2852855s) [seg_pos={0 0}, upload=330171]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Sep 22 05:07:22.500217377 (375h38m58.539790507s) [upload=330171]: Done uploading segments... Sep 22 05:07:22.500253296 (375h38m58.539826426s) [upload=330171]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 5.800691592s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [............XXXX.X.XX.....X.X....................X..........] duration: 5.985127064s FAIL (2025-09-22 05:38:14 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== 3.337157461s) [upload=330511]: Starting upload Sep 22 05:29:47.297668972 (376h1m23.337242102s) [upload=330511]: Done starting upload Sep 22 05:29:47.297899231 (376h1m23.337472361s) [upload=330511]: Uploading segments... Sep 22 05:29:47.298002678 (376h1m23.337575818s) [upload=330511]: Got next segment. Inline: false Sep 22 05:29:47.298059145 (376h1m23.337632275s) [upload=330511]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 05:29:47.298066109 (376h1m23.337639229s) [upload=330511]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 05:29:47.552911815 (376h1m23.592484945s) [seg_pos={0 0}, upload=330511]: Begin upload segment... Sep 22 05:29:47.552975406 (376h1m23.592548536s) [seg_pos={0 0}, upload=330511]: Done begin upload segment. Sep 22 05:29:47.553013599 (376h1m23.592586719s) [noise=true , address=45.91.116.11:4117 , piece_id=NIFPLA2N3L6XDWTIFUM3YYRGBMFQJKWFVCGYXD2GBFYVYDXW6OEQ, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.553528389 (376h1m23.593101539s) [noise=true , address=145.79.227.6:4258 , piece_id=D7HMSQH7ZLLKNTUZGJLVHIW2KEQZDL3HSS6VO2BSNRZJAC3USFAQ, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.553925034 (376h1m23.593498154s) [noise=true , address=45.91.116.9:4261 , piece_id=REZZWYP25KZOP4UBJWJPHGCDUWCW3EBOJG2DOLCDYZY5B43KBMSQ, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.553961283 (376h1m23.593534413s) [noise=true , address=77.90.31.14:4663 , piece_id=LYGO7UTA55Q5Q7APPDJZNNWZ4S2XHAT4LS4V27VF27A3435DVJQA, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.554326960 (376h1m23.59390009s) [noise=true , address=77.90.31.16:4735 , piece_id=K5KYKKY3BQV6UVJK4G3DYC24DFFIEFQX7KGGFLS4P35KJRFHA6UQ, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.554351497 (376h1m23.593924627s) [noise=true , address=145.79.227.16:4912 , piece_id=AG2FUP2KFIJ7W55H2VRYTJYWQ57MAXCTEMTDWSR4FUJAYE3HFKBQ, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.554588828 (376h1m23.594161948s) [noise=true , address=145.79.227.12:4767 , piece_id=OUVFSM62B7UZ4U4WGFFBOHCPHXEXR6A6H2FQL3DTG5L3DEPUVYVQ, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.554724957 (376h1m23.594298087s) [noise=true , address=77.90.31.18:4807 , piece_id=JFOMLF75MIXEDP5CKDHH7J3PWGWESB7BIKK3SCAIOVNX74DJDC3A, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.554875986 (376h1m23.594449116s) [noise=true , address=145.79.227.3:4151 , piece_id=O6AU3AOBKJRIUMEAWPHMWWDDNRTA52WL72JNUTK5JFLVG2TNV5IA, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.555093950 (376h1m23.59466708s) [noise=true , address=145.79.227.17:4972 , piece_id=Q3KQQ57XWDUCXNHP33STZ7DN6VQ6FVYZTDPLINEPZASFEVY5XDJA, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.555135429 (376h1m23.594708559s) [noise=true , address=45.91.116.6:4973 , piece_id=WPWUFOFQHQZFRPTR33GTDST2NYCVPXKLDVCJQFPQ4A7PI44LJKOQ, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.555464797 (376h1m23.595037917s) [noise=true , address=45.91.116.14:4189 , piece_id=6VJ6XCCEV2CZSARNRF3ATZ4KCUI75QKWIWQWJL5QH4XTQG4MCLUQ, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.555726194 (376h1m23.595299314s) [noise=true , address=77.90.31.15:4699 , piece_id=6XZN64DOILMELBYDU4D72ER33XDKMRIJRSAXX5ITFYPG46YEBITQ, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.556009483 (376h1m23.595582614s) [noise=true , address=145.79.227.19:4077 , piece_id=EDBYIPZUOPFRRIZBZJJEP2U7NZFCUHRLDZESZ7J5CQZFKADNLE6Q, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.556364609 (376h1m23.59593773s) [noise=true , address=145.79.227.15:4875 , piece_id=ZEWO52263YKTTRYDVF4PV52JTPEDJR3XGVRG3BFE5QQKDQSCE2LQ, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.556611299 (376h1m23.596184429s) [noise=true , address=145.79.227.5:4222 , piece_id=CZNVNWHETTFQS6JR2YLE65LNQU4WX74DAROY5I3KAYHG4QDLDOKA, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:49.308510933 (376h1m25.348084063s) [upload=330511]: Next returned nil segment Sep 22 05:29:49.308533666 (376h1m25.348106796s) [upload=330511]: Waiting for error group managing segments... Sep 22 05:29:49.602590138 (376h1m25.642163268s) [noise=true , address=45.91.116.11:4117 , piece_id=NIFPLA2N3L6XDWTIFUM3YYRGBMFQJKWFVCGYXD2GBFYVYDXW6OEQ, seg_pos={0 0}, upload=330511]: Done uploading piece. err: <nil> Sep 22 05:29:49.633269677 (376h1m25.672842807s) [noise=true , address=77.90.31.15:4699 , piece_id=6XZN64DOILMELBYDU4D72ER33XDKMRIJRSAXX5ITFYPG46YEBITQ, seg_pos={0 0}, upload=330511]: Done uploading piece. err: <nil> Sep 22 05:29:49.637382749 (376h1m25.676955879s) [noise=true , address=45.91.116.6:4973 , piece_id=WPWUFOFQHQZFRPTR33GTDST2NYCVPXKLDVCJQFPQ4A7PI44LJKOQ, seg_pos={0 0}, upload=330511]: Done uploading piece. err: <nil> Sep 22 05:29:49.648543782 (376h1m25.688116902s) [noise=true , address=77.90.31.16:4735 , piece_id=K5KYKKY3BQV6UVJK4G3DYC24DFFIEFQX7KGGFLS4P35KJRFHA6UQ, seg_pos={0 0}, upload=330511]: Done uploading piece. err: <nil> Sep 22 05:29:49.653934647 (376h1m25.693507767s) [noise=true , address=45.91.116.14:4189 , piece_id=6VJ6XCCEV2CZSARNRF3ATZ4KCUI75QKWIWQWJL5QH4XTQG4MCLUQ, seg_pos={0 0}, upload=330511]: Done uploading piece. err: <nil> Sep 22 05:29:49.669096699 (376h1m25.708669829s) [noise=true , address=45.91.116.9:4261 , piece_id=REZZWYP25KZOP4UBJWJPHGCDUWCW3EBOJG2DOLCDYZY5B43KBMSQ, seg_pos={0 0}, upload=330511]: Done uploading piece. err: <nil> Sep 22 05:29:49.673672402 (376h1m25.713245532s) [noise=true , address=77.90.31.14:4663 , piece_id=LYGO7UTA55Q5Q7APPDJZNNWZ4S2XHAT4LS4V27VF27A3435DVJQA, seg_pos={0 0}, upload=330511]: Done uploading piece. err: <nil> Sep 22 05:29:49.741841689 (376h1m25.781414809s) [noise=true , address=77.90.31.18:4807 , piece_id=JFOMLF75MIXEDP5CKDHH7J3PWGWESB7BIKK3SCAIOVNX74DJDC3A, seg_pos={0 0}, upload=330511]: Done uploading piece. err: <nil> Sep 22 05:29:58.021015047 (376h1m34.060588167s) [noise=true , address=145.79.227.5:4222 , piece_id=CZNVNWHETTFQS6JR2YLE65LNQU4WX74DAROY5I3KAYHG4QDLDOKA, seg_pos={0 0}, upload=330511]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:145.79.227.5:4222): protocol: rpc: dial tcp 145.79.227.5:4222: connect: no route to host; conn closed: op; piecestore: piecestore close: manager closed: conn closed: op Sep 22 05:30:05.028605724 (376h1m41.068178854s) [noise=true , address=145.79.227.3:4151 , piece_id=O6AU3AOBKJRIUMEAWPHMWWDDNRTA52WL72JNUTK5JFLVG2TNV5IA, seg_pos={0 0}, upload=330511]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:145.79.227.3:4151): protocol: rpc: dial tcp 145.79.227.3:4151: connect: no route to host; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:30:16.586571260 (376h1m52.62614439s) [noise=true , address=145.79.227.12:4767 , piece_id=OUVFSM62B7UZ4U4WGFFBOHCPHXEXR6A6H2FQL3DTG5L3DEPUVYVQ, seg_pos={0 0}, upload=330511]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:145.79.227.12:4767): protocol: rpc: dial tcp 145.79.227.12:4767: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:30:16.586679978 (376h1m52.626253118s) [noise=true , address=145.79.227.6:4258 , piece_id=D7HMSQH7ZLLKNTUZGJLVHIW2KEQZDL3HSS6VO2BSNRZJAC3USFAQ, seg_pos={0 0}, upload=330511]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:145.79.227.6:4258): protocol: rpc: dial tcp 145.79.227.6:4258: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:30:16.586708171 (376h1m52.626281301s) [noise=true , address=145.79.227.16:4912 , piece_id=AG2FUP2KFIJ7W55H2VRYTJYWQ57MAXCTEMTDWSR4FUJAYE3HFKBQ, seg_pos={0 0}, upload=330511]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:145.79.227.16:4912): protocol: rpc: dial tcp 145.79.227.16:4912: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:30:16.586749971 (376h1m52.626323101s) [noise=true , address=145.79.227.15:4875 , piece_id=ZEWO52263YKTTRYDVF4PV52JTPEDJR3XGVRG3BFE5QQKDQSCE2LQ, seg_pos={0 0}, upload=330511]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: rpc: dial tcp 145.79.227.15:4875: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:30:16.586795768 (376h1m52.626368888s) [noise=true , address=145.79.227.17:4972 , piece_id=Q3KQQ57XWDUCXNHP33STZ7DN6VQ6FVYZTDPLINEPZASFEVY5XDJA, seg_pos={0 0}, upload=330511]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:145.79.227.17:4972): protocol: rpc: dial tcp 145.79.227.17:4972: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:30:16.586806088 (376h1m52.626379218s) [noise=true , address=145.79.227.19:4077 , piece_id=EDBYIPZUOPFRRIZBZJJEP2U7NZFCUHRLDZESZ7J5CQZFKADNLE6Q, seg_pos={0 0}, upload=330511]: Done uploading piece. err: ecclient: upload failed (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk, address:145.79.227.19:4077): protocol: rpc: dial tcp 145.79.227.19:4077: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:30:16.586831426 (376h1m52.626404556s) [seg_pos={0 0}, upload=330511]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Sep 22 05:30:16.834944071 (376h1m52.874517201s) [upload=330511]: Done uploading segments... Sep 22 05:30:16.834981573 (376h1m52.874554703s) [upload=330511]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 5.985149087s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)