Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 8.145698723s FAIL (2025-09-13 15:12:43 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 8.145721737s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [...................................X...X....................] duration: 17.831014417s FAIL (2025-09-13 15:12:31 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 3) ======LOGS====== 6OXJYXIXQSQK3DMHSGQ56ANOAOEGXEHD4EWCOZDXTTIL37SAA, seg_pos={0 0}, upload=148985]: Uploading piece... Sep 13 14:55:15.907845810 (169h26m51.94741893s) [noise=true , address=145.79.227.52:4475 , piece_id=3T3MRZJ2T74J7CB5EJ4N2CIIERKWTQ5KWIOLDNV2LXHYLO5IKHTA, seg_pos={0 0}, upload=148985]: Uploading piece... Sep 13 14:55:15.908062555 (169h26m51.947635675s) [noise=true , address=145.79.227.5:4250 , piece_id=EPZOZXRFVPSAPD6UKPOCCBOX7VJNB4DQRKV6YTRWMPOLQQ5IH6KQ, seg_pos={0 0}, upload=148985]: Uploading piece... Sep 13 14:55:15.908276674 (169h26m51.947849794s) [noise=true , address=145.79.227.30:4780 , piece_id=7EMNYDLOMVLIDGE5ZK7WFQCNGBFLZA5AF5EHSLHYE22XJ2MP56VQ, seg_pos={0 0}, upload=148985]: Uploading piece... Sep 13 14:55:15.908522154 (169h26m51.948095274s) [noise=true , address=77.90.31.17:4793 , piece_id=PWVTHYWQY6GFO5FCIBANYNSX72VNUGT4UXKYI5CEIIHNZGNB63OA, seg_pos={0 0}, upload=148985]: Uploading piece... Sep 13 14:55:17.522666509 (169h26m53.562239629s) [upload=148985]: Next returned nil segment Sep 13 14:55:17.522686247 (169h26m53.562259367s) [upload=148985]: Waiting for error group managing segments... Sep 13 14:55:18.048737965 (169h26m54.088311085s) [noise=true , address=77.90.31.17:4793 , piece_id=PWVTHYWQY6GFO5FCIBANYNSX72VNUGT4UXKYI5CEIIHNZGNB63OA, seg_pos={0 0}, upload=148985]: Done uploading piece. err: <nil> Sep 13 14:55:18.323717226 (169h26m54.363290346s) [noise=true , address=77.90.31.11:4559 , piece_id=OYMOQVJRZ6SMMF7O2DXXDEOVRN27B3FT5EIVMFFBHC47UPZUUMIQ, seg_pos={0 0}, upload=148985]: Done uploading piece. err: <nil> Sep 13 14:55:18.769215263 (169h26m54.808788383s) [noise=true , address=45.91.116.14:4221 , piece_id=Q7ZPEITLUERUPOI247KAARIL3GSAPW6JNEUF7FSHQWJHL5GTGIZQ, seg_pos={0 0}, upload=148985]: Done uploading piece. err: <nil> Sep 13 14:56:14.598217910 (169h27m50.63779103s) [noise=true , address=145.79.227.28:4730 , piece_id=5WCQINYSLLWZR3Z3QWWUKRKDPRWYQ6VQPIZ4YDAD47JHSKDSCAGA, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:1SXQt5FugmWj1YdhsvaaR7bDoxTBkqHKZpyXSmQqGdX5ZRhKxV, address:145.79.227.28:4730): protocol: read tcp 188.166.225.170:42820->145.79.227.28:4730: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 13 14:56:14.598245002 (169h27m50.637818122s) [noise=true , address=145.79.227.52:4475 , piece_id=3T3MRZJ2T74J7CB5EJ4N2CIIERKWTQ5KWIOLDNV2LXHYLO5IKHTA, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:12UYjE6PEB9gZKwTKcnx23ykF4jDMJzLQoKmiFiejWew5mirdCj, address:145.79.227.52:4475): protocol: rpc: dial tcp 145.79.227.52:4475: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 13 14:56:14.598273767 (169h27m50.637846887s) [noise=true , address=145.79.227.50:4386 , piece_id=74QVIDRRHFRIA5Z6ZAQ4EWQ2PZ5HZI5JXNJ5WR2DVFJQ4WGE2UPQ, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:19aPjJnya2Q3stEhiedDSth3NHV8phLRj5yY79hZv81UgDitMv, address:145.79.227.50:4386): protocol: write tcp 188.166.225.170:42008->145.79.227.50:4386: use of closed network connection; write tcp 188.166.225.170:42008->145.79.227.50:4386: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42008->145.79.227.50:4386: use of closed network connection Sep 13 14:56:14.598293245 (169h27m50.637866365s) [noise=true , address=145.79.227.5:4250 , piece_id=EPZOZXRFVPSAPD6UKPOCCBOX7VJNB4DQRKV6YTRWMPOLQQ5IH6KQ, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:1uoGjfqBLfoNrg2X1aYKYLKQjgFUnzLjHetCFc9sPjJa2qqKYM, address:145.79.227.5:4250): protocol: write tcp 188.166.225.170:58078->145.79.227.5:4250: use of closed network connection; write tcp 188.166.225.170:58078->145.79.227.5:4250: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58078->145.79.227.5:4250: use of closed network connection Sep 13 14:56:14.598308794 (169h27m50.637881914s) [noise=true , address=145.79.227.33:4894 , piece_id=5KD2BL5J7DT7JDLPDS2QUUMM26AFWSXJ2BDTJSUGZ253ZV7XOVKQ, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:12KWNyruYhmSJVvgUF7fBpGf4H7eYmtAeEXToYjd2dcEPcj1E44, address:145.79.227.33:4894): protocol: write tcp 188.166.225.170:53614->145.79.227.33:4894: use of closed network connection; write tcp 188.166.225.170:53614->145.79.227.33:4894: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53614->145.79.227.33:4894: use of closed network connection Sep 13 14:56:14.598327330 (169h27m50.63790045s) [noise=true , address=145.79.227.7:4308 , piece_id=FYL7PQOKW74EJ2QLKMN6EYJKIT7A6B2VQESVP2VFVDVJHPHXFGAQ, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:1PgA5SZkBvK673LsV7p2sXpbi1r79esNNZspPZTzaitqfbjtyY, address:145.79.227.7:4308): protocol: write tcp 188.166.225.170:37146->145.79.227.7:4308: use of closed network connection; write tcp 188.166.225.170:37146->145.79.227.7:4308: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37146->145.79.227.7:4308: use of closed network connection Sep 13 14:56:14.598358169 (169h27m50.637931289s) [noise=true , address=145.79.227.36:4544 , piece_id=WZRHCCBJDHID4R4UOB76OABJSH2QVC6WZ6OZWIUUMKWWBDVWEG4A, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:12RSxwxw1WXkNcu7b3dEgJhyNkFGYUfKF4QVKmK323L5g3EKFnG, address:145.79.227.36:4544): protocol: rpc: dial tcp 145.79.227.36:4544: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 13 14:56:14.598387545 (169h27m50.637960665s) [noise=true , address=145.79.227.2:4137 , piece_id=NH4RZXYD3LKQZLLP5SSA7HXC7FBBOQV6QLBE2B3R5HH5F23Q7E6A, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:12tkbodtFrNCuJbvZk8YnpSdJfVsKasNuKhg2wGD8rDkNPrHZ21, address:145.79.227.2:4137): protocol: rpc: dial tcp 145.79.227.2:4137: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 13 14:56:14.598421721 (169h27m50.637994841s) [noise=true , address=145.79.227.30:4780 , piece_id=7EMNYDLOMVLIDGE5ZK7WFQCNGBFLZA5AF5EHSLHYE22XJ2MP56VQ, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:125WCbZ6a2NaewiLAHhHuPo91sQEfC9CZzEBV6rwF1JY7z13AEY, address:145.79.227.30:4780): protocol: read tcp 188.166.225.170:43016->145.79.227.30:4780: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 13 14:56:14.598427161 (169h27m50.638000281s) [noise=true , address=145.79.227.19:4090 , piece_id=YZ2H4UUM3Z5ZMCC6LQDAPLHCT7OQQAMRSDBQ3CH6I6UT2HAN4TXQ, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:12qSkJk9HUyQon3UP1AA8mRcqJbNdkKJh6cEUZoHLGNniqDvNqB, address:145.79.227.19:4090): protocol: write tcp 188.166.225.170:37110->145.79.227.19:4090: use of closed network connection; write tcp 188.166.225.170:37110->145.79.227.19:4090: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37110->145.79.227.19:4090: use of closed network connection Sep 13 14:56:14.598456818 (169h27m50.638029938s) [noise=true , address=145.79.227.10:4425 , piece_id=CRES4FQPX4WMTDTUBCPRZ2R4OHTI7MMHS6XSCZ7UXKDLSSLEVQ3Q, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:13sAcGFjAXuesAkUn7NuLaaV8TAah3Mr1kqd33wwh9MTELqmpD, address:145.79.227.10:4425): protocol: read tcp 188.166.225.170:58348->145.79.227.10:4425: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 13 14:56:14.598463220 (169h27m50.63803635s) [noise=true , address=145.79.227.35:4501 , piece_id=UKSF2KK675OI35QE6QZTH7QAJQ4SD7E42ZPMY2JIANOHGEFF44UQ, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:1hmsSAiSp1CBFqKhVE2S3sFfqYYkGVAiV1GfrT4Rd7hBaFNPpT, address:145.79.227.35:4501): protocol: rpc: dial tcp 145.79.227.35:4501: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 13 14:56:14.598484230 (169h27m50.63805735s) [noise=true , address=145.79.227.14:4867 , piece_id=7BW6OXJYXIXQSQK3DMHSGQ56ANOAOEGXEHD4EWCOZDXTTIL37SAA, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:12Sz7PyMo81CmAAkiCwpZ3M3JaM9HMHJLwAAM5M5kv6nnKktpKE, address:145.79.227.14:4867): protocol: read tcp 188.166.225.170:60252->145.79.227.14:4867: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 13 14:56:14.598492155 (169h27m50.638065275s) [noise=true , address=145.79.227.18:4018 , piece_id=72FTA6TNIQAVS2NDA5XOR4J2ZEEBWZ7LLGT4EAQ2EN2YB3VXG5JA, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:1bFRHbXjTaxoEKc5gbQWYt6jwUg8jknScBZpc8PdKNSZhYqUqP, address:145.79.227.18:4018): protocol: read tcp 188.166.225.170:60682->145.79.227.18:4018: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 13 14:56:14.598496193 (169h27m50.638069313s) [noise=true , address=145.79.227.47:4286 , piece_id=D2PSYV2IN5MFZGGF2U47EPRTCNDIMHB74GR3UFQ64AJN37FE7BVA, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:12v2vJnE92VrJXQrbxnAxANHCVASuXiBKMUZECMVmEWEjsZRtsQ, address:145.79.227.47:4286): protocol: read tcp 188.166.225.170:33092->145.79.227.47:4286: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 13 14:56:14.598521301 (169h27m50.638094421s) [seg_pos={0 0}, upload=148985]: Done waiting for segment. successful: 3 optimal: 14 errs: <nil> Sep 13 14:56:14.862062924 (169h27m50.901636044s) [upload=148985]: Done uploading segments... Sep 13 14:56:14.862090337 (169h27m50.901663457s) [upload=148985]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 3) Duration: 17.831040808s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [.......X.X...X..X.............X..X...X.X....X.X.........XXX.] duration: 6.937873662s FAIL (2025-09-13 15:12:10 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.937891786s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 7.741593737s FAIL (2025-09-13 15:12:08 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 7.741617072s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [........X.X...X..X.............X..X...X.X....X.X.........XXX] duration: 30.000968023s FAIL (2025-09-13 15:12:03 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.0010234s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [....................................X...X...................] duration: 12.801459261s FAIL (2025-09-13 15:11:52 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 3) ======LOGS====== 6OXJYXIXQSQK3DMHSGQ56ANOAOEGXEHD4EWCOZDXTTIL37SAA, seg_pos={0 0}, upload=148985]: Uploading piece... Sep 13 14:55:15.907845810 (169h26m51.94741893s) [noise=true , address=145.79.227.52:4475 , piece_id=3T3MRZJ2T74J7CB5EJ4N2CIIERKWTQ5KWIOLDNV2LXHYLO5IKHTA, seg_pos={0 0}, upload=148985]: Uploading piece... Sep 13 14:55:15.908062555 (169h26m51.947635675s) [noise=true , address=145.79.227.5:4250 , piece_id=EPZOZXRFVPSAPD6UKPOCCBOX7VJNB4DQRKV6YTRWMPOLQQ5IH6KQ, seg_pos={0 0}, upload=148985]: Uploading piece... Sep 13 14:55:15.908276674 (169h26m51.947849794s) [noise=true , address=145.79.227.30:4780 , piece_id=7EMNYDLOMVLIDGE5ZK7WFQCNGBFLZA5AF5EHSLHYE22XJ2MP56VQ, seg_pos={0 0}, upload=148985]: Uploading piece... Sep 13 14:55:15.908522154 (169h26m51.948095274s) [noise=true , address=77.90.31.17:4793 , piece_id=PWVTHYWQY6GFO5FCIBANYNSX72VNUGT4UXKYI5CEIIHNZGNB63OA, seg_pos={0 0}, upload=148985]: Uploading piece... Sep 13 14:55:17.522666509 (169h26m53.562239629s) [upload=148985]: Next returned nil segment Sep 13 14:55:17.522686247 (169h26m53.562259367s) [upload=148985]: Waiting for error group managing segments... Sep 13 14:55:18.048737965 (169h26m54.088311085s) [noise=true , address=77.90.31.17:4793 , piece_id=PWVTHYWQY6GFO5FCIBANYNSX72VNUGT4UXKYI5CEIIHNZGNB63OA, seg_pos={0 0}, upload=148985]: Done uploading piece. err: <nil> Sep 13 14:55:18.323717226 (169h26m54.363290346s) [noise=true , address=77.90.31.11:4559 , piece_id=OYMOQVJRZ6SMMF7O2DXXDEOVRN27B3FT5EIVMFFBHC47UPZUUMIQ, seg_pos={0 0}, upload=148985]: Done uploading piece. err: <nil> Sep 13 14:55:18.769215263 (169h26m54.808788383s) [noise=true , address=45.91.116.14:4221 , piece_id=Q7ZPEITLUERUPOI247KAARIL3GSAPW6JNEUF7FSHQWJHL5GTGIZQ, seg_pos={0 0}, upload=148985]: Done uploading piece. err: <nil> Sep 13 14:56:14.598217910 (169h27m50.63779103s) [noise=true , address=145.79.227.28:4730 , piece_id=5WCQINYSLLWZR3Z3QWWUKRKDPRWYQ6VQPIZ4YDAD47JHSKDSCAGA, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:1SXQt5FugmWj1YdhsvaaR7bDoxTBkqHKZpyXSmQqGdX5ZRhKxV, address:145.79.227.28:4730): protocol: read tcp 188.166.225.170:42820->145.79.227.28:4730: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 13 14:56:14.598245002 (169h27m50.637818122s) [noise=true , address=145.79.227.52:4475 , piece_id=3T3MRZJ2T74J7CB5EJ4N2CIIERKWTQ5KWIOLDNV2LXHYLO5IKHTA, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:12UYjE6PEB9gZKwTKcnx23ykF4jDMJzLQoKmiFiejWew5mirdCj, address:145.79.227.52:4475): protocol: rpc: dial tcp 145.79.227.52:4475: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 13 14:56:14.598273767 (169h27m50.637846887s) [noise=true , address=145.79.227.50:4386 , piece_id=74QVIDRRHFRIA5Z6ZAQ4EWQ2PZ5HZI5JXNJ5WR2DVFJQ4WGE2UPQ, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:19aPjJnya2Q3stEhiedDSth3NHV8phLRj5yY79hZv81UgDitMv, address:145.79.227.50:4386): protocol: write tcp 188.166.225.170:42008->145.79.227.50:4386: use of closed network connection; write tcp 188.166.225.170:42008->145.79.227.50:4386: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42008->145.79.227.50:4386: use of closed network connection Sep 13 14:56:14.598293245 (169h27m50.637866365s) [noise=true , address=145.79.227.5:4250 , piece_id=EPZOZXRFVPSAPD6UKPOCCBOX7VJNB4DQRKV6YTRWMPOLQQ5IH6KQ, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:1uoGjfqBLfoNrg2X1aYKYLKQjgFUnzLjHetCFc9sPjJa2qqKYM, address:145.79.227.5:4250): protocol: write tcp 188.166.225.170:58078->145.79.227.5:4250: use of closed network connection; write tcp 188.166.225.170:58078->145.79.227.5:4250: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58078->145.79.227.5:4250: use of closed network connection Sep 13 14:56:14.598308794 (169h27m50.637881914s) [noise=true , address=145.79.227.33:4894 , piece_id=5KD2BL5J7DT7JDLPDS2QUUMM26AFWSXJ2BDTJSUGZ253ZV7XOVKQ, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:12KWNyruYhmSJVvgUF7fBpGf4H7eYmtAeEXToYjd2dcEPcj1E44, address:145.79.227.33:4894): protocol: write tcp 188.166.225.170:53614->145.79.227.33:4894: use of closed network connection; write tcp 188.166.225.170:53614->145.79.227.33:4894: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53614->145.79.227.33:4894: use of closed network connection Sep 13 14:56:14.598327330 (169h27m50.63790045s) [noise=true , address=145.79.227.7:4308 , piece_id=FYL7PQOKW74EJ2QLKMN6EYJKIT7A6B2VQESVP2VFVDVJHPHXFGAQ, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:1PgA5SZkBvK673LsV7p2sXpbi1r79esNNZspPZTzaitqfbjtyY, address:145.79.227.7:4308): protocol: write tcp 188.166.225.170:37146->145.79.227.7:4308: use of closed network connection; write tcp 188.166.225.170:37146->145.79.227.7:4308: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37146->145.79.227.7:4308: use of closed network connection Sep 13 14:56:14.598358169 (169h27m50.637931289s) [noise=true , address=145.79.227.36:4544 , piece_id=WZRHCCBJDHID4R4UOB76OABJSH2QVC6WZ6OZWIUUMKWWBDVWEG4A, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:12RSxwxw1WXkNcu7b3dEgJhyNkFGYUfKF4QVKmK323L5g3EKFnG, address:145.79.227.36:4544): protocol: rpc: dial tcp 145.79.227.36:4544: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 13 14:56:14.598387545 (169h27m50.637960665s) [noise=true , address=145.79.227.2:4137 , piece_id=NH4RZXYD3LKQZLLP5SSA7HXC7FBBOQV6QLBE2B3R5HH5F23Q7E6A, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:12tkbodtFrNCuJbvZk8YnpSdJfVsKasNuKhg2wGD8rDkNPrHZ21, address:145.79.227.2:4137): protocol: rpc: dial tcp 145.79.227.2:4137: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 13 14:56:14.598421721 (169h27m50.637994841s) [noise=true , address=145.79.227.30:4780 , piece_id=7EMNYDLOMVLIDGE5ZK7WFQCNGBFLZA5AF5EHSLHYE22XJ2MP56VQ, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:125WCbZ6a2NaewiLAHhHuPo91sQEfC9CZzEBV6rwF1JY7z13AEY, address:145.79.227.30:4780): protocol: read tcp 188.166.225.170:43016->145.79.227.30:4780: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 13 14:56:14.598427161 (169h27m50.638000281s) [noise=true , address=145.79.227.19:4090 , piece_id=YZ2H4UUM3Z5ZMCC6LQDAPLHCT7OQQAMRSDBQ3CH6I6UT2HAN4TXQ, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:12qSkJk9HUyQon3UP1AA8mRcqJbNdkKJh6cEUZoHLGNniqDvNqB, address:145.79.227.19:4090): protocol: write tcp 188.166.225.170:37110->145.79.227.19:4090: use of closed network connection; write tcp 188.166.225.170:37110->145.79.227.19:4090: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37110->145.79.227.19:4090: use of closed network connection Sep 13 14:56:14.598456818 (169h27m50.638029938s) [noise=true , address=145.79.227.10:4425 , piece_id=CRES4FQPX4WMTDTUBCPRZ2R4OHTI7MMHS6XSCZ7UXKDLSSLEVQ3Q, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:13sAcGFjAXuesAkUn7NuLaaV8TAah3Mr1kqd33wwh9MTELqmpD, address:145.79.227.10:4425): protocol: read tcp 188.166.225.170:58348->145.79.227.10:4425: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 13 14:56:14.598463220 (169h27m50.63803635s) [noise=true , address=145.79.227.35:4501 , piece_id=UKSF2KK675OI35QE6QZTH7QAJQ4SD7E42ZPMY2JIANOHGEFF44UQ, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:1hmsSAiSp1CBFqKhVE2S3sFfqYYkGVAiV1GfrT4Rd7hBaFNPpT, address:145.79.227.35:4501): protocol: rpc: dial tcp 145.79.227.35:4501: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 13 14:56:14.598484230 (169h27m50.63805735s) [noise=true , address=145.79.227.14:4867 , piece_id=7BW6OXJYXIXQSQK3DMHSGQ56ANOAOEGXEHD4EWCOZDXTTIL37SAA, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:12Sz7PyMo81CmAAkiCwpZ3M3JaM9HMHJLwAAM5M5kv6nnKktpKE, address:145.79.227.14:4867): protocol: read tcp 188.166.225.170:60252->145.79.227.14:4867: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 13 14:56:14.598492155 (169h27m50.638065275s) [noise=true , address=145.79.227.18:4018 , piece_id=72FTA6TNIQAVS2NDA5XOR4J2ZEEBWZ7LLGT4EAQ2EN2YB3VXG5JA, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:1bFRHbXjTaxoEKc5gbQWYt6jwUg8jknScBZpc8PdKNSZhYqUqP, address:145.79.227.18:4018): protocol: read tcp 188.166.225.170:60682->145.79.227.18:4018: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 13 14:56:14.598496193 (169h27m50.638069313s) [noise=true , address=145.79.227.47:4286 , piece_id=D2PSYV2IN5MFZGGF2U47EPRTCNDIMHB74GR3UFQ64AJN37FE7BVA, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:12v2vJnE92VrJXQrbxnAxANHCVASuXiBKMUZECMVmEWEjsZRtsQ, address:145.79.227.47:4286): protocol: read tcp 188.166.225.170:33092->145.79.227.47:4286: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 13 14:56:14.598521301 (169h27m50.638094421s) [seg_pos={0 0}, upload=148985]: Done waiting for segment. successful: 3 optimal: 14 errs: <nil> Sep 13 14:56:14.862062924 (169h27m50.901636044s) [upload=148985]: Done uploading segments... Sep 13 14:56:14.862090337 (169h27m50.901663457s) [upload=148985]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 3) Duration: 12.801485581s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 7.927760791s FAIL (2025-09-13 15:11:10 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 7.92777601s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [.........X.X...X..X.............X..X...X.X....X.X.........XX] duration: 30.001325761s FAIL (2025-09-13 15:10:50 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.001381327s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....................................X...X..................] duration: 10.444107834s FAIL (2025-09-13 15:10:39 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 3) ======LOGS====== 6OXJYXIXQSQK3DMHSGQ56ANOAOEGXEHD4EWCOZDXTTIL37SAA, seg_pos={0 0}, upload=148985]: Uploading piece... Sep 13 14:55:15.907845810 (169h26m51.94741893s) [noise=true , address=145.79.227.52:4475 , piece_id=3T3MRZJ2T74J7CB5EJ4N2CIIERKWTQ5KWIOLDNV2LXHYLO5IKHTA, seg_pos={0 0}, upload=148985]: Uploading piece... Sep 13 14:55:15.908062555 (169h26m51.947635675s) [noise=true , address=145.79.227.5:4250 , piece_id=EPZOZXRFVPSAPD6UKPOCCBOX7VJNB4DQRKV6YTRWMPOLQQ5IH6KQ, seg_pos={0 0}, upload=148985]: Uploading piece... Sep 13 14:55:15.908276674 (169h26m51.947849794s) [noise=true , address=145.79.227.30:4780 , piece_id=7EMNYDLOMVLIDGE5ZK7WFQCNGBFLZA5AF5EHSLHYE22XJ2MP56VQ, seg_pos={0 0}, upload=148985]: Uploading piece... Sep 13 14:55:15.908522154 (169h26m51.948095274s) [noise=true , address=77.90.31.17:4793 , piece_id=PWVTHYWQY6GFO5FCIBANYNSX72VNUGT4UXKYI5CEIIHNZGNB63OA, seg_pos={0 0}, upload=148985]: Uploading piece... Sep 13 14:55:17.522666509 (169h26m53.562239629s) [upload=148985]: Next returned nil segment Sep 13 14:55:17.522686247 (169h26m53.562259367s) [upload=148985]: Waiting for error group managing segments... Sep 13 14:55:18.048737965 (169h26m54.088311085s) [noise=true , address=77.90.31.17:4793 , piece_id=PWVTHYWQY6GFO5FCIBANYNSX72VNUGT4UXKYI5CEIIHNZGNB63OA, seg_pos={0 0}, upload=148985]: Done uploading piece. err: <nil> Sep 13 14:55:18.323717226 (169h26m54.363290346s) [noise=true , address=77.90.31.11:4559 , piece_id=OYMOQVJRZ6SMMF7O2DXXDEOVRN27B3FT5EIVMFFBHC47UPZUUMIQ, seg_pos={0 0}, upload=148985]: Done uploading piece. err: <nil> Sep 13 14:55:18.769215263 (169h26m54.808788383s) [noise=true , address=45.91.116.14:4221 , piece_id=Q7ZPEITLUERUPOI247KAARIL3GSAPW6JNEUF7FSHQWJHL5GTGIZQ, seg_pos={0 0}, upload=148985]: Done uploading piece. err: <nil> Sep 13 14:56:14.598217910 (169h27m50.63779103s) [noise=true , address=145.79.227.28:4730 , piece_id=5WCQINYSLLWZR3Z3QWWUKRKDPRWYQ6VQPIZ4YDAD47JHSKDSCAGA, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:1SXQt5FugmWj1YdhsvaaR7bDoxTBkqHKZpyXSmQqGdX5ZRhKxV, address:145.79.227.28:4730): protocol: read tcp 188.166.225.170:42820->145.79.227.28:4730: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 13 14:56:14.598245002 (169h27m50.637818122s) [noise=true , address=145.79.227.52:4475 , piece_id=3T3MRZJ2T74J7CB5EJ4N2CIIERKWTQ5KWIOLDNV2LXHYLO5IKHTA, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:12UYjE6PEB9gZKwTKcnx23ykF4jDMJzLQoKmiFiejWew5mirdCj, address:145.79.227.52:4475): protocol: rpc: dial tcp 145.79.227.52:4475: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 13 14:56:14.598273767 (169h27m50.637846887s) [noise=true , address=145.79.227.50:4386 , piece_id=74QVIDRRHFRIA5Z6ZAQ4EWQ2PZ5HZI5JXNJ5WR2DVFJQ4WGE2UPQ, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:19aPjJnya2Q3stEhiedDSth3NHV8phLRj5yY79hZv81UgDitMv, address:145.79.227.50:4386): protocol: write tcp 188.166.225.170:42008->145.79.227.50:4386: use of closed network connection; write tcp 188.166.225.170:42008->145.79.227.50:4386: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42008->145.79.227.50:4386: use of closed network connection Sep 13 14:56:14.598293245 (169h27m50.637866365s) [noise=true , address=145.79.227.5:4250 , piece_id=EPZOZXRFVPSAPD6UKPOCCBOX7VJNB4DQRKV6YTRWMPOLQQ5IH6KQ, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:1uoGjfqBLfoNrg2X1aYKYLKQjgFUnzLjHetCFc9sPjJa2qqKYM, address:145.79.227.5:4250): protocol: write tcp 188.166.225.170:58078->145.79.227.5:4250: use of closed network connection; write tcp 188.166.225.170:58078->145.79.227.5:4250: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58078->145.79.227.5:4250: use of closed network connection Sep 13 14:56:14.598308794 (169h27m50.637881914s) [noise=true , address=145.79.227.33:4894 , piece_id=5KD2BL5J7DT7JDLPDS2QUUMM26AFWSXJ2BDTJSUGZ253ZV7XOVKQ, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:12KWNyruYhmSJVvgUF7fBpGf4H7eYmtAeEXToYjd2dcEPcj1E44, address:145.79.227.33:4894): protocol: write tcp 188.166.225.170:53614->145.79.227.33:4894: use of closed network connection; write tcp 188.166.225.170:53614->145.79.227.33:4894: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53614->145.79.227.33:4894: use of closed network connection Sep 13 14:56:14.598327330 (169h27m50.63790045s) [noise=true , address=145.79.227.7:4308 , piece_id=FYL7PQOKW74EJ2QLKMN6EYJKIT7A6B2VQESVP2VFVDVJHPHXFGAQ, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:1PgA5SZkBvK673LsV7p2sXpbi1r79esNNZspPZTzaitqfbjtyY, address:145.79.227.7:4308): protocol: write tcp 188.166.225.170:37146->145.79.227.7:4308: use of closed network connection; write tcp 188.166.225.170:37146->145.79.227.7:4308: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37146->145.79.227.7:4308: use of closed network connection Sep 13 14:56:14.598358169 (169h27m50.637931289s) [noise=true , address=145.79.227.36:4544 , piece_id=WZRHCCBJDHID4R4UOB76OABJSH2QVC6WZ6OZWIUUMKWWBDVWEG4A, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:12RSxwxw1WXkNcu7b3dEgJhyNkFGYUfKF4QVKmK323L5g3EKFnG, address:145.79.227.36:4544): protocol: rpc: dial tcp 145.79.227.36:4544: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 13 14:56:14.598387545 (169h27m50.637960665s) [noise=true , address=145.79.227.2:4137 , piece_id=NH4RZXYD3LKQZLLP5SSA7HXC7FBBOQV6QLBE2B3R5HH5F23Q7E6A, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:12tkbodtFrNCuJbvZk8YnpSdJfVsKasNuKhg2wGD8rDkNPrHZ21, address:145.79.227.2:4137): protocol: rpc: dial tcp 145.79.227.2:4137: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 13 14:56:14.598421721 (169h27m50.637994841s) [noise=true , address=145.79.227.30:4780 , piece_id=7EMNYDLOMVLIDGE5ZK7WFQCNGBFLZA5AF5EHSLHYE22XJ2MP56VQ, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:125WCbZ6a2NaewiLAHhHuPo91sQEfC9CZzEBV6rwF1JY7z13AEY, address:145.79.227.30:4780): protocol: read tcp 188.166.225.170:43016->145.79.227.30:4780: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 13 14:56:14.598427161 (169h27m50.638000281s) [noise=true , address=145.79.227.19:4090 , piece_id=YZ2H4UUM3Z5ZMCC6LQDAPLHCT7OQQAMRSDBQ3CH6I6UT2HAN4TXQ, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:12qSkJk9HUyQon3UP1AA8mRcqJbNdkKJh6cEUZoHLGNniqDvNqB, address:145.79.227.19:4090): protocol: write tcp 188.166.225.170:37110->145.79.227.19:4090: use of closed network connection; write tcp 188.166.225.170:37110->145.79.227.19:4090: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37110->145.79.227.19:4090: use of closed network connection Sep 13 14:56:14.598456818 (169h27m50.638029938s) [noise=true , address=145.79.227.10:4425 , piece_id=CRES4FQPX4WMTDTUBCPRZ2R4OHTI7MMHS6XSCZ7UXKDLSSLEVQ3Q, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:13sAcGFjAXuesAkUn7NuLaaV8TAah3Mr1kqd33wwh9MTELqmpD, address:145.79.227.10:4425): protocol: read tcp 188.166.225.170:58348->145.79.227.10:4425: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 13 14:56:14.598463220 (169h27m50.63803635s) [noise=true , address=145.79.227.35:4501 , piece_id=UKSF2KK675OI35QE6QZTH7QAJQ4SD7E42ZPMY2JIANOHGEFF44UQ, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:1hmsSAiSp1CBFqKhVE2S3sFfqYYkGVAiV1GfrT4Rd7hBaFNPpT, address:145.79.227.35:4501): protocol: rpc: dial tcp 145.79.227.35:4501: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 13 14:56:14.598484230 (169h27m50.63805735s) [noise=true , address=145.79.227.14:4867 , piece_id=7BW6OXJYXIXQSQK3DMHSGQ56ANOAOEGXEHD4EWCOZDXTTIL37SAA, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:12Sz7PyMo81CmAAkiCwpZ3M3JaM9HMHJLwAAM5M5kv6nnKktpKE, address:145.79.227.14:4867): protocol: read tcp 188.166.225.170:60252->145.79.227.14:4867: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 13 14:56:14.598492155 (169h27m50.638065275s) [noise=true , address=145.79.227.18:4018 , piece_id=72FTA6TNIQAVS2NDA5XOR4J2ZEEBWZ7LLGT4EAQ2EN2YB3VXG5JA, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:1bFRHbXjTaxoEKc5gbQWYt6jwUg8jknScBZpc8PdKNSZhYqUqP, address:145.79.227.18:4018): protocol: read tcp 188.166.225.170:60682->145.79.227.18:4018: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 13 14:56:14.598496193 (169h27m50.638069313s) [noise=true , address=145.79.227.47:4286 , piece_id=D2PSYV2IN5MFZGGF2U47EPRTCNDIMHB74GR3UFQ64AJN37FE7BVA, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:12v2vJnE92VrJXQrbxnAxANHCVASuXiBKMUZECMVmEWEjsZRtsQ, address:145.79.227.47:4286): protocol: read tcp 188.166.225.170:33092->145.79.227.47:4286: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 13 14:56:14.598521301 (169h27m50.638094421s) [seg_pos={0 0}, upload=148985]: Done waiting for segment. successful: 3 optimal: 14 errs: <nil> Sep 13 14:56:14.862062924 (169h27m50.901636044s) [upload=148985]: Done uploading segments... Sep 13 14:56:14.862090337 (169h27m50.901663457s) [upload=148985]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 3) Duration: 10.444141428s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 7.754432515s FAIL (2025-09-13 15:10:05 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 7.754470017s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [..........X.X...X..X.............X..X...X.X....X.X.........X] duration: 30.001280309s FAIL (2025-09-13 15:09:57 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.001297473s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [......................................X...X.................] duration: 13.572005717s FAIL (2025-09-13 15:09:33 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 3) ======LOGS====== 6OXJYXIXQSQK3DMHSGQ56ANOAOEGXEHD4EWCOZDXTTIL37SAA, seg_pos={0 0}, upload=148985]: Uploading piece... Sep 13 14:55:15.907845810 (169h26m51.94741893s) [noise=true , address=145.79.227.52:4475 , piece_id=3T3MRZJ2T74J7CB5EJ4N2CIIERKWTQ5KWIOLDNV2LXHYLO5IKHTA, seg_pos={0 0}, upload=148985]: Uploading piece... Sep 13 14:55:15.908062555 (169h26m51.947635675s) [noise=true , address=145.79.227.5:4250 , piece_id=EPZOZXRFVPSAPD6UKPOCCBOX7VJNB4DQRKV6YTRWMPOLQQ5IH6KQ, seg_pos={0 0}, upload=148985]: Uploading piece... Sep 13 14:55:15.908276674 (169h26m51.947849794s) [noise=true , address=145.79.227.30:4780 , piece_id=7EMNYDLOMVLIDGE5ZK7WFQCNGBFLZA5AF5EHSLHYE22XJ2MP56VQ, seg_pos={0 0}, upload=148985]: Uploading piece... Sep 13 14:55:15.908522154 (169h26m51.948095274s) [noise=true , address=77.90.31.17:4793 , piece_id=PWVTHYWQY6GFO5FCIBANYNSX72VNUGT4UXKYI5CEIIHNZGNB63OA, seg_pos={0 0}, upload=148985]: Uploading piece... Sep 13 14:55:17.522666509 (169h26m53.562239629s) [upload=148985]: Next returned nil segment Sep 13 14:55:17.522686247 (169h26m53.562259367s) [upload=148985]: Waiting for error group managing segments... Sep 13 14:55:18.048737965 (169h26m54.088311085s) [noise=true , address=77.90.31.17:4793 , piece_id=PWVTHYWQY6GFO5FCIBANYNSX72VNUGT4UXKYI5CEIIHNZGNB63OA, seg_pos={0 0}, upload=148985]: Done uploading piece. err: <nil> Sep 13 14:55:18.323717226 (169h26m54.363290346s) [noise=true , address=77.90.31.11:4559 , piece_id=OYMOQVJRZ6SMMF7O2DXXDEOVRN27B3FT5EIVMFFBHC47UPZUUMIQ, seg_pos={0 0}, upload=148985]: Done uploading piece. err: <nil> Sep 13 14:55:18.769215263 (169h26m54.808788383s) [noise=true , address=45.91.116.14:4221 , piece_id=Q7ZPEITLUERUPOI247KAARIL3GSAPW6JNEUF7FSHQWJHL5GTGIZQ, seg_pos={0 0}, upload=148985]: Done uploading piece. err: <nil> Sep 13 14:56:14.598217910 (169h27m50.63779103s) [noise=true , address=145.79.227.28:4730 , piece_id=5WCQINYSLLWZR3Z3QWWUKRKDPRWYQ6VQPIZ4YDAD47JHSKDSCAGA, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:1SXQt5FugmWj1YdhsvaaR7bDoxTBkqHKZpyXSmQqGdX5ZRhKxV, address:145.79.227.28:4730): protocol: read tcp 188.166.225.170:42820->145.79.227.28:4730: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 13 14:56:14.598245002 (169h27m50.637818122s) [noise=true , address=145.79.227.52:4475 , piece_id=3T3MRZJ2T74J7CB5EJ4N2CIIERKWTQ5KWIOLDNV2LXHYLO5IKHTA, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:12UYjE6PEB9gZKwTKcnx23ykF4jDMJzLQoKmiFiejWew5mirdCj, address:145.79.227.52:4475): protocol: rpc: dial tcp 145.79.227.52:4475: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 13 14:56:14.598273767 (169h27m50.637846887s) [noise=true , address=145.79.227.50:4386 , piece_id=74QVIDRRHFRIA5Z6ZAQ4EWQ2PZ5HZI5JXNJ5WR2DVFJQ4WGE2UPQ, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:19aPjJnya2Q3stEhiedDSth3NHV8phLRj5yY79hZv81UgDitMv, address:145.79.227.50:4386): protocol: write tcp 188.166.225.170:42008->145.79.227.50:4386: use of closed network connection; write tcp 188.166.225.170:42008->145.79.227.50:4386: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42008->145.79.227.50:4386: use of closed network connection Sep 13 14:56:14.598293245 (169h27m50.637866365s) [noise=true , address=145.79.227.5:4250 , piece_id=EPZOZXRFVPSAPD6UKPOCCBOX7VJNB4DQRKV6YTRWMPOLQQ5IH6KQ, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:1uoGjfqBLfoNrg2X1aYKYLKQjgFUnzLjHetCFc9sPjJa2qqKYM, address:145.79.227.5:4250): protocol: write tcp 188.166.225.170:58078->145.79.227.5:4250: use of closed network connection; write tcp 188.166.225.170:58078->145.79.227.5:4250: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58078->145.79.227.5:4250: use of closed network connection Sep 13 14:56:14.598308794 (169h27m50.637881914s) [noise=true , address=145.79.227.33:4894 , piece_id=5KD2BL5J7DT7JDLPDS2QUUMM26AFWSXJ2BDTJSUGZ253ZV7XOVKQ, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:12KWNyruYhmSJVvgUF7fBpGf4H7eYmtAeEXToYjd2dcEPcj1E44, address:145.79.227.33:4894): protocol: write tcp 188.166.225.170:53614->145.79.227.33:4894: use of closed network connection; write tcp 188.166.225.170:53614->145.79.227.33:4894: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53614->145.79.227.33:4894: use of closed network connection Sep 13 14:56:14.598327330 (169h27m50.63790045s) [noise=true , address=145.79.227.7:4308 , piece_id=FYL7PQOKW74EJ2QLKMN6EYJKIT7A6B2VQESVP2VFVDVJHPHXFGAQ, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:1PgA5SZkBvK673LsV7p2sXpbi1r79esNNZspPZTzaitqfbjtyY, address:145.79.227.7:4308): protocol: write tcp 188.166.225.170:37146->145.79.227.7:4308: use of closed network connection; write tcp 188.166.225.170:37146->145.79.227.7:4308: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37146->145.79.227.7:4308: use of closed network connection Sep 13 14:56:14.598358169 (169h27m50.637931289s) [noise=true , address=145.79.227.36:4544 , piece_id=WZRHCCBJDHID4R4UOB76OABJSH2QVC6WZ6OZWIUUMKWWBDVWEG4A, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:12RSxwxw1WXkNcu7b3dEgJhyNkFGYUfKF4QVKmK323L5g3EKFnG, address:145.79.227.36:4544): protocol: rpc: dial tcp 145.79.227.36:4544: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 13 14:56:14.598387545 (169h27m50.637960665s) [noise=true , address=145.79.227.2:4137 , piece_id=NH4RZXYD3LKQZLLP5SSA7HXC7FBBOQV6QLBE2B3R5HH5F23Q7E6A, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:12tkbodtFrNCuJbvZk8YnpSdJfVsKasNuKhg2wGD8rDkNPrHZ21, address:145.79.227.2:4137): protocol: rpc: dial tcp 145.79.227.2:4137: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 13 14:56:14.598421721 (169h27m50.637994841s) [noise=true , address=145.79.227.30:4780 , piece_id=7EMNYDLOMVLIDGE5ZK7WFQCNGBFLZA5AF5EHSLHYE22XJ2MP56VQ, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:125WCbZ6a2NaewiLAHhHuPo91sQEfC9CZzEBV6rwF1JY7z13AEY, address:145.79.227.30:4780): protocol: read tcp 188.166.225.170:43016->145.79.227.30:4780: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 13 14:56:14.598427161 (169h27m50.638000281s) [noise=true , address=145.79.227.19:4090 , piece_id=YZ2H4UUM3Z5ZMCC6LQDAPLHCT7OQQAMRSDBQ3CH6I6UT2HAN4TXQ, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:12qSkJk9HUyQon3UP1AA8mRcqJbNdkKJh6cEUZoHLGNniqDvNqB, address:145.79.227.19:4090): protocol: write tcp 188.166.225.170:37110->145.79.227.19:4090: use of closed network connection; write tcp 188.166.225.170:37110->145.79.227.19:4090: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37110->145.79.227.19:4090: use of closed network connection Sep 13 14:56:14.598456818 (169h27m50.638029938s) [noise=true , address=145.79.227.10:4425 , piece_id=CRES4FQPX4WMTDTUBCPRZ2R4OHTI7MMHS6XSCZ7UXKDLSSLEVQ3Q, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:13sAcGFjAXuesAkUn7NuLaaV8TAah3Mr1kqd33wwh9MTELqmpD, address:145.79.227.10:4425): protocol: read tcp 188.166.225.170:58348->145.79.227.10:4425: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 13 14:56:14.598463220 (169h27m50.63803635s) [noise=true , address=145.79.227.35:4501 , piece_id=UKSF2KK675OI35QE6QZTH7QAJQ4SD7E42ZPMY2JIANOHGEFF44UQ, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:1hmsSAiSp1CBFqKhVE2S3sFfqYYkGVAiV1GfrT4Rd7hBaFNPpT, address:145.79.227.35:4501): protocol: rpc: dial tcp 145.79.227.35:4501: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 13 14:56:14.598484230 (169h27m50.63805735s) [noise=true , address=145.79.227.14:4867 , piece_id=7BW6OXJYXIXQSQK3DMHSGQ56ANOAOEGXEHD4EWCOZDXTTIL37SAA, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:12Sz7PyMo81CmAAkiCwpZ3M3JaM9HMHJLwAAM5M5kv6nnKktpKE, address:145.79.227.14:4867): protocol: read tcp 188.166.225.170:60252->145.79.227.14:4867: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 13 14:56:14.598492155 (169h27m50.638065275s) [noise=true , address=145.79.227.18:4018 , piece_id=72FTA6TNIQAVS2NDA5XOR4J2ZEEBWZ7LLGT4EAQ2EN2YB3VXG5JA, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:1bFRHbXjTaxoEKc5gbQWYt6jwUg8jknScBZpc8PdKNSZhYqUqP, address:145.79.227.18:4018): protocol: read tcp 188.166.225.170:60682->145.79.227.18:4018: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 13 14:56:14.598496193 (169h27m50.638069313s) [noise=true , address=145.79.227.47:4286 , piece_id=D2PSYV2IN5MFZGGF2U47EPRTCNDIMHB74GR3UFQ64AJN37FE7BVA, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:12v2vJnE92VrJXQrbxnAxANHCVASuXiBKMUZECMVmEWEjsZRtsQ, address:145.79.227.47:4286): protocol: read tcp 188.166.225.170:33092->145.79.227.47:4286: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 13 14:56:14.598521301 (169h27m50.638094421s) [seg_pos={0 0}, upload=148985]: Done waiting for segment. successful: 3 optimal: 14 errs: <nil> Sep 13 14:56:14.862062924 (169h27m50.901636044s) [upload=148985]: Done uploading segments... Sep 13 14:56:14.862090337 (169h27m50.901663457s) [upload=148985]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 3) Duration: 13.572031236s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [...........X.X...X..X.............X..X...X.X....X.X.........] duration: 7.34463516s FAIL (2025-09-13 15:09:00 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 7.344663937s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......................................X...X................] duration: 9.929053616s FAIL (2025-09-13 15:08:53 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 3) ======LOGS====== 6OXJYXIXQSQK3DMHSGQ56ANOAOEGXEHD4EWCOZDXTTIL37SAA, seg_pos={0 0}, upload=148985]: Uploading piece... Sep 13 14:55:15.907845810 (169h26m51.94741893s) [noise=true , address=145.79.227.52:4475 , piece_id=3T3MRZJ2T74J7CB5EJ4N2CIIERKWTQ5KWIOLDNV2LXHYLO5IKHTA, seg_pos={0 0}, upload=148985]: Uploading piece... Sep 13 14:55:15.908062555 (169h26m51.947635675s) [noise=true , address=145.79.227.5:4250 , piece_id=EPZOZXRFVPSAPD6UKPOCCBOX7VJNB4DQRKV6YTRWMPOLQQ5IH6KQ, seg_pos={0 0}, upload=148985]: Uploading piece... Sep 13 14:55:15.908276674 (169h26m51.947849794s) [noise=true , address=145.79.227.30:4780 , piece_id=7EMNYDLOMVLIDGE5ZK7WFQCNGBFLZA5AF5EHSLHYE22XJ2MP56VQ, seg_pos={0 0}, upload=148985]: Uploading piece... Sep 13 14:55:15.908522154 (169h26m51.948095274s) [noise=true , address=77.90.31.17:4793 , piece_id=PWVTHYWQY6GFO5FCIBANYNSX72VNUGT4UXKYI5CEIIHNZGNB63OA, seg_pos={0 0}, upload=148985]: Uploading piece... Sep 13 14:55:17.522666509 (169h26m53.562239629s) [upload=148985]: Next returned nil segment Sep 13 14:55:17.522686247 (169h26m53.562259367s) [upload=148985]: Waiting for error group managing segments... Sep 13 14:55:18.048737965 (169h26m54.088311085s) [noise=true , address=77.90.31.17:4793 , piece_id=PWVTHYWQY6GFO5FCIBANYNSX72VNUGT4UXKYI5CEIIHNZGNB63OA, seg_pos={0 0}, upload=148985]: Done uploading piece. err: <nil> Sep 13 14:55:18.323717226 (169h26m54.363290346s) [noise=true , address=77.90.31.11:4559 , piece_id=OYMOQVJRZ6SMMF7O2DXXDEOVRN27B3FT5EIVMFFBHC47UPZUUMIQ, seg_pos={0 0}, upload=148985]: Done uploading piece. err: <nil> Sep 13 14:55:18.769215263 (169h26m54.808788383s) [noise=true , address=45.91.116.14:4221 , piece_id=Q7ZPEITLUERUPOI247KAARIL3GSAPW6JNEUF7FSHQWJHL5GTGIZQ, seg_pos={0 0}, upload=148985]: Done uploading piece. err: <nil> Sep 13 14:56:14.598217910 (169h27m50.63779103s) [noise=true , address=145.79.227.28:4730 , piece_id=5WCQINYSLLWZR3Z3QWWUKRKDPRWYQ6VQPIZ4YDAD47JHSKDSCAGA, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:1SXQt5FugmWj1YdhsvaaR7bDoxTBkqHKZpyXSmQqGdX5ZRhKxV, address:145.79.227.28:4730): protocol: read tcp 188.166.225.170:42820->145.79.227.28:4730: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 13 14:56:14.598245002 (169h27m50.637818122s) [noise=true , address=145.79.227.52:4475 , piece_id=3T3MRZJ2T74J7CB5EJ4N2CIIERKWTQ5KWIOLDNV2LXHYLO5IKHTA, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:12UYjE6PEB9gZKwTKcnx23ykF4jDMJzLQoKmiFiejWew5mirdCj, address:145.79.227.52:4475): protocol: rpc: dial tcp 145.79.227.52:4475: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 13 14:56:14.598273767 (169h27m50.637846887s) [noise=true , address=145.79.227.50:4386 , piece_id=74QVIDRRHFRIA5Z6ZAQ4EWQ2PZ5HZI5JXNJ5WR2DVFJQ4WGE2UPQ, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:19aPjJnya2Q3stEhiedDSth3NHV8phLRj5yY79hZv81UgDitMv, address:145.79.227.50:4386): protocol: write tcp 188.166.225.170:42008->145.79.227.50:4386: use of closed network connection; write tcp 188.166.225.170:42008->145.79.227.50:4386: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42008->145.79.227.50:4386: use of closed network connection Sep 13 14:56:14.598293245 (169h27m50.637866365s) [noise=true , address=145.79.227.5:4250 , piece_id=EPZOZXRFVPSAPD6UKPOCCBOX7VJNB4DQRKV6YTRWMPOLQQ5IH6KQ, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:1uoGjfqBLfoNrg2X1aYKYLKQjgFUnzLjHetCFc9sPjJa2qqKYM, address:145.79.227.5:4250): protocol: write tcp 188.166.225.170:58078->145.79.227.5:4250: use of closed network connection; write tcp 188.166.225.170:58078->145.79.227.5:4250: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58078->145.79.227.5:4250: use of closed network connection Sep 13 14:56:14.598308794 (169h27m50.637881914s) [noise=true , address=145.79.227.33:4894 , piece_id=5KD2BL5J7DT7JDLPDS2QUUMM26AFWSXJ2BDTJSUGZ253ZV7XOVKQ, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:12KWNyruYhmSJVvgUF7fBpGf4H7eYmtAeEXToYjd2dcEPcj1E44, address:145.79.227.33:4894): protocol: write tcp 188.166.225.170:53614->145.79.227.33:4894: use of closed network connection; write tcp 188.166.225.170:53614->145.79.227.33:4894: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53614->145.79.227.33:4894: use of closed network connection Sep 13 14:56:14.598327330 (169h27m50.63790045s) [noise=true , address=145.79.227.7:4308 , piece_id=FYL7PQOKW74EJ2QLKMN6EYJKIT7A6B2VQESVP2VFVDVJHPHXFGAQ, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:1PgA5SZkBvK673LsV7p2sXpbi1r79esNNZspPZTzaitqfbjtyY, address:145.79.227.7:4308): protocol: write tcp 188.166.225.170:37146->145.79.227.7:4308: use of closed network connection; write tcp 188.166.225.170:37146->145.79.227.7:4308: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37146->145.79.227.7:4308: use of closed network connection Sep 13 14:56:14.598358169 (169h27m50.637931289s) [noise=true , address=145.79.227.36:4544 , piece_id=WZRHCCBJDHID4R4UOB76OABJSH2QVC6WZ6OZWIUUMKWWBDVWEG4A, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:12RSxwxw1WXkNcu7b3dEgJhyNkFGYUfKF4QVKmK323L5g3EKFnG, address:145.79.227.36:4544): protocol: rpc: dial tcp 145.79.227.36:4544: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 13 14:56:14.598387545 (169h27m50.637960665s) [noise=true , address=145.79.227.2:4137 , piece_id=NH4RZXYD3LKQZLLP5SSA7HXC7FBBOQV6QLBE2B3R5HH5F23Q7E6A, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:12tkbodtFrNCuJbvZk8YnpSdJfVsKasNuKhg2wGD8rDkNPrHZ21, address:145.79.227.2:4137): protocol: rpc: dial tcp 145.79.227.2:4137: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 13 14:56:14.598421721 (169h27m50.637994841s) [noise=true , address=145.79.227.30:4780 , piece_id=7EMNYDLOMVLIDGE5ZK7WFQCNGBFLZA5AF5EHSLHYE22XJ2MP56VQ, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:125WCbZ6a2NaewiLAHhHuPo91sQEfC9CZzEBV6rwF1JY7z13AEY, address:145.79.227.30:4780): protocol: read tcp 188.166.225.170:43016->145.79.227.30:4780: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 13 14:56:14.598427161 (169h27m50.638000281s) [noise=true , address=145.79.227.19:4090 , piece_id=YZ2H4UUM3Z5ZMCC6LQDAPLHCT7OQQAMRSDBQ3CH6I6UT2HAN4TXQ, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:12qSkJk9HUyQon3UP1AA8mRcqJbNdkKJh6cEUZoHLGNniqDvNqB, address:145.79.227.19:4090): protocol: write tcp 188.166.225.170:37110->145.79.227.19:4090: use of closed network connection; write tcp 188.166.225.170:37110->145.79.227.19:4090: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37110->145.79.227.19:4090: use of closed network connection Sep 13 14:56:14.598456818 (169h27m50.638029938s) [noise=true , address=145.79.227.10:4425 , piece_id=CRES4FQPX4WMTDTUBCPRZ2R4OHTI7MMHS6XSCZ7UXKDLSSLEVQ3Q, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:13sAcGFjAXuesAkUn7NuLaaV8TAah3Mr1kqd33wwh9MTELqmpD, address:145.79.227.10:4425): protocol: read tcp 188.166.225.170:58348->145.79.227.10:4425: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 13 14:56:14.598463220 (169h27m50.63803635s) [noise=true , address=145.79.227.35:4501 , piece_id=UKSF2KK675OI35QE6QZTH7QAJQ4SD7E42ZPMY2JIANOHGEFF44UQ, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:1hmsSAiSp1CBFqKhVE2S3sFfqYYkGVAiV1GfrT4Rd7hBaFNPpT, address:145.79.227.35:4501): protocol: rpc: dial tcp 145.79.227.35:4501: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 13 14:56:14.598484230 (169h27m50.63805735s) [noise=true , address=145.79.227.14:4867 , piece_id=7BW6OXJYXIXQSQK3DMHSGQ56ANOAOEGXEHD4EWCOZDXTTIL37SAA, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:12Sz7PyMo81CmAAkiCwpZ3M3JaM9HMHJLwAAM5M5kv6nnKktpKE, address:145.79.227.14:4867): protocol: read tcp 188.166.225.170:60252->145.79.227.14:4867: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 13 14:56:14.598492155 (169h27m50.638065275s) [noise=true , address=145.79.227.18:4018 , piece_id=72FTA6TNIQAVS2NDA5XOR4J2ZEEBWZ7LLGT4EAQ2EN2YB3VXG5JA, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:1bFRHbXjTaxoEKc5gbQWYt6jwUg8jknScBZpc8PdKNSZhYqUqP, address:145.79.227.18:4018): protocol: read tcp 188.166.225.170:60682->145.79.227.18:4018: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 13 14:56:14.598496193 (169h27m50.638069313s) [noise=true , address=145.79.227.47:4286 , piece_id=D2PSYV2IN5MFZGGF2U47EPRTCNDIMHB74GR3UFQ64AJN37FE7BVA, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:12v2vJnE92VrJXQrbxnAxANHCVASuXiBKMUZECMVmEWEjsZRtsQ, address:145.79.227.47:4286): protocol: read tcp 188.166.225.170:33092->145.79.227.47:4286: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 13 14:56:14.598521301 (169h27m50.638094421s) [seg_pos={0 0}, upload=148985]: Done waiting for segment. successful: 3 optimal: 14 errs: <nil> Sep 13 14:56:14.862062924 (169h27m50.901636044s) [upload=148985]: Done uploading segments... Sep 13 14:56:14.862090337 (169h27m50.901663457s) [upload=148985]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 3) Duration: 9.929085467s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 8.170102522s FAIL (2025-09-13 15:08:51 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 8.170124275s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [........................................X...X...............] duration: 9.376693123s FAIL (2025-09-13 15:08:19 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 3) ======LOGS====== 6OXJYXIXQSQK3DMHSGQ56ANOAOEGXEHD4EWCOZDXTTIL37SAA, seg_pos={0 0}, upload=148985]: Uploading piece... Sep 13 14:55:15.907845810 (169h26m51.94741893s) [noise=true , address=145.79.227.52:4475 , piece_id=3T3MRZJ2T74J7CB5EJ4N2CIIERKWTQ5KWIOLDNV2LXHYLO5IKHTA, seg_pos={0 0}, upload=148985]: Uploading piece... Sep 13 14:55:15.908062555 (169h26m51.947635675s) [noise=true , address=145.79.227.5:4250 , piece_id=EPZOZXRFVPSAPD6UKPOCCBOX7VJNB4DQRKV6YTRWMPOLQQ5IH6KQ, seg_pos={0 0}, upload=148985]: Uploading piece... Sep 13 14:55:15.908276674 (169h26m51.947849794s) [noise=true , address=145.79.227.30:4780 , piece_id=7EMNYDLOMVLIDGE5ZK7WFQCNGBFLZA5AF5EHSLHYE22XJ2MP56VQ, seg_pos={0 0}, upload=148985]: Uploading piece... Sep 13 14:55:15.908522154 (169h26m51.948095274s) [noise=true , address=77.90.31.17:4793 , piece_id=PWVTHYWQY6GFO5FCIBANYNSX72VNUGT4UXKYI5CEIIHNZGNB63OA, seg_pos={0 0}, upload=148985]: Uploading piece... Sep 13 14:55:17.522666509 (169h26m53.562239629s) [upload=148985]: Next returned nil segment Sep 13 14:55:17.522686247 (169h26m53.562259367s) [upload=148985]: Waiting for error group managing segments... Sep 13 14:55:18.048737965 (169h26m54.088311085s) [noise=true , address=77.90.31.17:4793 , piece_id=PWVTHYWQY6GFO5FCIBANYNSX72VNUGT4UXKYI5CEIIHNZGNB63OA, seg_pos={0 0}, upload=148985]: Done uploading piece. err: <nil> Sep 13 14:55:18.323717226 (169h26m54.363290346s) [noise=true , address=77.90.31.11:4559 , piece_id=OYMOQVJRZ6SMMF7O2DXXDEOVRN27B3FT5EIVMFFBHC47UPZUUMIQ, seg_pos={0 0}, upload=148985]: Done uploading piece. err: <nil> Sep 13 14:55:18.769215263 (169h26m54.808788383s) [noise=true , address=45.91.116.14:4221 , piece_id=Q7ZPEITLUERUPOI247KAARIL3GSAPW6JNEUF7FSHQWJHL5GTGIZQ, seg_pos={0 0}, upload=148985]: Done uploading piece. err: <nil> Sep 13 14:56:14.598217910 (169h27m50.63779103s) [noise=true , address=145.79.227.28:4730 , piece_id=5WCQINYSLLWZR3Z3QWWUKRKDPRWYQ6VQPIZ4YDAD47JHSKDSCAGA, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:1SXQt5FugmWj1YdhsvaaR7bDoxTBkqHKZpyXSmQqGdX5ZRhKxV, address:145.79.227.28:4730): protocol: read tcp 188.166.225.170:42820->145.79.227.28:4730: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 13 14:56:14.598245002 (169h27m50.637818122s) [noise=true , address=145.79.227.52:4475 , piece_id=3T3MRZJ2T74J7CB5EJ4N2CIIERKWTQ5KWIOLDNV2LXHYLO5IKHTA, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:12UYjE6PEB9gZKwTKcnx23ykF4jDMJzLQoKmiFiejWew5mirdCj, address:145.79.227.52:4475): protocol: rpc: dial tcp 145.79.227.52:4475: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 13 14:56:14.598273767 (169h27m50.637846887s) [noise=true , address=145.79.227.50:4386 , piece_id=74QVIDRRHFRIA5Z6ZAQ4EWQ2PZ5HZI5JXNJ5WR2DVFJQ4WGE2UPQ, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:19aPjJnya2Q3stEhiedDSth3NHV8phLRj5yY79hZv81UgDitMv, address:145.79.227.50:4386): protocol: write tcp 188.166.225.170:42008->145.79.227.50:4386: use of closed network connection; write tcp 188.166.225.170:42008->145.79.227.50:4386: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42008->145.79.227.50:4386: use of closed network connection Sep 13 14:56:14.598293245 (169h27m50.637866365s) [noise=true , address=145.79.227.5:4250 , piece_id=EPZOZXRFVPSAPD6UKPOCCBOX7VJNB4DQRKV6YTRWMPOLQQ5IH6KQ, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:1uoGjfqBLfoNrg2X1aYKYLKQjgFUnzLjHetCFc9sPjJa2qqKYM, address:145.79.227.5:4250): protocol: write tcp 188.166.225.170:58078->145.79.227.5:4250: use of closed network connection; write tcp 188.166.225.170:58078->145.79.227.5:4250: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58078->145.79.227.5:4250: use of closed network connection Sep 13 14:56:14.598308794 (169h27m50.637881914s) [noise=true , address=145.79.227.33:4894 , piece_id=5KD2BL5J7DT7JDLPDS2QUUMM26AFWSXJ2BDTJSUGZ253ZV7XOVKQ, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:12KWNyruYhmSJVvgUF7fBpGf4H7eYmtAeEXToYjd2dcEPcj1E44, address:145.79.227.33:4894): protocol: write tcp 188.166.225.170:53614->145.79.227.33:4894: use of closed network connection; write tcp 188.166.225.170:53614->145.79.227.33:4894: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53614->145.79.227.33:4894: use of closed network connection Sep 13 14:56:14.598327330 (169h27m50.63790045s) [noise=true , address=145.79.227.7:4308 , piece_id=FYL7PQOKW74EJ2QLKMN6EYJKIT7A6B2VQESVP2VFVDVJHPHXFGAQ, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:1PgA5SZkBvK673LsV7p2sXpbi1r79esNNZspPZTzaitqfbjtyY, address:145.79.227.7:4308): protocol: write tcp 188.166.225.170:37146->145.79.227.7:4308: use of closed network connection; write tcp 188.166.225.170:37146->145.79.227.7:4308: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37146->145.79.227.7:4308: use of closed network connection Sep 13 14:56:14.598358169 (169h27m50.637931289s) [noise=true , address=145.79.227.36:4544 , piece_id=WZRHCCBJDHID4R4UOB76OABJSH2QVC6WZ6OZWIUUMKWWBDVWEG4A, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:12RSxwxw1WXkNcu7b3dEgJhyNkFGYUfKF4QVKmK323L5g3EKFnG, address:145.79.227.36:4544): protocol: rpc: dial tcp 145.79.227.36:4544: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 13 14:56:14.598387545 (169h27m50.637960665s) [noise=true , address=145.79.227.2:4137 , piece_id=NH4RZXYD3LKQZLLP5SSA7HXC7FBBOQV6QLBE2B3R5HH5F23Q7E6A, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:12tkbodtFrNCuJbvZk8YnpSdJfVsKasNuKhg2wGD8rDkNPrHZ21, address:145.79.227.2:4137): protocol: rpc: dial tcp 145.79.227.2:4137: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 13 14:56:14.598421721 (169h27m50.637994841s) [noise=true , address=145.79.227.30:4780 , piece_id=7EMNYDLOMVLIDGE5ZK7WFQCNGBFLZA5AF5EHSLHYE22XJ2MP56VQ, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:125WCbZ6a2NaewiLAHhHuPo91sQEfC9CZzEBV6rwF1JY7z13AEY, address:145.79.227.30:4780): protocol: read tcp 188.166.225.170:43016->145.79.227.30:4780: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 13 14:56:14.598427161 (169h27m50.638000281s) [noise=true , address=145.79.227.19:4090 , piece_id=YZ2H4UUM3Z5ZMCC6LQDAPLHCT7OQQAMRSDBQ3CH6I6UT2HAN4TXQ, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:12qSkJk9HUyQon3UP1AA8mRcqJbNdkKJh6cEUZoHLGNniqDvNqB, address:145.79.227.19:4090): protocol: write tcp 188.166.225.170:37110->145.79.227.19:4090: use of closed network connection; write tcp 188.166.225.170:37110->145.79.227.19:4090: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37110->145.79.227.19:4090: use of closed network connection Sep 13 14:56:14.598456818 (169h27m50.638029938s) [noise=true , address=145.79.227.10:4425 , piece_id=CRES4FQPX4WMTDTUBCPRZ2R4OHTI7MMHS6XSCZ7UXKDLSSLEVQ3Q, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:13sAcGFjAXuesAkUn7NuLaaV8TAah3Mr1kqd33wwh9MTELqmpD, address:145.79.227.10:4425): protocol: read tcp 188.166.225.170:58348->145.79.227.10:4425: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 13 14:56:14.598463220 (169h27m50.63803635s) [noise=true , address=145.79.227.35:4501 , piece_id=UKSF2KK675OI35QE6QZTH7QAJQ4SD7E42ZPMY2JIANOHGEFF44UQ, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:1hmsSAiSp1CBFqKhVE2S3sFfqYYkGVAiV1GfrT4Rd7hBaFNPpT, address:145.79.227.35:4501): protocol: rpc: dial tcp 145.79.227.35:4501: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 13 14:56:14.598484230 (169h27m50.63805735s) [noise=true , address=145.79.227.14:4867 , piece_id=7BW6OXJYXIXQSQK3DMHSGQ56ANOAOEGXEHD4EWCOZDXTTIL37SAA, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:12Sz7PyMo81CmAAkiCwpZ3M3JaM9HMHJLwAAM5M5kv6nnKktpKE, address:145.79.227.14:4867): protocol: read tcp 188.166.225.170:60252->145.79.227.14:4867: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 13 14:56:14.598492155 (169h27m50.638065275s) [noise=true , address=145.79.227.18:4018 , piece_id=72FTA6TNIQAVS2NDA5XOR4J2ZEEBWZ7LLGT4EAQ2EN2YB3VXG5JA, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:1bFRHbXjTaxoEKc5gbQWYt6jwUg8jknScBZpc8PdKNSZhYqUqP, address:145.79.227.18:4018): protocol: read tcp 188.166.225.170:60682->145.79.227.18:4018: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 13 14:56:14.598496193 (169h27m50.638069313s) [noise=true , address=145.79.227.47:4286 , piece_id=D2PSYV2IN5MFZGGF2U47EPRTCNDIMHB74GR3UFQ64AJN37FE7BVA, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:12v2vJnE92VrJXQrbxnAxANHCVASuXiBKMUZECMVmEWEjsZRtsQ, address:145.79.227.47:4286): protocol: read tcp 188.166.225.170:33092->145.79.227.47:4286: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 13 14:56:14.598521301 (169h27m50.638094421s) [seg_pos={0 0}, upload=148985]: Done waiting for segment. successful: 3 optimal: 14 errs: <nil> Sep 13 14:56:14.862062924 (169h27m50.901636044s) [upload=148985]: Done uploading segments... Sep 13 14:56:14.862090337 (169h27m50.901663457s) [upload=148985]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 3) Duration: 9.376720074s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [............X.X...X..X.............X..X...X.X....X.X........] duration: 6.490498726s FAIL (2025-09-13 15:07:58 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.490608467s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 7.12195535s FAIL (2025-09-13 15:07:52 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 7.121970778s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........................................X...X..............] duration: 10.274690895s FAIL (2025-09-13 15:07:18 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 3) ======LOGS====== 6OXJYXIXQSQK3DMHSGQ56ANOAOEGXEHD4EWCOZDXTTIL37SAA, seg_pos={0 0}, upload=148985]: Uploading piece... Sep 13 14:55:15.907845810 (169h26m51.94741893s) [noise=true , address=145.79.227.52:4475 , piece_id=3T3MRZJ2T74J7CB5EJ4N2CIIERKWTQ5KWIOLDNV2LXHYLO5IKHTA, seg_pos={0 0}, upload=148985]: Uploading piece... Sep 13 14:55:15.908062555 (169h26m51.947635675s) [noise=true , address=145.79.227.5:4250 , piece_id=EPZOZXRFVPSAPD6UKPOCCBOX7VJNB4DQRKV6YTRWMPOLQQ5IH6KQ, seg_pos={0 0}, upload=148985]: Uploading piece... Sep 13 14:55:15.908276674 (169h26m51.947849794s) [noise=true , address=145.79.227.30:4780 , piece_id=7EMNYDLOMVLIDGE5ZK7WFQCNGBFLZA5AF5EHSLHYE22XJ2MP56VQ, seg_pos={0 0}, upload=148985]: Uploading piece... Sep 13 14:55:15.908522154 (169h26m51.948095274s) [noise=true , address=77.90.31.17:4793 , piece_id=PWVTHYWQY6GFO5FCIBANYNSX72VNUGT4UXKYI5CEIIHNZGNB63OA, seg_pos={0 0}, upload=148985]: Uploading piece... Sep 13 14:55:17.522666509 (169h26m53.562239629s) [upload=148985]: Next returned nil segment Sep 13 14:55:17.522686247 (169h26m53.562259367s) [upload=148985]: Waiting for error group managing segments... Sep 13 14:55:18.048737965 (169h26m54.088311085s) [noise=true , address=77.90.31.17:4793 , piece_id=PWVTHYWQY6GFO5FCIBANYNSX72VNUGT4UXKYI5CEIIHNZGNB63OA, seg_pos={0 0}, upload=148985]: Done uploading piece. err: <nil> Sep 13 14:55:18.323717226 (169h26m54.363290346s) [noise=true , address=77.90.31.11:4559 , piece_id=OYMOQVJRZ6SMMF7O2DXXDEOVRN27B3FT5EIVMFFBHC47UPZUUMIQ, seg_pos={0 0}, upload=148985]: Done uploading piece. err: <nil> Sep 13 14:55:18.769215263 (169h26m54.808788383s) [noise=true , address=45.91.116.14:4221 , piece_id=Q7ZPEITLUERUPOI247KAARIL3GSAPW6JNEUF7FSHQWJHL5GTGIZQ, seg_pos={0 0}, upload=148985]: Done uploading piece. err: <nil> Sep 13 14:56:14.598217910 (169h27m50.63779103s) [noise=true , address=145.79.227.28:4730 , piece_id=5WCQINYSLLWZR3Z3QWWUKRKDPRWYQ6VQPIZ4YDAD47JHSKDSCAGA, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:1SXQt5FugmWj1YdhsvaaR7bDoxTBkqHKZpyXSmQqGdX5ZRhKxV, address:145.79.227.28:4730): protocol: read tcp 188.166.225.170:42820->145.79.227.28:4730: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 13 14:56:14.598245002 (169h27m50.637818122s) [noise=true , address=145.79.227.52:4475 , piece_id=3T3MRZJ2T74J7CB5EJ4N2CIIERKWTQ5KWIOLDNV2LXHYLO5IKHTA, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:12UYjE6PEB9gZKwTKcnx23ykF4jDMJzLQoKmiFiejWew5mirdCj, address:145.79.227.52:4475): protocol: rpc: dial tcp 145.79.227.52:4475: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 13 14:56:14.598273767 (169h27m50.637846887s) [noise=true , address=145.79.227.50:4386 , piece_id=74QVIDRRHFRIA5Z6ZAQ4EWQ2PZ5HZI5JXNJ5WR2DVFJQ4WGE2UPQ, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:19aPjJnya2Q3stEhiedDSth3NHV8phLRj5yY79hZv81UgDitMv, address:145.79.227.50:4386): protocol: write tcp 188.166.225.170:42008->145.79.227.50:4386: use of closed network connection; write tcp 188.166.225.170:42008->145.79.227.50:4386: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42008->145.79.227.50:4386: use of closed network connection Sep 13 14:56:14.598293245 (169h27m50.637866365s) [noise=true , address=145.79.227.5:4250 , piece_id=EPZOZXRFVPSAPD6UKPOCCBOX7VJNB4DQRKV6YTRWMPOLQQ5IH6KQ, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:1uoGjfqBLfoNrg2X1aYKYLKQjgFUnzLjHetCFc9sPjJa2qqKYM, address:145.79.227.5:4250): protocol: write tcp 188.166.225.170:58078->145.79.227.5:4250: use of closed network connection; write tcp 188.166.225.170:58078->145.79.227.5:4250: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58078->145.79.227.5:4250: use of closed network connection Sep 13 14:56:14.598308794 (169h27m50.637881914s) [noise=true , address=145.79.227.33:4894 , piece_id=5KD2BL5J7DT7JDLPDS2QUUMM26AFWSXJ2BDTJSUGZ253ZV7XOVKQ, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:12KWNyruYhmSJVvgUF7fBpGf4H7eYmtAeEXToYjd2dcEPcj1E44, address:145.79.227.33:4894): protocol: write tcp 188.166.225.170:53614->145.79.227.33:4894: use of closed network connection; write tcp 188.166.225.170:53614->145.79.227.33:4894: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53614->145.79.227.33:4894: use of closed network connection Sep 13 14:56:14.598327330 (169h27m50.63790045s) [noise=true , address=145.79.227.7:4308 , piece_id=FYL7PQOKW74EJ2QLKMN6EYJKIT7A6B2VQESVP2VFVDVJHPHXFGAQ, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:1PgA5SZkBvK673LsV7p2sXpbi1r79esNNZspPZTzaitqfbjtyY, address:145.79.227.7:4308): protocol: write tcp 188.166.225.170:37146->145.79.227.7:4308: use of closed network connection; write tcp 188.166.225.170:37146->145.79.227.7:4308: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37146->145.79.227.7:4308: use of closed network connection Sep 13 14:56:14.598358169 (169h27m50.637931289s) [noise=true , address=145.79.227.36:4544 , piece_id=WZRHCCBJDHID4R4UOB76OABJSH2QVC6WZ6OZWIUUMKWWBDVWEG4A, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:12RSxwxw1WXkNcu7b3dEgJhyNkFGYUfKF4QVKmK323L5g3EKFnG, address:145.79.227.36:4544): protocol: rpc: dial tcp 145.79.227.36:4544: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 13 14:56:14.598387545 (169h27m50.637960665s) [noise=true , address=145.79.227.2:4137 , piece_id=NH4RZXYD3LKQZLLP5SSA7HXC7FBBOQV6QLBE2B3R5HH5F23Q7E6A, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:12tkbodtFrNCuJbvZk8YnpSdJfVsKasNuKhg2wGD8rDkNPrHZ21, address:145.79.227.2:4137): protocol: rpc: dial tcp 145.79.227.2:4137: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 13 14:56:14.598421721 (169h27m50.637994841s) [noise=true , address=145.79.227.30:4780 , piece_id=7EMNYDLOMVLIDGE5ZK7WFQCNGBFLZA5AF5EHSLHYE22XJ2MP56VQ, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:125WCbZ6a2NaewiLAHhHuPo91sQEfC9CZzEBV6rwF1JY7z13AEY, address:145.79.227.30:4780): protocol: read tcp 188.166.225.170:43016->145.79.227.30:4780: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 13 14:56:14.598427161 (169h27m50.638000281s) [noise=true , address=145.79.227.19:4090 , piece_id=YZ2H4UUM3Z5ZMCC6LQDAPLHCT7OQQAMRSDBQ3CH6I6UT2HAN4TXQ, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:12qSkJk9HUyQon3UP1AA8mRcqJbNdkKJh6cEUZoHLGNniqDvNqB, address:145.79.227.19:4090): protocol: write tcp 188.166.225.170:37110->145.79.227.19:4090: use of closed network connection; write tcp 188.166.225.170:37110->145.79.227.19:4090: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37110->145.79.227.19:4090: use of closed network connection Sep 13 14:56:14.598456818 (169h27m50.638029938s) [noise=true , address=145.79.227.10:4425 , piece_id=CRES4FQPX4WMTDTUBCPRZ2R4OHTI7MMHS6XSCZ7UXKDLSSLEVQ3Q, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:13sAcGFjAXuesAkUn7NuLaaV8TAah3Mr1kqd33wwh9MTELqmpD, address:145.79.227.10:4425): protocol: read tcp 188.166.225.170:58348->145.79.227.10:4425: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 13 14:56:14.598463220 (169h27m50.63803635s) [noise=true , address=145.79.227.35:4501 , piece_id=UKSF2KK675OI35QE6QZTH7QAJQ4SD7E42ZPMY2JIANOHGEFF44UQ, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:1hmsSAiSp1CBFqKhVE2S3sFfqYYkGVAiV1GfrT4Rd7hBaFNPpT, address:145.79.227.35:4501): protocol: rpc: dial tcp 145.79.227.35:4501: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 13 14:56:14.598484230 (169h27m50.63805735s) [noise=true , address=145.79.227.14:4867 , piece_id=7BW6OXJYXIXQSQK3DMHSGQ56ANOAOEGXEHD4EWCOZDXTTIL37SAA, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:12Sz7PyMo81CmAAkiCwpZ3M3JaM9HMHJLwAAM5M5kv6nnKktpKE, address:145.79.227.14:4867): protocol: read tcp 188.166.225.170:60252->145.79.227.14:4867: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 13 14:56:14.598492155 (169h27m50.638065275s) [noise=true , address=145.79.227.18:4018 , piece_id=72FTA6TNIQAVS2NDA5XOR4J2ZEEBWZ7LLGT4EAQ2EN2YB3VXG5JA, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:1bFRHbXjTaxoEKc5gbQWYt6jwUg8jknScBZpc8PdKNSZhYqUqP, address:145.79.227.18:4018): protocol: read tcp 188.166.225.170:60682->145.79.227.18:4018: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 13 14:56:14.598496193 (169h27m50.638069313s) [noise=true , address=145.79.227.47:4286 , piece_id=D2PSYV2IN5MFZGGF2U47EPRTCNDIMHB74GR3UFQ64AJN37FE7BVA, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:12v2vJnE92VrJXQrbxnAxANHCVASuXiBKMUZECMVmEWEjsZRtsQ, address:145.79.227.47:4286): protocol: read tcp 188.166.225.170:33092->145.79.227.47:4286: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 13 14:56:14.598521301 (169h27m50.638094421s) [seg_pos={0 0}, upload=148985]: Done waiting for segment. successful: 3 optimal: 14 errs: <nil> Sep 13 14:56:14.862062924 (169h27m50.901636044s) [upload=148985]: Done uploading segments... Sep 13 14:56:14.862090337 (169h27m50.901663457s) [upload=148985]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 3) Duration: 10.274715982s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [.............X.X...X..X.............X..X...X.X....X.X.......] duration: 7.476083338s FAIL (2025-09-13 15:07:18 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 7.476106661s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 7.591675041s FAIL (2025-09-13 15:07:15 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 7.591699508s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 7.103917014s FAIL (2025-09-13 15:06:59 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 7.103939557s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [..............X.X...X..X.............X..X...X.X....X.X......] duration: 7.781698304s FAIL (2025-09-13 15:06:28 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 7.781721228s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........................................X...X.............] duration: 10.384489372s FAIL (2025-09-13 15:06:19 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 3) ======LOGS====== 6OXJYXIXQSQK3DMHSGQ56ANOAOEGXEHD4EWCOZDXTTIL37SAA, seg_pos={0 0}, upload=148985]: Uploading piece... Sep 13 14:55:15.907845810 (169h26m51.94741893s) [noise=true , address=145.79.227.52:4475 , piece_id=3T3MRZJ2T74J7CB5EJ4N2CIIERKWTQ5KWIOLDNV2LXHYLO5IKHTA, seg_pos={0 0}, upload=148985]: Uploading piece... Sep 13 14:55:15.908062555 (169h26m51.947635675s) [noise=true , address=145.79.227.5:4250 , piece_id=EPZOZXRFVPSAPD6UKPOCCBOX7VJNB4DQRKV6YTRWMPOLQQ5IH6KQ, seg_pos={0 0}, upload=148985]: Uploading piece... Sep 13 14:55:15.908276674 (169h26m51.947849794s) [noise=true , address=145.79.227.30:4780 , piece_id=7EMNYDLOMVLIDGE5ZK7WFQCNGBFLZA5AF5EHSLHYE22XJ2MP56VQ, seg_pos={0 0}, upload=148985]: Uploading piece... Sep 13 14:55:15.908522154 (169h26m51.948095274s) [noise=true , address=77.90.31.17:4793 , piece_id=PWVTHYWQY6GFO5FCIBANYNSX72VNUGT4UXKYI5CEIIHNZGNB63OA, seg_pos={0 0}, upload=148985]: Uploading piece... Sep 13 14:55:17.522666509 (169h26m53.562239629s) [upload=148985]: Next returned nil segment Sep 13 14:55:17.522686247 (169h26m53.562259367s) [upload=148985]: Waiting for error group managing segments... Sep 13 14:55:18.048737965 (169h26m54.088311085s) [noise=true , address=77.90.31.17:4793 , piece_id=PWVTHYWQY6GFO5FCIBANYNSX72VNUGT4UXKYI5CEIIHNZGNB63OA, seg_pos={0 0}, upload=148985]: Done uploading piece. err: <nil> Sep 13 14:55:18.323717226 (169h26m54.363290346s) [noise=true , address=77.90.31.11:4559 , piece_id=OYMOQVJRZ6SMMF7O2DXXDEOVRN27B3FT5EIVMFFBHC47UPZUUMIQ, seg_pos={0 0}, upload=148985]: Done uploading piece. err: <nil> Sep 13 14:55:18.769215263 (169h26m54.808788383s) [noise=true , address=45.91.116.14:4221 , piece_id=Q7ZPEITLUERUPOI247KAARIL3GSAPW6JNEUF7FSHQWJHL5GTGIZQ, seg_pos={0 0}, upload=148985]: Done uploading piece. err: <nil> Sep 13 14:56:14.598217910 (169h27m50.63779103s) [noise=true , address=145.79.227.28:4730 , piece_id=5WCQINYSLLWZR3Z3QWWUKRKDPRWYQ6VQPIZ4YDAD47JHSKDSCAGA, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:1SXQt5FugmWj1YdhsvaaR7bDoxTBkqHKZpyXSmQqGdX5ZRhKxV, address:145.79.227.28:4730): protocol: read tcp 188.166.225.170:42820->145.79.227.28:4730: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 13 14:56:14.598245002 (169h27m50.637818122s) [noise=true , address=145.79.227.52:4475 , piece_id=3T3MRZJ2T74J7CB5EJ4N2CIIERKWTQ5KWIOLDNV2LXHYLO5IKHTA, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:12UYjE6PEB9gZKwTKcnx23ykF4jDMJzLQoKmiFiejWew5mirdCj, address:145.79.227.52:4475): protocol: rpc: dial tcp 145.79.227.52:4475: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 13 14:56:14.598273767 (169h27m50.637846887s) [noise=true , address=145.79.227.50:4386 , piece_id=74QVIDRRHFRIA5Z6ZAQ4EWQ2PZ5HZI5JXNJ5WR2DVFJQ4WGE2UPQ, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:19aPjJnya2Q3stEhiedDSth3NHV8phLRj5yY79hZv81UgDitMv, address:145.79.227.50:4386): protocol: write tcp 188.166.225.170:42008->145.79.227.50:4386: use of closed network connection; write tcp 188.166.225.170:42008->145.79.227.50:4386: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42008->145.79.227.50:4386: use of closed network connection Sep 13 14:56:14.598293245 (169h27m50.637866365s) [noise=true , address=145.79.227.5:4250 , piece_id=EPZOZXRFVPSAPD6UKPOCCBOX7VJNB4DQRKV6YTRWMPOLQQ5IH6KQ, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:1uoGjfqBLfoNrg2X1aYKYLKQjgFUnzLjHetCFc9sPjJa2qqKYM, address:145.79.227.5:4250): protocol: write tcp 188.166.225.170:58078->145.79.227.5:4250: use of closed network connection; write tcp 188.166.225.170:58078->145.79.227.5:4250: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58078->145.79.227.5:4250: use of closed network connection Sep 13 14:56:14.598308794 (169h27m50.637881914s) [noise=true , address=145.79.227.33:4894 , piece_id=5KD2BL5J7DT7JDLPDS2QUUMM26AFWSXJ2BDTJSUGZ253ZV7XOVKQ, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:12KWNyruYhmSJVvgUF7fBpGf4H7eYmtAeEXToYjd2dcEPcj1E44, address:145.79.227.33:4894): protocol: write tcp 188.166.225.170:53614->145.79.227.33:4894: use of closed network connection; write tcp 188.166.225.170:53614->145.79.227.33:4894: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53614->145.79.227.33:4894: use of closed network connection Sep 13 14:56:14.598327330 (169h27m50.63790045s) [noise=true , address=145.79.227.7:4308 , piece_id=FYL7PQOKW74EJ2QLKMN6EYJKIT7A6B2VQESVP2VFVDVJHPHXFGAQ, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:1PgA5SZkBvK673LsV7p2sXpbi1r79esNNZspPZTzaitqfbjtyY, address:145.79.227.7:4308): protocol: write tcp 188.166.225.170:37146->145.79.227.7:4308: use of closed network connection; write tcp 188.166.225.170:37146->145.79.227.7:4308: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37146->145.79.227.7:4308: use of closed network connection Sep 13 14:56:14.598358169 (169h27m50.637931289s) [noise=true , address=145.79.227.36:4544 , piece_id=WZRHCCBJDHID4R4UOB76OABJSH2QVC6WZ6OZWIUUMKWWBDVWEG4A, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:12RSxwxw1WXkNcu7b3dEgJhyNkFGYUfKF4QVKmK323L5g3EKFnG, address:145.79.227.36:4544): protocol: rpc: dial tcp 145.79.227.36:4544: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 13 14:56:14.598387545 (169h27m50.637960665s) [noise=true , address=145.79.227.2:4137 , piece_id=NH4RZXYD3LKQZLLP5SSA7HXC7FBBOQV6QLBE2B3R5HH5F23Q7E6A, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:12tkbodtFrNCuJbvZk8YnpSdJfVsKasNuKhg2wGD8rDkNPrHZ21, address:145.79.227.2:4137): protocol: rpc: dial tcp 145.79.227.2:4137: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 13 14:56:14.598421721 (169h27m50.637994841s) [noise=true , address=145.79.227.30:4780 , piece_id=7EMNYDLOMVLIDGE5ZK7WFQCNGBFLZA5AF5EHSLHYE22XJ2MP56VQ, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:125WCbZ6a2NaewiLAHhHuPo91sQEfC9CZzEBV6rwF1JY7z13AEY, address:145.79.227.30:4780): protocol: read tcp 188.166.225.170:43016->145.79.227.30:4780: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 13 14:56:14.598427161 (169h27m50.638000281s) [noise=true , address=145.79.227.19:4090 , piece_id=YZ2H4UUM3Z5ZMCC6LQDAPLHCT7OQQAMRSDBQ3CH6I6UT2HAN4TXQ, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:12qSkJk9HUyQon3UP1AA8mRcqJbNdkKJh6cEUZoHLGNniqDvNqB, address:145.79.227.19:4090): protocol: write tcp 188.166.225.170:37110->145.79.227.19:4090: use of closed network connection; write tcp 188.166.225.170:37110->145.79.227.19:4090: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37110->145.79.227.19:4090: use of closed network connection Sep 13 14:56:14.598456818 (169h27m50.638029938s) [noise=true , address=145.79.227.10:4425 , piece_id=CRES4FQPX4WMTDTUBCPRZ2R4OHTI7MMHS6XSCZ7UXKDLSSLEVQ3Q, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:13sAcGFjAXuesAkUn7NuLaaV8TAah3Mr1kqd33wwh9MTELqmpD, address:145.79.227.10:4425): protocol: read tcp 188.166.225.170:58348->145.79.227.10:4425: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 13 14:56:14.598463220 (169h27m50.63803635s) [noise=true , address=145.79.227.35:4501 , piece_id=UKSF2KK675OI35QE6QZTH7QAJQ4SD7E42ZPMY2JIANOHGEFF44UQ, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:1hmsSAiSp1CBFqKhVE2S3sFfqYYkGVAiV1GfrT4Rd7hBaFNPpT, address:145.79.227.35:4501): protocol: rpc: dial tcp 145.79.227.35:4501: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 13 14:56:14.598484230 (169h27m50.63805735s) [noise=true , address=145.79.227.14:4867 , piece_id=7BW6OXJYXIXQSQK3DMHSGQ56ANOAOEGXEHD4EWCOZDXTTIL37SAA, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:12Sz7PyMo81CmAAkiCwpZ3M3JaM9HMHJLwAAM5M5kv6nnKktpKE, address:145.79.227.14:4867): protocol: read tcp 188.166.225.170:60252->145.79.227.14:4867: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 13 14:56:14.598492155 (169h27m50.638065275s) [noise=true , address=145.79.227.18:4018 , piece_id=72FTA6TNIQAVS2NDA5XOR4J2ZEEBWZ7LLGT4EAQ2EN2YB3VXG5JA, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:1bFRHbXjTaxoEKc5gbQWYt6jwUg8jknScBZpc8PdKNSZhYqUqP, address:145.79.227.18:4018): protocol: read tcp 188.166.225.170:60682->145.79.227.18:4018: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 13 14:56:14.598496193 (169h27m50.638069313s) [noise=true , address=145.79.227.47:4286 , piece_id=D2PSYV2IN5MFZGGF2U47EPRTCNDIMHB74GR3UFQ64AJN37FE7BVA, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:12v2vJnE92VrJXQrbxnAxANHCVASuXiBKMUZECMVmEWEjsZRtsQ, address:145.79.227.47:4286): protocol: read tcp 188.166.225.170:33092->145.79.227.47:4286: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 13 14:56:14.598521301 (169h27m50.638094421s) [seg_pos={0 0}, upload=148985]: Done waiting for segment. successful: 3 optimal: 14 errs: <nil> Sep 13 14:56:14.862062924 (169h27m50.901636044s) [upload=148985]: Done uploading segments... Sep 13 14:56:14.862090337 (169h27m50.901663457s) [upload=148985]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 3) Duration: 10.384519592s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [...............X.X...X..X.............X..X...X.X....X.X.....] duration: 7.45118593s FAIL (2025-09-13 15:06:07 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 7.45135314s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 7.155673989s FAIL (2025-09-13 15:06:05 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 7.155699005s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 7.205854188s FAIL (2025-09-13 15:05:25 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 7.205937216s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [...........................................X...X............] duration: 15.254443755s FAIL (2025-09-13 15:05:17 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 3) ======LOGS====== 6OXJYXIXQSQK3DMHSGQ56ANOAOEGXEHD4EWCOZDXTTIL37SAA, seg_pos={0 0}, upload=148985]: Uploading piece... Sep 13 14:55:15.907845810 (169h26m51.94741893s) [noise=true , address=145.79.227.52:4475 , piece_id=3T3MRZJ2T74J7CB5EJ4N2CIIERKWTQ5KWIOLDNV2LXHYLO5IKHTA, seg_pos={0 0}, upload=148985]: Uploading piece... Sep 13 14:55:15.908062555 (169h26m51.947635675s) [noise=true , address=145.79.227.5:4250 , piece_id=EPZOZXRFVPSAPD6UKPOCCBOX7VJNB4DQRKV6YTRWMPOLQQ5IH6KQ, seg_pos={0 0}, upload=148985]: Uploading piece... Sep 13 14:55:15.908276674 (169h26m51.947849794s) [noise=true , address=145.79.227.30:4780 , piece_id=7EMNYDLOMVLIDGE5ZK7WFQCNGBFLZA5AF5EHSLHYE22XJ2MP56VQ, seg_pos={0 0}, upload=148985]: Uploading piece... Sep 13 14:55:15.908522154 (169h26m51.948095274s) [noise=true , address=77.90.31.17:4793 , piece_id=PWVTHYWQY6GFO5FCIBANYNSX72VNUGT4UXKYI5CEIIHNZGNB63OA, seg_pos={0 0}, upload=148985]: Uploading piece... Sep 13 14:55:17.522666509 (169h26m53.562239629s) [upload=148985]: Next returned nil segment Sep 13 14:55:17.522686247 (169h26m53.562259367s) [upload=148985]: Waiting for error group managing segments... Sep 13 14:55:18.048737965 (169h26m54.088311085s) [noise=true , address=77.90.31.17:4793 , piece_id=PWVTHYWQY6GFO5FCIBANYNSX72VNUGT4UXKYI5CEIIHNZGNB63OA, seg_pos={0 0}, upload=148985]: Done uploading piece. err: <nil> Sep 13 14:55:18.323717226 (169h26m54.363290346s) [noise=true , address=77.90.31.11:4559 , piece_id=OYMOQVJRZ6SMMF7O2DXXDEOVRN27B3FT5EIVMFFBHC47UPZUUMIQ, seg_pos={0 0}, upload=148985]: Done uploading piece. err: <nil> Sep 13 14:55:18.769215263 (169h26m54.808788383s) [noise=true , address=45.91.116.14:4221 , piece_id=Q7ZPEITLUERUPOI247KAARIL3GSAPW6JNEUF7FSHQWJHL5GTGIZQ, seg_pos={0 0}, upload=148985]: Done uploading piece. err: <nil> Sep 13 14:56:14.598217910 (169h27m50.63779103s) [noise=true , address=145.79.227.28:4730 , piece_id=5WCQINYSLLWZR3Z3QWWUKRKDPRWYQ6VQPIZ4YDAD47JHSKDSCAGA, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:1SXQt5FugmWj1YdhsvaaR7bDoxTBkqHKZpyXSmQqGdX5ZRhKxV, address:145.79.227.28:4730): protocol: read tcp 188.166.225.170:42820->145.79.227.28:4730: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 13 14:56:14.598245002 (169h27m50.637818122s) [noise=true , address=145.79.227.52:4475 , piece_id=3T3MRZJ2T74J7CB5EJ4N2CIIERKWTQ5KWIOLDNV2LXHYLO5IKHTA, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:12UYjE6PEB9gZKwTKcnx23ykF4jDMJzLQoKmiFiejWew5mirdCj, address:145.79.227.52:4475): protocol: rpc: dial tcp 145.79.227.52:4475: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 13 14:56:14.598273767 (169h27m50.637846887s) [noise=true , address=145.79.227.50:4386 , piece_id=74QVIDRRHFRIA5Z6ZAQ4EWQ2PZ5HZI5JXNJ5WR2DVFJQ4WGE2UPQ, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:19aPjJnya2Q3stEhiedDSth3NHV8phLRj5yY79hZv81UgDitMv, address:145.79.227.50:4386): protocol: write tcp 188.166.225.170:42008->145.79.227.50:4386: use of closed network connection; write tcp 188.166.225.170:42008->145.79.227.50:4386: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42008->145.79.227.50:4386: use of closed network connection Sep 13 14:56:14.598293245 (169h27m50.637866365s) [noise=true , address=145.79.227.5:4250 , piece_id=EPZOZXRFVPSAPD6UKPOCCBOX7VJNB4DQRKV6YTRWMPOLQQ5IH6KQ, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:1uoGjfqBLfoNrg2X1aYKYLKQjgFUnzLjHetCFc9sPjJa2qqKYM, address:145.79.227.5:4250): protocol: write tcp 188.166.225.170:58078->145.79.227.5:4250: use of closed network connection; write tcp 188.166.225.170:58078->145.79.227.5:4250: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58078->145.79.227.5:4250: use of closed network connection Sep 13 14:56:14.598308794 (169h27m50.637881914s) [noise=true , address=145.79.227.33:4894 , piece_id=5KD2BL5J7DT7JDLPDS2QUUMM26AFWSXJ2BDTJSUGZ253ZV7XOVKQ, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:12KWNyruYhmSJVvgUF7fBpGf4H7eYmtAeEXToYjd2dcEPcj1E44, address:145.79.227.33:4894): protocol: write tcp 188.166.225.170:53614->145.79.227.33:4894: use of closed network connection; write tcp 188.166.225.170:53614->145.79.227.33:4894: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53614->145.79.227.33:4894: use of closed network connection Sep 13 14:56:14.598327330 (169h27m50.63790045s) [noise=true , address=145.79.227.7:4308 , piece_id=FYL7PQOKW74EJ2QLKMN6EYJKIT7A6B2VQESVP2VFVDVJHPHXFGAQ, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:1PgA5SZkBvK673LsV7p2sXpbi1r79esNNZspPZTzaitqfbjtyY, address:145.79.227.7:4308): protocol: write tcp 188.166.225.170:37146->145.79.227.7:4308: use of closed network connection; write tcp 188.166.225.170:37146->145.79.227.7:4308: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37146->145.79.227.7:4308: use of closed network connection Sep 13 14:56:14.598358169 (169h27m50.637931289s) [noise=true , address=145.79.227.36:4544 , piece_id=WZRHCCBJDHID4R4UOB76OABJSH2QVC6WZ6OZWIUUMKWWBDVWEG4A, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:12RSxwxw1WXkNcu7b3dEgJhyNkFGYUfKF4QVKmK323L5g3EKFnG, address:145.79.227.36:4544): protocol: rpc: dial tcp 145.79.227.36:4544: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 13 14:56:14.598387545 (169h27m50.637960665s) [noise=true , address=145.79.227.2:4137 , piece_id=NH4RZXYD3LKQZLLP5SSA7HXC7FBBOQV6QLBE2B3R5HH5F23Q7E6A, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:12tkbodtFrNCuJbvZk8YnpSdJfVsKasNuKhg2wGD8rDkNPrHZ21, address:145.79.227.2:4137): protocol: rpc: dial tcp 145.79.227.2:4137: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 13 14:56:14.598421721 (169h27m50.637994841s) [noise=true , address=145.79.227.30:4780 , piece_id=7EMNYDLOMVLIDGE5ZK7WFQCNGBFLZA5AF5EHSLHYE22XJ2MP56VQ, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:125WCbZ6a2NaewiLAHhHuPo91sQEfC9CZzEBV6rwF1JY7z13AEY, address:145.79.227.30:4780): protocol: read tcp 188.166.225.170:43016->145.79.227.30:4780: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 13 14:56:14.598427161 (169h27m50.638000281s) [noise=true , address=145.79.227.19:4090 , piece_id=YZ2H4UUM3Z5ZMCC6LQDAPLHCT7OQQAMRSDBQ3CH6I6UT2HAN4TXQ, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:12qSkJk9HUyQon3UP1AA8mRcqJbNdkKJh6cEUZoHLGNniqDvNqB, address:145.79.227.19:4090): protocol: write tcp 188.166.225.170:37110->145.79.227.19:4090: use of closed network connection; write tcp 188.166.225.170:37110->145.79.227.19:4090: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37110->145.79.227.19:4090: use of closed network connection Sep 13 14:56:14.598456818 (169h27m50.638029938s) [noise=true , address=145.79.227.10:4425 , piece_id=CRES4FQPX4WMTDTUBCPRZ2R4OHTI7MMHS6XSCZ7UXKDLSSLEVQ3Q, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:13sAcGFjAXuesAkUn7NuLaaV8TAah3Mr1kqd33wwh9MTELqmpD, address:145.79.227.10:4425): protocol: read tcp 188.166.225.170:58348->145.79.227.10:4425: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 13 14:56:14.598463220 (169h27m50.63803635s) [noise=true , address=145.79.227.35:4501 , piece_id=UKSF2KK675OI35QE6QZTH7QAJQ4SD7E42ZPMY2JIANOHGEFF44UQ, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:1hmsSAiSp1CBFqKhVE2S3sFfqYYkGVAiV1GfrT4Rd7hBaFNPpT, address:145.79.227.35:4501): protocol: rpc: dial tcp 145.79.227.35:4501: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 13 14:56:14.598484230 (169h27m50.63805735s) [noise=true , address=145.79.227.14:4867 , piece_id=7BW6OXJYXIXQSQK3DMHSGQ56ANOAOEGXEHD4EWCOZDXTTIL37SAA, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:12Sz7PyMo81CmAAkiCwpZ3M3JaM9HMHJLwAAM5M5kv6nnKktpKE, address:145.79.227.14:4867): protocol: read tcp 188.166.225.170:60252->145.79.227.14:4867: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 13 14:56:14.598492155 (169h27m50.638065275s) [noise=true , address=145.79.227.18:4018 , piece_id=72FTA6TNIQAVS2NDA5XOR4J2ZEEBWZ7LLGT4EAQ2EN2YB3VXG5JA, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:1bFRHbXjTaxoEKc5gbQWYt6jwUg8jknScBZpc8PdKNSZhYqUqP, address:145.79.227.18:4018): protocol: read tcp 188.166.225.170:60682->145.79.227.18:4018: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 13 14:56:14.598496193 (169h27m50.638069313s) [noise=true , address=145.79.227.47:4286 , piece_id=D2PSYV2IN5MFZGGF2U47EPRTCNDIMHB74GR3UFQ64AJN37FE7BVA, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:12v2vJnE92VrJXQrbxnAxANHCVASuXiBKMUZECMVmEWEjsZRtsQ, address:145.79.227.47:4286): protocol: read tcp 188.166.225.170:33092->145.79.227.47:4286: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 13 14:56:14.598521301 (169h27m50.638094421s) [seg_pos={0 0}, upload=148985]: Done waiting for segment. successful: 3 optimal: 14 errs: <nil> Sep 13 14:56:14.862062924 (169h27m50.901636044s) [upload=148985]: Done uploading segments... Sep 13 14:56:14.862090337 (169h27m50.901663457s) [upload=148985]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 3) Duration: 15.254525431s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [................X.X...X..X.............X..X...X.X....X.X....] duration: 6.861232491s FAIL (2025-09-13 15:05:11 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.861276035s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 8.227802691s FAIL (2025-09-13 15:05:09 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 8.227823242s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [............................................X...X...........] duration: 13.46806119s FAIL (2025-09-13 15:04:42 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 3) ======LOGS====== 6OXJYXIXQSQK3DMHSGQ56ANOAOEGXEHD4EWCOZDXTTIL37SAA, seg_pos={0 0}, upload=148985]: Uploading piece... Sep 13 14:55:15.907845810 (169h26m51.94741893s) [noise=true , address=145.79.227.52:4475 , piece_id=3T3MRZJ2T74J7CB5EJ4N2CIIERKWTQ5KWIOLDNV2LXHYLO5IKHTA, seg_pos={0 0}, upload=148985]: Uploading piece... Sep 13 14:55:15.908062555 (169h26m51.947635675s) [noise=true , address=145.79.227.5:4250 , piece_id=EPZOZXRFVPSAPD6UKPOCCBOX7VJNB4DQRKV6YTRWMPOLQQ5IH6KQ, seg_pos={0 0}, upload=148985]: Uploading piece... Sep 13 14:55:15.908276674 (169h26m51.947849794s) [noise=true , address=145.79.227.30:4780 , piece_id=7EMNYDLOMVLIDGE5ZK7WFQCNGBFLZA5AF5EHSLHYE22XJ2MP56VQ, seg_pos={0 0}, upload=148985]: Uploading piece... Sep 13 14:55:15.908522154 (169h26m51.948095274s) [noise=true , address=77.90.31.17:4793 , piece_id=PWVTHYWQY6GFO5FCIBANYNSX72VNUGT4UXKYI5CEIIHNZGNB63OA, seg_pos={0 0}, upload=148985]: Uploading piece... Sep 13 14:55:17.522666509 (169h26m53.562239629s) [upload=148985]: Next returned nil segment Sep 13 14:55:17.522686247 (169h26m53.562259367s) [upload=148985]: Waiting for error group managing segments... Sep 13 14:55:18.048737965 (169h26m54.088311085s) [noise=true , address=77.90.31.17:4793 , piece_id=PWVTHYWQY6GFO5FCIBANYNSX72VNUGT4UXKYI5CEIIHNZGNB63OA, seg_pos={0 0}, upload=148985]: Done uploading piece. err: <nil> Sep 13 14:55:18.323717226 (169h26m54.363290346s) [noise=true , address=77.90.31.11:4559 , piece_id=OYMOQVJRZ6SMMF7O2DXXDEOVRN27B3FT5EIVMFFBHC47UPZUUMIQ, seg_pos={0 0}, upload=148985]: Done uploading piece. err: <nil> Sep 13 14:55:18.769215263 (169h26m54.808788383s) [noise=true , address=45.91.116.14:4221 , piece_id=Q7ZPEITLUERUPOI247KAARIL3GSAPW6JNEUF7FSHQWJHL5GTGIZQ, seg_pos={0 0}, upload=148985]: Done uploading piece. err: <nil> Sep 13 14:56:14.598217910 (169h27m50.63779103s) [noise=true , address=145.79.227.28:4730 , piece_id=5WCQINYSLLWZR3Z3QWWUKRKDPRWYQ6VQPIZ4YDAD47JHSKDSCAGA, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:1SXQt5FugmWj1YdhsvaaR7bDoxTBkqHKZpyXSmQqGdX5ZRhKxV, address:145.79.227.28:4730): protocol: read tcp 188.166.225.170:42820->145.79.227.28:4730: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 13 14:56:14.598245002 (169h27m50.637818122s) [noise=true , address=145.79.227.52:4475 , piece_id=3T3MRZJ2T74J7CB5EJ4N2CIIERKWTQ5KWIOLDNV2LXHYLO5IKHTA, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:12UYjE6PEB9gZKwTKcnx23ykF4jDMJzLQoKmiFiejWew5mirdCj, address:145.79.227.52:4475): protocol: rpc: dial tcp 145.79.227.52:4475: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 13 14:56:14.598273767 (169h27m50.637846887s) [noise=true , address=145.79.227.50:4386 , piece_id=74QVIDRRHFRIA5Z6ZAQ4EWQ2PZ5HZI5JXNJ5WR2DVFJQ4WGE2UPQ, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:19aPjJnya2Q3stEhiedDSth3NHV8phLRj5yY79hZv81UgDitMv, address:145.79.227.50:4386): protocol: write tcp 188.166.225.170:42008->145.79.227.50:4386: use of closed network connection; write tcp 188.166.225.170:42008->145.79.227.50:4386: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42008->145.79.227.50:4386: use of closed network connection Sep 13 14:56:14.598293245 (169h27m50.637866365s) [noise=true , address=145.79.227.5:4250 , piece_id=EPZOZXRFVPSAPD6UKPOCCBOX7VJNB4DQRKV6YTRWMPOLQQ5IH6KQ, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:1uoGjfqBLfoNrg2X1aYKYLKQjgFUnzLjHetCFc9sPjJa2qqKYM, address:145.79.227.5:4250): protocol: write tcp 188.166.225.170:58078->145.79.227.5:4250: use of closed network connection; write tcp 188.166.225.170:58078->145.79.227.5:4250: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58078->145.79.227.5:4250: use of closed network connection Sep 13 14:56:14.598308794 (169h27m50.637881914s) [noise=true , address=145.79.227.33:4894 , piece_id=5KD2BL5J7DT7JDLPDS2QUUMM26AFWSXJ2BDTJSUGZ253ZV7XOVKQ, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:12KWNyruYhmSJVvgUF7fBpGf4H7eYmtAeEXToYjd2dcEPcj1E44, address:145.79.227.33:4894): protocol: write tcp 188.166.225.170:53614->145.79.227.33:4894: use of closed network connection; write tcp 188.166.225.170:53614->145.79.227.33:4894: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53614->145.79.227.33:4894: use of closed network connection Sep 13 14:56:14.598327330 (169h27m50.63790045s) [noise=true , address=145.79.227.7:4308 , piece_id=FYL7PQOKW74EJ2QLKMN6EYJKIT7A6B2VQESVP2VFVDVJHPHXFGAQ, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:1PgA5SZkBvK673LsV7p2sXpbi1r79esNNZspPZTzaitqfbjtyY, address:145.79.227.7:4308): protocol: write tcp 188.166.225.170:37146->145.79.227.7:4308: use of closed network connection; write tcp 188.166.225.170:37146->145.79.227.7:4308: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37146->145.79.227.7:4308: use of closed network connection Sep 13 14:56:14.598358169 (169h27m50.637931289s) [noise=true , address=145.79.227.36:4544 , piece_id=WZRHCCBJDHID4R4UOB76OABJSH2QVC6WZ6OZWIUUMKWWBDVWEG4A, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:12RSxwxw1WXkNcu7b3dEgJhyNkFGYUfKF4QVKmK323L5g3EKFnG, address:145.79.227.36:4544): protocol: rpc: dial tcp 145.79.227.36:4544: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 13 14:56:14.598387545 (169h27m50.637960665s) [noise=true , address=145.79.227.2:4137 , piece_id=NH4RZXYD3LKQZLLP5SSA7HXC7FBBOQV6QLBE2B3R5HH5F23Q7E6A, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:12tkbodtFrNCuJbvZk8YnpSdJfVsKasNuKhg2wGD8rDkNPrHZ21, address:145.79.227.2:4137): protocol: rpc: dial tcp 145.79.227.2:4137: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 13 14:56:14.598421721 (169h27m50.637994841s) [noise=true , address=145.79.227.30:4780 , piece_id=7EMNYDLOMVLIDGE5ZK7WFQCNGBFLZA5AF5EHSLHYE22XJ2MP56VQ, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:125WCbZ6a2NaewiLAHhHuPo91sQEfC9CZzEBV6rwF1JY7z13AEY, address:145.79.227.30:4780): protocol: read tcp 188.166.225.170:43016->145.79.227.30:4780: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 13 14:56:14.598427161 (169h27m50.638000281s) [noise=true , address=145.79.227.19:4090 , piece_id=YZ2H4UUM3Z5ZMCC6LQDAPLHCT7OQQAMRSDBQ3CH6I6UT2HAN4TXQ, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:12qSkJk9HUyQon3UP1AA8mRcqJbNdkKJh6cEUZoHLGNniqDvNqB, address:145.79.227.19:4090): protocol: write tcp 188.166.225.170:37110->145.79.227.19:4090: use of closed network connection; write tcp 188.166.225.170:37110->145.79.227.19:4090: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37110->145.79.227.19:4090: use of closed network connection Sep 13 14:56:14.598456818 (169h27m50.638029938s) [noise=true , address=145.79.227.10:4425 , piece_id=CRES4FQPX4WMTDTUBCPRZ2R4OHTI7MMHS6XSCZ7UXKDLSSLEVQ3Q, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:13sAcGFjAXuesAkUn7NuLaaV8TAah3Mr1kqd33wwh9MTELqmpD, address:145.79.227.10:4425): protocol: read tcp 188.166.225.170:58348->145.79.227.10:4425: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 13 14:56:14.598463220 (169h27m50.63803635s) [noise=true , address=145.79.227.35:4501 , piece_id=UKSF2KK675OI35QE6QZTH7QAJQ4SD7E42ZPMY2JIANOHGEFF44UQ, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:1hmsSAiSp1CBFqKhVE2S3sFfqYYkGVAiV1GfrT4Rd7hBaFNPpT, address:145.79.227.35:4501): protocol: rpc: dial tcp 145.79.227.35:4501: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 13 14:56:14.598484230 (169h27m50.63805735s) [noise=true , address=145.79.227.14:4867 , piece_id=7BW6OXJYXIXQSQK3DMHSGQ56ANOAOEGXEHD4EWCOZDXTTIL37SAA, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:12Sz7PyMo81CmAAkiCwpZ3M3JaM9HMHJLwAAM5M5kv6nnKktpKE, address:145.79.227.14:4867): protocol: read tcp 188.166.225.170:60252->145.79.227.14:4867: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 13 14:56:14.598492155 (169h27m50.638065275s) [noise=true , address=145.79.227.18:4018 , piece_id=72FTA6TNIQAVS2NDA5XOR4J2ZEEBWZ7LLGT4EAQ2EN2YB3VXG5JA, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:1bFRHbXjTaxoEKc5gbQWYt6jwUg8jknScBZpc8PdKNSZhYqUqP, address:145.79.227.18:4018): protocol: read tcp 188.166.225.170:60682->145.79.227.18:4018: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 13 14:56:14.598496193 (169h27m50.638069313s) [noise=true , address=145.79.227.47:4286 , piece_id=D2PSYV2IN5MFZGGF2U47EPRTCNDIMHB74GR3UFQ64AJN37FE7BVA, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:12v2vJnE92VrJXQrbxnAxANHCVASuXiBKMUZECMVmEWEjsZRtsQ, address:145.79.227.47:4286): protocol: read tcp 188.166.225.170:33092->145.79.227.47:4286: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 13 14:56:14.598521301 (169h27m50.638094421s) [seg_pos={0 0}, upload=148985]: Done waiting for segment. successful: 3 optimal: 14 errs: <nil> Sep 13 14:56:14.862062924 (169h27m50.901636044s) [upload=148985]: Done uploading segments... Sep 13 14:56:14.862090337 (169h27m50.901663457s) [upload=148985]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 3) Duration: 13.468093311s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [.................X.X...X..X.............X..X...X.X....X.X...] duration: 6.47111223s FAIL (2025-09-13 15:04:08 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.471138631s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 8.72134933s FAIL (2025-09-13 15:04:00 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 8.721380729s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [..................X.X...X..X.............X..X...X.X....X.X..] duration: 6.704082335s FAIL (2025-09-13 15:03:49 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.704110128s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.............................................X...X..........] duration: 14.681791843s FAIL (2025-09-13 15:03:40 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 3) ======LOGS====== 6OXJYXIXQSQK3DMHSGQ56ANOAOEGXEHD4EWCOZDXTTIL37SAA, seg_pos={0 0}, upload=148985]: Uploading piece... Sep 13 14:55:15.907845810 (169h26m51.94741893s) [noise=true , address=145.79.227.52:4475 , piece_id=3T3MRZJ2T74J7CB5EJ4N2CIIERKWTQ5KWIOLDNV2LXHYLO5IKHTA, seg_pos={0 0}, upload=148985]: Uploading piece... Sep 13 14:55:15.908062555 (169h26m51.947635675s) [noise=true , address=145.79.227.5:4250 , piece_id=EPZOZXRFVPSAPD6UKPOCCBOX7VJNB4DQRKV6YTRWMPOLQQ5IH6KQ, seg_pos={0 0}, upload=148985]: Uploading piece... Sep 13 14:55:15.908276674 (169h26m51.947849794s) [noise=true , address=145.79.227.30:4780 , piece_id=7EMNYDLOMVLIDGE5ZK7WFQCNGBFLZA5AF5EHSLHYE22XJ2MP56VQ, seg_pos={0 0}, upload=148985]: Uploading piece... Sep 13 14:55:15.908522154 (169h26m51.948095274s) [noise=true , address=77.90.31.17:4793 , piece_id=PWVTHYWQY6GFO5FCIBANYNSX72VNUGT4UXKYI5CEIIHNZGNB63OA, seg_pos={0 0}, upload=148985]: Uploading piece... Sep 13 14:55:17.522666509 (169h26m53.562239629s) [upload=148985]: Next returned nil segment Sep 13 14:55:17.522686247 (169h26m53.562259367s) [upload=148985]: Waiting for error group managing segments... Sep 13 14:55:18.048737965 (169h26m54.088311085s) [noise=true , address=77.90.31.17:4793 , piece_id=PWVTHYWQY6GFO5FCIBANYNSX72VNUGT4UXKYI5CEIIHNZGNB63OA, seg_pos={0 0}, upload=148985]: Done uploading piece. err: <nil> Sep 13 14:55:18.323717226 (169h26m54.363290346s) [noise=true , address=77.90.31.11:4559 , piece_id=OYMOQVJRZ6SMMF7O2DXXDEOVRN27B3FT5EIVMFFBHC47UPZUUMIQ, seg_pos={0 0}, upload=148985]: Done uploading piece. err: <nil> Sep 13 14:55:18.769215263 (169h26m54.808788383s) [noise=true , address=45.91.116.14:4221 , piece_id=Q7ZPEITLUERUPOI247KAARIL3GSAPW6JNEUF7FSHQWJHL5GTGIZQ, seg_pos={0 0}, upload=148985]: Done uploading piece. err: <nil> Sep 13 14:56:14.598217910 (169h27m50.63779103s) [noise=true , address=145.79.227.28:4730 , piece_id=5WCQINYSLLWZR3Z3QWWUKRKDPRWYQ6VQPIZ4YDAD47JHSKDSCAGA, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:1SXQt5FugmWj1YdhsvaaR7bDoxTBkqHKZpyXSmQqGdX5ZRhKxV, address:145.79.227.28:4730): protocol: read tcp 188.166.225.170:42820->145.79.227.28:4730: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 13 14:56:14.598245002 (169h27m50.637818122s) [noise=true , address=145.79.227.52:4475 , piece_id=3T3MRZJ2T74J7CB5EJ4N2CIIERKWTQ5KWIOLDNV2LXHYLO5IKHTA, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:12UYjE6PEB9gZKwTKcnx23ykF4jDMJzLQoKmiFiejWew5mirdCj, address:145.79.227.52:4475): protocol: rpc: dial tcp 145.79.227.52:4475: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 13 14:56:14.598273767 (169h27m50.637846887s) [noise=true , address=145.79.227.50:4386 , piece_id=74QVIDRRHFRIA5Z6ZAQ4EWQ2PZ5HZI5JXNJ5WR2DVFJQ4WGE2UPQ, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:19aPjJnya2Q3stEhiedDSth3NHV8phLRj5yY79hZv81UgDitMv, address:145.79.227.50:4386): protocol: write tcp 188.166.225.170:42008->145.79.227.50:4386: use of closed network connection; write tcp 188.166.225.170:42008->145.79.227.50:4386: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42008->145.79.227.50:4386: use of closed network connection Sep 13 14:56:14.598293245 (169h27m50.637866365s) [noise=true , address=145.79.227.5:4250 , piece_id=EPZOZXRFVPSAPD6UKPOCCBOX7VJNB4DQRKV6YTRWMPOLQQ5IH6KQ, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:1uoGjfqBLfoNrg2X1aYKYLKQjgFUnzLjHetCFc9sPjJa2qqKYM, address:145.79.227.5:4250): protocol: write tcp 188.166.225.170:58078->145.79.227.5:4250: use of closed network connection; write tcp 188.166.225.170:58078->145.79.227.5:4250: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58078->145.79.227.5:4250: use of closed network connection Sep 13 14:56:14.598308794 (169h27m50.637881914s) [noise=true , address=145.79.227.33:4894 , piece_id=5KD2BL5J7DT7JDLPDS2QUUMM26AFWSXJ2BDTJSUGZ253ZV7XOVKQ, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:12KWNyruYhmSJVvgUF7fBpGf4H7eYmtAeEXToYjd2dcEPcj1E44, address:145.79.227.33:4894): protocol: write tcp 188.166.225.170:53614->145.79.227.33:4894: use of closed network connection; write tcp 188.166.225.170:53614->145.79.227.33:4894: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53614->145.79.227.33:4894: use of closed network connection Sep 13 14:56:14.598327330 (169h27m50.63790045s) [noise=true , address=145.79.227.7:4308 , piece_id=FYL7PQOKW74EJ2QLKMN6EYJKIT7A6B2VQESVP2VFVDVJHPHXFGAQ, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:1PgA5SZkBvK673LsV7p2sXpbi1r79esNNZspPZTzaitqfbjtyY, address:145.79.227.7:4308): protocol: write tcp 188.166.225.170:37146->145.79.227.7:4308: use of closed network connection; write tcp 188.166.225.170:37146->145.79.227.7:4308: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37146->145.79.227.7:4308: use of closed network connection Sep 13 14:56:14.598358169 (169h27m50.637931289s) [noise=true , address=145.79.227.36:4544 , piece_id=WZRHCCBJDHID4R4UOB76OABJSH2QVC6WZ6OZWIUUMKWWBDVWEG4A, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:12RSxwxw1WXkNcu7b3dEgJhyNkFGYUfKF4QVKmK323L5g3EKFnG, address:145.79.227.36:4544): protocol: rpc: dial tcp 145.79.227.36:4544: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 13 14:56:14.598387545 (169h27m50.637960665s) [noise=true , address=145.79.227.2:4137 , piece_id=NH4RZXYD3LKQZLLP5SSA7HXC7FBBOQV6QLBE2B3R5HH5F23Q7E6A, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:12tkbodtFrNCuJbvZk8YnpSdJfVsKasNuKhg2wGD8rDkNPrHZ21, address:145.79.227.2:4137): protocol: rpc: dial tcp 145.79.227.2:4137: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 13 14:56:14.598421721 (169h27m50.637994841s) [noise=true , address=145.79.227.30:4780 , piece_id=7EMNYDLOMVLIDGE5ZK7WFQCNGBFLZA5AF5EHSLHYE22XJ2MP56VQ, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:125WCbZ6a2NaewiLAHhHuPo91sQEfC9CZzEBV6rwF1JY7z13AEY, address:145.79.227.30:4780): protocol: read tcp 188.166.225.170:43016->145.79.227.30:4780: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 13 14:56:14.598427161 (169h27m50.638000281s) [noise=true , address=145.79.227.19:4090 , piece_id=YZ2H4UUM3Z5ZMCC6LQDAPLHCT7OQQAMRSDBQ3CH6I6UT2HAN4TXQ, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:12qSkJk9HUyQon3UP1AA8mRcqJbNdkKJh6cEUZoHLGNniqDvNqB, address:145.79.227.19:4090): protocol: write tcp 188.166.225.170:37110->145.79.227.19:4090: use of closed network connection; write tcp 188.166.225.170:37110->145.79.227.19:4090: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37110->145.79.227.19:4090: use of closed network connection Sep 13 14:56:14.598456818 (169h27m50.638029938s) [noise=true , address=145.79.227.10:4425 , piece_id=CRES4FQPX4WMTDTUBCPRZ2R4OHTI7MMHS6XSCZ7UXKDLSSLEVQ3Q, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:13sAcGFjAXuesAkUn7NuLaaV8TAah3Mr1kqd33wwh9MTELqmpD, address:145.79.227.10:4425): protocol: read tcp 188.166.225.170:58348->145.79.227.10:4425: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 13 14:56:14.598463220 (169h27m50.63803635s) [noise=true , address=145.79.227.35:4501 , piece_id=UKSF2KK675OI35QE6QZTH7QAJQ4SD7E42ZPMY2JIANOHGEFF44UQ, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:1hmsSAiSp1CBFqKhVE2S3sFfqYYkGVAiV1GfrT4Rd7hBaFNPpT, address:145.79.227.35:4501): protocol: rpc: dial tcp 145.79.227.35:4501: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 13 14:56:14.598484230 (169h27m50.63805735s) [noise=true , address=145.79.227.14:4867 , piece_id=7BW6OXJYXIXQSQK3DMHSGQ56ANOAOEGXEHD4EWCOZDXTTIL37SAA, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:12Sz7PyMo81CmAAkiCwpZ3M3JaM9HMHJLwAAM5M5kv6nnKktpKE, address:145.79.227.14:4867): protocol: read tcp 188.166.225.170:60252->145.79.227.14:4867: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 13 14:56:14.598492155 (169h27m50.638065275s) [noise=true , address=145.79.227.18:4018 , piece_id=72FTA6TNIQAVS2NDA5XOR4J2ZEEBWZ7LLGT4EAQ2EN2YB3VXG5JA, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:1bFRHbXjTaxoEKc5gbQWYt6jwUg8jknScBZpc8PdKNSZhYqUqP, address:145.79.227.18:4018): protocol: read tcp 188.166.225.170:60682->145.79.227.18:4018: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 13 14:56:14.598496193 (169h27m50.638069313s) [noise=true , address=145.79.227.47:4286 , piece_id=D2PSYV2IN5MFZGGF2U47EPRTCNDIMHB74GR3UFQ64AJN37FE7BVA, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:12v2vJnE92VrJXQrbxnAxANHCVASuXiBKMUZECMVmEWEjsZRtsQ, address:145.79.227.47:4286): protocol: read tcp 188.166.225.170:33092->145.79.227.47:4286: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 13 14:56:14.598521301 (169h27m50.638094421s) [seg_pos={0 0}, upload=148985]: Done waiting for segment. successful: 3 optimal: 14 errs: <nil> Sep 13 14:56:14.862062924 (169h27m50.901636044s) [upload=148985]: Done uploading segments... Sep 13 14:56:14.862090337 (169h27m50.901663457s) [upload=148985]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 3) Duration: 14.681826539s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 7.953711768s FAIL (2025-09-13 15:03:03 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 7.953731707s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [..............................................X...X.........] duration: 13.529089359s FAIL (2025-09-13 15:02:41 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 3) ======LOGS====== 6OXJYXIXQSQK3DMHSGQ56ANOAOEGXEHD4EWCOZDXTTIL37SAA, seg_pos={0 0}, upload=148985]: Uploading piece... Sep 13 14:55:15.907845810 (169h26m51.94741893s) [noise=true , address=145.79.227.52:4475 , piece_id=3T3MRZJ2T74J7CB5EJ4N2CIIERKWTQ5KWIOLDNV2LXHYLO5IKHTA, seg_pos={0 0}, upload=148985]: Uploading piece... Sep 13 14:55:15.908062555 (169h26m51.947635675s) [noise=true , address=145.79.227.5:4250 , piece_id=EPZOZXRFVPSAPD6UKPOCCBOX7VJNB4DQRKV6YTRWMPOLQQ5IH6KQ, seg_pos={0 0}, upload=148985]: Uploading piece... Sep 13 14:55:15.908276674 (169h26m51.947849794s) [noise=true , address=145.79.227.30:4780 , piece_id=7EMNYDLOMVLIDGE5ZK7WFQCNGBFLZA5AF5EHSLHYE22XJ2MP56VQ, seg_pos={0 0}, upload=148985]: Uploading piece... Sep 13 14:55:15.908522154 (169h26m51.948095274s) [noise=true , address=77.90.31.17:4793 , piece_id=PWVTHYWQY6GFO5FCIBANYNSX72VNUGT4UXKYI5CEIIHNZGNB63OA, seg_pos={0 0}, upload=148985]: Uploading piece... Sep 13 14:55:17.522666509 (169h26m53.562239629s) [upload=148985]: Next returned nil segment Sep 13 14:55:17.522686247 (169h26m53.562259367s) [upload=148985]: Waiting for error group managing segments... Sep 13 14:55:18.048737965 (169h26m54.088311085s) [noise=true , address=77.90.31.17:4793 , piece_id=PWVTHYWQY6GFO5FCIBANYNSX72VNUGT4UXKYI5CEIIHNZGNB63OA, seg_pos={0 0}, upload=148985]: Done uploading piece. err: <nil> Sep 13 14:55:18.323717226 (169h26m54.363290346s) [noise=true , address=77.90.31.11:4559 , piece_id=OYMOQVJRZ6SMMF7O2DXXDEOVRN27B3FT5EIVMFFBHC47UPZUUMIQ, seg_pos={0 0}, upload=148985]: Done uploading piece. err: <nil> Sep 13 14:55:18.769215263 (169h26m54.808788383s) [noise=true , address=45.91.116.14:4221 , piece_id=Q7ZPEITLUERUPOI247KAARIL3GSAPW6JNEUF7FSHQWJHL5GTGIZQ, seg_pos={0 0}, upload=148985]: Done uploading piece. err: <nil> Sep 13 14:56:14.598217910 (169h27m50.63779103s) [noise=true , address=145.79.227.28:4730 , piece_id=5WCQINYSLLWZR3Z3QWWUKRKDPRWYQ6VQPIZ4YDAD47JHSKDSCAGA, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:1SXQt5FugmWj1YdhsvaaR7bDoxTBkqHKZpyXSmQqGdX5ZRhKxV, address:145.79.227.28:4730): protocol: read tcp 188.166.225.170:42820->145.79.227.28:4730: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 13 14:56:14.598245002 (169h27m50.637818122s) [noise=true , address=145.79.227.52:4475 , piece_id=3T3MRZJ2T74J7CB5EJ4N2CIIERKWTQ5KWIOLDNV2LXHYLO5IKHTA, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:12UYjE6PEB9gZKwTKcnx23ykF4jDMJzLQoKmiFiejWew5mirdCj, address:145.79.227.52:4475): protocol: rpc: dial tcp 145.79.227.52:4475: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 13 14:56:14.598273767 (169h27m50.637846887s) [noise=true , address=145.79.227.50:4386 , piece_id=74QVIDRRHFRIA5Z6ZAQ4EWQ2PZ5HZI5JXNJ5WR2DVFJQ4WGE2UPQ, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:19aPjJnya2Q3stEhiedDSth3NHV8phLRj5yY79hZv81UgDitMv, address:145.79.227.50:4386): protocol: write tcp 188.166.225.170:42008->145.79.227.50:4386: use of closed network connection; write tcp 188.166.225.170:42008->145.79.227.50:4386: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42008->145.79.227.50:4386: use of closed network connection Sep 13 14:56:14.598293245 (169h27m50.637866365s) [noise=true , address=145.79.227.5:4250 , piece_id=EPZOZXRFVPSAPD6UKPOCCBOX7VJNB4DQRKV6YTRWMPOLQQ5IH6KQ, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:1uoGjfqBLfoNrg2X1aYKYLKQjgFUnzLjHetCFc9sPjJa2qqKYM, address:145.79.227.5:4250): protocol: write tcp 188.166.225.170:58078->145.79.227.5:4250: use of closed network connection; write tcp 188.166.225.170:58078->145.79.227.5:4250: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58078->145.79.227.5:4250: use of closed network connection Sep 13 14:56:14.598308794 (169h27m50.637881914s) [noise=true , address=145.79.227.33:4894 , piece_id=5KD2BL5J7DT7JDLPDS2QUUMM26AFWSXJ2BDTJSUGZ253ZV7XOVKQ, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:12KWNyruYhmSJVvgUF7fBpGf4H7eYmtAeEXToYjd2dcEPcj1E44, address:145.79.227.33:4894): protocol: write tcp 188.166.225.170:53614->145.79.227.33:4894: use of closed network connection; write tcp 188.166.225.170:53614->145.79.227.33:4894: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53614->145.79.227.33:4894: use of closed network connection Sep 13 14:56:14.598327330 (169h27m50.63790045s) [noise=true , address=145.79.227.7:4308 , piece_id=FYL7PQOKW74EJ2QLKMN6EYJKIT7A6B2VQESVP2VFVDVJHPHXFGAQ, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:1PgA5SZkBvK673LsV7p2sXpbi1r79esNNZspPZTzaitqfbjtyY, address:145.79.227.7:4308): protocol: write tcp 188.166.225.170:37146->145.79.227.7:4308: use of closed network connection; write tcp 188.166.225.170:37146->145.79.227.7:4308: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37146->145.79.227.7:4308: use of closed network connection Sep 13 14:56:14.598358169 (169h27m50.637931289s) [noise=true , address=145.79.227.36:4544 , piece_id=WZRHCCBJDHID4R4UOB76OABJSH2QVC6WZ6OZWIUUMKWWBDVWEG4A, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:12RSxwxw1WXkNcu7b3dEgJhyNkFGYUfKF4QVKmK323L5g3EKFnG, address:145.79.227.36:4544): protocol: rpc: dial tcp 145.79.227.36:4544: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 13 14:56:14.598387545 (169h27m50.637960665s) [noise=true , address=145.79.227.2:4137 , piece_id=NH4RZXYD3LKQZLLP5SSA7HXC7FBBOQV6QLBE2B3R5HH5F23Q7E6A, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:12tkbodtFrNCuJbvZk8YnpSdJfVsKasNuKhg2wGD8rDkNPrHZ21, address:145.79.227.2:4137): protocol: rpc: dial tcp 145.79.227.2:4137: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 13 14:56:14.598421721 (169h27m50.637994841s) [noise=true , address=145.79.227.30:4780 , piece_id=7EMNYDLOMVLIDGE5ZK7WFQCNGBFLZA5AF5EHSLHYE22XJ2MP56VQ, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:125WCbZ6a2NaewiLAHhHuPo91sQEfC9CZzEBV6rwF1JY7z13AEY, address:145.79.227.30:4780): protocol: read tcp 188.166.225.170:43016->145.79.227.30:4780: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 13 14:56:14.598427161 (169h27m50.638000281s) [noise=true , address=145.79.227.19:4090 , piece_id=YZ2H4UUM3Z5ZMCC6LQDAPLHCT7OQQAMRSDBQ3CH6I6UT2HAN4TXQ, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:12qSkJk9HUyQon3UP1AA8mRcqJbNdkKJh6cEUZoHLGNniqDvNqB, address:145.79.227.19:4090): protocol: write tcp 188.166.225.170:37110->145.79.227.19:4090: use of closed network connection; write tcp 188.166.225.170:37110->145.79.227.19:4090: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37110->145.79.227.19:4090: use of closed network connection Sep 13 14:56:14.598456818 (169h27m50.638029938s) [noise=true , address=145.79.227.10:4425 , piece_id=CRES4FQPX4WMTDTUBCPRZ2R4OHTI7MMHS6XSCZ7UXKDLSSLEVQ3Q, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:13sAcGFjAXuesAkUn7NuLaaV8TAah3Mr1kqd33wwh9MTELqmpD, address:145.79.227.10:4425): protocol: read tcp 188.166.225.170:58348->145.79.227.10:4425: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 13 14:56:14.598463220 (169h27m50.63803635s) [noise=true , address=145.79.227.35:4501 , piece_id=UKSF2KK675OI35QE6QZTH7QAJQ4SD7E42ZPMY2JIANOHGEFF44UQ, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:1hmsSAiSp1CBFqKhVE2S3sFfqYYkGVAiV1GfrT4Rd7hBaFNPpT, address:145.79.227.35:4501): protocol: rpc: dial tcp 145.79.227.35:4501: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 13 14:56:14.598484230 (169h27m50.63805735s) [noise=true , address=145.79.227.14:4867 , piece_id=7BW6OXJYXIXQSQK3DMHSGQ56ANOAOEGXEHD4EWCOZDXTTIL37SAA, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:12Sz7PyMo81CmAAkiCwpZ3M3JaM9HMHJLwAAM5M5kv6nnKktpKE, address:145.79.227.14:4867): protocol: read tcp 188.166.225.170:60252->145.79.227.14:4867: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 13 14:56:14.598492155 (169h27m50.638065275s) [noise=true , address=145.79.227.18:4018 , piece_id=72FTA6TNIQAVS2NDA5XOR4J2ZEEBWZ7LLGT4EAQ2EN2YB3VXG5JA, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:1bFRHbXjTaxoEKc5gbQWYt6jwUg8jknScBZpc8PdKNSZhYqUqP, address:145.79.227.18:4018): protocol: read tcp 188.166.225.170:60682->145.79.227.18:4018: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 13 14:56:14.598496193 (169h27m50.638069313s) [noise=true , address=145.79.227.47:4286 , piece_id=D2PSYV2IN5MFZGGF2U47EPRTCNDIMHB74GR3UFQ64AJN37FE7BVA, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:12v2vJnE92VrJXQrbxnAxANHCVASuXiBKMUZECMVmEWEjsZRtsQ, address:145.79.227.47:4286): protocol: read tcp 188.166.225.170:33092->145.79.227.47:4286: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 13 14:56:14.598521301 (169h27m50.638094421s) [seg_pos={0 0}, upload=148985]: Done waiting for segment. successful: 3 optimal: 14 errs: <nil> Sep 13 14:56:14.862062924 (169h27m50.901636044s) [upload=148985]: Done uploading segments... Sep 13 14:56:14.862090337 (169h27m50.901663457s) [upload=148985]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 3) Duration: 13.529137772s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [...................X.X...X..X.............X..X...X.X....X.X.] duration: 6.893979281s FAIL (2025-09-13 15:02:38 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.894000542s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [....................X.X...X..X.............X..X...X.X....X.X] duration: 30.001580696s FAIL (2025-09-13 15:02:26 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.001594422s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [...............................................X...X........] duration: 11.21049539s FAIL (2025-09-13 15:02:13 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 3) ======LOGS====== 6OXJYXIXQSQK3DMHSGQ56ANOAOEGXEHD4EWCOZDXTTIL37SAA, seg_pos={0 0}, upload=148985]: Uploading piece... Sep 13 14:55:15.907845810 (169h26m51.94741893s) [noise=true , address=145.79.227.52:4475 , piece_id=3T3MRZJ2T74J7CB5EJ4N2CIIERKWTQ5KWIOLDNV2LXHYLO5IKHTA, seg_pos={0 0}, upload=148985]: Uploading piece... Sep 13 14:55:15.908062555 (169h26m51.947635675s) [noise=true , address=145.79.227.5:4250 , piece_id=EPZOZXRFVPSAPD6UKPOCCBOX7VJNB4DQRKV6YTRWMPOLQQ5IH6KQ, seg_pos={0 0}, upload=148985]: Uploading piece... Sep 13 14:55:15.908276674 (169h26m51.947849794s) [noise=true , address=145.79.227.30:4780 , piece_id=7EMNYDLOMVLIDGE5ZK7WFQCNGBFLZA5AF5EHSLHYE22XJ2MP56VQ, seg_pos={0 0}, upload=148985]: Uploading piece... Sep 13 14:55:15.908522154 (169h26m51.948095274s) [noise=true , address=77.90.31.17:4793 , piece_id=PWVTHYWQY6GFO5FCIBANYNSX72VNUGT4UXKYI5CEIIHNZGNB63OA, seg_pos={0 0}, upload=148985]: Uploading piece... Sep 13 14:55:17.522666509 (169h26m53.562239629s) [upload=148985]: Next returned nil segment Sep 13 14:55:17.522686247 (169h26m53.562259367s) [upload=148985]: Waiting for error group managing segments... Sep 13 14:55:18.048737965 (169h26m54.088311085s) [noise=true , address=77.90.31.17:4793 , piece_id=PWVTHYWQY6GFO5FCIBANYNSX72VNUGT4UXKYI5CEIIHNZGNB63OA, seg_pos={0 0}, upload=148985]: Done uploading piece. err: <nil> Sep 13 14:55:18.323717226 (169h26m54.363290346s) [noise=true , address=77.90.31.11:4559 , piece_id=OYMOQVJRZ6SMMF7O2DXXDEOVRN27B3FT5EIVMFFBHC47UPZUUMIQ, seg_pos={0 0}, upload=148985]: Done uploading piece. err: <nil> Sep 13 14:55:18.769215263 (169h26m54.808788383s) [noise=true , address=45.91.116.14:4221 , piece_id=Q7ZPEITLUERUPOI247KAARIL3GSAPW6JNEUF7FSHQWJHL5GTGIZQ, seg_pos={0 0}, upload=148985]: Done uploading piece. err: <nil> Sep 13 14:56:14.598217910 (169h27m50.63779103s) [noise=true , address=145.79.227.28:4730 , piece_id=5WCQINYSLLWZR3Z3QWWUKRKDPRWYQ6VQPIZ4YDAD47JHSKDSCAGA, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:1SXQt5FugmWj1YdhsvaaR7bDoxTBkqHKZpyXSmQqGdX5ZRhKxV, address:145.79.227.28:4730): protocol: read tcp 188.166.225.170:42820->145.79.227.28:4730: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 13 14:56:14.598245002 (169h27m50.637818122s) [noise=true , address=145.79.227.52:4475 , piece_id=3T3MRZJ2T74J7CB5EJ4N2CIIERKWTQ5KWIOLDNV2LXHYLO5IKHTA, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:12UYjE6PEB9gZKwTKcnx23ykF4jDMJzLQoKmiFiejWew5mirdCj, address:145.79.227.52:4475): protocol: rpc: dial tcp 145.79.227.52:4475: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 13 14:56:14.598273767 (169h27m50.637846887s) [noise=true , address=145.79.227.50:4386 , piece_id=74QVIDRRHFRIA5Z6ZAQ4EWQ2PZ5HZI5JXNJ5WR2DVFJQ4WGE2UPQ, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:19aPjJnya2Q3stEhiedDSth3NHV8phLRj5yY79hZv81UgDitMv, address:145.79.227.50:4386): protocol: write tcp 188.166.225.170:42008->145.79.227.50:4386: use of closed network connection; write tcp 188.166.225.170:42008->145.79.227.50:4386: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42008->145.79.227.50:4386: use of closed network connection Sep 13 14:56:14.598293245 (169h27m50.637866365s) [noise=true , address=145.79.227.5:4250 , piece_id=EPZOZXRFVPSAPD6UKPOCCBOX7VJNB4DQRKV6YTRWMPOLQQ5IH6KQ, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:1uoGjfqBLfoNrg2X1aYKYLKQjgFUnzLjHetCFc9sPjJa2qqKYM, address:145.79.227.5:4250): protocol: write tcp 188.166.225.170:58078->145.79.227.5:4250: use of closed network connection; write tcp 188.166.225.170:58078->145.79.227.5:4250: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58078->145.79.227.5:4250: use of closed network connection Sep 13 14:56:14.598308794 (169h27m50.637881914s) [noise=true , address=145.79.227.33:4894 , piece_id=5KD2BL5J7DT7JDLPDS2QUUMM26AFWSXJ2BDTJSUGZ253ZV7XOVKQ, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:12KWNyruYhmSJVvgUF7fBpGf4H7eYmtAeEXToYjd2dcEPcj1E44, address:145.79.227.33:4894): protocol: write tcp 188.166.225.170:53614->145.79.227.33:4894: use of closed network connection; write tcp 188.166.225.170:53614->145.79.227.33:4894: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53614->145.79.227.33:4894: use of closed network connection Sep 13 14:56:14.598327330 (169h27m50.63790045s) [noise=true , address=145.79.227.7:4308 , piece_id=FYL7PQOKW74EJ2QLKMN6EYJKIT7A6B2VQESVP2VFVDVJHPHXFGAQ, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:1PgA5SZkBvK673LsV7p2sXpbi1r79esNNZspPZTzaitqfbjtyY, address:145.79.227.7:4308): protocol: write tcp 188.166.225.170:37146->145.79.227.7:4308: use of closed network connection; write tcp 188.166.225.170:37146->145.79.227.7:4308: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37146->145.79.227.7:4308: use of closed network connection Sep 13 14:56:14.598358169 (169h27m50.637931289s) [noise=true , address=145.79.227.36:4544 , piece_id=WZRHCCBJDHID4R4UOB76OABJSH2QVC6WZ6OZWIUUMKWWBDVWEG4A, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:12RSxwxw1WXkNcu7b3dEgJhyNkFGYUfKF4QVKmK323L5g3EKFnG, address:145.79.227.36:4544): protocol: rpc: dial tcp 145.79.227.36:4544: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 13 14:56:14.598387545 (169h27m50.637960665s) [noise=true , address=145.79.227.2:4137 , piece_id=NH4RZXYD3LKQZLLP5SSA7HXC7FBBOQV6QLBE2B3R5HH5F23Q7E6A, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:12tkbodtFrNCuJbvZk8YnpSdJfVsKasNuKhg2wGD8rDkNPrHZ21, address:145.79.227.2:4137): protocol: rpc: dial tcp 145.79.227.2:4137: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 13 14:56:14.598421721 (169h27m50.637994841s) [noise=true , address=145.79.227.30:4780 , piece_id=7EMNYDLOMVLIDGE5ZK7WFQCNGBFLZA5AF5EHSLHYE22XJ2MP56VQ, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:125WCbZ6a2NaewiLAHhHuPo91sQEfC9CZzEBV6rwF1JY7z13AEY, address:145.79.227.30:4780): protocol: read tcp 188.166.225.170:43016->145.79.227.30:4780: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 13 14:56:14.598427161 (169h27m50.638000281s) [noise=true , address=145.79.227.19:4090 , piece_id=YZ2H4UUM3Z5ZMCC6LQDAPLHCT7OQQAMRSDBQ3CH6I6UT2HAN4TXQ, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:12qSkJk9HUyQon3UP1AA8mRcqJbNdkKJh6cEUZoHLGNniqDvNqB, address:145.79.227.19:4090): protocol: write tcp 188.166.225.170:37110->145.79.227.19:4090: use of closed network connection; write tcp 188.166.225.170:37110->145.79.227.19:4090: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37110->145.79.227.19:4090: use of closed network connection Sep 13 14:56:14.598456818 (169h27m50.638029938s) [noise=true , address=145.79.227.10:4425 , piece_id=CRES4FQPX4WMTDTUBCPRZ2R4OHTI7MMHS6XSCZ7UXKDLSSLEVQ3Q, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:13sAcGFjAXuesAkUn7NuLaaV8TAah3Mr1kqd33wwh9MTELqmpD, address:145.79.227.10:4425): protocol: read tcp 188.166.225.170:58348->145.79.227.10:4425: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 13 14:56:14.598463220 (169h27m50.63803635s) [noise=true , address=145.79.227.35:4501 , piece_id=UKSF2KK675OI35QE6QZTH7QAJQ4SD7E42ZPMY2JIANOHGEFF44UQ, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:1hmsSAiSp1CBFqKhVE2S3sFfqYYkGVAiV1GfrT4Rd7hBaFNPpT, address:145.79.227.35:4501): protocol: rpc: dial tcp 145.79.227.35:4501: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 13 14:56:14.598484230 (169h27m50.63805735s) [noise=true , address=145.79.227.14:4867 , piece_id=7BW6OXJYXIXQSQK3DMHSGQ56ANOAOEGXEHD4EWCOZDXTTIL37SAA, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:12Sz7PyMo81CmAAkiCwpZ3M3JaM9HMHJLwAAM5M5kv6nnKktpKE, address:145.79.227.14:4867): protocol: read tcp 188.166.225.170:60252->145.79.227.14:4867: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 13 14:56:14.598492155 (169h27m50.638065275s) [noise=true , address=145.79.227.18:4018 , piece_id=72FTA6TNIQAVS2NDA5XOR4J2ZEEBWZ7LLGT4EAQ2EN2YB3VXG5JA, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:1bFRHbXjTaxoEKc5gbQWYt6jwUg8jknScBZpc8PdKNSZhYqUqP, address:145.79.227.18:4018): protocol: read tcp 188.166.225.170:60682->145.79.227.18:4018: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 13 14:56:14.598496193 (169h27m50.638069313s) [noise=true , address=145.79.227.47:4286 , piece_id=D2PSYV2IN5MFZGGF2U47EPRTCNDIMHB74GR3UFQ64AJN37FE7BVA, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:12v2vJnE92VrJXQrbxnAxANHCVASuXiBKMUZECMVmEWEjsZRtsQ, address:145.79.227.47:4286): protocol: read tcp 188.166.225.170:33092->145.79.227.47:4286: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 13 14:56:14.598521301 (169h27m50.638094421s) [seg_pos={0 0}, upload=148985]: Done waiting for segment. successful: 3 optimal: 14 errs: <nil> Sep 13 14:56:14.862062924 (169h27m50.901636044s) [upload=148985]: Done uploading segments... Sep 13 14:56:14.862090337 (169h27m50.901663457s) [upload=148985]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 3) Duration: 11.210601134s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 7.625901558s FAIL (2025-09-13 15:02:07 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 7.625923449s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [X....................X.X...X..X.............X..X...X.X....X.] duration: 7.123100718s FAIL (2025-09-13 15:01:37 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 7.1231286s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 8.296704486s FAIL (2025-09-13 15:01:18 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 8.296728113s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [................................................X...X.......] duration: 10.190544969s FAIL (2025-09-13 15:01:12 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 3) ======LOGS====== 6OXJYXIXQSQK3DMHSGQ56ANOAOEGXEHD4EWCOZDXTTIL37SAA, seg_pos={0 0}, upload=148985]: Uploading piece... Sep 13 14:55:15.907845810 (169h26m51.94741893s) [noise=true , address=145.79.227.52:4475 , piece_id=3T3MRZJ2T74J7CB5EJ4N2CIIERKWTQ5KWIOLDNV2LXHYLO5IKHTA, seg_pos={0 0}, upload=148985]: Uploading piece... Sep 13 14:55:15.908062555 (169h26m51.947635675s) [noise=true , address=145.79.227.5:4250 , piece_id=EPZOZXRFVPSAPD6UKPOCCBOX7VJNB4DQRKV6YTRWMPOLQQ5IH6KQ, seg_pos={0 0}, upload=148985]: Uploading piece... Sep 13 14:55:15.908276674 (169h26m51.947849794s) [noise=true , address=145.79.227.30:4780 , piece_id=7EMNYDLOMVLIDGE5ZK7WFQCNGBFLZA5AF5EHSLHYE22XJ2MP56VQ, seg_pos={0 0}, upload=148985]: Uploading piece... Sep 13 14:55:15.908522154 (169h26m51.948095274s) [noise=true , address=77.90.31.17:4793 , piece_id=PWVTHYWQY6GFO5FCIBANYNSX72VNUGT4UXKYI5CEIIHNZGNB63OA, seg_pos={0 0}, upload=148985]: Uploading piece... Sep 13 14:55:17.522666509 (169h26m53.562239629s) [upload=148985]: Next returned nil segment Sep 13 14:55:17.522686247 (169h26m53.562259367s) [upload=148985]: Waiting for error group managing segments... Sep 13 14:55:18.048737965 (169h26m54.088311085s) [noise=true , address=77.90.31.17:4793 , piece_id=PWVTHYWQY6GFO5FCIBANYNSX72VNUGT4UXKYI5CEIIHNZGNB63OA, seg_pos={0 0}, upload=148985]: Done uploading piece. err: <nil> Sep 13 14:55:18.323717226 (169h26m54.363290346s) [noise=true , address=77.90.31.11:4559 , piece_id=OYMOQVJRZ6SMMF7O2DXXDEOVRN27B3FT5EIVMFFBHC47UPZUUMIQ, seg_pos={0 0}, upload=148985]: Done uploading piece. err: <nil> Sep 13 14:55:18.769215263 (169h26m54.808788383s) [noise=true , address=45.91.116.14:4221 , piece_id=Q7ZPEITLUERUPOI247KAARIL3GSAPW6JNEUF7FSHQWJHL5GTGIZQ, seg_pos={0 0}, upload=148985]: Done uploading piece. err: <nil> Sep 13 14:56:14.598217910 (169h27m50.63779103s) [noise=true , address=145.79.227.28:4730 , piece_id=5WCQINYSLLWZR3Z3QWWUKRKDPRWYQ6VQPIZ4YDAD47JHSKDSCAGA, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:1SXQt5FugmWj1YdhsvaaR7bDoxTBkqHKZpyXSmQqGdX5ZRhKxV, address:145.79.227.28:4730): protocol: read tcp 188.166.225.170:42820->145.79.227.28:4730: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 13 14:56:14.598245002 (169h27m50.637818122s) [noise=true , address=145.79.227.52:4475 , piece_id=3T3MRZJ2T74J7CB5EJ4N2CIIERKWTQ5KWIOLDNV2LXHYLO5IKHTA, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:12UYjE6PEB9gZKwTKcnx23ykF4jDMJzLQoKmiFiejWew5mirdCj, address:145.79.227.52:4475): protocol: rpc: dial tcp 145.79.227.52:4475: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 13 14:56:14.598273767 (169h27m50.637846887s) [noise=true , address=145.79.227.50:4386 , piece_id=74QVIDRRHFRIA5Z6ZAQ4EWQ2PZ5HZI5JXNJ5WR2DVFJQ4WGE2UPQ, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:19aPjJnya2Q3stEhiedDSth3NHV8phLRj5yY79hZv81UgDitMv, address:145.79.227.50:4386): protocol: write tcp 188.166.225.170:42008->145.79.227.50:4386: use of closed network connection; write tcp 188.166.225.170:42008->145.79.227.50:4386: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42008->145.79.227.50:4386: use of closed network connection Sep 13 14:56:14.598293245 (169h27m50.637866365s) [noise=true , address=145.79.227.5:4250 , piece_id=EPZOZXRFVPSAPD6UKPOCCBOX7VJNB4DQRKV6YTRWMPOLQQ5IH6KQ, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:1uoGjfqBLfoNrg2X1aYKYLKQjgFUnzLjHetCFc9sPjJa2qqKYM, address:145.79.227.5:4250): protocol: write tcp 188.166.225.170:58078->145.79.227.5:4250: use of closed network connection; write tcp 188.166.225.170:58078->145.79.227.5:4250: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58078->145.79.227.5:4250: use of closed network connection Sep 13 14:56:14.598308794 (169h27m50.637881914s) [noise=true , address=145.79.227.33:4894 , piece_id=5KD2BL5J7DT7JDLPDS2QUUMM26AFWSXJ2BDTJSUGZ253ZV7XOVKQ, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:12KWNyruYhmSJVvgUF7fBpGf4H7eYmtAeEXToYjd2dcEPcj1E44, address:145.79.227.33:4894): protocol: write tcp 188.166.225.170:53614->145.79.227.33:4894: use of closed network connection; write tcp 188.166.225.170:53614->145.79.227.33:4894: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53614->145.79.227.33:4894: use of closed network connection Sep 13 14:56:14.598327330 (169h27m50.63790045s) [noise=true , address=145.79.227.7:4308 , piece_id=FYL7PQOKW74EJ2QLKMN6EYJKIT7A6B2VQESVP2VFVDVJHPHXFGAQ, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:1PgA5SZkBvK673LsV7p2sXpbi1r79esNNZspPZTzaitqfbjtyY, address:145.79.227.7:4308): protocol: write tcp 188.166.225.170:37146->145.79.227.7:4308: use of closed network connection; write tcp 188.166.225.170:37146->145.79.227.7:4308: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37146->145.79.227.7:4308: use of closed network connection Sep 13 14:56:14.598358169 (169h27m50.637931289s) [noise=true , address=145.79.227.36:4544 , piece_id=WZRHCCBJDHID4R4UOB76OABJSH2QVC6WZ6OZWIUUMKWWBDVWEG4A, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:12RSxwxw1WXkNcu7b3dEgJhyNkFGYUfKF4QVKmK323L5g3EKFnG, address:145.79.227.36:4544): protocol: rpc: dial tcp 145.79.227.36:4544: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 13 14:56:14.598387545 (169h27m50.637960665s) [noise=true , address=145.79.227.2:4137 , piece_id=NH4RZXYD3LKQZLLP5SSA7HXC7FBBOQV6QLBE2B3R5HH5F23Q7E6A, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:12tkbodtFrNCuJbvZk8YnpSdJfVsKasNuKhg2wGD8rDkNPrHZ21, address:145.79.227.2:4137): protocol: rpc: dial tcp 145.79.227.2:4137: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 13 14:56:14.598421721 (169h27m50.637994841s) [noise=true , address=145.79.227.30:4780 , piece_id=7EMNYDLOMVLIDGE5ZK7WFQCNGBFLZA5AF5EHSLHYE22XJ2MP56VQ, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:125WCbZ6a2NaewiLAHhHuPo91sQEfC9CZzEBV6rwF1JY7z13AEY, address:145.79.227.30:4780): protocol: read tcp 188.166.225.170:43016->145.79.227.30:4780: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 13 14:56:14.598427161 (169h27m50.638000281s) [noise=true , address=145.79.227.19:4090 , piece_id=YZ2H4UUM3Z5ZMCC6LQDAPLHCT7OQQAMRSDBQ3CH6I6UT2HAN4TXQ, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:12qSkJk9HUyQon3UP1AA8mRcqJbNdkKJh6cEUZoHLGNniqDvNqB, address:145.79.227.19:4090): protocol: write tcp 188.166.225.170:37110->145.79.227.19:4090: use of closed network connection; write tcp 188.166.225.170:37110->145.79.227.19:4090: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37110->145.79.227.19:4090: use of closed network connection Sep 13 14:56:14.598456818 (169h27m50.638029938s) [noise=true , address=145.79.227.10:4425 , piece_id=CRES4FQPX4WMTDTUBCPRZ2R4OHTI7MMHS6XSCZ7UXKDLSSLEVQ3Q, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:13sAcGFjAXuesAkUn7NuLaaV8TAah3Mr1kqd33wwh9MTELqmpD, address:145.79.227.10:4425): protocol: read tcp 188.166.225.170:58348->145.79.227.10:4425: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 13 14:56:14.598463220 (169h27m50.63803635s) [noise=true , address=145.79.227.35:4501 , piece_id=UKSF2KK675OI35QE6QZTH7QAJQ4SD7E42ZPMY2JIANOHGEFF44UQ, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:1hmsSAiSp1CBFqKhVE2S3sFfqYYkGVAiV1GfrT4Rd7hBaFNPpT, address:145.79.227.35:4501): protocol: rpc: dial tcp 145.79.227.35:4501: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 13 14:56:14.598484230 (169h27m50.63805735s) [noise=true , address=145.79.227.14:4867 , piece_id=7BW6OXJYXIXQSQK3DMHSGQ56ANOAOEGXEHD4EWCOZDXTTIL37SAA, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:12Sz7PyMo81CmAAkiCwpZ3M3JaM9HMHJLwAAM5M5kv6nnKktpKE, address:145.79.227.14:4867): protocol: read tcp 188.166.225.170:60252->145.79.227.14:4867: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 13 14:56:14.598492155 (169h27m50.638065275s) [noise=true , address=145.79.227.18:4018 , piece_id=72FTA6TNIQAVS2NDA5XOR4J2ZEEBWZ7LLGT4EAQ2EN2YB3VXG5JA, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:1bFRHbXjTaxoEKc5gbQWYt6jwUg8jknScBZpc8PdKNSZhYqUqP, address:145.79.227.18:4018): protocol: read tcp 188.166.225.170:60682->145.79.227.18:4018: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 13 14:56:14.598496193 (169h27m50.638069313s) [noise=true , address=145.79.227.47:4286 , piece_id=D2PSYV2IN5MFZGGF2U47EPRTCNDIMHB74GR3UFQ64AJN37FE7BVA, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:12v2vJnE92VrJXQrbxnAxANHCVASuXiBKMUZECMVmEWEjsZRtsQ, address:145.79.227.47:4286): protocol: read tcp 188.166.225.170:33092->145.79.227.47:4286: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 13 14:56:14.598521301 (169h27m50.638094421s) [seg_pos={0 0}, upload=148985]: Done waiting for segment. successful: 3 optimal: 14 errs: <nil> Sep 13 14:56:14.862062924 (169h27m50.901636044s) [upload=148985]: Done uploading segments... Sep 13 14:56:14.862090337 (169h27m50.901663457s) [upload=148985]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 3) Duration: 10.190575256s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [XX....................X.X...X..X.............X..X...X.X....X] duration: 30.000468444s FAIL (2025-09-13 15:00:46 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.000490297s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 7.343376401s FAIL (2025-09-13 15:00:24 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 7.343397643s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.................................................X...X......] duration: 14.420558561s FAIL (2025-09-13 15:00:06 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 3) ======LOGS====== 6OXJYXIXQSQK3DMHSGQ56ANOAOEGXEHD4EWCOZDXTTIL37SAA, seg_pos={0 0}, upload=148985]: Uploading piece... Sep 13 14:55:15.907845810 (169h26m51.94741893s) [noise=true , address=145.79.227.52:4475 , piece_id=3T3MRZJ2T74J7CB5EJ4N2CIIERKWTQ5KWIOLDNV2LXHYLO5IKHTA, seg_pos={0 0}, upload=148985]: Uploading piece... Sep 13 14:55:15.908062555 (169h26m51.947635675s) [noise=true , address=145.79.227.5:4250 , piece_id=EPZOZXRFVPSAPD6UKPOCCBOX7VJNB4DQRKV6YTRWMPOLQQ5IH6KQ, seg_pos={0 0}, upload=148985]: Uploading piece... Sep 13 14:55:15.908276674 (169h26m51.947849794s) [noise=true , address=145.79.227.30:4780 , piece_id=7EMNYDLOMVLIDGE5ZK7WFQCNGBFLZA5AF5EHSLHYE22XJ2MP56VQ, seg_pos={0 0}, upload=148985]: Uploading piece... Sep 13 14:55:15.908522154 (169h26m51.948095274s) [noise=true , address=77.90.31.17:4793 , piece_id=PWVTHYWQY6GFO5FCIBANYNSX72VNUGT4UXKYI5CEIIHNZGNB63OA, seg_pos={0 0}, upload=148985]: Uploading piece... Sep 13 14:55:17.522666509 (169h26m53.562239629s) [upload=148985]: Next returned nil segment Sep 13 14:55:17.522686247 (169h26m53.562259367s) [upload=148985]: Waiting for error group managing segments... Sep 13 14:55:18.048737965 (169h26m54.088311085s) [noise=true , address=77.90.31.17:4793 , piece_id=PWVTHYWQY6GFO5FCIBANYNSX72VNUGT4UXKYI5CEIIHNZGNB63OA, seg_pos={0 0}, upload=148985]: Done uploading piece. err: <nil> Sep 13 14:55:18.323717226 (169h26m54.363290346s) [noise=true , address=77.90.31.11:4559 , piece_id=OYMOQVJRZ6SMMF7O2DXXDEOVRN27B3FT5EIVMFFBHC47UPZUUMIQ, seg_pos={0 0}, upload=148985]: Done uploading piece. err: <nil> Sep 13 14:55:18.769215263 (169h26m54.808788383s) [noise=true , address=45.91.116.14:4221 , piece_id=Q7ZPEITLUERUPOI247KAARIL3GSAPW6JNEUF7FSHQWJHL5GTGIZQ, seg_pos={0 0}, upload=148985]: Done uploading piece. err: <nil> Sep 13 14:56:14.598217910 (169h27m50.63779103s) [noise=true , address=145.79.227.28:4730 , piece_id=5WCQINYSLLWZR3Z3QWWUKRKDPRWYQ6VQPIZ4YDAD47JHSKDSCAGA, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:1SXQt5FugmWj1YdhsvaaR7bDoxTBkqHKZpyXSmQqGdX5ZRhKxV, address:145.79.227.28:4730): protocol: read tcp 188.166.225.170:42820->145.79.227.28:4730: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 13 14:56:14.598245002 (169h27m50.637818122s) [noise=true , address=145.79.227.52:4475 , piece_id=3T3MRZJ2T74J7CB5EJ4N2CIIERKWTQ5KWIOLDNV2LXHYLO5IKHTA, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:12UYjE6PEB9gZKwTKcnx23ykF4jDMJzLQoKmiFiejWew5mirdCj, address:145.79.227.52:4475): protocol: rpc: dial tcp 145.79.227.52:4475: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 13 14:56:14.598273767 (169h27m50.637846887s) [noise=true , address=145.79.227.50:4386 , piece_id=74QVIDRRHFRIA5Z6ZAQ4EWQ2PZ5HZI5JXNJ5WR2DVFJQ4WGE2UPQ, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:19aPjJnya2Q3stEhiedDSth3NHV8phLRj5yY79hZv81UgDitMv, address:145.79.227.50:4386): protocol: write tcp 188.166.225.170:42008->145.79.227.50:4386: use of closed network connection; write tcp 188.166.225.170:42008->145.79.227.50:4386: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42008->145.79.227.50:4386: use of closed network connection Sep 13 14:56:14.598293245 (169h27m50.637866365s) [noise=true , address=145.79.227.5:4250 , piece_id=EPZOZXRFVPSAPD6UKPOCCBOX7VJNB4DQRKV6YTRWMPOLQQ5IH6KQ, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:1uoGjfqBLfoNrg2X1aYKYLKQjgFUnzLjHetCFc9sPjJa2qqKYM, address:145.79.227.5:4250): protocol: write tcp 188.166.225.170:58078->145.79.227.5:4250: use of closed network connection; write tcp 188.166.225.170:58078->145.79.227.5:4250: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58078->145.79.227.5:4250: use of closed network connection Sep 13 14:56:14.598308794 (169h27m50.637881914s) [noise=true , address=145.79.227.33:4894 , piece_id=5KD2BL5J7DT7JDLPDS2QUUMM26AFWSXJ2BDTJSUGZ253ZV7XOVKQ, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:12KWNyruYhmSJVvgUF7fBpGf4H7eYmtAeEXToYjd2dcEPcj1E44, address:145.79.227.33:4894): protocol: write tcp 188.166.225.170:53614->145.79.227.33:4894: use of closed network connection; write tcp 188.166.225.170:53614->145.79.227.33:4894: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53614->145.79.227.33:4894: use of closed network connection Sep 13 14:56:14.598327330 (169h27m50.63790045s) [noise=true , address=145.79.227.7:4308 , piece_id=FYL7PQOKW74EJ2QLKMN6EYJKIT7A6B2VQESVP2VFVDVJHPHXFGAQ, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:1PgA5SZkBvK673LsV7p2sXpbi1r79esNNZspPZTzaitqfbjtyY, address:145.79.227.7:4308): protocol: write tcp 188.166.225.170:37146->145.79.227.7:4308: use of closed network connection; write tcp 188.166.225.170:37146->145.79.227.7:4308: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37146->145.79.227.7:4308: use of closed network connection Sep 13 14:56:14.598358169 (169h27m50.637931289s) [noise=true , address=145.79.227.36:4544 , piece_id=WZRHCCBJDHID4R4UOB76OABJSH2QVC6WZ6OZWIUUMKWWBDVWEG4A, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:12RSxwxw1WXkNcu7b3dEgJhyNkFGYUfKF4QVKmK323L5g3EKFnG, address:145.79.227.36:4544): protocol: rpc: dial tcp 145.79.227.36:4544: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 13 14:56:14.598387545 (169h27m50.637960665s) [noise=true , address=145.79.227.2:4137 , piece_id=NH4RZXYD3LKQZLLP5SSA7HXC7FBBOQV6QLBE2B3R5HH5F23Q7E6A, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:12tkbodtFrNCuJbvZk8YnpSdJfVsKasNuKhg2wGD8rDkNPrHZ21, address:145.79.227.2:4137): protocol: rpc: dial tcp 145.79.227.2:4137: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 13 14:56:14.598421721 (169h27m50.637994841s) [noise=true , address=145.79.227.30:4780 , piece_id=7EMNYDLOMVLIDGE5ZK7WFQCNGBFLZA5AF5EHSLHYE22XJ2MP56VQ, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:125WCbZ6a2NaewiLAHhHuPo91sQEfC9CZzEBV6rwF1JY7z13AEY, address:145.79.227.30:4780): protocol: read tcp 188.166.225.170:43016->145.79.227.30:4780: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 13 14:56:14.598427161 (169h27m50.638000281s) [noise=true , address=145.79.227.19:4090 , piece_id=YZ2H4UUM3Z5ZMCC6LQDAPLHCT7OQQAMRSDBQ3CH6I6UT2HAN4TXQ, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:12qSkJk9HUyQon3UP1AA8mRcqJbNdkKJh6cEUZoHLGNniqDvNqB, address:145.79.227.19:4090): protocol: write tcp 188.166.225.170:37110->145.79.227.19:4090: use of closed network connection; write tcp 188.166.225.170:37110->145.79.227.19:4090: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37110->145.79.227.19:4090: use of closed network connection Sep 13 14:56:14.598456818 (169h27m50.638029938s) [noise=true , address=145.79.227.10:4425 , piece_id=CRES4FQPX4WMTDTUBCPRZ2R4OHTI7MMHS6XSCZ7UXKDLSSLEVQ3Q, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:13sAcGFjAXuesAkUn7NuLaaV8TAah3Mr1kqd33wwh9MTELqmpD, address:145.79.227.10:4425): protocol: read tcp 188.166.225.170:58348->145.79.227.10:4425: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 13 14:56:14.598463220 (169h27m50.63803635s) [noise=true , address=145.79.227.35:4501 , piece_id=UKSF2KK675OI35QE6QZTH7QAJQ4SD7E42ZPMY2JIANOHGEFF44UQ, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:1hmsSAiSp1CBFqKhVE2S3sFfqYYkGVAiV1GfrT4Rd7hBaFNPpT, address:145.79.227.35:4501): protocol: rpc: dial tcp 145.79.227.35:4501: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 13 14:56:14.598484230 (169h27m50.63805735s) [noise=true , address=145.79.227.14:4867 , piece_id=7BW6OXJYXIXQSQK3DMHSGQ56ANOAOEGXEHD4EWCOZDXTTIL37SAA, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:12Sz7PyMo81CmAAkiCwpZ3M3JaM9HMHJLwAAM5M5kv6nnKktpKE, address:145.79.227.14:4867): protocol: read tcp 188.166.225.170:60252->145.79.227.14:4867: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 13 14:56:14.598492155 (169h27m50.638065275s) [noise=true , address=145.79.227.18:4018 , piece_id=72FTA6TNIQAVS2NDA5XOR4J2ZEEBWZ7LLGT4EAQ2EN2YB3VXG5JA, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:1bFRHbXjTaxoEKc5gbQWYt6jwUg8jknScBZpc8PdKNSZhYqUqP, address:145.79.227.18:4018): protocol: read tcp 188.166.225.170:60682->145.79.227.18:4018: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 13 14:56:14.598496193 (169h27m50.638069313s) [noise=true , address=145.79.227.47:4286 , piece_id=D2PSYV2IN5MFZGGF2U47EPRTCNDIMHB74GR3UFQ64AJN37FE7BVA, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:12v2vJnE92VrJXQrbxnAxANHCVASuXiBKMUZECMVmEWEjsZRtsQ, address:145.79.227.47:4286): protocol: read tcp 188.166.225.170:33092->145.79.227.47:4286: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 13 14:56:14.598521301 (169h27m50.638094421s) [seg_pos={0 0}, upload=148985]: Done waiting for segment. successful: 3 optimal: 14 errs: <nil> Sep 13 14:56:14.862062924 (169h27m50.901636044s) [upload=148985]: Done uploading segments... Sep 13 14:56:14.862090337 (169h27m50.901663457s) [upload=148985]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 3) Duration: 14.420581845s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 7.279281964s FAIL (2025-09-13 14:59:24 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 7.279304508s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [..................................................X...X.....] duration: 12.850009024s FAIL (2025-09-13 14:59:14 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 3) ======LOGS====== 6OXJYXIXQSQK3DMHSGQ56ANOAOEGXEHD4EWCOZDXTTIL37SAA, seg_pos={0 0}, upload=148985]: Uploading piece... Sep 13 14:55:15.907845810 (169h26m51.94741893s) [noise=true , address=145.79.227.52:4475 , piece_id=3T3MRZJ2T74J7CB5EJ4N2CIIERKWTQ5KWIOLDNV2LXHYLO5IKHTA, seg_pos={0 0}, upload=148985]: Uploading piece... Sep 13 14:55:15.908062555 (169h26m51.947635675s) [noise=true , address=145.79.227.5:4250 , piece_id=EPZOZXRFVPSAPD6UKPOCCBOX7VJNB4DQRKV6YTRWMPOLQQ5IH6KQ, seg_pos={0 0}, upload=148985]: Uploading piece... Sep 13 14:55:15.908276674 (169h26m51.947849794s) [noise=true , address=145.79.227.30:4780 , piece_id=7EMNYDLOMVLIDGE5ZK7WFQCNGBFLZA5AF5EHSLHYE22XJ2MP56VQ, seg_pos={0 0}, upload=148985]: Uploading piece... Sep 13 14:55:15.908522154 (169h26m51.948095274s) [noise=true , address=77.90.31.17:4793 , piece_id=PWVTHYWQY6GFO5FCIBANYNSX72VNUGT4UXKYI5CEIIHNZGNB63OA, seg_pos={0 0}, upload=148985]: Uploading piece... Sep 13 14:55:17.522666509 (169h26m53.562239629s) [upload=148985]: Next returned nil segment Sep 13 14:55:17.522686247 (169h26m53.562259367s) [upload=148985]: Waiting for error group managing segments... Sep 13 14:55:18.048737965 (169h26m54.088311085s) [noise=true , address=77.90.31.17:4793 , piece_id=PWVTHYWQY6GFO5FCIBANYNSX72VNUGT4UXKYI5CEIIHNZGNB63OA, seg_pos={0 0}, upload=148985]: Done uploading piece. err: <nil> Sep 13 14:55:18.323717226 (169h26m54.363290346s) [noise=true , address=77.90.31.11:4559 , piece_id=OYMOQVJRZ6SMMF7O2DXXDEOVRN27B3FT5EIVMFFBHC47UPZUUMIQ, seg_pos={0 0}, upload=148985]: Done uploading piece. err: <nil> Sep 13 14:55:18.769215263 (169h26m54.808788383s) [noise=true , address=45.91.116.14:4221 , piece_id=Q7ZPEITLUERUPOI247KAARIL3GSAPW6JNEUF7FSHQWJHL5GTGIZQ, seg_pos={0 0}, upload=148985]: Done uploading piece. err: <nil> Sep 13 14:56:14.598217910 (169h27m50.63779103s) [noise=true , address=145.79.227.28:4730 , piece_id=5WCQINYSLLWZR3Z3QWWUKRKDPRWYQ6VQPIZ4YDAD47JHSKDSCAGA, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:1SXQt5FugmWj1YdhsvaaR7bDoxTBkqHKZpyXSmQqGdX5ZRhKxV, address:145.79.227.28:4730): protocol: read tcp 188.166.225.170:42820->145.79.227.28:4730: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 13 14:56:14.598245002 (169h27m50.637818122s) [noise=true , address=145.79.227.52:4475 , piece_id=3T3MRZJ2T74J7CB5EJ4N2CIIERKWTQ5KWIOLDNV2LXHYLO5IKHTA, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:12UYjE6PEB9gZKwTKcnx23ykF4jDMJzLQoKmiFiejWew5mirdCj, address:145.79.227.52:4475): protocol: rpc: dial tcp 145.79.227.52:4475: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 13 14:56:14.598273767 (169h27m50.637846887s) [noise=true , address=145.79.227.50:4386 , piece_id=74QVIDRRHFRIA5Z6ZAQ4EWQ2PZ5HZI5JXNJ5WR2DVFJQ4WGE2UPQ, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:19aPjJnya2Q3stEhiedDSth3NHV8phLRj5yY79hZv81UgDitMv, address:145.79.227.50:4386): protocol: write tcp 188.166.225.170:42008->145.79.227.50:4386: use of closed network connection; write tcp 188.166.225.170:42008->145.79.227.50:4386: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42008->145.79.227.50:4386: use of closed network connection Sep 13 14:56:14.598293245 (169h27m50.637866365s) [noise=true , address=145.79.227.5:4250 , piece_id=EPZOZXRFVPSAPD6UKPOCCBOX7VJNB4DQRKV6YTRWMPOLQQ5IH6KQ, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:1uoGjfqBLfoNrg2X1aYKYLKQjgFUnzLjHetCFc9sPjJa2qqKYM, address:145.79.227.5:4250): protocol: write tcp 188.166.225.170:58078->145.79.227.5:4250: use of closed network connection; write tcp 188.166.225.170:58078->145.79.227.5:4250: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58078->145.79.227.5:4250: use of closed network connection Sep 13 14:56:14.598308794 (169h27m50.637881914s) [noise=true , address=145.79.227.33:4894 , piece_id=5KD2BL5J7DT7JDLPDS2QUUMM26AFWSXJ2BDTJSUGZ253ZV7XOVKQ, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:12KWNyruYhmSJVvgUF7fBpGf4H7eYmtAeEXToYjd2dcEPcj1E44, address:145.79.227.33:4894): protocol: write tcp 188.166.225.170:53614->145.79.227.33:4894: use of closed network connection; write tcp 188.166.225.170:53614->145.79.227.33:4894: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53614->145.79.227.33:4894: use of closed network connection Sep 13 14:56:14.598327330 (169h27m50.63790045s) [noise=true , address=145.79.227.7:4308 , piece_id=FYL7PQOKW74EJ2QLKMN6EYJKIT7A6B2VQESVP2VFVDVJHPHXFGAQ, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:1PgA5SZkBvK673LsV7p2sXpbi1r79esNNZspPZTzaitqfbjtyY, address:145.79.227.7:4308): protocol: write tcp 188.166.225.170:37146->145.79.227.7:4308: use of closed network connection; write tcp 188.166.225.170:37146->145.79.227.7:4308: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37146->145.79.227.7:4308: use of closed network connection Sep 13 14:56:14.598358169 (169h27m50.637931289s) [noise=true , address=145.79.227.36:4544 , piece_id=WZRHCCBJDHID4R4UOB76OABJSH2QVC6WZ6OZWIUUMKWWBDVWEG4A, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:12RSxwxw1WXkNcu7b3dEgJhyNkFGYUfKF4QVKmK323L5g3EKFnG, address:145.79.227.36:4544): protocol: rpc: dial tcp 145.79.227.36:4544: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 13 14:56:14.598387545 (169h27m50.637960665s) [noise=true , address=145.79.227.2:4137 , piece_id=NH4RZXYD3LKQZLLP5SSA7HXC7FBBOQV6QLBE2B3R5HH5F23Q7E6A, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:12tkbodtFrNCuJbvZk8YnpSdJfVsKasNuKhg2wGD8rDkNPrHZ21, address:145.79.227.2:4137): protocol: rpc: dial tcp 145.79.227.2:4137: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 13 14:56:14.598421721 (169h27m50.637994841s) [noise=true , address=145.79.227.30:4780 , piece_id=7EMNYDLOMVLIDGE5ZK7WFQCNGBFLZA5AF5EHSLHYE22XJ2MP56VQ, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:125WCbZ6a2NaewiLAHhHuPo91sQEfC9CZzEBV6rwF1JY7z13AEY, address:145.79.227.30:4780): protocol: read tcp 188.166.225.170:43016->145.79.227.30:4780: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 13 14:56:14.598427161 (169h27m50.638000281s) [noise=true , address=145.79.227.19:4090 , piece_id=YZ2H4UUM3Z5ZMCC6LQDAPLHCT7OQQAMRSDBQ3CH6I6UT2HAN4TXQ, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:12qSkJk9HUyQon3UP1AA8mRcqJbNdkKJh6cEUZoHLGNniqDvNqB, address:145.79.227.19:4090): protocol: write tcp 188.166.225.170:37110->145.79.227.19:4090: use of closed network connection; write tcp 188.166.225.170:37110->145.79.227.19:4090: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37110->145.79.227.19:4090: use of closed network connection Sep 13 14:56:14.598456818 (169h27m50.638029938s) [noise=true , address=145.79.227.10:4425 , piece_id=CRES4FQPX4WMTDTUBCPRZ2R4OHTI7MMHS6XSCZ7UXKDLSSLEVQ3Q, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:13sAcGFjAXuesAkUn7NuLaaV8TAah3Mr1kqd33wwh9MTELqmpD, address:145.79.227.10:4425): protocol: read tcp 188.166.225.170:58348->145.79.227.10:4425: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 13 14:56:14.598463220 (169h27m50.63803635s) [noise=true , address=145.79.227.35:4501 , piece_id=UKSF2KK675OI35QE6QZTH7QAJQ4SD7E42ZPMY2JIANOHGEFF44UQ, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:1hmsSAiSp1CBFqKhVE2S3sFfqYYkGVAiV1GfrT4Rd7hBaFNPpT, address:145.79.227.35:4501): protocol: rpc: dial tcp 145.79.227.35:4501: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 13 14:56:14.598484230 (169h27m50.63805735s) [noise=true , address=145.79.227.14:4867 , piece_id=7BW6OXJYXIXQSQK3DMHSGQ56ANOAOEGXEHD4EWCOZDXTTIL37SAA, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:12Sz7PyMo81CmAAkiCwpZ3M3JaM9HMHJLwAAM5M5kv6nnKktpKE, address:145.79.227.14:4867): protocol: read tcp 188.166.225.170:60252->145.79.227.14:4867: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 13 14:56:14.598492155 (169h27m50.638065275s) [noise=true , address=145.79.227.18:4018 , piece_id=72FTA6TNIQAVS2NDA5XOR4J2ZEEBWZ7LLGT4EAQ2EN2YB3VXG5JA, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:1bFRHbXjTaxoEKc5gbQWYt6jwUg8jknScBZpc8PdKNSZhYqUqP, address:145.79.227.18:4018): protocol: read tcp 188.166.225.170:60682->145.79.227.18:4018: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 13 14:56:14.598496193 (169h27m50.638069313s) [noise=true , address=145.79.227.47:4286 , piece_id=D2PSYV2IN5MFZGGF2U47EPRTCNDIMHB74GR3UFQ64AJN37FE7BVA, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:12v2vJnE92VrJXQrbxnAxANHCVASuXiBKMUZECMVmEWEjsZRtsQ, address:145.79.227.47:4286): protocol: read tcp 188.166.225.170:33092->145.79.227.47:4286: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 13 14:56:14.598521301 (169h27m50.638094421s) [seg_pos={0 0}, upload=148985]: Done waiting for segment. successful: 3 optimal: 14 errs: <nil> Sep 13 14:56:14.862062924 (169h27m50.901636044s) [upload=148985]: Done uploading segments... Sep 13 14:56:14.862090337 (169h27m50.901663457s) [upload=148985]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 3) Duration: 12.850036688s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [.XX....................X.X...X..X.............X..X...X.X....] duration: 6.642662878s FAIL (2025-09-13 14:59:09 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.642694119s edge gateway us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)