Orbiter History

checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [......XXX.X..X.....XXXXX...XXX.X.....X.X..X..X.X....X.X....X] duration: 30.001222699s FAIL (2025-07-26 23:52:31 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.001242749s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 57 out of 60 is failed (threshold=3) [...XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.23540559s FAIL (2025-07-26 23:52:25 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 9) ======LOGS====== piece_id=NF5N54WE3QGNS7WAIM5M3W7B5F2NNRT2B3RYDLYJKC5C5CLAY6EA, seg_pos={0 0}, upload=57802]: Uploading piece... Jul 26 23:51:26.202077401 (60h15m35.924640441s) [noise=true , address=79.127.205.245:4246 , piece_id=T7LN7ABTBCXEIPRVJ4RFHG3X5CMUFHPK6CZT6SLAUNTVT424TE5A, seg_pos={0 0}, upload=57802]: Uploading piece... Jul 26 23:51:26.202371564 (60h15m35.924934594s) [noise=true , address=104.36.82.55:4214 , piece_id=UNQISRA64SQL62RQ4ECWZUNR55YQMQU5LQCAXYCJLNBENM2EMDKA, seg_pos={0 0}, upload=57802]: Uploading piece... Jul 26 23:51:26.202571246 (60h15m35.925134286s) [noise=true , address=104.36.82.54:4157 , piece_id=7WEWW2XCMYJFNEKULG6DWTWCQL7QGGTJXWL4BPH4XEGS77RPV3ZQ, seg_pos={0 0}, upload=57802]: Uploading piece... Jul 26 23:51:26.202651500 (60h15m35.92521454s) [noise=true , address=79.127.205.233:4644 , piece_id=OTG5MZB6A5JNIBUDM4QLGGGWBS3HIRHIQ45H4CPJHVQZRBWUGYRA, seg_pos={0 0}, upload=57802]: Uploading piece... Jul 26 23:51:26.202949039 (60h15m35.925512079s) [noise=true , address=79.127.219.38:4001 , piece_id=LPVWDXVWI5UXI4KUVJIQP63FH4VQND4KDN4ZKSCIF3WSNZP6AGZQ, seg_pos={0 0}, upload=57802]: Uploading piece... Jul 26 23:51:26.203006579 (60h15m35.925569639s) [noise=true , address=79.127.205.234:4601 , piece_id=LX6QSNLCVFKVTDHBMCK3XUMMKCOISMNKCGDKGZIXDFXTIT5AQX2Q, seg_pos={0 0}, upload=57802]: Uploading piece... Jul 26 23:51:26.203193808 (60h15m35.925756848s) [noise=true , address=104.36.82.56:4251 , piece_id=4C3A7ULUADUVDQWQ2BZYJMKIKAFGIS6OFIM674KYXTSUL6FC4CGQ, seg_pos={0 0}, upload=57802]: Uploading piece... Jul 26 23:51:26.203431602 (60h15m35.925994642s) [noise=true , address=79.127.220.101:4048 , piece_id=XD75PBT3NQYCP3ZQVK3XYC4SLUQ4NAZN6YFDQQBMEXM5CD4HXAOA, seg_pos={0 0}, upload=57802]: Uploading piece... Jul 26 23:51:26.203455037 (60h15m35.926018067s) [noise=true , address=104.36.82.52:4092 , piece_id=FSGIDZJWN3PJTIFHCV7JFYPUREMML2LNWXW323WREPUZB3Z2KYTA, seg_pos={0 0}, upload=57802]: Uploading piece... Jul 26 23:51:26.203712681 (60h15m35.926275721s) [noise=true , address=104.36.82.46:4515 , piece_id=Z4GBYTIXLVRFE7TYCY5S77VVGOHPHAZBLK2DTF2XPZIZAICMHSMA, seg_pos={0 0}, upload=57802]: Uploading piece... Jul 26 23:51:26.203894788 (60h15m35.926457818s) [noise=true , address=79.127.205.246:4281 , piece_id=DQIP6GPALSTEE5RFPW3RUQOABBBINWFGLAACUHR575DPY6WJZZHQ, seg_pos={0 0}, upload=57802]: Uploading piece... Jul 26 23:51:26.203952980 (60h15m35.92651602s) [noise=true , address=45.91.116.10:4093 , piece_id=ZFHOAN6BM6MXNYZ3TFC4PKZGLKRHCSDDJF447EKSSKWLRZINJDWA, seg_pos={0 0}, upload=57802]: Uploading piece... Jul 26 23:51:26.204248616 (60h15m35.926811656s) [noise=true , address=104.36.82.20:4469 , piece_id=X5EPP77XFUJ4VAZJCAHF2BXNEVDY4JABOEW6J3WWOWT7FZXKI3UA, seg_pos={0 0}, upload=57802]: Uploading piece... Jul 26 23:51:26.204586021 (60h15m35.927149061s) [noise=true , address=104.36.82.47:4546 , piece_id=OGDLOUXC2SADM3RD2SCT5BOW3MZ7GHWIZ3Q72ZYSJAN6HN6VQF7Q, seg_pos={0 0}, upload=57802]: Uploading piece... Jul 26 23:51:26.205047554 (60h15m35.927610594s) [noise=true , address=104.36.82.11:4138 , piece_id=DKRGAQOII4VIVFLXUUGI25S7DTCQ2RXXKMSNRMB6KL6OLVMXG63Q, seg_pos={0 0}, upload=57802]: Uploading piece... Jul 26 23:51:26.205372626 (60h15m35.927935666s) [noise=true , address=104.36.82.21:4479 , piece_id=VLWJAD7OFSRQW4N5WV5BNXP4QJF56V7K3M5O3G6FQFHM5JIGUUQA, seg_pos={0 0}, upload=57802]: Uploading piece... Jul 26 23:51:27.302283947 (60h15m37.024846987s) [upload=57802]: Next returned nil segment Jul 26 23:51:27.302294738 (60h15m37.024857778s) [upload=57802]: Waiting for error group managing segments... Jul 26 23:51:27.591183990 (60h15m37.31374702s) [noise=true , address=79.127.205.233:4644 , piece_id=OTG5MZB6A5JNIBUDM4QLGGGWBS3HIRHIQ45H4CPJHVQZRBWUGYRA, seg_pos={0 0}, upload=57802]: Done uploading piece. err: <nil> Jul 26 23:51:27.599057915 (60h15m37.321620955s) [noise=true , address=79.127.205.245:4246 , piece_id=T7LN7ABTBCXEIPRVJ4RFHG3X5CMUFHPK6CZT6SLAUNTVT424TE5A, seg_pos={0 0}, upload=57802]: Done uploading piece. err: <nil> Jul 26 23:51:27.636069910 (60h15m37.35863298s) [noise=true , address=79.127.205.234:4601 , piece_id=LX6QSNLCVFKVTDHBMCK3XUMMKCOISMNKCGDKGZIXDFXTIT5AQX2Q, seg_pos={0 0}, upload=57802]: Done uploading piece. err: <nil> Jul 26 23:51:27.655914489 (60h15m37.378477519s) [noise=true , address=79.127.205.246:4281 , piece_id=DQIP6GPALSTEE5RFPW3RUQOABBBINWFGLAACUHR575DPY6WJZZHQ, seg_pos={0 0}, upload=57802]: Done uploading piece. err: <nil> Jul 26 23:51:27.757744640 (60h15m37.48030767s) [noise=true , address=64.49.13.137:4502 , piece_id=NF5N54WE3QGNS7WAIM5M3W7B5F2NNRT2B3RYDLYJKC5C5CLAY6EA, seg_pos={0 0}, upload=57802]: Done uploading piece. err: <nil> Jul 26 23:51:28.248289139 (60h15m37.970852179s) [noise=true , address=77.90.31.11:4562 , piece_id=7UVOEIUYBTJ4W4BJ4WDT56JOJGFYXYH2FKCHSXZEDPTEKQFOHXUQ, seg_pos={0 0}, upload=57802]: Done uploading piece. err: <nil> Jul 26 23:51:28.468041050 (60h15m38.190604101s) [noise=true , address=79.127.219.38:4001 , piece_id=LPVWDXVWI5UXI4KUVJIQP63FH4VQND4KDN4ZKSCIF3WSNZP6AGZQ, seg_pos={0 0}, upload=57802]: Done uploading piece. err: <nil> Jul 26 23:51:31.745595829 (60h15m41.468158869s) [noise=true , address=79.127.220.101:4048 , piece_id=XD75PBT3NQYCP3ZQVK3XYC4SLUQ4NAZN6YFDQQBMEXM5CD4HXAOA, seg_pos={0 0}, upload=57802]: Done uploading piece. err: <nil> Jul 26 23:51:38.626616778 (60h15m48.349179818s) [noise=true , address=45.91.116.10:4093 , piece_id=ZFHOAN6BM6MXNYZ3TFC4PKZGLKRHCSDDJF447EKSSKWLRZINJDWA, seg_pos={0 0}, upload=57802]: Done uploading piece. err: <nil> Jul 26 23:52:25.110504015 (60h16m34.833067055s) [noise=true , address=104.36.82.20:4469 , piece_id=X5EPP77XFUJ4VAZJCAHF2BXNEVDY4JABOEW6J3WWOWT7FZXKI3UA, seg_pos={0 0}, upload=57802]: Done uploading piece. err: ecclient: upload failed (node:1A6EEfrxXWy6k1Ki5UrjR5tKgRYsVXsSG9za2bQvKuejRpawHT, address:104.36.82.20:4469): protocol: rpc: dial tcp 104.36.82.20:4469: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:52:25.110661958 (60h16m34.833224998s) [noise=true , address=104.36.82.21:4479 , piece_id=VLWJAD7OFSRQW4N5WV5BNXP4QJF56V7K3M5O3G6FQFHM5JIGUUQA, seg_pos={0 0}, upload=57802]: Done uploading piece. err: ecclient: upload failed (node:12VCopFEoHk99GsQNQPuWvTQZRD1bpXi8HHLKXaXRdvFThTxmAk, address:104.36.82.21:4479): protocol: rpc: dial tcp 104.36.82.21:4479: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:52:25.110697756 (60h16m34.833260796s) [noise=true , address=104.36.82.54:4157 , piece_id=7WEWW2XCMYJFNEKULG6DWTWCQL7QGGTJXWL4BPH4XEGS77RPV3ZQ, seg_pos={0 0}, upload=57802]: Done uploading piece. err: ecclient: upload failed (node:1ZrzADk2mKG4R9tjxUKjQeJvb4WmfjLksGFyvstHRfYbBC7XNz, address:104.36.82.54:4157): protocol: write tcp 188.166.225.170:48912->104.36.82.54:4157: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:52:25.110915482 (60h16m34.833478513s) [noise=true , address=104.36.82.55:4214 , piece_id=UNQISRA64SQL62RQ4ECWZUNR55YQMQU5LQCAXYCJLNBENM2EMDKA, seg_pos={0 0}, upload=57802]: Done uploading piece. err: ecclient: upload failed (node:12LV4vwjXQvjzxs1jtSpNXZ6VvzLXbQBCYQBn5gzb7jh2DD9qpi, address:104.36.82.55:4214): protocol: rpc: dial tcp 104.36.82.55:4214: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:52:25.111045992 (60h16m34.833609033s) [noise=true , address=104.36.82.11:4138 , piece_id=DKRGAQOII4VIVFLXUUGI25S7DTCQ2RXXKMSNRMB6KL6OLVMXG63Q, seg_pos={0 0}, upload=57802]: Done uploading piece. err: ecclient: upload failed (node:1H28XSTvFGa3XhavfhythHWs95cqjazu2RuqpsjNCmdSFvRLoh, address:104.36.82.11:4138): protocol: rpc: dial tcp 104.36.82.11:4138: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:52:25.111088123 (60h16m34.833651163s) [noise=true , address=104.36.82.52:4092 , piece_id=FSGIDZJWN3PJTIFHCV7JFYPUREMML2LNWXW323WREPUZB3Z2KYTA, seg_pos={0 0}, upload=57802]: Done uploading piece. err: ecclient: upload failed (node:12wYaUu5QpFU8T4mZt1iqeHqVJKJ5CyVW5vkeTWf1a8G3pLsx3P, address:104.36.82.52:4092): protocol: rpc: dial tcp 104.36.82.52:4092: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:52:25.111192693 (60h16m34.833755733s) [noise=true , address=104.36.82.47:4546 , piece_id=OGDLOUXC2SADM3RD2SCT5BOW3MZ7GHWIZ3Q72ZYSJAN6HN6VQF7Q, seg_pos={0 0}, upload=57802]: Done uploading piece. err: ecclient: upload failed (node:1FJGmqUskGmxYctVCEaYhRHq98m1VCVZiWL1ShEMJzmieaYo1u, address:104.36.82.47:4546): protocol: rpc: dial tcp 104.36.82.47:4546: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:52:25.111325016 (60h16m34.833888056s) [noise=true , address=104.36.82.56:4251 , piece_id=4C3A7ULUADUVDQWQ2BZYJMKIKAFGIS6OFIM674KYXTSUL6FC4CGQ, seg_pos={0 0}, upload=57802]: Done uploading piece. err: ecclient: upload failed (node:12vvYKd1KXrz8cYXRJSL9KXiYQtY95aB2GFPGFi9c9Nj1Raj6J4, address:104.36.82.56:4251): protocol: rpc: dial tcp 104.36.82.56:4251: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:52:25.111386313 (60h16m34.833949353s) [noise=true , address=104.36.82.46:4515 , piece_id=Z4GBYTIXLVRFE7TYCY5S77VVGOHPHAZBLK2DTF2XPZIZAICMHSMA, seg_pos={0 0}, upload=57802]: Done uploading piece. err: ecclient: upload failed (node:1iDufTPd97E4DAupki6D6DxNZS8S1gKwynNBYynsauhbw5waUs, address:104.36.82.46:4515): protocol: write tcp 188.166.225.170:59108->104.36.82.46:4515: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:52:25.111410329 (60h16m34.833973359s) [seg_pos={0 0}, upload=57802]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil> Jul 26 23:52:25.343612370 (60h16m35.06617542s) [upload=57802]: Done uploading segments... Jul 26 23:52:25.343643649 (60h16m35.066206689s) [upload=57802]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9) Duration: 1m0.235421351s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 2.047373108s FAIL (2025-07-26 23:52:25 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K Duration: 2.04740103s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 57 out of 60 is failed (threshold=3) [...XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.227614277s FAIL (2025-07-26 23:52:14 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 11) Duration: 30.22762657s core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 21 out of 60 is failed (threshold=3) [.XX.XX..XXX............XXXX.....XXX..X.....X......X...XX..XX] duration: 1m0.000547749s FAIL (2025-07-26 23:52:04 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.000567035s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 58 out of 60 is failed (threshold=3) [.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXX] duration: 30.241058145s FAIL (2025-07-26 23:51:56 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== 6520690 (60h15m37.01908375s) [upload=57803]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 23:51:27.296528715 (60h15m37.019091795s) [upload=57803]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:51:27.529487436 (60h15m37.252050476s) [seg_pos={0 0}, upload=57803]: Begin upload segment... Jul 26 23:51:27.529539145 (60h15m37.252102185s) [seg_pos={0 0}, upload=57803]: Done begin upload segment. Jul 26 23:51:27.529572418 (60h15m37.252135458s) [noise=true , address=104.36.82.8:4972 , piece_id=42S6BNK6SD2UXOC2Y7HYXZLHBHINMP7X6KQJJEIABC6REUP4LXOA, seg_pos={0 0}, upload=57803]: Uploading piece... Jul 26 23:51:27.529639536 (60h15m37.252202606s) [noise=true , address=104.36.82.15:4259 , piece_id=WMI2RLTP2YGTJAPXWA7Q74C3ABD6K63GTYJUERKYNKY6EDD27TRA, seg_pos={0 0}, upload=57803]: Uploading piece... Jul 26 23:51:27.530133321 (60h15m37.252696361s) [noise=true , address=104.36.82.3:4765 , piece_id=XWN2JA2I6CHSWEZW2MLE2KGXCX5P7IEMATQAZUSA2P5FJ5VPLKXQ, seg_pos={0 0}, upload=57803]: Uploading piece... Jul 26 23:51:27.530542815 (60h15m37.253105855s) [noise=true , address=77.90.31.19:4842 , piece_id=XB4PB4Z7FFF7Y254HLIMQFMEOWXZMYPLPAJOHVUNDHOP36UQ5C4Q, seg_pos={0 0}, upload=57803]: Uploading piece... Jul 26 23:51:27.531101934 (60h15m37.253664974s) [noise=true , address=104.36.82.7:4910 , piece_id=L36NJRUEDZAYSPGC3YLPY56DNDDEL4E6P3XHDR7FBLZYK7VZXBLQ, seg_pos={0 0}, upload=57803]: Uploading piece... Jul 26 23:51:27.531264475 (60h15m37.253827505s) [noise=true , address=104.36.82.2:4729 , piece_id=D5SDQ7K6RALVBHJZMFE4K6ELIK6PW3LZL6ICXFUVKKMIJBMQ3UHQ, seg_pos={0 0}, upload=57803]: Uploading piece... Jul 26 23:51:27.531490478 (60h15m37.254053518s) [noise=true , address=77.90.31.12:4592 , piece_id=VEIAPKIOFJQSH5KGEN6SKCXY6ELKHXO4VQ5HQMTHXC3VHT6XBZVA, seg_pos={0 0}, upload=57803]: Uploading piece... Jul 26 23:51:27.531782797 (60h15m37.254345837s) [noise=true , address=104.36.82.12:4151 , piece_id=4PSBN45NK3AIUCS3SXLCGD26SMFL7K342QN5UE2QGCFL4YEVNRYQ, seg_pos={0 0}, upload=57803]: Uploading piece... Jul 26 23:51:27.532113440 (60h15m37.25467648s) [noise=true , address=104.36.82.5:4839 , piece_id=MYSZH23XYNZX5CC4URD5CVLSMQZ7P7UIAWNCNEMANGBQR4PHEABA, seg_pos={0 0}, upload=57803]: Uploading piece... Jul 26 23:51:27.532476614 (60h15m37.255039654s) [noise=true , address=45.91.116.5:4298 , piece_id=ZEQQNCLT3FKERBGYPPEMUGBJVQ25N62IKFFPXEWA55CBQ5MILTCQ, seg_pos={0 0}, upload=57803]: Uploading piece... Jul 26 23:51:27.532841613 (60h15m37.255404653s) [noise=true , address=45.91.116.4:4939 , piece_id=A2UKYLAHIZQH2K3NIIM2HWPJEWJYOBJEDYEP6NXUY3EUP72X4NAA, seg_pos={0 0}, upload=57803]: Uploading piece... Jul 26 23:51:27.533132489 (60h15m37.255695529s) [noise=true , address=45.91.116.12:4225 , piece_id=P6WC5CMUNNBHBDWZVOAVYXO5M23T5MVC53AIBZMFJJVOOT3X5CEQ, seg_pos={0 0}, upload=57803]: Uploading piece... Jul 26 23:51:27.533416293 (60h15m37.255979343s) [noise=true , address=45.91.116.10:4082 , piece_id=XEUMK2W5ENY3GX24RNU34KCTQM3XDM5TKMNOZCZPFR2AXNTTPMMA, seg_pos={0 0}, upload=57803]: Uploading piece... Jul 26 23:51:27.533746354 (60h15m37.256309395s) [noise=true , address=77.90.31.17:4770 , piece_id=AKHLVB37MDIAKZXBHLEQ6XJWTZ44RJ6W3YMYVJY5B6PDQRGUOBOA, seg_pos={0 0}, upload=57803]: Uploading piece... Jul 26 23:51:27.534089891 (60h15m37.256652921s) [noise=true , address=45.91.116.7:4009 , piece_id=4U7DXRVFYYRXDHY35U6HZSU4NXV2M3HG7SB5QA74W4F5F55UO5RQ, seg_pos={0 0}, upload=57803]: Uploading piece... Jul 26 23:51:27.534360299 (60h15m37.256923329s) [noise=true , address=77.90.31.16:4733 , piece_id=PTRNMVSV3M5NPRCU6UUTJGLRBW7AYFCROWDP7QI4QMT6CILSDSJQ, seg_pos={0 0}, upload=57803]: Uploading piece... Jul 26 23:51:27.784546280 (60h15m37.50710932s) [noise=true , address=77.90.31.16:4733 , piece_id=PTRNMVSV3M5NPRCU6UUTJGLRBW7AYFCROWDP7QI4QMT6CILSDSJQ, seg_pos={0 0}, upload=57803]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: write tcp 188.166.225.170:40040->77.90.31.16:4733: write: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:51:29.268576606 (60h15m38.991139636s) [upload=57803]: Next returned nil segment Jul 26 23:51:29.268591144 (60h15m38.991154184s) [upload=57803]: Waiting for error group managing segments... Jul 26 23:51:29.543139817 (60h15m39.265702847s) [noise=true , address=45.91.116.12:4225 , piece_id=P6WC5CMUNNBHBDWZVOAVYXO5M23T5MVC53AIBZMFJJVOOT3X5CEQ, seg_pos={0 0}, upload=57803]: Done uploading piece. err: <nil> Jul 26 23:51:29.551341781 (60h15m39.273904821s) [noise=true , address=45.91.116.10:4082 , piece_id=XEUMK2W5ENY3GX24RNU34KCTQM3XDM5TKMNOZCZPFR2AXNTTPMMA, seg_pos={0 0}, upload=57803]: Done uploading piece. err: <nil> Jul 26 23:51:29.557703793 (60h15m39.280266833s) [noise=true , address=45.91.116.4:4939 , piece_id=A2UKYLAHIZQH2K3NIIM2HWPJEWJYOBJEDYEP6NXUY3EUP72X4NAA, seg_pos={0 0}, upload=57803]: Done uploading piece. err: <nil> Jul 26 23:51:29.634773630 (60h15m39.35733667s) [noise=true , address=45.91.116.7:4009 , piece_id=4U7DXRVFYYRXDHY35U6HZSU4NXV2M3HG7SB5QA74W4F5F55UO5RQ, seg_pos={0 0}, upload=57803]: Done uploading piece. err: <nil> Jul 26 23:51:29.784726876 (60h15m39.507289916s) [noise=true , address=45.91.116.5:4298 , piece_id=ZEQQNCLT3FKERBGYPPEMUGBJVQ25N62IKFFPXEWA55CBQ5MILTCQ, seg_pos={0 0}, upload=57803]: Done uploading piece. err: <nil> Jul 26 23:51:31.096376166 (60h15m40.818939206s) [noise=true , address=77.90.31.19:4842 , piece_id=XB4PB4Z7FFF7Y254HLIMQFMEOWXZMYPLPAJOHVUNDHOP36UQ5C4Q, seg_pos={0 0}, upload=57803]: Done uploading piece. err: <nil> Jul 26 23:51:31.735712068 (60h15m41.458275108s) [noise=true , address=77.90.31.12:4592 , piece_id=VEIAPKIOFJQSH5KGEN6SKCXY6ELKHXO4VQ5HQMTHXC3VHT6XBZVA, seg_pos={0 0}, upload=57803]: Done uploading piece. err: <nil> Jul 26 23:51:56.656977872 (60h16m6.379540912s) [noise=true , address=104.36.82.7:4910 , piece_id=L36NJRUEDZAYSPGC3YLPY56DNDDEL4E6P3XHDR7FBLZYK7VZXBLQ, seg_pos={0 0}, upload=57803]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: rpc: dial tcp 104.36.82.7:4910: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:51:56.657157174 (60h16m6.379720204s) [noise=true , address=104.36.82.2:4729 , piece_id=D5SDQ7K6RALVBHJZMFE4K6ELIK6PW3LZL6ICXFUVKKMIJBMQ3UHQ, seg_pos={0 0}, upload=57803]: Done uploading piece. err: ecclient: upload failed (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR, address:104.36.82.2:4729): protocol: rpc: dial tcp 104.36.82.2:4729: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:51:56.657580715 (60h16m6.380143755s) [noise=true , address=104.36.82.15:4259 , piece_id=WMI2RLTP2YGTJAPXWA7Q74C3ABD6K63GTYJUERKYNKY6EDD27TRA, seg_pos={0 0}, upload=57803]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:104.36.82.15:4259): protocol: write tcp 188.166.225.170:33172->104.36.82.15:4259: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:51:56.657679343 (60h16m6.380242394s) [noise=true , address=104.36.82.8:4972 , piece_id=42S6BNK6SD2UXOC2Y7HYXZLHBHINMP7X6KQJJEIABC6REUP4LXOA, seg_pos={0 0}, upload=57803]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:104.36.82.8:4972): protocol: write tcp 188.166.225.170:42656->104.36.82.8:4972: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:51:56.657731263 (60h16m6.380294303s) [noise=true , address=104.36.82.12:4151 , piece_id=4PSBN45NK3AIUCS3SXLCGD26SMFL7K342QN5UE2QGCFL4YEVNRYQ, seg_pos={0 0}, upload=57803]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:104.36.82.12:4151): protocol: write tcp 188.166.225.170:57842->104.36.82.12:4151: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:51:56.657882743 (60h16m6.380445783s) [noise=true , address=77.90.31.17:4770 , piece_id=AKHLVB37MDIAKZXBHLEQ6XJWTZ44RJ6W3YMYVJY5B6PDQRGUOBOA, seg_pos={0 0}, upload=57803]: Done uploading piece. err: ecclient: upload failed (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, address:77.90.31.17:4770): protocol: rpc: dial tcp 77.90.31.17:4770: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:51:56.657989146 (60h16m6.380552186s) [noise=true , address=104.36.82.5:4839 , piece_id=MYSZH23XYNZX5CC4URD5CVLSMQZ7P7UIAWNCNEMANGBQR4PHEABA, seg_pos={0 0}, upload=57803]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:104.36.82.5:4839): protocol: write tcp 188.166.225.170:56348->104.36.82.5:4839: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:51:56.658032880 (60h16m6.38059592s) [noise=true , address=104.36.82.3:4765 , piece_id=XWN2JA2I6CHSWEZW2MLE2KGXCX5P7IEMATQAZUSA2P5FJ5VPLKXQ, seg_pos={0 0}, upload=57803]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:104.36.82.3:4765): protocol: write tcp 188.166.225.170:36474->104.36.82.3:4765: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:51:56.658058209 (60h16m6.380621249s) [seg_pos={0 0}, upload=57803]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Jul 26 23:51:56.895999731 (60h16m6.618562771s) [upload=57803]: Done uploading segments... Jul 26 23:51:56.896028326 (60h16m6.618591366s) [upload=57803]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration: 30.241078464s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 1.923997917s FAIL (2025-07-26 23:51:48 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K Duration: 1.924023445s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 56 out of 60 is failed (threshold=3) [....XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.250466774s FAIL (2025-07-26 23:51:44 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 11) Duration: 30.250497944s core uplink us1 vivint
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [.......XXX.X..X.....XXXXX...XXX.X.....X.X..X..X.X....X.X....] duration: 19.466147237s FAIL (2025-07-26 23:51:32 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 19.466165934s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [X..........................................................X] duration: 1.378608611s FAIL (2025-07-26 23:51:28 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K Duration: 1.378634102s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 57 out of 60 is failed (threshold=3) [X...XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.244564102s FAIL (2025-07-26 23:51:24 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== 55:4216 , piece_id=WARKAH4EZBBF7PFADYBF7VU5OEQZBNPXI4KYN4DAL2O5KQEWUJNA, seg_pos={0 0}, upload=57789]: Uploading piece... Jul 26 23:50:25.714419246 (60h14m35.436982286s) [noise=true , address=77.90.31.19:4871 , piece_id=Y67WCCNSGN45NG2NM4D5CZW2XXV776FOG3ZG5H7L4OAKXTSIC5TQ, seg_pos={0 0}, upload=57789]: Uploading piece... Jul 26 23:50:25.714713319 (60h14m35.437276359s) [noise=true , address=104.36.82.53:4133 , piece_id=BV4FIT75DHXTSEUKFWSHFGSPV3MIEEJUZVYDU5BR2WFCFICKLC4A, seg_pos={0 0}, upload=57789]: Uploading piece... Jul 26 23:50:25.714841153 (60h14m35.437404193s) [noise=true , address=77.90.31.18:4812 , piece_id=Q2A3B6RGZBXPPY7ZTXKOV76DJQBNGAVLT2VAW7MLFBGG3X5DGL7A, seg_pos={0 0}, upload=57789]: Uploading piece... Jul 26 23:50:25.715000418 (60h14m35.437563458s) [noise=true , address=77.90.31.19:4875 , piece_id=BBRXL6VVGFBIRB2TOJOJ5PE7Y6FJ2MUROHLJAQFTBKZGYXEM2ILA, seg_pos={0 0}, upload=57789]: Uploading piece... Jul 26 23:50:25.715110679 (60h14m35.437673719s) [noise=true , address=79.127.219.47:4048 , piece_id=UVTKJFBGID3WYM5C2QDQWZMF3ULPIXBLFHR5NYTKUY7NXQG466LA, seg_pos={0 0}, upload=57789]: Uploading piece... Jul 26 23:50:25.715292257 (60h14m35.437855287s) [noise=true , address=77.90.31.15:4713 , piece_id=JMQ4JST7YMZMII4SO6XK7TAAWXQR3C2B4AF2FGJUX6GLAZKFAY3Q, seg_pos={0 0}, upload=57789]: Uploading piece... Jul 26 23:50:25.715370216 (60h14m35.437933256s) [noise=true , address=64.49.13.139:4478 , piece_id=HIOLEGWTH45S3LSOZCL3SWSYRCRUH3MTMJBINU4Q5TGWEV5REXNA, seg_pos={0 0}, upload=57789]: Uploading piece... Jul 26 23:50:25.715556191 (60h14m35.438119231s) [noise=true , address=79.127.201.215:4343 , piece_id=Q7X3JTJTNUIHJWLTDAINP5N6MEDEOTLSTNL77JIKF3GO3J7HS3DA, seg_pos={0 0}, upload=57789]: Uploading piece... Jul 26 23:50:25.715617409 (60h14m35.438180449s) [noise=true , address=104.36.82.54:4185 , piece_id=2ABULXGDLWUF3C26ISRIAH7WUB65AODANMFAE65JJZA6EY27C44Q, seg_pos={0 0}, upload=57789]: Uploading piece... Jul 26 23:50:25.715761384 (60h14m35.438324424s) [noise=true , address=79.127.219.38:4006 , piece_id=UUYH7KNHTNSEEAVAFWJ4K5BJNIPHNRXBS7UBKD6M5DZQ77I2KN2A, seg_pos={0 0}, upload=57789]: Uploading piece... Jul 26 23:50:25.715828613 (60h14m35.438391653s) [noise=true , address=77.90.31.19:4867 , piece_id=WRA22BV33O3PUNRZCHIZVL5IJ7RFPUZD5XTECVNRCW3B22W64XFA, seg_pos={0 0}, upload=57789]: Uploading piece... Jul 26 23:50:25.715965234 (60h14m35.438528264s) [noise=true , address=104.36.82.10:4111 , piece_id=6UL4A2OVJYSGTDXLTOSSFGTKK3RWOFVL4TADOSIQDMVO6WHZJE7A, seg_pos={0 0}, upload=57789]: Uploading piece... Jul 26 23:50:25.716048373 (60h14m35.438611403s) [noise=true , address=79.127.205.234:4606 , piece_id=AIDQQVTKC6STT4GGYIJS7XD2MQW2WXGIG777DGRNTLNQXLXOQNPQ, seg_pos={0 0}, upload=57789]: Uploading piece... Jul 26 23:50:25.716196477 (60h14m35.438759517s) [noise=true , address=104.36.82.52:4089 , piece_id=2HMLVYIATCF6BT2P6N4XR7DEBCKC6SYOD4PXLADM3DBHDRAHNV6A, seg_pos={0 0}, upload=57789]: Uploading piece... Jul 26 23:50:25.716276069 (60h14m35.438839109s) [noise=true , address=104.36.82.56:4243 , piece_id=RH6QFS6I7ULX7UGS5I5SKGSAC6EF3WCLU2R523ND3ZF4VPJUHTOA, seg_pos={0 0}, upload=57789]: Uploading piece... Jul 26 23:50:25.716415867 (60h14m35.438978907s) [noise=true , address=79.127.205.245:4238 , piece_id=C3DRPTTTZ6BXGZQHI4UFYA6OHXBQSLPPSKSATIWQU6RAKG75WGZQ, seg_pos={0 0}, upload=57789]: Uploading piece... Jul 26 23:50:25.716627050 (60h14m35.43919009s) [noise=true , address=104.36.82.46:4513 , piece_id=Q75EESO3BE4XOZN6E72SFO76CN6X4SZ4CXOIFEUXYWHW3EZEDWDA, seg_pos={0 0}, upload=57789]: Uploading piece... Jul 26 23:50:26.850510150 (60h14m36.57307318s) [upload=57789]: Next returned nil segment Jul 26 23:50:26.850524116 (60h14m36.573087156s) [upload=57789]: Waiting for error group managing segments... Jul 26 23:50:27.147311530 (60h14m36.86987457s) [noise=true , address=79.127.205.245:4238 , piece_id=C3DRPTTTZ6BXGZQHI4UFYA6OHXBQSLPPSKSATIWQU6RAKG75WGZQ, seg_pos={0 0}, upload=57789]: Done uploading piece. err: <nil> Jul 26 23:50:27.528012504 (60h14m37.250575534s) [noise=true , address=79.127.205.234:4606 , piece_id=AIDQQVTKC6STT4GGYIJS7XD2MQW2WXGIG777DGRNTLNQXLXOQNPQ, seg_pos={0 0}, upload=57789]: Done uploading piece. err: <nil> Jul 26 23:50:27.655696593 (60h14m37.378259633s) [noise=true , address=77.90.31.15:4713 , piece_id=JMQ4JST7YMZMII4SO6XK7TAAWXQR3C2B4AF2FGJUX6GLAZKFAY3Q, seg_pos={0 0}, upload=57789]: Done uploading piece. err: <nil> Jul 26 23:50:27.923518987 (60h14m37.646082027s) [noise=true , address=79.127.219.38:4006 , piece_id=UUYH7KNHTNSEEAVAFWJ4K5BJNIPHNRXBS7UBKD6M5DZQ77I2KN2A, seg_pos={0 0}, upload=57789]: Done uploading piece. err: <nil> Jul 26 23:50:29.929070609 (60h14m39.651633649s) [noise=true , address=79.127.219.47:4048 , piece_id=UVTKJFBGID3WYM5C2QDQWZMF3ULPIXBLFHR5NYTKUY7NXQG466LA, seg_pos={0 0}, upload=57789]: Done uploading piece. err: <nil> Jul 26 23:50:32.945213466 (60h14m42.667776496s) [noise=true , address=77.90.31.19:4871 , piece_id=Y67WCCNSGN45NG2NM4D5CZW2XXV776FOG3ZG5H7L4OAKXTSIC5TQ, seg_pos={0 0}, upload=57789]: Done uploading piece. err: <nil> Jul 26 23:50:35.166042209 (60h14m44.88860527s) [noise=true , address=64.49.13.139:4478 , piece_id=HIOLEGWTH45S3LSOZCL3SWSYRCRUH3MTMJBINU4Q5TGWEV5REXNA, seg_pos={0 0}, upload=57789]: Done uploading piece. err: <nil> Jul 26 23:50:37.182142054 (60h14m46.904705094s) [noise=true , address=77.90.31.19:4867 , piece_id=WRA22BV33O3PUNRZCHIZVL5IJ7RFPUZD5XTECVNRCW3B22W64XFA, seg_pos={0 0}, upload=57789]: Done uploading piece. err: <nil> Jul 26 23:50:43.857120243 (60h14m53.579683283s) [noise=true , address=77.90.31.19:4875 , piece_id=BBRXL6VVGFBIRB2TOJOJ5PE7Y6FJ2MUROHLJAQFTBKZGYXEM2ILA, seg_pos={0 0}, upload=57789]: Done uploading piece. err: <nil> Jul 26 23:51:06.314212932 (60h15m16.036775972s) [noise=true , address=79.127.201.215:4343 , piece_id=Q7X3JTJTNUIHJWLTDAINP5N6MEDEOTLSTNL77JIKF3GO3J7HS3DA, seg_pos={0 0}, upload=57789]: Done uploading piece. err: <nil> Jul 26 23:51:24.626829479 (60h15m34.349392519s) [noise=true , address=104.36.82.46:4513 , piece_id=Q75EESO3BE4XOZN6E72SFO76CN6X4SZ4CXOIFEUXYWHW3EZEDWDA, seg_pos={0 0}, upload=57789]: Done uploading piece. err: ecclient: upload failed (node:12pyyW8Gdqg6j9YGMtCUBhFnoNXryvPMUy19x815dp2HV3Xqr8h, address:104.36.82.46:4513): protocol: rpc: dial tcp 104.36.82.46:4513: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:51:24.626875486 (60h15m34.349438547s) [noise=true , address=104.36.82.55:4216 , piece_id=WARKAH4EZBBF7PFADYBF7VU5OEQZBNPXI4KYN4DAL2O5KQEWUJNA, seg_pos={0 0}, upload=57789]: Done uploading piece. err: ecclient: upload failed (node:12vYNo8tVKDfSphJzsryTLhezUDVw3DA1NaK7ZXcXYcEw4pf1Pt, address:104.36.82.55:4216): protocol: rpc: dial tcp 104.36.82.55:4216: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:51:24.626917708 (60h15m34.349480758s) [noise=true , address=104.36.82.10:4111 , piece_id=6UL4A2OVJYSGTDXLTOSSFGTKK3RWOFVL4TADOSIQDMVO6WHZJE7A, seg_pos={0 0}, upload=57789]: Done uploading piece. err: ecclient: upload failed (node:128oYxFebZ8HAoUNDf2Tmqi23Q1kuo2XgzWxKVRCGwaiuZBWtAw, address:104.36.82.10:4111): protocol: rpc: dial tcp 104.36.82.10:4111: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:51:24.626947895 (60h15m34.349510976s) [noise=true , address=104.36.82.54:4185 , piece_id=2ABULXGDLWUF3C26ISRIAH7WUB65AODANMFAE65JJZA6EY27C44Q, seg_pos={0 0}, upload=57789]: Done uploading piece. err: ecclient: upload failed (node:1cLdyyVLrp1yBmgsfBrfYc6N2RFbBichYoBbo2F7EZfaBXtkfk, address:104.36.82.54:4185): protocol: rpc: dial tcp 104.36.82.54:4185: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:51:24.626990177 (60h15m34.349553217s) [noise=true , address=104.36.82.53:4133 , piece_id=BV4FIT75DHXTSEUKFWSHFGSPV3MIEEJUZVYDU5BR2WFCFICKLC4A, seg_pos={0 0}, upload=57789]: Done uploading piece. err: ecclient: upload failed (node:12usLZNRYviwS7dfk6soCgE7XP5LeyVcSUpjhWAk3stm61tjR7g, address:104.36.82.53:4133): protocol: write tcp 188.166.225.170:45056->104.36.82.53:4133: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:51:24.627018842 (60h15m34.349581902s) [noise=true , address=104.36.82.52:4089 , piece_id=2HMLVYIATCF6BT2P6N4XR7DEBCKC6SYOD4PXLADM3DBHDRAHNV6A, seg_pos={0 0}, upload=57789]: Done uploading piece. err: ecclient: upload failed (node:12L2DTyJJ3v7KRjA424PVwQn9mpeq5Pq4ZTCerhrStgqgc3cVFk, address:104.36.82.52:4089): protocol: write tcp 188.166.225.170:48734->104.36.82.52:4089: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:51:24.627034331 (60h15m34.349597391s) [noise=true , address=77.90.31.18:4812 , piece_id=Q2A3B6RGZBXPPY7ZTXKOV76DJQBNGAVLT2VAW7MLFBGG3X5DGL7A, seg_pos={0 0}, upload=57789]: Done uploading piece. err: ecclient: upload failed (node:128qNJJSSwMBPvN8wmTcUeMWAJxVmpjWjwA6WTKxBNCmT2FdFJn, address:77.90.31.18:4812): protocol: rpc: dial tcp 77.90.31.18:4812: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:51:24.627059148 (60h15m34.349622188s) [noise=true , address=104.36.82.56:4243 , piece_id=RH6QFS6I7ULX7UGS5I5SKGSAC6EF3WCLU2R523ND3ZF4VPJUHTOA, seg_pos={0 0}, upload=57789]: Done uploading piece. err: ecclient: upload failed (node:12Fyogwa9NcRSUwxTPFTGL68Dbn1FEAqs8K48CNtTyKQEgVwY1c, address:104.36.82.56:4243): protocol: rpc: dial tcp 104.36.82.56:4243: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:51:24.627088665 (60h15m34.349651705s) [seg_pos={0 0}, upload=57789]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Jul 26 23:51:24.869401292 (60h15m34.591964342s) [upload=57789]: Done uploading segments... Jul 26 23:51:24.869449484 (60h15m34.592012524s) [upload=57789]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 1m0.244583229s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 55 out of 60 is failed (threshold=3) [.....XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.233442722s FAIL (2025-07-26 23:51:09 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 11) Duration: 30.23350346s core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 20 out of 60 is failed (threshold=3) [..XX.XX..XXX............XXXX.....XXX..X.....X......X...XX..X] duration: 1m0.000506221s FAIL (2025-07-26 23:50:59 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.000528383s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 57 out of 60 is failed (threshold=3) [..XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXX] duration: 30.235341783s FAIL (2025-07-26 23:50:57 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== 47746 (60h14m13.616810796s) [seg_pos={0 0}, upload=57776]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Jul 26 23:50:04.128677420 (60h14m13.851240491s) [upload=57776]: Done uploading segments... Jul 26 23:50:04.128712968 (60h14m13.851276018s) [upload=57776]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Jul 26 23:50:27.630267198 (60h14m37.352830238s) [upload=57791]: Starting upload Jul 26 23:50:27.630337093 (60h14m37.352900133s) [upload=57791]: Done starting upload Jul 26 23:50:27.630430070 (60h14m37.35299311s) [upload=57791]: Uploading segments... Jul 26 23:50:27.630576790 (60h14m37.35313983s) [upload=57791]: Got next segment. Inline: false Jul 26 23:50:27.630628519 (60h14m37.35319155s) [upload=57791]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 23:50:27.630633519 (60h14m37.353196549s) [upload=57791]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:50:27.864919278 (60h14m37.587482318s) [seg_pos={0 0}, upload=57791]: Begin upload segment... Jul 26 23:50:27.864978522 (60h14m37.587541562s) [noise=true , address=45.91.116.10:4082 , piece_id=6N7ALVSE7RKXLIQITQYT2OZPI2UCMFQ4YBOUX36IS3DTSCGSGHVA, seg_pos={0 0}, upload=57791]: Uploading piece... Jul 26 23:50:27.865012477 (60h14m37.587575517s) [noise=true , address=45.91.116.10:4083 , piece_id=XQUVOBA2BPMK5FDDPY5YBECZLGC6AAHKNKHXCA5UXU2WFJXZ73EQ, seg_pos={0 0}, upload=57791]: Uploading piece... Jul 26 23:50:27.865338110 (60h14m37.58790115s) [noise=true , address=104.36.82.15:4259 , piece_id=YFRLCI4J45R2NBCP26CCYFCAYOOGMAJPPOTCZO24DEJU7I55HWHA, seg_pos={0 0}, upload=57791]: Uploading piece... Jul 26 23:50:27.865504959 (60h14m37.588067999s) [noise=true , address=77.90.31.13:4628 , piece_id=5HDNSHLSHW2RQ6BL4J3PLSVW5GE647OXKK77R4OWEUZ7RFMGD23A, seg_pos={0 0}, upload=57791]: Uploading piece... Jul 26 23:50:27.865581144 (60h14m37.588144184s) [seg_pos={0 0}, upload=57791]: Done begin upload segment. Jul 26 23:50:27.865956142 (60h14m37.588519182s) [noise=true , address=77.90.31.14:4661 , piece_id=HB56HDXK52WHICSHVTLEU5WMEMMIM2VPH2YPUFQUMUTXOW32KENA, seg_pos={0 0}, upload=57791]: Uploading piece... Jul 26 23:50:27.866296914 (60h14m37.588859954s) [noise=true , address=77.90.31.19:4842 , piece_id=NUZTP4UWWOUCQ6OGUC7K4W7DPOPZ25VVYOWUST5CW67EPAKUSPIA, seg_pos={0 0}, upload=57791]: Uploading piece... Jul 26 23:50:27.867658640 (60h14m37.59022168s) [noise=true , address=104.36.82.13:4185 , piece_id=BLODFMRHW4JYGFR4IMHFIOQUB7ANVFTTNTBCL3HANKFL4EVDB66A, seg_pos={0 0}, upload=57791]: Uploading piece... Jul 26 23:50:27.867956700 (60h14m37.59051974s) [noise=true , address=45.91.116.8:4045 , piece_id=WRQGUOFVOGFX5AIMFLPG3VRWPTZYYDHWASIZD4I3BR2Z54546PLA, seg_pos={0 0}, upload=57791]: Uploading piece... Jul 26 23:50:27.868303132 (60h14m37.590866172s) [noise=true , address=104.36.82.16:4294 , piece_id=5D4RC7DIWAL4HSRCZXJGPAS5IEVORTPCZ2GSNQ4HZMC7SNAH3ZJQ, seg_pos={0 0}, upload=57791]: Uploading piece... Jul 26 23:50:27.868619318 (60h14m37.591182358s) [noise=true , address=104.36.82.3:4765 , piece_id=OG2UDQ7Z5DWYDLSAAUCAQRKWIYNIMOUEZ47ACNVXOQVXNN2TAMPQ, seg_pos={0 0}, upload=57791]: Uploading piece... Jul 26 23:50:27.868970059 (60h14m37.591533109s) [noise=true , address=77.90.31.16:4733 , piece_id=CWY4BMAUJ5ORA7JYMTJH7GFHMS2Z4H6L2MNVVS5B4FDQIWN73VYA, seg_pos={0 0}, upload=57791]: Uploading piece... Jul 26 23:50:27.869266185 (60h14m37.591829225s) [noise=true , address=45.91.116.5:4299 , piece_id=LCAZBCMT3BUSFBJLZZCT2UXFVTZIEEQ5WZ5JREV4YFOZVDOXXVBQ, seg_pos={0 0}, upload=57791]: Uploading piece... Jul 26 23:50:27.869653566 (60h14m37.592216606s) [noise=true , address=104.36.82.4:4801 , piece_id=NEWTAXU6F434WD2NI6PXGVWKXJAFFEM6P6MRP5USX4HEDVXNJV4Q, seg_pos={0 0}, upload=57791]: Uploading piece... Jul 26 23:50:27.869940375 (60h14m37.592503415s) [noise=true , address=45.91.116.13:4153 , piece_id=XHYDTD4YNU5Z2RPX4GZSMJG66LT3PMRMGIUCVL52UVTF3BJKFQ3A, seg_pos={0 0}, upload=57791]: Uploading piece... Jul 26 23:50:27.870244738 (60h14m37.592807778s) [noise=true , address=45.91.116.4:4937 , piece_id=7M3LRS5WSN6DE2NRU2OR5HX775SZ3W4T7HPHNRULLZCYLX5D7RXA, seg_pos={0 0}, upload=57791]: Uploading piece... Jul 26 23:50:27.870489847 (60h14m37.593052887s) [noise=true , address=77.90.31.17:4770 , piece_id=RCOQDCFC3RLHLRPQWYNXTHIXQL5RYUXMNWIDZAGTJFLJNKF5UCGA, seg_pos={0 0}, upload=57791]: Uploading piece... Jul 26 23:50:28.122341304 (60h14m37.844904344s) [noise=true , address=77.90.31.16:4733 , piece_id=CWY4BMAUJ5ORA7JYMTJH7GFHMS2Z4H6L2MNVVS5B4FDQIWN73VYA, seg_pos={0 0}, upload=57791]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: write tcp 188.166.225.170:57908->77.90.31.16:4733: write: connection refused; manager closed: conn closed: op Jul 26 23:50:29.608714001 (60h14m39.331277041s) [upload=57791]: Next returned nil segment Jul 26 23:50:29.608731585 (60h14m39.331294625s) [upload=57791]: Waiting for error group managing segments... Jul 26 23:50:29.889332038 (60h14m39.611895128s) [noise=true , address=45.91.116.10:4082 , piece_id=6N7ALVSE7RKXLIQITQYT2OZPI2UCMFQ4YBOUX36IS3DTSCGSGHVA, seg_pos={0 0}, upload=57791]: Done uploading piece. err: <nil> Jul 26 23:50:29.900629874 (60h14m39.623192914s) [noise=true , address=45.91.116.4:4937 , piece_id=7M3LRS5WSN6DE2NRU2OR5HX775SZ3W4T7HPHNRULLZCYLX5D7RXA, seg_pos={0 0}, upload=57791]: Done uploading piece. err: <nil> Jul 26 23:50:29.964734245 (60h14m39.687297275s) [noise=true , address=45.91.116.8:4045 , piece_id=WRQGUOFVOGFX5AIMFLPG3VRWPTZYYDHWASIZD4I3BR2Z54546PLA, seg_pos={0 0}, upload=57791]: Done uploading piece. err: <nil> Jul 26 23:50:29.965112979 (60h14m39.68767602s) [noise=true , address=45.91.116.10:4083 , piece_id=XQUVOBA2BPMK5FDDPY5YBECZLGC6AAHKNKHXCA5UXU2WFJXZ73EQ, seg_pos={0 0}, upload=57791]: Done uploading piece. err: <nil> Jul 26 23:50:30.126374281 (60h14m39.848937321s) [noise=true , address=77.90.31.14:4661 , piece_id=HB56HDXK52WHICSHVTLEU5WMEMMIM2VPH2YPUFQUMUTXOW32KENA, seg_pos={0 0}, upload=57791]: Done uploading piece. err: <nil> Jul 26 23:50:30.153277655 (60h14m39.875840695s) [noise=true , address=45.91.116.5:4299 , piece_id=LCAZBCMT3BUSFBJLZZCT2UXFVTZIEEQ5WZ5JREV4YFOZVDOXXVBQ, seg_pos={0 0}, upload=57791]: Done uploading piece. err: <nil> Jul 26 23:50:30.199141486 (60h14m39.921704526s) [noise=true , address=45.91.116.13:4153 , piece_id=XHYDTD4YNU5Z2RPX4GZSMJG66LT3PMRMGIUCVL52UVTF3BJKFQ3A, seg_pos={0 0}, upload=57791]: Done uploading piece. err: <nil> Jul 26 23:50:31.249255800 (60h14m40.971818881s) [noise=true , address=77.90.31.19:4842 , piece_id=NUZTP4UWWOUCQ6OGUC7K4W7DPOPZ25VVYOWUST5CW67EPAKUSPIA, seg_pos={0 0}, upload=57791]: Done uploading piece. err: <nil> Jul 26 23:50:31.487298428 (60h14m41.209861468s) [noise=true , address=77.90.31.17:4770 , piece_id=RCOQDCFC3RLHLRPQWYNXTHIXQL5RYUXMNWIDZAGTJFLJNKF5UCGA, seg_pos={0 0}, upload=57791]: Done uploading piece. err: <nil> Jul 26 23:50:31.831075386 (60h14m41.553638426s) [noise=true , address=77.90.31.13:4628 , piece_id=5HDNSHLSHW2RQ6BL4J3PLSVW5GE647OXKK77R4OWEUZ7RFMGD23A, seg_pos={0 0}, upload=57791]: Done uploading piece. err: <nil> Jul 26 23:50:56.983182635 (60h15m6.705745675s) [noise=true , address=104.36.82.3:4765 , piece_id=OG2UDQ7Z5DWYDLSAAUCAQRKWIYNIMOUEZ47ACNVXOQVXNN2TAMPQ, seg_pos={0 0}, upload=57791]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:104.36.82.3:4765): protocol: write tcp 188.166.225.170:57544->104.36.82.3:4765: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:50:56.983365064 (60h15m6.705928104s) [noise=true , address=104.36.82.4:4801 , piece_id=NEWTAXU6F434WD2NI6PXGVWKXJAFFEM6P6MRP5USX4HEDVXNJV4Q, seg_pos={0 0}, upload=57791]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: write tcp 188.166.225.170:42106->104.36.82.4:4801: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:50:56.983410019 (60h15m6.705973059s) [noise=true , address=104.36.82.16:4294 , piece_id=5D4RC7DIWAL4HSRCZXJGPAS5IEVORTPCZ2GSNQ4HZMC7SNAH3ZJQ, seg_pos={0 0}, upload=57791]: Done uploading piece. err: ecclient: upload failed (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, address:104.36.82.16:4294): protocol: rpc: dial tcp 104.36.82.16:4294: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:50:56.983421772 (60h15m6.705984813s) [noise=true , address=104.36.82.15:4259 , piece_id=YFRLCI4J45R2NBCP26CCYFCAYOOGMAJPPOTCZO24DEJU7I55HWHA, seg_pos={0 0}, upload=57791]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:104.36.82.15:4259): protocol: write tcp 188.166.225.170:48690->104.36.82.15:4259: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:50:56.983448003 (60h15m6.706011043s) [noise=true , address=104.36.82.13:4185 , piece_id=BLODFMRHW4JYGFR4IMHFIOQUB7ANVFTTNTBCL3HANKFL4EVDB66A, seg_pos={0 0}, upload=57791]: Done uploading piece. err: ecclient: upload failed (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN, address:104.36.82.13:4185): protocol: write tcp 188.166.225.170:60984->104.36.82.13:4185: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:50:56.983476988 (60h15m6.706040028s) [seg_pos={0 0}, upload=57791]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Jul 26 23:50:57.216586257 (60h15m6.939149307s) [upload=57791]: Done uploading segments... Jul 26 23:50:57.216617727 (60h15m6.939180787s) [upload=57791]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 30.235355489s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [........XXX.X..X.....XXXXX...XXX.X.....X.X..X..X.X....X.X...] duration: 6.154974183s FAIL (2025-07-26 23:50:52 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.154994942s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 57 out of 60 is failed (threshold=3) [XX...XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.233104269s FAIL (2025-07-26 23:50:17 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== t segment. Inline: false Jul 26 23:49:17.737374442 (60h13m27.459937482s) [upload=57772]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 23:49:17.737378681 (60h13m27.459941721s) [upload=57772]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:49:18.181093811 (60h13m27.903656841s) [seg_pos={0 0}, upload=57772]: Begin upload segment... Jul 26 23:49:18.181156540 (60h13m27.903719611s) [seg_pos={0 0}, upload=57772]: Done begin upload segment. Jul 26 23:49:18.181194864 (60h13m27.903757944s) [noise=true , address=77.90.31.10:4540 , piece_id=LDCVDX3KIGIAZHETWKGTVNYUFIT6O24LIJYUVWA4GS4IWBPSMS2Q, seg_pos={0 0}, upload=57772]: Uploading piece... Jul 26 23:49:18.181231654 (60h13m27.903794694s) [noise=true , address=79.127.205.245:4240 , piece_id=TNTPE6S2RKIUEERL6LEYSCTZY2I3J4GCJON3PJNA3CEZNECJWJ6Q, seg_pos={0 0}, upload=57772]: Uploading piece... Jul 26 23:49:18.181547128 (60h13m27.904110168s) [noise=true , address=79.127.219.38:4018 , piece_id=CR5VEOG2GLWIU42L46ER2RTTSZPPNLPZKWJYPRUH2Z7GIUWDIBTQ, seg_pos={0 0}, upload=57772]: Uploading piece... Jul 26 23:49:18.181662118 (60h13m27.904225158s) [noise=true , address=45.91.116.10:4093 , piece_id=DVWRCX3PPFPXVUTUHDRJZDWBCXM743WZJET4OG7H34CDCBE47VWA, seg_pos={0 0}, upload=57772]: Uploading piece... Jul 26 23:49:18.181880034 (60h13m27.904443064s) [noise=true , address=79.127.205.235:4684 , piece_id=JHTATC4OFXHATUGQ4V5QSWJ2I3PPGOOEV7Q463E22DMNOH5ODIWA, seg_pos={0 0}, upload=57772]: Uploading piece... Jul 26 23:49:18.181944919 (60h13m27.904507959s) [noise=true , address=104.36.82.47:4552 , piece_id=7M7D53CDHTJYRNWU62M74IKP3W553O4L2CTIEJNMLM3BAU6UJBYA, seg_pos={0 0}, upload=57772]: Uploading piece... Jul 26 23:49:18.182166132 (60h13m27.904729172s) [noise=true , address=79.127.205.233:4640 , piece_id=7R3LL624HPHDQ4VYTWVMDHN3CA7QFJIJPUGCNQE4AJFVIHOFCEVQ, seg_pos={0 0}, upload=57772]: Uploading piece... Jul 26 23:49:18.182236847 (60h13m27.904799887s) [noise=true , address=77.90.31.21:4896 , piece_id=NHFJFIQTL44IY54IK7BU6DS56M64K7SQDOENZPGRNYYOBH24YT7Q, seg_pos={0 0}, upload=57772]: Uploading piece... Jul 26 23:49:18.182414508 (60h13m27.904977548s) [noise=true , address=104.36.82.53:4131 , piece_id=LZZWE6ZHNV3DIZFUJV5JYKCSM4OAL4MDCXJCEQCPFCLM2MAF552Q, seg_pos={0 0}, upload=57772]: Uploading piece... Jul 26 23:49:18.182500302 (60h13m27.905063342s) [noise=true , address=104.36.82.46:4492 , piece_id=BGYEX5N4SLAHZDLYQYIKB22OUUWSO5QYC22KFFWBALQT3C3XYNDA, seg_pos={0 0}, upload=57772]: Uploading piece... Jul 26 23:49:18.182672240 (60h13m27.90523528s) [noise=true , address=79.127.205.234:4601 , piece_id=LEVOMFARRBMTQ4GQMXG7FQ6HFPYN3Y5VHZACN6XCHPSYRFOQRJ5Q, seg_pos={0 0}, upload=57772]: Uploading piece... Jul 26 23:49:18.182751512 (60h13m27.905314552s) [noise=true , address=45.91.116.9:4277 , piece_id=6XVA4CLXUZEQ2ABZURXWXXB6FA5EWON3KBSS7VE6U6KB3KXPFRKQ, seg_pos={0 0}, upload=57772]: Uploading piece... Jul 26 23:49:18.182880910 (60h13m27.90544395s) [noise=true , address=77.90.31.16:4737 , piece_id=UD3QASQ6JJDPLZY3KA5HKHA4TTCFW3R3ZKX6PXBGNDGTGOIYUM2Q, seg_pos={0 0}, upload=57772]: Uploading piece... Jul 26 23:49:18.182969650 (60h13m27.90553269s) [noise=true , address=104.36.82.38:4715 , piece_id=ITVPZMXMD4OIVMY25YB7TZS7XWIX62GZXVB2YFRAMMF36XIPJKWA, seg_pos={0 0}, upload=57772]: Uploading piece... Jul 26 23:49:18.183084499 (60h13m27.905647539s) [noise=true , address=79.127.205.246:4283 , piece_id=RVQWJGLJ5HKGQM5RSVBQDVOKAQWXJSRUVWA5Q5BGEJ5CQVUHJ42A, seg_pos={0 0}, upload=57772]: Uploading piece... Jul 26 23:49:18.183195792 (60h13m27.905758832s) [noise=true , address=64.49.13.140:4508 , piece_id=CP4KLEZFK74V5SIYZ72INZUGDEGPPOZ22RQIMBNCFYWGDKRNQY5A, seg_pos={0 0}, upload=57772]: Uploading piece... Jul 26 23:49:18.183290594 (60h13m27.905853634s) [noise=true , address=64.49.13.138:4483 , piece_id=TDEA67BAEYGFJKTL2LRGTHAOBHZPVWJE2ZK3M5VZNCJIUWU6GBOA, seg_pos={0 0}, upload=57772]: Uploading piece... Jul 26 23:49:18.183488993 (60h13m27.906052033s) [noise=true , address=104.36.82.54:4164 , piece_id=HDDDDWBJXUOU63VTXOROL6ITYRXMKIYZJSLXHBHGIYKGYURPMNZQ, seg_pos={0 0}, upload=57772]: Uploading piece... Jul 26 23:49:19.286588874 (60h13m29.009151914s) [upload=57772]: Next returned nil segment Jul 26 23:49:19.286603623 (60h13m29.009166663s) [upload=57772]: Waiting for error group managing segments... Jul 26 23:49:19.579924503 (60h13m29.302487533s) [noise=true , address=79.127.205.246:4283 , piece_id=RVQWJGLJ5HKGQM5RSVBQDVOKAQWXJSRUVWA5Q5BGEJ5CQVUHJ42A, seg_pos={0 0}, upload=57772]: Done uploading piece. err: <nil> Jul 26 23:49:19.606866501 (60h13m29.329429541s) [noise=true , address=79.127.205.233:4640 , piece_id=7R3LL624HPHDQ4VYTWVMDHN3CA7QFJIJPUGCNQE4AJFVIHOFCEVQ, seg_pos={0 0}, upload=57772]: Done uploading piece. err: <nil> Jul 26 23:49:19.608725498 (60h13m29.331288528s) [noise=true , address=79.127.205.234:4601 , piece_id=LEVOMFARRBMTQ4GQMXG7FQ6HFPYN3Y5VHZACN6XCHPSYRFOQRJ5Q, seg_pos={0 0}, upload=57772]: Done uploading piece. err: <nil> Jul 26 23:49:19.715244832 (60h13m29.437807862s) [noise=true , address=79.127.205.235:4684 , piece_id=JHTATC4OFXHATUGQ4V5QSWJ2I3PPGOOEV7Q463E22DMNOH5ODIWA, seg_pos={0 0}, upload=57772]: Done uploading piece. err: <nil> Jul 26 23:49:19.799476091 (60h13m29.522039131s) [noise=true , address=79.127.205.245:4240 , piece_id=TNTPE6S2RKIUEERL6LEYSCTZY2I3J4GCJON3PJNA3CEZNECJWJ6Q, seg_pos={0 0}, upload=57772]: Done uploading piece. err: <nil> Jul 26 23:49:20.065322155 (60h13m29.787885195s) [noise=true , address=77.90.31.10:4540 , piece_id=LDCVDX3KIGIAZHETWKGTVNYUFIT6O24LIJYUVWA4GS4IWBPSMS2Q, seg_pos={0 0}, upload=57772]: Done uploading piece. err: <nil> Jul 26 23:49:20.156927764 (60h13m29.879490804s) [noise=true , address=79.127.219.38:4018 , piece_id=CR5VEOG2GLWIU42L46ER2RTTSZPPNLPZKWJYPRUH2Z7GIUWDIBTQ, seg_pos={0 0}, upload=57772]: Done uploading piece. err: <nil> Jul 26 23:49:20.225711848 (60h13m29.948274898s) [noise=true , address=77.90.31.21:4896 , piece_id=NHFJFIQTL44IY54IK7BU6DS56M64K7SQDOENZPGRNYYOBH24YT7Q, seg_pos={0 0}, upload=57772]: Done uploading piece. err: <nil> Jul 26 23:49:20.307120107 (60h13m30.029683147s) [noise=true , address=45.91.116.9:4277 , piece_id=6XVA4CLXUZEQ2ABZURXWXXB6FA5EWON3KBSS7VE6U6KB3KXPFRKQ, seg_pos={0 0}, upload=57772]: Done uploading piece. err: <nil> Jul 26 23:49:21.535458487 (60h13m31.258021527s) [noise=true , address=64.49.13.138:4483 , piece_id=TDEA67BAEYGFJKTL2LRGTHAOBHZPVWJE2ZK3M5VZNCJIUWU6GBOA, seg_pos={0 0}, upload=57772]: Done uploading piece. err: ecclient: upload failed (node:12Bq23DceaRvomaGDHoZr3sYzSHHJEQr2wE4LzSEYk9rUm1DhLa, address:64.49.13.138:4483): protocol: storage node overloaded, request limit: 2000 Jul 26 23:49:29.867801842 (60h13m39.590364882s) [noise=true , address=64.49.13.140:4508 , piece_id=CP4KLEZFK74V5SIYZ72INZUGDEGPPOZ22RQIMBNCFYWGDKRNQY5A, seg_pos={0 0}, upload=57772]: Done uploading piece. err: <nil> Jul 26 23:49:33.705919336 (60h13m43.428482376s) [noise=true , address=45.91.116.10:4093 , piece_id=DVWRCX3PPFPXVUTUHDRJZDWBCXM743WZJET4OG7H34CDCBE47VWA, seg_pos={0 0}, upload=57772]: Done uploading piece. err: <nil> Jul 26 23:50:17.097200779 (60h14m26.819763809s) [noise=true , address=104.36.82.46:4492 , piece_id=BGYEX5N4SLAHZDLYQYIKB22OUUWSO5QYC22KFFWBALQT3C3XYNDA, seg_pos={0 0}, upload=57772]: Done uploading piece. err: ecclient: upload failed (node:1roa5GGYmqCX2AxSrqFgqyV2CtBEYZavuGTseiwDy4pjkT6bip, address:104.36.82.46:4492): protocol: write tcp 188.166.225.170:47964->104.36.82.46:4492: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:50:17.097253470 (60h14m26.81981651s) [noise=true , address=104.36.82.47:4552 , piece_id=7M7D53CDHTJYRNWU62M74IKP3W553O4L2CTIEJNMLM3BAU6UJBYA, seg_pos={0 0}, upload=57772]: Done uploading piece. err: ecclient: upload failed (node:1hoXgBFoGKKknrobbqfArxNhrsiUHKiiVZGWCnvxDSgBcae8gw, address:104.36.82.47:4552): protocol: context deadline exceeded; context deadline exceeded Jul 26 23:50:17.097291522 (60h14m26.819854562s) [noise=true , address=77.90.31.16:4737 , piece_id=UD3QASQ6JJDPLZY3KA5HKHA4TTCFW3R3ZKX6PXBGNDGTGOIYUM2Q, seg_pos={0 0}, upload=57772]: Done uploading piece. err: ecclient: upload failed (node:12pAVncc8nskMA4guej8rBKjXSY6qrCScMbq8edx6dpWk4vf8rM, address:77.90.31.16:4737): protocol: context deadline exceeded; context deadline exceeded Jul 26 23:50:17.097330037 (60h14m26.819893077s) [noise=true , address=104.36.82.53:4131 , piece_id=LZZWE6ZHNV3DIZFUJV5JYKCSM4OAL4MDCXJCEQCPFCLM2MAF552Q, seg_pos={0 0}, upload=57772]: Done uploading piece. err: ecclient: upload failed (node:1uhyc5ozrF9ndeEnQXtjc9o3tsYiAnW1dzeuHV3kv1JS4T4a8C, address:104.36.82.53:4131): protocol: context deadline exceeded; context deadline exceeded Jul 26 23:50:17.097385373 (60h14m26.819948403s) [noise=true , address=104.36.82.38:4715 , piece_id=ITVPZMXMD4OIVMY25YB7TZS7XWIX62GZXVB2YFRAMMF36XIPJKWA, seg_pos={0 0}, upload=57772]: Done uploading piece. err: ecclient: upload failed (node:1dtsPeGJbDSsaR1rPCFE42aCvH8o6AsnRMSK8Dp2RmcpFxo3CR, address:104.36.82.38:4715): protocol: context deadline exceeded; context deadline exceeded Jul 26 23:50:17.097454013 (60h14m26.820017063s) [noise=true , address=104.36.82.54:4164 , piece_id=HDDDDWBJXUOU63VTXOROL6ITYRXMKIYZJSLXHBHGIYKGYURPMNZQ, seg_pos={0 0}, upload=57772]: Done uploading piece. err: ecclient: upload failed (node:1gUXWffNccBoXWftiMWHHSTeEpA5zrd2cHz4uXX5j75GeEK9s2, address:104.36.82.54:4164): protocol: rpc: dial tcp 104.36.82.54:4164: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Jul 26 23:50:17.097485013 (60h14m26.820048063s) [seg_pos={0 0}, upload=57772]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Jul 26 23:50:17.327686067 (60h14m27.050249107s) [upload=57772]: Done uploading segments... Jul 26 23:50:17.327716605 (60h14m27.050279716s) [upload=57772]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 1m0.233118558s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 54 out of 60 is failed (threshold=3) [......XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.236784958s FAIL (2025-07-26 23:50:17 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 11) Duration: 30.236804915s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 1.907244041s FAIL (2025-07-26 23:50:17 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K Duration: 1.907273427s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 56 out of 60 is failed (threshold=3) [...XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXX] duration: 30.237427942s FAIL (2025-07-26 23:50:04 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== 104.36.82.8:4970: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:49:24.387845200 (60h13m34.11040824s) [seg_pos={0 0}, upload=57768]: Done waiting for segment. successful: 5 optimal: 13 errs: <nil> Jul 26 23:49:24.617081988 (60h13m34.339645049s) [upload=57768]: Done uploading segments... Jul 26 23:49:24.617113809 (60h13m34.339676869s) [upload=57768]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 5) Jul 26 23:49:34.540639957 (60h13m44.263202997s) [upload=57776]: Starting upload Jul 26 23:49:34.540698118 (60h13m44.263261158s) [upload=57776]: Done starting upload Jul 26 23:49:34.540744848 (60h13m44.263307888s) [upload=57776]: Uploading segments... Jul 26 23:49:34.540868174 (60h13m44.263431214s) [upload=57776]: Got next segment. Inline: false Jul 26 23:49:34.540892300 (60h13m44.26345534s) [upload=57776]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 23:49:34.540898732 (60h13m44.263461772s) [upload=57776]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:49:34.774445628 (60h13m44.497008668s) [seg_pos={0 0}, upload=57776]: Begin upload segment... Jul 26 23:49:34.774517455 (60h13m44.497080535s) [seg_pos={0 0}, upload=57776]: Done begin upload segment. Jul 26 23:49:34.774553445 (60h13m44.497116485s) [noise=true , address=104.36.82.11:4115 , piece_id=5E22UBHNGS47QCBIFMH26QQJ5LWSJFAZSJPBGLOBNHR7XQDEIEZQ, seg_pos={0 0}, upload=57776]: Uploading piece... Jul 26 23:49:34.774560428 (60h13m44.497123468s) [noise=true , address=45.91.116.9:4262 , piece_id=UJGCDKGJBLBE55TAJA6ZY4K34GGYXAJJVTDGJ7AWCDW3T2QSW2VQ, seg_pos={0 0}, upload=57776]: Uploading piece... Jul 26 23:49:34.775340811 (60h13m44.497903901s) [noise=true , address=77.90.31.12:4592 , piece_id=777DAWJPBLLS3UBH2CJTHV2N6S6OW6Y36F2SDJ6CUAF6SYQIZ75Q, seg_pos={0 0}, upload=57776]: Uploading piece... Jul 26 23:49:34.775896915 (60h13m44.498459955s) [noise=true , address=45.91.116.8:4048 , piece_id=4UPJBI334YEAWOK3AZAOQPQBQ2OFKI7PUU2NI7GD4WYLWWOKBAIA, seg_pos={0 0}, upload=57776]: Uploading piece... Jul 26 23:49:34.776080135 (60h13m44.498643215s) [noise=true , address=45.91.116.7:4010 , piece_id=C5UHQEDB5TPTYYOXPWOA6GVES3FSQ4AFGZZKCN5MRMU6BZ5E52KQ, seg_pos={0 0}, upload=57776]: Uploading piece... Jul 26 23:49:34.776420577 (60h13m44.498983637s) [noise=true , address=45.91.116.10:4081 , piece_id=ZAWKAUVWFC6LC4W34W6HUWYADMO43GQ6BAM4OXM26WAAHK4X3PEQ, seg_pos={0 0}, upload=57776]: Uploading piece... Jul 26 23:49:34.776667099 (60h13m44.499230159s) [noise=true , address=45.91.116.8:4047 , piece_id=KWMB64ORHH3TIW5UGDABJRYEA5M3PDFZ3I74WH2L33G27WEYG2IQ, seg_pos={0 0}, upload=57776]: Uploading piece... Jul 26 23:49:34.777047065 (60h13m44.499610105s) [noise=true , address=104.36.82.5:4837 , piece_id=UTF567EI7QUWFU4YLNHACZ3TSMV4Y3GMNN7EA2FPEG7UDRLGDIGA, seg_pos={0 0}, upload=57776]: Uploading piece... Jul 26 23:49:34.777440118 (60h13m44.500003158s) [noise=true , address=104.36.82.3:4765 , piece_id=BK4RXP2BRWPQVXQLHTKHCIDC7XU6BQYDQLTMVMEXQGCCY5MELHZA, seg_pos={0 0}, upload=57776]: Uploading piece... Jul 26 23:49:34.777523067 (60h13m44.500086107s) [noise=true , address=104.36.82.8:4971 , piece_id=QGQHBOWSKRJ5TZJVF3MAB4ZP3QMXUW3DPCQIS5IY2YVFWWGJVDCQ, seg_pos={0 0}, upload=57776]: Uploading piece... Jul 26 23:49:34.777881392 (60h13m44.500444432s) [noise=true , address=77.90.31.11:4554 , piece_id=7BMFSLU76HD5TZ2OMYAAEEB2ZLLWJB4HZ6BTTG7ZO55FY44ZPVAA, seg_pos={0 0}, upload=57776]: Uploading piece... Jul 26 23:49:34.778181176 (60h13m44.500744216s) [noise=true , address=77.90.31.19:4841 , piece_id=5J6WX73QVE6XOV76UAFDUGUKW7ROKECRLB43DVBIVHHZNPBTOVDQ, seg_pos={0 0}, upload=57776]: Uploading piece... Jul 26 23:49:34.778343346 (60h13m44.500906386s) [noise=true , address=45.91.116.7:4009 , piece_id=EBCHBNFXMWYLONTDE7NQDPKGT3FRXNZNXWPQXYEPZWWBWUC6N66Q, seg_pos={0 0}, upload=57776]: Uploading piece... Jul 26 23:49:34.779227027 (60h13m44.501790067s) [noise=true , address=45.91.116.11:4118 , piece_id=BA6ZXCDHMGFTY3HSZBBASYKYH2WGXBBFURIVU37CXUEWENOGR53Q, seg_pos={0 0}, upload=57776]: Uploading piece... Jul 26 23:49:34.779930212 (60h13m44.502493252s) [noise=true , address=104.36.82.12:4150 , piece_id=EECQR654G6IC54PZJSCANHZM4XZDAS7JMZJY6IPIPGVFGZZOYELQ, seg_pos={0 0}, upload=57776]: Uploading piece... Jul 26 23:49:34.780619520 (60h13m44.50318256s) [noise=true , address=77.90.31.16:4735 , piece_id=6BKJKSS7ZMS2FG4V33PB2VWY5GIDZA3NEOW5ZH6ZSXURGRFU6D7Q, seg_pos={0 0}, upload=57776]: Uploading piece... Jul 26 23:49:36.509732575 (60h13m46.232295615s) [upload=57776]: Next returned nil segment Jul 26 23:49:36.509752653 (60h13m46.232315683s) [upload=57776]: Waiting for error group managing segments... Jul 26 23:49:36.860750898 (60h13m46.583313938s) [noise=true , address=45.91.116.8:4047 , piece_id=KWMB64ORHH3TIW5UGDABJRYEA5M3PDFZ3I74WH2L33G27WEYG2IQ, seg_pos={0 0}, upload=57776]: Done uploading piece. err: <nil> Jul 26 23:49:36.875721120 (60h13m46.59828416s) [noise=true , address=45.91.116.7:4009 , piece_id=EBCHBNFXMWYLONTDE7NQDPKGT3FRXNZNXWPQXYEPZWWBWUC6N66Q, seg_pos={0 0}, upload=57776]: Done uploading piece. err: <nil> Jul 26 23:49:36.885947617 (60h13m46.608510657s) [noise=true , address=45.91.116.11:4118 , piece_id=BA6ZXCDHMGFTY3HSZBBASYKYH2WGXBBFURIVU37CXUEWENOGR53Q, seg_pos={0 0}, upload=57776]: Done uploading piece. err: <nil> Jul 26 23:49:36.887562096 (60h13m46.610125136s) [noise=true , address=77.90.31.11:4554 , piece_id=7BMFSLU76HD5TZ2OMYAAEEB2ZLLWJB4HZ6BTTG7ZO55FY44ZPVAA, seg_pos={0 0}, upload=57776]: Done uploading piece. err: <nil> Jul 26 23:49:36.893518301 (60h13m46.616081341s) [noise=true , address=45.91.116.7:4010 , piece_id=C5UHQEDB5TPTYYOXPWOA6GVES3FSQ4AFGZZKCN5MRMU6BZ5E52KQ, seg_pos={0 0}, upload=57776]: Done uploading piece. err: <nil> Jul 26 23:49:36.925893916 (60h13m46.648456956s) [noise=true , address=45.91.116.9:4262 , piece_id=UJGCDKGJBLBE55TAJA6ZY4K34GGYXAJJVTDGJ7AWCDW3T2QSW2VQ, seg_pos={0 0}, upload=57776]: Done uploading piece. err: <nil> Jul 26 23:49:36.956433678 (60h13m46.678996718s) [noise=true , address=45.91.116.8:4048 , piece_id=4UPJBI334YEAWOK3AZAOQPQBQ2OFKI7PUU2NI7GD4WYLWWOKBAIA, seg_pos={0 0}, upload=57776]: Done uploading piece. err: <nil> Jul 26 23:49:37.078426567 (60h13m46.800989607s) [noise=true , address=45.91.116.10:4081 , piece_id=ZAWKAUVWFC6LC4W34W6HUWYADMO43GQ6BAM4OXM26WAAHK4X3PEQ, seg_pos={0 0}, upload=57776]: Done uploading piece. err: <nil> Jul 26 23:49:38.190358256 (60h13m47.912921326s) [noise=true , address=77.90.31.19:4841 , piece_id=5J6WX73QVE6XOV76UAFDUGUKW7ROKECRLB43DVBIVHHZNPBTOVDQ, seg_pos={0 0}, upload=57776]: Done uploading piece. err: <nil> Jul 26 23:49:52.979833887 (60h14m2.702396927s) [noise=true , address=77.90.31.12:4592 , piece_id=777DAWJPBLLS3UBH2CJTHV2N6S6OW6Y36F2SDJ6CUAF6SYQIZ75Q, seg_pos={0 0}, upload=57776]: Done uploading piece. err: <nil> Jul 26 23:50:03.893998179 (60h14m13.616561229s) [noise=true , address=104.36.82.8:4971 , piece_id=QGQHBOWSKRJ5TZJVF3MAB4ZP3QMXUW3DPCQIS5IY2YVFWWGJVDCQ, seg_pos={0 0}, upload=57776]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:104.36.82.8:4971): protocol: rpc: dial tcp 104.36.82.8:4971: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:50:03.894048926 (60h14m13.616611966s) [noise=true , address=104.36.82.3:4765 , piece_id=BK4RXP2BRWPQVXQLHTKHCIDC7XU6BQYDQLTMVMEXQGCCY5MELHZA, seg_pos={0 0}, upload=57776]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:104.36.82.3:4765): protocol: rpc: dial tcp 104.36.82.3:4765: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:50:03.894084944 (60h14m13.616647985s) [noise=true , address=104.36.82.12:4150 , piece_id=EECQR654G6IC54PZJSCANHZM4XZDAS7JMZJY6IPIPGVFGZZOYELQ, seg_pos={0 0}, upload=57776]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:104.36.82.12:4150): protocol: rpc: dial tcp 104.36.82.12:4150: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:50:03.894127096 (60h14m13.616690136s) [noise=true , address=104.36.82.5:4837 , piece_id=UTF567EI7QUWFU4YLNHACZ3TSMV4Y3GMNN7EA2FPEG7UDRLGDIGA, seg_pos={0 0}, upload=57776]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:104.36.82.5:4837): protocol: write tcp 188.166.225.170:54202->104.36.82.5:4837: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:50:03.894172382 (60h14m13.616735422s) [noise=true , address=104.36.82.11:4115 , piece_id=5E22UBHNGS47QCBIFMH26QQJ5LWSJFAZSJPBGLOBNHR7XQDEIEZQ, seg_pos={0 0}, upload=57776]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:104.36.82.11:4115): protocol: rpc: dial tcp 104.36.82.11:4115: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:50:03.894188994 (60h14m13.616752034s) [noise=true , address=77.90.31.16:4735 , piece_id=6BKJKSS7ZMS2FG4V33PB2VWY5GIDZA3NEOW5ZH6ZSXURGRFU6D7Q, seg_pos={0 0}, upload=57776]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: rpc: dial tcp 77.90.31.16:4735: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:50:03.894247746 (60h14m13.616810796s) [seg_pos={0 0}, upload=57776]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Jul 26 23:50:04.128677420 (60h14m13.851240491s) [upload=57776]: Done uploading segments... Jul 26 23:50:04.128712968 (60h14m13.851276018s) [upload=57776]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 30.237454774s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 20 out of 60 is failed (threshold=3) [X..XX.XX..XXX............XXXX.....XXX..X.....X......X...XX..] duration: 13.961944454s FAIL (2025-07-26 23:49:50 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 13.961958883s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 53 out of 60 is failed (threshold=3) [.......XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.236315416s FAIL (2025-07-26 23:49:47 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 7) Duration: 30.236336806s core uplink us1 vivint
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [.........XXX.X..X.....XXXXX...XXX.X.....X.X..X..X.X....X.X..] duration: 6.531893573s FAIL (2025-07-26 23:49:42 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.531934332s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 1.808696131s FAIL (2025-07-26 23:49:35 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K Duration: 1.808722069s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 55 out of 60 is failed (threshold=3) [....XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXX] duration: 30.231928652s FAIL (2025-07-26 23:49:24 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 5) ======LOGS====== oad=57768]: Begin upload segment... Jul 26 23:48:55.276746000 (60h13m4.99930904s) [seg_pos={0 0}, upload=57768]: Done begin upload segment. Jul 26 23:48:55.276767812 (60h13m4.999330852s) [noise=true , address=45.91.116.12:4227 , piece_id=ALHHYYQFSGDXK5NG32XEQHS4EICKGOPJCP4PDUDW6MBF7XAIT3PA, seg_pos={0 0}, upload=57768]: Uploading piece... Jul 26 23:48:55.276907319 (60h13m4.999470359s) [noise=true , address=104.36.82.9:4005 , piece_id=2DNW26TZGGTR6AYTMJVWXUBQ7ESOSOVL3S6ZNRI27KVXY5YEXUAQ, seg_pos={0 0}, upload=57768]: Uploading piece... Jul 26 23:48:55.277122881 (60h13m4.999685921s) [noise=true , address=104.36.82.12:4151 , piece_id=RLDGPU2SGVOZBY2NIR3FXUVDCRVQACWVDKNTECNWX2NNDUO6KQ2Q, seg_pos={0 0}, upload=57768]: Uploading piece... Jul 26 23:48:55.277309118 (60h13m4.999872158s) [noise=true , address=45.91.116.4:4938 , piece_id=HGWHVUBWTHSCIRHSD5XYJAPWOFHH65RGX3HTMTYLKIUBIIZORFPQ, seg_pos={0 0}, upload=57768]: Uploading piece... Jul 26 23:48:55.277356729 (60h13m4.999919769s) [noise=true , address=77.90.31.14:4662 , piece_id=FIPA3VMRHJ7IELG57SGK2BZJHI4I7JTCZNVRMYHM74YSEG6UCDEQ, seg_pos={0 0}, upload=57768]: Uploading piece... Jul 26 23:48:55.277612809 (60h13m5.000175849s) [noise=true , address=104.36.82.4:4802 , piece_id=AOJV245MRRNT7FFZ4DEI4M7YY7E3PUWBU4BA43DA7YKZRKZW5FVQ, seg_pos={0 0}, upload=57768]: Uploading piece... Jul 26 23:48:55.277738269 (60h13m5.000301299s) [noise=true , address=104.36.82.11:4114 , piece_id=IWZZ5GDJS727IBFXNGQF7EZCZGTLWRF2ZLTANCC27DHW3EJ45DSQ, seg_pos={0 0}, upload=57768]: Uploading piece... Jul 26 23:48:55.278044465 (60h13m5.000607525s) [noise=true , address=45.91.116.7:4009 , piece_id=2NFSP35KL5P34ZKJHCGRWABFQFHNBO6TFZUFPITOQ2PXZTHGQL4Q, seg_pos={0 0}, upload=57768]: Uploading piece... Jul 26 23:48:55.278062218 (60h13m5.000625258s) [noise=true , address=104.36.82.16:4295 , piece_id=GDX6SYSZZJCETOXBRAIKABFR5OJNLVAJLQCHRVPQJSNYCARQUYJQ, seg_pos={0 0}, upload=57768]: Uploading piece... Jul 26 23:48:55.278099480 (60h13m5.00066252s) [noise=true , address=104.36.82.5:4840 , piece_id=SIECCLAJ2KQQYCTTZI2LEDMFNV63CSJVIXS2NT2Q7R2K4THIGOWA, seg_pos={0 0}, upload=57768]: Uploading piece... Jul 26 23:48:55.278436305 (60h13m5.000999335s) [noise=true , address=104.36.82.8:4970 , piece_id=3QZZY7YQEKDBOBDLYSNZVD3WQY3OK4BRLOPFTPZD3STFHLTE5WWQ, seg_pos={0 0}, upload=57768]: Uploading piece... Jul 26 23:48:55.278592564 (60h13m5.001155604s) [noise=true , address=77.90.31.16:4734 , piece_id=N4UNOCVG6OK4DFZFU365HKCKYTW7YVKULF7WLDIZT4SZQYVLJE3Q, seg_pos={0 0}, upload=57768]: Uploading piece... Jul 26 23:48:55.278765404 (60h13m5.001328444s) [noise=true , address=45.91.116.5:4298 , piece_id=Q5NVO5IWMHI3LNPOVMN4AQFCCRERRAZDVEGUAC52A3BY7YTLV76A, seg_pos={0 0}, upload=57768]: Uploading piece... Jul 26 23:48:55.278975186 (60h13m5.001538236s) [noise=true , address=104.36.82.6:4874 , piece_id=CWA5IBSFJ6VMXHJWMH6DQQV4SIGYTVEOVNV3CXAVZDLNNH3ZEUDA, seg_pos={0 0}, upload=57768]: Uploading piece... Jul 26 23:48:55.279045941 (60h13m5.001608981s) [noise=true , address=104.36.82.2:4731 , piece_id=2HW3N5MVMQEE3LRJ6U7PFH7CQHIH2DJ6QST3S34YSEGWI5XVZRBA, seg_pos={0 0}, upload=57768]: Uploading piece... Jul 26 23:48:55.279430917 (60h13m5.001993957s) [noise=true , address=104.36.82.12:4149 , piece_id=L2JZOCSXEP46RWCO2QLTASX7UBWD36CLA6EO4XH4ENXVYVOTE5MA, seg_pos={0 0}, upload=57768]: Uploading piece... Jul 26 23:48:57.002424170 (60h13m6.72498721s) [upload=57768]: Next returned nil segment Jul 26 23:48:57.002440161 (60h13m6.725003191s) [upload=57768]: Waiting for error group managing segments... Jul 26 23:48:57.293344809 (60h13m7.015907859s) [noise=true , address=45.91.116.4:4938 , piece_id=HGWHVUBWTHSCIRHSD5XYJAPWOFHH65RGX3HTMTYLKIUBIIZORFPQ, seg_pos={0 0}, upload=57768]: Done uploading piece. err: <nil> Jul 26 23:48:57.376406051 (60h13m7.098969091s) [noise=true , address=77.90.31.14:4662 , piece_id=FIPA3VMRHJ7IELG57SGK2BZJHI4I7JTCZNVRMYHM74YSEG6UCDEQ, seg_pos={0 0}, upload=57768]: Done uploading piece. err: <nil> Jul 26 23:48:57.393367152 (60h13m7.115930183s) [noise=true , address=45.91.116.12:4227 , piece_id=ALHHYYQFSGDXK5NG32XEQHS4EICKGOPJCP4PDUDW6MBF7XAIT3PA, seg_pos={0 0}, upload=57768]: Done uploading piece. err: <nil> Jul 26 23:48:57.393508012 (60h13m7.116071042s) [noise=true , address=45.91.116.7:4009 , piece_id=2NFSP35KL5P34ZKJHCGRWABFQFHNBO6TFZUFPITOQ2PXZTHGQL4Q, seg_pos={0 0}, upload=57768]: Done uploading piece. err: <nil> Jul 26 23:48:57.525399901 (60h13m7.247962941s) [noise=true , address=45.91.116.5:4298 , piece_id=Q5NVO5IWMHI3LNPOVMN4AQFCCRERRAZDVEGUAC52A3BY7YTLV76A, seg_pos={0 0}, upload=57768]: Done uploading piece. err: <nil> Jul 26 23:49:24.387601474 (60h13m34.110164514s) [noise=true , address=104.36.82.11:4114 , piece_id=IWZZ5GDJS727IBFXNGQF7EZCZGTLWRF2ZLTANCC27DHW3EJ45DSQ, seg_pos={0 0}, upload=57768]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: rpc: dial tcp 104.36.82.11:4114: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:49:24.387642552 (60h13m34.110205592s) [noise=true , address=104.36.82.6:4874 , piece_id=CWA5IBSFJ6VMXHJWMH6DQQV4SIGYTVEOVNV3CXAVZDLNNH3ZEUDA, seg_pos={0 0}, upload=57768]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:104.36.82.6:4874): protocol: rpc: dial tcp 104.36.82.6:4874: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:49:24.387675796 (60h13m34.110238826s) [noise=true , address=104.36.82.12:4149 , piece_id=L2JZOCSXEP46RWCO2QLTASX7UBWD36CLA6EO4XH4ENXVYVOTE5MA, seg_pos={0 0}, upload=57768]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:104.36.82.12:4149): protocol: rpc: dial tcp 104.36.82.12:4149: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:49:24.387711575 (60h13m34.110274615s) [noise=true , address=77.90.31.16:4734 , piece_id=N4UNOCVG6OK4DFZFU365HKCKYTW7YVKULF7WLDIZT4SZQYVLJE3Q, seg_pos={0 0}, upload=57768]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: rpc: dial tcp 77.90.31.16:4734: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:49:24.387720051 (60h13m34.110283091s) [noise=true , address=104.36.82.4:4802 , piece_id=AOJV245MRRNT7FFZ4DEI4M7YY7E3PUWBU4BA43DA7YKZRKZW5FVQ, seg_pos={0 0}, upload=57768]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: rpc: dial tcp 104.36.82.4:4802: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:49:24.387741833 (60h13m34.110304873s) [noise=true , address=104.36.82.9:4005 , piece_id=2DNW26TZGGTR6AYTMJVWXUBQ7ESOSOVL3S6ZNRI27KVXY5YEXUAQ, seg_pos={0 0}, upload=57768]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:104.36.82.9:4005): protocol: rpc: dial tcp 104.36.82.9:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:49:24.387775006 (60h13m34.110338046s) [noise=true , address=104.36.82.12:4151 , piece_id=RLDGPU2SGVOZBY2NIR3FXUVDCRVQACWVDKNTECNWX2NNDUO6KQ2Q, seg_pos={0 0}, upload=57768]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:104.36.82.12:4151): protocol: write tcp 188.166.225.170:34480->104.36.82.12:4151: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:49:24.387780586 (60h13m34.110343627s) [noise=true , address=104.36.82.16:4295 , piece_id=GDX6SYSZZJCETOXBRAIKABFR5OJNLVAJLQCHRVPQJSNYCARQUYJQ, seg_pos={0 0}, upload=57768]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:104.36.82.16:4295): protocol: write tcp 188.166.225.170:41962->104.36.82.16:4295: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:49:24.387803050 (60h13m34.11036609s) [noise=true , address=104.36.82.2:4731 , piece_id=2HW3N5MVMQEE3LRJ6U7PFH7CQHIH2DJ6QST3S34YSEGWI5XVZRBA, seg_pos={0 0}, upload=57768]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:104.36.82.2:4731): protocol: rpc: dial tcp 104.36.82.2:4731: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:49:24.387822798 (60h13m34.110385838s) [noise=true , address=104.36.82.5:4840 , piece_id=SIECCLAJ2KQQYCTTZI2LEDMFNV63CSJVIXS2NT2Q7R2K4THIGOWA, seg_pos={0 0}, upload=57768]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:104.36.82.5:4840): protocol: rpc: dial tcp 104.36.82.5:4840: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:49:24.387826074 (60h13m34.110389114s) [noise=true , address=104.36.82.8:4970 , piece_id=3QZZY7YQEKDBOBDLYSNZVD3WQY3OK4BRLOPFTPZD3STFHLTE5WWQ, seg_pos={0 0}, upload=57768]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:104.36.82.8:4970): protocol: rpc: dial tcp 104.36.82.8:4970: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:49:24.387845200 (60h13m34.11040824s) [seg_pos={0 0}, upload=57768]: Done waiting for segment. successful: 5 optimal: 13 errs: <nil> Jul 26 23:49:24.617081988 (60h13m34.339645049s) [upload=57768]: Done uploading segments... Jul 26 23:49:24.617113809 (60h13m34.339676869s) [upload=57768]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 5) Duration: 30.23194821s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [..........XXX.X..X.....XXXXX...XXX.X.....X.X..X..X.X....X.X.] duration: 6.335970041s FAIL (2025-07-26 23:49:22 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.335988156s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 21 out of 60 is failed (threshold=3) [XX..XX.XX..XXX............XXXX.....XXX..X.....X......X...XX.] duration: 51.763027816s FAIL (2025-07-26 23:49:21 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 51.763060929s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 57 out of 60 is failed (threshold=3) [XXX...XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.230956526s FAIL (2025-07-26 23:49:16 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== 6 23:48:17.719543681 (60h12m27.442106721s) [noise=true , address=104.36.82.47:4520 , piece_id=U3WRHO47O4QVJWPCNFX7KBQNQF65P4LU4FSZEEH7WWS6PCQWZRJA, seg_pos={0 0}, upload=57757]: Uploading piece... Jul 26 23:48:17.719551756 (60h12m27.442114796s) [noise=true , address=77.90.31.18:4828 , piece_id=KTPZUYXC6KNH3AU5Y6RKLB4H63VFGSOPJEV6RB2NEVHS46MHXURQ, seg_pos={0 0}, upload=57757]: Uploading piece... Jul 26 23:48:17.719586562 (60h12m27.442149602s) [noise=true , address=79.127.205.246:4258 , piece_id=IZIZIGZ7OTIYBCI3ZIMQC6QRJ5UA2SDLJF4HTIPVUDUVZVT5TK7Q, seg_pos={0 0}, upload=57757]: Uploading piece... Jul 26 23:48:17.719809459 (60h12m27.4423725s) [noise=true , address=64.49.13.140:4508 , piece_id=P4RG4KTZRXYWC45ECHO4VOQB6LVVWV7BVK2SK6WJQ2LNTQ5TXVUA, seg_pos={0 0}, upload=57757]: Uploading piece... Jul 26 23:48:17.720013159 (60h12m27.442576199s) [noise=true , address=77.90.31.21:4891 , piece_id=S6L7BE3ATPBW7H3WFJ7CU7VRKTENAVAV5HXGMJCNN4AEWX3UJ7EQ, seg_pos={0 0}, upload=57757]: Uploading piece... Jul 26 23:48:17.720219523 (60h12m27.442782563s) [noise=true , address=104.36.82.53:4122 , piece_id=6AVULRGQPFICYBPDN6MKB6YXMIGK6P4FFQGFSJTUSDRRXYJQHXSQ, seg_pos={0 0}, upload=57757]: Uploading piece... Jul 26 23:48:17.721022430 (60h12m27.44358547s) [noise=true , address=77.90.31.19:4856 , piece_id=OQTHFYQT7S4MI5GVSGXO5LYSVKEFJO5DQM74RO3BN2CPBWFOZYSQ, seg_pos={0 0}, upload=57757]: Uploading piece... Jul 26 23:48:17.721247601 (60h12m27.443810641s) [noise=true , address=79.127.205.234:4614 , piece_id=IQQLENC7UU6W3YJADFER5Y27BFOYNNP3IGFK4VUQVMNJQWAKQQYA, seg_pos={0 0}, upload=57757]: Uploading piece... Jul 26 23:48:17.721409731 (60h12m27.443972771s) [noise=true , address=45.91.116.5:4306 , piece_id=FZ6JOZZZCMSXZZYCCZHEQOYUUJW24M44M6TS5TZ7HPNKTUL672JA, seg_pos={0 0}, upload=57757]: Uploading piece... Jul 26 23:48:17.721690598 (60h12m27.444253638s) [noise=true , address=77.90.31.16:4760 , piece_id=Y4J2JQVDA334KLNQ6WU5H5NNVRV5OS2277PFU4RYXM45BGK5X3GA, seg_pos={0 0}, upload=57757]: Uploading piece... Jul 26 23:48:17.722055125 (60h12m27.444618155s) [noise=true , address=79.127.205.235:4682 , piece_id=ZSF7E4G2DXIEQL35PWWE3FIX7JDXD663MJBGYTPCZARM3A3AZKUA, seg_pos={0 0}, upload=57757]: Uploading piece... Jul 26 23:48:17.722374657 (60h12m27.444937687s) [noise=true , address=79.127.201.215:4344 , piece_id=MR6JVTPKU5KZQ2GG44224W5SXQ63KIFJDIHW65APHQHPBEMQVSWQ, seg_pos={0 0}, upload=57757]: Uploading piece... Jul 26 23:48:17.722570151 (60h12m27.445133191s) [noise=true , address=104.36.82.54:4165 , piece_id=6JS2EUFEPM3I3FARMYYNLZXT3JTV4FLWS4ATR3FVHCE4LCAOAO6A, seg_pos={0 0}, upload=57757]: Uploading piece... Jul 26 23:48:17.725004769 (60h12m27.447567809s) [noise=true , address=104.36.82.55:4193 , piece_id=WBNIMAVKR4NVE3L6QCSDRADXL5ZSOKWV664IVLV3K7TCPLB63VQA, seg_pos={0 0}, upload=57757]: Uploading piece... Jul 26 23:48:17.725329230 (60h12m27.44789227s) [noise=true , address=79.127.219.51:4210 , piece_id=CVKOHW7TJYS5NH67IAM6HFHG2JA34UFO2DN6EKBMMKVO6MUV7ZVA, seg_pos={0 0}, upload=57757]: Uploading piece... Jul 26 23:48:17.725522079 (60h12m27.448085119s) [noise=true , address=104.36.82.56:4244 , piece_id=ZJAJKUDT45RTWD4IBGCYJ4XN7A573OP6OGAMJ56NNXEBV2KVQH4A, seg_pos={0 0}, upload=57757]: Uploading piece... Jul 26 23:48:17.725716440 (60h12m27.448279481s) [noise=true , address=45.91.116.9:4273 , piece_id=ST3W2TID2Y4EYWIJIFC6I272PBMEWBJ2LHDW2MQ3AEO3Z5WL5C5A, seg_pos={0 0}, upload=57757]: Uploading piece... Jul 26 23:48:17.725910041 (60h12m27.448473071s) [noise=true , address=79.127.220.97:4361 , piece_id=U5NRYFLUIEMGEG2TM4RIOA2RSLRECY32VV4T46BBN7LT267ADSVA, seg_pos={0 0}, upload=57757]: Uploading piece... Jul 26 23:48:18.807813297 (60h12m28.530376337s) [upload=57757]: Next returned nil segment Jul 26 23:48:18.807822905 (60h12m28.530385945s) [upload=57757]: Waiting for error group managing segments... Jul 26 23:48:19.084250384 (60h12m28.806813414s) [noise=true , address=79.127.205.246:4258 , piece_id=IZIZIGZ7OTIYBCI3ZIMQC6QRJ5UA2SDLJF4HTIPVUDUVZVT5TK7Q, seg_pos={0 0}, upload=57757]: Done uploading piece. err: <nil> Jul 26 23:48:19.106201653 (60h12m28.828764693s) [noise=true , address=79.127.205.234:4614 , piece_id=IQQLENC7UU6W3YJADFER5Y27BFOYNNP3IGFK4VUQVMNJQWAKQQYA, seg_pos={0 0}, upload=57757]: Done uploading piece. err: <nil> Jul 26 23:48:19.255303440 (60h12m28.9778665s) [noise=true , address=45.91.116.5:4306 , piece_id=FZ6JOZZZCMSXZZYCCZHEQOYUUJW24M44M6TS5TZ7HPNKTUL672JA, seg_pos={0 0}, upload=57757]: Done uploading piece. err: ecclient: upload failed (node:1QuFvRW9wGKdKRXPK8UdR8uqTm53CnHaF9CsUz2bUh2Ptw9r1F, address:45.91.116.5:4306): protocol: storage node overloaded, request limit: 2000 Jul 26 23:48:19.269898746 (60h12m28.992461786s) [noise=true , address=79.127.205.235:4682 , piece_id=ZSF7E4G2DXIEQL35PWWE3FIX7JDXD663MJBGYTPCZARM3A3AZKUA, seg_pos={0 0}, upload=57757]: Done uploading piece. err: <nil> Jul 26 23:48:19.825101194 (60h12m29.547664235s) [noise=true , address=77.90.31.21:4891 , piece_id=S6L7BE3ATPBW7H3WFJ7CU7VRKTENAVAV5HXGMJCNN4AEWX3UJ7EQ, seg_pos={0 0}, upload=57757]: Done uploading piece. err: <nil> Jul 26 23:48:20.346978643 (60h12m30.069541683s) [noise=true , address=45.91.116.9:4273 , piece_id=ST3W2TID2Y4EYWIJIFC6I272PBMEWBJ2LHDW2MQ3AEO3Z5WL5C5A, seg_pos={0 0}, upload=57757]: Done uploading piece. err: <nil> Jul 26 23:48:21.382564750 (60h12m31.10512779s) [noise=true , address=79.127.219.51:4210 , piece_id=CVKOHW7TJYS5NH67IAM6HFHG2JA34UFO2DN6EKBMMKVO6MUV7ZVA, seg_pos={0 0}, upload=57757]: Done uploading piece. err: <nil> Jul 26 23:48:23.967400500 (60h12m33.68996354s) [noise=true , address=79.127.220.97:4361 , piece_id=U5NRYFLUIEMGEG2TM4RIOA2RSLRECY32VV4T46BBN7LT267ADSVA, seg_pos={0 0}, upload=57757]: Done uploading piece. err: <nil> Jul 26 23:48:28.951039704 (60h12m38.673602744s) [noise=true , address=64.49.13.140:4508 , piece_id=P4RG4KTZRXYWC45ECHO4VOQB6LVVWV7BVK2SK6WJQ2LNTQ5TXVUA, seg_pos={0 0}, upload=57757]: Done uploading piece. err: <nil> Jul 26 23:48:31.858402303 (60h12m41.580965333s) [noise=true , address=77.90.31.19:4856 , piece_id=OQTHFYQT7S4MI5GVSGXO5LYSVKEFJO5DQM74RO3BN2CPBWFOZYSQ, seg_pos={0 0}, upload=57757]: Done uploading piece. err: <nil> Jul 26 23:48:41.495001929 (60h12m51.217564969s) [noise=true , address=79.127.201.215:4344 , piece_id=MR6JVTPKU5KZQ2GG44224W5SXQ63KIFJDIHW65APHQHPBEMQVSWQ, seg_pos={0 0}, upload=57757]: Done uploading piece. err: <nil> Jul 26 23:49:16.633860439 (60h13m26.356423479s) [noise=true , address=77.90.31.18:4828 , piece_id=KTPZUYXC6KNH3AU5Y6RKLB4H63VFGSOPJEV6RB2NEVHS46MHXURQ, seg_pos={0 0}, upload=57757]: Done uploading piece. err: ecclient: upload failed (node:1jwTDXPShXfNdfMx3UXi22gQphBpFre1MhRsHyjq6xRz4wwTr9, address:77.90.31.18:4828): protocol: rpc: dial tcp 77.90.31.18:4828: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:49:16.633926105 (60h13m26.356489135s) [noise=true , address=104.36.82.56:4244 , piece_id=ZJAJKUDT45RTWD4IBGCYJ4XN7A573OP6OGAMJ56NNXEBV2KVQH4A, seg_pos={0 0}, upload=57757]: Done uploading piece. err: ecclient: upload failed (node:12S1wh3PFQLPosjWmPs5iLfm4S5J6abXvoCfkhYz3KdvXCTaPRu, address:104.36.82.56:4244): protocol: rpc: dial tcp 104.36.82.56:4244: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:49:16.633933679 (60h13m26.356496719s) [noise=true , address=104.36.82.47:4520 , piece_id=U3WRHO47O4QVJWPCNFX7KBQNQF65P4LU4FSZEEH7WWS6PCQWZRJA, seg_pos={0 0}, upload=57757]: Done uploading piece. err: ecclient: upload failed (node:12eNRsboM5zFiyBVPLuQfgb5QSAc5bZFizK9KJHudNd9nstVGyy, address:104.36.82.47:4520): protocol: rpc: dial tcp 104.36.82.47:4520: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:49:16.633961332 (60h13m26.356524362s) [noise=true , address=104.36.82.53:4122 , piece_id=6AVULRGQPFICYBPDN6MKB6YXMIGK6P4FFQGFSJTUSDRRXYJQHXSQ, seg_pos={0 0}, upload=57757]: Done uploading piece. err: ecclient: upload failed (node:124d52DhtbtwipVpZuWdpFTQAttR18hy1K1BbokNRX4R1bLtzTi, address:104.36.82.53:4122): protocol: rpc: dial tcp 104.36.82.53:4122: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:49:16.633987553 (60h13m26.356550593s) [noise=true , address=104.36.82.54:4165 , piece_id=6JS2EUFEPM3I3FARMYYNLZXT3JTV4FLWS4ATR3FVHCE4LCAOAO6A, seg_pos={0 0}, upload=57757]: Done uploading piece. err: ecclient: upload failed (node:12wGUMXVEubEtki4qGnjGEsbYFB1iUauZcu3oDRKrCQsJMV3bV2, address:104.36.82.54:4165): protocol: write tcp 188.166.225.170:50062->104.36.82.54:4165: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:49:16.633997401 (60h13m26.356560441s) [noise=true , address=104.36.82.55:4193 , piece_id=WBNIMAVKR4NVE3L6QCSDRADXL5ZSOKWV664IVLV3K7TCPLB63VQA, seg_pos={0 0}, upload=57757]: Done uploading piece. err: ecclient: upload failed (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8, address:104.36.82.55:4193): protocol: rpc: dial tcp 104.36.82.55:4193: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:49:16.634027118 (60h13m26.356590158s) [noise=true , address=77.90.31.16:4760 , piece_id=Y4J2JQVDA334KLNQ6WU5H5NNVRV5OS2277PFU4RYXM45BGK5X3GA, seg_pos={0 0}, upload=57757]: Done uploading piece. err: ecclient: upload failed (node:1e6p1TD5CqUyWdZjkrxh2kRqMuyDiMUr9eimfeayDAKwhtBrZ8, address:77.90.31.16:4760): protocol: write tcp 188.166.225.170:40810->77.90.31.16:4760: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:49:16.634058267 (60h13m26.356621307s) [seg_pos={0 0}, upload=57757]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Jul 26 23:49:16.863127736 (60h13m26.585690776s) [upload=57757]: Done uploading segments... Jul 26 23:49:16.863150580 (60h13m26.58571362s) [upload=57757]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 1m0.230967336s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 52 out of 60 is failed (threshold=3) [........XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.240002235s FAIL (2025-07-26 23:49:09 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 6) Duration: 30.240024037s core uplink us1 vivint
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [...........XXX.X..X.....XXXXX...XXX.X.....X.X..X..X.X....X.X] duration: 30.000426135s FAIL (2025-07-26 23:49:07 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.000452525s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 1.838328124s FAIL (2025-07-26 23:48:33 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K Duration: 1.838351998s core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 21 out of 60 is failed (threshold=3) [.XX..XX.XX..XXX............XXXX.....XXX..X.....X......X...XX] duration: 1m0.000470584s FAIL (2025-07-26 23:48:28 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.000484339s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 57 out of 60 is failed (threshold=3) [XXXX...XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.233679297s FAIL (2025-07-26 23:48:16 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 9) ======LOGS====== , piece_id=YD23ZM4YGGZRPTB4A4T3AR3BGIXIW4DP74PWV7TPXSYIQ5JEMAGQ, seg_pos={0 0}, upload=57741]: Uploading piece... Jul 26 23:47:17.259925565 (60h11m26.982488605s) [noise=true , address=45.91.116.14:4206 , piece_id=3ILJBGGGZ5NKBU6XOKRDFCWRJI63R2FXDTQC6APD66BVSKFQLVVQ, seg_pos={0 0}, upload=57741]: Uploading piece... Jul 26 23:47:17.260273300 (60h11m26.98283634s) [noise=true , address=104.36.82.52:4111 , piece_id=XTPMI3VV24DSHDKPE72MCQQOMS5DWHZTTJHXOV74WH4VL62VBXVA, seg_pos={0 0}, upload=57741]: Uploading piece... Jul 26 23:47:17.260583654 (60h11m26.983146694s) [noise=true , address=64.49.13.138:4483 , piece_id=V5JDEGVO2QEVGE7J2NKGHU7YHL5F6Q3BA54CJ3KMQKY7ELGSVZ4A, seg_pos={0 0}, upload=57741]: Uploading piece... Jul 26 23:47:17.261046479 (60h11m26.983609519s) [noise=true , address=104.36.82.55:4211 , piece_id=2ZUESMC5WIN547EWANQ6OUSKQPTTUNHCEJ6YCH7ZRMTM4SLGQI3Q, seg_pos={0 0}, upload=57741]: Uploading piece... Jul 26 23:47:17.261292229 (60h11m26.983855269s) [noise=true , address=79.127.201.215:4359 , piece_id=SZ7YD4TEGUVEE4YK5PE6U6AL3V4XCOLDA52B4LRY7TPVR3T6HDLQ, seg_pos={0 0}, upload=57741]: Uploading piece... Jul 26 23:47:17.261536146 (60h11m26.984099186s) [noise=true , address=104.36.82.46:4516 , piece_id=HFRHP7SU5CVDB3MKPYHVUSZDDMC2J4SYL3MTFO5K2YBSMR4VOYAQ, seg_pos={0 0}, upload=57741]: Uploading piece... Jul 26 23:47:17.261758351 (60h11m26.984321391s) [noise=true , address=156.146.43.225:4448 , piece_id=ITMJZEK7WVGDOS65BNUBWT5NOTXN44R5NV2TXNOOGHHO7E4RIRQQ, seg_pos={0 0}, upload=57741]: Uploading piece... Jul 26 23:47:17.262002678 (60h11m26.984565719s) [noise=true , address=104.36.82.47:4543 , piece_id=32E26YLYHTG3PGCUBLLJEAM2OW4IOLJ7M3TXFNBSCB6IG55HQOKA, seg_pos={0 0}, upload=57741]: Uploading piece... Jul 26 23:47:17.262220485 (60h11m26.984783525s) [noise=true , address=45.91.116.9:4273 , piece_id=LIMUVVGWBQSJXKQIM6HXUX7UVCC467OBRMWSVMF7XJN7AFVXBH6A, seg_pos={0 0}, upload=57741]: Uploading piece... Jul 26 23:47:17.262433563 (60h11m26.984996603s) [noise=true , address=77.90.31.21:4886 , piece_id=Q7POHTUWMZ6B23PJCWKOPIE5XKGU7JMYE53A5BHKFN6JHPX4I64Q, seg_pos={0 0}, upload=57741]: Uploading piece... Jul 26 23:47:17.262647563 (60h11m26.985210603s) [noise=true , address=79.127.205.234:4610 , piece_id=SV7BVNOD7MF6JWICIXHM2KVVMT5ZIDYVPPF6AAUV4QXV3LPOYOJA, seg_pos={0 0}, upload=57741]: Uploading piece... Jul 26 23:47:17.262866412 (60h11m26.985429452s) [noise=true , address=77.90.31.15:4703 , piece_id=PVGX4JMLVDX6WIVFX7E3Y5XDFAU3AXCM7WU4MT7HH22J245OXWDA, seg_pos={0 0}, upload=57741]: Uploading piece... Jul 26 23:47:17.263095239 (60h11m26.985658279s) [noise=true , address=104.36.82.53:4150 , piece_id=3VPLKKB2WOK5PXMLXWUPG7BDGAOZOHJZR53MPRLG7XI6T5UYBZDA, seg_pos={0 0}, upload=57741]: Uploading piece... Jul 26 23:47:17.263319127 (60h11m26.985882167s) [noise=true , address=45.91.116.14:4194 , piece_id=IMPZIHLMCQMGYVDYZGFVE7Z44KFRCKJYYD4TCGO2MHKRNNDZD6SA, seg_pos={0 0}, upload=57741]: Uploading piece... Jul 26 23:47:17.263535471 (60h11m26.986098511s) [noise=true , address=79.127.219.38:4018 , piece_id=LAVSPQRBBUA4ZYAHTPS4M3EQZRSPWBNK26SFVITCB2S5ANVFTV3Q, seg_pos={0 0}, upload=57741]: Uploading piece... Jul 26 23:47:17.263755052 (60h11m26.986318092s) [noise=true , address=104.36.82.54:4186 , piece_id=2CHSCHUTFW7OOXMATBXYUL4L7BBDKXKSZW33MYJI7EJJCSPSTMZQ, seg_pos={0 0}, upload=57741]: Uploading piece... Jul 26 23:47:18.400670162 (60h11m28.123233192s) [upload=57741]: Next returned nil segment Jul 26 23:47:18.400687235 (60h11m28.123250275s) [upload=57741]: Waiting for error group managing segments... Jul 26 23:47:18.464488978 (60h11m28.187052018s) [noise=true , address=45.91.116.5:4310 , piece_id=CMCCDVGAVJJBBFGHLFBPLIUFREIJ5RTYLR2VAZOV44OEDV4FDQUA, seg_pos={0 0}, upload=57741]: Done uploading piece. err: ecclient: upload failed (node:127RMUc62xRq9RU5z3XfkZsq5sehmv3kP8o53tr9BxVcPE3GQCY, address:45.91.116.5:4310): protocol: storage node overloaded, request limit: 2000 Jul 26 23:47:18.473222066 (60h11m28.195785106s) [noise=true , address=156.146.43.225:4448 , piece_id=ITMJZEK7WVGDOS65BNUBWT5NOTXN44R5NV2TXNOOGHHO7E4RIRQQ, seg_pos={0 0}, upload=57741]: Done uploading piece. err: ecclient: upload failed (node:1irai8UTYknGTUkKHtKbBNTJTYMQsNTFnzrTEaWCMRDq1NNpGh, address:156.146.43.225:4448): protocol: storage node overloaded, request limit: 2000 Jul 26 23:47:18.691660173 (60h11m28.414223254s) [noise=true , address=79.127.205.234:4610 , piece_id=SV7BVNOD7MF6JWICIXHM2KVVMT5ZIDYVPPF6AAUV4QXV3LPOYOJA, seg_pos={0 0}, upload=57741]: Done uploading piece. err: <nil> Jul 26 23:47:19.251632850 (60h11m28.97419588s) [noise=true , address=77.90.31.15:4703 , piece_id=PVGX4JMLVDX6WIVFX7E3Y5XDFAU3AXCM7WU4MT7HH22J245OXWDA, seg_pos={0 0}, upload=57741]: Done uploading piece. err: <nil> Jul 26 23:47:19.296407367 (60h11m29.018970407s) [noise=true , address=77.90.31.21:4886 , piece_id=Q7POHTUWMZ6B23PJCWKOPIE5XKGU7JMYE53A5BHKFN6JHPX4I64Q, seg_pos={0 0}, upload=57741]: Done uploading piece. err: <nil> Jul 26 23:47:19.439619385 (60h11m29.162182425s) [noise=true , address=79.127.219.38:4018 , piece_id=LAVSPQRBBUA4ZYAHTPS4M3EQZRSPWBNK26SFVITCB2S5ANVFTV3Q, seg_pos={0 0}, upload=57741]: Done uploading piece. err: <nil> Jul 26 23:47:21.413719713 (60h11m31.136282753s) [noise=true , address=45.91.116.9:4273 , piece_id=LIMUVVGWBQSJXKQIM6HXUX7UVCC467OBRMWSVMF7XJN7AFVXBH6A, seg_pos={0 0}, upload=57741]: Done uploading piece. err: <nil> Jul 26 23:47:23.359200953 (60h11m33.081763993s) [noise=true , address=64.49.13.138:4483 , piece_id=V5JDEGVO2QEVGE7J2NKGHU7YHL5F6Q3BA54CJ3KMQKY7ELGSVZ4A, seg_pos={0 0}, upload=57741]: Done uploading piece. err: <nil> Jul 26 23:47:26.609732399 (60h11m36.332295439s) [noise=true , address=45.91.116.14:4194 , piece_id=IMPZIHLMCQMGYVDYZGFVE7Z44KFRCKJYYD4TCGO2MHKRNNDZD6SA, seg_pos={0 0}, upload=57741]: Done uploading piece. err: <nil> Jul 26 23:47:28.991584836 (60h11m38.714147876s) [noise=true , address=45.91.116.14:4206 , piece_id=3ILJBGGGZ5NKBU6XOKRDFCWRJI63R2FXDTQC6APD66BVSKFQLVVQ, seg_pos={0 0}, upload=57741]: Done uploading piece. err: <nil> Jul 26 23:48:01.448362709 (60h12m11.17092575s) [noise=true , address=79.127.201.215:4359 , piece_id=SZ7YD4TEGUVEE4YK5PE6U6AL3V4XCOLDA52B4LRY7TPVR3T6HDLQ, seg_pos={0 0}, upload=57741]: Done uploading piece. err: <nil> Jul 26 23:48:16.173278012 (60h12m25.895841042s) [noise=true , address=104.36.82.47:4543 , piece_id=32E26YLYHTG3PGCUBLLJEAM2OW4IOLJ7M3TXFNBSCB6IG55HQOKA, seg_pos={0 0}, upload=57741]: Done uploading piece. err: ecclient: upload failed (node:12KxnBTVxhbwA79Gd8V7Mh6bWybtqttqgo5agHoHewXHAey5y7D, address:104.36.82.47:4543): protocol: write tcp 188.166.225.170:60224->104.36.82.47:4543: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:48:16.173309342 (60h12m25.895872382s) [noise=true , address=104.36.82.55:4211 , piece_id=2ZUESMC5WIN547EWANQ6OUSKQPTTUNHCEJ6YCH7ZRMTM4SLGQI3Q, seg_pos={0 0}, upload=57741]: Done uploading piece. err: ecclient: upload failed (node:1269tDiDZTJSwmka4HLKMEaHErdCaUqXEYTFimGWhtjai9o4dAB, address:104.36.82.55:4211): protocol: write tcp 188.166.225.170:37874->104.36.82.55:4211: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:48:16.173316256 (60h12m25.895879296s) [noise=true , address=104.36.82.46:4516 , piece_id=HFRHP7SU5CVDB3MKPYHVUSZDDMC2J4SYL3MTFO5K2YBSMR4VOYAQ, seg_pos={0 0}, upload=57741]: Done uploading piece. err: ecclient: upload failed (node:12FRYABJbf6iHTm3WiYcL192kFth6rHjrcMAS3ahF8ddm4ox6Uj, address:104.36.82.46:4516): protocol: write tcp 188.166.225.170:51730->104.36.82.46:4516: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:48:16.173342857 (60h12m25.895905887s) [noise=true , address=104.36.82.15:4279 , piece_id=YD23ZM4YGGZRPTB4A4T3AR3BGIXIW4DP74PWV7TPXSYIQ5JEMAGQ, seg_pos={0 0}, upload=57741]: Done uploading piece. err: ecclient: upload failed (node:12uiA6tmZtmHDNaNtkhRtH31q4MJxn3C4bfrLAXmYz5mKmofHoG, address:104.36.82.15:4279): protocol: write tcp 188.166.225.170:39096->104.36.82.15:4279: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:48:16.173350431 (60h12m25.895913461s) [noise=true , address=104.36.82.54:4186 , piece_id=2CHSCHUTFW7OOXMATBXYUL4L7BBDKXKSZW33MYJI7EJJCSPSTMZQ, seg_pos={0 0}, upload=57741]: Done uploading piece. err: ecclient: upload failed (node:12Gu3vXy2JUkKDaF4MkMCgVmYspW4Ez8EAfv6e4GeCkrwYTe5T, address:104.36.82.54:4186): protocol: rpc: dial tcp 104.36.82.54:4186: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:48:16.173368446 (60h12m25.895931486s) [noise=true , address=104.36.82.53:4150 , piece_id=3VPLKKB2WOK5PXMLXWUPG7BDGAOZOHJZR53MPRLG7XI6T5UYBZDA, seg_pos={0 0}, upload=57741]: Done uploading piece. err: ecclient: upload failed (node:1jDz53H9KVPn5dBA1gqj66ZQEKbo9dNLucS9c3u8WvY1Xn7jH, address:104.36.82.53:4150): protocol: rpc: dial tcp 104.36.82.53:4150: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:48:16.173377733 (60h12m25.895940763s) [noise=true , address=104.36.82.52:4111 , piece_id=XTPMI3VV24DSHDKPE72MCQQOMS5DWHZTTJHXOV74WH4VL62VBXVA, seg_pos={0 0}, upload=57741]: Done uploading piece. err: ecclient: upload failed (node:12krqZcFoBPfLFbLGQUAjPMzHJ4HKqsKPRg6Xap7nuShaEKBumH, address:104.36.82.52:4111): protocol: write tcp 188.166.225.170:43770->104.36.82.52:4111: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:48:16.173397582 (60h12m25.895960612s) [seg_pos={0 0}, upload=57741]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil> Jul 26 23:48:16.405230535 (60h12m26.127793575s) [upload=57741]: Done uploading segments... Jul 26 23:48:16.405256335 (60h12m26.127819365s) [upload=57741]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9) Duration: 1m0.233692072s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 54 out of 60 is failed (threshold=3) [.....XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXX] duration: 30.232576039s FAIL (2025-07-26 23:48:12 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== e:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: read tcp 188.166.225.170:48530->77.90.31.18:4806: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:47:40.541715359 (60h11m50.264278389s) [seg_pos={0 0}, upload=57738]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Jul 26 23:47:40.904931641 (60h11m50.627494681s) [upload=57738]: Done uploading segments... Jul 26 23:47:40.904955066 (60h11m50.627518106s) [upload=57738]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Jul 26 23:47:43.325198508 (60h11m53.047761578s) [upload=57748]: Starting upload Jul 26 23:47:43.325316624 (60h11m53.047879714s) [upload=57748]: Done starting upload Jul 26 23:47:43.325422516 (60h11m53.047985546s) [upload=57748]: Uploading segments... Jul 26 23:47:43.325948162 (60h11m53.048511212s) [upload=57748]: Got next segment. Inline: false Jul 26 23:47:43.326029227 (60h11m53.048592267s) [upload=57748]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 23:47:43.326035961 (60h11m53.048599011s) [upload=57748]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:47:43.560602296 (60h11m53.283165346s) [seg_pos={0 0}, upload=57748]: Begin upload segment... Jul 26 23:47:43.560670727 (60h11m53.283233777s) [seg_pos={0 0}, upload=57748]: Done begin upload segment. Jul 26 23:47:43.560710022 (60h11m53.283273072s) [noise=true , address=45.91.116.7:4009 , piece_id=5VCWTQLCHO5J5NQRJSW52TDDY5ALZSMO7GEUG7O52KFGQ7MB7JYA, seg_pos={0 0}, upload=57748]: Uploading piece... Jul 26 23:47:43.560739007 (60h11m53.283302047s) [noise=true , address=104.36.82.6:4873 , piece_id=ITIFV5ZVWP6SE4FF7BKFQ2BXAITUSQX3KZZKJAGD4MZMQY5ENK5A, seg_pos={0 0}, upload=57748]: Uploading piece... Jul 26 23:47:43.561078087 (60h11m53.283641127s) [noise=true , address=104.36.82.5:4837 , piece_id=G4EZ2R6L34C3C73PKWQ5E7O3XE7GOAH3MAZGU427ETZ2IBMCBICA, seg_pos={0 0}, upload=57748]: Uploading piece... Jul 26 23:47:43.561336531 (60h11m53.283899561s) [noise=true , address=45.91.116.8:4045 , piece_id=UODTBHHAEN4KSXCEHNGDAYQSTKN23I2JGD3J55KKTYSSZ3P7YAVA, seg_pos={0 0}, upload=57748]: Uploading piece... Jul 26 23:47:43.561342462 (60h11m53.283905502s) [noise=true , address=45.91.116.6:4973 , piece_id=OMSL7CAVK3YNLICMC6UASC2SFGCAAS23EKXKN77723EHWUD23QSQ, seg_pos={0 0}, upload=57748]: Uploading piece... Jul 26 23:47:43.561632858 (60h11m53.284195898s) [noise=true , address=77.90.31.18:4807 , piece_id=QJLUVMAMINJRHL65V5HOVRPFZSTUINKRAAXQTNIJDECO42TWDP3Q, seg_pos={0 0}, upload=57748]: Uploading piece... Jul 26 23:47:43.561657555 (60h11m53.284220595s) [noise=true , address=45.91.116.8:4048 , piece_id=BJ6PGE5BROLF7SIRUPDQKBT5VW5TLDAOOZO7TT2LISUVACXAOUUQ, seg_pos={0 0}, upload=57748]: Uploading piece... Jul 26 23:47:43.561997616 (60h11m53.284560656s) [noise=true , address=104.36.82.6:4876 , piece_id=YRJUVLXAM6ZZEIAKLHIUGHEUTZ76ZQYV4OHC26HM4KVPJ7Y2P62A, seg_pos={0 0}, upload=57748]: Uploading piece... Jul 26 23:47:43.562013587 (60h11m53.284576627s) [noise=true , address=77.90.31.14:4661 , piece_id=MVDJ2OLJZ66D3V27753S5XOCXV74TT4CZHFGX5FLVE73XKPRIZPA, seg_pos={0 0}, upload=57748]: Uploading piece... Jul 26 23:47:43.562275778 (60h11m53.284838818s) [noise=true , address=77.90.31.21:4879 , piece_id=4FQNWCSLDGYU34Y7C2MFQ3QLGG67TRBR5HTZHVNC65C47VIKTBLA, seg_pos={0 0}, upload=57748]: Uploading piece... Jul 26 23:47:43.562321055 (60h11m53.284884125s) [noise=true , address=104.36.82.11:4113 , piece_id=IMGM74ALMTOFTNH5EW756W55CAFIOES3744F7DIHG6VLFU2ID2SQ, seg_pos={0 0}, upload=57748]: Uploading piece... Jul 26 23:47:43.562530204 (60h11m53.285093244s) [noise=true , address=45.91.116.10:4082 , piece_id=Y6VU3OVATPJX3MRMCCAYYRGR3QHYJQSJHSRWC5JKE4JL3WHLUQJA, seg_pos={0 0}, upload=57748]: Uploading piece... Jul 26 23:47:43.562680382 (60h11m53.285243412s) [noise=true , address=77.90.31.19:4843 , piece_id=LX5HNCHV7EIA72SVGZMXCRQIFWKIZER2THA4NNPNFOUDKLZ6C63A, seg_pos={0 0}, upload=57748]: Uploading piece... Jul 26 23:47:43.562741839 (60h11m53.28530488s) [noise=true , address=45.91.116.4:4937 , piece_id=XV3IJJRHCFRPP5RH7PURI7AXZGZ3URZVJVYLUQIMN654PUUTHLEQ, seg_pos={0 0}, upload=57748]: Uploading piece... Jul 26 23:47:43.562949587 (60h11m53.285512627s) [noise=true , address=45.91.116.6:4975 , piece_id=SP3I2CSBXEK3TZMNLNB77G4AM52Z42R67W6LHKS5Q77OZOSXMRPA, seg_pos={0 0}, upload=57748]: Uploading piece... Jul 26 23:47:43.563773434 (60h11m53.286336464s) [noise=true , address=45.91.116.14:4189 , piece_id=QQBX5SFFXSNHMLNGCUWCQNCWP2452RFJDSNGZAIIYIBBMM3VB65A, seg_pos={0 0}, upload=57748]: Uploading piece... Jul 26 23:47:45.253656767 (60h11m54.976219807s) [upload=57748]: Next returned nil segment Jul 26 23:47:45.253669290 (60h11m54.97623233s) [upload=57748]: Waiting for error group managing segments... Jul 26 23:47:45.580548168 (60h11m55.303111208s) [noise=true , address=45.91.116.10:4082 , piece_id=Y6VU3OVATPJX3MRMCCAYYRGR3QHYJQSJHSRWC5JKE4JL3WHLUQJA, seg_pos={0 0}, upload=57748]: Done uploading piece. err: <nil> Jul 26 23:47:45.588792292 (60h11m55.311355332s) [noise=true , address=45.91.116.6:4975 , piece_id=SP3I2CSBXEK3TZMNLNB77G4AM52Z42R67W6LHKS5Q77OZOSXMRPA, seg_pos={0 0}, upload=57748]: Done uploading piece. err: <nil> Jul 26 23:47:45.593099402 (60h11m55.315662442s) [noise=true , address=45.91.116.4:4937 , piece_id=XV3IJJRHCFRPP5RH7PURI7AXZGZ3URZVJVYLUQIMN654PUUTHLEQ, seg_pos={0 0}, upload=57748]: Done uploading piece. err: <nil> Jul 26 23:47:45.593503526 (60h11m55.316066566s) [noise=true , address=45.91.116.8:4045 , piece_id=UODTBHHAEN4KSXCEHNGDAYQSTKN23I2JGD3J55KKTYSSZ3P7YAVA, seg_pos={0 0}, upload=57748]: Done uploading piece. err: <nil> Jul 26 23:47:45.668071687 (60h11m55.390634727s) [noise=true , address=45.91.116.6:4973 , piece_id=OMSL7CAVK3YNLICMC6UASC2SFGCAAS23EKXKN77723EHWUD23QSQ, seg_pos={0 0}, upload=57748]: Done uploading piece. err: <nil> Jul 26 23:47:45.668218978 (60h11m55.390782019s) [noise=true , address=77.90.31.14:4661 , piece_id=MVDJ2OLJZ66D3V27753S5XOCXV74TT4CZHFGX5FLVE73XKPRIZPA, seg_pos={0 0}, upload=57748]: Done uploading piece. err: <nil> Jul 26 23:47:45.680087928 (60h11m55.402650968s) [noise=true , address=45.91.116.7:4009 , piece_id=5VCWTQLCHO5J5NQRJSW52TDDY5ALZSMO7GEUG7O52KFGQ7MB7JYA, seg_pos={0 0}, upload=57748]: Done uploading piece. err: <nil> Jul 26 23:47:45.708269558 (60h11m55.430832598s) [noise=true , address=45.91.116.14:4189 , piece_id=QQBX5SFFXSNHMLNGCUWCQNCWP2452RFJDSNGZAIIYIBBMM3VB65A, seg_pos={0 0}, upload=57748]: Done uploading piece. err: <nil> Jul 26 23:47:45.735786215 (60h11m55.458349255s) [noise=true , address=45.91.116.8:4048 , piece_id=BJ6PGE5BROLF7SIRUPDQKBT5VW5TLDAOOZO7TT2LISUVACXAOUUQ, seg_pos={0 0}, upload=57748]: Done uploading piece. err: <nil> Jul 26 23:47:45.750643862 (60h11m55.473206902s) [noise=true , address=77.90.31.21:4879 , piece_id=4FQNWCSLDGYU34Y7C2MFQ3QLGG67TRBR5HTZHVNC65C47VIKTBLA, seg_pos={0 0}, upload=57748]: Done uploading piece. err: <nil> Jul 26 23:47:47.184027825 (60h11m56.906590865s) [noise=true , address=77.90.31.19:4843 , piece_id=LX5HNCHV7EIA72SVGZMXCRQIFWKIZER2THA4NNPNFOUDKLZ6C63A, seg_pos={0 0}, upload=57748]: Done uploading piece. err: <nil> Jul 26 23:48:12.670290334 (60h12m22.392853364s) [noise=true , address=77.90.31.18:4807 , piece_id=QJLUVMAMINJRHL65V5HOVRPFZSTUINKRAAXQTNIJDECO42TWDP3Q, seg_pos={0 0}, upload=57748]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: write tcp 188.166.225.170:47162->77.90.31.18:4807: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:48:12.670351250 (60h12m22.39291429s) [noise=true , address=104.36.82.5:4837 , piece_id=G4EZ2R6L34C3C73PKWQ5E7O3XE7GOAH3MAZGU427ETZ2IBMCBICA, seg_pos={0 0}, upload=57748]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:104.36.82.5:4837): protocol: write tcp 188.166.225.170:35416->104.36.82.5:4837: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:48:12.670362502 (60h12m22.392925532s) [noise=true , address=104.36.82.6:4876 , piece_id=YRJUVLXAM6ZZEIAKLHIUGHEUTZ76ZQYV4OHC26HM4KVPJ7Y2P62A, seg_pos={0 0}, upload=57748]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: write tcp 188.166.225.170:40070->104.36.82.6:4876: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:48:12.670398701 (60h12m22.392961741s) [noise=true , address=104.36.82.11:4113 , piece_id=IMGM74ALMTOFTNH5EW756W55CAFIOES3744F7DIHG6VLFU2ID2SQ, seg_pos={0 0}, upload=57748]: Done uploading piece. err: ecclient: upload failed (node:1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV, address:104.36.82.11:4113): protocol: rpc: dial tcp 104.36.82.11:4113: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:48:12.670415233 (60h12m22.392978273s) [noise=true , address=104.36.82.6:4873 , piece_id=ITIFV5ZVWP6SE4FF7BKFQ2BXAITUSQX3KZZKJAGD4MZMQY5ENK5A, seg_pos={0 0}, upload=57748]: Done uploading piece. err: ecclient: upload failed (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3, address:104.36.82.6:4873): protocol: rpc: dial tcp 104.36.82.6:4873: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:48:12.670440100 (60h12m22.39300314s) [seg_pos={0 0}, upload=57748]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 26 23:48:12.901580522 (60h12m22.624143562s) [upload=57748]: Done uploading segments... Jul 26 23:48:12.901608245 (60h12m22.624171275s) [upload=57748]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 30.23259184s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 51 out of 60 is failed (threshold=3) [.........XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.235520533s FAIL (2025-07-26 23:48:11 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.235546423s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 2.022256825s FAIL (2025-07-26 23:48:03 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K Duration: 2.022290059s core uplink us1 vivint
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [............XXX.X..X.....XXXXX...XXX.X.....X.X..X..X.X....X.] duration: 8.119464173s FAIL (2025-07-26 23:47:43 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 8.119484722s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 53 out of 60 is failed (threshold=3) [......XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXX] duration: 30.365148478s FAIL (2025-07-26 23:47:40 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== 7029955 (60h10m55.389592995s) [upload=57728]: Done uploading segments... Jul 26 23:46:45.667054953 (60h10m55.389617993s) [upload=57728]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Jul 26 23:47:11.186968024 (60h11m20.909531064s) [upload=57738]: Starting upload Jul 26 23:47:11.187023239 (60h11m20.909586279s) [upload=57738]: Done starting upload Jul 26 23:47:11.187126748 (60h11m20.909689798s) [upload=57738]: Uploading segments... Jul 26 23:47:11.187257157 (60h11m20.909820207s) [upload=57738]: Got next segment. Inline: false Jul 26 23:47:11.187297755 (60h11m20.909860795s) [upload=57738]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 23:47:11.187304317 (60h11m20.909867357s) [upload=57738]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:47:11.484458041 (60h11m21.207021081s) [seg_pos={0 0}, upload=57738]: Begin upload segment... Jul 26 23:47:11.484497867 (60h11m21.207060947s) [seg_pos={0 0}, upload=57738]: Done begin upload segment. Jul 26 23:47:11.484518376 (60h11m21.207081416s) [noise=true , address=45.91.116.13:4154 , piece_id=YWLJD6ZW66H7ZMW5NHVKXXV3OZZOAQOUDJ42YMHQRY7DBDSEK65A, seg_pos={0 0}, upload=57738]: Uploading piece... Jul 26 23:47:11.484950273 (60h11m21.207513323s) [noise=true , address=45.91.116.12:4225 , piece_id=7CG6KLP6BMUMO2CNTGNOOJSZYUWZIGZNBUF6GAR4KUTQDZQ4EMLA, seg_pos={0 0}, upload=57738]: Uploading piece... Jul 26 23:47:11.485217864 (60h11m21.207780904s) [noise=true , address=77.90.31.18:4806 , piece_id=XCBDPEMBZGTDSH5S5IAXXTHSNMRS4CIG56UY7ASFJNDUGEW6XXUQ, seg_pos={0 0}, upload=57738]: Uploading piece... Jul 26 23:47:11.485479645 (60h11m21.208042686s) [noise=true , address=45.91.116.7:4009 , piece_id=BYZ6H5YHNHB2JVADGMN2KTC4BSCA4Y6GKMJZZLRF3TG7JX74FL6Q, seg_pos={0 0}, upload=57738]: Uploading piece... Jul 26 23:47:11.485741055 (60h11m21.208304095s) [noise=true , address=77.90.31.14:4661 , piece_id=ZMAQBP4VPBAHALI5ULY2BLHDW7B7SKHGCOIPHCKYL4LCDF5I6KZQ, seg_pos={0 0}, upload=57738]: Uploading piece... Jul 26 23:47:11.485926570 (60h11m21.20848967s) [noise=true , address=45.91.116.8:4047 , piece_id=U45PMTJ3VMEHYU7WCHBGVBOOFYFR2PRPN5L3MK3R3GKDHE64X3LA, seg_pos={0 0}, upload=57738]: Uploading piece... Jul 26 23:47:11.486537810 (60h11m21.20910085s) [noise=true , address=104.36.82.5:4839 , piece_id=QUCHCVMSO3JZHPOQEBDW3JCZCLMFWTALYCJELFFCZBIOETVHKGTA, seg_pos={0 0}, upload=57738]: Uploading piece... Jul 26 23:47:11.487016205 (60h11m21.209579246s) [noise=true , address=77.90.31.10:4517 , piece_id=37FYOW57Q5JSF7CVIY5LOMOB6LU4SDII67GYJUG65K7WD7KYTKRA, seg_pos={0 0}, upload=57738]: Uploading piece... Jul 26 23:47:11.487523496 (60h11m21.210086536s) [noise=true , address=45.91.116.13:4155 , piece_id=7HOYKE4U2QREIMDW25QLGTDKPZTJ7BASOH6SCM5JI7Q7MQ6KWBBA, seg_pos={0 0}, upload=57738]: Uploading piece... Jul 26 23:47:11.487876181 (60h11m21.210439221s) [noise=true , address=104.36.82.3:4765 , piece_id=SZGZ47TSCSAU6Z3UI4LB2AJIXEYRK2SK67H4M65JGLTJQLNO2YJQ, seg_pos={0 0}, upload=57738]: Uploading piece... Jul 26 23:47:11.488279052 (60h11m21.210842092s) [noise=true , address=104.36.82.2:4731 , piece_id=N4MNBERLWY5EXS7WQ5FC6S2PRMCSNIYLXYUHWAYY4H7JIE2HUR7Q, seg_pos={0 0}, upload=57738]: Uploading piece... Jul 26 23:47:11.488609163 (60h11m21.211172203s) [noise=true , address=45.91.116.5:4298 , piece_id=RUYY2T7X3HZAFXTG2GIYFD3V4RUWSNLGXIW3JMTM2SMJASAY2ARA, seg_pos={0 0}, upload=57738]: Uploading piece... Jul 26 23:47:11.488704906 (60h11m21.211267956s) [noise=true , address=104.36.82.11:4113 , piece_id=OSNH3V5UYMSRB2YCACPNTQ2YDCPHWOXK2OOP5MMQLUOYVII7S22Q, seg_pos={0 0}, upload=57738]: Uploading piece... Jul 26 23:47:11.488934715 (60h11m21.211497755s) [noise=true , address=45.91.116.11:4119 , piece_id=SGKGYVIRJ3XYI6FVQ34OY3IDPX6M2VFYVYDINMBA7QE2IBZZ6NUA, seg_pos={0 0}, upload=57738]: Uploading piece... Jul 26 23:47:11.489237916 (60h11m21.211800957s) [noise=true , address=104.36.82.15:4257 , piece_id=GIXA4H7GJPSROYK6TZWQFLMAZ2E2SPDQLLKXYV3KOYYX6QRFQM5Q, seg_pos={0 0}, upload=57738]: Uploading piece... Jul 26 23:47:11.489526618 (60h11m21.212089658s) [noise=true , address=104.36.82.9:4006 , piece_id=ZOAHQYYWKANR2CIFGRV5LGAH4Z4ZW3O4ERVY47YA5XKOMFQK4L3Q, seg_pos={0 0}, upload=57738]: Uploading piece... Jul 26 23:47:13.195410326 (60h11m22.917973366s) [upload=57738]: Next returned nil segment Jul 26 23:47:13.195427780 (60h11m22.91799082s) [upload=57738]: Waiting for error group managing segments... Jul 26 23:47:13.595535195 (60h11m23.318098235s) [noise=true , address=45.91.116.13:4155 , piece_id=7HOYKE4U2QREIMDW25QLGTDKPZTJ7BASOH6SCM5JI7Q7MQ6KWBBA, seg_pos={0 0}, upload=57738]: Done uploading piece. err: <nil> Jul 26 23:47:13.595940521 (60h11m23.318503561s) [noise=true , address=45.91.116.12:4225 , piece_id=7CG6KLP6BMUMO2CNTGNOOJSZYUWZIGZNBUF6GAR4KUTQDZQ4EMLA, seg_pos={0 0}, upload=57738]: Done uploading piece. err: <nil> Jul 26 23:47:13.598170267 (60h11m23.320733307s) [noise=true , address=45.91.116.8:4047 , piece_id=U45PMTJ3VMEHYU7WCHBGVBOOFYFR2PRPN5L3MK3R3GKDHE64X3LA, seg_pos={0 0}, upload=57738]: Done uploading piece. err: <nil> Jul 26 23:47:13.598635156 (60h11m23.321198196s) [noise=true , address=45.91.116.7:4009 , piece_id=BYZ6H5YHNHB2JVADGMN2KTC4BSCA4Y6GKMJZZLRF3TG7JX74FL6Q, seg_pos={0 0}, upload=57738]: Done uploading piece. err: <nil> Jul 26 23:47:13.598884493 (60h11m23.321447533s) [noise=true , address=77.90.31.14:4661 , piece_id=ZMAQBP4VPBAHALI5ULY2BLHDW7B7SKHGCOIPHCKYL4LCDF5I6KZQ, seg_pos={0 0}, upload=57738]: Done uploading piece. err: <nil> Jul 26 23:47:13.599954681 (60h11m23.322517711s) [noise=true , address=45.91.116.13:4154 , piece_id=YWLJD6ZW66H7ZMW5NHVKXXV3OZZOAQOUDJ42YMHQRY7DBDSEK65A, seg_pos={0 0}, upload=57738]: Done uploading piece. err: <nil> Jul 26 23:47:13.612720616 (60h11m23.335283656s) [noise=true , address=77.90.31.10:4517 , piece_id=37FYOW57Q5JSF7CVIY5LOMOB6LU4SDII67GYJUG65K7WD7KYTKRA, seg_pos={0 0}, upload=57738]: Done uploading piece. err: <nil> Jul 26 23:47:13.725287690 (60h11m23.44785073s) [noise=true , address=45.91.116.11:4119 , piece_id=SGKGYVIRJ3XYI6FVQ34OY3IDPX6M2VFYVYDINMBA7QE2IBZZ6NUA, seg_pos={0 0}, upload=57738]: Done uploading piece. err: <nil> Jul 26 23:47:13.990246987 (60h11m23.712810027s) [noise=true , address=45.91.116.5:4298 , piece_id=RUYY2T7X3HZAFXTG2GIYFD3V4RUWSNLGXIW3JMTM2SMJASAY2ARA, seg_pos={0 0}, upload=57738]: Done uploading piece. err: <nil> Jul 26 23:47:40.541429002 (60h11m50.263992042s) [noise=true , address=104.36.82.11:4113 , piece_id=OSNH3V5UYMSRB2YCACPNTQ2YDCPHWOXK2OOP5MMQLUOYVII7S22Q, seg_pos={0 0}, upload=57738]: Done uploading piece. err: ecclient: upload failed (node:1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV, address:104.36.82.11:4113): protocol: rpc: dial tcp 104.36.82.11:4113: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:47:40.541484658 (60h11m50.264047698s) [noise=true , address=104.36.82.3:4765 , piece_id=SZGZ47TSCSAU6Z3UI4LB2AJIXEYRK2SK67H4M65JGLTJQLNO2YJQ, seg_pos={0 0}, upload=57738]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:104.36.82.3:4765): protocol: rpc: dial tcp 104.36.82.3:4765: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:47:40.541520286 (60h11m50.264083326s) [noise=true , address=104.36.82.2:4731 , piece_id=N4MNBERLWY5EXS7WQ5FC6S2PRMCSNIYLXYUHWAYY4H7JIE2HUR7Q, seg_pos={0 0}, upload=57738]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:104.36.82.2:4731): protocol: rpc: dial tcp 104.36.82.2:4731: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:47:40.541550925 (60h11m50.264113965s) [noise=true , address=104.36.82.15:4257 , piece_id=GIXA4H7GJPSROYK6TZWQFLMAZ2E2SPDQLLKXYV3KOYYX6QRFQM5Q, seg_pos={0 0}, upload=57738]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:104.36.82.15:4257): protocol: rpc: dial tcp 104.36.82.15:4257: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:47:40.541643142 (60h11m50.264206182s) [noise=true , address=104.36.82.5:4839 , piece_id=QUCHCVMSO3JZHPOQEBDW3JCZCLMFWTALYCJELFFCZBIOETVHKGTA, seg_pos={0 0}, upload=57738]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:104.36.82.5:4839): protocol: rpc: dial tcp 104.36.82.5:4839: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:47:40.541672939 (60h11m50.264235979s) [noise=true , address=104.36.82.9:4006 , piece_id=ZOAHQYYWKANR2CIFGRV5LGAH4Z4ZW3O4ERVY47YA5XKOMFQK4L3Q, seg_pos={0 0}, upload=57738]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: write tcp 188.166.225.170:35544->104.36.82.9:4006: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:47:40.541703086 (60h11m50.264266116s) [noise=true , address=77.90.31.18:4806 , piece_id=XCBDPEMBZGTDSH5S5IAXXTHSNMRS4CIG56UY7ASFJNDUGEW6XXUQ, seg_pos={0 0}, upload=57738]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: read tcp 188.166.225.170:48530->77.90.31.18:4806: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:47:40.541715359 (60h11m50.264278389s) [seg_pos={0 0}, upload=57738]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Jul 26 23:47:40.904931641 (60h11m50.627494681s) [upload=57738]: Done uploading segments... Jul 26 23:47:40.904955066 (60h11m50.627518106s) [upload=57738]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 30.36517113s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 20 out of 60 is failed (threshold=3) [..XX..XX.XX..XXX............XXXX.....XXX..X.....X......X...X] duration: 1m0.001494296s FAIL (2025-07-26 23:47:28 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.001518343s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 1.82953405s FAIL (2025-07-26 23:47:25 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K Duration: 1.829559848s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 57 out of 60 is failed (threshold=3) [XXXXX...XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.227541718s FAIL (2025-07-26 23:47:15 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== tBegin Jul 26 23:46:16.785360477 (60h10m26.507923507s) [seg_pos={0 0}, upload=57729]: Begin upload segment... Jul 26 23:46:16.785417796 (60h10m26.507980836s) [seg_pos={0 0}, upload=57729]: Done begin upload segment. Jul 26 23:46:16.785440901 (60h10m26.508003941s) [noise=true , address=64.49.13.138:4482 , piece_id=SAYCX2NROKX35TY44SVBEBCDTWGJH4TBMPNE2DHOKRQE6QAIIPZA, seg_pos={0 0}, upload=57729]: Uploading piece... Jul 26 23:46:16.785500324 (60h10m26.508063404s) [noise=true , address=104.36.82.47:4548 , piece_id=SCPLK6ZV5D6INCWUZYVKIDO24AOOOP6SVTSR2FNWODU3W7GF77TQ, seg_pos={0 0}, upload=57729]: Uploading piece... Jul 26 23:46:16.785844693 (60h10m26.508407734s) [noise=true , address=79.127.219.51:4210 , piece_id=KCIIQD5OJ2JR4TUKPDEGULQXK7MOOZVE6UFDWHNAUZMDY5QWCLBQ, seg_pos={0 0}, upload=57729]: Uploading piece... Jul 26 23:46:16.786108468 (60h10m26.508671498s) [noise=true , address=45.91.116.9:4277 , piece_id=OZAPOY6FR6LQS5PV45TT26TZRFPIOFZF3UTCM5Q5BB3TN7QGXIYA, seg_pos={0 0}, upload=57729]: Uploading piece... Jul 26 23:46:16.786364368 (60h10m26.508927398s) [noise=true , address=77.90.31.11:4558 , piece_id=7WNE7DEOH2L3RBJ4KVRRYSN2OBSL7Y5PEOI42XXW7XH4VEV7WREA, seg_pos={0 0}, upload=57729]: Uploading piece... Jul 26 23:46:16.786377442 (60h10m26.508940512s) [noise=true , address=79.127.205.233:4633 , piece_id=A7RTI4GROCTKPJ4EQ5RHDKUHVCDH2O7OI7EYSX7KCOMCQWKA4A4A, seg_pos={0 0}, upload=57729]: Uploading piece... Jul 26 23:46:16.786663330 (60h10m26.50922636s) [noise=true , address=77.90.31.17:4801 , piece_id=MHG4TOW5NV5BVQENIQYCUD7DNBMHSCPYKJSPYMLGW73EVCM3NTIA, seg_pos={0 0}, upload=57729]: Uploading piece... Jul 26 23:46:16.786906955 (60h10m26.509469995s) [noise=true , address=79.127.205.235:4691 , piece_id=M7WJVQJBJCM5KFHQMJXWDE7WEIZ2UP6FNJGG2UK5HQVHUMZX4NEA, seg_pos={0 0}, upload=57729]: Uploading piece... Jul 26 23:46:16.786982290 (60h10m26.50954533s) [noise=true , address=104.36.82.20:4457 , piece_id=QLMAAV4PMXRESEGGU5OCQTCOPEGABENRRROUZSECMFU425HZ2UWQ, seg_pos={0 0}, upload=57729]: Uploading piece... Jul 26 23:46:16.787129922 (60h10m26.509692952s) [noise=true , address=77.90.31.10:4538 , piece_id=ESVGTFK2DYGC4DFDQQJQT4M7T4MD3HLX4JHGNRGUQLCTUMAVVZRQ, seg_pos={0 0}, upload=57729]: Uploading piece... Jul 26 23:46:16.787341697 (60h10m26.509904737s) [noise=true , address=79.127.219.38:4013 , piece_id=QU3VBSO4CC3WTWRXDHLE22NJX5SHHDUVMN2Q6KQRULZ7H3BV2WDA, seg_pos={0 0}, upload=57729]: Uploading piece... Jul 26 23:46:16.787555306 (60h10m26.510118346s) [noise=true , address=64.49.13.137:4502 , piece_id=QV4TPCWTAOVQ3SRXT5YDUGSPQAP56QUX4Y25DSODBGA5H6VPMQ2A, seg_pos={0 0}, upload=57729]: Uploading piece... Jul 26 23:46:16.787776099 (60h10m26.510339139s) [noise=true , address=104.36.82.56:4256 , piece_id=GV7YIV6OFBM3IVYENUHTMEPBGJR2M4ZREAL6PRJ5KQ72MCLY67TQ, seg_pos={0 0}, upload=57729]: Uploading piece... Jul 26 23:46:16.787983125 (60h10m26.510546165s) [noise=true , address=104.36.82.55:4224 , piece_id=FJXNYKNAFMTYPABDI4V6DD5TEEHPCMKWWZQB24KDJ7P6WF33ZSAQ, seg_pos={0 0}, upload=57729]: Uploading piece... Jul 26 23:46:16.788241830 (60h10m26.51080486s) [noise=true , address=64.49.13.140:4514 , piece_id=DE2SMOOIJFZOQESB34BKHNIWKPOTUJ5AZPYFG3YPEFKMC42TD53A, seg_pos={0 0}, upload=57729]: Uploading piece... Jul 26 23:46:16.788389893 (60h10m26.510952953s) [noise=true , address=104.36.82.46:4500 , piece_id=TDHYD5GASBBOOS57J6QJI55JZN3RITQDSH5L7SCHN6FZ64I3PZFA, seg_pos={0 0}, upload=57729]: Uploading piece... Jul 26 23:46:16.788733260 (60h10m26.5112963s) [noise=true , address=79.127.201.215:4346 , piece_id=4KLKIIUOI3QRIWJH7TECJKQGTLUWSHOHKW6CIR54ZGHHKLX3TD3Q, seg_pos={0 0}, upload=57729]: Uploading piece... Jul 26 23:46:16.788971495 (60h10m26.511534535s) [noise=true , address=104.36.82.54:4180 , piece_id=I6HMNTSATACVU6VK6WYIRUYK6A3QTVE627DOMVAFJXEM4D5YKDQQ, seg_pos={0 0}, upload=57729]: Uploading piece... Jul 26 23:46:17.913585960 (60h10m27.636149s) [upload=57729]: Next returned nil segment Jul 26 23:46:17.913604927 (60h10m27.636167967s) [upload=57729]: Waiting for error group managing segments... Jul 26 23:46:18.207231742 (60h10m27.929794782s) [noise=true , address=79.127.205.235:4691 , piece_id=M7WJVQJBJCM5KFHQMJXWDE7WEIZ2UP6FNJGG2UK5HQVHUMZX4NEA, seg_pos={0 0}, upload=57729]: Done uploading piece. err: <nil> Jul 26 23:46:18.221244593 (60h10m27.943807634s) [noise=true , address=79.127.205.233:4633 , piece_id=A7RTI4GROCTKPJ4EQ5RHDKUHVCDH2O7OI7EYSX7KCOMCQWKA4A4A, seg_pos={0 0}, upload=57729]: Done uploading piece. err: <nil> Jul 26 23:46:18.255145166 (60h10m27.977708206s) [noise=true , address=64.49.13.138:4482 , piece_id=SAYCX2NROKX35TY44SVBEBCDTWGJH4TBMPNE2DHOKRQE6QAIIPZA, seg_pos={0 0}, upload=57729]: Done uploading piece. err: <nil> Jul 26 23:46:18.728283091 (60h10m28.450846131s) [noise=true , address=79.127.219.38:4013 , piece_id=QU3VBSO4CC3WTWRXDHLE22NJX5SHHDUVMN2Q6KQRULZ7H3BV2WDA, seg_pos={0 0}, upload=57729]: Done uploading piece. err: <nil> Jul 26 23:46:18.805069325 (60h10m28.527632365s) [noise=true , address=77.90.31.10:4538 , piece_id=ESVGTFK2DYGC4DFDQQJQT4M7T4MD3HLX4JHGNRGUQLCTUMAVVZRQ, seg_pos={0 0}, upload=57729]: Done uploading piece. err: <nil> Jul 26 23:46:18.913081766 (60h10m28.635644806s) [noise=true , address=77.90.31.11:4558 , piece_id=7WNE7DEOH2L3RBJ4KVRRYSN2OBSL7Y5PEOI42XXW7XH4VEV7WREA, seg_pos={0 0}, upload=57729]: Done uploading piece. err: <nil> Jul 26 23:46:18.917900876 (60h10m28.640463917s) [noise=true , address=77.90.31.17:4801 , piece_id=MHG4TOW5NV5BVQENIQYCUD7DNBMHSCPYKJSPYMLGW73EVCM3NTIA, seg_pos={0 0}, upload=57729]: Done uploading piece. err: <nil> Jul 26 23:46:20.436282136 (60h10m30.158845166s) [noise=true , address=79.127.219.51:4210 , piece_id=KCIIQD5OJ2JR4TUKPDEGULQXK7MOOZVE6UFDWHNAUZMDY5QWCLBQ, seg_pos={0 0}, upload=57729]: Done uploading piece. err: <nil> Jul 26 23:46:24.668042435 (60h10m34.390605475s) [noise=true , address=45.91.116.9:4277 , piece_id=OZAPOY6FR6LQS5PV45TT26TZRFPIOFZF3UTCM5Q5BB3TN7QGXIYA, seg_pos={0 0}, upload=57729]: Done uploading piece. err: <nil> Jul 26 23:46:25.480343038 (60h10m35.202906098s) [noise=true , address=64.49.13.140:4514 , piece_id=DE2SMOOIJFZOQESB34BKHNIWKPOTUJ5AZPYFG3YPEFKMC42TD53A, seg_pos={0 0}, upload=57729]: Done uploading piece. err: <nil> Jul 26 23:46:29.152797306 (60h10m38.875360366s) [noise=true , address=64.49.13.137:4502 , piece_id=QV4TPCWTAOVQ3SRXT5YDUGSPQAP56QUX4Y25DSODBGA5H6VPMQ2A, seg_pos={0 0}, upload=57729]: Done uploading piece. err: <nil> Jul 26 23:46:49.252077155 (60h10m58.974640205s) [noise=true , address=79.127.201.215:4346 , piece_id=4KLKIIUOI3QRIWJH7TECJKQGTLUWSHOHKW6CIR54ZGHHKLX3TD3Q, seg_pos={0 0}, upload=57729]: Done uploading piece. err: <nil> Jul 26 23:47:15.702133660 (60h11m25.4246967s) [noise=true , address=104.36.82.54:4180 , piece_id=I6HMNTSATACVU6VK6WYIRUYK6A3QTVE627DOMVAFJXEM4D5YKDQQ, seg_pos={0 0}, upload=57729]: Done uploading piece. err: ecclient: upload failed (node:1rhkcipPBZNFScBjBhWdigzSMwENSmzhCucnC2iuyc9udpweE3, address:104.36.82.54:4180): protocol: write tcp 188.166.225.170:38690->104.36.82.54:4180: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:47:15.702185410 (60h11m25.42474845s) [noise=true , address=104.36.82.55:4224 , piece_id=FJXNYKNAFMTYPABDI4V6DD5TEEHPCMKWWZQB24KDJ7P6WF33ZSAQ, seg_pos={0 0}, upload=57729]: Done uploading piece. err: ecclient: upload failed (node:12EjKmUiv4pxkVCB4WoiXim4uFpxrBPyfiSS4efTY3HKSE3jb5y, address:104.36.82.55:4224): protocol: write tcp 188.166.225.170:57470->104.36.82.55:4224: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:47:15.702255985 (60h11m25.424819025s) [noise=true , address=104.36.82.47:4548 , piece_id=SCPLK6ZV5D6INCWUZYVKIDO24AOOOP6SVTSR2FNWODU3W7GF77TQ, seg_pos={0 0}, upload=57729]: Done uploading piece. err: ecclient: upload failed (node:1P7nZqYyfKc2CT4Sf8bWjEM6zk34vVz9UMifuKRybUjWKQqekT, address:104.36.82.47:4548): protocol: write tcp 188.166.225.170:48308->104.36.82.47:4548: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:47:15.702301321 (60h11m25.424864351s) [noise=true , address=104.36.82.56:4256 , piece_id=GV7YIV6OFBM3IVYENUHTMEPBGJR2M4ZREAL6PRJ5KQ72MCLY67TQ, seg_pos={0 0}, upload=57729]: Done uploading piece. err: ecclient: upload failed (node:11N1BR6rgiYye7CnuM5xDJogUXoUp4FgXrXM41GdhfrRCbbgT8, address:104.36.82.56:4256): protocol: write tcp 188.166.225.170:59998->104.36.82.56:4256: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:47:15.702306821 (60h11m25.424869862s) [noise=true , address=104.36.82.46:4500 , piece_id=TDHYD5GASBBOOS57J6QJI55JZN3RITQDSH5L7SCHN6FZ64I3PZFA, seg_pos={0 0}, upload=57729]: Done uploading piece. err: ecclient: upload failed (node:12B7ox4ehqxJ2mwqeXj8C9N9GuSU5etNC23iT1pJnTgMLGBAGQK, address:104.36.82.46:4500): protocol: write tcp 188.166.225.170:51800->104.36.82.46:4500: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:47:15.702333703 (60h11m25.424896743s) [noise=true , address=104.36.82.20:4457 , piece_id=QLMAAV4PMXRESEGGU5OCQTCOPEGABENRRROUZSECMFU425HZ2UWQ, seg_pos={0 0}, upload=57729]: Done uploading piece. err: ecclient: upload failed (node:16sDrSF9mwW9cQ2h8N5TfUKrnwPAwtEfed6AwCEAT95r5iY9vQ, address:104.36.82.20:4457): protocol: write tcp 188.166.225.170:43296->104.36.82.20:4457: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:47:15.702365845 (60h11m25.424928875s) [seg_pos={0 0}, upload=57729]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jul 26 23:47:15.928029317 (60h11m25.650592357s) [upload=57729]: Done uploading segments... Jul 26 23:47:15.928049916 (60h11m25.650612956s) [upload=57729]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 1m0.227555445s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 50 out of 60 is failed (threshold=3) [..........XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.24281864s FAIL (2025-07-26 23:47:04 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 9) Duration: 30.242843176s core uplink us1 vivint
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [.............XXX.X..X.....XXXXX...XXX.X.....X.X..X..X.X....X] duration: 30.001553992s FAIL (2025-07-26 23:46:50 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.001587658s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 52 out of 60 is failed (threshold=3) [.......XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XX] duration: 30.230043333s FAIL (2025-07-26 23:46:45 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 23:46:16.086175288 (60h10m25.808738328s) [upload=57728]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:46:16.319155716 (60h10m26.041718756s) [seg_pos={0 0}, upload=57728]: Begin upload segment... Jul 26 23:46:16.319195222 (60h10m26.041758262s) [seg_pos={0 0}, upload=57728]: Done begin upload segment. Jul 26 23:46:16.319217716 (60h10m26.041780756s) [noise=true , address=77.90.31.15:4699 , piece_id=2DGPNEHIKWJIUD7LTV7UMN6U5HCS5XDM3RLYZBJUKUIUPXDTQ5JA, seg_pos={0 0}, upload=57728]: Uploading piece... Jul 26 23:46:16.319232263 (60h10m26.041795293s) [noise=true , address=104.36.82.11:4113 , piece_id=7WWLW6WWXKZ2HLKLAAB42Y2QCUQVSMXU4H45WSIHGE4FCZHANSUQ, seg_pos={0 0}, upload=57728]: Uploading piece... Jul 26 23:46:16.319550693 (60h10m26.042113733s) [noise=true , address=104.36.82.6:4873 , piece_id=W5HEDK47TRZ5PV6XE5AIDS4UJBPVAHR2TDZSZT6XSYLYA66J2DIA, seg_pos={0 0}, upload=57728]: Uploading piece... Jul 26 23:46:16.319556954 (60h10m26.042119994s) [noise=true , address=45.91.116.13:4155 , piece_id=MOHC3A3Q5MGWSWQ7PRVYMOKQ2FHMV46R4VGIILR5EXHUKKFMPO2Q, seg_pos={0 0}, upload=57728]: Uploading piece... Jul 26 23:46:16.319793025 (60h10m26.042356066s) [noise=true , address=104.36.82.3:4767 , piece_id=CJZX4AXJDAOAVPERGI2WEQUEFREQIN3QHHLNE73HK55ILTUSGHOQ, seg_pos={0 0}, upload=57728]: Uploading piece... Jul 26 23:46:16.319806111 (60h10m26.042369151s) [noise=true , address=104.36.82.10:4078 , piece_id=JY5YHQZAJG6IHB5ZJOLFKSIDS44UISZIGNI32X4EZSSULXPXTCSQ, seg_pos={0 0}, upload=57728]: Uploading piece... Jul 26 23:46:16.320046361 (60h10m26.042609401s) [noise=true , address=104.36.82.13:4187 , piece_id=35TFH3ZSWAGDEG6QAHOWCG36EGDKO6A4GUMR5FM32YIT5LX5JKOQ, seg_pos={0 0}, upload=57728]: Uploading piece... Jul 26 23:46:16.320049787 (60h10m26.042612817s) [noise=true , address=77.90.31.14:4663 , piece_id=KC2ECSCDX2ZQDM2XRR4L4LG3VI3FEYWMFMKRGRAAYQW5JFGW4ELA, seg_pos={0 0}, upload=57728]: Uploading piece... Jul 26 23:46:16.320337307 (60h10m26.042900347s) [noise=true , address=104.36.82.8:4972 , piece_id=L7EZLIIBAWYSIV6CWRVU7WUOFTRHYNQTVY6WOTKPCVU4E2LZXKYA, seg_pos={0 0}, upload=57728]: Uploading piece... Jul 26 23:46:16.320357977 (60h10m26.042921017s) [noise=true , address=77.90.31.19:4843 , piece_id=4HPTVGNCOUDTNKNQE7T5ZZW2S4SPBMLD4CV3DC337NMN63D7QC7A, seg_pos={0 0}, upload=57728]: Uploading piece... Jul 26 23:46:16.320593939 (60h10m26.043156979s) [noise=true , address=104.36.82.7:4912 , piece_id=WW5JZY7B3LPXG55XABEW5723HGPTZJ4KSISINHUVWVFMBKZHEWZQ, seg_pos={0 0}, upload=57728]: Uploading piece... Jul 26 23:46:16.320616501 (60h10m26.043179541s) [noise=true , address=77.90.31.16:4733 , piece_id=ZWEJWNVMCYC7ORI3IJK5PJ74COXXRHRDCPLKEFQEI7E4DN2V4KQA, seg_pos={0 0}, upload=57728]: Uploading piece... Jul 26 23:46:16.320865828 (60h10m26.043428878s) [noise=true , address=104.36.82.5:4840 , piece_id=GWQUJR3DWAGOMPQ44K252DQYEFP65OR4MMEGY4E326DUV3J3JQSA, seg_pos={0 0}, upload=57728]: Uploading piece... Jul 26 23:46:16.320919371 (60h10m26.043482411s) [noise=true , address=104.36.82.4:4801 , piece_id=7KEO4YVG2TVHYTDE4WYHYVAFWPVFNW2F5BPGW265XWBFNHB4WDDQ, seg_pos={0 0}, upload=57728]: Uploading piece... Jul 26 23:46:16.321101630 (60h10m26.04366467s) [noise=true , address=45.91.116.6:4973 , piece_id=SIBO6RWPXSYU6ZZCWYNQ3UAGGTDVX3IHVCOQAUVPJJIVNNUYY4PQ, seg_pos={0 0}, upload=57728]: Uploading piece... Jul 26 23:46:16.321291022 (60h10m26.043854062s) [noise=true , address=45.91.116.4:4939 , piece_id=JJECXAEQXOGGG4W2N6PYL3YZZWR6LF3XUB3TCRE5BRBZ35WGHNNQ, seg_pos={0 0}, upload=57728]: Uploading piece... Jul 26 23:46:18.069969048 (60h10m27.792532088s) [upload=57728]: Next returned nil segment Jul 26 23:46:18.069994346 (60h10m27.792557376s) [upload=57728]: Waiting for error group managing segments... Jul 26 23:46:18.444036061 (60h10m28.166599091s) [noise=true , address=45.91.116.4:4939 , piece_id=JJECXAEQXOGGG4W2N6PYL3YZZWR6LF3XUB3TCRE5BRBZ35WGHNNQ, seg_pos={0 0}, upload=57728]: Done uploading piece. err: <nil> Jul 26 23:46:18.477896906 (60h10m28.200459936s) [noise=true , address=77.90.31.14:4663 , piece_id=KC2ECSCDX2ZQDM2XRR4L4LG3VI3FEYWMFMKRGRAAYQW5JFGW4ELA, seg_pos={0 0}, upload=57728]: Done uploading piece. err: <nil> Jul 26 23:46:18.494896351 (60h10m28.217459391s) [noise=true , address=45.91.116.6:4973 , piece_id=SIBO6RWPXSYU6ZZCWYNQ3UAGGTDVX3IHVCOQAUVPJJIVNNUYY4PQ, seg_pos={0 0}, upload=57728]: Done uploading piece. err: <nil> Jul 26 23:46:18.499106557 (60h10m28.221669597s) [noise=true , address=77.90.31.15:4699 , piece_id=2DGPNEHIKWJIUD7LTV7UMN6U5HCS5XDM3RLYZBJUKUIUPXDTQ5JA, seg_pos={0 0}, upload=57728]: Done uploading piece. err: <nil> Jul 26 23:46:18.640892356 (60h10m28.363455396s) [noise=true , address=45.91.116.13:4155 , piece_id=MOHC3A3Q5MGWSWQ7PRVYMOKQ2FHMV46R4VGIILR5EXHUKKFMPO2Q, seg_pos={0 0}, upload=57728]: Done uploading piece. err: <nil> Jul 26 23:46:19.794363843 (60h10m29.516926893s) [noise=true , address=77.90.31.19:4843 , piece_id=4HPTVGNCOUDTNKNQE7T5ZZW2S4SPBMLD4CV3DC337NMN63D7QC7A, seg_pos={0 0}, upload=57728]: Done uploading piece. err: <nil> Jul 26 23:46:45.438276230 (60h10m55.16083927s) [noise=true , address=77.90.31.16:4733 , piece_id=ZWEJWNVMCYC7ORI3IJK5PJ74COXXRHRDCPLKEFQEI7E4DN2V4KQA, seg_pos={0 0}, upload=57728]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: rpc: dial tcp 77.90.31.16:4733: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:46:45.438317438 (60h10m55.160880478s) [noise=true , address=104.36.82.4:4801 , piece_id=7KEO4YVG2TVHYTDE4WYHYVAFWPVFNW2F5BPGW265XWBFNHB4WDDQ, seg_pos={0 0}, upload=57728]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: rpc: dial tcp 104.36.82.4:4801: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:46:45.438345952 (60h10m55.160908992s) [noise=true , address=104.36.82.13:4187 , piece_id=35TFH3ZSWAGDEG6QAHOWCG36EGDKO6A4GUMR5FM32YIT5LX5JKOQ, seg_pos={0 0}, upload=57728]: Done uploading piece. err: ecclient: upload failed (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, address:104.36.82.13:4187): protocol: rpc: dial tcp 104.36.82.13:4187: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:46:45.438375609 (60h10m55.160938649s) [noise=true , address=104.36.82.7:4912 , piece_id=WW5JZY7B3LPXG55XABEW5723HGPTZJ4KSISINHUVWVFMBKZHEWZQ, seg_pos={0 0}, upload=57728]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:104.36.82.7:4912): protocol: write tcp 188.166.225.170:33510->104.36.82.7:4912: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:46:45.438403202 (60h10m55.160966242s) [noise=true , address=104.36.82.6:4873 , piece_id=W5HEDK47TRZ5PV6XE5AIDS4UJBPVAHR2TDZSZT6XSYLYA66J2DIA, seg_pos={0 0}, upload=57728]: Done uploading piece. err: ecclient: upload failed (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3, address:104.36.82.6:4873): protocol: rpc: dial tcp 104.36.82.6:4873: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:46:45.438418301 (60h10m55.160981341s) [noise=true , address=104.36.82.8:4972 , piece_id=L7EZLIIBAWYSIV6CWRVU7WUOFTRHYNQTVY6WOTKPCVU4E2LZXKYA, seg_pos={0 0}, upload=57728]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:104.36.82.8:4972): protocol: rpc: dial tcp 104.36.82.8:4972: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:46:45.438432147 (60h10m55.160995177s) [noise=true , address=104.36.82.3:4767 , piece_id=CJZX4AXJDAOAVPERGI2WEQUEFREQIN3QHHLNE73HK55ILTUSGHOQ, seg_pos={0 0}, upload=57728]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:104.36.82.3:4767): protocol: rpc: dial tcp 104.36.82.3:4767: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:46:45.438453859 (60h10m55.161016899s) [noise=true , address=104.36.82.5:4840 , piece_id=GWQUJR3DWAGOMPQ44K252DQYEFP65OR4MMEGY4E326DUV3J3JQSA, seg_pos={0 0}, upload=57728]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:104.36.82.5:4840): protocol: rpc: dial tcp 104.36.82.5:4840: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:46:45.438461523 (60h10m55.161024553s) [noise=true , address=104.36.82.10:4078 , piece_id=JY5YHQZAJG6IHB5ZJOLFKSIDS44UISZIGNI32X4EZSSULXPXTCSQ, seg_pos={0 0}, upload=57728]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: rpc: dial tcp 104.36.82.10:4078: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:46:45.438498966 (60h10m55.161061996s) [noise=true , address=104.36.82.11:4113 , piece_id=7WWLW6WWXKZ2HLKLAAB42Y2QCUQVSMXU4H45WSIHGE4FCZHANSUQ, seg_pos={0 0}, upload=57728]: Done uploading piece. err: ecclient: upload failed (node:1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV, address:104.36.82.11:4113): protocol: rpc: dial tcp 104.36.82.11:4113: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:46:45.438523171 (60h10m55.161086211s) [seg_pos={0 0}, upload=57728]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Jul 26 23:46:45.667029955 (60h10m55.389592995s) [upload=57728]: Done uploading segments... Jul 26 23:46:45.667054953 (60h10m55.389617993s) [upload=57728]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 30.230066668s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 1.955587145s FAIL (2025-07-26 23:46:35 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K Duration: 1.955617483s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 57 out of 60 is failed (threshold=3) [XXXXXX...XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.235616957s FAIL (2025-07-26 23:46:15 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== 2965311s) [upload=57712]: Got next segment. Inline: false Jul 26 23:45:15.907124877 (60h9m25.629687907s) [upload=57712]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 23:45:15.907129857 (60h9m25.629692897s) [upload=57712]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:45:16.355925889 (60h9m26.078488929s) [seg_pos={0 0}, upload=57712]: Begin upload segment... Jul 26 23:45:16.355975635 (60h9m26.078538675s) [seg_pos={0 0}, upload=57712]: Done begin upload segment. Jul 26 23:45:16.355996425 (60h9m26.078559465s) [noise=true , address=79.127.201.214:4321 , piece_id=KUQVZRH7L4DWZLBAIELDSX5GR4JTMPYPBORGHYKTBH6DTEZLKCPA, seg_pos={0 0}, upload=57712]: Uploading piece... Jul 26 23:45:16.356028305 (60h9m26.078591346s) [noise=true , address=64.49.13.138:4483 , piece_id=VQW5DD7DEM3C26T3L5I4HTVVQ3OFULU4RZGTBVHJ26VXFPMVCH4Q, seg_pos={0 0}, upload=57712]: Uploading piece... Jul 26 23:45:16.356390278 (60h9m26.078953388s) [noise=true , address=79.127.219.47:4048 , piece_id=2P7KG2LUVWJO5VMUXMTH3LQ5U37BC7SAMA2IPV4QNHKFWBYNT3FQ, seg_pos={0 0}, upload=57712]: Uploading piece... Jul 26 23:45:16.356721832 (60h9m26.079284872s) [noise=true , address=79.127.205.235:4675 , piece_id=BM7P4RG3JNU4TGLJZJ3W6A5X4GDCXAEHLXDSERFMDB4M7X6ZQRZQ, seg_pos={0 0}, upload=57712]: Uploading piece... Jul 26 23:45:16.357028639 (60h9m26.079591669s) [noise=true , address=79.127.201.215:4345 , piece_id=4LTCQLZJRKRJ7DVF5X6XN6FL3BMNZ7KT5MQQJCSKH62UFEHV6BRA, seg_pos={0 0}, upload=57712]: Uploading piece... Jul 26 23:45:16.357282685 (60h9m26.079845725s) [noise=true , address=104.36.82.46:4496 , piece_id=K2M3U6XOHXVCAZEZCDOA5VA2JNLGPQPR4CRJMWGJ4UGOE72CELVQ, seg_pos={0 0}, upload=57712]: Uploading piece... Jul 26 23:45:16.357534007 (60h9m26.080097047s) [noise=true , address=79.127.219.38:4013 , piece_id=YZPTGNXMYP4DLBBUBJONDFQD7OSZAI6MHOESYPB2RFPYDPDCBZAQ, seg_pos={0 0}, upload=57712]: Uploading piece... Jul 26 23:45:16.357771530 (60h9m26.08033456s) [noise=true , address=79.127.205.233:4653 , piece_id=PTHAPDNMR3GWOMUAXP6ZI2267PH6WNBQUOCKI6AFDJTJI43CGJBQ, seg_pos={0 0}, upload=57712]: Uploading piece... Jul 26 23:45:16.357994978 (60h9m26.080558018s) [noise=true , address=64.49.13.138:4482 , piece_id=XCG7GREXB3OMC7WBHHLS6DI2JAJXBGIFEAJPG4LSE2PA3KV6CD7A, seg_pos={0 0}, upload=57712]: Uploading piece... Jul 26 23:45:16.358195933 (60h9m26.080758973s) [noise=true , address=104.36.82.53:4127 , piece_id=KWOATCKD6ZDKZB54TRA3Z7KIPNILIXY2TZVOAIEOXZ4JF3UYCR3A, seg_pos={0 0}, upload=57712]: Uploading piece... Jul 26 23:45:16.358399342 (60h9m26.080962382s) [noise=true , address=79.127.205.246:4275 , piece_id=XNPUOMMEAMXB4TR7D4IGQUJGSCICUDCIZMISA5WH2JC44YVGWIUQ, seg_pos={0 0}, upload=57712]: Uploading piece... Jul 26 23:45:16.359093830 (60h9m26.08165687s) [noise=true , address=104.36.82.56:4259 , piece_id=75RNI5V5AUNXLXHGS75OPBBVMQWLUY6FO4ZGONKFOOLF5IRSA6ZA, seg_pos={0 0}, upload=57712]: Uploading piece... Jul 26 23:45:16.359326346 (60h9m26.081889386s) [noise=true , address=104.36.82.52:4107 , piece_id=UO6ACROWHIW3FOBGEB2IWRWWJXRFHGGR2OFP35V75JMZU2EOEJNA, seg_pos={0 0}, upload=57712]: Uploading piece... Jul 26 23:45:16.359550324 (60h9m26.082113364s) [noise=true , address=79.127.205.234:4606 , piece_id=YNCLNG7FKGXRNGVLGONQECIW4OPEQU3MI5DDMVER267V6SQ3TYAA, seg_pos={0 0}, upload=57712]: Uploading piece... Jul 26 23:45:16.359739256 (60h9m26.082302296s) [noise=true , address=104.36.82.47:4545 , piece_id=YDJ2ZR6UCADBVK6PWVJILJNIMMFAJBSAVHGVMOJGA34QTTZKDPQQ, seg_pos={0 0}, upload=57712]: Uploading piece... Jul 26 23:45:16.359928568 (60h9m26.082491608s) [noise=true , address=104.36.82.54:4168 , piece_id=WJBQX2JWWX24JALWR34CFQJTEZKDPXW7SGV5VFWREKRVBU7EW5OQ, seg_pos={0 0}, upload=57712]: Uploading piece... Jul 26 23:45:16.360112369 (60h9m26.082675409s) [noise=true , address=64.49.13.137:4502 , piece_id=6XOSWYZ2XAFSVUV3RRFHFTFPZYRTC56MYN4GMTMAQDAG4LIBOTLA, seg_pos={0 0}, upload=57712]: Uploading piece... Jul 26 23:45:16.360316079 (60h9m26.082879119s) [noise=true , address=79.127.205.245:4246 , piece_id=SB7BXKPHW63XMIWPKTPYDQD3OYDGRCQ626OA4JCVTQCVG7URDU2Q, seg_pos={0 0}, upload=57712]: Uploading piece... Jul 26 23:45:17.464763097 (60h9m27.187326137s) [upload=57712]: Next returned nil segment Jul 26 23:45:17.464781933 (60h9m27.187344973s) [upload=57712]: Waiting for error group managing segments... Jul 26 23:45:17.767303667 (60h9m27.489866697s) [noise=true , address=79.127.205.235:4675 , piece_id=BM7P4RG3JNU4TGLJZJ3W6A5X4GDCXAEHLXDSERFMDB4M7X6ZQRZQ, seg_pos={0 0}, upload=57712]: Done uploading piece. err: <nil> Jul 26 23:45:17.781245523 (60h9m27.503808563s) [noise=true , address=79.127.205.234:4606 , piece_id=YNCLNG7FKGXRNGVLGONQECIW4OPEQU3MI5DDMVER267V6SQ3TYAA, seg_pos={0 0}, upload=57712]: Done uploading piece. err: <nil> Jul 26 23:45:17.810089980 (60h9m27.53265302s) [noise=true , address=79.127.205.245:4246 , piece_id=SB7BXKPHW63XMIWPKTPYDQD3OYDGRCQ626OA4JCVTQCVG7URDU2Q, seg_pos={0 0}, upload=57712]: Done uploading piece. err: <nil> Jul 26 23:45:17.817243518 (60h9m27.539806558s) [noise=true , address=79.127.205.246:4275 , piece_id=XNPUOMMEAMXB4TR7D4IGQUJGSCICUDCIZMISA5WH2JC44YVGWIUQ, seg_pos={0 0}, upload=57712]: Done uploading piece. err: <nil> Jul 26 23:45:18.026286159 (60h9m27.748849199s) [noise=true , address=79.127.205.233:4653 , piece_id=PTHAPDNMR3GWOMUAXP6ZI2267PH6WNBQUOCKI6AFDJTJI43CGJBQ, seg_pos={0 0}, upload=57712]: Done uploading piece. err: <nil> Jul 26 23:45:18.283139888 (60h9m28.005702928s) [noise=true , address=79.127.219.38:4013 , piece_id=YZPTGNXMYP4DLBBUBJONDFQD7OSZAI6MHOESYPB2RFPYDPDCBZAQ, seg_pos={0 0}, upload=57712]: Done uploading piece. err: <nil> Jul 26 23:45:19.121308426 (60h9m28.843871466s) [noise=true , address=64.49.13.138:4482 , piece_id=XCG7GREXB3OMC7WBHHLS6DI2JAJXBGIFEAJPG4LSE2PA3KV6CD7A, seg_pos={0 0}, upload=57712]: Done uploading piece. err: <nil> Jul 26 23:45:20.827651392 (60h9m30.550214432s) [noise=true , address=79.127.201.214:4321 , piece_id=KUQVZRH7L4DWZLBAIELDSX5GR4JTMPYPBORGHYKTBH6DTEZLKCPA, seg_pos={0 0}, upload=57712]: Done uploading piece. err: <nil> Jul 26 23:45:21.255261067 (60h9m30.977824107s) [noise=true , address=79.127.219.47:4048 , piece_id=2P7KG2LUVWJO5VMUXMTH3LQ5U37BC7SAMA2IPV4QNHKFWBYNT3FQ, seg_pos={0 0}, upload=57712]: Done uploading piece. err: <nil> Jul 26 23:45:24.027122863 (60h9m33.749685903s) [noise=true , address=64.49.13.137:4502 , piece_id=6XOSWYZ2XAFSVUV3RRFHFTFPZYRTC56MYN4GMTMAQDAG4LIBOTLA, seg_pos={0 0}, upload=57712]: Done uploading piece. err: <nil> Jul 26 23:45:25.911414378 (60h9m35.633977418s) [noise=true , address=64.49.13.138:4483 , piece_id=VQW5DD7DEM3C26T3L5I4HTVVQ3OFULU4RZGTBVHJ26VXFPMVCH4Q, seg_pos={0 0}, upload=57712]: Done uploading piece. err: <nil> Jul 26 23:45:42.530059239 (60h9m52.252622279s) [noise=true , address=79.127.201.215:4345 , piece_id=4LTCQLZJRKRJ7DVF5X6XN6FL3BMNZ7KT5MQQJCSKH62UFEHV6BRA, seg_pos={0 0}, upload=57712]: Done uploading piece. err: <nil> Jul 26 23:46:15.252443877 (60h10m24.975006907s) [noise=true , address=104.36.82.53:4127 , piece_id=KWOATCKD6ZDKZB54TRA3Z7KIPNILIXY2TZVOAIEOXZ4JF3UYCR3A, seg_pos={0 0}, upload=57712]: Done uploading piece. err: ecclient: upload failed (node:12VM12t6Dq6SyApviDYSSUqWnnzW3wTfA4o8VsvJaah4fLj9EWh, address:104.36.82.53:4127): protocol: rpc: dial tcp 104.36.82.53:4127: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:46:15.252482521 (60h10m24.975045561s) [noise=true , address=104.36.82.46:4496 , piece_id=K2M3U6XOHXVCAZEZCDOA5VA2JNLGPQPR4CRJMWGJ4UGOE72CELVQ, seg_pos={0 0}, upload=57712]: Done uploading piece. err: ecclient: upload failed (node:12V8XhSWkyGUVYFR8MykifFAP1YAfGCwzqucDcMMfzw3xsVW77Z, address:104.36.82.46:4496): protocol: rpc: dial tcp 104.36.82.46:4496: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:46:15.252500776 (60h10m24.975063806s) [noise=true , address=104.36.82.47:4545 , piece_id=YDJ2ZR6UCADBVK6PWVJILJNIMMFAJBSAVHGVMOJGA34QTTZKDPQQ, seg_pos={0 0}, upload=57712]: Done uploading piece. err: ecclient: upload failed (node:12euvmhGegnPYufoKKQ1on9FTv2AhuicSVYuWyXtDiY9BLdDMYE, address:104.36.82.47:4545): protocol: rpc: dial tcp 104.36.82.47:4545: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:46:15.252544770 (60h10m24.97510781s) [noise=true , address=104.36.82.52:4107 , piece_id=UO6ACROWHIW3FOBGEB2IWRWWJXRFHGGR2OFP35V75JMZU2EOEJNA, seg_pos={0 0}, upload=57712]: Done uploading piece. err: ecclient: upload failed (node:12YVUtzGaKXr1Leb1URHZuaQXZv4zTtYwnv59j5GaAfd9zoK2Kj, address:104.36.82.52:4107): protocol: rpc: dial tcp 104.36.82.52:4107: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:46:15.252549439 (60h10m24.975112479s) [noise=true , address=104.36.82.56:4259 , piece_id=75RNI5V5AUNXLXHGS75OPBBVMQWLUY6FO4ZGONKFOOLF5IRSA6ZA, seg_pos={0 0}, upload=57712]: Done uploading piece. err: ecclient: upload failed (node:12BCJsPfe1NNUxVUfQWKbvkzviQWYgMxkRZjDhKJ7KFZc1GmnjV, address:104.36.82.56:4259): protocol: write tcp 188.166.225.170:44084->104.36.82.56:4259: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:46:15.252645413 (60h10m24.975208453s) [noise=true , address=104.36.82.54:4168 , piece_id=WJBQX2JWWX24JALWR34CFQJTEZKDPXW7SGV5VFWREKRVBU7EW5OQ, seg_pos={0 0}, upload=57712]: Done uploading piece. err: ecclient: upload failed (node:1XH68Yk8yh2Qa8vcPRYAuLHdGR7Jy4KtSBKYKgwAJpdDvVu7up, address:104.36.82.54:4168): protocol: context deadline exceeded; context deadline exceeded Jul 26 23:46:15.252663347 (60h10m24.975226377s) [seg_pos={0 0}, upload=57712]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jul 26 23:46:15.486286086 (60h10m25.208849126s) [upload=57712]: Done uploading segments... Jul 26 23:46:15.486310353 (60h10m25.208873393s) [upload=57712]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 1m0.235628921s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 49 out of 60 is failed (threshold=3) [...........XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.232946613s FAIL (2025-07-26 23:46:15 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 8) Duration: 30.232974807s core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 19 out of 60 is failed (threshold=3) [...XX..XX.XX..XXX............XXXX.....XXX..X.....X......X...] duration: 18.246118954s FAIL (2025-07-26 23:46:15 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 18.246142177s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 52 out of 60 is failed (threshold=3) [X.......XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.X] duration: 30.244491196s FAIL (2025-07-26 23:45:55 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== 9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX): protocol: write tcp 188.166.225.170:40654->104.36.82.8:4969: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:45:03.518363619 (60h9m13.240926659s) [seg_pos={0 0}, upload=57708]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 26 23:45:03.518442328 (60h9m13.241005378s) [upload=57708]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 26 23:45:03.518447008 (60h9m13.241010048s) [upload=57708]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 23:45:03.984265689 (60h9m13.706828729s) [upload=57708]: Done uploading segments... Jul 26 23:45:03.984290857 (60h9m13.706853907s) [upload=57708]: Upload finished. err: <nil> Jul 26 23:45:25.947536189 (60h9m35.670099229s) [upload=57714]: Starting upload Jul 26 23:45:25.947587507 (60h9m35.670150547s) [upload=57714]: Done starting upload Jul 26 23:45:25.947756801 (60h9m35.670319831s) [upload=57714]: Uploading segments... Jul 26 23:45:25.947786398 (60h9m35.670349438s) [upload=57714]: Got next segment. Inline: false Jul 26 23:45:25.947807859 (60h9m35.670370899s) [upload=57714]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 23:45:25.947812167 (60h9m35.670375207s) [upload=57714]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:45:26.182198920 (60h9m35.90476196s) [seg_pos={0 0}, upload=57714]: Begin upload segment... Jul 26 23:45:26.182242494 (60h9m35.904805524s) [seg_pos={0 0}, upload=57714]: Done begin upload segment. Jul 26 23:45:26.182262772 (60h9m35.904825812s) [noise=true , address=104.36.82.5:4839 , piece_id=6XL2XNANKJVGFLM6BBYQDLCJRS6TYL5YSEE5MPYUA44W4AIHSNSA, seg_pos={0 0}, upload=57714]: Uploading piece... Jul 26 23:45:26.182615907 (60h9m35.905178948s) [noise=true , address=77.90.31.11:4553 , piece_id=V6XJWWU42GBLDSWT2ACJY37BLCFPOHBDJ7T2GIHIM7A5GVTUJKIQ, seg_pos={0 0}, upload=57714]: Uploading piece... Jul 26 23:45:26.182872880 (60h9m35.90543592s) [noise=true , address=104.36.82.3:4767 , piece_id=XJYV37QNQFYWHFDKFVIIEEGAPKB2L673JOYY4QM53M4Z3B5CEZ3Q, seg_pos={0 0}, upload=57714]: Uploading piece... Jul 26 23:45:26.183123670 (60h9m35.90568671s) [noise=true , address=45.91.116.13:4153 , piece_id=GTVECMHO7743L57HRZ3Y4HZ6MJRCTDY243NFSNRGQY6WO6BDC32Q, seg_pos={0 0}, upload=57714]: Uploading piece... Jul 26 23:45:26.183370201 (60h9m35.905933241s) [noise=true , address=104.36.82.13:4185 , piece_id=KC5HNAHKFDGU6BDQB46VZJGXCELNMGHC5TNCSXUZTQJKIN3MSH7A, seg_pos={0 0}, upload=57714]: Uploading piece... Jul 26 23:45:26.183597726 (60h9m35.906160756s) [noise=true , address=45.91.116.6:4974 , piece_id=UTIDFKZOTTR4BZ42PCCQFLLGI62CRYA37OTNKECZ63YIOXIVVOSQ, seg_pos={0 0}, upload=57714]: Uploading piece... Jul 26 23:45:26.183804451 (60h9m35.906367491s) [noise=true , address=77.90.31.10:4518 , piece_id=I6AHWGCTIGIID4ZGQUH4OB3FAVAXF2GOSU2CG7WWLICOM2YG4RWA, seg_pos={0 0}, upload=57714]: Uploading piece... Jul 26 23:45:26.184016778 (60h9m35.906579818s) [noise=true , address=45.91.116.8:4047 , piece_id=U3PKYQDD445DOGFHFUVJGIKWBEBS7CMN2XLXOV3ED3QIHE3NLVHQ, seg_pos={0 0}, upload=57714]: Uploading piece... Jul 26 23:45:26.184272537 (60h9m35.906835577s) [noise=true , address=104.36.82.9:4005 , piece_id=SX2OXSRJD7MLXOPRDPJGBEJ4FTTAYJXKIELY6FHFC2AX3QPCQQDA, seg_pos={0 0}, upload=57714]: Uploading piece... Jul 26 23:45:26.184511775 (60h9m35.907074805s) [noise=true , address=104.36.82.10:4077 , piece_id=PY55WCZ6KWUUAIIG5KB4QATBWJFAL6S7OL4HAN7Q7PQOJ72CHORA, seg_pos={0 0}, upload=57714]: Uploading piece... Jul 26 23:45:26.184691920 (60h9m35.90725496s) [noise=true , address=45.91.116.7:4009 , piece_id=E5BZHDOQFAUTIBOE57DTLSZHIZ2EKHUGC5WXIE5QRLCV76M4YG3Q, seg_pos={0 0}, upload=57714]: Uploading piece... Jul 26 23:45:26.184870702 (60h9m35.907433732s) [noise=true , address=77.90.31.13:4627 , piece_id=ZQ3CHR3IJJAKB65HYTELT3ELMBUH6G33PUOHR3MWF5AOY7RUECMQ, seg_pos={0 0}, upload=57714]: Uploading piece... Jul 26 23:45:26.185049123 (60h9m35.907612164s) [noise=true , address=45.91.116.5:4299 , piece_id=SQYAC6AOWATK2IPTJRIJ3B6V6FPFFCYPPEUHDRDAYVCYNUG7K4IA, seg_pos={0 0}, upload=57714]: Uploading piece... Jul 26 23:45:26.185232363 (60h9m35.907795393s) [noise=true , address=45.91.116.12:4226 , piece_id=KXCKT5HFOJV3YIGQKC3DFIWYSRDH5L4VIPTO27CPMYUTYWBDMXTQ, seg_pos={0 0}, upload=57714]: Uploading piece... Jul 26 23:45:26.185422718 (60h9m35.907985758s) [noise=true , address=77.90.31.10:4520 , piece_id=3Q7ZWQGE43TVV3VSNNXWGFVI5ZRGCH6IHY4TRFTOCBL7IRRA7XNQ, seg_pos={0 0}, upload=57714]: Uploading piece... Jul 26 23:45:26.185602963 (60h9m35.908166003s) [noise=true , address=45.91.116.14:4191 , piece_id=6UR3633BWPEDT75YEB43SMYGLK255ZWUA344SKZL76WK6OAOSABA, seg_pos={0 0}, upload=57714]: Uploading piece... Jul 26 23:45:27.896021848 (60h9m37.618584888s) [upload=57714]: Next returned nil segment Jul 26 23:45:27.896039572 (60h9m37.618602612s) [upload=57714]: Waiting for error group managing segments... Jul 26 23:45:28.196988599 (60h9m37.919551639s) [noise=true , address=45.91.116.6:4974 , piece_id=UTIDFKZOTTR4BZ42PCCQFLLGI62CRYA37OTNKECZ63YIOXIVVOSQ, seg_pos={0 0}, upload=57714]: Done uploading piece. err: <nil> Jul 26 23:45:28.202231519 (60h9m37.924794549s) [noise=true , address=45.91.116.13:4153 , piece_id=GTVECMHO7743L57HRZ3Y4HZ6MJRCTDY243NFSNRGQY6WO6BDC32Q, seg_pos={0 0}, upload=57714]: Done uploading piece. err: <nil> Jul 26 23:45:28.212874422 (60h9m37.935437462s) [noise=true , address=77.90.31.10:4520 , piece_id=3Q7ZWQGE43TVV3VSNNXWGFVI5ZRGCH6IHY4TRFTOCBL7IRRA7XNQ, seg_pos={0 0}, upload=57714]: Done uploading piece. err: <nil> Jul 26 23:45:28.280579824 (60h9m38.003142854s) [noise=true , address=45.91.116.8:4047 , piece_id=U3PKYQDD445DOGFHFUVJGIKWBEBS7CMN2XLXOV3ED3QIHE3NLVHQ, seg_pos={0 0}, upload=57714]: Done uploading piece. err: <nil> Jul 26 23:45:28.284121771 (60h9m38.006684811s) [noise=true , address=45.91.116.12:4226 , piece_id=KXCKT5HFOJV3YIGQKC3DFIWYSRDH5L4VIPTO27CPMYUTYWBDMXTQ, seg_pos={0 0}, upload=57714]: Done uploading piece. err: <nil> Jul 26 23:45:28.287335089 (60h9m38.009898129s) [noise=true , address=77.90.31.11:4553 , piece_id=V6XJWWU42GBLDSWT2ACJY37BLCFPOHBDJ7T2GIHIM7A5GVTUJKIQ, seg_pos={0 0}, upload=57714]: Done uploading piece. err: <nil> Jul 26 23:45:28.293408348 (60h9m38.015971398s) [noise=true , address=45.91.116.7:4009 , piece_id=E5BZHDOQFAUTIBOE57DTLSZHIZ2EKHUGC5WXIE5QRLCV76M4YG3Q, seg_pos={0 0}, upload=57714]: Done uploading piece. err: <nil> Jul 26 23:45:28.339195482 (60h9m38.061758522s) [noise=true , address=45.91.116.5:4299 , piece_id=SQYAC6AOWATK2IPTJRIJ3B6V6FPFFCYPPEUHDRDAYVCYNUG7K4IA, seg_pos={0 0}, upload=57714]: Done uploading piece. err: <nil> Jul 26 23:45:28.492402834 (60h9m38.214965874s) [noise=true , address=77.90.31.10:4518 , piece_id=I6AHWGCTIGIID4ZGQUH4OB3FAVAXF2GOSU2CG7WWLICOM2YG4RWA, seg_pos={0 0}, upload=57714]: Done uploading piece. err: <nil> Jul 26 23:45:28.656682400 (60h9m38.37924544s) [noise=true , address=45.91.116.14:4191 , piece_id=6UR3633BWPEDT75YEB43SMYGLK255ZWUA344SKZL76WK6OAOSABA, seg_pos={0 0}, upload=57714]: Done uploading piece. err: <nil> Jul 26 23:45:29.679389658 (60h9m39.401952698s) [noise=true , address=77.90.31.13:4627 , piece_id=ZQ3CHR3IJJAKB65HYTELT3ELMBUH6G33PUOHR3MWF5AOY7RUECMQ, seg_pos={0 0}, upload=57714]: Done uploading piece. err: <nil> Jul 26 23:45:55.293236149 (60h10m5.015799189s) [noise=true , address=104.36.82.9:4005 , piece_id=SX2OXSRJD7MLXOPRDPJGBEJ4FTTAYJXKIELY6FHFC2AX3QPCQQDA, seg_pos={0 0}, upload=57714]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:104.36.82.9:4005): protocol: rpc: dial tcp 104.36.82.9:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:45:55.293279321 (60h10m5.015842361s) [noise=true , address=104.36.82.5:4839 , piece_id=6XL2XNANKJVGFLM6BBYQDLCJRS6TYL5YSEE5MPYUA44W4AIHSNSA, seg_pos={0 0}, upload=57714]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:104.36.82.5:4839): protocol: rpc: dial tcp 104.36.82.5:4839: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:45:55.293308308 (60h10m5.015871348s) [noise=true , address=104.36.82.13:4185 , piece_id=KC5HNAHKFDGU6BDQB46VZJGXCELNMGHC5TNCSXUZTQJKIN3MSH7A, seg_pos={0 0}, upload=57714]: Done uploading piece. err: ecclient: upload failed (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN, address:104.36.82.13:4185): protocol: rpc: dial tcp 104.36.82.13:4185: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:45:55.293330339 (60h10m5.015893379s) [noise=true , address=104.36.82.3:4767 , piece_id=XJYV37QNQFYWHFDKFVIIEEGAPKB2L673JOYY4QM53M4Z3B5CEZ3Q, seg_pos={0 0}, upload=57714]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:104.36.82.3:4767): protocol: rpc: dial tcp 104.36.82.3:4767: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:45:55.293352502 (60h10m5.015915542s) [noise=true , address=104.36.82.10:4077 , piece_id=PY55WCZ6KWUUAIIG5KB4QATBWJFAL6S7OL4HAN7Q7PQOJ72CHORA, seg_pos={0 0}, upload=57714]: Done uploading piece. err: ecclient: upload failed (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk, address:104.36.82.10:4077): protocol: write tcp 188.166.225.170:44228->104.36.82.10:4077: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:45:55.293384733 (60h10m5.015947773s) [seg_pos={0 0}, upload=57714]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 26 23:45:55.536427667 (60h10m5.258990697s) [upload=57714]: Done uploading segments... Jul 26 23:45:55.536452414 (60h10m5.259015454s) [upload=57714]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 30.244515432s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 1.923999228s FAIL (2025-07-26 23:45:51 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K Duration: 1.924025759s core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 19 out of 60 is failed (threshold=3) [....XX..XX.XX..XXX............XXXX.....XXX..X.....X......X..] duration: 41.584654834s FAIL (2025-07-26 23:45:51 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 41.584674723s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)