Orbiter History

checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [...........................................XX.X.XX.XX..X....] duration: 6.409685493s FAIL (2025-08-04 16:22:11 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.409710491s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 5.559899049s FAIL (2025-08-04 16:22:09 UTC) orbiter warning: Operation: Commit uplink: failed to upload enough pieces (needed at least 14 but got 12) Duration: 5.559921641s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 5.997500932s FAIL (2025-08-04 16:22:02 UTC) orbiter warning: Operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 4.36.82.36:4625 , piece_id=FCK7QGPSKJ7VDK4D7BVNF44BRPQCQS5V5ICAJHELAQHDBSQV5R3Q, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.146274823 (118h10m17.083933322s) [noise=true , address=45.91.116.7:4026 , piece_id=CB6TYC4A6DQOFFKVFPNMF5MH5MGMJHU5Z25B4QXQJI5ETDWLQJJQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.146672965 (118h10m17.084331464s) [noise=true , address=104.36.82.16:4321 , piece_id=42MZPFPEBQNQRJUHQJLZQP2UWGDV2OAXT6Z4KTWPPWLALISASN5Q, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.146726166 (118h10m17.084384665s) [noise=true , address=104.36.82.55:4206 , piece_id=KSAKYO6MALVA5REEKKJZ7MGX7YIRC2PDI5PNHFHRASPBD47V5SOQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.147136371 (118h10m17.08479488s) [noise=true , address=104.36.82.42:4823 , piece_id=66Q6G7YBCVUEAHU3Y5LHPR5MIT7H7NQWI2TWWNHKZT5FYS4DNQBA, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.147479628 (118h10m17.085138127s) [noise=true , address=193.149.164.143:4695 , piece_id=3QTYKOTBCOJ4CRM76FR2XKONKQVYTQQMPTRH3E6MYJR4ODBPY3AQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.147821592 (118h10m17.085480091s) [noise=true , address=104.36.82.62:4411 , piece_id=CIJJ4SMWKFDOIZZ5UEYIXFZHUNAZKRMM2P36IJIT5BHUX5STDZ4A, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.148134220 (118h10m17.085792719s) [noise=true , address=104.36.82.55:4224 , piece_id=NJYMQ2467DXHUEC6ELXITFJF6WWKM4XFEONRJVRF6QSJBDEMC7JA, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.148548573 (118h10m17.086207072s) [noise=true , address=104.36.82.13:4193 , piece_id=WJA72IWFD3TUA2WPGJGB6A7AHJXI4PGOK62YDMKSMDPUAVHUHW4Q, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.148889576 (118h10m17.086548085s) [noise=true , address=104.36.82.39:4728 , piece_id=ZCL6YOUXY4P6HTYUTTILZR4TLG44CDTXWLOJJWOHPFSORA2IBCBA, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.149205931 (118h10m17.08686443s) [noise=true , address=104.36.82.59:4326 , piece_id=CS7FVO3IRXKZFB2EZM2HQBVEVM4XTTFI5Y7TH3NJZGIYMREVY7UQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.149509752 (118h10m17.087168261s) [noise=true , address=104.36.82.45:4930 , piece_id=LBMDUGYMQI7PBBGEOT22S7OGNVRIYS66USSYFAZOSGI445GWKCYA, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.149866545 (118h10m17.087525044s) [noise=true , address=104.36.82.23:4474 , piece_id=PLEKQKMEQKQDW72BNLLY4JOOPFI6CPNNGT74M5EFFNDQQFWELXUQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.150193601 (118h10m17.08785211s) [noise=true , address=104.36.82.13:4212 , piece_id=ER6VIEGFSG2F7542M5P66RS3QAUQYV7BWPQ4YOX55Q3XX6DDCTGA, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.150515557 (118h10m17.088174066s) [noise=true , address=104.36.82.46:4485 , piece_id=P4FMDR5NCQOLKSXQBLE7I7BPEZJ2URAPPVNKOR53ZPROBCE4YAJQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.150935149 (118h10m17.088593648s) [noise=true , address=104.36.82.54:4181 , piece_id=AHW774W42KLBATPDVRQRKH4MQWV5YP2XEEUY6VASDMYJ2BKXCS2Q, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.151249151 (118h10m17.08890765s) [noise=true , address=104.36.82.44:4886 , piece_id=Q3BIO4GNMY64IWXC7VZ5BAYKYRJJ3GKFV7Z4IOG7SP62IUJHXN2A, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.151276363 (118h10m17.088934862s) [noise=true , address=104.36.82.9:4019 , piece_id=SBCPU6XTMB4CRRTFKDBYFQQ2XZZNE3I6AF7VVSGYC356SPQNVHUQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:28.640156348 (118h10m18.577814857s) [upload=113540]: Next returned nil segment Aug 4 16:07:28.640223306 (118h10m18.577881825s) [upload=113540]: Waiting for error group managing segments... Aug 4 16:07:29.062127352 (118h10m18.999785841s) [noise=true , address=104.36.82.16:4321 , piece_id=42MZPFPEBQNQRJUHQJLZQP2UWGDV2OAXT6Z4KTWPPWLALISASN5Q, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.080565981 (118h10m19.01822448s) [noise=true , address=104.36.82.54:4181 , piece_id=AHW774W42KLBATPDVRQRKH4MQWV5YP2XEEUY6VASDMYJ2BKXCS2Q, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.081329852 (118h10m19.018988352s) [noise=true , address=104.36.82.62:4411 , piece_id=CIJJ4SMWKFDOIZZ5UEYIXFZHUNAZKRMM2P36IJIT5BHUX5STDZ4A, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.081846201 (118h10m19.0195047s) [noise=true , address=104.36.82.55:4224 , piece_id=NJYMQ2467DXHUEC6ELXITFJF6WWKM4XFEONRJVRF6QSJBDEMC7JA, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.091017397 (118h10m19.028675896s) [noise=true , address=104.36.82.46:4485 , piece_id=P4FMDR5NCQOLKSXQBLE7I7BPEZJ2URAPPVNKOR53ZPROBCE4YAJQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.105092769 (118h10m19.042751268s) [noise=true , address=104.36.82.13:4212 , piece_id=ER6VIEGFSG2F7542M5P66RS3QAUQYV7BWPQ4YOX55Q3XX6DDCTGA, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.107141759 (118h10m19.044800248s) [noise=true , address=104.36.82.23:4474 , piece_id=PLEKQKMEQKQDW72BNLLY4JOOPFI6CPNNGT74M5EFFNDQQFWELXUQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.116502399 (118h10m19.054160898s) [noise=true , address=45.91.116.7:4026 , piece_id=CB6TYC4A6DQOFFKVFPNMF5MH5MGMJHU5Z25B4QXQJI5ETDWLQJJQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.153680253 (118h10m19.091338752s) [noise=true , address=104.36.82.9:4019 , piece_id=SBCPU6XTMB4CRRTFKDBYFQQ2XZZNE3I6AF7VVSGYC356SPQNVHUQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.210191177 (118h10m19.147849686s) [noise=true , address=104.36.82.36:4625 , piece_id=FCK7QGPSKJ7VDK4D7BVNF44BRPQCQS5V5ICAJHELAQHDBSQV5R3Q, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.219317350 (118h10m19.156975849s) [noise=true , address=104.36.82.55:4206 , piece_id=KSAKYO6MALVA5REEKKJZ7MGX7YIRC2PDI5PNHFHRASPBD47V5SOQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:31.340384557 (118h10m21.278043046s) [noise=true , address=104.36.82.59:4326 , piece_id=CS7FVO3IRXKZFB2EZM2HQBVEVM4XTTFI5Y7TH3NJZGIYMREVY7UQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:32.464238230 (118h10m22.401896729s) [noise=true , address=104.36.82.13:4193 , piece_id=WJA72IWFD3TUA2WPGJGB6A7AHJXI4PGOK62YDMKSMDPUAVHUHW4Q, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:08:26.041033880 (118h11m15.978692379s) [noise=true , address=193.149.164.143:4695 , piece_id=3QTYKOTBCOJ4CRM76FR2XKONKQVYTQQMPTRH3E6MYJR4ODBPY3AQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: ecclient: upload failed (node:12TZfgz24gnZr4UryzEKw6gu94njnmCvafX5X64MKFSJqaUWSVh, address:193.149.164.143:4695): protocol: write tcp 188.166.225.170:43298->193.149.164.143:4695: use of closed network connection; write tcp 188.166.225.170:43298->193.149.164.143:4695: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43298->193.149.164.143:4695: use of closed network connection Aug 4 16:08:26.041079005 (118h11m15.978737504s) [noise=true , address=104.36.82.44:4886 , piece_id=Q3BIO4GNMY64IWXC7VZ5BAYKYRJJ3GKFV7Z4IOG7SP62IUJHXN2A, seg_pos={0 0}, upload=113540]: Done uploading piece. err: ecclient: upload failed (node:1n2cXhqd5EWbNPTXem83jQUjYq5ZaGKBidbVCQzxYfR3rWddGR, address:104.36.82.44:4886): protocol: write tcp 188.166.225.170:38030->104.36.82.44:4886: use of closed network connection; write tcp 188.166.225.170:38030->104.36.82.44:4886: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38030->104.36.82.44:4886: use of closed network connection Aug 4 16:08:26.041128580 (118h11m15.97878711s) [noise=true , address=104.36.82.39:4728 , piece_id=ZCL6YOUXY4P6HTYUTTILZR4TLG44CDTXWLOJJWOHPFSORA2IBCBA, seg_pos={0 0}, upload=113540]: Done uploading piece. err: ecclient: upload failed (node:1YoLzpb4mMYZ8TsdxnMocekZZ1gLfTrdCM8mxTDj4tY6khZGvi, address:104.36.82.39:4728): protocol: write tcp 188.166.225.170:34326->104.36.82.39:4728: use of closed network connection; write tcp 188.166.225.170:34326->104.36.82.39:4728: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34326->104.36.82.39:4728: use of closed network connection Aug 4 16:08:26.041195678 (118h11m15.978854187s) [noise=true , address=104.36.82.45:4930 , piece_id=LBMDUGYMQI7PBBGEOT22S7OGNVRIYS66USSYFAZOSGI445GWKCYA, seg_pos={0 0}, upload=113540]: Done uploading piece. err: ecclient: upload failed (node:12qaYFSpezuLAc5zJfAYruzad6nTpYFDpog7bmKMVTA8MVm76NB, address:104.36.82.45:4930): protocol: expected piece hash; context deadline exceeded; EOF Aug 4 16:08:26.041203374 (118h11m15.978861913s) [noise=true , address=104.36.82.42:4823 , piece_id=66Q6G7YBCVUEAHU3Y5LHPR5MIT7H7NQWI2TWWNHKZT5FYS4DNQBA, seg_pos={0 0}, upload=113540]: Done uploading piece. err: ecclient: upload failed (node:1fkA5DEj9RqsR6N85z7azvujy8tPkUhB5GCqBpUmFb4vvwbd8y, address:104.36.82.42:4823): protocol: write tcp 188.166.225.170:37230->104.36.82.42:4823: use of closed network connection; write tcp 188.166.225.170:37230->104.36.82.42:4823: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37230->104.36.82.42:4823: use of closed network connection Aug 4 16:08:26.041258960 (118h11m15.978917489s) [seg_pos={0 0}, upload=113540]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Aug 4 16:08:26.274088873 (118h11m16.211747372s) [upload=113540]: Done uploading segments... Aug 4 16:08:26.274113650 (118h11m16.211772139s) [upload=113540]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.997529626s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 5.45343994s FAIL (2025-08-04 16:22:00 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.453467091s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 1.854621336s FAIL (2025-08-04 16:21:59 UTC) orbiter warning: Operation: DownloadObject uplink: object not found ("orbiter_us_0/1K") Duration: 1.854649792s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 2.370117496s FAIL (2025-08-04 16:21:55 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.370134999s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [............................................XX.X.XX.XX..X...] duration: 6.031420768s FAIL (2025-08-04 16:21:49 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.031441337s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 6.094713958s FAIL (2025-08-04 16:21:32 UTC) orbiter warning: Operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 4.36.82.36:4625 , piece_id=FCK7QGPSKJ7VDK4D7BVNF44BRPQCQS5V5ICAJHELAQHDBSQV5R3Q, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.146274823 (118h10m17.083933322s) [noise=true , address=45.91.116.7:4026 , piece_id=CB6TYC4A6DQOFFKVFPNMF5MH5MGMJHU5Z25B4QXQJI5ETDWLQJJQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.146672965 (118h10m17.084331464s) [noise=true , address=104.36.82.16:4321 , piece_id=42MZPFPEBQNQRJUHQJLZQP2UWGDV2OAXT6Z4KTWPPWLALISASN5Q, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.146726166 (118h10m17.084384665s) [noise=true , address=104.36.82.55:4206 , piece_id=KSAKYO6MALVA5REEKKJZ7MGX7YIRC2PDI5PNHFHRASPBD47V5SOQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.147136371 (118h10m17.08479488s) [noise=true , address=104.36.82.42:4823 , piece_id=66Q6G7YBCVUEAHU3Y5LHPR5MIT7H7NQWI2TWWNHKZT5FYS4DNQBA, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.147479628 (118h10m17.085138127s) [noise=true , address=193.149.164.143:4695 , piece_id=3QTYKOTBCOJ4CRM76FR2XKONKQVYTQQMPTRH3E6MYJR4ODBPY3AQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.147821592 (118h10m17.085480091s) [noise=true , address=104.36.82.62:4411 , piece_id=CIJJ4SMWKFDOIZZ5UEYIXFZHUNAZKRMM2P36IJIT5BHUX5STDZ4A, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.148134220 (118h10m17.085792719s) [noise=true , address=104.36.82.55:4224 , piece_id=NJYMQ2467DXHUEC6ELXITFJF6WWKM4XFEONRJVRF6QSJBDEMC7JA, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.148548573 (118h10m17.086207072s) [noise=true , address=104.36.82.13:4193 , piece_id=WJA72IWFD3TUA2WPGJGB6A7AHJXI4PGOK62YDMKSMDPUAVHUHW4Q, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.148889576 (118h10m17.086548085s) [noise=true , address=104.36.82.39:4728 , piece_id=ZCL6YOUXY4P6HTYUTTILZR4TLG44CDTXWLOJJWOHPFSORA2IBCBA, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.149205931 (118h10m17.08686443s) [noise=true , address=104.36.82.59:4326 , piece_id=CS7FVO3IRXKZFB2EZM2HQBVEVM4XTTFI5Y7TH3NJZGIYMREVY7UQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.149509752 (118h10m17.087168261s) [noise=true , address=104.36.82.45:4930 , piece_id=LBMDUGYMQI7PBBGEOT22S7OGNVRIYS66USSYFAZOSGI445GWKCYA, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.149866545 (118h10m17.087525044s) [noise=true , address=104.36.82.23:4474 , piece_id=PLEKQKMEQKQDW72BNLLY4JOOPFI6CPNNGT74M5EFFNDQQFWELXUQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.150193601 (118h10m17.08785211s) [noise=true , address=104.36.82.13:4212 , piece_id=ER6VIEGFSG2F7542M5P66RS3QAUQYV7BWPQ4YOX55Q3XX6DDCTGA, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.150515557 (118h10m17.088174066s) [noise=true , address=104.36.82.46:4485 , piece_id=P4FMDR5NCQOLKSXQBLE7I7BPEZJ2URAPPVNKOR53ZPROBCE4YAJQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.150935149 (118h10m17.088593648s) [noise=true , address=104.36.82.54:4181 , piece_id=AHW774W42KLBATPDVRQRKH4MQWV5YP2XEEUY6VASDMYJ2BKXCS2Q, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.151249151 (118h10m17.08890765s) [noise=true , address=104.36.82.44:4886 , piece_id=Q3BIO4GNMY64IWXC7VZ5BAYKYRJJ3GKFV7Z4IOG7SP62IUJHXN2A, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.151276363 (118h10m17.088934862s) [noise=true , address=104.36.82.9:4019 , piece_id=SBCPU6XTMB4CRRTFKDBYFQQ2XZZNE3I6AF7VVSGYC356SPQNVHUQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:28.640156348 (118h10m18.577814857s) [upload=113540]: Next returned nil segment Aug 4 16:07:28.640223306 (118h10m18.577881825s) [upload=113540]: Waiting for error group managing segments... Aug 4 16:07:29.062127352 (118h10m18.999785841s) [noise=true , address=104.36.82.16:4321 , piece_id=42MZPFPEBQNQRJUHQJLZQP2UWGDV2OAXT6Z4KTWPPWLALISASN5Q, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.080565981 (118h10m19.01822448s) [noise=true , address=104.36.82.54:4181 , piece_id=AHW774W42KLBATPDVRQRKH4MQWV5YP2XEEUY6VASDMYJ2BKXCS2Q, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.081329852 (118h10m19.018988352s) [noise=true , address=104.36.82.62:4411 , piece_id=CIJJ4SMWKFDOIZZ5UEYIXFZHUNAZKRMM2P36IJIT5BHUX5STDZ4A, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.081846201 (118h10m19.0195047s) [noise=true , address=104.36.82.55:4224 , piece_id=NJYMQ2467DXHUEC6ELXITFJF6WWKM4XFEONRJVRF6QSJBDEMC7JA, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.091017397 (118h10m19.028675896s) [noise=true , address=104.36.82.46:4485 , piece_id=P4FMDR5NCQOLKSXQBLE7I7BPEZJ2URAPPVNKOR53ZPROBCE4YAJQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.105092769 (118h10m19.042751268s) [noise=true , address=104.36.82.13:4212 , piece_id=ER6VIEGFSG2F7542M5P66RS3QAUQYV7BWPQ4YOX55Q3XX6DDCTGA, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.107141759 (118h10m19.044800248s) [noise=true , address=104.36.82.23:4474 , piece_id=PLEKQKMEQKQDW72BNLLY4JOOPFI6CPNNGT74M5EFFNDQQFWELXUQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.116502399 (118h10m19.054160898s) [noise=true , address=45.91.116.7:4026 , piece_id=CB6TYC4A6DQOFFKVFPNMF5MH5MGMJHU5Z25B4QXQJI5ETDWLQJJQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.153680253 (118h10m19.091338752s) [noise=true , address=104.36.82.9:4019 , piece_id=SBCPU6XTMB4CRRTFKDBYFQQ2XZZNE3I6AF7VVSGYC356SPQNVHUQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.210191177 (118h10m19.147849686s) [noise=true , address=104.36.82.36:4625 , piece_id=FCK7QGPSKJ7VDK4D7BVNF44BRPQCQS5V5ICAJHELAQHDBSQV5R3Q, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.219317350 (118h10m19.156975849s) [noise=true , address=104.36.82.55:4206 , piece_id=KSAKYO6MALVA5REEKKJZ7MGX7YIRC2PDI5PNHFHRASPBD47V5SOQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:31.340384557 (118h10m21.278043046s) [noise=true , address=104.36.82.59:4326 , piece_id=CS7FVO3IRXKZFB2EZM2HQBVEVM4XTTFI5Y7TH3NJZGIYMREVY7UQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:32.464238230 (118h10m22.401896729s) [noise=true , address=104.36.82.13:4193 , piece_id=WJA72IWFD3TUA2WPGJGB6A7AHJXI4PGOK62YDMKSMDPUAVHUHW4Q, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:08:26.041033880 (118h11m15.978692379s) [noise=true , address=193.149.164.143:4695 , piece_id=3QTYKOTBCOJ4CRM76FR2XKONKQVYTQQMPTRH3E6MYJR4ODBPY3AQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: ecclient: upload failed (node:12TZfgz24gnZr4UryzEKw6gu94njnmCvafX5X64MKFSJqaUWSVh, address:193.149.164.143:4695): protocol: write tcp 188.166.225.170:43298->193.149.164.143:4695: use of closed network connection; write tcp 188.166.225.170:43298->193.149.164.143:4695: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43298->193.149.164.143:4695: use of closed network connection Aug 4 16:08:26.041079005 (118h11m15.978737504s) [noise=true , address=104.36.82.44:4886 , piece_id=Q3BIO4GNMY64IWXC7VZ5BAYKYRJJ3GKFV7Z4IOG7SP62IUJHXN2A, seg_pos={0 0}, upload=113540]: Done uploading piece. err: ecclient: upload failed (node:1n2cXhqd5EWbNPTXem83jQUjYq5ZaGKBidbVCQzxYfR3rWddGR, address:104.36.82.44:4886): protocol: write tcp 188.166.225.170:38030->104.36.82.44:4886: use of closed network connection; write tcp 188.166.225.170:38030->104.36.82.44:4886: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38030->104.36.82.44:4886: use of closed network connection Aug 4 16:08:26.041128580 (118h11m15.97878711s) [noise=true , address=104.36.82.39:4728 , piece_id=ZCL6YOUXY4P6HTYUTTILZR4TLG44CDTXWLOJJWOHPFSORA2IBCBA, seg_pos={0 0}, upload=113540]: Done uploading piece. err: ecclient: upload failed (node:1YoLzpb4mMYZ8TsdxnMocekZZ1gLfTrdCM8mxTDj4tY6khZGvi, address:104.36.82.39:4728): protocol: write tcp 188.166.225.170:34326->104.36.82.39:4728: use of closed network connection; write tcp 188.166.225.170:34326->104.36.82.39:4728: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34326->104.36.82.39:4728: use of closed network connection Aug 4 16:08:26.041195678 (118h11m15.978854187s) [noise=true , address=104.36.82.45:4930 , piece_id=LBMDUGYMQI7PBBGEOT22S7OGNVRIYS66USSYFAZOSGI445GWKCYA, seg_pos={0 0}, upload=113540]: Done uploading piece. err: ecclient: upload failed (node:12qaYFSpezuLAc5zJfAYruzad6nTpYFDpog7bmKMVTA8MVm76NB, address:104.36.82.45:4930): protocol: expected piece hash; context deadline exceeded; EOF Aug 4 16:08:26.041203374 (118h11m15.978861913s) [noise=true , address=104.36.82.42:4823 , piece_id=66Q6G7YBCVUEAHU3Y5LHPR5MIT7H7NQWI2TWWNHKZT5FYS4DNQBA, seg_pos={0 0}, upload=113540]: Done uploading piece. err: ecclient: upload failed (node:1fkA5DEj9RqsR6N85z7azvujy8tPkUhB5GCqBpUmFb4vvwbd8y, address:104.36.82.42:4823): protocol: write tcp 188.166.225.170:37230->104.36.82.42:4823: use of closed network connection; write tcp 188.166.225.170:37230->104.36.82.42:4823: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37230->104.36.82.42:4823: use of closed network connection Aug 4 16:08:26.041258960 (118h11m15.978917489s) [seg_pos={0 0}, upload=113540]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Aug 4 16:08:26.274088873 (118h11m16.211747372s) [upload=113540]: Done uploading segments... Aug 4 16:08:26.274113650 (118h11m16.211772139s) [upload=113540]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 6.094732884s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 5.406952498s FAIL (2025-08-04 16:21:21 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.406972856s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 5.664889619s FAIL (2025-08-04 16:21:15 UTC) orbiter warning: Operation: Commit uplink: failed to upload enough pieces (needed at least 14 but got 12) Duration: 5.66491647s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [............................................XX.....X.XXX....] duration: 6.087807993s FAIL (2025-08-04 16:21:12 UTC) orbiter warning: Operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== 66.225.170:57862->104.36.82.13:4186: use of closed network connection Aug 4 16:18:29.634260260 (118h21m19.571918749s) [seg_pos={0 0}, upload=113704]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Aug 4 16:18:29.864378444 (118h21m19.802036943s) [upload=113704]: Done uploading segments... Aug 4 16:18:29.864404103 (118h21m19.802062602s) [upload=113704]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Aug 4 16:18:30.773685844 (118h21m20.711344333s) [upload=113714]: Starting upload Aug 4 16:18:30.773843035 (118h21m20.711501534s) [upload=113714]: Done starting upload Aug 4 16:18:30.773924561 (118h21m20.71158306s) [upload=113714]: Uploading segments... Aug 4 16:18:30.774094206 (118h21m20.711752695s) [upload=113714]: Got next segment. Inline: false Aug 4 16:18:30.774126056 (118h21m20.711784555s) [upload=113714]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 4 16:18:30.774130344 (118h21m20.711788843s) [upload=113714]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 4 16:18:31.011515803 (118h21m20.949174302s) [seg_pos={0 0}, upload=113714]: Begin upload segment... Aug 4 16:18:31.011581389 (118h21m20.949239898s) [seg_pos={0 0}, upload=113714]: Done begin upload segment. Aug 4 16:18:31.011604403 (118h21m20.949262912s) [noise=true , address=77.90.31.17:4770 , piece_id=TMEGOD5OLRWVC56VIWJTPP2WBFBBMEXWCKACT7K26T2HHARZNSQA, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.011720936 (118h21m20.949379435s) [noise=true , address=77.90.31.13:4628 , piece_id=PFEVUZQ4IX6Q7LHKD6VSEW6J2BA2GADLR53SRXIYIYL5XSCQCJNQ, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.011997064 (118h21m20.949655563s) [noise=true , address=77.90.31.21:4879 , piece_id=AXYOW2DPJQVDB4ZOURPUWC7Q3KPXXPVYV2V4E3SIFAQ27B7RY3XA, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.012225821 (118h21m20.94988432s) [noise=true , address=77.90.31.14:4663 , piece_id=3L2HY4QB7EUZ4E4JEYKUPRBQF6QRE2TCFN255LMXAR3YBF4AE65Q, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.012308239 (118h21m20.949966738s) [noise=true , address=77.90.31.19:4841 , piece_id=23RE2PCFCBQOLS7JFWHAX7U3THWIA34JYDQGRV2SQEBJV4RYDDUA, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.012418300 (118h21m20.950076799s) [noise=true , address=77.90.31.21:4877 , piece_id=YTBYW3NK4SXLO3CNQT7YND7XD2IKNN4OTWN6WYASBTJUIMPKUM2Q, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.012613513 (118h21m20.950272012s) [noise=true , address=45.91.116.5:4298 , piece_id=2WPFWY3R5RJC2QO5BXP6IULTUM366JEDEYHEBQDUGPEKFYNEDWIQ, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.012753160 (118h21m20.950411659s) [noise=true , address=45.91.116.10:4082 , piece_id=FRB3EVRTMRTR6VATM7GRKXPO2FXDOQCCPG3DASJRYWZBVMBKNTXQ, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.012811131 (118h21m20.95046963s) [noise=true , address=77.90.31.12:4589 , piece_id=AAMATJFUK77O6VQ4AF57WHRIGQQYUXRHEESXZFYFLPOHS2IGBFYQ, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.013004381 (118h21m20.95066287s) [noise=true , address=45.91.116.8:4045 , piece_id=YCPPIATKRNXMTWXGTCELS2EZPXRPC7RVZZT2YMZZHS6NUH2USLJA, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.013163866 (118h21m20.950822365s) [noise=true , address=45.91.116.14:4190 , piece_id=7QBTI3SCRTWBVGE77THDPVEKBE6F36UOXGXPRTJLWLKSHDAV6HEQ, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.013531159 (118h21m20.951189658s) [noise=true , address=77.90.31.15:4697 , piece_id=6PROJBDPAFJCBHAAQ45RQ2BDIDDUT2KD6UEQTTONFJ6INIK57YAA, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.013872873 (118h21m20.951531372s) [noise=true , address=104.36.82.10:4078 , piece_id=TQPRFOOXJMBYB5NMAXB3QZLQ5G5ABNARWYXEAAQEILVNM4FEYOBA, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.014172115 (118h21m20.951830614s) [noise=true , address=104.36.82.7:4911 , piece_id=BJRLBG4JVK33LDNCJTEAVSQV4U772YELSTGYN6WH4QLPH5C6AJUA, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.015248565 (118h21m20.952907064s) [noise=true , address=45.91.116.11:4118 , piece_id=3DC3BORHVWAGF2PAS2GLEX3NMFMJETBRQQWLNPWLDHFNCCNLXAWA, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.015496128 (118h21m20.953154627s) [noise=true , address=77.90.31.18:4806 , piece_id=3COKTLXWWA3ZSBCOYWYKT5GDUG6H5USPSHBFYVIOL4SKHIBI4CCA, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:32.741770833 (118h21m22.679429352s) [upload=113714]: Next returned nil segment Aug 4 16:18:32.741792233 (118h21m22.679450763s) [upload=113714]: Waiting for error group managing segments... Aug 4 16:18:33.031304408 (118h21m22.968962907s) [noise=true , address=45.91.116.8:4045 , piece_id=YCPPIATKRNXMTWXGTCELS2EZPXRPC7RVZZT2YMZZHS6NUH2USLJA, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:33.033040921 (118h21m22.97069942s) [noise=true , address=45.91.116.5:4298 , piece_id=2WPFWY3R5RJC2QO5BXP6IULTUM366JEDEYHEBQDUGPEKFYNEDWIQ, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:33.034686720 (118h21m22.972345209s) [noise=true , address=77.90.31.12:4589 , piece_id=AAMATJFUK77O6VQ4AF57WHRIGQQYUXRHEESXZFYFLPOHS2IGBFYQ, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:33.036548412 (118h21m22.974206911s) [noise=true , address=77.90.31.21:4877 , piece_id=YTBYW3NK4SXLO3CNQT7YND7XD2IKNN4OTWN6WYASBTJUIMPKUM2Q, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:33.059147582 (118h21m22.996806071s) [noise=true , address=77.90.31.15:4697 , piece_id=6PROJBDPAFJCBHAAQ45RQ2BDIDDUT2KD6UEQTTONFJ6INIK57YAA, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:33.171247436 (118h21m23.108905935s) [noise=true , address=77.90.31.13:4628 , piece_id=PFEVUZQ4IX6Q7LHKD6VSEW6J2BA2GADLR53SRXIYIYL5XSCQCJNQ, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:33.219388326 (118h21m23.157046825s) [noise=true , address=104.36.82.10:4078 , piece_id=TQPRFOOXJMBYB5NMAXB3QZLQ5G5ABNARWYXEAAQEILVNM4FEYOBA, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:33.997614596 (118h21m23.935273095s) [noise=true , address=77.90.31.21:4879 , piece_id=AXYOW2DPJQVDB4ZOURPUWC7Q3KPXXPVYV2V4E3SIFAQ27B7RY3XA, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:34.018191777 (118h21m23.955850276s) [noise=true , address=104.36.82.7:4911 , piece_id=BJRLBG4JVK33LDNCJTEAVSQV4U772YELSTGYN6WH4QLPH5C6AJUA, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:44.685373447 (118h21m34.623031946s) [noise=true , address=77.90.31.14:4663 , piece_id=3L2HY4QB7EUZ4E4JEYKUPRBQF6QRE2TCFN255LMXAR3YBF4AE65Q, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:45.924412567 (118h21m35.862071066s) [noise=true , address=77.90.31.18:4806 , piece_id=3COKTLXWWA3ZSBCOYWYKT5GDUG6H5USPSHBFYVIOL4SKHIBI4CCA, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:53.143254472 (118h21m43.080913011s) [noise=true , address=77.90.31.17:4770 , piece_id=TMEGOD5OLRWVC56VIWJTPP2WBFBBMEXWCKACT7K26T2HHARZNSQA, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:57.690047932 (118h21m47.627706431s) [noise=true , address=77.90.31.19:4841 , piece_id=23RE2PCFCBQOLS7JFWHAX7U3THWIA34JYDQGRV2SQEBJV4RYDDUA, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:57.690075375 (118h21m47.627733874s) [seg_pos={0 0}, upload=113714]: Segment reached optimal threshold of 13 pieces. Aug 4 16:18:57.690245260 (118h21m47.627903749s) [noise=true , address=45.91.116.14:4190 , piece_id=7QBTI3SCRTWBVGE77THDPVEKBE6F36UOXGXPRTJLWLKSHDAV6HEQ, seg_pos={0 0}, upload=113714]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: expected piece hash; context canceled; EOF Aug 4 16:18:57.690312398 (118h21m47.627970897s) [noise=true , address=45.91.116.11:4118 , piece_id=3DC3BORHVWAGF2PAS2GLEX3NMFMJETBRQQWLNPWLDHFNCCNLXAWA, seg_pos={0 0}, upload=113714]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p): protocol: write tcp 188.166.225.170:37362->45.91.116.11:4118: use of closed network connection; write tcp 188.166.225.170:37362->45.91.116.11:4118: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37362->45.91.116.11:4118: use of closed network connection Aug 4 16:18:57.690359428 (118h21m47.628017927s) [noise=true , address=45.91.116.10:4082 , piece_id=FRB3EVRTMRTR6VATM7GRKXPO2FXDOQCCPG3DASJRYWZBVMBKNTXQ, seg_pos={0 0}, upload=113714]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: write tcp 188.166.225.170:46480->45.91.116.10:4082: use of closed network connection; write tcp 188.166.225.170:46480->45.91.116.10:4082: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46480->45.91.116.10:4082: use of closed network connection Aug 4 16:18:57.690376110 (118h21m47.628034609s) [seg_pos={0 0}, upload=113714]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 4 16:18:57.690439481 (118h21m47.62809798s) [upload=113714]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 4 16:18:57.690442958 (118h21m47.628101457s) [upload=113714]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 4 16:18:58.023198180 (118h21m47.960856679s) [upload=113714]: Done uploading segments... Aug 4 16:18:58.023217707 (118h21m47.960876196s) [upload=113714]: Upload finished. err: <nil> Duration: 6.087838321s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 2.318386001s FAIL (2025-08-04 16:21:08 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.318710401s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 1.871142396s FAIL (2025-08-04 16:20:57 UTC) orbiter warning: Operation: DownloadObject uplink: object not found ("orbiter_us_0/1K") Duration: 1.871166441s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 5.536092569s FAIL (2025-08-04 16:20:56 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.536115203s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [.............................................XX.X.XX.XX..X..] duration: 6.005212114s FAIL (2025-08-04 16:20:45 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.005233726s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 2.515884004s FAIL (2025-08-04 16:20:32 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.515912558s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 5.511638176s FAIL (2025-08-04 16:20:30 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.511658004s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.............................................XX.....X.XXX...] duration: 5.809596495s FAIL (2025-08-04 16:20:22 UTC) orbiter warning: Operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== 66.225.170:57862->104.36.82.13:4186: use of closed network connection Aug 4 16:18:29.634260260 (118h21m19.571918749s) [seg_pos={0 0}, upload=113704]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Aug 4 16:18:29.864378444 (118h21m19.802036943s) [upload=113704]: Done uploading segments... Aug 4 16:18:29.864404103 (118h21m19.802062602s) [upload=113704]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Aug 4 16:18:30.773685844 (118h21m20.711344333s) [upload=113714]: Starting upload Aug 4 16:18:30.773843035 (118h21m20.711501534s) [upload=113714]: Done starting upload Aug 4 16:18:30.773924561 (118h21m20.71158306s) [upload=113714]: Uploading segments... Aug 4 16:18:30.774094206 (118h21m20.711752695s) [upload=113714]: Got next segment. Inline: false Aug 4 16:18:30.774126056 (118h21m20.711784555s) [upload=113714]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 4 16:18:30.774130344 (118h21m20.711788843s) [upload=113714]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 4 16:18:31.011515803 (118h21m20.949174302s) [seg_pos={0 0}, upload=113714]: Begin upload segment... Aug 4 16:18:31.011581389 (118h21m20.949239898s) [seg_pos={0 0}, upload=113714]: Done begin upload segment. Aug 4 16:18:31.011604403 (118h21m20.949262912s) [noise=true , address=77.90.31.17:4770 , piece_id=TMEGOD5OLRWVC56VIWJTPP2WBFBBMEXWCKACT7K26T2HHARZNSQA, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.011720936 (118h21m20.949379435s) [noise=true , address=77.90.31.13:4628 , piece_id=PFEVUZQ4IX6Q7LHKD6VSEW6J2BA2GADLR53SRXIYIYL5XSCQCJNQ, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.011997064 (118h21m20.949655563s) [noise=true , address=77.90.31.21:4879 , piece_id=AXYOW2DPJQVDB4ZOURPUWC7Q3KPXXPVYV2V4E3SIFAQ27B7RY3XA, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.012225821 (118h21m20.94988432s) [noise=true , address=77.90.31.14:4663 , piece_id=3L2HY4QB7EUZ4E4JEYKUPRBQF6QRE2TCFN255LMXAR3YBF4AE65Q, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.012308239 (118h21m20.949966738s) [noise=true , address=77.90.31.19:4841 , piece_id=23RE2PCFCBQOLS7JFWHAX7U3THWIA34JYDQGRV2SQEBJV4RYDDUA, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.012418300 (118h21m20.950076799s) [noise=true , address=77.90.31.21:4877 , piece_id=YTBYW3NK4SXLO3CNQT7YND7XD2IKNN4OTWN6WYASBTJUIMPKUM2Q, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.012613513 (118h21m20.950272012s) [noise=true , address=45.91.116.5:4298 , piece_id=2WPFWY3R5RJC2QO5BXP6IULTUM366JEDEYHEBQDUGPEKFYNEDWIQ, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.012753160 (118h21m20.950411659s) [noise=true , address=45.91.116.10:4082 , piece_id=FRB3EVRTMRTR6VATM7GRKXPO2FXDOQCCPG3DASJRYWZBVMBKNTXQ, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.012811131 (118h21m20.95046963s) [noise=true , address=77.90.31.12:4589 , piece_id=AAMATJFUK77O6VQ4AF57WHRIGQQYUXRHEESXZFYFLPOHS2IGBFYQ, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.013004381 (118h21m20.95066287s) [noise=true , address=45.91.116.8:4045 , piece_id=YCPPIATKRNXMTWXGTCELS2EZPXRPC7RVZZT2YMZZHS6NUH2USLJA, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.013163866 (118h21m20.950822365s) [noise=true , address=45.91.116.14:4190 , piece_id=7QBTI3SCRTWBVGE77THDPVEKBE6F36UOXGXPRTJLWLKSHDAV6HEQ, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.013531159 (118h21m20.951189658s) [noise=true , address=77.90.31.15:4697 , piece_id=6PROJBDPAFJCBHAAQ45RQ2BDIDDUT2KD6UEQTTONFJ6INIK57YAA, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.013872873 (118h21m20.951531372s) [noise=true , address=104.36.82.10:4078 , piece_id=TQPRFOOXJMBYB5NMAXB3QZLQ5G5ABNARWYXEAAQEILVNM4FEYOBA, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.014172115 (118h21m20.951830614s) [noise=true , address=104.36.82.7:4911 , piece_id=BJRLBG4JVK33LDNCJTEAVSQV4U772YELSTGYN6WH4QLPH5C6AJUA, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.015248565 (118h21m20.952907064s) [noise=true , address=45.91.116.11:4118 , piece_id=3DC3BORHVWAGF2PAS2GLEX3NMFMJETBRQQWLNPWLDHFNCCNLXAWA, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.015496128 (118h21m20.953154627s) [noise=true , address=77.90.31.18:4806 , piece_id=3COKTLXWWA3ZSBCOYWYKT5GDUG6H5USPSHBFYVIOL4SKHIBI4CCA, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:32.741770833 (118h21m22.679429352s) [upload=113714]: Next returned nil segment Aug 4 16:18:32.741792233 (118h21m22.679450763s) [upload=113714]: Waiting for error group managing segments... Aug 4 16:18:33.031304408 (118h21m22.968962907s) [noise=true , address=45.91.116.8:4045 , piece_id=YCPPIATKRNXMTWXGTCELS2EZPXRPC7RVZZT2YMZZHS6NUH2USLJA, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:33.033040921 (118h21m22.97069942s) [noise=true , address=45.91.116.5:4298 , piece_id=2WPFWY3R5RJC2QO5BXP6IULTUM366JEDEYHEBQDUGPEKFYNEDWIQ, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:33.034686720 (118h21m22.972345209s) [noise=true , address=77.90.31.12:4589 , piece_id=AAMATJFUK77O6VQ4AF57WHRIGQQYUXRHEESXZFYFLPOHS2IGBFYQ, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:33.036548412 (118h21m22.974206911s) [noise=true , address=77.90.31.21:4877 , piece_id=YTBYW3NK4SXLO3CNQT7YND7XD2IKNN4OTWN6WYASBTJUIMPKUM2Q, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:33.059147582 (118h21m22.996806071s) [noise=true , address=77.90.31.15:4697 , piece_id=6PROJBDPAFJCBHAAQ45RQ2BDIDDUT2KD6UEQTTONFJ6INIK57YAA, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:33.171247436 (118h21m23.108905935s) [noise=true , address=77.90.31.13:4628 , piece_id=PFEVUZQ4IX6Q7LHKD6VSEW6J2BA2GADLR53SRXIYIYL5XSCQCJNQ, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:33.219388326 (118h21m23.157046825s) [noise=true , address=104.36.82.10:4078 , piece_id=TQPRFOOXJMBYB5NMAXB3QZLQ5G5ABNARWYXEAAQEILVNM4FEYOBA, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:33.997614596 (118h21m23.935273095s) [noise=true , address=77.90.31.21:4879 , piece_id=AXYOW2DPJQVDB4ZOURPUWC7Q3KPXXPVYV2V4E3SIFAQ27B7RY3XA, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:34.018191777 (118h21m23.955850276s) [noise=true , address=104.36.82.7:4911 , piece_id=BJRLBG4JVK33LDNCJTEAVSQV4U772YELSTGYN6WH4QLPH5C6AJUA, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:44.685373447 (118h21m34.623031946s) [noise=true , address=77.90.31.14:4663 , piece_id=3L2HY4QB7EUZ4E4JEYKUPRBQF6QRE2TCFN255LMXAR3YBF4AE65Q, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:45.924412567 (118h21m35.862071066s) [noise=true , address=77.90.31.18:4806 , piece_id=3COKTLXWWA3ZSBCOYWYKT5GDUG6H5USPSHBFYVIOL4SKHIBI4CCA, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:53.143254472 (118h21m43.080913011s) [noise=true , address=77.90.31.17:4770 , piece_id=TMEGOD5OLRWVC56VIWJTPP2WBFBBMEXWCKACT7K26T2HHARZNSQA, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:57.690047932 (118h21m47.627706431s) [noise=true , address=77.90.31.19:4841 , piece_id=23RE2PCFCBQOLS7JFWHAX7U3THWIA34JYDQGRV2SQEBJV4RYDDUA, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:57.690075375 (118h21m47.627733874s) [seg_pos={0 0}, upload=113714]: Segment reached optimal threshold of 13 pieces. Aug 4 16:18:57.690245260 (118h21m47.627903749s) [noise=true , address=45.91.116.14:4190 , piece_id=7QBTI3SCRTWBVGE77THDPVEKBE6F36UOXGXPRTJLWLKSHDAV6HEQ, seg_pos={0 0}, upload=113714]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: expected piece hash; context canceled; EOF Aug 4 16:18:57.690312398 (118h21m47.627970897s) [noise=true , address=45.91.116.11:4118 , piece_id=3DC3BORHVWAGF2PAS2GLEX3NMFMJETBRQQWLNPWLDHFNCCNLXAWA, seg_pos={0 0}, upload=113714]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p): protocol: write tcp 188.166.225.170:37362->45.91.116.11:4118: use of closed network connection; write tcp 188.166.225.170:37362->45.91.116.11:4118: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37362->45.91.116.11:4118: use of closed network connection Aug 4 16:18:57.690359428 (118h21m47.628017927s) [noise=true , address=45.91.116.10:4082 , piece_id=FRB3EVRTMRTR6VATM7GRKXPO2FXDOQCCPG3DASJRYWZBVMBKNTXQ, seg_pos={0 0}, upload=113714]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: write tcp 188.166.225.170:46480->45.91.116.10:4082: use of closed network connection; write tcp 188.166.225.170:46480->45.91.116.10:4082: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46480->45.91.116.10:4082: use of closed network connection Aug 4 16:18:57.690376110 (118h21m47.628034609s) [seg_pos={0 0}, upload=113714]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 4 16:18:57.690439481 (118h21m47.62809798s) [upload=113714]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 4 16:18:57.690442958 (118h21m47.628101457s) [upload=113714]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 4 16:18:58.023198180 (118h21m47.960856679s) [upload=113714]: Done uploading segments... Aug 4 16:18:58.023217707 (118h21m47.960876196s) [upload=113714]: Upload finished. err: <nil> Duration: 5.809622255s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 5.830453524s FAIL (2025-08-04 16:20:17 UTC) orbiter warning: Operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 4.36.82.36:4625 , piece_id=FCK7QGPSKJ7VDK4D7BVNF44BRPQCQS5V5ICAJHELAQHDBSQV5R3Q, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.146274823 (118h10m17.083933322s) [noise=true , address=45.91.116.7:4026 , piece_id=CB6TYC4A6DQOFFKVFPNMF5MH5MGMJHU5Z25B4QXQJI5ETDWLQJJQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.146672965 (118h10m17.084331464s) [noise=true , address=104.36.82.16:4321 , piece_id=42MZPFPEBQNQRJUHQJLZQP2UWGDV2OAXT6Z4KTWPPWLALISASN5Q, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.146726166 (118h10m17.084384665s) [noise=true , address=104.36.82.55:4206 , piece_id=KSAKYO6MALVA5REEKKJZ7MGX7YIRC2PDI5PNHFHRASPBD47V5SOQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.147136371 (118h10m17.08479488s) [noise=true , address=104.36.82.42:4823 , piece_id=66Q6G7YBCVUEAHU3Y5LHPR5MIT7H7NQWI2TWWNHKZT5FYS4DNQBA, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.147479628 (118h10m17.085138127s) [noise=true , address=193.149.164.143:4695 , piece_id=3QTYKOTBCOJ4CRM76FR2XKONKQVYTQQMPTRH3E6MYJR4ODBPY3AQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.147821592 (118h10m17.085480091s) [noise=true , address=104.36.82.62:4411 , piece_id=CIJJ4SMWKFDOIZZ5UEYIXFZHUNAZKRMM2P36IJIT5BHUX5STDZ4A, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.148134220 (118h10m17.085792719s) [noise=true , address=104.36.82.55:4224 , piece_id=NJYMQ2467DXHUEC6ELXITFJF6WWKM4XFEONRJVRF6QSJBDEMC7JA, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.148548573 (118h10m17.086207072s) [noise=true , address=104.36.82.13:4193 , piece_id=WJA72IWFD3TUA2WPGJGB6A7AHJXI4PGOK62YDMKSMDPUAVHUHW4Q, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.148889576 (118h10m17.086548085s) [noise=true , address=104.36.82.39:4728 , piece_id=ZCL6YOUXY4P6HTYUTTILZR4TLG44CDTXWLOJJWOHPFSORA2IBCBA, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.149205931 (118h10m17.08686443s) [noise=true , address=104.36.82.59:4326 , piece_id=CS7FVO3IRXKZFB2EZM2HQBVEVM4XTTFI5Y7TH3NJZGIYMREVY7UQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.149509752 (118h10m17.087168261s) [noise=true , address=104.36.82.45:4930 , piece_id=LBMDUGYMQI7PBBGEOT22S7OGNVRIYS66USSYFAZOSGI445GWKCYA, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.149866545 (118h10m17.087525044s) [noise=true , address=104.36.82.23:4474 , piece_id=PLEKQKMEQKQDW72BNLLY4JOOPFI6CPNNGT74M5EFFNDQQFWELXUQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.150193601 (118h10m17.08785211s) [noise=true , address=104.36.82.13:4212 , piece_id=ER6VIEGFSG2F7542M5P66RS3QAUQYV7BWPQ4YOX55Q3XX6DDCTGA, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.150515557 (118h10m17.088174066s) [noise=true , address=104.36.82.46:4485 , piece_id=P4FMDR5NCQOLKSXQBLE7I7BPEZJ2URAPPVNKOR53ZPROBCE4YAJQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.150935149 (118h10m17.088593648s) [noise=true , address=104.36.82.54:4181 , piece_id=AHW774W42KLBATPDVRQRKH4MQWV5YP2XEEUY6VASDMYJ2BKXCS2Q, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.151249151 (118h10m17.08890765s) [noise=true , address=104.36.82.44:4886 , piece_id=Q3BIO4GNMY64IWXC7VZ5BAYKYRJJ3GKFV7Z4IOG7SP62IUJHXN2A, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.151276363 (118h10m17.088934862s) [noise=true , address=104.36.82.9:4019 , piece_id=SBCPU6XTMB4CRRTFKDBYFQQ2XZZNE3I6AF7VVSGYC356SPQNVHUQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:28.640156348 (118h10m18.577814857s) [upload=113540]: Next returned nil segment Aug 4 16:07:28.640223306 (118h10m18.577881825s) [upload=113540]: Waiting for error group managing segments... Aug 4 16:07:29.062127352 (118h10m18.999785841s) [noise=true , address=104.36.82.16:4321 , piece_id=42MZPFPEBQNQRJUHQJLZQP2UWGDV2OAXT6Z4KTWPPWLALISASN5Q, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.080565981 (118h10m19.01822448s) [noise=true , address=104.36.82.54:4181 , piece_id=AHW774W42KLBATPDVRQRKH4MQWV5YP2XEEUY6VASDMYJ2BKXCS2Q, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.081329852 (118h10m19.018988352s) [noise=true , address=104.36.82.62:4411 , piece_id=CIJJ4SMWKFDOIZZ5UEYIXFZHUNAZKRMM2P36IJIT5BHUX5STDZ4A, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.081846201 (118h10m19.0195047s) [noise=true , address=104.36.82.55:4224 , piece_id=NJYMQ2467DXHUEC6ELXITFJF6WWKM4XFEONRJVRF6QSJBDEMC7JA, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.091017397 (118h10m19.028675896s) [noise=true , address=104.36.82.46:4485 , piece_id=P4FMDR5NCQOLKSXQBLE7I7BPEZJ2URAPPVNKOR53ZPROBCE4YAJQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.105092769 (118h10m19.042751268s) [noise=true , address=104.36.82.13:4212 , piece_id=ER6VIEGFSG2F7542M5P66RS3QAUQYV7BWPQ4YOX55Q3XX6DDCTGA, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.107141759 (118h10m19.044800248s) [noise=true , address=104.36.82.23:4474 , piece_id=PLEKQKMEQKQDW72BNLLY4JOOPFI6CPNNGT74M5EFFNDQQFWELXUQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.116502399 (118h10m19.054160898s) [noise=true , address=45.91.116.7:4026 , piece_id=CB6TYC4A6DQOFFKVFPNMF5MH5MGMJHU5Z25B4QXQJI5ETDWLQJJQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.153680253 (118h10m19.091338752s) [noise=true , address=104.36.82.9:4019 , piece_id=SBCPU6XTMB4CRRTFKDBYFQQ2XZZNE3I6AF7VVSGYC356SPQNVHUQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.210191177 (118h10m19.147849686s) [noise=true , address=104.36.82.36:4625 , piece_id=FCK7QGPSKJ7VDK4D7BVNF44BRPQCQS5V5ICAJHELAQHDBSQV5R3Q, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.219317350 (118h10m19.156975849s) [noise=true , address=104.36.82.55:4206 , piece_id=KSAKYO6MALVA5REEKKJZ7MGX7YIRC2PDI5PNHFHRASPBD47V5SOQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:31.340384557 (118h10m21.278043046s) [noise=true , address=104.36.82.59:4326 , piece_id=CS7FVO3IRXKZFB2EZM2HQBVEVM4XTTFI5Y7TH3NJZGIYMREVY7UQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:32.464238230 (118h10m22.401896729s) [noise=true , address=104.36.82.13:4193 , piece_id=WJA72IWFD3TUA2WPGJGB6A7AHJXI4PGOK62YDMKSMDPUAVHUHW4Q, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:08:26.041033880 (118h11m15.978692379s) [noise=true , address=193.149.164.143:4695 , piece_id=3QTYKOTBCOJ4CRM76FR2XKONKQVYTQQMPTRH3E6MYJR4ODBPY3AQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: ecclient: upload failed (node:12TZfgz24gnZr4UryzEKw6gu94njnmCvafX5X64MKFSJqaUWSVh, address:193.149.164.143:4695): protocol: write tcp 188.166.225.170:43298->193.149.164.143:4695: use of closed network connection; write tcp 188.166.225.170:43298->193.149.164.143:4695: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43298->193.149.164.143:4695: use of closed network connection Aug 4 16:08:26.041079005 (118h11m15.978737504s) [noise=true , address=104.36.82.44:4886 , piece_id=Q3BIO4GNMY64IWXC7VZ5BAYKYRJJ3GKFV7Z4IOG7SP62IUJHXN2A, seg_pos={0 0}, upload=113540]: Done uploading piece. err: ecclient: upload failed (node:1n2cXhqd5EWbNPTXem83jQUjYq5ZaGKBidbVCQzxYfR3rWddGR, address:104.36.82.44:4886): protocol: write tcp 188.166.225.170:38030->104.36.82.44:4886: use of closed network connection; write tcp 188.166.225.170:38030->104.36.82.44:4886: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38030->104.36.82.44:4886: use of closed network connection Aug 4 16:08:26.041128580 (118h11m15.97878711s) [noise=true , address=104.36.82.39:4728 , piece_id=ZCL6YOUXY4P6HTYUTTILZR4TLG44CDTXWLOJJWOHPFSORA2IBCBA, seg_pos={0 0}, upload=113540]: Done uploading piece. err: ecclient: upload failed (node:1YoLzpb4mMYZ8TsdxnMocekZZ1gLfTrdCM8mxTDj4tY6khZGvi, address:104.36.82.39:4728): protocol: write tcp 188.166.225.170:34326->104.36.82.39:4728: use of closed network connection; write tcp 188.166.225.170:34326->104.36.82.39:4728: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34326->104.36.82.39:4728: use of closed network connection Aug 4 16:08:26.041195678 (118h11m15.978854187s) [noise=true , address=104.36.82.45:4930 , piece_id=LBMDUGYMQI7PBBGEOT22S7OGNVRIYS66USSYFAZOSGI445GWKCYA, seg_pos={0 0}, upload=113540]: Done uploading piece. err: ecclient: upload failed (node:12qaYFSpezuLAc5zJfAYruzad6nTpYFDpog7bmKMVTA8MVm76NB, address:104.36.82.45:4930): protocol: expected piece hash; context deadline exceeded; EOF Aug 4 16:08:26.041203374 (118h11m15.978861913s) [noise=true , address=104.36.82.42:4823 , piece_id=66Q6G7YBCVUEAHU3Y5LHPR5MIT7H7NQWI2TWWNHKZT5FYS4DNQBA, seg_pos={0 0}, upload=113540]: Done uploading piece. err: ecclient: upload failed (node:1fkA5DEj9RqsR6N85z7azvujy8tPkUhB5GCqBpUmFb4vvwbd8y, address:104.36.82.42:4823): protocol: write tcp 188.166.225.170:37230->104.36.82.42:4823: use of closed network connection; write tcp 188.166.225.170:37230->104.36.82.42:4823: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37230->104.36.82.42:4823: use of closed network connection Aug 4 16:08:26.041258960 (118h11m15.978917489s) [seg_pos={0 0}, upload=113540]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Aug 4 16:08:26.274088873 (118h11m16.211747372s) [upload=113540]: Done uploading segments... Aug 4 16:08:26.274113650 (118h11m16.211772139s) [upload=113540]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.830487559s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 5.835839241s FAIL (2025-08-04 16:20:04 UTC) orbiter warning: Operation: Commit uplink: failed to upload enough pieces (needed at least 14 but got 12) Duration: 5.835869389s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [..............................................XX.....X.XXX..] duration: 6.129416102s FAIL (2025-08-04 16:20:03 UTC) orbiter warning: Operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== 66.225.170:57862->104.36.82.13:4186: use of closed network connection Aug 4 16:18:29.634260260 (118h21m19.571918749s) [seg_pos={0 0}, upload=113704]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Aug 4 16:18:29.864378444 (118h21m19.802036943s) [upload=113704]: Done uploading segments... Aug 4 16:18:29.864404103 (118h21m19.802062602s) [upload=113704]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Aug 4 16:18:30.773685844 (118h21m20.711344333s) [upload=113714]: Starting upload Aug 4 16:18:30.773843035 (118h21m20.711501534s) [upload=113714]: Done starting upload Aug 4 16:18:30.773924561 (118h21m20.71158306s) [upload=113714]: Uploading segments... Aug 4 16:18:30.774094206 (118h21m20.711752695s) [upload=113714]: Got next segment. Inline: false Aug 4 16:18:30.774126056 (118h21m20.711784555s) [upload=113714]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 4 16:18:30.774130344 (118h21m20.711788843s) [upload=113714]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 4 16:18:31.011515803 (118h21m20.949174302s) [seg_pos={0 0}, upload=113714]: Begin upload segment... Aug 4 16:18:31.011581389 (118h21m20.949239898s) [seg_pos={0 0}, upload=113714]: Done begin upload segment. Aug 4 16:18:31.011604403 (118h21m20.949262912s) [noise=true , address=77.90.31.17:4770 , piece_id=TMEGOD5OLRWVC56VIWJTPP2WBFBBMEXWCKACT7K26T2HHARZNSQA, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.011720936 (118h21m20.949379435s) [noise=true , address=77.90.31.13:4628 , piece_id=PFEVUZQ4IX6Q7LHKD6VSEW6J2BA2GADLR53SRXIYIYL5XSCQCJNQ, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.011997064 (118h21m20.949655563s) [noise=true , address=77.90.31.21:4879 , piece_id=AXYOW2DPJQVDB4ZOURPUWC7Q3KPXXPVYV2V4E3SIFAQ27B7RY3XA, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.012225821 (118h21m20.94988432s) [noise=true , address=77.90.31.14:4663 , piece_id=3L2HY4QB7EUZ4E4JEYKUPRBQF6QRE2TCFN255LMXAR3YBF4AE65Q, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.012308239 (118h21m20.949966738s) [noise=true , address=77.90.31.19:4841 , piece_id=23RE2PCFCBQOLS7JFWHAX7U3THWIA34JYDQGRV2SQEBJV4RYDDUA, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.012418300 (118h21m20.950076799s) [noise=true , address=77.90.31.21:4877 , piece_id=YTBYW3NK4SXLO3CNQT7YND7XD2IKNN4OTWN6WYASBTJUIMPKUM2Q, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.012613513 (118h21m20.950272012s) [noise=true , address=45.91.116.5:4298 , piece_id=2WPFWY3R5RJC2QO5BXP6IULTUM366JEDEYHEBQDUGPEKFYNEDWIQ, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.012753160 (118h21m20.950411659s) [noise=true , address=45.91.116.10:4082 , piece_id=FRB3EVRTMRTR6VATM7GRKXPO2FXDOQCCPG3DASJRYWZBVMBKNTXQ, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.012811131 (118h21m20.95046963s) [noise=true , address=77.90.31.12:4589 , piece_id=AAMATJFUK77O6VQ4AF57WHRIGQQYUXRHEESXZFYFLPOHS2IGBFYQ, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.013004381 (118h21m20.95066287s) [noise=true , address=45.91.116.8:4045 , piece_id=YCPPIATKRNXMTWXGTCELS2EZPXRPC7RVZZT2YMZZHS6NUH2USLJA, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.013163866 (118h21m20.950822365s) [noise=true , address=45.91.116.14:4190 , piece_id=7QBTI3SCRTWBVGE77THDPVEKBE6F36UOXGXPRTJLWLKSHDAV6HEQ, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.013531159 (118h21m20.951189658s) [noise=true , address=77.90.31.15:4697 , piece_id=6PROJBDPAFJCBHAAQ45RQ2BDIDDUT2KD6UEQTTONFJ6INIK57YAA, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.013872873 (118h21m20.951531372s) [noise=true , address=104.36.82.10:4078 , piece_id=TQPRFOOXJMBYB5NMAXB3QZLQ5G5ABNARWYXEAAQEILVNM4FEYOBA, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.014172115 (118h21m20.951830614s) [noise=true , address=104.36.82.7:4911 , piece_id=BJRLBG4JVK33LDNCJTEAVSQV4U772YELSTGYN6WH4QLPH5C6AJUA, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.015248565 (118h21m20.952907064s) [noise=true , address=45.91.116.11:4118 , piece_id=3DC3BORHVWAGF2PAS2GLEX3NMFMJETBRQQWLNPWLDHFNCCNLXAWA, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.015496128 (118h21m20.953154627s) [noise=true , address=77.90.31.18:4806 , piece_id=3COKTLXWWA3ZSBCOYWYKT5GDUG6H5USPSHBFYVIOL4SKHIBI4CCA, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:32.741770833 (118h21m22.679429352s) [upload=113714]: Next returned nil segment Aug 4 16:18:32.741792233 (118h21m22.679450763s) [upload=113714]: Waiting for error group managing segments... Aug 4 16:18:33.031304408 (118h21m22.968962907s) [noise=true , address=45.91.116.8:4045 , piece_id=YCPPIATKRNXMTWXGTCELS2EZPXRPC7RVZZT2YMZZHS6NUH2USLJA, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:33.033040921 (118h21m22.97069942s) [noise=true , address=45.91.116.5:4298 , piece_id=2WPFWY3R5RJC2QO5BXP6IULTUM366JEDEYHEBQDUGPEKFYNEDWIQ, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:33.034686720 (118h21m22.972345209s) [noise=true , address=77.90.31.12:4589 , piece_id=AAMATJFUK77O6VQ4AF57WHRIGQQYUXRHEESXZFYFLPOHS2IGBFYQ, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:33.036548412 (118h21m22.974206911s) [noise=true , address=77.90.31.21:4877 , piece_id=YTBYW3NK4SXLO3CNQT7YND7XD2IKNN4OTWN6WYASBTJUIMPKUM2Q, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:33.059147582 (118h21m22.996806071s) [noise=true , address=77.90.31.15:4697 , piece_id=6PROJBDPAFJCBHAAQ45RQ2BDIDDUT2KD6UEQTTONFJ6INIK57YAA, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:33.171247436 (118h21m23.108905935s) [noise=true , address=77.90.31.13:4628 , piece_id=PFEVUZQ4IX6Q7LHKD6VSEW6J2BA2GADLR53SRXIYIYL5XSCQCJNQ, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:33.219388326 (118h21m23.157046825s) [noise=true , address=104.36.82.10:4078 , piece_id=TQPRFOOXJMBYB5NMAXB3QZLQ5G5ABNARWYXEAAQEILVNM4FEYOBA, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:33.997614596 (118h21m23.935273095s) [noise=true , address=77.90.31.21:4879 , piece_id=AXYOW2DPJQVDB4ZOURPUWC7Q3KPXXPVYV2V4E3SIFAQ27B7RY3XA, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:34.018191777 (118h21m23.955850276s) [noise=true , address=104.36.82.7:4911 , piece_id=BJRLBG4JVK33LDNCJTEAVSQV4U772YELSTGYN6WH4QLPH5C6AJUA, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:44.685373447 (118h21m34.623031946s) [noise=true , address=77.90.31.14:4663 , piece_id=3L2HY4QB7EUZ4E4JEYKUPRBQF6QRE2TCFN255LMXAR3YBF4AE65Q, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:45.924412567 (118h21m35.862071066s) [noise=true , address=77.90.31.18:4806 , piece_id=3COKTLXWWA3ZSBCOYWYKT5GDUG6H5USPSHBFYVIOL4SKHIBI4CCA, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:53.143254472 (118h21m43.080913011s) [noise=true , address=77.90.31.17:4770 , piece_id=TMEGOD5OLRWVC56VIWJTPP2WBFBBMEXWCKACT7K26T2HHARZNSQA, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:57.690047932 (118h21m47.627706431s) [noise=true , address=77.90.31.19:4841 , piece_id=23RE2PCFCBQOLS7JFWHAX7U3THWIA34JYDQGRV2SQEBJV4RYDDUA, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:57.690075375 (118h21m47.627733874s) [seg_pos={0 0}, upload=113714]: Segment reached optimal threshold of 13 pieces. Aug 4 16:18:57.690245260 (118h21m47.627903749s) [noise=true , address=45.91.116.14:4190 , piece_id=7QBTI3SCRTWBVGE77THDPVEKBE6F36UOXGXPRTJLWLKSHDAV6HEQ, seg_pos={0 0}, upload=113714]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: expected piece hash; context canceled; EOF Aug 4 16:18:57.690312398 (118h21m47.627970897s) [noise=true , address=45.91.116.11:4118 , piece_id=3DC3BORHVWAGF2PAS2GLEX3NMFMJETBRQQWLNPWLDHFNCCNLXAWA, seg_pos={0 0}, upload=113714]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p): protocol: write tcp 188.166.225.170:37362->45.91.116.11:4118: use of closed network connection; write tcp 188.166.225.170:37362->45.91.116.11:4118: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37362->45.91.116.11:4118: use of closed network connection Aug 4 16:18:57.690359428 (118h21m47.628017927s) [noise=true , address=45.91.116.10:4082 , piece_id=FRB3EVRTMRTR6VATM7GRKXPO2FXDOQCCPG3DASJRYWZBVMBKNTXQ, seg_pos={0 0}, upload=113714]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: write tcp 188.166.225.170:46480->45.91.116.10:4082: use of closed network connection; write tcp 188.166.225.170:46480->45.91.116.10:4082: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46480->45.91.116.10:4082: use of closed network connection Aug 4 16:18:57.690376110 (118h21m47.628034609s) [seg_pos={0 0}, upload=113714]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 4 16:18:57.690439481 (118h21m47.62809798s) [upload=113714]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 4 16:18:57.690442958 (118h21m47.628101457s) [upload=113714]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 4 16:18:58.023198180 (118h21m47.960856679s) [upload=113714]: Done uploading segments... Aug 4 16:18:58.023217707 (118h21m47.960876196s) [upload=113714]: Upload finished. err: <nil> Duration: 6.129446089s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 2.360113376s FAIL (2025-08-04 16:20:01 UTC) orbiter warning: Operation: DownloadObject uplink: object not found ("orbiter_us_0/1K") Duration: 2.360145868s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [..............................................XX.X.XX.XX..X.] duration: 6.590201776s FAIL (2025-08-04 16:19:53 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.590224178s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 5.605559712s FAIL (2025-08-04 16:19:39 UTC) orbiter warning: Operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 4.36.82.36:4625 , piece_id=FCK7QGPSKJ7VDK4D7BVNF44BRPQCQS5V5ICAJHELAQHDBSQV5R3Q, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.146274823 (118h10m17.083933322s) [noise=true , address=45.91.116.7:4026 , piece_id=CB6TYC4A6DQOFFKVFPNMF5MH5MGMJHU5Z25B4QXQJI5ETDWLQJJQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.146672965 (118h10m17.084331464s) [noise=true , address=104.36.82.16:4321 , piece_id=42MZPFPEBQNQRJUHQJLZQP2UWGDV2OAXT6Z4KTWPPWLALISASN5Q, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.146726166 (118h10m17.084384665s) [noise=true , address=104.36.82.55:4206 , piece_id=KSAKYO6MALVA5REEKKJZ7MGX7YIRC2PDI5PNHFHRASPBD47V5SOQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.147136371 (118h10m17.08479488s) [noise=true , address=104.36.82.42:4823 , piece_id=66Q6G7YBCVUEAHU3Y5LHPR5MIT7H7NQWI2TWWNHKZT5FYS4DNQBA, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.147479628 (118h10m17.085138127s) [noise=true , address=193.149.164.143:4695 , piece_id=3QTYKOTBCOJ4CRM76FR2XKONKQVYTQQMPTRH3E6MYJR4ODBPY3AQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.147821592 (118h10m17.085480091s) [noise=true , address=104.36.82.62:4411 , piece_id=CIJJ4SMWKFDOIZZ5UEYIXFZHUNAZKRMM2P36IJIT5BHUX5STDZ4A, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.148134220 (118h10m17.085792719s) [noise=true , address=104.36.82.55:4224 , piece_id=NJYMQ2467DXHUEC6ELXITFJF6WWKM4XFEONRJVRF6QSJBDEMC7JA, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.148548573 (118h10m17.086207072s) [noise=true , address=104.36.82.13:4193 , piece_id=WJA72IWFD3TUA2WPGJGB6A7AHJXI4PGOK62YDMKSMDPUAVHUHW4Q, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.148889576 (118h10m17.086548085s) [noise=true , address=104.36.82.39:4728 , piece_id=ZCL6YOUXY4P6HTYUTTILZR4TLG44CDTXWLOJJWOHPFSORA2IBCBA, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.149205931 (118h10m17.08686443s) [noise=true , address=104.36.82.59:4326 , piece_id=CS7FVO3IRXKZFB2EZM2HQBVEVM4XTTFI5Y7TH3NJZGIYMREVY7UQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.149509752 (118h10m17.087168261s) [noise=true , address=104.36.82.45:4930 , piece_id=LBMDUGYMQI7PBBGEOT22S7OGNVRIYS66USSYFAZOSGI445GWKCYA, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.149866545 (118h10m17.087525044s) [noise=true , address=104.36.82.23:4474 , piece_id=PLEKQKMEQKQDW72BNLLY4JOOPFI6CPNNGT74M5EFFNDQQFWELXUQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.150193601 (118h10m17.08785211s) [noise=true , address=104.36.82.13:4212 , piece_id=ER6VIEGFSG2F7542M5P66RS3QAUQYV7BWPQ4YOX55Q3XX6DDCTGA, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.150515557 (118h10m17.088174066s) [noise=true , address=104.36.82.46:4485 , piece_id=P4FMDR5NCQOLKSXQBLE7I7BPEZJ2URAPPVNKOR53ZPROBCE4YAJQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.150935149 (118h10m17.088593648s) [noise=true , address=104.36.82.54:4181 , piece_id=AHW774W42KLBATPDVRQRKH4MQWV5YP2XEEUY6VASDMYJ2BKXCS2Q, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.151249151 (118h10m17.08890765s) [noise=true , address=104.36.82.44:4886 , piece_id=Q3BIO4GNMY64IWXC7VZ5BAYKYRJJ3GKFV7Z4IOG7SP62IUJHXN2A, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.151276363 (118h10m17.088934862s) [noise=true , address=104.36.82.9:4019 , piece_id=SBCPU6XTMB4CRRTFKDBYFQQ2XZZNE3I6AF7VVSGYC356SPQNVHUQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:28.640156348 (118h10m18.577814857s) [upload=113540]: Next returned nil segment Aug 4 16:07:28.640223306 (118h10m18.577881825s) [upload=113540]: Waiting for error group managing segments... Aug 4 16:07:29.062127352 (118h10m18.999785841s) [noise=true , address=104.36.82.16:4321 , piece_id=42MZPFPEBQNQRJUHQJLZQP2UWGDV2OAXT6Z4KTWPPWLALISASN5Q, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.080565981 (118h10m19.01822448s) [noise=true , address=104.36.82.54:4181 , piece_id=AHW774W42KLBATPDVRQRKH4MQWV5YP2XEEUY6VASDMYJ2BKXCS2Q, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.081329852 (118h10m19.018988352s) [noise=true , address=104.36.82.62:4411 , piece_id=CIJJ4SMWKFDOIZZ5UEYIXFZHUNAZKRMM2P36IJIT5BHUX5STDZ4A, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.081846201 (118h10m19.0195047s) [noise=true , address=104.36.82.55:4224 , piece_id=NJYMQ2467DXHUEC6ELXITFJF6WWKM4XFEONRJVRF6QSJBDEMC7JA, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.091017397 (118h10m19.028675896s) [noise=true , address=104.36.82.46:4485 , piece_id=P4FMDR5NCQOLKSXQBLE7I7BPEZJ2URAPPVNKOR53ZPROBCE4YAJQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.105092769 (118h10m19.042751268s) [noise=true , address=104.36.82.13:4212 , piece_id=ER6VIEGFSG2F7542M5P66RS3QAUQYV7BWPQ4YOX55Q3XX6DDCTGA, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.107141759 (118h10m19.044800248s) [noise=true , address=104.36.82.23:4474 , piece_id=PLEKQKMEQKQDW72BNLLY4JOOPFI6CPNNGT74M5EFFNDQQFWELXUQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.116502399 (118h10m19.054160898s) [noise=true , address=45.91.116.7:4026 , piece_id=CB6TYC4A6DQOFFKVFPNMF5MH5MGMJHU5Z25B4QXQJI5ETDWLQJJQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.153680253 (118h10m19.091338752s) [noise=true , address=104.36.82.9:4019 , piece_id=SBCPU6XTMB4CRRTFKDBYFQQ2XZZNE3I6AF7VVSGYC356SPQNVHUQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.210191177 (118h10m19.147849686s) [noise=true , address=104.36.82.36:4625 , piece_id=FCK7QGPSKJ7VDK4D7BVNF44BRPQCQS5V5ICAJHELAQHDBSQV5R3Q, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.219317350 (118h10m19.156975849s) [noise=true , address=104.36.82.55:4206 , piece_id=KSAKYO6MALVA5REEKKJZ7MGX7YIRC2PDI5PNHFHRASPBD47V5SOQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:31.340384557 (118h10m21.278043046s) [noise=true , address=104.36.82.59:4326 , piece_id=CS7FVO3IRXKZFB2EZM2HQBVEVM4XTTFI5Y7TH3NJZGIYMREVY7UQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:32.464238230 (118h10m22.401896729s) [noise=true , address=104.36.82.13:4193 , piece_id=WJA72IWFD3TUA2WPGJGB6A7AHJXI4PGOK62YDMKSMDPUAVHUHW4Q, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:08:26.041033880 (118h11m15.978692379s) [noise=true , address=193.149.164.143:4695 , piece_id=3QTYKOTBCOJ4CRM76FR2XKONKQVYTQQMPTRH3E6MYJR4ODBPY3AQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: ecclient: upload failed (node:12TZfgz24gnZr4UryzEKw6gu94njnmCvafX5X64MKFSJqaUWSVh, address:193.149.164.143:4695): protocol: write tcp 188.166.225.170:43298->193.149.164.143:4695: use of closed network connection; write tcp 188.166.225.170:43298->193.149.164.143:4695: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43298->193.149.164.143:4695: use of closed network connection Aug 4 16:08:26.041079005 (118h11m15.978737504s) [noise=true , address=104.36.82.44:4886 , piece_id=Q3BIO4GNMY64IWXC7VZ5BAYKYRJJ3GKFV7Z4IOG7SP62IUJHXN2A, seg_pos={0 0}, upload=113540]: Done uploading piece. err: ecclient: upload failed (node:1n2cXhqd5EWbNPTXem83jQUjYq5ZaGKBidbVCQzxYfR3rWddGR, address:104.36.82.44:4886): protocol: write tcp 188.166.225.170:38030->104.36.82.44:4886: use of closed network connection; write tcp 188.166.225.170:38030->104.36.82.44:4886: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38030->104.36.82.44:4886: use of closed network connection Aug 4 16:08:26.041128580 (118h11m15.97878711s) [noise=true , address=104.36.82.39:4728 , piece_id=ZCL6YOUXY4P6HTYUTTILZR4TLG44CDTXWLOJJWOHPFSORA2IBCBA, seg_pos={0 0}, upload=113540]: Done uploading piece. err: ecclient: upload failed (node:1YoLzpb4mMYZ8TsdxnMocekZZ1gLfTrdCM8mxTDj4tY6khZGvi, address:104.36.82.39:4728): protocol: write tcp 188.166.225.170:34326->104.36.82.39:4728: use of closed network connection; write tcp 188.166.225.170:34326->104.36.82.39:4728: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34326->104.36.82.39:4728: use of closed network connection Aug 4 16:08:26.041195678 (118h11m15.978854187s) [noise=true , address=104.36.82.45:4930 , piece_id=LBMDUGYMQI7PBBGEOT22S7OGNVRIYS66USSYFAZOSGI445GWKCYA, seg_pos={0 0}, upload=113540]: Done uploading piece. err: ecclient: upload failed (node:12qaYFSpezuLAc5zJfAYruzad6nTpYFDpog7bmKMVTA8MVm76NB, address:104.36.82.45:4930): protocol: expected piece hash; context deadline exceeded; EOF Aug 4 16:08:26.041203374 (118h11m15.978861913s) [noise=true , address=104.36.82.42:4823 , piece_id=66Q6G7YBCVUEAHU3Y5LHPR5MIT7H7NQWI2TWWNHKZT5FYS4DNQBA, seg_pos={0 0}, upload=113540]: Done uploading piece. err: ecclient: upload failed (node:1fkA5DEj9RqsR6N85z7azvujy8tPkUhB5GCqBpUmFb4vvwbd8y, address:104.36.82.42:4823): protocol: write tcp 188.166.225.170:37230->104.36.82.42:4823: use of closed network connection; write tcp 188.166.225.170:37230->104.36.82.42:4823: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37230->104.36.82.42:4823: use of closed network connection Aug 4 16:08:26.041258960 (118h11m15.978917489s) [seg_pos={0 0}, upload=113540]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Aug 4 16:08:26.274088873 (118h11m16.211747372s) [upload=113540]: Done uploading segments... Aug 4 16:08:26.274113650 (118h11m16.211772139s) [upload=113540]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.605587956s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 5.594060821s FAIL (2025-08-04 16:19:38 UTC) orbiter warning: Operation: Commit uplink: failed to upload enough pieces (needed at least 14 but got 12) Duration: 5.594087421s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 40.792828455s FAIL (2025-08-04 16:19:31 UTC) orbiter warning: Operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 4.36.82.36:4625 , piece_id=FCK7QGPSKJ7VDK4D7BVNF44BRPQCQS5V5ICAJHELAQHDBSQV5R3Q, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.146274823 (118h10m17.083933322s) [noise=true , address=45.91.116.7:4026 , piece_id=CB6TYC4A6DQOFFKVFPNMF5MH5MGMJHU5Z25B4QXQJI5ETDWLQJJQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.146672965 (118h10m17.084331464s) [noise=true , address=104.36.82.16:4321 , piece_id=42MZPFPEBQNQRJUHQJLZQP2UWGDV2OAXT6Z4KTWPPWLALISASN5Q, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.146726166 (118h10m17.084384665s) [noise=true , address=104.36.82.55:4206 , piece_id=KSAKYO6MALVA5REEKKJZ7MGX7YIRC2PDI5PNHFHRASPBD47V5SOQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.147136371 (118h10m17.08479488s) [noise=true , address=104.36.82.42:4823 , piece_id=66Q6G7YBCVUEAHU3Y5LHPR5MIT7H7NQWI2TWWNHKZT5FYS4DNQBA, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.147479628 (118h10m17.085138127s) [noise=true , address=193.149.164.143:4695 , piece_id=3QTYKOTBCOJ4CRM76FR2XKONKQVYTQQMPTRH3E6MYJR4ODBPY3AQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.147821592 (118h10m17.085480091s) [noise=true , address=104.36.82.62:4411 , piece_id=CIJJ4SMWKFDOIZZ5UEYIXFZHUNAZKRMM2P36IJIT5BHUX5STDZ4A, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.148134220 (118h10m17.085792719s) [noise=true , address=104.36.82.55:4224 , piece_id=NJYMQ2467DXHUEC6ELXITFJF6WWKM4XFEONRJVRF6QSJBDEMC7JA, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.148548573 (118h10m17.086207072s) [noise=true , address=104.36.82.13:4193 , piece_id=WJA72IWFD3TUA2WPGJGB6A7AHJXI4PGOK62YDMKSMDPUAVHUHW4Q, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.148889576 (118h10m17.086548085s) [noise=true , address=104.36.82.39:4728 , piece_id=ZCL6YOUXY4P6HTYUTTILZR4TLG44CDTXWLOJJWOHPFSORA2IBCBA, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.149205931 (118h10m17.08686443s) [noise=true , address=104.36.82.59:4326 , piece_id=CS7FVO3IRXKZFB2EZM2HQBVEVM4XTTFI5Y7TH3NJZGIYMREVY7UQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.149509752 (118h10m17.087168261s) [noise=true , address=104.36.82.45:4930 , piece_id=LBMDUGYMQI7PBBGEOT22S7OGNVRIYS66USSYFAZOSGI445GWKCYA, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.149866545 (118h10m17.087525044s) [noise=true , address=104.36.82.23:4474 , piece_id=PLEKQKMEQKQDW72BNLLY4JOOPFI6CPNNGT74M5EFFNDQQFWELXUQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.150193601 (118h10m17.08785211s) [noise=true , address=104.36.82.13:4212 , piece_id=ER6VIEGFSG2F7542M5P66RS3QAUQYV7BWPQ4YOX55Q3XX6DDCTGA, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.150515557 (118h10m17.088174066s) [noise=true , address=104.36.82.46:4485 , piece_id=P4FMDR5NCQOLKSXQBLE7I7BPEZJ2URAPPVNKOR53ZPROBCE4YAJQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.150935149 (118h10m17.088593648s) [noise=true , address=104.36.82.54:4181 , piece_id=AHW774W42KLBATPDVRQRKH4MQWV5YP2XEEUY6VASDMYJ2BKXCS2Q, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.151249151 (118h10m17.08890765s) [noise=true , address=104.36.82.44:4886 , piece_id=Q3BIO4GNMY64IWXC7VZ5BAYKYRJJ3GKFV7Z4IOG7SP62IUJHXN2A, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.151276363 (118h10m17.088934862s) [noise=true , address=104.36.82.9:4019 , piece_id=SBCPU6XTMB4CRRTFKDBYFQQ2XZZNE3I6AF7VVSGYC356SPQNVHUQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:28.640156348 (118h10m18.577814857s) [upload=113540]: Next returned nil segment Aug 4 16:07:28.640223306 (118h10m18.577881825s) [upload=113540]: Waiting for error group managing segments... Aug 4 16:07:29.062127352 (118h10m18.999785841s) [noise=true , address=104.36.82.16:4321 , piece_id=42MZPFPEBQNQRJUHQJLZQP2UWGDV2OAXT6Z4KTWPPWLALISASN5Q, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.080565981 (118h10m19.01822448s) [noise=true , address=104.36.82.54:4181 , piece_id=AHW774W42KLBATPDVRQRKH4MQWV5YP2XEEUY6VASDMYJ2BKXCS2Q, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.081329852 (118h10m19.018988352s) [noise=true , address=104.36.82.62:4411 , piece_id=CIJJ4SMWKFDOIZZ5UEYIXFZHUNAZKRMM2P36IJIT5BHUX5STDZ4A, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.081846201 (118h10m19.0195047s) [noise=true , address=104.36.82.55:4224 , piece_id=NJYMQ2467DXHUEC6ELXITFJF6WWKM4XFEONRJVRF6QSJBDEMC7JA, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.091017397 (118h10m19.028675896s) [noise=true , address=104.36.82.46:4485 , piece_id=P4FMDR5NCQOLKSXQBLE7I7BPEZJ2URAPPVNKOR53ZPROBCE4YAJQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.105092769 (118h10m19.042751268s) [noise=true , address=104.36.82.13:4212 , piece_id=ER6VIEGFSG2F7542M5P66RS3QAUQYV7BWPQ4YOX55Q3XX6DDCTGA, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.107141759 (118h10m19.044800248s) [noise=true , address=104.36.82.23:4474 , piece_id=PLEKQKMEQKQDW72BNLLY4JOOPFI6CPNNGT74M5EFFNDQQFWELXUQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.116502399 (118h10m19.054160898s) [noise=true , address=45.91.116.7:4026 , piece_id=CB6TYC4A6DQOFFKVFPNMF5MH5MGMJHU5Z25B4QXQJI5ETDWLQJJQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.153680253 (118h10m19.091338752s) [noise=true , address=104.36.82.9:4019 , piece_id=SBCPU6XTMB4CRRTFKDBYFQQ2XZZNE3I6AF7VVSGYC356SPQNVHUQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.210191177 (118h10m19.147849686s) [noise=true , address=104.36.82.36:4625 , piece_id=FCK7QGPSKJ7VDK4D7BVNF44BRPQCQS5V5ICAJHELAQHDBSQV5R3Q, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.219317350 (118h10m19.156975849s) [noise=true , address=104.36.82.55:4206 , piece_id=KSAKYO6MALVA5REEKKJZ7MGX7YIRC2PDI5PNHFHRASPBD47V5SOQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:31.340384557 (118h10m21.278043046s) [noise=true , address=104.36.82.59:4326 , piece_id=CS7FVO3IRXKZFB2EZM2HQBVEVM4XTTFI5Y7TH3NJZGIYMREVY7UQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:32.464238230 (118h10m22.401896729s) [noise=true , address=104.36.82.13:4193 , piece_id=WJA72IWFD3TUA2WPGJGB6A7AHJXI4PGOK62YDMKSMDPUAVHUHW4Q, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:08:26.041033880 (118h11m15.978692379s) [noise=true , address=193.149.164.143:4695 , piece_id=3QTYKOTBCOJ4CRM76FR2XKONKQVYTQQMPTRH3E6MYJR4ODBPY3AQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: ecclient: upload failed (node:12TZfgz24gnZr4UryzEKw6gu94njnmCvafX5X64MKFSJqaUWSVh, address:193.149.164.143:4695): protocol: write tcp 188.166.225.170:43298->193.149.164.143:4695: use of closed network connection; write tcp 188.166.225.170:43298->193.149.164.143:4695: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43298->193.149.164.143:4695: use of closed network connection Aug 4 16:08:26.041079005 (118h11m15.978737504s) [noise=true , address=104.36.82.44:4886 , piece_id=Q3BIO4GNMY64IWXC7VZ5BAYKYRJJ3GKFV7Z4IOG7SP62IUJHXN2A, seg_pos={0 0}, upload=113540]: Done uploading piece. err: ecclient: upload failed (node:1n2cXhqd5EWbNPTXem83jQUjYq5ZaGKBidbVCQzxYfR3rWddGR, address:104.36.82.44:4886): protocol: write tcp 188.166.225.170:38030->104.36.82.44:4886: use of closed network connection; write tcp 188.166.225.170:38030->104.36.82.44:4886: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38030->104.36.82.44:4886: use of closed network connection Aug 4 16:08:26.041128580 (118h11m15.97878711s) [noise=true , address=104.36.82.39:4728 , piece_id=ZCL6YOUXY4P6HTYUTTILZR4TLG44CDTXWLOJJWOHPFSORA2IBCBA, seg_pos={0 0}, upload=113540]: Done uploading piece. err: ecclient: upload failed (node:1YoLzpb4mMYZ8TsdxnMocekZZ1gLfTrdCM8mxTDj4tY6khZGvi, address:104.36.82.39:4728): protocol: write tcp 188.166.225.170:34326->104.36.82.39:4728: use of closed network connection; write tcp 188.166.225.170:34326->104.36.82.39:4728: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34326->104.36.82.39:4728: use of closed network connection Aug 4 16:08:26.041195678 (118h11m15.978854187s) [noise=true , address=104.36.82.45:4930 , piece_id=LBMDUGYMQI7PBBGEOT22S7OGNVRIYS66USSYFAZOSGI445GWKCYA, seg_pos={0 0}, upload=113540]: Done uploading piece. err: ecclient: upload failed (node:12qaYFSpezuLAc5zJfAYruzad6nTpYFDpog7bmKMVTA8MVm76NB, address:104.36.82.45:4930): protocol: expected piece hash; context deadline exceeded; EOF Aug 4 16:08:26.041203374 (118h11m15.978861913s) [noise=true , address=104.36.82.42:4823 , piece_id=66Q6G7YBCVUEAHU3Y5LHPR5MIT7H7NQWI2TWWNHKZT5FYS4DNQBA, seg_pos={0 0}, upload=113540]: Done uploading piece. err: ecclient: upload failed (node:1fkA5DEj9RqsR6N85z7azvujy8tPkUhB5GCqBpUmFb4vvwbd8y, address:104.36.82.42:4823): protocol: write tcp 188.166.225.170:37230->104.36.82.42:4823: use of closed network connection; write tcp 188.166.225.170:37230->104.36.82.42:4823: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37230->104.36.82.42:4823: use of closed network connection Aug 4 16:08:26.041258960 (118h11m15.978917489s) [seg_pos={0 0}, upload=113540]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Aug 4 16:08:26.274088873 (118h11m16.211747372s) [upload=113540]: Done uploading segments... Aug 4 16:08:26.274113650 (118h11m16.211772139s) [upload=113540]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 40.792850387s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 2.59076217s FAIL (2025-08-04 16:19:31 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.590783322s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [...............................................XX.....X.XXX.] duration: 16.867414882s FAIL (2025-08-04 16:19:30 UTC) orbiter warning: Operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== 66.225.170:57862->104.36.82.13:4186: use of closed network connection Aug 4 16:18:29.634260260 (118h21m19.571918749s) [seg_pos={0 0}, upload=113704]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Aug 4 16:18:29.864378444 (118h21m19.802036943s) [upload=113704]: Done uploading segments... Aug 4 16:18:29.864404103 (118h21m19.802062602s) [upload=113704]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Aug 4 16:18:30.773685844 (118h21m20.711344333s) [upload=113714]: Starting upload Aug 4 16:18:30.773843035 (118h21m20.711501534s) [upload=113714]: Done starting upload Aug 4 16:18:30.773924561 (118h21m20.71158306s) [upload=113714]: Uploading segments... Aug 4 16:18:30.774094206 (118h21m20.711752695s) [upload=113714]: Got next segment. Inline: false Aug 4 16:18:30.774126056 (118h21m20.711784555s) [upload=113714]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 4 16:18:30.774130344 (118h21m20.711788843s) [upload=113714]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 4 16:18:31.011515803 (118h21m20.949174302s) [seg_pos={0 0}, upload=113714]: Begin upload segment... Aug 4 16:18:31.011581389 (118h21m20.949239898s) [seg_pos={0 0}, upload=113714]: Done begin upload segment. Aug 4 16:18:31.011604403 (118h21m20.949262912s) [noise=true , address=77.90.31.17:4770 , piece_id=TMEGOD5OLRWVC56VIWJTPP2WBFBBMEXWCKACT7K26T2HHARZNSQA, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.011720936 (118h21m20.949379435s) [noise=true , address=77.90.31.13:4628 , piece_id=PFEVUZQ4IX6Q7LHKD6VSEW6J2BA2GADLR53SRXIYIYL5XSCQCJNQ, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.011997064 (118h21m20.949655563s) [noise=true , address=77.90.31.21:4879 , piece_id=AXYOW2DPJQVDB4ZOURPUWC7Q3KPXXPVYV2V4E3SIFAQ27B7RY3XA, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.012225821 (118h21m20.94988432s) [noise=true , address=77.90.31.14:4663 , piece_id=3L2HY4QB7EUZ4E4JEYKUPRBQF6QRE2TCFN255LMXAR3YBF4AE65Q, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.012308239 (118h21m20.949966738s) [noise=true , address=77.90.31.19:4841 , piece_id=23RE2PCFCBQOLS7JFWHAX7U3THWIA34JYDQGRV2SQEBJV4RYDDUA, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.012418300 (118h21m20.950076799s) [noise=true , address=77.90.31.21:4877 , piece_id=YTBYW3NK4SXLO3CNQT7YND7XD2IKNN4OTWN6WYASBTJUIMPKUM2Q, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.012613513 (118h21m20.950272012s) [noise=true , address=45.91.116.5:4298 , piece_id=2WPFWY3R5RJC2QO5BXP6IULTUM366JEDEYHEBQDUGPEKFYNEDWIQ, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.012753160 (118h21m20.950411659s) [noise=true , address=45.91.116.10:4082 , piece_id=FRB3EVRTMRTR6VATM7GRKXPO2FXDOQCCPG3DASJRYWZBVMBKNTXQ, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.012811131 (118h21m20.95046963s) [noise=true , address=77.90.31.12:4589 , piece_id=AAMATJFUK77O6VQ4AF57WHRIGQQYUXRHEESXZFYFLPOHS2IGBFYQ, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.013004381 (118h21m20.95066287s) [noise=true , address=45.91.116.8:4045 , piece_id=YCPPIATKRNXMTWXGTCELS2EZPXRPC7RVZZT2YMZZHS6NUH2USLJA, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.013163866 (118h21m20.950822365s) [noise=true , address=45.91.116.14:4190 , piece_id=7QBTI3SCRTWBVGE77THDPVEKBE6F36UOXGXPRTJLWLKSHDAV6HEQ, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.013531159 (118h21m20.951189658s) [noise=true , address=77.90.31.15:4697 , piece_id=6PROJBDPAFJCBHAAQ45RQ2BDIDDUT2KD6UEQTTONFJ6INIK57YAA, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.013872873 (118h21m20.951531372s) [noise=true , address=104.36.82.10:4078 , piece_id=TQPRFOOXJMBYB5NMAXB3QZLQ5G5ABNARWYXEAAQEILVNM4FEYOBA, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.014172115 (118h21m20.951830614s) [noise=true , address=104.36.82.7:4911 , piece_id=BJRLBG4JVK33LDNCJTEAVSQV4U772YELSTGYN6WH4QLPH5C6AJUA, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.015248565 (118h21m20.952907064s) [noise=true , address=45.91.116.11:4118 , piece_id=3DC3BORHVWAGF2PAS2GLEX3NMFMJETBRQQWLNPWLDHFNCCNLXAWA, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.015496128 (118h21m20.953154627s) [noise=true , address=77.90.31.18:4806 , piece_id=3COKTLXWWA3ZSBCOYWYKT5GDUG6H5USPSHBFYVIOL4SKHIBI4CCA, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:32.741770833 (118h21m22.679429352s) [upload=113714]: Next returned nil segment Aug 4 16:18:32.741792233 (118h21m22.679450763s) [upload=113714]: Waiting for error group managing segments... Aug 4 16:18:33.031304408 (118h21m22.968962907s) [noise=true , address=45.91.116.8:4045 , piece_id=YCPPIATKRNXMTWXGTCELS2EZPXRPC7RVZZT2YMZZHS6NUH2USLJA, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:33.033040921 (118h21m22.97069942s) [noise=true , address=45.91.116.5:4298 , piece_id=2WPFWY3R5RJC2QO5BXP6IULTUM366JEDEYHEBQDUGPEKFYNEDWIQ, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:33.034686720 (118h21m22.972345209s) [noise=true , address=77.90.31.12:4589 , piece_id=AAMATJFUK77O6VQ4AF57WHRIGQQYUXRHEESXZFYFLPOHS2IGBFYQ, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:33.036548412 (118h21m22.974206911s) [noise=true , address=77.90.31.21:4877 , piece_id=YTBYW3NK4SXLO3CNQT7YND7XD2IKNN4OTWN6WYASBTJUIMPKUM2Q, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:33.059147582 (118h21m22.996806071s) [noise=true , address=77.90.31.15:4697 , piece_id=6PROJBDPAFJCBHAAQ45RQ2BDIDDUT2KD6UEQTTONFJ6INIK57YAA, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:33.171247436 (118h21m23.108905935s) [noise=true , address=77.90.31.13:4628 , piece_id=PFEVUZQ4IX6Q7LHKD6VSEW6J2BA2GADLR53SRXIYIYL5XSCQCJNQ, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:33.219388326 (118h21m23.157046825s) [noise=true , address=104.36.82.10:4078 , piece_id=TQPRFOOXJMBYB5NMAXB3QZLQ5G5ABNARWYXEAAQEILVNM4FEYOBA, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:33.997614596 (118h21m23.935273095s) [noise=true , address=77.90.31.21:4879 , piece_id=AXYOW2DPJQVDB4ZOURPUWC7Q3KPXXPVYV2V4E3SIFAQ27B7RY3XA, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:34.018191777 (118h21m23.955850276s) [noise=true , address=104.36.82.7:4911 , piece_id=BJRLBG4JVK33LDNCJTEAVSQV4U772YELSTGYN6WH4QLPH5C6AJUA, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:44.685373447 (118h21m34.623031946s) [noise=true , address=77.90.31.14:4663 , piece_id=3L2HY4QB7EUZ4E4JEYKUPRBQF6QRE2TCFN255LMXAR3YBF4AE65Q, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:45.924412567 (118h21m35.862071066s) [noise=true , address=77.90.31.18:4806 , piece_id=3COKTLXWWA3ZSBCOYWYKT5GDUG6H5USPSHBFYVIOL4SKHIBI4CCA, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:53.143254472 (118h21m43.080913011s) [noise=true , address=77.90.31.17:4770 , piece_id=TMEGOD5OLRWVC56VIWJTPP2WBFBBMEXWCKACT7K26T2HHARZNSQA, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:57.690047932 (118h21m47.627706431s) [noise=true , address=77.90.31.19:4841 , piece_id=23RE2PCFCBQOLS7JFWHAX7U3THWIA34JYDQGRV2SQEBJV4RYDDUA, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:57.690075375 (118h21m47.627733874s) [seg_pos={0 0}, upload=113714]: Segment reached optimal threshold of 13 pieces. Aug 4 16:18:57.690245260 (118h21m47.627903749s) [noise=true , address=45.91.116.14:4190 , piece_id=7QBTI3SCRTWBVGE77THDPVEKBE6F36UOXGXPRTJLWLKSHDAV6HEQ, seg_pos={0 0}, upload=113714]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: expected piece hash; context canceled; EOF Aug 4 16:18:57.690312398 (118h21m47.627970897s) [noise=true , address=45.91.116.11:4118 , piece_id=3DC3BORHVWAGF2PAS2GLEX3NMFMJETBRQQWLNPWLDHFNCCNLXAWA, seg_pos={0 0}, upload=113714]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p): protocol: write tcp 188.166.225.170:37362->45.91.116.11:4118: use of closed network connection; write tcp 188.166.225.170:37362->45.91.116.11:4118: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37362->45.91.116.11:4118: use of closed network connection Aug 4 16:18:57.690359428 (118h21m47.628017927s) [noise=true , address=45.91.116.10:4082 , piece_id=FRB3EVRTMRTR6VATM7GRKXPO2FXDOQCCPG3DASJRYWZBVMBKNTXQ, seg_pos={0 0}, upload=113714]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: write tcp 188.166.225.170:46480->45.91.116.10:4082: use of closed network connection; write tcp 188.166.225.170:46480->45.91.116.10:4082: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46480->45.91.116.10:4082: use of closed network connection Aug 4 16:18:57.690376110 (118h21m47.628034609s) [seg_pos={0 0}, upload=113714]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 4 16:18:57.690439481 (118h21m47.62809798s) [upload=113714]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 4 16:18:57.690442958 (118h21m47.628101457s) [upload=113714]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 4 16:18:58.023198180 (118h21m47.960856679s) [upload=113714]: Done uploading segments... Aug 4 16:18:58.023217707 (118h21m47.960876196s) [upload=113714]: Upload finished. err: <nil> Duration: 16.867449528s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 5.48565384s FAIL (2025-08-04 16:19:28 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.485676134s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [...............................................XX.X.XX.XX..X] duration: 30.001246136s FAIL (2025-08-04 16:19:20 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.001265493s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 5.541752979s FAIL (2025-08-04 16:19:05 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.541777255s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [................................................XX.....X.XXX] duration: 30.002156933s FAIL (2025-08-04 16:19:00 UTC) orbiter warning: Operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== 66.225.170:57862->104.36.82.13:4186: use of closed network connection Aug 4 16:18:29.634260260 (118h21m19.571918749s) [seg_pos={0 0}, upload=113704]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Aug 4 16:18:29.864378444 (118h21m19.802036943s) [upload=113704]: Done uploading segments... Aug 4 16:18:29.864404103 (118h21m19.802062602s) [upload=113704]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Aug 4 16:18:30.773685844 (118h21m20.711344333s) [upload=113714]: Starting upload Aug 4 16:18:30.773843035 (118h21m20.711501534s) [upload=113714]: Done starting upload Aug 4 16:18:30.773924561 (118h21m20.71158306s) [upload=113714]: Uploading segments... Aug 4 16:18:30.774094206 (118h21m20.711752695s) [upload=113714]: Got next segment. Inline: false Aug 4 16:18:30.774126056 (118h21m20.711784555s) [upload=113714]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 4 16:18:30.774130344 (118h21m20.711788843s) [upload=113714]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 4 16:18:31.011515803 (118h21m20.949174302s) [seg_pos={0 0}, upload=113714]: Begin upload segment... Aug 4 16:18:31.011581389 (118h21m20.949239898s) [seg_pos={0 0}, upload=113714]: Done begin upload segment. Aug 4 16:18:31.011604403 (118h21m20.949262912s) [noise=true , address=77.90.31.17:4770 , piece_id=TMEGOD5OLRWVC56VIWJTPP2WBFBBMEXWCKACT7K26T2HHARZNSQA, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.011720936 (118h21m20.949379435s) [noise=true , address=77.90.31.13:4628 , piece_id=PFEVUZQ4IX6Q7LHKD6VSEW6J2BA2GADLR53SRXIYIYL5XSCQCJNQ, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.011997064 (118h21m20.949655563s) [noise=true , address=77.90.31.21:4879 , piece_id=AXYOW2DPJQVDB4ZOURPUWC7Q3KPXXPVYV2V4E3SIFAQ27B7RY3XA, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.012225821 (118h21m20.94988432s) [noise=true , address=77.90.31.14:4663 , piece_id=3L2HY4QB7EUZ4E4JEYKUPRBQF6QRE2TCFN255LMXAR3YBF4AE65Q, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.012308239 (118h21m20.949966738s) [noise=true , address=77.90.31.19:4841 , piece_id=23RE2PCFCBQOLS7JFWHAX7U3THWIA34JYDQGRV2SQEBJV4RYDDUA, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.012418300 (118h21m20.950076799s) [noise=true , address=77.90.31.21:4877 , piece_id=YTBYW3NK4SXLO3CNQT7YND7XD2IKNN4OTWN6WYASBTJUIMPKUM2Q, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.012613513 (118h21m20.950272012s) [noise=true , address=45.91.116.5:4298 , piece_id=2WPFWY3R5RJC2QO5BXP6IULTUM366JEDEYHEBQDUGPEKFYNEDWIQ, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.012753160 (118h21m20.950411659s) [noise=true , address=45.91.116.10:4082 , piece_id=FRB3EVRTMRTR6VATM7GRKXPO2FXDOQCCPG3DASJRYWZBVMBKNTXQ, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.012811131 (118h21m20.95046963s) [noise=true , address=77.90.31.12:4589 , piece_id=AAMATJFUK77O6VQ4AF57WHRIGQQYUXRHEESXZFYFLPOHS2IGBFYQ, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.013004381 (118h21m20.95066287s) [noise=true , address=45.91.116.8:4045 , piece_id=YCPPIATKRNXMTWXGTCELS2EZPXRPC7RVZZT2YMZZHS6NUH2USLJA, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.013163866 (118h21m20.950822365s) [noise=true , address=45.91.116.14:4190 , piece_id=7QBTI3SCRTWBVGE77THDPVEKBE6F36UOXGXPRTJLWLKSHDAV6HEQ, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.013531159 (118h21m20.951189658s) [noise=true , address=77.90.31.15:4697 , piece_id=6PROJBDPAFJCBHAAQ45RQ2BDIDDUT2KD6UEQTTONFJ6INIK57YAA, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.013872873 (118h21m20.951531372s) [noise=true , address=104.36.82.10:4078 , piece_id=TQPRFOOXJMBYB5NMAXB3QZLQ5G5ABNARWYXEAAQEILVNM4FEYOBA, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.014172115 (118h21m20.951830614s) [noise=true , address=104.36.82.7:4911 , piece_id=BJRLBG4JVK33LDNCJTEAVSQV4U772YELSTGYN6WH4QLPH5C6AJUA, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.015248565 (118h21m20.952907064s) [noise=true , address=45.91.116.11:4118 , piece_id=3DC3BORHVWAGF2PAS2GLEX3NMFMJETBRQQWLNPWLDHFNCCNLXAWA, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:31.015496128 (118h21m20.953154627s) [noise=true , address=77.90.31.18:4806 , piece_id=3COKTLXWWA3ZSBCOYWYKT5GDUG6H5USPSHBFYVIOL4SKHIBI4CCA, seg_pos={0 0}, upload=113714]: Uploading piece... Aug 4 16:18:32.741770833 (118h21m22.679429352s) [upload=113714]: Next returned nil segment Aug 4 16:18:32.741792233 (118h21m22.679450763s) [upload=113714]: Waiting for error group managing segments... Aug 4 16:18:33.031304408 (118h21m22.968962907s) [noise=true , address=45.91.116.8:4045 , piece_id=YCPPIATKRNXMTWXGTCELS2EZPXRPC7RVZZT2YMZZHS6NUH2USLJA, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:33.033040921 (118h21m22.97069942s) [noise=true , address=45.91.116.5:4298 , piece_id=2WPFWY3R5RJC2QO5BXP6IULTUM366JEDEYHEBQDUGPEKFYNEDWIQ, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:33.034686720 (118h21m22.972345209s) [noise=true , address=77.90.31.12:4589 , piece_id=AAMATJFUK77O6VQ4AF57WHRIGQQYUXRHEESXZFYFLPOHS2IGBFYQ, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:33.036548412 (118h21m22.974206911s) [noise=true , address=77.90.31.21:4877 , piece_id=YTBYW3NK4SXLO3CNQT7YND7XD2IKNN4OTWN6WYASBTJUIMPKUM2Q, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:33.059147582 (118h21m22.996806071s) [noise=true , address=77.90.31.15:4697 , piece_id=6PROJBDPAFJCBHAAQ45RQ2BDIDDUT2KD6UEQTTONFJ6INIK57YAA, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:33.171247436 (118h21m23.108905935s) [noise=true , address=77.90.31.13:4628 , piece_id=PFEVUZQ4IX6Q7LHKD6VSEW6J2BA2GADLR53SRXIYIYL5XSCQCJNQ, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:33.219388326 (118h21m23.157046825s) [noise=true , address=104.36.82.10:4078 , piece_id=TQPRFOOXJMBYB5NMAXB3QZLQ5G5ABNARWYXEAAQEILVNM4FEYOBA, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:33.997614596 (118h21m23.935273095s) [noise=true , address=77.90.31.21:4879 , piece_id=AXYOW2DPJQVDB4ZOURPUWC7Q3KPXXPVYV2V4E3SIFAQ27B7RY3XA, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:34.018191777 (118h21m23.955850276s) [noise=true , address=104.36.82.7:4911 , piece_id=BJRLBG4JVK33LDNCJTEAVSQV4U772YELSTGYN6WH4QLPH5C6AJUA, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:44.685373447 (118h21m34.623031946s) [noise=true , address=77.90.31.14:4663 , piece_id=3L2HY4QB7EUZ4E4JEYKUPRBQF6QRE2TCFN255LMXAR3YBF4AE65Q, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:45.924412567 (118h21m35.862071066s) [noise=true , address=77.90.31.18:4806 , piece_id=3COKTLXWWA3ZSBCOYWYKT5GDUG6H5USPSHBFYVIOL4SKHIBI4CCA, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:53.143254472 (118h21m43.080913011s) [noise=true , address=77.90.31.17:4770 , piece_id=TMEGOD5OLRWVC56VIWJTPP2WBFBBMEXWCKACT7K26T2HHARZNSQA, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:57.690047932 (118h21m47.627706431s) [noise=true , address=77.90.31.19:4841 , piece_id=23RE2PCFCBQOLS7JFWHAX7U3THWIA34JYDQGRV2SQEBJV4RYDDUA, seg_pos={0 0}, upload=113714]: Done uploading piece. err: <nil> Aug 4 16:18:57.690075375 (118h21m47.627733874s) [seg_pos={0 0}, upload=113714]: Segment reached optimal threshold of 13 pieces. Aug 4 16:18:57.690245260 (118h21m47.627903749s) [noise=true , address=45.91.116.14:4190 , piece_id=7QBTI3SCRTWBVGE77THDPVEKBE6F36UOXGXPRTJLWLKSHDAV6HEQ, seg_pos={0 0}, upload=113714]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: expected piece hash; context canceled; EOF Aug 4 16:18:57.690312398 (118h21m47.627970897s) [noise=true , address=45.91.116.11:4118 , piece_id=3DC3BORHVWAGF2PAS2GLEX3NMFMJETBRQQWLNPWLDHFNCCNLXAWA, seg_pos={0 0}, upload=113714]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p): protocol: write tcp 188.166.225.170:37362->45.91.116.11:4118: use of closed network connection; write tcp 188.166.225.170:37362->45.91.116.11:4118: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37362->45.91.116.11:4118: use of closed network connection Aug 4 16:18:57.690359428 (118h21m47.628017927s) [noise=true , address=45.91.116.10:4082 , piece_id=FRB3EVRTMRTR6VATM7GRKXPO2FXDOQCCPG3DASJRYWZBVMBKNTXQ, seg_pos={0 0}, upload=113714]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: write tcp 188.166.225.170:46480->45.91.116.10:4082: use of closed network connection; write tcp 188.166.225.170:46480->45.91.116.10:4082: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46480->45.91.116.10:4082: use of closed network connection Aug 4 16:18:57.690376110 (118h21m47.628034609s) [seg_pos={0 0}, upload=113714]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 4 16:18:57.690439481 (118h21m47.62809798s) [upload=113714]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 4 16:18:57.690442958 (118h21m47.628101457s) [upload=113714]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 4 16:18:58.023198180 (118h21m47.960856679s) [upload=113714]: Done uploading segments... Aug 4 16:18:58.023217707 (118h21m47.960876196s) [upload=113714]: Upload finished. err: <nil> Duration: 30.002167373s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 30.000709431s FAIL (2025-08-04 16:18:54 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.000731061s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [X.............................X.............................] duration: 1.894665404s FAIL (2025-08-04 16:18:48 UTC) orbiter warning: Operation: DownloadObject uplink: object not found ("orbiter_us_0/1K") Duration: 1.894690001s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 5.858276392s FAIL (2025-08-04 16:18:47 UTC) orbiter warning: Operation: Commit uplink: failed to upload enough pieces (needed at least 14 but got 12) Duration: 5.858305738s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.................................................XX.....X.XX] duration: 30.231705952s FAIL (2025-08-04 16:18:29 UTC) orbiter warning: Operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== uploading segments... Aug 4 16:17:35.581776351 (118h20m25.51943485s) [upload=113692]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Aug 4 16:18:00.287522009 (118h20m50.225180539s) [upload=113704]: Starting upload Aug 4 16:18:00.287582745 (118h20m50.225241245s) [upload=113704]: Done starting upload Aug 4 16:18:00.287653541 (118h20m50.22531204s) [upload=113704]: Uploading segments... Aug 4 16:18:00.287780584 (118h20m50.225439083s) [upload=113704]: Got next segment. Inline: false Aug 4 16:18:00.287817695 (118h20m50.225476185s) [upload=113704]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 4 16:18:00.287824618 (118h20m50.225483118s) [upload=113704]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 4 16:18:00.553255517 (118h20m50.490914016s) [seg_pos={0 0}, upload=113704]: Begin upload segment... Aug 4 16:18:00.553310652 (118h20m50.490969152s) [seg_pos={0 0}, upload=113704]: Done begin upload segment. Aug 4 16:18:00.553322185 (118h20m50.490980684s) [noise=true , address=45.91.116.13:4153 , piece_id=Y757SCLBLQGTFHIMF5MP36SE2B4A6ZK6SEQBFJCEHFNOUQVHFZJA, seg_pos={0 0}, upload=113704]: Uploading piece... Aug 4 16:18:00.553331553 (118h20m50.490990062s) [noise=true , address=45.91.116.12:4226 , piece_id=IQDNYG7YMFR2OUCN3ERAT624ZSLFESWCXREHEUZ4INUJMK6BWMCQ, seg_pos={0 0}, upload=113704]: Uploading piece... Aug 4 16:18:00.553858200 (118h20m50.491516709s) [noise=true , address=45.91.116.13:4154 , piece_id=SHFSVYDFUDZG4LDFURLZ5KXV2EZDWET5LVA6FUKD3WHUAF7BT42Q, seg_pos={0 0}, upload=113704]: Uploading piece... Aug 4 16:18:00.554100063 (118h20m50.491758562s) [noise=true , address=45.91.116.5:4297 , piece_id=ZVNFJRERYO36HYPU4IFSMME5GK4ODF52RIRHCQS4YQEATLZ2XMAQ, seg_pos={0 0}, upload=113704]: Uploading piece... Aug 4 16:18:00.554240161 (118h20m50.49189866s) [noise=true , address=77.90.31.16:4734 , piece_id=HMBKVVPJW2Q3R24ZJ5KU2NYQSHAV4JVQC2TVPKMUP74KDV3Z73EQ, seg_pos={0 0}, upload=113704]: Uploading piece... Aug 4 16:18:00.554327869 (118h20m50.491986368s) [noise=true , address=45.91.116.8:4045 , piece_id=B7OKO3VDZC64BDUMFU2VRMYSG6K3UP7WIWTJQP7676KUJW36QQIA, seg_pos={0 0}, upload=113704]: Uploading piece... Aug 4 16:18:00.554528333 (118h20m50.492186832s) [noise=true , address=77.90.31.13:4627 , piece_id=FIKWGTVIF5HLVDYACGHY6T2YGRW3GOIV5SQ4O4D75WNVLCGEDITA, seg_pos={0 0}, upload=113704]: Uploading piece... Aug 4 16:18:00.554556527 (118h20m50.492215026s) [noise=true , address=104.36.82.13:4186 , piece_id=UQULRLZYWGJZNJXMNBPC2GS3J6JNDKR2Q3MABFGLHX45RIYF2BQA, seg_pos={0 0}, upload=113704]: Uploading piece... Aug 4 16:18:00.554719178 (118h20m50.492377677s) [noise=true , address=45.91.116.9:4261 , piece_id=GELDKRVI2KCMDVRDZOA4QDULN5ZMXEPHFP6BPZGCFQZB3LMCCC7Q, seg_pos={0 0}, upload=113704]: Uploading piece... Aug 4 16:18:00.554835621 (118h20m50.49249412s) [noise=true , address=77.90.31.19:4842 , piece_id=SFSU266OG3DJ5ZYCFZMWRNJNYK4NEMSDR3YI5S562W2YK4XGQMNA, seg_pos={0 0}, upload=113704]: Uploading piece... Aug 4 16:18:00.554922367 (118h20m50.492580866s) [noise=true , address=45.91.116.8:4048 , piece_id=XCGJ43LRJWYZBN3P5E54A3HQHGGIGHZLODWJSGCC43RSHPGRYJ7Q, seg_pos={0 0}, upload=113704]: Uploading piece... Aug 4 16:18:00.555125565 (118h20m50.492784064s) [noise=true , address=77.90.31.11:4555 , piece_id=MTU24WEIUJ6PJJD4SIRHTIGA2R6KIZET7MAYA4Z34JCVZOZMGFQQ, seg_pos={0 0}, upload=113704]: Uploading piece... Aug 4 16:18:00.555358992 (118h20m50.493017481s) [noise=true , address=45.91.116.4:4940 , piece_id=7QCLBWSOKFY6IF336YF44EWRRFODEGRRFFKXFK7AHEUWDJJUSKZA, seg_pos={0 0}, upload=113704]: Uploading piece... Aug 4 16:18:00.555612577 (118h20m50.493271076s) [noise=true , address=77.90.31.21:4877 , piece_id=7CNN2MGWGFLMCVL5GZPUA5MROLFBGGHJ4EB2EMSL56PTFSEK2SBA, seg_pos={0 0}, upload=113704]: Uploading piece... Aug 4 16:18:00.555847307 (118h20m50.493505806s) [noise=true , address=45.91.116.14:4189 , piece_id=SI3SIHWQBUB52MFD47OWQWDHX6FQYZ76FKBB3DVPEIF6DFYO2HJQ, seg_pos={0 0}, upload=113704]: Uploading piece... Aug 4 16:18:00.555860311 (118h20m50.49351881s) [noise=true , address=104.36.82.9:4005 , piece_id=TFFYLNUGRUMBDN6MWTZ2DSTJGIQVLO46N4FU3NYLBW2DVMIQUYYA, seg_pos={0 0}, upload=113704]: Uploading piece... Aug 4 16:18:02.276289541 (118h20m52.21394804s) [upload=113704]: Next returned nil segment Aug 4 16:18:02.276307175 (118h20m52.213965674s) [upload=113704]: Waiting for error group managing segments... Aug 4 16:18:02.552989501 (118h20m52.490648s) [noise=true , address=45.91.116.13:4153 , piece_id=Y757SCLBLQGTFHIMF5MP36SE2B4A6ZK6SEQBFJCEHFNOUQVHFZJA, seg_pos={0 0}, upload=113704]: Done uploading piece. err: <nil> Aug 4 16:18:02.578761602 (118h20m52.516420101s) [noise=true , address=45.91.116.5:4297 , piece_id=ZVNFJRERYO36HYPU4IFSMME5GK4ODF52RIRHCQS4YQEATLZ2XMAQ, seg_pos={0 0}, upload=113704]: Done uploading piece. err: <nil> Aug 4 16:18:02.579112032 (118h20m52.516770531s) [noise=true , address=77.90.31.11:4555 , piece_id=MTU24WEIUJ6PJJD4SIRHTIGA2R6KIZET7MAYA4Z34JCVZOZMGFQQ, seg_pos={0 0}, upload=113704]: Done uploading piece. err: <nil> Aug 4 16:18:02.583355341 (118h20m52.52101384s) [noise=true , address=45.91.116.13:4154 , piece_id=SHFSVYDFUDZG4LDFURLZ5KXV2EZDWET5LVA6FUKD3WHUAF7BT42Q, seg_pos={0 0}, upload=113704]: Done uploading piece. err: <nil> Aug 4 16:18:02.583922516 (118h20m52.521581025s) [noise=true , address=77.90.31.16:4734 , piece_id=HMBKVVPJW2Q3R24ZJ5KU2NYQSHAV4JVQC2TVPKMUP74KDV3Z73EQ, seg_pos={0 0}, upload=113704]: Done uploading piece. err: <nil> Aug 4 16:18:02.583960338 (118h20m52.521618837s) [noise=true , address=45.91.116.9:4261 , piece_id=GELDKRVI2KCMDVRDZOA4QDULN5ZMXEPHFP6BPZGCFQZB3LMCCC7Q, seg_pos={0 0}, upload=113704]: Done uploading piece. err: <nil> Aug 4 16:18:02.586128106 (118h20m52.523786605s) [noise=true , address=45.91.116.8:4048 , piece_id=XCGJ43LRJWYZBN3P5E54A3HQHGGIGHZLODWJSGCC43RSHPGRYJ7Q, seg_pos={0 0}, upload=113704]: Done uploading piece. err: <nil> Aug 4 16:18:02.588062358 (118h20m52.525720857s) [noise=true , address=45.91.116.12:4226 , piece_id=IQDNYG7YMFR2OUCN3ERAT624ZSLFESWCXREHEUZ4INUJMK6BWMCQ, seg_pos={0 0}, upload=113704]: Done uploading piece. err: <nil> Aug 4 16:18:02.678916862 (118h20m52.616575361s) [noise=true , address=104.36.82.9:4005 , piece_id=TFFYLNUGRUMBDN6MWTZ2DSTJGIQVLO46N4FU3NYLBW2DVMIQUYYA, seg_pos={0 0}, upload=113704]: Done uploading piece. err: <nil> Aug 4 16:18:02.751163245 (118h20m52.688821744s) [noise=true , address=77.90.31.13:4627 , piece_id=FIKWGTVIF5HLVDYACGHY6T2YGRW3GOIV5SQ4O4D75WNVLCGEDITA, seg_pos={0 0}, upload=113704]: Done uploading piece. err: <nil> Aug 4 16:18:03.402649426 (118h20m53.340307926s) [noise=true , address=45.91.116.14:4189 , piece_id=SI3SIHWQBUB52MFD47OWQWDHX6FQYZ76FKBB3DVPEIF6DFYO2HJQ, seg_pos={0 0}, upload=113704]: Done uploading piece. err: <nil> Aug 4 16:18:03.639973047 (118h20m53.577631546s) [noise=true , address=77.90.31.21:4877 , piece_id=7CNN2MGWGFLMCVL5GZPUA5MROLFBGGHJ4EB2EMSL56PTFSEK2SBA, seg_pos={0 0}, upload=113704]: Done uploading piece. err: <nil> Aug 4 16:18:29.634047583 (118h21m19.571706072s) [noise=true , address=77.90.31.19:4842 , piece_id=SFSU266OG3DJ5ZYCFZMWRNJNYK4NEMSDR3YI5S562W2YK4XGQMNA, seg_pos={0 0}, upload=113704]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: write tcp 188.166.225.170:55224->77.90.31.19:4842: use of closed network connection; write tcp 188.166.225.170:55224->77.90.31.19:4842: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55224->77.90.31.19:4842: use of closed network connection Aug 4 16:18:29.634142284 (118h21m19.571800783s) [noise=true , address=45.91.116.4:4940 , piece_id=7QCLBWSOKFY6IF336YF44EWRRFODEGRRFFKXFK7AHEUWDJJUSKZA, seg_pos={0 0}, upload=113704]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: write tcp 188.166.225.170:34808->45.91.116.4:4940: use of closed network connection; write tcp 188.166.225.170:34808->45.91.116.4:4940: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34808->45.91.116.4:4940: use of closed network connection Aug 4 16:18:29.634194554 (118h21m19.571853053s) [noise=true , address=45.91.116.8:4045 , piece_id=B7OKO3VDZC64BDUMFU2VRMYSG6K3UP7WIWTJQP7676KUJW36QQIA, seg_pos={0 0}, upload=113704]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: write tcp 188.166.225.170:42566->45.91.116.8:4045: use of closed network connection; write tcp 188.166.225.170:42566->45.91.116.8:4045: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42566->45.91.116.8:4045: use of closed network connection Aug 4 16:18:29.634211257 (118h21m19.571869746s) [noise=true , address=104.36.82.13:4186 , piece_id=UQULRLZYWGJZNJXMNBPC2GS3J6JNDKR2Q3MABFGLHX45RIYF2BQA, seg_pos={0 0}, upload=113704]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:104.36.82.13:4186): protocol: write tcp 188.166.225.170:57862->104.36.82.13:4186: use of closed network connection; write tcp 188.166.225.170:57862->104.36.82.13:4186: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57862->104.36.82.13:4186: use of closed network connection Aug 4 16:18:29.634260260 (118h21m19.571918749s) [seg_pos={0 0}, upload=113704]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Aug 4 16:18:29.864378444 (118h21m19.802036943s) [upload=113704]: Done uploading segments... Aug 4 16:18:29.864404103 (118h21m19.802062602s) [upload=113704]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.23173133s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 6.263765135s FAIL (2025-08-04 16:18:28 UTC) orbiter warning: Operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 4.36.82.36:4625 , piece_id=FCK7QGPSKJ7VDK4D7BVNF44BRPQCQS5V5ICAJHELAQHDBSQV5R3Q, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.146274823 (118h10m17.083933322s) [noise=true , address=45.91.116.7:4026 , piece_id=CB6TYC4A6DQOFFKVFPNMF5MH5MGMJHU5Z25B4QXQJI5ETDWLQJJQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.146672965 (118h10m17.084331464s) [noise=true , address=104.36.82.16:4321 , piece_id=42MZPFPEBQNQRJUHQJLZQP2UWGDV2OAXT6Z4KTWPPWLALISASN5Q, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.146726166 (118h10m17.084384665s) [noise=true , address=104.36.82.55:4206 , piece_id=KSAKYO6MALVA5REEKKJZ7MGX7YIRC2PDI5PNHFHRASPBD47V5SOQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.147136371 (118h10m17.08479488s) [noise=true , address=104.36.82.42:4823 , piece_id=66Q6G7YBCVUEAHU3Y5LHPR5MIT7H7NQWI2TWWNHKZT5FYS4DNQBA, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.147479628 (118h10m17.085138127s) [noise=true , address=193.149.164.143:4695 , piece_id=3QTYKOTBCOJ4CRM76FR2XKONKQVYTQQMPTRH3E6MYJR4ODBPY3AQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.147821592 (118h10m17.085480091s) [noise=true , address=104.36.82.62:4411 , piece_id=CIJJ4SMWKFDOIZZ5UEYIXFZHUNAZKRMM2P36IJIT5BHUX5STDZ4A, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.148134220 (118h10m17.085792719s) [noise=true , address=104.36.82.55:4224 , piece_id=NJYMQ2467DXHUEC6ELXITFJF6WWKM4XFEONRJVRF6QSJBDEMC7JA, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.148548573 (118h10m17.086207072s) [noise=true , address=104.36.82.13:4193 , piece_id=WJA72IWFD3TUA2WPGJGB6A7AHJXI4PGOK62YDMKSMDPUAVHUHW4Q, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.148889576 (118h10m17.086548085s) [noise=true , address=104.36.82.39:4728 , piece_id=ZCL6YOUXY4P6HTYUTTILZR4TLG44CDTXWLOJJWOHPFSORA2IBCBA, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.149205931 (118h10m17.08686443s) [noise=true , address=104.36.82.59:4326 , piece_id=CS7FVO3IRXKZFB2EZM2HQBVEVM4XTTFI5Y7TH3NJZGIYMREVY7UQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.149509752 (118h10m17.087168261s) [noise=true , address=104.36.82.45:4930 , piece_id=LBMDUGYMQI7PBBGEOT22S7OGNVRIYS66USSYFAZOSGI445GWKCYA, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.149866545 (118h10m17.087525044s) [noise=true , address=104.36.82.23:4474 , piece_id=PLEKQKMEQKQDW72BNLLY4JOOPFI6CPNNGT74M5EFFNDQQFWELXUQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.150193601 (118h10m17.08785211s) [noise=true , address=104.36.82.13:4212 , piece_id=ER6VIEGFSG2F7542M5P66RS3QAUQYV7BWPQ4YOX55Q3XX6DDCTGA, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.150515557 (118h10m17.088174066s) [noise=true , address=104.36.82.46:4485 , piece_id=P4FMDR5NCQOLKSXQBLE7I7BPEZJ2URAPPVNKOR53ZPROBCE4YAJQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.150935149 (118h10m17.088593648s) [noise=true , address=104.36.82.54:4181 , piece_id=AHW774W42KLBATPDVRQRKH4MQWV5YP2XEEUY6VASDMYJ2BKXCS2Q, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.151249151 (118h10m17.08890765s) [noise=true , address=104.36.82.44:4886 , piece_id=Q3BIO4GNMY64IWXC7VZ5BAYKYRJJ3GKFV7Z4IOG7SP62IUJHXN2A, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.151276363 (118h10m17.088934862s) [noise=true , address=104.36.82.9:4019 , piece_id=SBCPU6XTMB4CRRTFKDBYFQQ2XZZNE3I6AF7VVSGYC356SPQNVHUQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:28.640156348 (118h10m18.577814857s) [upload=113540]: Next returned nil segment Aug 4 16:07:28.640223306 (118h10m18.577881825s) [upload=113540]: Waiting for error group managing segments... Aug 4 16:07:29.062127352 (118h10m18.999785841s) [noise=true , address=104.36.82.16:4321 , piece_id=42MZPFPEBQNQRJUHQJLZQP2UWGDV2OAXT6Z4KTWPPWLALISASN5Q, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.080565981 (118h10m19.01822448s) [noise=true , address=104.36.82.54:4181 , piece_id=AHW774W42KLBATPDVRQRKH4MQWV5YP2XEEUY6VASDMYJ2BKXCS2Q, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.081329852 (118h10m19.018988352s) [noise=true , address=104.36.82.62:4411 , piece_id=CIJJ4SMWKFDOIZZ5UEYIXFZHUNAZKRMM2P36IJIT5BHUX5STDZ4A, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.081846201 (118h10m19.0195047s) [noise=true , address=104.36.82.55:4224 , piece_id=NJYMQ2467DXHUEC6ELXITFJF6WWKM4XFEONRJVRF6QSJBDEMC7JA, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.091017397 (118h10m19.028675896s) [noise=true , address=104.36.82.46:4485 , piece_id=P4FMDR5NCQOLKSXQBLE7I7BPEZJ2URAPPVNKOR53ZPROBCE4YAJQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.105092769 (118h10m19.042751268s) [noise=true , address=104.36.82.13:4212 , piece_id=ER6VIEGFSG2F7542M5P66RS3QAUQYV7BWPQ4YOX55Q3XX6DDCTGA, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.107141759 (118h10m19.044800248s) [noise=true , address=104.36.82.23:4474 , piece_id=PLEKQKMEQKQDW72BNLLY4JOOPFI6CPNNGT74M5EFFNDQQFWELXUQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.116502399 (118h10m19.054160898s) [noise=true , address=45.91.116.7:4026 , piece_id=CB6TYC4A6DQOFFKVFPNMF5MH5MGMJHU5Z25B4QXQJI5ETDWLQJJQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.153680253 (118h10m19.091338752s) [noise=true , address=104.36.82.9:4019 , piece_id=SBCPU6XTMB4CRRTFKDBYFQQ2XZZNE3I6AF7VVSGYC356SPQNVHUQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.210191177 (118h10m19.147849686s) [noise=true , address=104.36.82.36:4625 , piece_id=FCK7QGPSKJ7VDK4D7BVNF44BRPQCQS5V5ICAJHELAQHDBSQV5R3Q, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.219317350 (118h10m19.156975849s) [noise=true , address=104.36.82.55:4206 , piece_id=KSAKYO6MALVA5REEKKJZ7MGX7YIRC2PDI5PNHFHRASPBD47V5SOQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:31.340384557 (118h10m21.278043046s) [noise=true , address=104.36.82.59:4326 , piece_id=CS7FVO3IRXKZFB2EZM2HQBVEVM4XTTFI5Y7TH3NJZGIYMREVY7UQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:32.464238230 (118h10m22.401896729s) [noise=true , address=104.36.82.13:4193 , piece_id=WJA72IWFD3TUA2WPGJGB6A7AHJXI4PGOK62YDMKSMDPUAVHUHW4Q, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:08:26.041033880 (118h11m15.978692379s) [noise=true , address=193.149.164.143:4695 , piece_id=3QTYKOTBCOJ4CRM76FR2XKONKQVYTQQMPTRH3E6MYJR4ODBPY3AQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: ecclient: upload failed (node:12TZfgz24gnZr4UryzEKw6gu94njnmCvafX5X64MKFSJqaUWSVh, address:193.149.164.143:4695): protocol: write tcp 188.166.225.170:43298->193.149.164.143:4695: use of closed network connection; write tcp 188.166.225.170:43298->193.149.164.143:4695: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43298->193.149.164.143:4695: use of closed network connection Aug 4 16:08:26.041079005 (118h11m15.978737504s) [noise=true , address=104.36.82.44:4886 , piece_id=Q3BIO4GNMY64IWXC7VZ5BAYKYRJJ3GKFV7Z4IOG7SP62IUJHXN2A, seg_pos={0 0}, upload=113540]: Done uploading piece. err: ecclient: upload failed (node:1n2cXhqd5EWbNPTXem83jQUjYq5ZaGKBidbVCQzxYfR3rWddGR, address:104.36.82.44:4886): protocol: write tcp 188.166.225.170:38030->104.36.82.44:4886: use of closed network connection; write tcp 188.166.225.170:38030->104.36.82.44:4886: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38030->104.36.82.44:4886: use of closed network connection Aug 4 16:08:26.041128580 (118h11m15.97878711s) [noise=true , address=104.36.82.39:4728 , piece_id=ZCL6YOUXY4P6HTYUTTILZR4TLG44CDTXWLOJJWOHPFSORA2IBCBA, seg_pos={0 0}, upload=113540]: Done uploading piece. err: ecclient: upload failed (node:1YoLzpb4mMYZ8TsdxnMocekZZ1gLfTrdCM8mxTDj4tY6khZGvi, address:104.36.82.39:4728): protocol: write tcp 188.166.225.170:34326->104.36.82.39:4728: use of closed network connection; write tcp 188.166.225.170:34326->104.36.82.39:4728: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34326->104.36.82.39:4728: use of closed network connection Aug 4 16:08:26.041195678 (118h11m15.978854187s) [noise=true , address=104.36.82.45:4930 , piece_id=LBMDUGYMQI7PBBGEOT22S7OGNVRIYS66USSYFAZOSGI445GWKCYA, seg_pos={0 0}, upload=113540]: Done uploading piece. err: ecclient: upload failed (node:12qaYFSpezuLAc5zJfAYruzad6nTpYFDpog7bmKMVTA8MVm76NB, address:104.36.82.45:4930): protocol: expected piece hash; context deadline exceeded; EOF Aug 4 16:08:26.041203374 (118h11m15.978861913s) [noise=true , address=104.36.82.42:4823 , piece_id=66Q6G7YBCVUEAHU3Y5LHPR5MIT7H7NQWI2TWWNHKZT5FYS4DNQBA, seg_pos={0 0}, upload=113540]: Done uploading piece. err: ecclient: upload failed (node:1fkA5DEj9RqsR6N85z7azvujy8tPkUhB5GCqBpUmFb4vvwbd8y, address:104.36.82.42:4823): protocol: write tcp 188.166.225.170:37230->104.36.82.42:4823: use of closed network connection; write tcp 188.166.225.170:37230->104.36.82.42:4823: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37230->104.36.82.42:4823: use of closed network connection Aug 4 16:08:26.041258960 (118h11m15.978917489s) [seg_pos={0 0}, upload=113540]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Aug 4 16:08:26.274088873 (118h11m16.211747372s) [upload=113540]: Done uploading segments... Aug 4 16:08:26.274113650 (118h11m16.211772139s) [upload=113540]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 6.263782499s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 5.695310199s FAIL (2025-08-04 16:18:21 UTC) orbiter warning: Operation: Commit uplink: failed to upload enough pieces (needed at least 14 but got 12) Duration: 5.695337131s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [.X.............................X............................] duration: 1.88466533s FAIL (2025-08-04 16:18:14 UTC) orbiter warning: Operation: DownloadObject uplink: object not found ("orbiter_us_0/1K") Duration: 1.884686299s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 5.589802575s FAIL (2025-08-04 16:18:02 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.589860556s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [................................................XX.X.XX.XX..] duration: 10.762033595s FAIL (2025-08-04 16:17:57 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 10.762045048s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 5.608377281s FAIL (2025-08-04 16:17:55 UTC) orbiter warning: Operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 4.36.82.36:4625 , piece_id=FCK7QGPSKJ7VDK4D7BVNF44BRPQCQS5V5ICAJHELAQHDBSQV5R3Q, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.146274823 (118h10m17.083933322s) [noise=true , address=45.91.116.7:4026 , piece_id=CB6TYC4A6DQOFFKVFPNMF5MH5MGMJHU5Z25B4QXQJI5ETDWLQJJQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.146672965 (118h10m17.084331464s) [noise=true , address=104.36.82.16:4321 , piece_id=42MZPFPEBQNQRJUHQJLZQP2UWGDV2OAXT6Z4KTWPPWLALISASN5Q, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.146726166 (118h10m17.084384665s) [noise=true , address=104.36.82.55:4206 , piece_id=KSAKYO6MALVA5REEKKJZ7MGX7YIRC2PDI5PNHFHRASPBD47V5SOQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.147136371 (118h10m17.08479488s) [noise=true , address=104.36.82.42:4823 , piece_id=66Q6G7YBCVUEAHU3Y5LHPR5MIT7H7NQWI2TWWNHKZT5FYS4DNQBA, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.147479628 (118h10m17.085138127s) [noise=true , address=193.149.164.143:4695 , piece_id=3QTYKOTBCOJ4CRM76FR2XKONKQVYTQQMPTRH3E6MYJR4ODBPY3AQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.147821592 (118h10m17.085480091s) [noise=true , address=104.36.82.62:4411 , piece_id=CIJJ4SMWKFDOIZZ5UEYIXFZHUNAZKRMM2P36IJIT5BHUX5STDZ4A, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.148134220 (118h10m17.085792719s) [noise=true , address=104.36.82.55:4224 , piece_id=NJYMQ2467DXHUEC6ELXITFJF6WWKM4XFEONRJVRF6QSJBDEMC7JA, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.148548573 (118h10m17.086207072s) [noise=true , address=104.36.82.13:4193 , piece_id=WJA72IWFD3TUA2WPGJGB6A7AHJXI4PGOK62YDMKSMDPUAVHUHW4Q, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.148889576 (118h10m17.086548085s) [noise=true , address=104.36.82.39:4728 , piece_id=ZCL6YOUXY4P6HTYUTTILZR4TLG44CDTXWLOJJWOHPFSORA2IBCBA, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.149205931 (118h10m17.08686443s) [noise=true , address=104.36.82.59:4326 , piece_id=CS7FVO3IRXKZFB2EZM2HQBVEVM4XTTFI5Y7TH3NJZGIYMREVY7UQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.149509752 (118h10m17.087168261s) [noise=true , address=104.36.82.45:4930 , piece_id=LBMDUGYMQI7PBBGEOT22S7OGNVRIYS66USSYFAZOSGI445GWKCYA, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.149866545 (118h10m17.087525044s) [noise=true , address=104.36.82.23:4474 , piece_id=PLEKQKMEQKQDW72BNLLY4JOOPFI6CPNNGT74M5EFFNDQQFWELXUQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.150193601 (118h10m17.08785211s) [noise=true , address=104.36.82.13:4212 , piece_id=ER6VIEGFSG2F7542M5P66RS3QAUQYV7BWPQ4YOX55Q3XX6DDCTGA, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.150515557 (118h10m17.088174066s) [noise=true , address=104.36.82.46:4485 , piece_id=P4FMDR5NCQOLKSXQBLE7I7BPEZJ2URAPPVNKOR53ZPROBCE4YAJQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.150935149 (118h10m17.088593648s) [noise=true , address=104.36.82.54:4181 , piece_id=AHW774W42KLBATPDVRQRKH4MQWV5YP2XEEUY6VASDMYJ2BKXCS2Q, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.151249151 (118h10m17.08890765s) [noise=true , address=104.36.82.44:4886 , piece_id=Q3BIO4GNMY64IWXC7VZ5BAYKYRJJ3GKFV7Z4IOG7SP62IUJHXN2A, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.151276363 (118h10m17.088934862s) [noise=true , address=104.36.82.9:4019 , piece_id=SBCPU6XTMB4CRRTFKDBYFQQ2XZZNE3I6AF7VVSGYC356SPQNVHUQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:28.640156348 (118h10m18.577814857s) [upload=113540]: Next returned nil segment Aug 4 16:07:28.640223306 (118h10m18.577881825s) [upload=113540]: Waiting for error group managing segments... Aug 4 16:07:29.062127352 (118h10m18.999785841s) [noise=true , address=104.36.82.16:4321 , piece_id=42MZPFPEBQNQRJUHQJLZQP2UWGDV2OAXT6Z4KTWPPWLALISASN5Q, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.080565981 (118h10m19.01822448s) [noise=true , address=104.36.82.54:4181 , piece_id=AHW774W42KLBATPDVRQRKH4MQWV5YP2XEEUY6VASDMYJ2BKXCS2Q, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.081329852 (118h10m19.018988352s) [noise=true , address=104.36.82.62:4411 , piece_id=CIJJ4SMWKFDOIZZ5UEYIXFZHUNAZKRMM2P36IJIT5BHUX5STDZ4A, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.081846201 (118h10m19.0195047s) [noise=true , address=104.36.82.55:4224 , piece_id=NJYMQ2467DXHUEC6ELXITFJF6WWKM4XFEONRJVRF6QSJBDEMC7JA, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.091017397 (118h10m19.028675896s) [noise=true , address=104.36.82.46:4485 , piece_id=P4FMDR5NCQOLKSXQBLE7I7BPEZJ2URAPPVNKOR53ZPROBCE4YAJQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.105092769 (118h10m19.042751268s) [noise=true , address=104.36.82.13:4212 , piece_id=ER6VIEGFSG2F7542M5P66RS3QAUQYV7BWPQ4YOX55Q3XX6DDCTGA, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.107141759 (118h10m19.044800248s) [noise=true , address=104.36.82.23:4474 , piece_id=PLEKQKMEQKQDW72BNLLY4JOOPFI6CPNNGT74M5EFFNDQQFWELXUQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.116502399 (118h10m19.054160898s) [noise=true , address=45.91.116.7:4026 , piece_id=CB6TYC4A6DQOFFKVFPNMF5MH5MGMJHU5Z25B4QXQJI5ETDWLQJJQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.153680253 (118h10m19.091338752s) [noise=true , address=104.36.82.9:4019 , piece_id=SBCPU6XTMB4CRRTFKDBYFQQ2XZZNE3I6AF7VVSGYC356SPQNVHUQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.210191177 (118h10m19.147849686s) [noise=true , address=104.36.82.36:4625 , piece_id=FCK7QGPSKJ7VDK4D7BVNF44BRPQCQS5V5ICAJHELAQHDBSQV5R3Q, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.219317350 (118h10m19.156975849s) [noise=true , address=104.36.82.55:4206 , piece_id=KSAKYO6MALVA5REEKKJZ7MGX7YIRC2PDI5PNHFHRASPBD47V5SOQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:31.340384557 (118h10m21.278043046s) [noise=true , address=104.36.82.59:4326 , piece_id=CS7FVO3IRXKZFB2EZM2HQBVEVM4XTTFI5Y7TH3NJZGIYMREVY7UQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:32.464238230 (118h10m22.401896729s) [noise=true , address=104.36.82.13:4193 , piece_id=WJA72IWFD3TUA2WPGJGB6A7AHJXI4PGOK62YDMKSMDPUAVHUHW4Q, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:08:26.041033880 (118h11m15.978692379s) [noise=true , address=193.149.164.143:4695 , piece_id=3QTYKOTBCOJ4CRM76FR2XKONKQVYTQQMPTRH3E6MYJR4ODBPY3AQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: ecclient: upload failed (node:12TZfgz24gnZr4UryzEKw6gu94njnmCvafX5X64MKFSJqaUWSVh, address:193.149.164.143:4695): protocol: write tcp 188.166.225.170:43298->193.149.164.143:4695: use of closed network connection; write tcp 188.166.225.170:43298->193.149.164.143:4695: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43298->193.149.164.143:4695: use of closed network connection Aug 4 16:08:26.041079005 (118h11m15.978737504s) [noise=true , address=104.36.82.44:4886 , piece_id=Q3BIO4GNMY64IWXC7VZ5BAYKYRJJ3GKFV7Z4IOG7SP62IUJHXN2A, seg_pos={0 0}, upload=113540]: Done uploading piece. err: ecclient: upload failed (node:1n2cXhqd5EWbNPTXem83jQUjYq5ZaGKBidbVCQzxYfR3rWddGR, address:104.36.82.44:4886): protocol: write tcp 188.166.225.170:38030->104.36.82.44:4886: use of closed network connection; write tcp 188.166.225.170:38030->104.36.82.44:4886: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38030->104.36.82.44:4886: use of closed network connection Aug 4 16:08:26.041128580 (118h11m15.97878711s) [noise=true , address=104.36.82.39:4728 , piece_id=ZCL6YOUXY4P6HTYUTTILZR4TLG44CDTXWLOJJWOHPFSORA2IBCBA, seg_pos={0 0}, upload=113540]: Done uploading piece. err: ecclient: upload failed (node:1YoLzpb4mMYZ8TsdxnMocekZZ1gLfTrdCM8mxTDj4tY6khZGvi, address:104.36.82.39:4728): protocol: write tcp 188.166.225.170:34326->104.36.82.39:4728: use of closed network connection; write tcp 188.166.225.170:34326->104.36.82.39:4728: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34326->104.36.82.39:4728: use of closed network connection Aug 4 16:08:26.041195678 (118h11m15.978854187s) [noise=true , address=104.36.82.45:4930 , piece_id=LBMDUGYMQI7PBBGEOT22S7OGNVRIYS66USSYFAZOSGI445GWKCYA, seg_pos={0 0}, upload=113540]: Done uploading piece. err: ecclient: upload failed (node:12qaYFSpezuLAc5zJfAYruzad6nTpYFDpog7bmKMVTA8MVm76NB, address:104.36.82.45:4930): protocol: expected piece hash; context deadline exceeded; EOF Aug 4 16:08:26.041203374 (118h11m15.978861913s) [noise=true , address=104.36.82.42:4823 , piece_id=66Q6G7YBCVUEAHU3Y5LHPR5MIT7H7NQWI2TWWNHKZT5FYS4DNQBA, seg_pos={0 0}, upload=113540]: Done uploading piece. err: ecclient: upload failed (node:1fkA5DEj9RqsR6N85z7azvujy8tPkUhB5GCqBpUmFb4vvwbd8y, address:104.36.82.42:4823): protocol: write tcp 188.166.225.170:37230->104.36.82.42:4823: use of closed network connection; write tcp 188.166.225.170:37230->104.36.82.42:4823: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37230->104.36.82.42:4823: use of closed network connection Aug 4 16:08:26.041258960 (118h11m15.978917489s) [seg_pos={0 0}, upload=113540]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Aug 4 16:08:26.274088873 (118h11m16.211747372s) [upload=113540]: Done uploading segments... Aug 4 16:08:26.274113650 (118h11m16.211772139s) [upload=113540]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.608410356s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 7.193775612s FAIL (2025-08-04 16:17:45 UTC) orbiter warning: Operation: Commit uplink: failed to upload enough pieces (needed at least 14 but got 12) Duration: 7.193823003s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 5.71153104s FAIL (2025-08-04 16:17:39 UTC) orbiter warning: Operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 4.36.82.36:4625 , piece_id=FCK7QGPSKJ7VDK4D7BVNF44BRPQCQS5V5ICAJHELAQHDBSQV5R3Q, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.146274823 (118h10m17.083933322s) [noise=true , address=45.91.116.7:4026 , piece_id=CB6TYC4A6DQOFFKVFPNMF5MH5MGMJHU5Z25B4QXQJI5ETDWLQJJQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.146672965 (118h10m17.084331464s) [noise=true , address=104.36.82.16:4321 , piece_id=42MZPFPEBQNQRJUHQJLZQP2UWGDV2OAXT6Z4KTWPPWLALISASN5Q, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.146726166 (118h10m17.084384665s) [noise=true , address=104.36.82.55:4206 , piece_id=KSAKYO6MALVA5REEKKJZ7MGX7YIRC2PDI5PNHFHRASPBD47V5SOQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.147136371 (118h10m17.08479488s) [noise=true , address=104.36.82.42:4823 , piece_id=66Q6G7YBCVUEAHU3Y5LHPR5MIT7H7NQWI2TWWNHKZT5FYS4DNQBA, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.147479628 (118h10m17.085138127s) [noise=true , address=193.149.164.143:4695 , piece_id=3QTYKOTBCOJ4CRM76FR2XKONKQVYTQQMPTRH3E6MYJR4ODBPY3AQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.147821592 (118h10m17.085480091s) [noise=true , address=104.36.82.62:4411 , piece_id=CIJJ4SMWKFDOIZZ5UEYIXFZHUNAZKRMM2P36IJIT5BHUX5STDZ4A, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.148134220 (118h10m17.085792719s) [noise=true , address=104.36.82.55:4224 , piece_id=NJYMQ2467DXHUEC6ELXITFJF6WWKM4XFEONRJVRF6QSJBDEMC7JA, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.148548573 (118h10m17.086207072s) [noise=true , address=104.36.82.13:4193 , piece_id=WJA72IWFD3TUA2WPGJGB6A7AHJXI4PGOK62YDMKSMDPUAVHUHW4Q, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.148889576 (118h10m17.086548085s) [noise=true , address=104.36.82.39:4728 , piece_id=ZCL6YOUXY4P6HTYUTTILZR4TLG44CDTXWLOJJWOHPFSORA2IBCBA, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.149205931 (118h10m17.08686443s) [noise=true , address=104.36.82.59:4326 , piece_id=CS7FVO3IRXKZFB2EZM2HQBVEVM4XTTFI5Y7TH3NJZGIYMREVY7UQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.149509752 (118h10m17.087168261s) [noise=true , address=104.36.82.45:4930 , piece_id=LBMDUGYMQI7PBBGEOT22S7OGNVRIYS66USSYFAZOSGI445GWKCYA, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.149866545 (118h10m17.087525044s) [noise=true , address=104.36.82.23:4474 , piece_id=PLEKQKMEQKQDW72BNLLY4JOOPFI6CPNNGT74M5EFFNDQQFWELXUQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.150193601 (118h10m17.08785211s) [noise=true , address=104.36.82.13:4212 , piece_id=ER6VIEGFSG2F7542M5P66RS3QAUQYV7BWPQ4YOX55Q3XX6DDCTGA, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.150515557 (118h10m17.088174066s) [noise=true , address=104.36.82.46:4485 , piece_id=P4FMDR5NCQOLKSXQBLE7I7BPEZJ2URAPPVNKOR53ZPROBCE4YAJQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.150935149 (118h10m17.088593648s) [noise=true , address=104.36.82.54:4181 , piece_id=AHW774W42KLBATPDVRQRKH4MQWV5YP2XEEUY6VASDMYJ2BKXCS2Q, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.151249151 (118h10m17.08890765s) [noise=true , address=104.36.82.44:4886 , piece_id=Q3BIO4GNMY64IWXC7VZ5BAYKYRJJ3GKFV7Z4IOG7SP62IUJHXN2A, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.151276363 (118h10m17.088934862s) [noise=true , address=104.36.82.9:4019 , piece_id=SBCPU6XTMB4CRRTFKDBYFQQ2XZZNE3I6AF7VVSGYC356SPQNVHUQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:28.640156348 (118h10m18.577814857s) [upload=113540]: Next returned nil segment Aug 4 16:07:28.640223306 (118h10m18.577881825s) [upload=113540]: Waiting for error group managing segments... Aug 4 16:07:29.062127352 (118h10m18.999785841s) [noise=true , address=104.36.82.16:4321 , piece_id=42MZPFPEBQNQRJUHQJLZQP2UWGDV2OAXT6Z4KTWPPWLALISASN5Q, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.080565981 (118h10m19.01822448s) [noise=true , address=104.36.82.54:4181 , piece_id=AHW774W42KLBATPDVRQRKH4MQWV5YP2XEEUY6VASDMYJ2BKXCS2Q, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.081329852 (118h10m19.018988352s) [noise=true , address=104.36.82.62:4411 , piece_id=CIJJ4SMWKFDOIZZ5UEYIXFZHUNAZKRMM2P36IJIT5BHUX5STDZ4A, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.081846201 (118h10m19.0195047s) [noise=true , address=104.36.82.55:4224 , piece_id=NJYMQ2467DXHUEC6ELXITFJF6WWKM4XFEONRJVRF6QSJBDEMC7JA, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.091017397 (118h10m19.028675896s) [noise=true , address=104.36.82.46:4485 , piece_id=P4FMDR5NCQOLKSXQBLE7I7BPEZJ2URAPPVNKOR53ZPROBCE4YAJQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.105092769 (118h10m19.042751268s) [noise=true , address=104.36.82.13:4212 , piece_id=ER6VIEGFSG2F7542M5P66RS3QAUQYV7BWPQ4YOX55Q3XX6DDCTGA, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.107141759 (118h10m19.044800248s) [noise=true , address=104.36.82.23:4474 , piece_id=PLEKQKMEQKQDW72BNLLY4JOOPFI6CPNNGT74M5EFFNDQQFWELXUQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.116502399 (118h10m19.054160898s) [noise=true , address=45.91.116.7:4026 , piece_id=CB6TYC4A6DQOFFKVFPNMF5MH5MGMJHU5Z25B4QXQJI5ETDWLQJJQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.153680253 (118h10m19.091338752s) [noise=true , address=104.36.82.9:4019 , piece_id=SBCPU6XTMB4CRRTFKDBYFQQ2XZZNE3I6AF7VVSGYC356SPQNVHUQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.210191177 (118h10m19.147849686s) [noise=true , address=104.36.82.36:4625 , piece_id=FCK7QGPSKJ7VDK4D7BVNF44BRPQCQS5V5ICAJHELAQHDBSQV5R3Q, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.219317350 (118h10m19.156975849s) [noise=true , address=104.36.82.55:4206 , piece_id=KSAKYO6MALVA5REEKKJZ7MGX7YIRC2PDI5PNHFHRASPBD47V5SOQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:31.340384557 (118h10m21.278043046s) [noise=true , address=104.36.82.59:4326 , piece_id=CS7FVO3IRXKZFB2EZM2HQBVEVM4XTTFI5Y7TH3NJZGIYMREVY7UQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:32.464238230 (118h10m22.401896729s) [noise=true , address=104.36.82.13:4193 , piece_id=WJA72IWFD3TUA2WPGJGB6A7AHJXI4PGOK62YDMKSMDPUAVHUHW4Q, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:08:26.041033880 (118h11m15.978692379s) [noise=true , address=193.149.164.143:4695 , piece_id=3QTYKOTBCOJ4CRM76FR2XKONKQVYTQQMPTRH3E6MYJR4ODBPY3AQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: ecclient: upload failed (node:12TZfgz24gnZr4UryzEKw6gu94njnmCvafX5X64MKFSJqaUWSVh, address:193.149.164.143:4695): protocol: write tcp 188.166.225.170:43298->193.149.164.143:4695: use of closed network connection; write tcp 188.166.225.170:43298->193.149.164.143:4695: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43298->193.149.164.143:4695: use of closed network connection Aug 4 16:08:26.041079005 (118h11m15.978737504s) [noise=true , address=104.36.82.44:4886 , piece_id=Q3BIO4GNMY64IWXC7VZ5BAYKYRJJ3GKFV7Z4IOG7SP62IUJHXN2A, seg_pos={0 0}, upload=113540]: Done uploading piece. err: ecclient: upload failed (node:1n2cXhqd5EWbNPTXem83jQUjYq5ZaGKBidbVCQzxYfR3rWddGR, address:104.36.82.44:4886): protocol: write tcp 188.166.225.170:38030->104.36.82.44:4886: use of closed network connection; write tcp 188.166.225.170:38030->104.36.82.44:4886: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38030->104.36.82.44:4886: use of closed network connection Aug 4 16:08:26.041128580 (118h11m15.97878711s) [noise=true , address=104.36.82.39:4728 , piece_id=ZCL6YOUXY4P6HTYUTTILZR4TLG44CDTXWLOJJWOHPFSORA2IBCBA, seg_pos={0 0}, upload=113540]: Done uploading piece. err: ecclient: upload failed (node:1YoLzpb4mMYZ8TsdxnMocekZZ1gLfTrdCM8mxTDj4tY6khZGvi, address:104.36.82.39:4728): protocol: write tcp 188.166.225.170:34326->104.36.82.39:4728: use of closed network connection; write tcp 188.166.225.170:34326->104.36.82.39:4728: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34326->104.36.82.39:4728: use of closed network connection Aug 4 16:08:26.041195678 (118h11m15.978854187s) [noise=true , address=104.36.82.45:4930 , piece_id=LBMDUGYMQI7PBBGEOT22S7OGNVRIYS66USSYFAZOSGI445GWKCYA, seg_pos={0 0}, upload=113540]: Done uploading piece. err: ecclient: upload failed (node:12qaYFSpezuLAc5zJfAYruzad6nTpYFDpog7bmKMVTA8MVm76NB, address:104.36.82.45:4930): protocol: expected piece hash; context deadline exceeded; EOF Aug 4 16:08:26.041203374 (118h11m15.978861913s) [noise=true , address=104.36.82.42:4823 , piece_id=66Q6G7YBCVUEAHU3Y5LHPR5MIT7H7NQWI2TWWNHKZT5FYS4DNQBA, seg_pos={0 0}, upload=113540]: Done uploading piece. err: ecclient: upload failed (node:1fkA5DEj9RqsR6N85z7azvujy8tPkUhB5GCqBpUmFb4vvwbd8y, address:104.36.82.42:4823): protocol: write tcp 188.166.225.170:37230->104.36.82.42:4823: use of closed network connection; write tcp 188.166.225.170:37230->104.36.82.42:4823: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37230->104.36.82.42:4823: use of closed network connection Aug 4 16:08:26.041258960 (118h11m15.978917489s) [seg_pos={0 0}, upload=113540]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Aug 4 16:08:26.274088873 (118h11m16.211747372s) [upload=113540]: Done uploading segments... Aug 4 16:08:26.274113650 (118h11m16.211772139s) [upload=113540]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.711620501s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 5.513039039s FAIL (2025-08-04 16:17:38 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.513056271s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [..................................................XX.....X.X] duration: 30.238061856s FAIL (2025-08-04 16:17:35 UTC) orbiter warning: Operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 9101416 (118h19m28.546759915s) [upload=113681]: Done uploading segments... Aug 4 16:16:38.609120443 (118h19m28.546778932s) [upload=113681]: Upload finished. err: <nil> Aug 4 16:17:05.998053134 (118h19m55.935711663s) [upload=113692]: Starting upload Aug 4 16:17:05.998173415 (118h19m55.935831914s) [upload=113692]: Done starting upload Aug 4 16:17:05.998260210 (118h19m55.93591877s) [upload=113692]: Uploading segments... Aug 4 16:17:05.998601433 (118h19m55.936259952s) [upload=113692]: Got next segment. Inline: false Aug 4 16:17:05.998658682 (118h19m55.936317201s) [upload=113692]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 4 16:17:05.998667330 (118h19m55.936325879s) [upload=113692]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 4 16:17:06.238820923 (118h19m56.176479422s) [seg_pos={0 0}, upload=113692]: Begin upload segment... Aug 4 16:17:06.238865489 (118h19m56.176523988s) [seg_pos={0 0}, upload=113692]: Done begin upload segment. Aug 4 16:17:06.238890357 (118h19m56.176548856s) [noise=true , address=45.91.116.10:4081 , piece_id=SPCXBEJMWGBUZJ5V5N6LZ556A3FOREOACVDVGQ4WFTO5DYB2HCXA, seg_pos={0 0}, upload=113692]: Uploading piece... Aug 4 16:17:06.238955331 (118h19m56.17661383s) [noise=true , address=77.90.31.13:4627 , piece_id=DCUKITRW7KW3ELY5F6Y6WV2R6N5JIQTFL2MOIL3A7B5CEPMA7IBQ, seg_pos={0 0}, upload=113692]: Uploading piece... Aug 4 16:17:06.239263350 (118h19m56.176921849s) [noise=true , address=104.36.82.2:4732 , piece_id=X22BQ7F4IGXKWPTRXURREONCSDOJHNB735ZJMZRYGTKCVY5IXGHQ, seg_pos={0 0}, upload=113692]: Uploading piece... Aug 4 16:17:06.239464465 (118h19m56.177122954s) [noise=true , address=77.90.31.14:4662 , piece_id=ZFG5RDMFHAQ6VYZJJYZIBQKS4BIHSE4YZ6CDNU32DUKCBPCAK7FA, seg_pos={0 0}, upload=113692]: Uploading piece... Aug 4 16:17:06.239532956 (118h19m56.177191455s) [noise=true , address=77.90.31.17:4771 , piece_id=J55SVVDJNDMKL3GSCHJ6XCCCHTURNZ4ETBLEL4GBKJLKTSEXHY4A, seg_pos={0 0}, upload=113692]: Uploading piece... Aug 4 16:17:06.239788194 (118h19m56.177446693s) [noise=true , address=45.91.116.13:4154 , piece_id=PCE3GCPLTOJWS6YP37JWPRKXOU6RG7ARUGRX33LJKAJYIESBBC6Q, seg_pos={0 0}, upload=113692]: Uploading piece... Aug 4 16:17:06.239823923 (118h19m56.177482422s) [noise=true , address=104.36.82.16:4295 , piece_id=X6FJ7VU6GCD64DOSFZP53H6EA4K2R3YZTZMBD7PL5B5KVFAU3S5Q, seg_pos={0 0}, upload=113692]: Uploading piece... Aug 4 16:17:06.240034686 (118h19m56.177693185s) [noise=true , address=45.91.116.6:4973 , piece_id=SQLGWOHR6Z4HUM47764MSCREL6S4SVA2OHGJIRMVRZJ3BNDOMCLQ, seg_pos={0 0}, upload=113692]: Uploading piece... Aug 4 16:17:06.240273823 (118h19m56.177932322s) [noise=true , address=45.91.116.12:4226 , piece_id=OIRF34NSWFFAX77PMLWCM4KI7YXLKC7FZSMPL443P2X7CXOQPUYA, seg_pos={0 0}, upload=113692]: Uploading piece... Aug 4 16:17:06.240365408 (118h19m56.178024038s) [noise=true , address=104.36.82.2:4730 , piece_id=S3GZJB6UX4LLA6FZFF2GQEWA4BTFNDPDS65W4PTMEU6LUBNXODDQ, seg_pos={0 0}, upload=113692]: Uploading piece... Aug 4 16:17:06.240586992 (118h19m56.178245491s) [noise=true , address=45.91.116.7:4011 , piece_id=VNBRJ2J6IMRNQZNTHBZEWIIM6IHHA4GNOF7PO3GGVUORJFTFCAPA, seg_pos={0 0}, upload=113692]: Uploading piece... Aug 4 16:17:06.240742490 (118h19m56.178400989s) [noise=true , address=104.36.82.11:4113 , piece_id=RJ65BS7G52KFUASC5U4ZY2OOPTRNHWSJRUWUCBOK27XYOQ6BEQQQ, seg_pos={0 0}, upload=113692]: Uploading piece... Aug 4 16:17:06.240888679 (118h19m56.178547178s) [noise=true , address=45.91.116.4:4939 , piece_id=2PDSOLXDDBNAZSWZFKGFPKVH5WUREFSZVOQ3IUEQ45UC6PS4XZAQ, seg_pos={0 0}, upload=113692]: Uploading piece... Aug 4 16:17:06.241025501 (118h19m56.178684s) [noise=true , address=77.90.31.15:4699 , piece_id=DY5NO5OBWQVVN3DEWL5S4U6RWEUZG7YOQBYV63RNPBYCEKJKO5IA, seg_pos={0 0}, upload=113692]: Uploading piece... Aug 4 16:17:06.241232267 (118h19m56.178890766s) [noise=true , address=45.91.116.8:4048 , piece_id=2LRVVSAJY37IXQWKJTLLZLLRVJCUT4W2IK7Z7P6DDBGH3OPDBUCA, seg_pos={0 0}, upload=113692]: Uploading piece... Aug 4 16:17:06.241269799 (118h19m56.178928298s) [noise=true , address=77.90.31.10:4517 , piece_id=EFBTZDMH7ULHUDAXBZRQ57UOOYFZFUWKP6IPDDNDEPD2V5BMNDGA, seg_pos={0 0}, upload=113692]: Uploading piece... Aug 4 16:17:07.929863629 (118h19m57.867522128s) [upload=113692]: Next returned nil segment Aug 4 16:17:07.929879760 (118h19m57.867538249s) [upload=113692]: Waiting for error group managing segments... Aug 4 16:17:08.255553915 (118h19m58.193212414s) [noise=true , address=77.90.31.10:4517 , piece_id=EFBTZDMH7ULHUDAXBZRQ57UOOYFZFUWKP6IPDDNDEPD2V5BMNDGA, seg_pos={0 0}, upload=113692]: Done uploading piece. err: <nil> Aug 4 16:17:08.269512842 (118h19m58.207171332s) [noise=true , address=45.91.116.6:4973 , piece_id=SQLGWOHR6Z4HUM47764MSCREL6S4SVA2OHGJIRMVRZJ3BNDOMCLQ, seg_pos={0 0}, upload=113692]: Done uploading piece. err: <nil> Aug 4 16:17:08.275215553 (118h19m58.212874052s) [noise=true , address=45.91.116.12:4226 , piece_id=OIRF34NSWFFAX77PMLWCM4KI7YXLKC7FZSMPL443P2X7CXOQPUYA, seg_pos={0 0}, upload=113692]: Done uploading piece. err: <nil> Aug 4 16:17:08.281380589 (118h19m58.219039088s) [noise=true , address=45.91.116.7:4011 , piece_id=VNBRJ2J6IMRNQZNTHBZEWIIM6IHHA4GNOF7PO3GGVUORJFTFCAPA, seg_pos={0 0}, upload=113692]: Done uploading piece. err: <nil> Aug 4 16:17:08.299391320 (118h19m58.237049819s) [noise=true , address=104.36.82.2:4732 , piece_id=X22BQ7F4IGXKWPTRXURREONCSDOJHNB735ZJMZRYGTKCVY5IXGHQ, seg_pos={0 0}, upload=113692]: Done uploading piece. err: <nil> Aug 4 16:17:08.311687867 (118h19m58.249346366s) [noise=true , address=104.36.82.16:4295 , piece_id=X6FJ7VU6GCD64DOSFZP53H6EA4K2R3YZTZMBD7PL5B5KVFAU3S5Q, seg_pos={0 0}, upload=113692]: Done uploading piece. err: <nil> Aug 4 16:17:08.451613662 (118h19m58.389272161s) [noise=true , address=104.36.82.11:4113 , piece_id=RJ65BS7G52KFUASC5U4ZY2OOPTRNHWSJRUWUCBOK27XYOQ6BEQQQ, seg_pos={0 0}, upload=113692]: Done uploading piece. err: <nil> Aug 4 16:17:09.380387010 (118h19m59.318045509s) [noise=true , address=77.90.31.17:4771 , piece_id=J55SVVDJNDMKL3GSCHJ6XCCCHTURNZ4ETBLEL4GBKJLKTSEXHY4A, seg_pos={0 0}, upload=113692]: Done uploading piece. err: <nil> Aug 4 16:17:10.008820665 (118h19m59.946479164s) [noise=true , address=77.90.31.14:4662 , piece_id=ZFG5RDMFHAQ6VYZJJYZIBQKS4BIHSE4YZ6CDNU32DUKCBPCAK7FA, seg_pos={0 0}, upload=113692]: Done uploading piece. err: <nil> Aug 4 16:17:10.134642744 (118h20m0.072301243s) [noise=true , address=77.90.31.13:4627 , piece_id=DCUKITRW7KW3ELY5F6Y6WV2R6N5JIQTFL2MOIL3A7B5CEPMA7IBQ, seg_pos={0 0}, upload=113692]: Done uploading piece. err: <nil> Aug 4 16:17:10.332473572 (118h20m0.270132071s) [noise=true , address=45.91.116.8:4048 , piece_id=2LRVVSAJY37IXQWKJTLLZLLRVJCUT4W2IK7Z7P6DDBGH3OPDBUCA, seg_pos={0 0}, upload=113692]: Done uploading piece. err: <nil> Aug 4 16:17:12.482017276 (118h20m2.419675765s) [noise=true , address=45.91.116.13:4154 , piece_id=PCE3GCPLTOJWS6YP37JWPRKXOU6RG7ARUGRX33LJKAJYIESBBC6Q, seg_pos={0 0}, upload=113692]: Done uploading piece. err: <nil> Aug 4 16:17:35.345966027 (118h20m25.283624526s) [noise=true , address=45.91.116.10:4081 , piece_id=SPCXBEJMWGBUZJ5V5N6LZ556A3FOREOACVDVGQ4WFTO5DYB2HCXA, seg_pos={0 0}, upload=113692]: Done uploading piece. err: ecclient: upload failed (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG, address:45.91.116.10:4081): protocol: write tcp 188.166.225.170:41146->45.91.116.10:4081: use of closed network connection; write tcp 188.166.225.170:41146->45.91.116.10:4081: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41146->45.91.116.10:4081: use of closed network connection Aug 4 16:17:35.346059295 (118h20m25.283717794s) [noise=true , address=77.90.31.15:4699 , piece_id=DY5NO5OBWQVVN3DEWL5S4U6RWEUZG7YOQBYV63RNPBYCEKJKO5IA, seg_pos={0 0}, upload=113692]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: write tcp 188.166.225.170:36558->77.90.31.15:4699: use of closed network connection; write tcp 188.166.225.170:36558->77.90.31.15:4699: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36558->77.90.31.15:4699: use of closed network connection Aug 4 16:17:35.346131072 (118h20m25.283789571s) [noise=true , address=104.36.82.2:4730 , piece_id=S3GZJB6UX4LLA6FZFF2GQEWA4BTFNDPDS65W4PTMEU6LUBNXODDQ, seg_pos={0 0}, upload=113692]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:104.36.82.2:4730): protocol: write tcp 188.166.225.170:47430->104.36.82.2:4730: use of closed network connection; write tcp 188.166.225.170:47430->104.36.82.2:4730: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47430->104.36.82.2:4730: use of closed network connection Aug 4 16:17:35.346143917 (118h20m25.283802416s) [noise=true , address=45.91.116.4:4939 , piece_id=2PDSOLXDDBNAZSWZFKGFPKVH5WUREFSZVOQ3IUEQ45UC6PS4XZAQ, seg_pos={0 0}, upload=113692]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: write tcp 188.166.225.170:59874->45.91.116.4:4939: use of closed network connection; write tcp 188.166.225.170:59874->45.91.116.4:4939: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59874->45.91.116.4:4939: use of closed network connection Aug 4 16:17:35.346185527 (118h20m25.283844026s) [seg_pos={0 0}, upload=113692]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Aug 4 16:17:35.581733629 (118h20m25.519392118s) [upload=113692]: Done uploading segments... Aug 4 16:17:35.581776351 (118h20m25.51943485s) [upload=113692]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.238085542s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [..X.............................X...........................] duration: 1.961975931s FAIL (2025-08-04 16:17:16 UTC) orbiter warning: Operation: DownloadObject uplink: object not found ("orbiter_us_0/1K") Duration: 1.962010627s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 6.854157201s FAIL (2025-08-04 16:17:04 UTC) orbiter warning: Operation: Commit uplink: failed to upload enough pieces (needed at least 14 but got 12) Duration: 6.854173513s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [.................................................XX.X.XX.XX.] duration: 12.20256398s FAIL (2025-08-04 16:16:44 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 12.20257973s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...................................................XX.....X.] duration: 15.68892097s FAIL (2025-08-04 16:16:41 UTC) orbiter warning: Operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 1773 (118h17m27.941760272s) [upload=113655]: Done uploading segments... Aug 4 16:14:38.004119036 (118h17m27.941777535s) [upload=113655]: Upload finished. err: <nil> Aug 4 16:15:32.302616649 (118h18m22.240275148s) [upload=113669]: Starting upload Aug 4 16:15:32.302693065 (118h18m22.240351564s) [upload=113669]: Done starting upload Aug 4 16:15:32.302798066 (118h18m22.240456566s) [upload=113669]: Uploading segments... Aug 4 16:15:32.303449322 (118h18m22.241107821s) [upload=113669]: Got next segment. Inline: false Aug 4 16:15:32.303543322 (118h18m22.241201821s) [upload=113669]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 4 16:15:32.303549073 (118h18m22.241207572s) [upload=113669]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 4 16:15:32.539789831 (118h18m22.47744833s) [seg_pos={0 0}, upload=113669]: Begin upload segment... Aug 4 16:15:32.539839626 (118h18m22.477498125s) [seg_pos={0 0}, upload=113669]: Done begin upload segment. Aug 4 16:15:32.539860957 (118h18m22.477519456s) [noise=true , address=77.90.31.19:4843 , piece_id=FVR7EP4XTKJNRYKRFUIV4Y2DBYI67FJ3EHKZVJSLOGRVM7S6M3KQ, seg_pos={0 0}, upload=113669]: Uploading piece... Aug 4 16:15:32.539899040 (118h18m22.477557529s) [noise=true , address=45.91.116.4:4940 , piece_id=RGXORMHLSXGPZ2OE7U7FMBLAM6FC3TQ2GPWUY6ER2OXGFE7SQYHA, seg_pos={0 0}, upload=113669]: Uploading piece... Aug 4 16:15:32.540296541 (118h18m22.47795504s) [noise=true , address=77.90.31.16:4735 , piece_id=2SAWIJVGRQNDE35L2KQZIBLAYAR56YRJ7TQUBWFGHPGU2SZUNQHA, seg_pos={0 0}, upload=113669]: Uploading piece... Aug 4 16:15:32.540452799 (118h18m22.478111298s) [noise=true , address=77.90.31.13:4627 , piece_id=KSEAMM2S7WQJTIA7W62C2OLYY6IYNEK6IJEY5YH3SQRII3FKTH3Q, seg_pos={0 0}, upload=113669]: Uploading piece... Aug 4 16:15:32.540615701 (118h18m22.47827419s) [noise=true , address=45.91.116.7:4011 , piece_id=WEXVVFXFV6P733TCS5KGBVTLSNCBUDV5BYDRPNHCJ6YXJ5YE66HA, seg_pos={0 0}, upload=113669]: Uploading piece... Aug 4 16:15:32.540829069 (118h18m22.478487568s) [noise=true , address=104.36.82.11:4113 , piece_id=SNHPZDFGMXWTNFPYU6RE3W2LFNTXHJZPA5QZTYZ4SEXCEMLKSDTQ, seg_pos={0 0}, upload=113669]: Uploading piece... Aug 4 16:15:32.540895115 (118h18m22.478553614s) [noise=true , address=104.36.82.9:4006 , piece_id=U2EDQZHMXT6AFVN5SRGFZUCO57ILACJP3VDCVULZBX6WDSV4XW4Q, seg_pos={0 0}, upload=113669]: Uploading piece... Aug 4 16:15:32.541147618 (118h18m22.478806117s) [noise=true , address=45.91.116.10:4081 , piece_id=WIW36M6BBTKUKOABD2GLECEORY7UGMOIJIIYAGPHPGOCLTOFHWBA, seg_pos={0 0}, upload=113669]: Uploading piece... Aug 4 16:15:32.541151206 (118h18m22.478809705s) [noise=true , address=45.91.116.5:4297 , piece_id=HQPA4GTU6XJ4ZHV7CSBZOLWGS4F3RRDSQU7QZ26IMYXJIDONMTWA, seg_pos={0 0}, upload=113669]: Uploading piece... Aug 4 16:15:32.541986113 (118h18m22.479644602s) [noise=true , address=77.90.31.14:4661 , piece_id=BKOCMYQV56DZXRHIHDY4OYRHPF5IVVQJRDDUO5JQXF2FBQQYAGCQ, seg_pos={0 0}, upload=113669]: Uploading piece... Aug 4 16:15:32.542171167 (118h18m22.479829666s) [noise=true , address=77.90.31.10:4519 , piece_id=BB72QDQ7SB3BPN365GK2AGZBLRRRONM2VYMACVN33GVMDM6HIWGQ, seg_pos={0 0}, upload=113669]: Uploading piece... Aug 4 16:15:32.542309612 (118h18m22.479968111s) [noise=true , address=77.90.31.15:4698 , piece_id=OITYF76UF6L4FDEJNIPZXU2PWNPL4ARVHCH74UVIT6EBBSYQDZ6Q, seg_pos={0 0}, upload=113669]: Uploading piece... Aug 4 16:15:32.542643841 (118h18m22.480302341s) [noise=true , address=104.36.82.15:4259 , piece_id=ZPBF63TYCMS75EM37ODUHINUXYDEC4V7BSOO3ZF7SEXBEL5SFQWA, seg_pos={0 0}, upload=113669]: Uploading piece... Aug 4 16:15:32.543281982 (118h18m22.480940471s) [noise=true , address=77.90.31.10:4517 , piece_id=6IA36O5LXNTPZGU5W72J5BJMI346WVQEKVDPRUCJMJWQNBACDIOA, seg_pos={0 0}, upload=113669]: Uploading piece... Aug 4 16:15:32.544607649 (118h18m22.482266138s) [noise=true , address=45.91.116.12:4227 , piece_id=FYMNWNR7QAIDRA7ENCFDIWDIDHVIN6SCP3DTJPNKV5YIORDG36GA, seg_pos={0 0}, upload=113669]: Uploading piece... Aug 4 16:15:32.545301176 (118h18m22.482959665s) [noise=true , address=77.90.31.11:4553 , piece_id=ZZPOA5Q5CK6MKYUXLEVJBBAAWVBA7B4CFB5546JJABS6ILPEUAKA, seg_pos={0 0}, upload=113669]: Uploading piece... Aug 4 16:15:34.275056410 (118h18m24.212714909s) [upload=113669]: Next returned nil segment Aug 4 16:15:34.275073813 (118h18m24.212732302s) [upload=113669]: Waiting for error group managing segments... Aug 4 16:15:34.554758063 (118h18m24.492416562s) [noise=true , address=77.90.31.15:4698 , piece_id=OITYF76UF6L4FDEJNIPZXU2PWNPL4ARVHCH74UVIT6EBBSYQDZ6Q, seg_pos={0 0}, upload=113669]: Done uploading piece. err: <nil> Aug 4 16:15:34.556381058 (118h18m24.494039547s) [noise=true , address=77.90.31.14:4661 , piece_id=BKOCMYQV56DZXRHIHDY4OYRHPF5IVVQJRDDUO5JQXF2FBQQYAGCQ, seg_pos={0 0}, upload=113669]: Done uploading piece. err: <nil> Aug 4 16:15:34.559542938 (118h18m24.497201448s) [noise=true , address=45.91.116.5:4297 , piece_id=HQPA4GTU6XJ4ZHV7CSBZOLWGS4F3RRDSQU7QZ26IMYXJIDONMTWA, seg_pos={0 0}, upload=113669]: Done uploading piece. err: <nil> Aug 4 16:15:34.574012413 (118h18m24.511670912s) [noise=true , address=104.36.82.11:4113 , piece_id=SNHPZDFGMXWTNFPYU6RE3W2LFNTXHJZPA5QZTYZ4SEXCEMLKSDTQ, seg_pos={0 0}, upload=113669]: Done uploading piece. err: <nil> Aug 4 16:15:34.591923804 (118h18m24.529582303s) [noise=true , address=77.90.31.10:4519 , piece_id=BB72QDQ7SB3BPN365GK2AGZBLRRRONM2VYMACVN33GVMDM6HIWGQ, seg_pos={0 0}, upload=113669]: Done uploading piece. err: <nil> Aug 4 16:15:34.740603818 (118h18m24.678262317s) [noise=true , address=104.36.82.9:4006 , piece_id=U2EDQZHMXT6AFVN5SRGFZUCO57ILACJP3VDCVULZBX6WDSV4XW4Q, seg_pos={0 0}, upload=113669]: Done uploading piece. err: <nil> Aug 4 16:15:34.770186750 (118h18m24.707845249s) [noise=true , address=45.91.116.7:4011 , piece_id=WEXVVFXFV6P733TCS5KGBVTLSNCBUDV5BYDRPNHCJ6YXJ5YE66HA, seg_pos={0 0}, upload=113669]: Done uploading piece. err: <nil> Aug 4 16:15:34.877238737 (118h18m24.814897236s) [noise=true , address=77.90.31.10:4517 , piece_id=6IA36O5LXNTPZGU5W72J5BJMI346WVQEKVDPRUCJMJWQNBACDIOA, seg_pos={0 0}, upload=113669]: Done uploading piece. err: <nil> Aug 4 16:15:35.259555828 (118h18m25.197214328s) [noise=true , address=77.90.31.11:4553 , piece_id=ZZPOA5Q5CK6MKYUXLEVJBBAAWVBA7B4CFB5546JJABS6ILPEUAKA, seg_pos={0 0}, upload=113669]: Done uploading piece. err: <nil> Aug 4 16:15:35.556139422 (118h18m25.493797921s) [noise=true , address=45.91.116.4:4940 , piece_id=RGXORMHLSXGPZ2OE7U7FMBLAM6FC3TQ2GPWUY6ER2OXGFE7SQYHA, seg_pos={0 0}, upload=113669]: Done uploading piece. err: <nil> Aug 4 16:15:35.593880413 (118h18m25.531538912s) [noise=true , address=45.91.116.12:4227 , piece_id=FYMNWNR7QAIDRA7ENCFDIWDIDHVIN6SCP3DTJPNKV5YIORDG36GA, seg_pos={0 0}, upload=113669]: Done uploading piece. err: <nil> Aug 4 16:15:36.055685242 (118h18m25.993343741s) [noise=true , address=104.36.82.15:4259 , piece_id=ZPBF63TYCMS75EM37ODUHINUXYDEC4V7BSOO3ZF7SEXBEL5SFQWA, seg_pos={0 0}, upload=113669]: Done uploading piece. err: <nil> Aug 4 16:16:01.655205548 (118h18m51.592864057s) [noise=true , address=45.91.116.10:4081 , piece_id=WIW36M6BBTKUKOABD2GLECEORY7UGMOIJIIYAGPHPGOCLTOFHWBA, seg_pos={0 0}, upload=113669]: Done uploading piece. err: ecclient: upload failed (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG, address:45.91.116.10:4081): protocol: write tcp 188.166.225.170:35324->45.91.116.10:4081: use of closed network connection; write tcp 188.166.225.170:35324->45.91.116.10:4081: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35324->45.91.116.10:4081: use of closed network connection Aug 4 16:16:01.655391584 (118h18m51.593050093s) [noise=true , address=77.90.31.13:4627 , piece_id=KSEAMM2S7WQJTIA7W62C2OLYY6IYNEK6IJEY5YH3SQRII3FKTH3Q, seg_pos={0 0}, upload=113669]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: write tcp 188.166.225.170:55624->77.90.31.13:4627: use of closed network connection; write tcp 188.166.225.170:55624->77.90.31.13:4627: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55624->77.90.31.13:4627: use of closed network connection Aug 4 16:16:01.655479321 (118h18m51.59313782s) [noise=true , address=77.90.31.19:4843 , piece_id=FVR7EP4XTKJNRYKRFUIV4Y2DBYI67FJ3EHKZVJSLOGRVM7S6M3KQ, seg_pos={0 0}, upload=113669]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: write tcp 188.166.225.170:42918->77.90.31.19:4843: use of closed network connection; write tcp 188.166.225.170:42918->77.90.31.19:4843: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42918->77.90.31.19:4843: use of closed network connection Aug 4 16:16:01.655511203 (118h18m51.593169702s) [noise=true , address=77.90.31.16:4735 , piece_id=2SAWIJVGRQNDE35L2KQZIBLAYAR56YRJ7TQUBWFGHPGU2SZUNQHA, seg_pos={0 0}, upload=113669]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 188.166.225.170:44366->77.90.31.16:4735: use of closed network connection; write tcp 188.166.225.170:44366->77.90.31.16:4735: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44366->77.90.31.16:4735: use of closed network connection Aug 4 16:16:01.655543364 (118h18m51.593201914s) [seg_pos={0 0}, upload=113669]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Aug 4 16:16:01.885476855 (118h18m51.823135364s) [upload=113669]: Done uploading segments... Aug 4 16:16:01.885512302 (118h18m51.823170811s) [upload=113669]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 15.688944807s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)