Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 6.943261995s FAIL (2025-08-04 16:16:33 UTC) orbiter warning: Operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 4.36.82.36:4625 , piece_id=FCK7QGPSKJ7VDK4D7BVNF44BRPQCQS5V5ICAJHELAQHDBSQV5R3Q, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.146274823 (118h10m17.083933322s) [noise=true , address=45.91.116.7:4026 , piece_id=CB6TYC4A6DQOFFKVFPNMF5MH5MGMJHU5Z25B4QXQJI5ETDWLQJJQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.146672965 (118h10m17.084331464s) [noise=true , address=104.36.82.16:4321 , piece_id=42MZPFPEBQNQRJUHQJLZQP2UWGDV2OAXT6Z4KTWPPWLALISASN5Q, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.146726166 (118h10m17.084384665s) [noise=true , address=104.36.82.55:4206 , piece_id=KSAKYO6MALVA5REEKKJZ7MGX7YIRC2PDI5PNHFHRASPBD47V5SOQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.147136371 (118h10m17.08479488s) [noise=true , address=104.36.82.42:4823 , piece_id=66Q6G7YBCVUEAHU3Y5LHPR5MIT7H7NQWI2TWWNHKZT5FYS4DNQBA, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.147479628 (118h10m17.085138127s) [noise=true , address=193.149.164.143:4695 , piece_id=3QTYKOTBCOJ4CRM76FR2XKONKQVYTQQMPTRH3E6MYJR4ODBPY3AQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.147821592 (118h10m17.085480091s) [noise=true , address=104.36.82.62:4411 , piece_id=CIJJ4SMWKFDOIZZ5UEYIXFZHUNAZKRMM2P36IJIT5BHUX5STDZ4A, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.148134220 (118h10m17.085792719s) [noise=true , address=104.36.82.55:4224 , piece_id=NJYMQ2467DXHUEC6ELXITFJF6WWKM4XFEONRJVRF6QSJBDEMC7JA, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.148548573 (118h10m17.086207072s) [noise=true , address=104.36.82.13:4193 , piece_id=WJA72IWFD3TUA2WPGJGB6A7AHJXI4PGOK62YDMKSMDPUAVHUHW4Q, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.148889576 (118h10m17.086548085s) [noise=true , address=104.36.82.39:4728 , piece_id=ZCL6YOUXY4P6HTYUTTILZR4TLG44CDTXWLOJJWOHPFSORA2IBCBA, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.149205931 (118h10m17.08686443s) [noise=true , address=104.36.82.59:4326 , piece_id=CS7FVO3IRXKZFB2EZM2HQBVEVM4XTTFI5Y7TH3NJZGIYMREVY7UQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.149509752 (118h10m17.087168261s) [noise=true , address=104.36.82.45:4930 , piece_id=LBMDUGYMQI7PBBGEOT22S7OGNVRIYS66USSYFAZOSGI445GWKCYA, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.149866545 (118h10m17.087525044s) [noise=true , address=104.36.82.23:4474 , piece_id=PLEKQKMEQKQDW72BNLLY4JOOPFI6CPNNGT74M5EFFNDQQFWELXUQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.150193601 (118h10m17.08785211s) [noise=true , address=104.36.82.13:4212 , piece_id=ER6VIEGFSG2F7542M5P66RS3QAUQYV7BWPQ4YOX55Q3XX6DDCTGA, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.150515557 (118h10m17.088174066s) [noise=true , address=104.36.82.46:4485 , piece_id=P4FMDR5NCQOLKSXQBLE7I7BPEZJ2URAPPVNKOR53ZPROBCE4YAJQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.150935149 (118h10m17.088593648s) [noise=true , address=104.36.82.54:4181 , piece_id=AHW774W42KLBATPDVRQRKH4MQWV5YP2XEEUY6VASDMYJ2BKXCS2Q, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.151249151 (118h10m17.08890765s) [noise=true , address=104.36.82.44:4886 , piece_id=Q3BIO4GNMY64IWXC7VZ5BAYKYRJJ3GKFV7Z4IOG7SP62IUJHXN2A, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.151276363 (118h10m17.088934862s) [noise=true , address=104.36.82.9:4019 , piece_id=SBCPU6XTMB4CRRTFKDBYFQQ2XZZNE3I6AF7VVSGYC356SPQNVHUQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:28.640156348 (118h10m18.577814857s) [upload=113540]: Next returned nil segment Aug 4 16:07:28.640223306 (118h10m18.577881825s) [upload=113540]: Waiting for error group managing segments... Aug 4 16:07:29.062127352 (118h10m18.999785841s) [noise=true , address=104.36.82.16:4321 , piece_id=42MZPFPEBQNQRJUHQJLZQP2UWGDV2OAXT6Z4KTWPPWLALISASN5Q, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.080565981 (118h10m19.01822448s) [noise=true , address=104.36.82.54:4181 , piece_id=AHW774W42KLBATPDVRQRKH4MQWV5YP2XEEUY6VASDMYJ2BKXCS2Q, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.081329852 (118h10m19.018988352s) [noise=true , address=104.36.82.62:4411 , piece_id=CIJJ4SMWKFDOIZZ5UEYIXFZHUNAZKRMM2P36IJIT5BHUX5STDZ4A, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.081846201 (118h10m19.0195047s) [noise=true , address=104.36.82.55:4224 , piece_id=NJYMQ2467DXHUEC6ELXITFJF6WWKM4XFEONRJVRF6QSJBDEMC7JA, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.091017397 (118h10m19.028675896s) [noise=true , address=104.36.82.46:4485 , piece_id=P4FMDR5NCQOLKSXQBLE7I7BPEZJ2URAPPVNKOR53ZPROBCE4YAJQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.105092769 (118h10m19.042751268s) [noise=true , address=104.36.82.13:4212 , piece_id=ER6VIEGFSG2F7542M5P66RS3QAUQYV7BWPQ4YOX55Q3XX6DDCTGA, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.107141759 (118h10m19.044800248s) [noise=true , address=104.36.82.23:4474 , piece_id=PLEKQKMEQKQDW72BNLLY4JOOPFI6CPNNGT74M5EFFNDQQFWELXUQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.116502399 (118h10m19.054160898s) [noise=true , address=45.91.116.7:4026 , piece_id=CB6TYC4A6DQOFFKVFPNMF5MH5MGMJHU5Z25B4QXQJI5ETDWLQJJQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.153680253 (118h10m19.091338752s) [noise=true , address=104.36.82.9:4019 , piece_id=SBCPU6XTMB4CRRTFKDBYFQQ2XZZNE3I6AF7VVSGYC356SPQNVHUQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.210191177 (118h10m19.147849686s) [noise=true , address=104.36.82.36:4625 , piece_id=FCK7QGPSKJ7VDK4D7BVNF44BRPQCQS5V5ICAJHELAQHDBSQV5R3Q, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.219317350 (118h10m19.156975849s) [noise=true , address=104.36.82.55:4206 , piece_id=KSAKYO6MALVA5REEKKJZ7MGX7YIRC2PDI5PNHFHRASPBD47V5SOQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:31.340384557 (118h10m21.278043046s) [noise=true , address=104.36.82.59:4326 , piece_id=CS7FVO3IRXKZFB2EZM2HQBVEVM4XTTFI5Y7TH3NJZGIYMREVY7UQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:32.464238230 (118h10m22.401896729s) [noise=true , address=104.36.82.13:4193 , piece_id=WJA72IWFD3TUA2WPGJGB6A7AHJXI4PGOK62YDMKSMDPUAVHUHW4Q, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:08:26.041033880 (118h11m15.978692379s) [noise=true , address=193.149.164.143:4695 , piece_id=3QTYKOTBCOJ4CRM76FR2XKONKQVYTQQMPTRH3E6MYJR4ODBPY3AQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: ecclient: upload failed (node:12TZfgz24gnZr4UryzEKw6gu94njnmCvafX5X64MKFSJqaUWSVh, address:193.149.164.143:4695): protocol: write tcp 188.166.225.170:43298->193.149.164.143:4695: use of closed network connection; write tcp 188.166.225.170:43298->193.149.164.143:4695: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43298->193.149.164.143:4695: use of closed network connection Aug 4 16:08:26.041079005 (118h11m15.978737504s) [noise=true , address=104.36.82.44:4886 , piece_id=Q3BIO4GNMY64IWXC7VZ5BAYKYRJJ3GKFV7Z4IOG7SP62IUJHXN2A, seg_pos={0 0}, upload=113540]: Done uploading piece. err: ecclient: upload failed (node:1n2cXhqd5EWbNPTXem83jQUjYq5ZaGKBidbVCQzxYfR3rWddGR, address:104.36.82.44:4886): protocol: write tcp 188.166.225.170:38030->104.36.82.44:4886: use of closed network connection; write tcp 188.166.225.170:38030->104.36.82.44:4886: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38030->104.36.82.44:4886: use of closed network connection Aug 4 16:08:26.041128580 (118h11m15.97878711s) [noise=true , address=104.36.82.39:4728 , piece_id=ZCL6YOUXY4P6HTYUTTILZR4TLG44CDTXWLOJJWOHPFSORA2IBCBA, seg_pos={0 0}, upload=113540]: Done uploading piece. err: ecclient: upload failed (node:1YoLzpb4mMYZ8TsdxnMocekZZ1gLfTrdCM8mxTDj4tY6khZGvi, address:104.36.82.39:4728): protocol: write tcp 188.166.225.170:34326->104.36.82.39:4728: use of closed network connection; write tcp 188.166.225.170:34326->104.36.82.39:4728: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34326->104.36.82.39:4728: use of closed network connection Aug 4 16:08:26.041195678 (118h11m15.978854187s) [noise=true , address=104.36.82.45:4930 , piece_id=LBMDUGYMQI7PBBGEOT22S7OGNVRIYS66USSYFAZOSGI445GWKCYA, seg_pos={0 0}, upload=113540]: Done uploading piece. err: ecclient: upload failed (node:12qaYFSpezuLAc5zJfAYruzad6nTpYFDpog7bmKMVTA8MVm76NB, address:104.36.82.45:4930): protocol: expected piece hash; context deadline exceeded; EOF Aug 4 16:08:26.041203374 (118h11m15.978861913s) [noise=true , address=104.36.82.42:4823 , piece_id=66Q6G7YBCVUEAHU3Y5LHPR5MIT7H7NQWI2TWWNHKZT5FYS4DNQBA, seg_pos={0 0}, upload=113540]: Done uploading piece. err: ecclient: upload failed (node:1fkA5DEj9RqsR6N85z7azvujy8tPkUhB5GCqBpUmFb4vvwbd8y, address:104.36.82.42:4823): protocol: write tcp 188.166.225.170:37230->104.36.82.42:4823: use of closed network connection; write tcp 188.166.225.170:37230->104.36.82.42:4823: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37230->104.36.82.42:4823: use of closed network connection Aug 4 16:08:26.041258960 (118h11m15.978917489s) [seg_pos={0 0}, upload=113540]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Aug 4 16:08:26.274088873 (118h11m16.211747372s) [upload=113540]: Done uploading segments... Aug 4 16:08:26.274113650 (118h11m16.211772139s) [upload=113540]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 6.943285208s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 6.65841164s FAIL (2025-08-04 16:16:33 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.658443351s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [...X.............................X..........................] duration: 1.921060484s FAIL (2025-08-04 16:16:32 UTC) orbiter warning: Operation: DownloadObject uplink: object not found ("orbiter_us_0/1K") Duration: 1.921087407s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 6.449387757s FAIL (2025-08-04 16:16:31 UTC) orbiter warning: Operation: Commit uplink: failed to upload enough pieces (needed at least 14 but got 12) Duration: 6.449418034s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 1m0.001534142s FAIL (2025-08-04 16:16:26 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.001561443s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [..................................................XX.X.XX.XX] duration: 30.00059886s FAIL (2025-08-04 16:16:20 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.000610621s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 13.602880519s FAIL (2025-08-04 16:16:16 UTC) orbiter warning: Operation: Commit uplink: failed to upload enough pieces (needed at least 14 but got 12) Duration: 13.602906819s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [....X.............................X.........................] duration: 1.913710023s FAIL (2025-08-04 16:16:09 UTC) orbiter warning: Operation: DownloadObject uplink: object not found ("orbiter_us_0/1K") Duration: 1.913739939s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [....................................................XX.....X] duration: 30.232261719s FAIL (2025-08-04 16:16:01 UTC) orbiter warning: Operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 1773 (118h17m27.941760272s) [upload=113655]: Done uploading segments... Aug 4 16:14:38.004119036 (118h17m27.941777535s) [upload=113655]: Upload finished. err: <nil> Aug 4 16:15:32.302616649 (118h18m22.240275148s) [upload=113669]: Starting upload Aug 4 16:15:32.302693065 (118h18m22.240351564s) [upload=113669]: Done starting upload Aug 4 16:15:32.302798066 (118h18m22.240456566s) [upload=113669]: Uploading segments... Aug 4 16:15:32.303449322 (118h18m22.241107821s) [upload=113669]: Got next segment. Inline: false Aug 4 16:15:32.303543322 (118h18m22.241201821s) [upload=113669]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 4 16:15:32.303549073 (118h18m22.241207572s) [upload=113669]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 4 16:15:32.539789831 (118h18m22.47744833s) [seg_pos={0 0}, upload=113669]: Begin upload segment... Aug 4 16:15:32.539839626 (118h18m22.477498125s) [seg_pos={0 0}, upload=113669]: Done begin upload segment. Aug 4 16:15:32.539860957 (118h18m22.477519456s) [noise=true , address=77.90.31.19:4843 , piece_id=FVR7EP4XTKJNRYKRFUIV4Y2DBYI67FJ3EHKZVJSLOGRVM7S6M3KQ, seg_pos={0 0}, upload=113669]: Uploading piece... Aug 4 16:15:32.539899040 (118h18m22.477557529s) [noise=true , address=45.91.116.4:4940 , piece_id=RGXORMHLSXGPZ2OE7U7FMBLAM6FC3TQ2GPWUY6ER2OXGFE7SQYHA, seg_pos={0 0}, upload=113669]: Uploading piece... Aug 4 16:15:32.540296541 (118h18m22.47795504s) [noise=true , address=77.90.31.16:4735 , piece_id=2SAWIJVGRQNDE35L2KQZIBLAYAR56YRJ7TQUBWFGHPGU2SZUNQHA, seg_pos={0 0}, upload=113669]: Uploading piece... Aug 4 16:15:32.540452799 (118h18m22.478111298s) [noise=true , address=77.90.31.13:4627 , piece_id=KSEAMM2S7WQJTIA7W62C2OLYY6IYNEK6IJEY5YH3SQRII3FKTH3Q, seg_pos={0 0}, upload=113669]: Uploading piece... Aug 4 16:15:32.540615701 (118h18m22.47827419s) [noise=true , address=45.91.116.7:4011 , piece_id=WEXVVFXFV6P733TCS5KGBVTLSNCBUDV5BYDRPNHCJ6YXJ5YE66HA, seg_pos={0 0}, upload=113669]: Uploading piece... Aug 4 16:15:32.540829069 (118h18m22.478487568s) [noise=true , address=104.36.82.11:4113 , piece_id=SNHPZDFGMXWTNFPYU6RE3W2LFNTXHJZPA5QZTYZ4SEXCEMLKSDTQ, seg_pos={0 0}, upload=113669]: Uploading piece... Aug 4 16:15:32.540895115 (118h18m22.478553614s) [noise=true , address=104.36.82.9:4006 , piece_id=U2EDQZHMXT6AFVN5SRGFZUCO57ILACJP3VDCVULZBX6WDSV4XW4Q, seg_pos={0 0}, upload=113669]: Uploading piece... Aug 4 16:15:32.541147618 (118h18m22.478806117s) [noise=true , address=45.91.116.10:4081 , piece_id=WIW36M6BBTKUKOABD2GLECEORY7UGMOIJIIYAGPHPGOCLTOFHWBA, seg_pos={0 0}, upload=113669]: Uploading piece... Aug 4 16:15:32.541151206 (118h18m22.478809705s) [noise=true , address=45.91.116.5:4297 , piece_id=HQPA4GTU6XJ4ZHV7CSBZOLWGS4F3RRDSQU7QZ26IMYXJIDONMTWA, seg_pos={0 0}, upload=113669]: Uploading piece... Aug 4 16:15:32.541986113 (118h18m22.479644602s) [noise=true , address=77.90.31.14:4661 , piece_id=BKOCMYQV56DZXRHIHDY4OYRHPF5IVVQJRDDUO5JQXF2FBQQYAGCQ, seg_pos={0 0}, upload=113669]: Uploading piece... Aug 4 16:15:32.542171167 (118h18m22.479829666s) [noise=true , address=77.90.31.10:4519 , piece_id=BB72QDQ7SB3BPN365GK2AGZBLRRRONM2VYMACVN33GVMDM6HIWGQ, seg_pos={0 0}, upload=113669]: Uploading piece... Aug 4 16:15:32.542309612 (118h18m22.479968111s) [noise=true , address=77.90.31.15:4698 , piece_id=OITYF76UF6L4FDEJNIPZXU2PWNPL4ARVHCH74UVIT6EBBSYQDZ6Q, seg_pos={0 0}, upload=113669]: Uploading piece... Aug 4 16:15:32.542643841 (118h18m22.480302341s) [noise=true , address=104.36.82.15:4259 , piece_id=ZPBF63TYCMS75EM37ODUHINUXYDEC4V7BSOO3ZF7SEXBEL5SFQWA, seg_pos={0 0}, upload=113669]: Uploading piece... Aug 4 16:15:32.543281982 (118h18m22.480940471s) [noise=true , address=77.90.31.10:4517 , piece_id=6IA36O5LXNTPZGU5W72J5BJMI346WVQEKVDPRUCJMJWQNBACDIOA, seg_pos={0 0}, upload=113669]: Uploading piece... Aug 4 16:15:32.544607649 (118h18m22.482266138s) [noise=true , address=45.91.116.12:4227 , piece_id=FYMNWNR7QAIDRA7ENCFDIWDIDHVIN6SCP3DTJPNKV5YIORDG36GA, seg_pos={0 0}, upload=113669]: Uploading piece... Aug 4 16:15:32.545301176 (118h18m22.482959665s) [noise=true , address=77.90.31.11:4553 , piece_id=ZZPOA5Q5CK6MKYUXLEVJBBAAWVBA7B4CFB5546JJABS6ILPEUAKA, seg_pos={0 0}, upload=113669]: Uploading piece... Aug 4 16:15:34.275056410 (118h18m24.212714909s) [upload=113669]: Next returned nil segment Aug 4 16:15:34.275073813 (118h18m24.212732302s) [upload=113669]: Waiting for error group managing segments... Aug 4 16:15:34.554758063 (118h18m24.492416562s) [noise=true , address=77.90.31.15:4698 , piece_id=OITYF76UF6L4FDEJNIPZXU2PWNPL4ARVHCH74UVIT6EBBSYQDZ6Q, seg_pos={0 0}, upload=113669]: Done uploading piece. err: <nil> Aug 4 16:15:34.556381058 (118h18m24.494039547s) [noise=true , address=77.90.31.14:4661 , piece_id=BKOCMYQV56DZXRHIHDY4OYRHPF5IVVQJRDDUO5JQXF2FBQQYAGCQ, seg_pos={0 0}, upload=113669]: Done uploading piece. err: <nil> Aug 4 16:15:34.559542938 (118h18m24.497201448s) [noise=true , address=45.91.116.5:4297 , piece_id=HQPA4GTU6XJ4ZHV7CSBZOLWGS4F3RRDSQU7QZ26IMYXJIDONMTWA, seg_pos={0 0}, upload=113669]: Done uploading piece. err: <nil> Aug 4 16:15:34.574012413 (118h18m24.511670912s) [noise=true , address=104.36.82.11:4113 , piece_id=SNHPZDFGMXWTNFPYU6RE3W2LFNTXHJZPA5QZTYZ4SEXCEMLKSDTQ, seg_pos={0 0}, upload=113669]: Done uploading piece. err: <nil> Aug 4 16:15:34.591923804 (118h18m24.529582303s) [noise=true , address=77.90.31.10:4519 , piece_id=BB72QDQ7SB3BPN365GK2AGZBLRRRONM2VYMACVN33GVMDM6HIWGQ, seg_pos={0 0}, upload=113669]: Done uploading piece. err: <nil> Aug 4 16:15:34.740603818 (118h18m24.678262317s) [noise=true , address=104.36.82.9:4006 , piece_id=U2EDQZHMXT6AFVN5SRGFZUCO57ILACJP3VDCVULZBX6WDSV4XW4Q, seg_pos={0 0}, upload=113669]: Done uploading piece. err: <nil> Aug 4 16:15:34.770186750 (118h18m24.707845249s) [noise=true , address=45.91.116.7:4011 , piece_id=WEXVVFXFV6P733TCS5KGBVTLSNCBUDV5BYDRPNHCJ6YXJ5YE66HA, seg_pos={0 0}, upload=113669]: Done uploading piece. err: <nil> Aug 4 16:15:34.877238737 (118h18m24.814897236s) [noise=true , address=77.90.31.10:4517 , piece_id=6IA36O5LXNTPZGU5W72J5BJMI346WVQEKVDPRUCJMJWQNBACDIOA, seg_pos={0 0}, upload=113669]: Done uploading piece. err: <nil> Aug 4 16:15:35.259555828 (118h18m25.197214328s) [noise=true , address=77.90.31.11:4553 , piece_id=ZZPOA5Q5CK6MKYUXLEVJBBAAWVBA7B4CFB5546JJABS6ILPEUAKA, seg_pos={0 0}, upload=113669]: Done uploading piece. err: <nil> Aug 4 16:15:35.556139422 (118h18m25.493797921s) [noise=true , address=45.91.116.4:4940 , piece_id=RGXORMHLSXGPZ2OE7U7FMBLAM6FC3TQ2GPWUY6ER2OXGFE7SQYHA, seg_pos={0 0}, upload=113669]: Done uploading piece. err: <nil> Aug 4 16:15:35.593880413 (118h18m25.531538912s) [noise=true , address=45.91.116.12:4227 , piece_id=FYMNWNR7QAIDRA7ENCFDIWDIDHVIN6SCP3DTJPNKV5YIORDG36GA, seg_pos={0 0}, upload=113669]: Done uploading piece. err: <nil> Aug 4 16:15:36.055685242 (118h18m25.993343741s) [noise=true , address=104.36.82.15:4259 , piece_id=ZPBF63TYCMS75EM37ODUHINUXYDEC4V7BSOO3ZF7SEXBEL5SFQWA, seg_pos={0 0}, upload=113669]: Done uploading piece. err: <nil> Aug 4 16:16:01.655205548 (118h18m51.592864057s) [noise=true , address=45.91.116.10:4081 , piece_id=WIW36M6BBTKUKOABD2GLECEORY7UGMOIJIIYAGPHPGOCLTOFHWBA, seg_pos={0 0}, upload=113669]: Done uploading piece. err: ecclient: upload failed (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG, address:45.91.116.10:4081): protocol: write tcp 188.166.225.170:35324->45.91.116.10:4081: use of closed network connection; write tcp 188.166.225.170:35324->45.91.116.10:4081: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35324->45.91.116.10:4081: use of closed network connection Aug 4 16:16:01.655391584 (118h18m51.593050093s) [noise=true , address=77.90.31.13:4627 , piece_id=KSEAMM2S7WQJTIA7W62C2OLYY6IYNEK6IJEY5YH3SQRII3FKTH3Q, seg_pos={0 0}, upload=113669]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: write tcp 188.166.225.170:55624->77.90.31.13:4627: use of closed network connection; write tcp 188.166.225.170:55624->77.90.31.13:4627: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55624->77.90.31.13:4627: use of closed network connection Aug 4 16:16:01.655479321 (118h18m51.59313782s) [noise=true , address=77.90.31.19:4843 , piece_id=FVR7EP4XTKJNRYKRFUIV4Y2DBYI67FJ3EHKZVJSLOGRVM7S6M3KQ, seg_pos={0 0}, upload=113669]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: write tcp 188.166.225.170:42918->77.90.31.19:4843: use of closed network connection; write tcp 188.166.225.170:42918->77.90.31.19:4843: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42918->77.90.31.19:4843: use of closed network connection Aug 4 16:16:01.655511203 (118h18m51.593169702s) [noise=true , address=77.90.31.16:4735 , piece_id=2SAWIJVGRQNDE35L2KQZIBLAYAR56YRJ7TQUBWFGHPGU2SZUNQHA, seg_pos={0 0}, upload=113669]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 188.166.225.170:44366->77.90.31.16:4735: use of closed network connection; write tcp 188.166.225.170:44366->77.90.31.16:4735: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44366->77.90.31.16:4735: use of closed network connection Aug 4 16:16:01.655543364 (118h18m51.593201914s) [seg_pos={0 0}, upload=113669]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Aug 4 16:16:01.885476855 (118h18m51.823135364s) [upload=113669]: Done uploading segments... Aug 4 16:16:01.885512302 (118h18m51.823170811s) [upload=113669]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.232281627s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [...................................................XX.X.XX.X] duration: 30.001054741s FAIL (2025-08-04 16:15:50 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.001076233s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 8.66866917s FAIL (2025-08-04 16:15:38 UTC) orbiter warning: Operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 4.36.82.36:4625 , piece_id=FCK7QGPSKJ7VDK4D7BVNF44BRPQCQS5V5ICAJHELAQHDBSQV5R3Q, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.146274823 (118h10m17.083933322s) [noise=true , address=45.91.116.7:4026 , piece_id=CB6TYC4A6DQOFFKVFPNMF5MH5MGMJHU5Z25B4QXQJI5ETDWLQJJQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.146672965 (118h10m17.084331464s) [noise=true , address=104.36.82.16:4321 , piece_id=42MZPFPEBQNQRJUHQJLZQP2UWGDV2OAXT6Z4KTWPPWLALISASN5Q, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.146726166 (118h10m17.084384665s) [noise=true , address=104.36.82.55:4206 , piece_id=KSAKYO6MALVA5REEKKJZ7MGX7YIRC2PDI5PNHFHRASPBD47V5SOQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.147136371 (118h10m17.08479488s) [noise=true , address=104.36.82.42:4823 , piece_id=66Q6G7YBCVUEAHU3Y5LHPR5MIT7H7NQWI2TWWNHKZT5FYS4DNQBA, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.147479628 (118h10m17.085138127s) [noise=true , address=193.149.164.143:4695 , piece_id=3QTYKOTBCOJ4CRM76FR2XKONKQVYTQQMPTRH3E6MYJR4ODBPY3AQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.147821592 (118h10m17.085480091s) [noise=true , address=104.36.82.62:4411 , piece_id=CIJJ4SMWKFDOIZZ5UEYIXFZHUNAZKRMM2P36IJIT5BHUX5STDZ4A, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.148134220 (118h10m17.085792719s) [noise=true , address=104.36.82.55:4224 , piece_id=NJYMQ2467DXHUEC6ELXITFJF6WWKM4XFEONRJVRF6QSJBDEMC7JA, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.148548573 (118h10m17.086207072s) [noise=true , address=104.36.82.13:4193 , piece_id=WJA72IWFD3TUA2WPGJGB6A7AHJXI4PGOK62YDMKSMDPUAVHUHW4Q, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.148889576 (118h10m17.086548085s) [noise=true , address=104.36.82.39:4728 , piece_id=ZCL6YOUXY4P6HTYUTTILZR4TLG44CDTXWLOJJWOHPFSORA2IBCBA, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.149205931 (118h10m17.08686443s) [noise=true , address=104.36.82.59:4326 , piece_id=CS7FVO3IRXKZFB2EZM2HQBVEVM4XTTFI5Y7TH3NJZGIYMREVY7UQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.149509752 (118h10m17.087168261s) [noise=true , address=104.36.82.45:4930 , piece_id=LBMDUGYMQI7PBBGEOT22S7OGNVRIYS66USSYFAZOSGI445GWKCYA, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.149866545 (118h10m17.087525044s) [noise=true , address=104.36.82.23:4474 , piece_id=PLEKQKMEQKQDW72BNLLY4JOOPFI6CPNNGT74M5EFFNDQQFWELXUQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.150193601 (118h10m17.08785211s) [noise=true , address=104.36.82.13:4212 , piece_id=ER6VIEGFSG2F7542M5P66RS3QAUQYV7BWPQ4YOX55Q3XX6DDCTGA, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.150515557 (118h10m17.088174066s) [noise=true , address=104.36.82.46:4485 , piece_id=P4FMDR5NCQOLKSXQBLE7I7BPEZJ2URAPPVNKOR53ZPROBCE4YAJQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.150935149 (118h10m17.088593648s) [noise=true , address=104.36.82.54:4181 , piece_id=AHW774W42KLBATPDVRQRKH4MQWV5YP2XEEUY6VASDMYJ2BKXCS2Q, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.151249151 (118h10m17.08890765s) [noise=true , address=104.36.82.44:4886 , piece_id=Q3BIO4GNMY64IWXC7VZ5BAYKYRJJ3GKFV7Z4IOG7SP62IUJHXN2A, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.151276363 (118h10m17.088934862s) [noise=true , address=104.36.82.9:4019 , piece_id=SBCPU6XTMB4CRRTFKDBYFQQ2XZZNE3I6AF7VVSGYC356SPQNVHUQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:28.640156348 (118h10m18.577814857s) [upload=113540]: Next returned nil segment Aug 4 16:07:28.640223306 (118h10m18.577881825s) [upload=113540]: Waiting for error group managing segments... Aug 4 16:07:29.062127352 (118h10m18.999785841s) [noise=true , address=104.36.82.16:4321 , piece_id=42MZPFPEBQNQRJUHQJLZQP2UWGDV2OAXT6Z4KTWPPWLALISASN5Q, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.080565981 (118h10m19.01822448s) [noise=true , address=104.36.82.54:4181 , piece_id=AHW774W42KLBATPDVRQRKH4MQWV5YP2XEEUY6VASDMYJ2BKXCS2Q, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.081329852 (118h10m19.018988352s) [noise=true , address=104.36.82.62:4411 , piece_id=CIJJ4SMWKFDOIZZ5UEYIXFZHUNAZKRMM2P36IJIT5BHUX5STDZ4A, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.081846201 (118h10m19.0195047s) [noise=true , address=104.36.82.55:4224 , piece_id=NJYMQ2467DXHUEC6ELXITFJF6WWKM4XFEONRJVRF6QSJBDEMC7JA, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.091017397 (118h10m19.028675896s) [noise=true , address=104.36.82.46:4485 , piece_id=P4FMDR5NCQOLKSXQBLE7I7BPEZJ2URAPPVNKOR53ZPROBCE4YAJQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.105092769 (118h10m19.042751268s) [noise=true , address=104.36.82.13:4212 , piece_id=ER6VIEGFSG2F7542M5P66RS3QAUQYV7BWPQ4YOX55Q3XX6DDCTGA, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.107141759 (118h10m19.044800248s) [noise=true , address=104.36.82.23:4474 , piece_id=PLEKQKMEQKQDW72BNLLY4JOOPFI6CPNNGT74M5EFFNDQQFWELXUQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.116502399 (118h10m19.054160898s) [noise=true , address=45.91.116.7:4026 , piece_id=CB6TYC4A6DQOFFKVFPNMF5MH5MGMJHU5Z25B4QXQJI5ETDWLQJJQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.153680253 (118h10m19.091338752s) [noise=true , address=104.36.82.9:4019 , piece_id=SBCPU6XTMB4CRRTFKDBYFQQ2XZZNE3I6AF7VVSGYC356SPQNVHUQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.210191177 (118h10m19.147849686s) [noise=true , address=104.36.82.36:4625 , piece_id=FCK7QGPSKJ7VDK4D7BVNF44BRPQCQS5V5ICAJHELAQHDBSQV5R3Q, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.219317350 (118h10m19.156975849s) [noise=true , address=104.36.82.55:4206 , piece_id=KSAKYO6MALVA5REEKKJZ7MGX7YIRC2PDI5PNHFHRASPBD47V5SOQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:31.340384557 (118h10m21.278043046s) [noise=true , address=104.36.82.59:4326 , piece_id=CS7FVO3IRXKZFB2EZM2HQBVEVM4XTTFI5Y7TH3NJZGIYMREVY7UQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:32.464238230 (118h10m22.401896729s) [noise=true , address=104.36.82.13:4193 , piece_id=WJA72IWFD3TUA2WPGJGB6A7AHJXI4PGOK62YDMKSMDPUAVHUHW4Q, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:08:26.041033880 (118h11m15.978692379s) [noise=true , address=193.149.164.143:4695 , piece_id=3QTYKOTBCOJ4CRM76FR2XKONKQVYTQQMPTRH3E6MYJR4ODBPY3AQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: ecclient: upload failed (node:12TZfgz24gnZr4UryzEKw6gu94njnmCvafX5X64MKFSJqaUWSVh, address:193.149.164.143:4695): protocol: write tcp 188.166.225.170:43298->193.149.164.143:4695: use of closed network connection; write tcp 188.166.225.170:43298->193.149.164.143:4695: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43298->193.149.164.143:4695: use of closed network connection Aug 4 16:08:26.041079005 (118h11m15.978737504s) [noise=true , address=104.36.82.44:4886 , piece_id=Q3BIO4GNMY64IWXC7VZ5BAYKYRJJ3GKFV7Z4IOG7SP62IUJHXN2A, seg_pos={0 0}, upload=113540]: Done uploading piece. err: ecclient: upload failed (node:1n2cXhqd5EWbNPTXem83jQUjYq5ZaGKBidbVCQzxYfR3rWddGR, address:104.36.82.44:4886): protocol: write tcp 188.166.225.170:38030->104.36.82.44:4886: use of closed network connection; write tcp 188.166.225.170:38030->104.36.82.44:4886: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38030->104.36.82.44:4886: use of closed network connection Aug 4 16:08:26.041128580 (118h11m15.97878711s) [noise=true , address=104.36.82.39:4728 , piece_id=ZCL6YOUXY4P6HTYUTTILZR4TLG44CDTXWLOJJWOHPFSORA2IBCBA, seg_pos={0 0}, upload=113540]: Done uploading piece. err: ecclient: upload failed (node:1YoLzpb4mMYZ8TsdxnMocekZZ1gLfTrdCM8mxTDj4tY6khZGvi, address:104.36.82.39:4728): protocol: write tcp 188.166.225.170:34326->104.36.82.39:4728: use of closed network connection; write tcp 188.166.225.170:34326->104.36.82.39:4728: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34326->104.36.82.39:4728: use of closed network connection Aug 4 16:08:26.041195678 (118h11m15.978854187s) [noise=true , address=104.36.82.45:4930 , piece_id=LBMDUGYMQI7PBBGEOT22S7OGNVRIYS66USSYFAZOSGI445GWKCYA, seg_pos={0 0}, upload=113540]: Done uploading piece. err: ecclient: upload failed (node:12qaYFSpezuLAc5zJfAYruzad6nTpYFDpog7bmKMVTA8MVm76NB, address:104.36.82.45:4930): protocol: expected piece hash; context deadline exceeded; EOF Aug 4 16:08:26.041203374 (118h11m15.978861913s) [noise=true , address=104.36.82.42:4823 , piece_id=66Q6G7YBCVUEAHU3Y5LHPR5MIT7H7NQWI2TWWNHKZT5FYS4DNQBA, seg_pos={0 0}, upload=113540]: Done uploading piece. err: ecclient: upload failed (node:1fkA5DEj9RqsR6N85z7azvujy8tPkUhB5GCqBpUmFb4vvwbd8y, address:104.36.82.42:4823): protocol: write tcp 188.166.225.170:37230->104.36.82.42:4823: use of closed network connection; write tcp 188.166.225.170:37230->104.36.82.42:4823: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37230->104.36.82.42:4823: use of closed network connection Aug 4 16:08:26.041258960 (118h11m15.978917489s) [seg_pos={0 0}, upload=113540]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Aug 4 16:08:26.274088873 (118h11m16.211747372s) [upload=113540]: Done uploading segments... Aug 4 16:08:26.274113650 (118h11m16.211772139s) [upload=113540]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 8.668696883s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [.....X.............................X........................] duration: 1.866835605s FAIL (2025-08-04 16:15:22 UTC) orbiter warning: Operation: DownloadObject uplink: object not found ("orbiter_us_0/1K") Duration: 1.866866725s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 15.85149131s FAIL (2025-08-04 16:15:20 UTC) orbiter warning: Operation: Commit uplink: failed to upload enough pieces (needed at least 14 but got 12) Duration: 15.851520617s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....................................................XX.....] duration: 5.666796451s FAIL (2025-08-04 16:14:40 UTC) orbiter warning: Operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== 16569229 (118h13m13.754227758s) [upload=113588]: Done starting upload Aug 4 16:10:23.816697144 (118h13m13.754355643s) [upload=113588]: Uploading segments... Aug 4 16:10:23.816823747 (118h13m13.754482246s) [upload=113588]: Got next segment. Inline: false Aug 4 16:10:23.816875515 (118h13m13.754534014s) [upload=113588]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 4 16:10:23.816882549 (118h13m13.754541068s) [upload=113588]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 4 16:10:24.051702429 (118h13m13.989360928s) [seg_pos={0 0}, upload=113588]: Begin upload segment... Aug 4 16:10:24.051745952 (118h13m13.989404451s) [seg_pos={0 0}, upload=113588]: Done begin upload segment. Aug 4 16:10:24.051767884 (118h13m13.989426383s) [noise=true , address=104.36.82.16:4293 , piece_id=UUQ6WYMZMYI6PYRGK5IK2OE6CX2AHAW45ZRCPA7LK3X2IIIM3ZJA, seg_pos={0 0}, upload=113588]: Uploading piece... Aug 4 16:10:24.051841024 (118h13m13.989499533s) [noise=true , address=104.36.82.7:4910 , piece_id=ZBYQ4SELARIGNS5Q3FBOOS3QIRVQ23VDCHMDK3TBLU5E7VREUPBA, seg_pos={0 0}, upload=113588]: Uploading piece... Aug 4 16:10:24.052106963 (118h13m13.989765462s) [noise=true , address=77.90.31.18:4805 , piece_id=OW2WLOK6KW2WUGVFEDEHMTUBSSUGGH4S5NQTYXKNOVMDXRYWHSEQ, seg_pos={0 0}, upload=113588]: Uploading piece... Aug 4 16:10:24.052435702 (118h13m13.990094201s) [noise=true , address=77.90.31.21:4877 , piece_id=GNPOUGIJ42Y47VOZ5EMFVIDEFAY5B3KQQPEZ3LS7XYBDTXBZCLKQ, seg_pos={0 0}, upload=113588]: Uploading piece... Aug 4 16:10:24.052526446 (118h13m13.990184945s) [noise=true , address=45.91.116.12:4225 , piece_id=3M2NI24IGXUHSDASHFYXDR2QPTNGEAQFT22AHIUAHVK2JVJNDQXQ, seg_pos={0 0}, upload=113588]: Uploading piece... Aug 4 16:10:24.052684418 (118h13m13.990342907s) [noise=true , address=45.91.116.10:4082 , piece_id=GKHLII3XXM72YPTIZF5ZFZESHXZQNB56E3CVZFAEXJC2WX6URD5A, seg_pos={0 0}, upload=113588]: Uploading piece... Aug 4 16:10:24.052926030 (118h13m13.990584529s) [noise=true , address=104.36.82.10:4077 , piece_id=UNAQGJCBMXKJIOAHLHZSG4RYC2UDR3B5OXAOUTI7FD5CXLFXR2UQ, seg_pos={0 0}, upload=113588]: Uploading piece... Aug 4 16:10:24.052950747 (118h13m13.990609246s) [noise=true , address=45.91.116.4:4938 , piece_id=SHP5IWA4HZTAAECBH2VRW525XCY6BVWIJOHE3U7WAD4ISJ63WUVA, seg_pos={0 0}, upload=113588]: Uploading piece... Aug 4 16:10:24.053160659 (118h13m13.990819158s) [noise=true , address=45.91.116.11:4119 , piece_id=WBFTLEUFEXKAR2U5SZLSOHPZEGV3QNWDLVJBL36XP2GEAK5L7RUA, seg_pos={0 0}, upload=113588]: Uploading piece... Aug 4 16:10:24.053383786 (118h13m13.991042285s) [noise=true , address=77.90.31.12:4589 , piece_id=BFYZ6N2AZEFVYNS4EQYPVKNLUIANMXZ7MPDSGYA5LB2XP2REAJUQ, seg_pos={0 0}, upload=113588]: Uploading piece... Aug 4 16:10:24.053410627 (118h13m13.991069126s) [noise=true , address=77.90.31.16:4735 , piece_id=XET5ROIUKXDV7MNQBYU2JPXLWHH74RD525PMYWC7VS7GZABL5P7A, seg_pos={0 0}, upload=113588]: Uploading piece... Aug 4 16:10:24.053613205 (118h13m13.991271704s) [noise=true , address=77.90.31.13:4627 , piece_id=E5LXNHOSI5URSZV7OBEWIQ75CEQWXDN2DLW4N2PUSY6MFHCNQ7YA, seg_pos={0 0}, upload=113588]: Uploading piece... Aug 4 16:10:24.053834499 (118h13m13.991492998s) [noise=true , address=104.36.82.6:4876 , piece_id=EWMXECZHZILAGIYJ5QJGYIGZSN4XP4LAD2PFY5CQHVARTAGTU5UA, seg_pos={0 0}, upload=113588]: Uploading piece... Aug 4 16:10:24.053873513 (118h13m13.991532012s) [noise=true , address=45.91.116.7:4011 , piece_id=57KLGTVQH77IOCAQPMQA66CPZPLEDKUIO4JKMUP6BOD4XSMPB5BQ, seg_pos={0 0}, upload=113588]: Uploading piece... Aug 4 16:10:24.054251246 (118h13m13.991909755s) [noise=true , address=77.90.31.10:4519 , piece_id=7HTU6S7JCLTAA7MMAYQBQP6F2PWHU3X7BU2NPHUUT2Q2MPFTLRWA, seg_pos={0 0}, upload=113588]: Uploading piece... Aug 4 16:10:24.054636593 (118h13m13.992295092s) [noise=true , address=45.91.116.8:4048 , piece_id=JGBCOCNYT4IZUJ6BCI3TEX7E46PYWQSS6HHM67YCOXQ3W5L4H5OA, seg_pos={0 0}, upload=113588]: Uploading piece... Aug 4 16:10:25.788117314 (118h13m15.725775813s) [upload=113588]: Next returned nil segment Aug 4 16:10:25.788134928 (118h13m15.725793427s) [upload=113588]: Waiting for error group managing segments... Aug 4 16:10:26.080627835 (118h13m16.018286335s) [noise=true , address=77.90.31.16:4735 , piece_id=XET5ROIUKXDV7MNQBYU2JPXLWHH74RD525PMYWC7VS7GZABL5P7A, seg_pos={0 0}, upload=113588]: Done uploading piece. err: <nil> Aug 4 16:10:26.081237151 (118h13m16.01889565s) [noise=true , address=77.90.31.21:4877 , piece_id=GNPOUGIJ42Y47VOZ5EMFVIDEFAY5B3KQQPEZ3LS7XYBDTXBZCLKQ, seg_pos={0 0}, upload=113588]: Done uploading piece. err: <nil> Aug 4 16:10:26.082642120 (118h13m16.020300619s) [noise=true , address=77.90.31.18:4805 , piece_id=OW2WLOK6KW2WUGVFEDEHMTUBSSUGGH4S5NQTYXKNOVMDXRYWHSEQ, seg_pos={0 0}, upload=113588]: Done uploading piece. err: <nil> Aug 4 16:10:26.090023893 (118h13m16.027682392s) [noise=true , address=45.91.116.11:4119 , piece_id=WBFTLEUFEXKAR2U5SZLSOHPZEGV3QNWDLVJBL36XP2GEAK5L7RUA, seg_pos={0 0}, upload=113588]: Done uploading piece. err: <nil> Aug 4 16:10:26.092155202 (118h13m16.029813701s) [noise=true , address=45.91.116.12:4225 , piece_id=3M2NI24IGXUHSDASHFYXDR2QPTNGEAQFT22AHIUAHVK2JVJNDQXQ, seg_pos={0 0}, upload=113588]: Done uploading piece. err: <nil> Aug 4 16:10:26.093301104 (118h13m16.030959593s) [noise=true , address=45.91.116.4:4938 , piece_id=SHP5IWA4HZTAAECBH2VRW525XCY6BVWIJOHE3U7WAD4ISJ63WUVA, seg_pos={0 0}, upload=113588]: Done uploading piece. err: <nil> Aug 4 16:10:26.096255166 (118h13m16.033913675s) [noise=true , address=104.36.82.6:4876 , piece_id=EWMXECZHZILAGIYJ5QJGYIGZSN4XP4LAD2PFY5CQHVARTAGTU5UA, seg_pos={0 0}, upload=113588]: Done uploading piece. err: <nil> Aug 4 16:10:26.112849236 (118h13m16.050507735s) [noise=true , address=104.36.82.16:4293 , piece_id=UUQ6WYMZMYI6PYRGK5IK2OE6CX2AHAW45ZRCPA7LK3X2IIIM3ZJA, seg_pos={0 0}, upload=113588]: Done uploading piece. err: <nil> Aug 4 16:10:26.171272489 (118h13m16.108930988s) [noise=true , address=77.90.31.12:4589 , piece_id=BFYZ6N2AZEFVYNS4EQYPVKNLUIANMXZ7MPDSGYA5LB2XP2REAJUQ, seg_pos={0 0}, upload=113588]: Done uploading piece. err: <nil> Aug 4 16:10:28.145791810 (118h13m18.083450309s) [noise=true , address=77.90.31.13:4627 , piece_id=E5LXNHOSI5URSZV7OBEWIQ75CEQWXDN2DLW4N2PUSY6MFHCNQ7YA, seg_pos={0 0}, upload=113588]: Done uploading piece. err: <nil> Aug 4 16:10:31.180407751 (118h13m21.11806625s) [noise=true , address=77.90.31.10:4519 , piece_id=7HTU6S7JCLTAA7MMAYQBQP6F2PWHU3X7BU2NPHUUT2Q2MPFTLRWA, seg_pos={0 0}, upload=113588]: Done uploading piece. err: <nil> Aug 4 16:10:53.161543551 (118h13m43.099202051s) [noise=true , address=104.36.82.7:4910 , piece_id=ZBYQ4SELARIGNS5Q3FBOOS3QIRVQ23VDCHMDK3TBLU5E7VREUPBA, seg_pos={0 0}, upload=113588]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: write tcp 188.166.225.170:52204->104.36.82.7:4910: use of closed network connection; write tcp 188.166.225.170:52204->104.36.82.7:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52204->104.36.82.7:4910: use of closed network connection Aug 4 16:10:53.161605039 (118h13m43.099263528s) [noise=true , address=104.36.82.10:4077 , piece_id=UNAQGJCBMXKJIOAHLHZSG4RYC2UDR3B5OXAOUTI7FD5CXLFXR2UQ, seg_pos={0 0}, upload=113588]: Done uploading piece. err: ecclient: upload failed (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk, address:104.36.82.10:4077): protocol: write tcp 188.166.225.170:41132->104.36.82.10:4077: use of closed network connection; write tcp 188.166.225.170:41132->104.36.82.10:4077: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41132->104.36.82.10:4077: use of closed network connection Aug 4 16:10:53.161771859 (118h13m43.099430348s) [noise=true , address=45.91.116.8:4048 , piece_id=JGBCOCNYT4IZUJ6BCI3TEX7E46PYWQSS6HHM67YCOXQ3W5L4H5OA, seg_pos={0 0}, upload=113588]: Done uploading piece. err: ecclient: upload failed (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, address:45.91.116.8:4048): protocol: write tcp 188.166.225.170:35774->45.91.116.8:4048: use of closed network connection; write tcp 188.166.225.170:35774->45.91.116.8:4048: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35774->45.91.116.8:4048: use of closed network connection Aug 4 16:10:53.161844948 (118h13m43.099503447s) [noise=true , address=45.91.116.7:4011 , piece_id=57KLGTVQH77IOCAQPMQA66CPZPLEDKUIO4JKMUP6BOD4XSMPB5BQ, seg_pos={0 0}, upload=113588]: Done uploading piece. err: ecclient: upload failed (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, address:45.91.116.7:4011): protocol: write tcp 188.166.225.170:50254->45.91.116.7:4011: use of closed network connection; write tcp 188.166.225.170:50254->45.91.116.7:4011: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50254->45.91.116.7:4011: use of closed network connection Aug 4 16:10:53.161881499 (118h13m43.099539998s) [noise=true , address=45.91.116.10:4082 , piece_id=GKHLII3XXM72YPTIZF5ZFZESHXZQNB56E3CVZFAEXJC2WX6URD5A, seg_pos={0 0}, upload=113588]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: read tcp 188.166.225.170:44088->45.91.116.10:4082: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 4 16:10:53.161905614 (118h13m43.099564103s) [seg_pos={0 0}, upload=113588]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Aug 4 16:10:53.443651470 (118h13m43.381309969s) [upload=113588]: Done uploading segments... Aug 4 16:10:53.443675736 (118h13m43.381334235s) [upload=113588]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.666821951s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 30.239822636s FAIL (2025-08-04 16:14:33 UTC) orbiter warning: Operation: Commit uplink: failed to upload enough pieces (needed at least 14 but got 12) Duration: 30.239843856s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 6.766927621s FAIL (2025-08-04 16:14:27 UTC) orbiter warning: Operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 4.36.82.36:4625 , piece_id=FCK7QGPSKJ7VDK4D7BVNF44BRPQCQS5V5ICAJHELAQHDBSQV5R3Q, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.146274823 (118h10m17.083933322s) [noise=true , address=45.91.116.7:4026 , piece_id=CB6TYC4A6DQOFFKVFPNMF5MH5MGMJHU5Z25B4QXQJI5ETDWLQJJQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.146672965 (118h10m17.084331464s) [noise=true , address=104.36.82.16:4321 , piece_id=42MZPFPEBQNQRJUHQJLZQP2UWGDV2OAXT6Z4KTWPPWLALISASN5Q, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.146726166 (118h10m17.084384665s) [noise=true , address=104.36.82.55:4206 , piece_id=KSAKYO6MALVA5REEKKJZ7MGX7YIRC2PDI5PNHFHRASPBD47V5SOQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.147136371 (118h10m17.08479488s) [noise=true , address=104.36.82.42:4823 , piece_id=66Q6G7YBCVUEAHU3Y5LHPR5MIT7H7NQWI2TWWNHKZT5FYS4DNQBA, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.147479628 (118h10m17.085138127s) [noise=true , address=193.149.164.143:4695 , piece_id=3QTYKOTBCOJ4CRM76FR2XKONKQVYTQQMPTRH3E6MYJR4ODBPY3AQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.147821592 (118h10m17.085480091s) [noise=true , address=104.36.82.62:4411 , piece_id=CIJJ4SMWKFDOIZZ5UEYIXFZHUNAZKRMM2P36IJIT5BHUX5STDZ4A, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.148134220 (118h10m17.085792719s) [noise=true , address=104.36.82.55:4224 , piece_id=NJYMQ2467DXHUEC6ELXITFJF6WWKM4XFEONRJVRF6QSJBDEMC7JA, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.148548573 (118h10m17.086207072s) [noise=true , address=104.36.82.13:4193 , piece_id=WJA72IWFD3TUA2WPGJGB6A7AHJXI4PGOK62YDMKSMDPUAVHUHW4Q, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.148889576 (118h10m17.086548085s) [noise=true , address=104.36.82.39:4728 , piece_id=ZCL6YOUXY4P6HTYUTTILZR4TLG44CDTXWLOJJWOHPFSORA2IBCBA, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.149205931 (118h10m17.08686443s) [noise=true , address=104.36.82.59:4326 , piece_id=CS7FVO3IRXKZFB2EZM2HQBVEVM4XTTFI5Y7TH3NJZGIYMREVY7UQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.149509752 (118h10m17.087168261s) [noise=true , address=104.36.82.45:4930 , piece_id=LBMDUGYMQI7PBBGEOT22S7OGNVRIYS66USSYFAZOSGI445GWKCYA, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.149866545 (118h10m17.087525044s) [noise=true , address=104.36.82.23:4474 , piece_id=PLEKQKMEQKQDW72BNLLY4JOOPFI6CPNNGT74M5EFFNDQQFWELXUQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.150193601 (118h10m17.08785211s) [noise=true , address=104.36.82.13:4212 , piece_id=ER6VIEGFSG2F7542M5P66RS3QAUQYV7BWPQ4YOX55Q3XX6DDCTGA, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.150515557 (118h10m17.088174066s) [noise=true , address=104.36.82.46:4485 , piece_id=P4FMDR5NCQOLKSXQBLE7I7BPEZJ2URAPPVNKOR53ZPROBCE4YAJQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.150935149 (118h10m17.088593648s) [noise=true , address=104.36.82.54:4181 , piece_id=AHW774W42KLBATPDVRQRKH4MQWV5YP2XEEUY6VASDMYJ2BKXCS2Q, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.151249151 (118h10m17.08890765s) [noise=true , address=104.36.82.44:4886 , piece_id=Q3BIO4GNMY64IWXC7VZ5BAYKYRJJ3GKFV7Z4IOG7SP62IUJHXN2A, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.151276363 (118h10m17.088934862s) [noise=true , address=104.36.82.9:4019 , piece_id=SBCPU6XTMB4CRRTFKDBYFQQ2XZZNE3I6AF7VVSGYC356SPQNVHUQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:28.640156348 (118h10m18.577814857s) [upload=113540]: Next returned nil segment Aug 4 16:07:28.640223306 (118h10m18.577881825s) [upload=113540]: Waiting for error group managing segments... Aug 4 16:07:29.062127352 (118h10m18.999785841s) [noise=true , address=104.36.82.16:4321 , piece_id=42MZPFPEBQNQRJUHQJLZQP2UWGDV2OAXT6Z4KTWPPWLALISASN5Q, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.080565981 (118h10m19.01822448s) [noise=true , address=104.36.82.54:4181 , piece_id=AHW774W42KLBATPDVRQRKH4MQWV5YP2XEEUY6VASDMYJ2BKXCS2Q, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.081329852 (118h10m19.018988352s) [noise=true , address=104.36.82.62:4411 , piece_id=CIJJ4SMWKFDOIZZ5UEYIXFZHUNAZKRMM2P36IJIT5BHUX5STDZ4A, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.081846201 (118h10m19.0195047s) [noise=true , address=104.36.82.55:4224 , piece_id=NJYMQ2467DXHUEC6ELXITFJF6WWKM4XFEONRJVRF6QSJBDEMC7JA, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.091017397 (118h10m19.028675896s) [noise=true , address=104.36.82.46:4485 , piece_id=P4FMDR5NCQOLKSXQBLE7I7BPEZJ2URAPPVNKOR53ZPROBCE4YAJQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.105092769 (118h10m19.042751268s) [noise=true , address=104.36.82.13:4212 , piece_id=ER6VIEGFSG2F7542M5P66RS3QAUQYV7BWPQ4YOX55Q3XX6DDCTGA, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.107141759 (118h10m19.044800248s) [noise=true , address=104.36.82.23:4474 , piece_id=PLEKQKMEQKQDW72BNLLY4JOOPFI6CPNNGT74M5EFFNDQQFWELXUQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.116502399 (118h10m19.054160898s) [noise=true , address=45.91.116.7:4026 , piece_id=CB6TYC4A6DQOFFKVFPNMF5MH5MGMJHU5Z25B4QXQJI5ETDWLQJJQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.153680253 (118h10m19.091338752s) [noise=true , address=104.36.82.9:4019 , piece_id=SBCPU6XTMB4CRRTFKDBYFQQ2XZZNE3I6AF7VVSGYC356SPQNVHUQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.210191177 (118h10m19.147849686s) [noise=true , address=104.36.82.36:4625 , piece_id=FCK7QGPSKJ7VDK4D7BVNF44BRPQCQS5V5ICAJHELAQHDBSQV5R3Q, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.219317350 (118h10m19.156975849s) [noise=true , address=104.36.82.55:4206 , piece_id=KSAKYO6MALVA5REEKKJZ7MGX7YIRC2PDI5PNHFHRASPBD47V5SOQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:31.340384557 (118h10m21.278043046s) [noise=true , address=104.36.82.59:4326 , piece_id=CS7FVO3IRXKZFB2EZM2HQBVEVM4XTTFI5Y7TH3NJZGIYMREVY7UQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:32.464238230 (118h10m22.401896729s) [noise=true , address=104.36.82.13:4193 , piece_id=WJA72IWFD3TUA2WPGJGB6A7AHJXI4PGOK62YDMKSMDPUAVHUHW4Q, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:08:26.041033880 (118h11m15.978692379s) [noise=true , address=193.149.164.143:4695 , piece_id=3QTYKOTBCOJ4CRM76FR2XKONKQVYTQQMPTRH3E6MYJR4ODBPY3AQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: ecclient: upload failed (node:12TZfgz24gnZr4UryzEKw6gu94njnmCvafX5X64MKFSJqaUWSVh, address:193.149.164.143:4695): protocol: write tcp 188.166.225.170:43298->193.149.164.143:4695: use of closed network connection; write tcp 188.166.225.170:43298->193.149.164.143:4695: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43298->193.149.164.143:4695: use of closed network connection Aug 4 16:08:26.041079005 (118h11m15.978737504s) [noise=true , address=104.36.82.44:4886 , piece_id=Q3BIO4GNMY64IWXC7VZ5BAYKYRJJ3GKFV7Z4IOG7SP62IUJHXN2A, seg_pos={0 0}, upload=113540]: Done uploading piece. err: ecclient: upload failed (node:1n2cXhqd5EWbNPTXem83jQUjYq5ZaGKBidbVCQzxYfR3rWddGR, address:104.36.82.44:4886): protocol: write tcp 188.166.225.170:38030->104.36.82.44:4886: use of closed network connection; write tcp 188.166.225.170:38030->104.36.82.44:4886: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38030->104.36.82.44:4886: use of closed network connection Aug 4 16:08:26.041128580 (118h11m15.97878711s) [noise=true , address=104.36.82.39:4728 , piece_id=ZCL6YOUXY4P6HTYUTTILZR4TLG44CDTXWLOJJWOHPFSORA2IBCBA, seg_pos={0 0}, upload=113540]: Done uploading piece. err: ecclient: upload failed (node:1YoLzpb4mMYZ8TsdxnMocekZZ1gLfTrdCM8mxTDj4tY6khZGvi, address:104.36.82.39:4728): protocol: write tcp 188.166.225.170:34326->104.36.82.39:4728: use of closed network connection; write tcp 188.166.225.170:34326->104.36.82.39:4728: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34326->104.36.82.39:4728: use of closed network connection Aug 4 16:08:26.041195678 (118h11m15.978854187s) [noise=true , address=104.36.82.45:4930 , piece_id=LBMDUGYMQI7PBBGEOT22S7OGNVRIYS66USSYFAZOSGI445GWKCYA, seg_pos={0 0}, upload=113540]: Done uploading piece. err: ecclient: upload failed (node:12qaYFSpezuLAc5zJfAYruzad6nTpYFDpog7bmKMVTA8MVm76NB, address:104.36.82.45:4930): protocol: expected piece hash; context deadline exceeded; EOF Aug 4 16:08:26.041203374 (118h11m15.978861913s) [noise=true , address=104.36.82.42:4823 , piece_id=66Q6G7YBCVUEAHU3Y5LHPR5MIT7H7NQWI2TWWNHKZT5FYS4DNQBA, seg_pos={0 0}, upload=113540]: Done uploading piece. err: ecclient: upload failed (node:1fkA5DEj9RqsR6N85z7azvujy8tPkUhB5GCqBpUmFb4vvwbd8y, address:104.36.82.42:4823): protocol: write tcp 188.166.225.170:37230->104.36.82.42:4823: use of closed network connection; write tcp 188.166.225.170:37230->104.36.82.42:4823: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37230->104.36.82.42:4823: use of closed network connection Aug 4 16:08:26.041258960 (118h11m15.978917489s) [seg_pos={0 0}, upload=113540]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Aug 4 16:08:26.274088873 (118h11m16.211747372s) [upload=113540]: Done uploading segments... Aug 4 16:08:26.274113650 (118h11m16.211772139s) [upload=113540]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 6.766954262s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [....................................................XX.X.XX.] duration: 6.023761773s FAIL (2025-08-04 16:14:23 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.02378057s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [......X.............................X.......................] duration: 1.88117489s FAIL (2025-08-04 16:14:22 UTC) orbiter warning: Operation: DownloadObject uplink: object not found ("orbiter_us_0/1K") Duration: 1.881205689s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 7.834415866s FAIL (2025-08-04 16:14:02 UTC) orbiter warning: Operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 4.36.82.36:4625 , piece_id=FCK7QGPSKJ7VDK4D7BVNF44BRPQCQS5V5ICAJHELAQHDBSQV5R3Q, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.146274823 (118h10m17.083933322s) [noise=true , address=45.91.116.7:4026 , piece_id=CB6TYC4A6DQOFFKVFPNMF5MH5MGMJHU5Z25B4QXQJI5ETDWLQJJQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.146672965 (118h10m17.084331464s) [noise=true , address=104.36.82.16:4321 , piece_id=42MZPFPEBQNQRJUHQJLZQP2UWGDV2OAXT6Z4KTWPPWLALISASN5Q, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.146726166 (118h10m17.084384665s) [noise=true , address=104.36.82.55:4206 , piece_id=KSAKYO6MALVA5REEKKJZ7MGX7YIRC2PDI5PNHFHRASPBD47V5SOQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.147136371 (118h10m17.08479488s) [noise=true , address=104.36.82.42:4823 , piece_id=66Q6G7YBCVUEAHU3Y5LHPR5MIT7H7NQWI2TWWNHKZT5FYS4DNQBA, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.147479628 (118h10m17.085138127s) [noise=true , address=193.149.164.143:4695 , piece_id=3QTYKOTBCOJ4CRM76FR2XKONKQVYTQQMPTRH3E6MYJR4ODBPY3AQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.147821592 (118h10m17.085480091s) [noise=true , address=104.36.82.62:4411 , piece_id=CIJJ4SMWKFDOIZZ5UEYIXFZHUNAZKRMM2P36IJIT5BHUX5STDZ4A, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.148134220 (118h10m17.085792719s) [noise=true , address=104.36.82.55:4224 , piece_id=NJYMQ2467DXHUEC6ELXITFJF6WWKM4XFEONRJVRF6QSJBDEMC7JA, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.148548573 (118h10m17.086207072s) [noise=true , address=104.36.82.13:4193 , piece_id=WJA72IWFD3TUA2WPGJGB6A7AHJXI4PGOK62YDMKSMDPUAVHUHW4Q, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.148889576 (118h10m17.086548085s) [noise=true , address=104.36.82.39:4728 , piece_id=ZCL6YOUXY4P6HTYUTTILZR4TLG44CDTXWLOJJWOHPFSORA2IBCBA, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.149205931 (118h10m17.08686443s) [noise=true , address=104.36.82.59:4326 , piece_id=CS7FVO3IRXKZFB2EZM2HQBVEVM4XTTFI5Y7TH3NJZGIYMREVY7UQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.149509752 (118h10m17.087168261s) [noise=true , address=104.36.82.45:4930 , piece_id=LBMDUGYMQI7PBBGEOT22S7OGNVRIYS66USSYFAZOSGI445GWKCYA, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.149866545 (118h10m17.087525044s) [noise=true , address=104.36.82.23:4474 , piece_id=PLEKQKMEQKQDW72BNLLY4JOOPFI6CPNNGT74M5EFFNDQQFWELXUQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.150193601 (118h10m17.08785211s) [noise=true , address=104.36.82.13:4212 , piece_id=ER6VIEGFSG2F7542M5P66RS3QAUQYV7BWPQ4YOX55Q3XX6DDCTGA, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.150515557 (118h10m17.088174066s) [noise=true , address=104.36.82.46:4485 , piece_id=P4FMDR5NCQOLKSXQBLE7I7BPEZJ2URAPPVNKOR53ZPROBCE4YAJQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.150935149 (118h10m17.088593648s) [noise=true , address=104.36.82.54:4181 , piece_id=AHW774W42KLBATPDVRQRKH4MQWV5YP2XEEUY6VASDMYJ2BKXCS2Q, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.151249151 (118h10m17.08890765s) [noise=true , address=104.36.82.44:4886 , piece_id=Q3BIO4GNMY64IWXC7VZ5BAYKYRJJ3GKFV7Z4IOG7SP62IUJHXN2A, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.151276363 (118h10m17.088934862s) [noise=true , address=104.36.82.9:4019 , piece_id=SBCPU6XTMB4CRRTFKDBYFQQ2XZZNE3I6AF7VVSGYC356SPQNVHUQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:28.640156348 (118h10m18.577814857s) [upload=113540]: Next returned nil segment Aug 4 16:07:28.640223306 (118h10m18.577881825s) [upload=113540]: Waiting for error group managing segments... Aug 4 16:07:29.062127352 (118h10m18.999785841s) [noise=true , address=104.36.82.16:4321 , piece_id=42MZPFPEBQNQRJUHQJLZQP2UWGDV2OAXT6Z4KTWPPWLALISASN5Q, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.080565981 (118h10m19.01822448s) [noise=true , address=104.36.82.54:4181 , piece_id=AHW774W42KLBATPDVRQRKH4MQWV5YP2XEEUY6VASDMYJ2BKXCS2Q, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.081329852 (118h10m19.018988352s) [noise=true , address=104.36.82.62:4411 , piece_id=CIJJ4SMWKFDOIZZ5UEYIXFZHUNAZKRMM2P36IJIT5BHUX5STDZ4A, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.081846201 (118h10m19.0195047s) [noise=true , address=104.36.82.55:4224 , piece_id=NJYMQ2467DXHUEC6ELXITFJF6WWKM4XFEONRJVRF6QSJBDEMC7JA, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.091017397 (118h10m19.028675896s) [noise=true , address=104.36.82.46:4485 , piece_id=P4FMDR5NCQOLKSXQBLE7I7BPEZJ2URAPPVNKOR53ZPROBCE4YAJQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.105092769 (118h10m19.042751268s) [noise=true , address=104.36.82.13:4212 , piece_id=ER6VIEGFSG2F7542M5P66RS3QAUQYV7BWPQ4YOX55Q3XX6DDCTGA, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.107141759 (118h10m19.044800248s) [noise=true , address=104.36.82.23:4474 , piece_id=PLEKQKMEQKQDW72BNLLY4JOOPFI6CPNNGT74M5EFFNDQQFWELXUQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.116502399 (118h10m19.054160898s) [noise=true , address=45.91.116.7:4026 , piece_id=CB6TYC4A6DQOFFKVFPNMF5MH5MGMJHU5Z25B4QXQJI5ETDWLQJJQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.153680253 (118h10m19.091338752s) [noise=true , address=104.36.82.9:4019 , piece_id=SBCPU6XTMB4CRRTFKDBYFQQ2XZZNE3I6AF7VVSGYC356SPQNVHUQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.210191177 (118h10m19.147849686s) [noise=true , address=104.36.82.36:4625 , piece_id=FCK7QGPSKJ7VDK4D7BVNF44BRPQCQS5V5ICAJHELAQHDBSQV5R3Q, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.219317350 (118h10m19.156975849s) [noise=true , address=104.36.82.55:4206 , piece_id=KSAKYO6MALVA5REEKKJZ7MGX7YIRC2PDI5PNHFHRASPBD47V5SOQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:31.340384557 (118h10m21.278043046s) [noise=true , address=104.36.82.59:4326 , piece_id=CS7FVO3IRXKZFB2EZM2HQBVEVM4XTTFI5Y7TH3NJZGIYMREVY7UQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:32.464238230 (118h10m22.401896729s) [noise=true , address=104.36.82.13:4193 , piece_id=WJA72IWFD3TUA2WPGJGB6A7AHJXI4PGOK62YDMKSMDPUAVHUHW4Q, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:08:26.041033880 (118h11m15.978692379s) [noise=true , address=193.149.164.143:4695 , piece_id=3QTYKOTBCOJ4CRM76FR2XKONKQVYTQQMPTRH3E6MYJR4ODBPY3AQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: ecclient: upload failed (node:12TZfgz24gnZr4UryzEKw6gu94njnmCvafX5X64MKFSJqaUWSVh, address:193.149.164.143:4695): protocol: write tcp 188.166.225.170:43298->193.149.164.143:4695: use of closed network connection; write tcp 188.166.225.170:43298->193.149.164.143:4695: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43298->193.149.164.143:4695: use of closed network connection Aug 4 16:08:26.041079005 (118h11m15.978737504s) [noise=true , address=104.36.82.44:4886 , piece_id=Q3BIO4GNMY64IWXC7VZ5BAYKYRJJ3GKFV7Z4IOG7SP62IUJHXN2A, seg_pos={0 0}, upload=113540]: Done uploading piece. err: ecclient: upload failed (node:1n2cXhqd5EWbNPTXem83jQUjYq5ZaGKBidbVCQzxYfR3rWddGR, address:104.36.82.44:4886): protocol: write tcp 188.166.225.170:38030->104.36.82.44:4886: use of closed network connection; write tcp 188.166.225.170:38030->104.36.82.44:4886: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38030->104.36.82.44:4886: use of closed network connection Aug 4 16:08:26.041128580 (118h11m15.97878711s) [noise=true , address=104.36.82.39:4728 , piece_id=ZCL6YOUXY4P6HTYUTTILZR4TLG44CDTXWLOJJWOHPFSORA2IBCBA, seg_pos={0 0}, upload=113540]: Done uploading piece. err: ecclient: upload failed (node:1YoLzpb4mMYZ8TsdxnMocekZZ1gLfTrdCM8mxTDj4tY6khZGvi, address:104.36.82.39:4728): protocol: write tcp 188.166.225.170:34326->104.36.82.39:4728: use of closed network connection; write tcp 188.166.225.170:34326->104.36.82.39:4728: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34326->104.36.82.39:4728: use of closed network connection Aug 4 16:08:26.041195678 (118h11m15.978854187s) [noise=true , address=104.36.82.45:4930 , piece_id=LBMDUGYMQI7PBBGEOT22S7OGNVRIYS66USSYFAZOSGI445GWKCYA, seg_pos={0 0}, upload=113540]: Done uploading piece. err: ecclient: upload failed (node:12qaYFSpezuLAc5zJfAYruzad6nTpYFDpog7bmKMVTA8MVm76NB, address:104.36.82.45:4930): protocol: expected piece hash; context deadline exceeded; EOF Aug 4 16:08:26.041203374 (118h11m15.978861913s) [noise=true , address=104.36.82.42:4823 , piece_id=66Q6G7YBCVUEAHU3Y5LHPR5MIT7H7NQWI2TWWNHKZT5FYS4DNQBA, seg_pos={0 0}, upload=113540]: Done uploading piece. err: ecclient: upload failed (node:1fkA5DEj9RqsR6N85z7azvujy8tPkUhB5GCqBpUmFb4vvwbd8y, address:104.36.82.42:4823): protocol: write tcp 188.166.225.170:37230->104.36.82.42:4823: use of closed network connection; write tcp 188.166.225.170:37230->104.36.82.42:4823: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37230->104.36.82.42:4823: use of closed network connection Aug 4 16:08:26.041258960 (118h11m15.978917489s) [seg_pos={0 0}, upload=113540]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Aug 4 16:08:26.274088873 (118h11m16.211747372s) [upload=113540]: Done uploading segments... Aug 4 16:08:26.274113650 (118h11m16.211772139s) [upload=113540]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 7.834445473s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.....................................................XX.X.XX] duration: 30.000687326s FAIL (2025-08-04 16:13:57 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.000708667s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 21.117676653s FAIL (2025-08-04 16:13:48 UTC) orbiter warning: Operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 4.36.82.36:4625 , piece_id=FCK7QGPSKJ7VDK4D7BVNF44BRPQCQS5V5ICAJHELAQHDBSQV5R3Q, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.146274823 (118h10m17.083933322s) [noise=true , address=45.91.116.7:4026 , piece_id=CB6TYC4A6DQOFFKVFPNMF5MH5MGMJHU5Z25B4QXQJI5ETDWLQJJQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.146672965 (118h10m17.084331464s) [noise=true , address=104.36.82.16:4321 , piece_id=42MZPFPEBQNQRJUHQJLZQP2UWGDV2OAXT6Z4KTWPPWLALISASN5Q, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.146726166 (118h10m17.084384665s) [noise=true , address=104.36.82.55:4206 , piece_id=KSAKYO6MALVA5REEKKJZ7MGX7YIRC2PDI5PNHFHRASPBD47V5SOQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.147136371 (118h10m17.08479488s) [noise=true , address=104.36.82.42:4823 , piece_id=66Q6G7YBCVUEAHU3Y5LHPR5MIT7H7NQWI2TWWNHKZT5FYS4DNQBA, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.147479628 (118h10m17.085138127s) [noise=true , address=193.149.164.143:4695 , piece_id=3QTYKOTBCOJ4CRM76FR2XKONKQVYTQQMPTRH3E6MYJR4ODBPY3AQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.147821592 (118h10m17.085480091s) [noise=true , address=104.36.82.62:4411 , piece_id=CIJJ4SMWKFDOIZZ5UEYIXFZHUNAZKRMM2P36IJIT5BHUX5STDZ4A, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.148134220 (118h10m17.085792719s) [noise=true , address=104.36.82.55:4224 , piece_id=NJYMQ2467DXHUEC6ELXITFJF6WWKM4XFEONRJVRF6QSJBDEMC7JA, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.148548573 (118h10m17.086207072s) [noise=true , address=104.36.82.13:4193 , piece_id=WJA72IWFD3TUA2WPGJGB6A7AHJXI4PGOK62YDMKSMDPUAVHUHW4Q, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.148889576 (118h10m17.086548085s) [noise=true , address=104.36.82.39:4728 , piece_id=ZCL6YOUXY4P6HTYUTTILZR4TLG44CDTXWLOJJWOHPFSORA2IBCBA, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.149205931 (118h10m17.08686443s) [noise=true , address=104.36.82.59:4326 , piece_id=CS7FVO3IRXKZFB2EZM2HQBVEVM4XTTFI5Y7TH3NJZGIYMREVY7UQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.149509752 (118h10m17.087168261s) [noise=true , address=104.36.82.45:4930 , piece_id=LBMDUGYMQI7PBBGEOT22S7OGNVRIYS66USSYFAZOSGI445GWKCYA, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.149866545 (118h10m17.087525044s) [noise=true , address=104.36.82.23:4474 , piece_id=PLEKQKMEQKQDW72BNLLY4JOOPFI6CPNNGT74M5EFFNDQQFWELXUQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.150193601 (118h10m17.08785211s) [noise=true , address=104.36.82.13:4212 , piece_id=ER6VIEGFSG2F7542M5P66RS3QAUQYV7BWPQ4YOX55Q3XX6DDCTGA, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.150515557 (118h10m17.088174066s) [noise=true , address=104.36.82.46:4485 , piece_id=P4FMDR5NCQOLKSXQBLE7I7BPEZJ2URAPPVNKOR53ZPROBCE4YAJQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.150935149 (118h10m17.088593648s) [noise=true , address=104.36.82.54:4181 , piece_id=AHW774W42KLBATPDVRQRKH4MQWV5YP2XEEUY6VASDMYJ2BKXCS2Q, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.151249151 (118h10m17.08890765s) [noise=true , address=104.36.82.44:4886 , piece_id=Q3BIO4GNMY64IWXC7VZ5BAYKYRJJ3GKFV7Z4IOG7SP62IUJHXN2A, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.151276363 (118h10m17.088934862s) [noise=true , address=104.36.82.9:4019 , piece_id=SBCPU6XTMB4CRRTFKDBYFQQ2XZZNE3I6AF7VVSGYC356SPQNVHUQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:28.640156348 (118h10m18.577814857s) [upload=113540]: Next returned nil segment Aug 4 16:07:28.640223306 (118h10m18.577881825s) [upload=113540]: Waiting for error group managing segments... Aug 4 16:07:29.062127352 (118h10m18.999785841s) [noise=true , address=104.36.82.16:4321 , piece_id=42MZPFPEBQNQRJUHQJLZQP2UWGDV2OAXT6Z4KTWPPWLALISASN5Q, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.080565981 (118h10m19.01822448s) [noise=true , address=104.36.82.54:4181 , piece_id=AHW774W42KLBATPDVRQRKH4MQWV5YP2XEEUY6VASDMYJ2BKXCS2Q, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.081329852 (118h10m19.018988352s) [noise=true , address=104.36.82.62:4411 , piece_id=CIJJ4SMWKFDOIZZ5UEYIXFZHUNAZKRMM2P36IJIT5BHUX5STDZ4A, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.081846201 (118h10m19.0195047s) [noise=true , address=104.36.82.55:4224 , piece_id=NJYMQ2467DXHUEC6ELXITFJF6WWKM4XFEONRJVRF6QSJBDEMC7JA, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.091017397 (118h10m19.028675896s) [noise=true , address=104.36.82.46:4485 , piece_id=P4FMDR5NCQOLKSXQBLE7I7BPEZJ2URAPPVNKOR53ZPROBCE4YAJQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.105092769 (118h10m19.042751268s) [noise=true , address=104.36.82.13:4212 , piece_id=ER6VIEGFSG2F7542M5P66RS3QAUQYV7BWPQ4YOX55Q3XX6DDCTGA, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.107141759 (118h10m19.044800248s) [noise=true , address=104.36.82.23:4474 , piece_id=PLEKQKMEQKQDW72BNLLY4JOOPFI6CPNNGT74M5EFFNDQQFWELXUQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.116502399 (118h10m19.054160898s) [noise=true , address=45.91.116.7:4026 , piece_id=CB6TYC4A6DQOFFKVFPNMF5MH5MGMJHU5Z25B4QXQJI5ETDWLQJJQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.153680253 (118h10m19.091338752s) [noise=true , address=104.36.82.9:4019 , piece_id=SBCPU6XTMB4CRRTFKDBYFQQ2XZZNE3I6AF7VVSGYC356SPQNVHUQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.210191177 (118h10m19.147849686s) [noise=true , address=104.36.82.36:4625 , piece_id=FCK7QGPSKJ7VDK4D7BVNF44BRPQCQS5V5ICAJHELAQHDBSQV5R3Q, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.219317350 (118h10m19.156975849s) [noise=true , address=104.36.82.55:4206 , piece_id=KSAKYO6MALVA5REEKKJZ7MGX7YIRC2PDI5PNHFHRASPBD47V5SOQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:31.340384557 (118h10m21.278043046s) [noise=true , address=104.36.82.59:4326 , piece_id=CS7FVO3IRXKZFB2EZM2HQBVEVM4XTTFI5Y7TH3NJZGIYMREVY7UQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:32.464238230 (118h10m22.401896729s) [noise=true , address=104.36.82.13:4193 , piece_id=WJA72IWFD3TUA2WPGJGB6A7AHJXI4PGOK62YDMKSMDPUAVHUHW4Q, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:08:26.041033880 (118h11m15.978692379s) [noise=true , address=193.149.164.143:4695 , piece_id=3QTYKOTBCOJ4CRM76FR2XKONKQVYTQQMPTRH3E6MYJR4ODBPY3AQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: ecclient: upload failed (node:12TZfgz24gnZr4UryzEKw6gu94njnmCvafX5X64MKFSJqaUWSVh, address:193.149.164.143:4695): protocol: write tcp 188.166.225.170:43298->193.149.164.143:4695: use of closed network connection; write tcp 188.166.225.170:43298->193.149.164.143:4695: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43298->193.149.164.143:4695: use of closed network connection Aug 4 16:08:26.041079005 (118h11m15.978737504s) [noise=true , address=104.36.82.44:4886 , piece_id=Q3BIO4GNMY64IWXC7VZ5BAYKYRJJ3GKFV7Z4IOG7SP62IUJHXN2A, seg_pos={0 0}, upload=113540]: Done uploading piece. err: ecclient: upload failed (node:1n2cXhqd5EWbNPTXem83jQUjYq5ZaGKBidbVCQzxYfR3rWddGR, address:104.36.82.44:4886): protocol: write tcp 188.166.225.170:38030->104.36.82.44:4886: use of closed network connection; write tcp 188.166.225.170:38030->104.36.82.44:4886: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38030->104.36.82.44:4886: use of closed network connection Aug 4 16:08:26.041128580 (118h11m15.97878711s) [noise=true , address=104.36.82.39:4728 , piece_id=ZCL6YOUXY4P6HTYUTTILZR4TLG44CDTXWLOJJWOHPFSORA2IBCBA, seg_pos={0 0}, upload=113540]: Done uploading piece. err: ecclient: upload failed (node:1YoLzpb4mMYZ8TsdxnMocekZZ1gLfTrdCM8mxTDj4tY6khZGvi, address:104.36.82.39:4728): protocol: write tcp 188.166.225.170:34326->104.36.82.39:4728: use of closed network connection; write tcp 188.166.225.170:34326->104.36.82.39:4728: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34326->104.36.82.39:4728: use of closed network connection Aug 4 16:08:26.041195678 (118h11m15.978854187s) [noise=true , address=104.36.82.45:4930 , piece_id=LBMDUGYMQI7PBBGEOT22S7OGNVRIYS66USSYFAZOSGI445GWKCYA, seg_pos={0 0}, upload=113540]: Done uploading piece. err: ecclient: upload failed (node:12qaYFSpezuLAc5zJfAYruzad6nTpYFDpog7bmKMVTA8MVm76NB, address:104.36.82.45:4930): protocol: expected piece hash; context deadline exceeded; EOF Aug 4 16:08:26.041203374 (118h11m15.978861913s) [noise=true , address=104.36.82.42:4823 , piece_id=66Q6G7YBCVUEAHU3Y5LHPR5MIT7H7NQWI2TWWNHKZT5FYS4DNQBA, seg_pos={0 0}, upload=113540]: Done uploading piece. err: ecclient: upload failed (node:1fkA5DEj9RqsR6N85z7azvujy8tPkUhB5GCqBpUmFb4vvwbd8y, address:104.36.82.42:4823): protocol: write tcp 188.166.225.170:37230->104.36.82.42:4823: use of closed network connection; write tcp 188.166.225.170:37230->104.36.82.42:4823: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37230->104.36.82.42:4823: use of closed network connection Aug 4 16:08:26.041258960 (118h11m15.978917489s) [seg_pos={0 0}, upload=113540]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Aug 4 16:08:26.274088873 (118h11m16.211747372s) [upload=113540]: Done uploading segments... Aug 4 16:08:26.274113650 (118h11m16.211772139s) [upload=113540]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 21.117703002s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [.......X.............................X......................] duration: 1.886168406s FAIL (2025-08-04 16:13:48 UTC) orbiter warning: Operation: DownloadObject uplink: object not found ("orbiter_us_0/1K") Duration: 1.886190397s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [......................................................XX....] duration: 5.793639667s FAIL (2025-08-04 16:13:39 UTC) orbiter warning: Operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== 16569229 (118h13m13.754227758s) [upload=113588]: Done starting upload Aug 4 16:10:23.816697144 (118h13m13.754355643s) [upload=113588]: Uploading segments... Aug 4 16:10:23.816823747 (118h13m13.754482246s) [upload=113588]: Got next segment. Inline: false Aug 4 16:10:23.816875515 (118h13m13.754534014s) [upload=113588]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 4 16:10:23.816882549 (118h13m13.754541068s) [upload=113588]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 4 16:10:24.051702429 (118h13m13.989360928s) [seg_pos={0 0}, upload=113588]: Begin upload segment... Aug 4 16:10:24.051745952 (118h13m13.989404451s) [seg_pos={0 0}, upload=113588]: Done begin upload segment. Aug 4 16:10:24.051767884 (118h13m13.989426383s) [noise=true , address=104.36.82.16:4293 , piece_id=UUQ6WYMZMYI6PYRGK5IK2OE6CX2AHAW45ZRCPA7LK3X2IIIM3ZJA, seg_pos={0 0}, upload=113588]: Uploading piece... Aug 4 16:10:24.051841024 (118h13m13.989499533s) [noise=true , address=104.36.82.7:4910 , piece_id=ZBYQ4SELARIGNS5Q3FBOOS3QIRVQ23VDCHMDK3TBLU5E7VREUPBA, seg_pos={0 0}, upload=113588]: Uploading piece... Aug 4 16:10:24.052106963 (118h13m13.989765462s) [noise=true , address=77.90.31.18:4805 , piece_id=OW2WLOK6KW2WUGVFEDEHMTUBSSUGGH4S5NQTYXKNOVMDXRYWHSEQ, seg_pos={0 0}, upload=113588]: Uploading piece... Aug 4 16:10:24.052435702 (118h13m13.990094201s) [noise=true , address=77.90.31.21:4877 , piece_id=GNPOUGIJ42Y47VOZ5EMFVIDEFAY5B3KQQPEZ3LS7XYBDTXBZCLKQ, seg_pos={0 0}, upload=113588]: Uploading piece... Aug 4 16:10:24.052526446 (118h13m13.990184945s) [noise=true , address=45.91.116.12:4225 , piece_id=3M2NI24IGXUHSDASHFYXDR2QPTNGEAQFT22AHIUAHVK2JVJNDQXQ, seg_pos={0 0}, upload=113588]: Uploading piece... Aug 4 16:10:24.052684418 (118h13m13.990342907s) [noise=true , address=45.91.116.10:4082 , piece_id=GKHLII3XXM72YPTIZF5ZFZESHXZQNB56E3CVZFAEXJC2WX6URD5A, seg_pos={0 0}, upload=113588]: Uploading piece... Aug 4 16:10:24.052926030 (118h13m13.990584529s) [noise=true , address=104.36.82.10:4077 , piece_id=UNAQGJCBMXKJIOAHLHZSG4RYC2UDR3B5OXAOUTI7FD5CXLFXR2UQ, seg_pos={0 0}, upload=113588]: Uploading piece... Aug 4 16:10:24.052950747 (118h13m13.990609246s) [noise=true , address=45.91.116.4:4938 , piece_id=SHP5IWA4HZTAAECBH2VRW525XCY6BVWIJOHE3U7WAD4ISJ63WUVA, seg_pos={0 0}, upload=113588]: Uploading piece... Aug 4 16:10:24.053160659 (118h13m13.990819158s) [noise=true , address=45.91.116.11:4119 , piece_id=WBFTLEUFEXKAR2U5SZLSOHPZEGV3QNWDLVJBL36XP2GEAK5L7RUA, seg_pos={0 0}, upload=113588]: Uploading piece... Aug 4 16:10:24.053383786 (118h13m13.991042285s) [noise=true , address=77.90.31.12:4589 , piece_id=BFYZ6N2AZEFVYNS4EQYPVKNLUIANMXZ7MPDSGYA5LB2XP2REAJUQ, seg_pos={0 0}, upload=113588]: Uploading piece... Aug 4 16:10:24.053410627 (118h13m13.991069126s) [noise=true , address=77.90.31.16:4735 , piece_id=XET5ROIUKXDV7MNQBYU2JPXLWHH74RD525PMYWC7VS7GZABL5P7A, seg_pos={0 0}, upload=113588]: Uploading piece... Aug 4 16:10:24.053613205 (118h13m13.991271704s) [noise=true , address=77.90.31.13:4627 , piece_id=E5LXNHOSI5URSZV7OBEWIQ75CEQWXDN2DLW4N2PUSY6MFHCNQ7YA, seg_pos={0 0}, upload=113588]: Uploading piece... Aug 4 16:10:24.053834499 (118h13m13.991492998s) [noise=true , address=104.36.82.6:4876 , piece_id=EWMXECZHZILAGIYJ5QJGYIGZSN4XP4LAD2PFY5CQHVARTAGTU5UA, seg_pos={0 0}, upload=113588]: Uploading piece... Aug 4 16:10:24.053873513 (118h13m13.991532012s) [noise=true , address=45.91.116.7:4011 , piece_id=57KLGTVQH77IOCAQPMQA66CPZPLEDKUIO4JKMUP6BOD4XSMPB5BQ, seg_pos={0 0}, upload=113588]: Uploading piece... Aug 4 16:10:24.054251246 (118h13m13.991909755s) [noise=true , address=77.90.31.10:4519 , piece_id=7HTU6S7JCLTAA7MMAYQBQP6F2PWHU3X7BU2NPHUUT2Q2MPFTLRWA, seg_pos={0 0}, upload=113588]: Uploading piece... Aug 4 16:10:24.054636593 (118h13m13.992295092s) [noise=true , address=45.91.116.8:4048 , piece_id=JGBCOCNYT4IZUJ6BCI3TEX7E46PYWQSS6HHM67YCOXQ3W5L4H5OA, seg_pos={0 0}, upload=113588]: Uploading piece... Aug 4 16:10:25.788117314 (118h13m15.725775813s) [upload=113588]: Next returned nil segment Aug 4 16:10:25.788134928 (118h13m15.725793427s) [upload=113588]: Waiting for error group managing segments... Aug 4 16:10:26.080627835 (118h13m16.018286335s) [noise=true , address=77.90.31.16:4735 , piece_id=XET5ROIUKXDV7MNQBYU2JPXLWHH74RD525PMYWC7VS7GZABL5P7A, seg_pos={0 0}, upload=113588]: Done uploading piece. err: <nil> Aug 4 16:10:26.081237151 (118h13m16.01889565s) [noise=true , address=77.90.31.21:4877 , piece_id=GNPOUGIJ42Y47VOZ5EMFVIDEFAY5B3KQQPEZ3LS7XYBDTXBZCLKQ, seg_pos={0 0}, upload=113588]: Done uploading piece. err: <nil> Aug 4 16:10:26.082642120 (118h13m16.020300619s) [noise=true , address=77.90.31.18:4805 , piece_id=OW2WLOK6KW2WUGVFEDEHMTUBSSUGGH4S5NQTYXKNOVMDXRYWHSEQ, seg_pos={0 0}, upload=113588]: Done uploading piece. err: <nil> Aug 4 16:10:26.090023893 (118h13m16.027682392s) [noise=true , address=45.91.116.11:4119 , piece_id=WBFTLEUFEXKAR2U5SZLSOHPZEGV3QNWDLVJBL36XP2GEAK5L7RUA, seg_pos={0 0}, upload=113588]: Done uploading piece. err: <nil> Aug 4 16:10:26.092155202 (118h13m16.029813701s) [noise=true , address=45.91.116.12:4225 , piece_id=3M2NI24IGXUHSDASHFYXDR2QPTNGEAQFT22AHIUAHVK2JVJNDQXQ, seg_pos={0 0}, upload=113588]: Done uploading piece. err: <nil> Aug 4 16:10:26.093301104 (118h13m16.030959593s) [noise=true , address=45.91.116.4:4938 , piece_id=SHP5IWA4HZTAAECBH2VRW525XCY6BVWIJOHE3U7WAD4ISJ63WUVA, seg_pos={0 0}, upload=113588]: Done uploading piece. err: <nil> Aug 4 16:10:26.096255166 (118h13m16.033913675s) [noise=true , address=104.36.82.6:4876 , piece_id=EWMXECZHZILAGIYJ5QJGYIGZSN4XP4LAD2PFY5CQHVARTAGTU5UA, seg_pos={0 0}, upload=113588]: Done uploading piece. err: <nil> Aug 4 16:10:26.112849236 (118h13m16.050507735s) [noise=true , address=104.36.82.16:4293 , piece_id=UUQ6WYMZMYI6PYRGK5IK2OE6CX2AHAW45ZRCPA7LK3X2IIIM3ZJA, seg_pos={0 0}, upload=113588]: Done uploading piece. err: <nil> Aug 4 16:10:26.171272489 (118h13m16.108930988s) [noise=true , address=77.90.31.12:4589 , piece_id=BFYZ6N2AZEFVYNS4EQYPVKNLUIANMXZ7MPDSGYA5LB2XP2REAJUQ, seg_pos={0 0}, upload=113588]: Done uploading piece. err: <nil> Aug 4 16:10:28.145791810 (118h13m18.083450309s) [noise=true , address=77.90.31.13:4627 , piece_id=E5LXNHOSI5URSZV7OBEWIQ75CEQWXDN2DLW4N2PUSY6MFHCNQ7YA, seg_pos={0 0}, upload=113588]: Done uploading piece. err: <nil> Aug 4 16:10:31.180407751 (118h13m21.11806625s) [noise=true , address=77.90.31.10:4519 , piece_id=7HTU6S7JCLTAA7MMAYQBQP6F2PWHU3X7BU2NPHUUT2Q2MPFTLRWA, seg_pos={0 0}, upload=113588]: Done uploading piece. err: <nil> Aug 4 16:10:53.161543551 (118h13m43.099202051s) [noise=true , address=104.36.82.7:4910 , piece_id=ZBYQ4SELARIGNS5Q3FBOOS3QIRVQ23VDCHMDK3TBLU5E7VREUPBA, seg_pos={0 0}, upload=113588]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: write tcp 188.166.225.170:52204->104.36.82.7:4910: use of closed network connection; write tcp 188.166.225.170:52204->104.36.82.7:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52204->104.36.82.7:4910: use of closed network connection Aug 4 16:10:53.161605039 (118h13m43.099263528s) [noise=true , address=104.36.82.10:4077 , piece_id=UNAQGJCBMXKJIOAHLHZSG4RYC2UDR3B5OXAOUTI7FD5CXLFXR2UQ, seg_pos={0 0}, upload=113588]: Done uploading piece. err: ecclient: upload failed (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk, address:104.36.82.10:4077): protocol: write tcp 188.166.225.170:41132->104.36.82.10:4077: use of closed network connection; write tcp 188.166.225.170:41132->104.36.82.10:4077: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41132->104.36.82.10:4077: use of closed network connection Aug 4 16:10:53.161771859 (118h13m43.099430348s) [noise=true , address=45.91.116.8:4048 , piece_id=JGBCOCNYT4IZUJ6BCI3TEX7E46PYWQSS6HHM67YCOXQ3W5L4H5OA, seg_pos={0 0}, upload=113588]: Done uploading piece. err: ecclient: upload failed (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, address:45.91.116.8:4048): protocol: write tcp 188.166.225.170:35774->45.91.116.8:4048: use of closed network connection; write tcp 188.166.225.170:35774->45.91.116.8:4048: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35774->45.91.116.8:4048: use of closed network connection Aug 4 16:10:53.161844948 (118h13m43.099503447s) [noise=true , address=45.91.116.7:4011 , piece_id=57KLGTVQH77IOCAQPMQA66CPZPLEDKUIO4JKMUP6BOD4XSMPB5BQ, seg_pos={0 0}, upload=113588]: Done uploading piece. err: ecclient: upload failed (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, address:45.91.116.7:4011): protocol: write tcp 188.166.225.170:50254->45.91.116.7:4011: use of closed network connection; write tcp 188.166.225.170:50254->45.91.116.7:4011: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50254->45.91.116.7:4011: use of closed network connection Aug 4 16:10:53.161881499 (118h13m43.099539998s) [noise=true , address=45.91.116.10:4082 , piece_id=GKHLII3XXM72YPTIZF5ZFZESHXZQNB56E3CVZFAEXJC2WX6URD5A, seg_pos={0 0}, upload=113588]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: read tcp 188.166.225.170:44088->45.91.116.10:4082: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 4 16:10:53.161905614 (118h13m43.099564103s) [seg_pos={0 0}, upload=113588]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Aug 4 16:10:53.443651470 (118h13m43.381309969s) [upload=113588]: Done uploading segments... Aug 4 16:10:53.443675736 (118h13m43.381334235s) [upload=113588]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.793664093s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 10.567472253s FAIL (2025-08-04 16:13:19 UTC) orbiter warning: Operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 4.36.82.36:4625 , piece_id=FCK7QGPSKJ7VDK4D7BVNF44BRPQCQS5V5ICAJHELAQHDBSQV5R3Q, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.146274823 (118h10m17.083933322s) [noise=true , address=45.91.116.7:4026 , piece_id=CB6TYC4A6DQOFFKVFPNMF5MH5MGMJHU5Z25B4QXQJI5ETDWLQJJQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.146672965 (118h10m17.084331464s) [noise=true , address=104.36.82.16:4321 , piece_id=42MZPFPEBQNQRJUHQJLZQP2UWGDV2OAXT6Z4KTWPPWLALISASN5Q, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.146726166 (118h10m17.084384665s) [noise=true , address=104.36.82.55:4206 , piece_id=KSAKYO6MALVA5REEKKJZ7MGX7YIRC2PDI5PNHFHRASPBD47V5SOQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.147136371 (118h10m17.08479488s) [noise=true , address=104.36.82.42:4823 , piece_id=66Q6G7YBCVUEAHU3Y5LHPR5MIT7H7NQWI2TWWNHKZT5FYS4DNQBA, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.147479628 (118h10m17.085138127s) [noise=true , address=193.149.164.143:4695 , piece_id=3QTYKOTBCOJ4CRM76FR2XKONKQVYTQQMPTRH3E6MYJR4ODBPY3AQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.147821592 (118h10m17.085480091s) [noise=true , address=104.36.82.62:4411 , piece_id=CIJJ4SMWKFDOIZZ5UEYIXFZHUNAZKRMM2P36IJIT5BHUX5STDZ4A, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.148134220 (118h10m17.085792719s) [noise=true , address=104.36.82.55:4224 , piece_id=NJYMQ2467DXHUEC6ELXITFJF6WWKM4XFEONRJVRF6QSJBDEMC7JA, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.148548573 (118h10m17.086207072s) [noise=true , address=104.36.82.13:4193 , piece_id=WJA72IWFD3TUA2WPGJGB6A7AHJXI4PGOK62YDMKSMDPUAVHUHW4Q, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.148889576 (118h10m17.086548085s) [noise=true , address=104.36.82.39:4728 , piece_id=ZCL6YOUXY4P6HTYUTTILZR4TLG44CDTXWLOJJWOHPFSORA2IBCBA, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.149205931 (118h10m17.08686443s) [noise=true , address=104.36.82.59:4326 , piece_id=CS7FVO3IRXKZFB2EZM2HQBVEVM4XTTFI5Y7TH3NJZGIYMREVY7UQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.149509752 (118h10m17.087168261s) [noise=true , address=104.36.82.45:4930 , piece_id=LBMDUGYMQI7PBBGEOT22S7OGNVRIYS66USSYFAZOSGI445GWKCYA, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.149866545 (118h10m17.087525044s) [noise=true , address=104.36.82.23:4474 , piece_id=PLEKQKMEQKQDW72BNLLY4JOOPFI6CPNNGT74M5EFFNDQQFWELXUQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.150193601 (118h10m17.08785211s) [noise=true , address=104.36.82.13:4212 , piece_id=ER6VIEGFSG2F7542M5P66RS3QAUQYV7BWPQ4YOX55Q3XX6DDCTGA, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.150515557 (118h10m17.088174066s) [noise=true , address=104.36.82.46:4485 , piece_id=P4FMDR5NCQOLKSXQBLE7I7BPEZJ2URAPPVNKOR53ZPROBCE4YAJQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.150935149 (118h10m17.088593648s) [noise=true , address=104.36.82.54:4181 , piece_id=AHW774W42KLBATPDVRQRKH4MQWV5YP2XEEUY6VASDMYJ2BKXCS2Q, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.151249151 (118h10m17.08890765s) [noise=true , address=104.36.82.44:4886 , piece_id=Q3BIO4GNMY64IWXC7VZ5BAYKYRJJ3GKFV7Z4IOG7SP62IUJHXN2A, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.151276363 (118h10m17.088934862s) [noise=true , address=104.36.82.9:4019 , piece_id=SBCPU6XTMB4CRRTFKDBYFQQ2XZZNE3I6AF7VVSGYC356SPQNVHUQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:28.640156348 (118h10m18.577814857s) [upload=113540]: Next returned nil segment Aug 4 16:07:28.640223306 (118h10m18.577881825s) [upload=113540]: Waiting for error group managing segments... Aug 4 16:07:29.062127352 (118h10m18.999785841s) [noise=true , address=104.36.82.16:4321 , piece_id=42MZPFPEBQNQRJUHQJLZQP2UWGDV2OAXT6Z4KTWPPWLALISASN5Q, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.080565981 (118h10m19.01822448s) [noise=true , address=104.36.82.54:4181 , piece_id=AHW774W42KLBATPDVRQRKH4MQWV5YP2XEEUY6VASDMYJ2BKXCS2Q, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.081329852 (118h10m19.018988352s) [noise=true , address=104.36.82.62:4411 , piece_id=CIJJ4SMWKFDOIZZ5UEYIXFZHUNAZKRMM2P36IJIT5BHUX5STDZ4A, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.081846201 (118h10m19.0195047s) [noise=true , address=104.36.82.55:4224 , piece_id=NJYMQ2467DXHUEC6ELXITFJF6WWKM4XFEONRJVRF6QSJBDEMC7JA, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.091017397 (118h10m19.028675896s) [noise=true , address=104.36.82.46:4485 , piece_id=P4FMDR5NCQOLKSXQBLE7I7BPEZJ2URAPPVNKOR53ZPROBCE4YAJQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.105092769 (118h10m19.042751268s) [noise=true , address=104.36.82.13:4212 , piece_id=ER6VIEGFSG2F7542M5P66RS3QAUQYV7BWPQ4YOX55Q3XX6DDCTGA, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.107141759 (118h10m19.044800248s) [noise=true , address=104.36.82.23:4474 , piece_id=PLEKQKMEQKQDW72BNLLY4JOOPFI6CPNNGT74M5EFFNDQQFWELXUQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.116502399 (118h10m19.054160898s) [noise=true , address=45.91.116.7:4026 , piece_id=CB6TYC4A6DQOFFKVFPNMF5MH5MGMJHU5Z25B4QXQJI5ETDWLQJJQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.153680253 (118h10m19.091338752s) [noise=true , address=104.36.82.9:4019 , piece_id=SBCPU6XTMB4CRRTFKDBYFQQ2XZZNE3I6AF7VVSGYC356SPQNVHUQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.210191177 (118h10m19.147849686s) [noise=true , address=104.36.82.36:4625 , piece_id=FCK7QGPSKJ7VDK4D7BVNF44BRPQCQS5V5ICAJHELAQHDBSQV5R3Q, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.219317350 (118h10m19.156975849s) [noise=true , address=104.36.82.55:4206 , piece_id=KSAKYO6MALVA5REEKKJZ7MGX7YIRC2PDI5PNHFHRASPBD47V5SOQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:31.340384557 (118h10m21.278043046s) [noise=true , address=104.36.82.59:4326 , piece_id=CS7FVO3IRXKZFB2EZM2HQBVEVM4XTTFI5Y7TH3NJZGIYMREVY7UQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:32.464238230 (118h10m22.401896729s) [noise=true , address=104.36.82.13:4193 , piece_id=WJA72IWFD3TUA2WPGJGB6A7AHJXI4PGOK62YDMKSMDPUAVHUHW4Q, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:08:26.041033880 (118h11m15.978692379s) [noise=true , address=193.149.164.143:4695 , piece_id=3QTYKOTBCOJ4CRM76FR2XKONKQVYTQQMPTRH3E6MYJR4ODBPY3AQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: ecclient: upload failed (node:12TZfgz24gnZr4UryzEKw6gu94njnmCvafX5X64MKFSJqaUWSVh, address:193.149.164.143:4695): protocol: write tcp 188.166.225.170:43298->193.149.164.143:4695: use of closed network connection; write tcp 188.166.225.170:43298->193.149.164.143:4695: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43298->193.149.164.143:4695: use of closed network connection Aug 4 16:08:26.041079005 (118h11m15.978737504s) [noise=true , address=104.36.82.44:4886 , piece_id=Q3BIO4GNMY64IWXC7VZ5BAYKYRJJ3GKFV7Z4IOG7SP62IUJHXN2A, seg_pos={0 0}, upload=113540]: Done uploading piece. err: ecclient: upload failed (node:1n2cXhqd5EWbNPTXem83jQUjYq5ZaGKBidbVCQzxYfR3rWddGR, address:104.36.82.44:4886): protocol: write tcp 188.166.225.170:38030->104.36.82.44:4886: use of closed network connection; write tcp 188.166.225.170:38030->104.36.82.44:4886: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38030->104.36.82.44:4886: use of closed network connection Aug 4 16:08:26.041128580 (118h11m15.97878711s) [noise=true , address=104.36.82.39:4728 , piece_id=ZCL6YOUXY4P6HTYUTTILZR4TLG44CDTXWLOJJWOHPFSORA2IBCBA, seg_pos={0 0}, upload=113540]: Done uploading piece. err: ecclient: upload failed (node:1YoLzpb4mMYZ8TsdxnMocekZZ1gLfTrdCM8mxTDj4tY6khZGvi, address:104.36.82.39:4728): protocol: write tcp 188.166.225.170:34326->104.36.82.39:4728: use of closed network connection; write tcp 188.166.225.170:34326->104.36.82.39:4728: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34326->104.36.82.39:4728: use of closed network connection Aug 4 16:08:26.041195678 (118h11m15.978854187s) [noise=true , address=104.36.82.45:4930 , piece_id=LBMDUGYMQI7PBBGEOT22S7OGNVRIYS66USSYFAZOSGI445GWKCYA, seg_pos={0 0}, upload=113540]: Done uploading piece. err: ecclient: upload failed (node:12qaYFSpezuLAc5zJfAYruzad6nTpYFDpog7bmKMVTA8MVm76NB, address:104.36.82.45:4930): protocol: expected piece hash; context deadline exceeded; EOF Aug 4 16:08:26.041203374 (118h11m15.978861913s) [noise=true , address=104.36.82.42:4823 , piece_id=66Q6G7YBCVUEAHU3Y5LHPR5MIT7H7NQWI2TWWNHKZT5FYS4DNQBA, seg_pos={0 0}, upload=113540]: Done uploading piece. err: ecclient: upload failed (node:1fkA5DEj9RqsR6N85z7azvujy8tPkUhB5GCqBpUmFb4vvwbd8y, address:104.36.82.42:4823): protocol: write tcp 188.166.225.170:37230->104.36.82.42:4823: use of closed network connection; write tcp 188.166.225.170:37230->104.36.82.42:4823: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37230->104.36.82.42:4823: use of closed network connection Aug 4 16:08:26.041258960 (118h11m15.978917489s) [seg_pos={0 0}, upload=113540]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Aug 4 16:08:26.274088873 (118h11m16.211747372s) [upload=113540]: Done uploading segments... Aug 4 16:08:26.274113650 (118h11m16.211772139s) [upload=113540]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 10.567488233s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [......................................................XX.X.X] duration: 30.000485224s FAIL (2025-08-04 16:13:07 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.000507384s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......................................................XX...] duration: 7.208766614s FAIL (2025-08-04 16:12:52 UTC) orbiter warning: Operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== 16569229 (118h13m13.754227758s) [upload=113588]: Done starting upload Aug 4 16:10:23.816697144 (118h13m13.754355643s) [upload=113588]: Uploading segments... Aug 4 16:10:23.816823747 (118h13m13.754482246s) [upload=113588]: Got next segment. Inline: false Aug 4 16:10:23.816875515 (118h13m13.754534014s) [upload=113588]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 4 16:10:23.816882549 (118h13m13.754541068s) [upload=113588]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 4 16:10:24.051702429 (118h13m13.989360928s) [seg_pos={0 0}, upload=113588]: Begin upload segment... Aug 4 16:10:24.051745952 (118h13m13.989404451s) [seg_pos={0 0}, upload=113588]: Done begin upload segment. Aug 4 16:10:24.051767884 (118h13m13.989426383s) [noise=true , address=104.36.82.16:4293 , piece_id=UUQ6WYMZMYI6PYRGK5IK2OE6CX2AHAW45ZRCPA7LK3X2IIIM3ZJA, seg_pos={0 0}, upload=113588]: Uploading piece... Aug 4 16:10:24.051841024 (118h13m13.989499533s) [noise=true , address=104.36.82.7:4910 , piece_id=ZBYQ4SELARIGNS5Q3FBOOS3QIRVQ23VDCHMDK3TBLU5E7VREUPBA, seg_pos={0 0}, upload=113588]: Uploading piece... Aug 4 16:10:24.052106963 (118h13m13.989765462s) [noise=true , address=77.90.31.18:4805 , piece_id=OW2WLOK6KW2WUGVFEDEHMTUBSSUGGH4S5NQTYXKNOVMDXRYWHSEQ, seg_pos={0 0}, upload=113588]: Uploading piece... Aug 4 16:10:24.052435702 (118h13m13.990094201s) [noise=true , address=77.90.31.21:4877 , piece_id=GNPOUGIJ42Y47VOZ5EMFVIDEFAY5B3KQQPEZ3LS7XYBDTXBZCLKQ, seg_pos={0 0}, upload=113588]: Uploading piece... Aug 4 16:10:24.052526446 (118h13m13.990184945s) [noise=true , address=45.91.116.12:4225 , piece_id=3M2NI24IGXUHSDASHFYXDR2QPTNGEAQFT22AHIUAHVK2JVJNDQXQ, seg_pos={0 0}, upload=113588]: Uploading piece... Aug 4 16:10:24.052684418 (118h13m13.990342907s) [noise=true , address=45.91.116.10:4082 , piece_id=GKHLII3XXM72YPTIZF5ZFZESHXZQNB56E3CVZFAEXJC2WX6URD5A, seg_pos={0 0}, upload=113588]: Uploading piece... Aug 4 16:10:24.052926030 (118h13m13.990584529s) [noise=true , address=104.36.82.10:4077 , piece_id=UNAQGJCBMXKJIOAHLHZSG4RYC2UDR3B5OXAOUTI7FD5CXLFXR2UQ, seg_pos={0 0}, upload=113588]: Uploading piece... Aug 4 16:10:24.052950747 (118h13m13.990609246s) [noise=true , address=45.91.116.4:4938 , piece_id=SHP5IWA4HZTAAECBH2VRW525XCY6BVWIJOHE3U7WAD4ISJ63WUVA, seg_pos={0 0}, upload=113588]: Uploading piece... Aug 4 16:10:24.053160659 (118h13m13.990819158s) [noise=true , address=45.91.116.11:4119 , piece_id=WBFTLEUFEXKAR2U5SZLSOHPZEGV3QNWDLVJBL36XP2GEAK5L7RUA, seg_pos={0 0}, upload=113588]: Uploading piece... Aug 4 16:10:24.053383786 (118h13m13.991042285s) [noise=true , address=77.90.31.12:4589 , piece_id=BFYZ6N2AZEFVYNS4EQYPVKNLUIANMXZ7MPDSGYA5LB2XP2REAJUQ, seg_pos={0 0}, upload=113588]: Uploading piece... Aug 4 16:10:24.053410627 (118h13m13.991069126s) [noise=true , address=77.90.31.16:4735 , piece_id=XET5ROIUKXDV7MNQBYU2JPXLWHH74RD525PMYWC7VS7GZABL5P7A, seg_pos={0 0}, upload=113588]: Uploading piece... Aug 4 16:10:24.053613205 (118h13m13.991271704s) [noise=true , address=77.90.31.13:4627 , piece_id=E5LXNHOSI5URSZV7OBEWIQ75CEQWXDN2DLW4N2PUSY6MFHCNQ7YA, seg_pos={0 0}, upload=113588]: Uploading piece... Aug 4 16:10:24.053834499 (118h13m13.991492998s) [noise=true , address=104.36.82.6:4876 , piece_id=EWMXECZHZILAGIYJ5QJGYIGZSN4XP4LAD2PFY5CQHVARTAGTU5UA, seg_pos={0 0}, upload=113588]: Uploading piece... Aug 4 16:10:24.053873513 (118h13m13.991532012s) [noise=true , address=45.91.116.7:4011 , piece_id=57KLGTVQH77IOCAQPMQA66CPZPLEDKUIO4JKMUP6BOD4XSMPB5BQ, seg_pos={0 0}, upload=113588]: Uploading piece... Aug 4 16:10:24.054251246 (118h13m13.991909755s) [noise=true , address=77.90.31.10:4519 , piece_id=7HTU6S7JCLTAA7MMAYQBQP6F2PWHU3X7BU2NPHUUT2Q2MPFTLRWA, seg_pos={0 0}, upload=113588]: Uploading piece... Aug 4 16:10:24.054636593 (118h13m13.992295092s) [noise=true , address=45.91.116.8:4048 , piece_id=JGBCOCNYT4IZUJ6BCI3TEX7E46PYWQSS6HHM67YCOXQ3W5L4H5OA, seg_pos={0 0}, upload=113588]: Uploading piece... Aug 4 16:10:25.788117314 (118h13m15.725775813s) [upload=113588]: Next returned nil segment Aug 4 16:10:25.788134928 (118h13m15.725793427s) [upload=113588]: Waiting for error group managing segments... Aug 4 16:10:26.080627835 (118h13m16.018286335s) [noise=true , address=77.90.31.16:4735 , piece_id=XET5ROIUKXDV7MNQBYU2JPXLWHH74RD525PMYWC7VS7GZABL5P7A, seg_pos={0 0}, upload=113588]: Done uploading piece. err: <nil> Aug 4 16:10:26.081237151 (118h13m16.01889565s) [noise=true , address=77.90.31.21:4877 , piece_id=GNPOUGIJ42Y47VOZ5EMFVIDEFAY5B3KQQPEZ3LS7XYBDTXBZCLKQ, seg_pos={0 0}, upload=113588]: Done uploading piece. err: <nil> Aug 4 16:10:26.082642120 (118h13m16.020300619s) [noise=true , address=77.90.31.18:4805 , piece_id=OW2WLOK6KW2WUGVFEDEHMTUBSSUGGH4S5NQTYXKNOVMDXRYWHSEQ, seg_pos={0 0}, upload=113588]: Done uploading piece. err: <nil> Aug 4 16:10:26.090023893 (118h13m16.027682392s) [noise=true , address=45.91.116.11:4119 , piece_id=WBFTLEUFEXKAR2U5SZLSOHPZEGV3QNWDLVJBL36XP2GEAK5L7RUA, seg_pos={0 0}, upload=113588]: Done uploading piece. err: <nil> Aug 4 16:10:26.092155202 (118h13m16.029813701s) [noise=true , address=45.91.116.12:4225 , piece_id=3M2NI24IGXUHSDASHFYXDR2QPTNGEAQFT22AHIUAHVK2JVJNDQXQ, seg_pos={0 0}, upload=113588]: Done uploading piece. err: <nil> Aug 4 16:10:26.093301104 (118h13m16.030959593s) [noise=true , address=45.91.116.4:4938 , piece_id=SHP5IWA4HZTAAECBH2VRW525XCY6BVWIJOHE3U7WAD4ISJ63WUVA, seg_pos={0 0}, upload=113588]: Done uploading piece. err: <nil> Aug 4 16:10:26.096255166 (118h13m16.033913675s) [noise=true , address=104.36.82.6:4876 , piece_id=EWMXECZHZILAGIYJ5QJGYIGZSN4XP4LAD2PFY5CQHVARTAGTU5UA, seg_pos={0 0}, upload=113588]: Done uploading piece. err: <nil> Aug 4 16:10:26.112849236 (118h13m16.050507735s) [noise=true , address=104.36.82.16:4293 , piece_id=UUQ6WYMZMYI6PYRGK5IK2OE6CX2AHAW45ZRCPA7LK3X2IIIM3ZJA, seg_pos={0 0}, upload=113588]: Done uploading piece. err: <nil> Aug 4 16:10:26.171272489 (118h13m16.108930988s) [noise=true , address=77.90.31.12:4589 , piece_id=BFYZ6N2AZEFVYNS4EQYPVKNLUIANMXZ7MPDSGYA5LB2XP2REAJUQ, seg_pos={0 0}, upload=113588]: Done uploading piece. err: <nil> Aug 4 16:10:28.145791810 (118h13m18.083450309s) [noise=true , address=77.90.31.13:4627 , piece_id=E5LXNHOSI5URSZV7OBEWIQ75CEQWXDN2DLW4N2PUSY6MFHCNQ7YA, seg_pos={0 0}, upload=113588]: Done uploading piece. err: <nil> Aug 4 16:10:31.180407751 (118h13m21.11806625s) [noise=true , address=77.90.31.10:4519 , piece_id=7HTU6S7JCLTAA7MMAYQBQP6F2PWHU3X7BU2NPHUUT2Q2MPFTLRWA, seg_pos={0 0}, upload=113588]: Done uploading piece. err: <nil> Aug 4 16:10:53.161543551 (118h13m43.099202051s) [noise=true , address=104.36.82.7:4910 , piece_id=ZBYQ4SELARIGNS5Q3FBOOS3QIRVQ23VDCHMDK3TBLU5E7VREUPBA, seg_pos={0 0}, upload=113588]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: write tcp 188.166.225.170:52204->104.36.82.7:4910: use of closed network connection; write tcp 188.166.225.170:52204->104.36.82.7:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52204->104.36.82.7:4910: use of closed network connection Aug 4 16:10:53.161605039 (118h13m43.099263528s) [noise=true , address=104.36.82.10:4077 , piece_id=UNAQGJCBMXKJIOAHLHZSG4RYC2UDR3B5OXAOUTI7FD5CXLFXR2UQ, seg_pos={0 0}, upload=113588]: Done uploading piece. err: ecclient: upload failed (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk, address:104.36.82.10:4077): protocol: write tcp 188.166.225.170:41132->104.36.82.10:4077: use of closed network connection; write tcp 188.166.225.170:41132->104.36.82.10:4077: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41132->104.36.82.10:4077: use of closed network connection Aug 4 16:10:53.161771859 (118h13m43.099430348s) [noise=true , address=45.91.116.8:4048 , piece_id=JGBCOCNYT4IZUJ6BCI3TEX7E46PYWQSS6HHM67YCOXQ3W5L4H5OA, seg_pos={0 0}, upload=113588]: Done uploading piece. err: ecclient: upload failed (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, address:45.91.116.8:4048): protocol: write tcp 188.166.225.170:35774->45.91.116.8:4048: use of closed network connection; write tcp 188.166.225.170:35774->45.91.116.8:4048: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35774->45.91.116.8:4048: use of closed network connection Aug 4 16:10:53.161844948 (118h13m43.099503447s) [noise=true , address=45.91.116.7:4011 , piece_id=57KLGTVQH77IOCAQPMQA66CPZPLEDKUIO4JKMUP6BOD4XSMPB5BQ, seg_pos={0 0}, upload=113588]: Done uploading piece. err: ecclient: upload failed (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, address:45.91.116.7:4011): protocol: write tcp 188.166.225.170:50254->45.91.116.7:4011: use of closed network connection; write tcp 188.166.225.170:50254->45.91.116.7:4011: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50254->45.91.116.7:4011: use of closed network connection Aug 4 16:10:53.161881499 (118h13m43.099539998s) [noise=true , address=45.91.116.10:4082 , piece_id=GKHLII3XXM72YPTIZF5ZFZESHXZQNB56E3CVZFAEXJC2WX6URD5A, seg_pos={0 0}, upload=113588]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: read tcp 188.166.225.170:44088->45.91.116.10:4082: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 4 16:10:53.161905614 (118h13m43.099564103s) [seg_pos={0 0}, upload=113588]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Aug 4 16:10:53.443651470 (118h13m43.381309969s) [upload=113588]: Done uploading segments... Aug 4 16:10:53.443675736 (118h13m43.381334235s) [upload=113588]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 7.208791532s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [........X.............................X.....................] duration: 1.847083012s FAIL (2025-08-04 16:12:48 UTC) orbiter warning: Operation: DownloadObject uplink: object not found ("orbiter_us_0/1K") Duration: 1.847111285s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 6.796242131s FAIL (2025-08-04 16:12:19 UTC) orbiter warning: Operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 4.36.82.36:4625 , piece_id=FCK7QGPSKJ7VDK4D7BVNF44BRPQCQS5V5ICAJHELAQHDBSQV5R3Q, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.146274823 (118h10m17.083933322s) [noise=true , address=45.91.116.7:4026 , piece_id=CB6TYC4A6DQOFFKVFPNMF5MH5MGMJHU5Z25B4QXQJI5ETDWLQJJQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.146672965 (118h10m17.084331464s) [noise=true , address=104.36.82.16:4321 , piece_id=42MZPFPEBQNQRJUHQJLZQP2UWGDV2OAXT6Z4KTWPPWLALISASN5Q, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.146726166 (118h10m17.084384665s) [noise=true , address=104.36.82.55:4206 , piece_id=KSAKYO6MALVA5REEKKJZ7MGX7YIRC2PDI5PNHFHRASPBD47V5SOQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.147136371 (118h10m17.08479488s) [noise=true , address=104.36.82.42:4823 , piece_id=66Q6G7YBCVUEAHU3Y5LHPR5MIT7H7NQWI2TWWNHKZT5FYS4DNQBA, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.147479628 (118h10m17.085138127s) [noise=true , address=193.149.164.143:4695 , piece_id=3QTYKOTBCOJ4CRM76FR2XKONKQVYTQQMPTRH3E6MYJR4ODBPY3AQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.147821592 (118h10m17.085480091s) [noise=true , address=104.36.82.62:4411 , piece_id=CIJJ4SMWKFDOIZZ5UEYIXFZHUNAZKRMM2P36IJIT5BHUX5STDZ4A, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.148134220 (118h10m17.085792719s) [noise=true , address=104.36.82.55:4224 , piece_id=NJYMQ2467DXHUEC6ELXITFJF6WWKM4XFEONRJVRF6QSJBDEMC7JA, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.148548573 (118h10m17.086207072s) [noise=true , address=104.36.82.13:4193 , piece_id=WJA72IWFD3TUA2WPGJGB6A7AHJXI4PGOK62YDMKSMDPUAVHUHW4Q, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.148889576 (118h10m17.086548085s) [noise=true , address=104.36.82.39:4728 , piece_id=ZCL6YOUXY4P6HTYUTTILZR4TLG44CDTXWLOJJWOHPFSORA2IBCBA, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.149205931 (118h10m17.08686443s) [noise=true , address=104.36.82.59:4326 , piece_id=CS7FVO3IRXKZFB2EZM2HQBVEVM4XTTFI5Y7TH3NJZGIYMREVY7UQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.149509752 (118h10m17.087168261s) [noise=true , address=104.36.82.45:4930 , piece_id=LBMDUGYMQI7PBBGEOT22S7OGNVRIYS66USSYFAZOSGI445GWKCYA, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.149866545 (118h10m17.087525044s) [noise=true , address=104.36.82.23:4474 , piece_id=PLEKQKMEQKQDW72BNLLY4JOOPFI6CPNNGT74M5EFFNDQQFWELXUQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.150193601 (118h10m17.08785211s) [noise=true , address=104.36.82.13:4212 , piece_id=ER6VIEGFSG2F7542M5P66RS3QAUQYV7BWPQ4YOX55Q3XX6DDCTGA, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.150515557 (118h10m17.088174066s) [noise=true , address=104.36.82.46:4485 , piece_id=P4FMDR5NCQOLKSXQBLE7I7BPEZJ2URAPPVNKOR53ZPROBCE4YAJQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.150935149 (118h10m17.088593648s) [noise=true , address=104.36.82.54:4181 , piece_id=AHW774W42KLBATPDVRQRKH4MQWV5YP2XEEUY6VASDMYJ2BKXCS2Q, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.151249151 (118h10m17.08890765s) [noise=true , address=104.36.82.44:4886 , piece_id=Q3BIO4GNMY64IWXC7VZ5BAYKYRJJ3GKFV7Z4IOG7SP62IUJHXN2A, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.151276363 (118h10m17.088934862s) [noise=true , address=104.36.82.9:4019 , piece_id=SBCPU6XTMB4CRRTFKDBYFQQ2XZZNE3I6AF7VVSGYC356SPQNVHUQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:28.640156348 (118h10m18.577814857s) [upload=113540]: Next returned nil segment Aug 4 16:07:28.640223306 (118h10m18.577881825s) [upload=113540]: Waiting for error group managing segments... Aug 4 16:07:29.062127352 (118h10m18.999785841s) [noise=true , address=104.36.82.16:4321 , piece_id=42MZPFPEBQNQRJUHQJLZQP2UWGDV2OAXT6Z4KTWPPWLALISASN5Q, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.080565981 (118h10m19.01822448s) [noise=true , address=104.36.82.54:4181 , piece_id=AHW774W42KLBATPDVRQRKH4MQWV5YP2XEEUY6VASDMYJ2BKXCS2Q, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.081329852 (118h10m19.018988352s) [noise=true , address=104.36.82.62:4411 , piece_id=CIJJ4SMWKFDOIZZ5UEYIXFZHUNAZKRMM2P36IJIT5BHUX5STDZ4A, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.081846201 (118h10m19.0195047s) [noise=true , address=104.36.82.55:4224 , piece_id=NJYMQ2467DXHUEC6ELXITFJF6WWKM4XFEONRJVRF6QSJBDEMC7JA, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.091017397 (118h10m19.028675896s) [noise=true , address=104.36.82.46:4485 , piece_id=P4FMDR5NCQOLKSXQBLE7I7BPEZJ2URAPPVNKOR53ZPROBCE4YAJQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.105092769 (118h10m19.042751268s) [noise=true , address=104.36.82.13:4212 , piece_id=ER6VIEGFSG2F7542M5P66RS3QAUQYV7BWPQ4YOX55Q3XX6DDCTGA, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.107141759 (118h10m19.044800248s) [noise=true , address=104.36.82.23:4474 , piece_id=PLEKQKMEQKQDW72BNLLY4JOOPFI6CPNNGT74M5EFFNDQQFWELXUQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.116502399 (118h10m19.054160898s) [noise=true , address=45.91.116.7:4026 , piece_id=CB6TYC4A6DQOFFKVFPNMF5MH5MGMJHU5Z25B4QXQJI5ETDWLQJJQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.153680253 (118h10m19.091338752s) [noise=true , address=104.36.82.9:4019 , piece_id=SBCPU6XTMB4CRRTFKDBYFQQ2XZZNE3I6AF7VVSGYC356SPQNVHUQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.210191177 (118h10m19.147849686s) [noise=true , address=104.36.82.36:4625 , piece_id=FCK7QGPSKJ7VDK4D7BVNF44BRPQCQS5V5ICAJHELAQHDBSQV5R3Q, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.219317350 (118h10m19.156975849s) [noise=true , address=104.36.82.55:4206 , piece_id=KSAKYO6MALVA5REEKKJZ7MGX7YIRC2PDI5PNHFHRASPBD47V5SOQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:31.340384557 (118h10m21.278043046s) [noise=true , address=104.36.82.59:4326 , piece_id=CS7FVO3IRXKZFB2EZM2HQBVEVM4XTTFI5Y7TH3NJZGIYMREVY7UQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:32.464238230 (118h10m22.401896729s) [noise=true , address=104.36.82.13:4193 , piece_id=WJA72IWFD3TUA2WPGJGB6A7AHJXI4PGOK62YDMKSMDPUAVHUHW4Q, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:08:26.041033880 (118h11m15.978692379s) [noise=true , address=193.149.164.143:4695 , piece_id=3QTYKOTBCOJ4CRM76FR2XKONKQVYTQQMPTRH3E6MYJR4ODBPY3AQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: ecclient: upload failed (node:12TZfgz24gnZr4UryzEKw6gu94njnmCvafX5X64MKFSJqaUWSVh, address:193.149.164.143:4695): protocol: write tcp 188.166.225.170:43298->193.149.164.143:4695: use of closed network connection; write tcp 188.166.225.170:43298->193.149.164.143:4695: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43298->193.149.164.143:4695: use of closed network connection Aug 4 16:08:26.041079005 (118h11m15.978737504s) [noise=true , address=104.36.82.44:4886 , piece_id=Q3BIO4GNMY64IWXC7VZ5BAYKYRJJ3GKFV7Z4IOG7SP62IUJHXN2A, seg_pos={0 0}, upload=113540]: Done uploading piece. err: ecclient: upload failed (node:1n2cXhqd5EWbNPTXem83jQUjYq5ZaGKBidbVCQzxYfR3rWddGR, address:104.36.82.44:4886): protocol: write tcp 188.166.225.170:38030->104.36.82.44:4886: use of closed network connection; write tcp 188.166.225.170:38030->104.36.82.44:4886: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38030->104.36.82.44:4886: use of closed network connection Aug 4 16:08:26.041128580 (118h11m15.97878711s) [noise=true , address=104.36.82.39:4728 , piece_id=ZCL6YOUXY4P6HTYUTTILZR4TLG44CDTXWLOJJWOHPFSORA2IBCBA, seg_pos={0 0}, upload=113540]: Done uploading piece. err: ecclient: upload failed (node:1YoLzpb4mMYZ8TsdxnMocekZZ1gLfTrdCM8mxTDj4tY6khZGvi, address:104.36.82.39:4728): protocol: write tcp 188.166.225.170:34326->104.36.82.39:4728: use of closed network connection; write tcp 188.166.225.170:34326->104.36.82.39:4728: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34326->104.36.82.39:4728: use of closed network connection Aug 4 16:08:26.041195678 (118h11m15.978854187s) [noise=true , address=104.36.82.45:4930 , piece_id=LBMDUGYMQI7PBBGEOT22S7OGNVRIYS66USSYFAZOSGI445GWKCYA, seg_pos={0 0}, upload=113540]: Done uploading piece. err: ecclient: upload failed (node:12qaYFSpezuLAc5zJfAYruzad6nTpYFDpog7bmKMVTA8MVm76NB, address:104.36.82.45:4930): protocol: expected piece hash; context deadline exceeded; EOF Aug 4 16:08:26.041203374 (118h11m15.978861913s) [noise=true , address=104.36.82.42:4823 , piece_id=66Q6G7YBCVUEAHU3Y5LHPR5MIT7H7NQWI2TWWNHKZT5FYS4DNQBA, seg_pos={0 0}, upload=113540]: Done uploading piece. err: ecclient: upload failed (node:1fkA5DEj9RqsR6N85z7azvujy8tPkUhB5GCqBpUmFb4vvwbd8y, address:104.36.82.42:4823): protocol: write tcp 188.166.225.170:37230->104.36.82.42:4823: use of closed network connection; write tcp 188.166.225.170:37230->104.36.82.42:4823: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37230->104.36.82.42:4823: use of closed network connection Aug 4 16:08:26.041258960 (118h11m15.978917489s) [seg_pos={0 0}, upload=113540]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Aug 4 16:08:26.274088873 (118h11m16.211747372s) [upload=113540]: Done uploading segments... Aug 4 16:08:26.274113650 (118h11m16.211772139s) [upload=113540]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 6.796278991s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [........................................................XX..] duration: 6.432923252s FAIL (2025-08-04 16:12:18 UTC) orbiter warning: Operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== 16569229 (118h13m13.754227758s) [upload=113588]: Done starting upload Aug 4 16:10:23.816697144 (118h13m13.754355643s) [upload=113588]: Uploading segments... Aug 4 16:10:23.816823747 (118h13m13.754482246s) [upload=113588]: Got next segment. Inline: false Aug 4 16:10:23.816875515 (118h13m13.754534014s) [upload=113588]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 4 16:10:23.816882549 (118h13m13.754541068s) [upload=113588]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 4 16:10:24.051702429 (118h13m13.989360928s) [seg_pos={0 0}, upload=113588]: Begin upload segment... Aug 4 16:10:24.051745952 (118h13m13.989404451s) [seg_pos={0 0}, upload=113588]: Done begin upload segment. Aug 4 16:10:24.051767884 (118h13m13.989426383s) [noise=true , address=104.36.82.16:4293 , piece_id=UUQ6WYMZMYI6PYRGK5IK2OE6CX2AHAW45ZRCPA7LK3X2IIIM3ZJA, seg_pos={0 0}, upload=113588]: Uploading piece... Aug 4 16:10:24.051841024 (118h13m13.989499533s) [noise=true , address=104.36.82.7:4910 , piece_id=ZBYQ4SELARIGNS5Q3FBOOS3QIRVQ23VDCHMDK3TBLU5E7VREUPBA, seg_pos={0 0}, upload=113588]: Uploading piece... Aug 4 16:10:24.052106963 (118h13m13.989765462s) [noise=true , address=77.90.31.18:4805 , piece_id=OW2WLOK6KW2WUGVFEDEHMTUBSSUGGH4S5NQTYXKNOVMDXRYWHSEQ, seg_pos={0 0}, upload=113588]: Uploading piece... Aug 4 16:10:24.052435702 (118h13m13.990094201s) [noise=true , address=77.90.31.21:4877 , piece_id=GNPOUGIJ42Y47VOZ5EMFVIDEFAY5B3KQQPEZ3LS7XYBDTXBZCLKQ, seg_pos={0 0}, upload=113588]: Uploading piece... Aug 4 16:10:24.052526446 (118h13m13.990184945s) [noise=true , address=45.91.116.12:4225 , piece_id=3M2NI24IGXUHSDASHFYXDR2QPTNGEAQFT22AHIUAHVK2JVJNDQXQ, seg_pos={0 0}, upload=113588]: Uploading piece... Aug 4 16:10:24.052684418 (118h13m13.990342907s) [noise=true , address=45.91.116.10:4082 , piece_id=GKHLII3XXM72YPTIZF5ZFZESHXZQNB56E3CVZFAEXJC2WX6URD5A, seg_pos={0 0}, upload=113588]: Uploading piece... Aug 4 16:10:24.052926030 (118h13m13.990584529s) [noise=true , address=104.36.82.10:4077 , piece_id=UNAQGJCBMXKJIOAHLHZSG4RYC2UDR3B5OXAOUTI7FD5CXLFXR2UQ, seg_pos={0 0}, upload=113588]: Uploading piece... Aug 4 16:10:24.052950747 (118h13m13.990609246s) [noise=true , address=45.91.116.4:4938 , piece_id=SHP5IWA4HZTAAECBH2VRW525XCY6BVWIJOHE3U7WAD4ISJ63WUVA, seg_pos={0 0}, upload=113588]: Uploading piece... Aug 4 16:10:24.053160659 (118h13m13.990819158s) [noise=true , address=45.91.116.11:4119 , piece_id=WBFTLEUFEXKAR2U5SZLSOHPZEGV3QNWDLVJBL36XP2GEAK5L7RUA, seg_pos={0 0}, upload=113588]: Uploading piece... Aug 4 16:10:24.053383786 (118h13m13.991042285s) [noise=true , address=77.90.31.12:4589 , piece_id=BFYZ6N2AZEFVYNS4EQYPVKNLUIANMXZ7MPDSGYA5LB2XP2REAJUQ, seg_pos={0 0}, upload=113588]: Uploading piece... Aug 4 16:10:24.053410627 (118h13m13.991069126s) [noise=true , address=77.90.31.16:4735 , piece_id=XET5ROIUKXDV7MNQBYU2JPXLWHH74RD525PMYWC7VS7GZABL5P7A, seg_pos={0 0}, upload=113588]: Uploading piece... Aug 4 16:10:24.053613205 (118h13m13.991271704s) [noise=true , address=77.90.31.13:4627 , piece_id=E5LXNHOSI5URSZV7OBEWIQ75CEQWXDN2DLW4N2PUSY6MFHCNQ7YA, seg_pos={0 0}, upload=113588]: Uploading piece... Aug 4 16:10:24.053834499 (118h13m13.991492998s) [noise=true , address=104.36.82.6:4876 , piece_id=EWMXECZHZILAGIYJ5QJGYIGZSN4XP4LAD2PFY5CQHVARTAGTU5UA, seg_pos={0 0}, upload=113588]: Uploading piece... Aug 4 16:10:24.053873513 (118h13m13.991532012s) [noise=true , address=45.91.116.7:4011 , piece_id=57KLGTVQH77IOCAQPMQA66CPZPLEDKUIO4JKMUP6BOD4XSMPB5BQ, seg_pos={0 0}, upload=113588]: Uploading piece... Aug 4 16:10:24.054251246 (118h13m13.991909755s) [noise=true , address=77.90.31.10:4519 , piece_id=7HTU6S7JCLTAA7MMAYQBQP6F2PWHU3X7BU2NPHUUT2Q2MPFTLRWA, seg_pos={0 0}, upload=113588]: Uploading piece... Aug 4 16:10:24.054636593 (118h13m13.992295092s) [noise=true , address=45.91.116.8:4048 , piece_id=JGBCOCNYT4IZUJ6BCI3TEX7E46PYWQSS6HHM67YCOXQ3W5L4H5OA, seg_pos={0 0}, upload=113588]: Uploading piece... Aug 4 16:10:25.788117314 (118h13m15.725775813s) [upload=113588]: Next returned nil segment Aug 4 16:10:25.788134928 (118h13m15.725793427s) [upload=113588]: Waiting for error group managing segments... Aug 4 16:10:26.080627835 (118h13m16.018286335s) [noise=true , address=77.90.31.16:4735 , piece_id=XET5ROIUKXDV7MNQBYU2JPXLWHH74RD525PMYWC7VS7GZABL5P7A, seg_pos={0 0}, upload=113588]: Done uploading piece. err: <nil> Aug 4 16:10:26.081237151 (118h13m16.01889565s) [noise=true , address=77.90.31.21:4877 , piece_id=GNPOUGIJ42Y47VOZ5EMFVIDEFAY5B3KQQPEZ3LS7XYBDTXBZCLKQ, seg_pos={0 0}, upload=113588]: Done uploading piece. err: <nil> Aug 4 16:10:26.082642120 (118h13m16.020300619s) [noise=true , address=77.90.31.18:4805 , piece_id=OW2WLOK6KW2WUGVFEDEHMTUBSSUGGH4S5NQTYXKNOVMDXRYWHSEQ, seg_pos={0 0}, upload=113588]: Done uploading piece. err: <nil> Aug 4 16:10:26.090023893 (118h13m16.027682392s) [noise=true , address=45.91.116.11:4119 , piece_id=WBFTLEUFEXKAR2U5SZLSOHPZEGV3QNWDLVJBL36XP2GEAK5L7RUA, seg_pos={0 0}, upload=113588]: Done uploading piece. err: <nil> Aug 4 16:10:26.092155202 (118h13m16.029813701s) [noise=true , address=45.91.116.12:4225 , piece_id=3M2NI24IGXUHSDASHFYXDR2QPTNGEAQFT22AHIUAHVK2JVJNDQXQ, seg_pos={0 0}, upload=113588]: Done uploading piece. err: <nil> Aug 4 16:10:26.093301104 (118h13m16.030959593s) [noise=true , address=45.91.116.4:4938 , piece_id=SHP5IWA4HZTAAECBH2VRW525XCY6BVWIJOHE3U7WAD4ISJ63WUVA, seg_pos={0 0}, upload=113588]: Done uploading piece. err: <nil> Aug 4 16:10:26.096255166 (118h13m16.033913675s) [noise=true , address=104.36.82.6:4876 , piece_id=EWMXECZHZILAGIYJ5QJGYIGZSN4XP4LAD2PFY5CQHVARTAGTU5UA, seg_pos={0 0}, upload=113588]: Done uploading piece. err: <nil> Aug 4 16:10:26.112849236 (118h13m16.050507735s) [noise=true , address=104.36.82.16:4293 , piece_id=UUQ6WYMZMYI6PYRGK5IK2OE6CX2AHAW45ZRCPA7LK3X2IIIM3ZJA, seg_pos={0 0}, upload=113588]: Done uploading piece. err: <nil> Aug 4 16:10:26.171272489 (118h13m16.108930988s) [noise=true , address=77.90.31.12:4589 , piece_id=BFYZ6N2AZEFVYNS4EQYPVKNLUIANMXZ7MPDSGYA5LB2XP2REAJUQ, seg_pos={0 0}, upload=113588]: Done uploading piece. err: <nil> Aug 4 16:10:28.145791810 (118h13m18.083450309s) [noise=true , address=77.90.31.13:4627 , piece_id=E5LXNHOSI5URSZV7OBEWIQ75CEQWXDN2DLW4N2PUSY6MFHCNQ7YA, seg_pos={0 0}, upload=113588]: Done uploading piece. err: <nil> Aug 4 16:10:31.180407751 (118h13m21.11806625s) [noise=true , address=77.90.31.10:4519 , piece_id=7HTU6S7JCLTAA7MMAYQBQP6F2PWHU3X7BU2NPHUUT2Q2MPFTLRWA, seg_pos={0 0}, upload=113588]: Done uploading piece. err: <nil> Aug 4 16:10:53.161543551 (118h13m43.099202051s) [noise=true , address=104.36.82.7:4910 , piece_id=ZBYQ4SELARIGNS5Q3FBOOS3QIRVQ23VDCHMDK3TBLU5E7VREUPBA, seg_pos={0 0}, upload=113588]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: write tcp 188.166.225.170:52204->104.36.82.7:4910: use of closed network connection; write tcp 188.166.225.170:52204->104.36.82.7:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52204->104.36.82.7:4910: use of closed network connection Aug 4 16:10:53.161605039 (118h13m43.099263528s) [noise=true , address=104.36.82.10:4077 , piece_id=UNAQGJCBMXKJIOAHLHZSG4RYC2UDR3B5OXAOUTI7FD5CXLFXR2UQ, seg_pos={0 0}, upload=113588]: Done uploading piece. err: ecclient: upload failed (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk, address:104.36.82.10:4077): protocol: write tcp 188.166.225.170:41132->104.36.82.10:4077: use of closed network connection; write tcp 188.166.225.170:41132->104.36.82.10:4077: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41132->104.36.82.10:4077: use of closed network connection Aug 4 16:10:53.161771859 (118h13m43.099430348s) [noise=true , address=45.91.116.8:4048 , piece_id=JGBCOCNYT4IZUJ6BCI3TEX7E46PYWQSS6HHM67YCOXQ3W5L4H5OA, seg_pos={0 0}, upload=113588]: Done uploading piece. err: ecclient: upload failed (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, address:45.91.116.8:4048): protocol: write tcp 188.166.225.170:35774->45.91.116.8:4048: use of closed network connection; write tcp 188.166.225.170:35774->45.91.116.8:4048: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35774->45.91.116.8:4048: use of closed network connection Aug 4 16:10:53.161844948 (118h13m43.099503447s) [noise=true , address=45.91.116.7:4011 , piece_id=57KLGTVQH77IOCAQPMQA66CPZPLEDKUIO4JKMUP6BOD4XSMPB5BQ, seg_pos={0 0}, upload=113588]: Done uploading piece. err: ecclient: upload failed (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, address:45.91.116.7:4011): protocol: write tcp 188.166.225.170:50254->45.91.116.7:4011: use of closed network connection; write tcp 188.166.225.170:50254->45.91.116.7:4011: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50254->45.91.116.7:4011: use of closed network connection Aug 4 16:10:53.161881499 (118h13m43.099539998s) [noise=true , address=45.91.116.10:4082 , piece_id=GKHLII3XXM72YPTIZF5ZFZESHXZQNB56E3CVZFAEXJC2WX6URD5A, seg_pos={0 0}, upload=113588]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: read tcp 188.166.225.170:44088->45.91.116.10:4082: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 4 16:10:53.161905614 (118h13m43.099564103s) [seg_pos={0 0}, upload=113588]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Aug 4 16:10:53.443651470 (118h13m43.381309969s) [upload=113588]: Done uploading segments... Aug 4 16:10:53.443675736 (118h13m43.381334235s) [upload=113588]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 6.432949421s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.......................................................XX.X.] duration: 11.969507416s FAIL (2025-08-04 16:11:59 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 11.969528348s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [.........X.............................X....................] duration: 1.863400463s FAIL (2025-08-04 16:11:52 UTC) orbiter warning: Operation: DownloadObject uplink: object not found ("orbiter_us_0/1K") Duration: 1.863462151s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [........................................................XX.X] duration: 30.001207129s FAIL (2025-08-04 16:11:42 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.001225325s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 7.653709151s FAIL (2025-08-04 16:11:37 UTC) orbiter warning: Operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 4.36.82.36:4625 , piece_id=FCK7QGPSKJ7VDK4D7BVNF44BRPQCQS5V5ICAJHELAQHDBSQV5R3Q, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.146274823 (118h10m17.083933322s) [noise=true , address=45.91.116.7:4026 , piece_id=CB6TYC4A6DQOFFKVFPNMF5MH5MGMJHU5Z25B4QXQJI5ETDWLQJJQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.146672965 (118h10m17.084331464s) [noise=true , address=104.36.82.16:4321 , piece_id=42MZPFPEBQNQRJUHQJLZQP2UWGDV2OAXT6Z4KTWPPWLALISASN5Q, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.146726166 (118h10m17.084384665s) [noise=true , address=104.36.82.55:4206 , piece_id=KSAKYO6MALVA5REEKKJZ7MGX7YIRC2PDI5PNHFHRASPBD47V5SOQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.147136371 (118h10m17.08479488s) [noise=true , address=104.36.82.42:4823 , piece_id=66Q6G7YBCVUEAHU3Y5LHPR5MIT7H7NQWI2TWWNHKZT5FYS4DNQBA, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.147479628 (118h10m17.085138127s) [noise=true , address=193.149.164.143:4695 , piece_id=3QTYKOTBCOJ4CRM76FR2XKONKQVYTQQMPTRH3E6MYJR4ODBPY3AQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.147821592 (118h10m17.085480091s) [noise=true , address=104.36.82.62:4411 , piece_id=CIJJ4SMWKFDOIZZ5UEYIXFZHUNAZKRMM2P36IJIT5BHUX5STDZ4A, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.148134220 (118h10m17.085792719s) [noise=true , address=104.36.82.55:4224 , piece_id=NJYMQ2467DXHUEC6ELXITFJF6WWKM4XFEONRJVRF6QSJBDEMC7JA, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.148548573 (118h10m17.086207072s) [noise=true , address=104.36.82.13:4193 , piece_id=WJA72IWFD3TUA2WPGJGB6A7AHJXI4PGOK62YDMKSMDPUAVHUHW4Q, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.148889576 (118h10m17.086548085s) [noise=true , address=104.36.82.39:4728 , piece_id=ZCL6YOUXY4P6HTYUTTILZR4TLG44CDTXWLOJJWOHPFSORA2IBCBA, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.149205931 (118h10m17.08686443s) [noise=true , address=104.36.82.59:4326 , piece_id=CS7FVO3IRXKZFB2EZM2HQBVEVM4XTTFI5Y7TH3NJZGIYMREVY7UQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.149509752 (118h10m17.087168261s) [noise=true , address=104.36.82.45:4930 , piece_id=LBMDUGYMQI7PBBGEOT22S7OGNVRIYS66USSYFAZOSGI445GWKCYA, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.149866545 (118h10m17.087525044s) [noise=true , address=104.36.82.23:4474 , piece_id=PLEKQKMEQKQDW72BNLLY4JOOPFI6CPNNGT74M5EFFNDQQFWELXUQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.150193601 (118h10m17.08785211s) [noise=true , address=104.36.82.13:4212 , piece_id=ER6VIEGFSG2F7542M5P66RS3QAUQYV7BWPQ4YOX55Q3XX6DDCTGA, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.150515557 (118h10m17.088174066s) [noise=true , address=104.36.82.46:4485 , piece_id=P4FMDR5NCQOLKSXQBLE7I7BPEZJ2URAPPVNKOR53ZPROBCE4YAJQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.150935149 (118h10m17.088593648s) [noise=true , address=104.36.82.54:4181 , piece_id=AHW774W42KLBATPDVRQRKH4MQWV5YP2XEEUY6VASDMYJ2BKXCS2Q, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.151249151 (118h10m17.08890765s) [noise=true , address=104.36.82.44:4886 , piece_id=Q3BIO4GNMY64IWXC7VZ5BAYKYRJJ3GKFV7Z4IOG7SP62IUJHXN2A, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.151276363 (118h10m17.088934862s) [noise=true , address=104.36.82.9:4019 , piece_id=SBCPU6XTMB4CRRTFKDBYFQQ2XZZNE3I6AF7VVSGYC356SPQNVHUQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:28.640156348 (118h10m18.577814857s) [upload=113540]: Next returned nil segment Aug 4 16:07:28.640223306 (118h10m18.577881825s) [upload=113540]: Waiting for error group managing segments... Aug 4 16:07:29.062127352 (118h10m18.999785841s) [noise=true , address=104.36.82.16:4321 , piece_id=42MZPFPEBQNQRJUHQJLZQP2UWGDV2OAXT6Z4KTWPPWLALISASN5Q, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.080565981 (118h10m19.01822448s) [noise=true , address=104.36.82.54:4181 , piece_id=AHW774W42KLBATPDVRQRKH4MQWV5YP2XEEUY6VASDMYJ2BKXCS2Q, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.081329852 (118h10m19.018988352s) [noise=true , address=104.36.82.62:4411 , piece_id=CIJJ4SMWKFDOIZZ5UEYIXFZHUNAZKRMM2P36IJIT5BHUX5STDZ4A, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.081846201 (118h10m19.0195047s) [noise=true , address=104.36.82.55:4224 , piece_id=NJYMQ2467DXHUEC6ELXITFJF6WWKM4XFEONRJVRF6QSJBDEMC7JA, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.091017397 (118h10m19.028675896s) [noise=true , address=104.36.82.46:4485 , piece_id=P4FMDR5NCQOLKSXQBLE7I7BPEZJ2URAPPVNKOR53ZPROBCE4YAJQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.105092769 (118h10m19.042751268s) [noise=true , address=104.36.82.13:4212 , piece_id=ER6VIEGFSG2F7542M5P66RS3QAUQYV7BWPQ4YOX55Q3XX6DDCTGA, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.107141759 (118h10m19.044800248s) [noise=true , address=104.36.82.23:4474 , piece_id=PLEKQKMEQKQDW72BNLLY4JOOPFI6CPNNGT74M5EFFNDQQFWELXUQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.116502399 (118h10m19.054160898s) [noise=true , address=45.91.116.7:4026 , piece_id=CB6TYC4A6DQOFFKVFPNMF5MH5MGMJHU5Z25B4QXQJI5ETDWLQJJQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.153680253 (118h10m19.091338752s) [noise=true , address=104.36.82.9:4019 , piece_id=SBCPU6XTMB4CRRTFKDBYFQQ2XZZNE3I6AF7VVSGYC356SPQNVHUQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.210191177 (118h10m19.147849686s) [noise=true , address=104.36.82.36:4625 , piece_id=FCK7QGPSKJ7VDK4D7BVNF44BRPQCQS5V5ICAJHELAQHDBSQV5R3Q, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.219317350 (118h10m19.156975849s) [noise=true , address=104.36.82.55:4206 , piece_id=KSAKYO6MALVA5REEKKJZ7MGX7YIRC2PDI5PNHFHRASPBD47V5SOQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:31.340384557 (118h10m21.278043046s) [noise=true , address=104.36.82.59:4326 , piece_id=CS7FVO3IRXKZFB2EZM2HQBVEVM4XTTFI5Y7TH3NJZGIYMREVY7UQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:32.464238230 (118h10m22.401896729s) [noise=true , address=104.36.82.13:4193 , piece_id=WJA72IWFD3TUA2WPGJGB6A7AHJXI4PGOK62YDMKSMDPUAVHUHW4Q, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:08:26.041033880 (118h11m15.978692379s) [noise=true , address=193.149.164.143:4695 , piece_id=3QTYKOTBCOJ4CRM76FR2XKONKQVYTQQMPTRH3E6MYJR4ODBPY3AQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: ecclient: upload failed (node:12TZfgz24gnZr4UryzEKw6gu94njnmCvafX5X64MKFSJqaUWSVh, address:193.149.164.143:4695): protocol: write tcp 188.166.225.170:43298->193.149.164.143:4695: use of closed network connection; write tcp 188.166.225.170:43298->193.149.164.143:4695: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43298->193.149.164.143:4695: use of closed network connection Aug 4 16:08:26.041079005 (118h11m15.978737504s) [noise=true , address=104.36.82.44:4886 , piece_id=Q3BIO4GNMY64IWXC7VZ5BAYKYRJJ3GKFV7Z4IOG7SP62IUJHXN2A, seg_pos={0 0}, upload=113540]: Done uploading piece. err: ecclient: upload failed (node:1n2cXhqd5EWbNPTXem83jQUjYq5ZaGKBidbVCQzxYfR3rWddGR, address:104.36.82.44:4886): protocol: write tcp 188.166.225.170:38030->104.36.82.44:4886: use of closed network connection; write tcp 188.166.225.170:38030->104.36.82.44:4886: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38030->104.36.82.44:4886: use of closed network connection Aug 4 16:08:26.041128580 (118h11m15.97878711s) [noise=true , address=104.36.82.39:4728 , piece_id=ZCL6YOUXY4P6HTYUTTILZR4TLG44CDTXWLOJJWOHPFSORA2IBCBA, seg_pos={0 0}, upload=113540]: Done uploading piece. err: ecclient: upload failed (node:1YoLzpb4mMYZ8TsdxnMocekZZ1gLfTrdCM8mxTDj4tY6khZGvi, address:104.36.82.39:4728): protocol: write tcp 188.166.225.170:34326->104.36.82.39:4728: use of closed network connection; write tcp 188.166.225.170:34326->104.36.82.39:4728: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34326->104.36.82.39:4728: use of closed network connection Aug 4 16:08:26.041195678 (118h11m15.978854187s) [noise=true , address=104.36.82.45:4930 , piece_id=LBMDUGYMQI7PBBGEOT22S7OGNVRIYS66USSYFAZOSGI445GWKCYA, seg_pos={0 0}, upload=113540]: Done uploading piece. err: ecclient: upload failed (node:12qaYFSpezuLAc5zJfAYruzad6nTpYFDpog7bmKMVTA8MVm76NB, address:104.36.82.45:4930): protocol: expected piece hash; context deadline exceeded; EOF Aug 4 16:08:26.041203374 (118h11m15.978861913s) [noise=true , address=104.36.82.42:4823 , piece_id=66Q6G7YBCVUEAHU3Y5LHPR5MIT7H7NQWI2TWWNHKZT5FYS4DNQBA, seg_pos={0 0}, upload=113540]: Done uploading piece. err: ecclient: upload failed (node:1fkA5DEj9RqsR6N85z7azvujy8tPkUhB5GCqBpUmFb4vvwbd8y, address:104.36.82.42:4823): protocol: write tcp 188.166.225.170:37230->104.36.82.42:4823: use of closed network connection; write tcp 188.166.225.170:37230->104.36.82.42:4823: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37230->104.36.82.42:4823: use of closed network connection Aug 4 16:08:26.041258960 (118h11m15.978917489s) [seg_pos={0 0}, upload=113540]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Aug 4 16:08:26.274088873 (118h11m16.211747372s) [upload=113540]: Done uploading segments... Aug 4 16:08:26.274113650 (118h11m16.211772139s) [upload=113540]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 7.653737916s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........................................................XX.] duration: 8.921603383s FAIL (2025-08-04 16:11:08 UTC) orbiter warning: Operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== 16569229 (118h13m13.754227758s) [upload=113588]: Done starting upload Aug 4 16:10:23.816697144 (118h13m13.754355643s) [upload=113588]: Uploading segments... Aug 4 16:10:23.816823747 (118h13m13.754482246s) [upload=113588]: Got next segment. Inline: false Aug 4 16:10:23.816875515 (118h13m13.754534014s) [upload=113588]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 4 16:10:23.816882549 (118h13m13.754541068s) [upload=113588]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 4 16:10:24.051702429 (118h13m13.989360928s) [seg_pos={0 0}, upload=113588]: Begin upload segment... Aug 4 16:10:24.051745952 (118h13m13.989404451s) [seg_pos={0 0}, upload=113588]: Done begin upload segment. Aug 4 16:10:24.051767884 (118h13m13.989426383s) [noise=true , address=104.36.82.16:4293 , piece_id=UUQ6WYMZMYI6PYRGK5IK2OE6CX2AHAW45ZRCPA7LK3X2IIIM3ZJA, seg_pos={0 0}, upload=113588]: Uploading piece... Aug 4 16:10:24.051841024 (118h13m13.989499533s) [noise=true , address=104.36.82.7:4910 , piece_id=ZBYQ4SELARIGNS5Q3FBOOS3QIRVQ23VDCHMDK3TBLU5E7VREUPBA, seg_pos={0 0}, upload=113588]: Uploading piece... Aug 4 16:10:24.052106963 (118h13m13.989765462s) [noise=true , address=77.90.31.18:4805 , piece_id=OW2WLOK6KW2WUGVFEDEHMTUBSSUGGH4S5NQTYXKNOVMDXRYWHSEQ, seg_pos={0 0}, upload=113588]: Uploading piece... Aug 4 16:10:24.052435702 (118h13m13.990094201s) [noise=true , address=77.90.31.21:4877 , piece_id=GNPOUGIJ42Y47VOZ5EMFVIDEFAY5B3KQQPEZ3LS7XYBDTXBZCLKQ, seg_pos={0 0}, upload=113588]: Uploading piece... Aug 4 16:10:24.052526446 (118h13m13.990184945s) [noise=true , address=45.91.116.12:4225 , piece_id=3M2NI24IGXUHSDASHFYXDR2QPTNGEAQFT22AHIUAHVK2JVJNDQXQ, seg_pos={0 0}, upload=113588]: Uploading piece... Aug 4 16:10:24.052684418 (118h13m13.990342907s) [noise=true , address=45.91.116.10:4082 , piece_id=GKHLII3XXM72YPTIZF5ZFZESHXZQNB56E3CVZFAEXJC2WX6URD5A, seg_pos={0 0}, upload=113588]: Uploading piece... Aug 4 16:10:24.052926030 (118h13m13.990584529s) [noise=true , address=104.36.82.10:4077 , piece_id=UNAQGJCBMXKJIOAHLHZSG4RYC2UDR3B5OXAOUTI7FD5CXLFXR2UQ, seg_pos={0 0}, upload=113588]: Uploading piece... Aug 4 16:10:24.052950747 (118h13m13.990609246s) [noise=true , address=45.91.116.4:4938 , piece_id=SHP5IWA4HZTAAECBH2VRW525XCY6BVWIJOHE3U7WAD4ISJ63WUVA, seg_pos={0 0}, upload=113588]: Uploading piece... Aug 4 16:10:24.053160659 (118h13m13.990819158s) [noise=true , address=45.91.116.11:4119 , piece_id=WBFTLEUFEXKAR2U5SZLSOHPZEGV3QNWDLVJBL36XP2GEAK5L7RUA, seg_pos={0 0}, upload=113588]: Uploading piece... Aug 4 16:10:24.053383786 (118h13m13.991042285s) [noise=true , address=77.90.31.12:4589 , piece_id=BFYZ6N2AZEFVYNS4EQYPVKNLUIANMXZ7MPDSGYA5LB2XP2REAJUQ, seg_pos={0 0}, upload=113588]: Uploading piece... Aug 4 16:10:24.053410627 (118h13m13.991069126s) [noise=true , address=77.90.31.16:4735 , piece_id=XET5ROIUKXDV7MNQBYU2JPXLWHH74RD525PMYWC7VS7GZABL5P7A, seg_pos={0 0}, upload=113588]: Uploading piece... Aug 4 16:10:24.053613205 (118h13m13.991271704s) [noise=true , address=77.90.31.13:4627 , piece_id=E5LXNHOSI5URSZV7OBEWIQ75CEQWXDN2DLW4N2PUSY6MFHCNQ7YA, seg_pos={0 0}, upload=113588]: Uploading piece... Aug 4 16:10:24.053834499 (118h13m13.991492998s) [noise=true , address=104.36.82.6:4876 , piece_id=EWMXECZHZILAGIYJ5QJGYIGZSN4XP4LAD2PFY5CQHVARTAGTU5UA, seg_pos={0 0}, upload=113588]: Uploading piece... Aug 4 16:10:24.053873513 (118h13m13.991532012s) [noise=true , address=45.91.116.7:4011 , piece_id=57KLGTVQH77IOCAQPMQA66CPZPLEDKUIO4JKMUP6BOD4XSMPB5BQ, seg_pos={0 0}, upload=113588]: Uploading piece... Aug 4 16:10:24.054251246 (118h13m13.991909755s) [noise=true , address=77.90.31.10:4519 , piece_id=7HTU6S7JCLTAA7MMAYQBQP6F2PWHU3X7BU2NPHUUT2Q2MPFTLRWA, seg_pos={0 0}, upload=113588]: Uploading piece... Aug 4 16:10:24.054636593 (118h13m13.992295092s) [noise=true , address=45.91.116.8:4048 , piece_id=JGBCOCNYT4IZUJ6BCI3TEX7E46PYWQSS6HHM67YCOXQ3W5L4H5OA, seg_pos={0 0}, upload=113588]: Uploading piece... Aug 4 16:10:25.788117314 (118h13m15.725775813s) [upload=113588]: Next returned nil segment Aug 4 16:10:25.788134928 (118h13m15.725793427s) [upload=113588]: Waiting for error group managing segments... Aug 4 16:10:26.080627835 (118h13m16.018286335s) [noise=true , address=77.90.31.16:4735 , piece_id=XET5ROIUKXDV7MNQBYU2JPXLWHH74RD525PMYWC7VS7GZABL5P7A, seg_pos={0 0}, upload=113588]: Done uploading piece. err: <nil> Aug 4 16:10:26.081237151 (118h13m16.01889565s) [noise=true , address=77.90.31.21:4877 , piece_id=GNPOUGIJ42Y47VOZ5EMFVIDEFAY5B3KQQPEZ3LS7XYBDTXBZCLKQ, seg_pos={0 0}, upload=113588]: Done uploading piece. err: <nil> Aug 4 16:10:26.082642120 (118h13m16.020300619s) [noise=true , address=77.90.31.18:4805 , piece_id=OW2WLOK6KW2WUGVFEDEHMTUBSSUGGH4S5NQTYXKNOVMDXRYWHSEQ, seg_pos={0 0}, upload=113588]: Done uploading piece. err: <nil> Aug 4 16:10:26.090023893 (118h13m16.027682392s) [noise=true , address=45.91.116.11:4119 , piece_id=WBFTLEUFEXKAR2U5SZLSOHPZEGV3QNWDLVJBL36XP2GEAK5L7RUA, seg_pos={0 0}, upload=113588]: Done uploading piece. err: <nil> Aug 4 16:10:26.092155202 (118h13m16.029813701s) [noise=true , address=45.91.116.12:4225 , piece_id=3M2NI24IGXUHSDASHFYXDR2QPTNGEAQFT22AHIUAHVK2JVJNDQXQ, seg_pos={0 0}, upload=113588]: Done uploading piece. err: <nil> Aug 4 16:10:26.093301104 (118h13m16.030959593s) [noise=true , address=45.91.116.4:4938 , piece_id=SHP5IWA4HZTAAECBH2VRW525XCY6BVWIJOHE3U7WAD4ISJ63WUVA, seg_pos={0 0}, upload=113588]: Done uploading piece. err: <nil> Aug 4 16:10:26.096255166 (118h13m16.033913675s) [noise=true , address=104.36.82.6:4876 , piece_id=EWMXECZHZILAGIYJ5QJGYIGZSN4XP4LAD2PFY5CQHVARTAGTU5UA, seg_pos={0 0}, upload=113588]: Done uploading piece. err: <nil> Aug 4 16:10:26.112849236 (118h13m16.050507735s) [noise=true , address=104.36.82.16:4293 , piece_id=UUQ6WYMZMYI6PYRGK5IK2OE6CX2AHAW45ZRCPA7LK3X2IIIM3ZJA, seg_pos={0 0}, upload=113588]: Done uploading piece. err: <nil> Aug 4 16:10:26.171272489 (118h13m16.108930988s) [noise=true , address=77.90.31.12:4589 , piece_id=BFYZ6N2AZEFVYNS4EQYPVKNLUIANMXZ7MPDSGYA5LB2XP2REAJUQ, seg_pos={0 0}, upload=113588]: Done uploading piece. err: <nil> Aug 4 16:10:28.145791810 (118h13m18.083450309s) [noise=true , address=77.90.31.13:4627 , piece_id=E5LXNHOSI5URSZV7OBEWIQ75CEQWXDN2DLW4N2PUSY6MFHCNQ7YA, seg_pos={0 0}, upload=113588]: Done uploading piece. err: <nil> Aug 4 16:10:31.180407751 (118h13m21.11806625s) [noise=true , address=77.90.31.10:4519 , piece_id=7HTU6S7JCLTAA7MMAYQBQP6F2PWHU3X7BU2NPHUUT2Q2MPFTLRWA, seg_pos={0 0}, upload=113588]: Done uploading piece. err: <nil> Aug 4 16:10:53.161543551 (118h13m43.099202051s) [noise=true , address=104.36.82.7:4910 , piece_id=ZBYQ4SELARIGNS5Q3FBOOS3QIRVQ23VDCHMDK3TBLU5E7VREUPBA, seg_pos={0 0}, upload=113588]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: write tcp 188.166.225.170:52204->104.36.82.7:4910: use of closed network connection; write tcp 188.166.225.170:52204->104.36.82.7:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52204->104.36.82.7:4910: use of closed network connection Aug 4 16:10:53.161605039 (118h13m43.099263528s) [noise=true , address=104.36.82.10:4077 , piece_id=UNAQGJCBMXKJIOAHLHZSG4RYC2UDR3B5OXAOUTI7FD5CXLFXR2UQ, seg_pos={0 0}, upload=113588]: Done uploading piece. err: ecclient: upload failed (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk, address:104.36.82.10:4077): protocol: write tcp 188.166.225.170:41132->104.36.82.10:4077: use of closed network connection; write tcp 188.166.225.170:41132->104.36.82.10:4077: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41132->104.36.82.10:4077: use of closed network connection Aug 4 16:10:53.161771859 (118h13m43.099430348s) [noise=true , address=45.91.116.8:4048 , piece_id=JGBCOCNYT4IZUJ6BCI3TEX7E46PYWQSS6HHM67YCOXQ3W5L4H5OA, seg_pos={0 0}, upload=113588]: Done uploading piece. err: ecclient: upload failed (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, address:45.91.116.8:4048): protocol: write tcp 188.166.225.170:35774->45.91.116.8:4048: use of closed network connection; write tcp 188.166.225.170:35774->45.91.116.8:4048: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35774->45.91.116.8:4048: use of closed network connection Aug 4 16:10:53.161844948 (118h13m43.099503447s) [noise=true , address=45.91.116.7:4011 , piece_id=57KLGTVQH77IOCAQPMQA66CPZPLEDKUIO4JKMUP6BOD4XSMPB5BQ, seg_pos={0 0}, upload=113588]: Done uploading piece. err: ecclient: upload failed (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, address:45.91.116.7:4011): protocol: write tcp 188.166.225.170:50254->45.91.116.7:4011: use of closed network connection; write tcp 188.166.225.170:50254->45.91.116.7:4011: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50254->45.91.116.7:4011: use of closed network connection Aug 4 16:10:53.161881499 (118h13m43.099539998s) [noise=true , address=45.91.116.10:4082 , piece_id=GKHLII3XXM72YPTIZF5ZFZESHXZQNB56E3CVZFAEXJC2WX6URD5A, seg_pos={0 0}, upload=113588]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: read tcp 188.166.225.170:44088->45.91.116.10:4082: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 4 16:10:53.161905614 (118h13m43.099564103s) [seg_pos={0 0}, upload=113588]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Aug 4 16:10:53.443651470 (118h13m43.381309969s) [upload=113588]: Done uploading segments... Aug 4 16:10:53.443675736 (118h13m43.381334235s) [upload=113588]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 8.921625886s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........................................................XX] duration: 30.284110618s FAIL (2025-08-04 16:10:53 UTC) orbiter warning: Operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== 16569229 (118h13m13.754227758s) [upload=113588]: Done starting upload Aug 4 16:10:23.816697144 (118h13m13.754355643s) [upload=113588]: Uploading segments... Aug 4 16:10:23.816823747 (118h13m13.754482246s) [upload=113588]: Got next segment. Inline: false Aug 4 16:10:23.816875515 (118h13m13.754534014s) [upload=113588]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 4 16:10:23.816882549 (118h13m13.754541068s) [upload=113588]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 4 16:10:24.051702429 (118h13m13.989360928s) [seg_pos={0 0}, upload=113588]: Begin upload segment... Aug 4 16:10:24.051745952 (118h13m13.989404451s) [seg_pos={0 0}, upload=113588]: Done begin upload segment. Aug 4 16:10:24.051767884 (118h13m13.989426383s) [noise=true , address=104.36.82.16:4293 , piece_id=UUQ6WYMZMYI6PYRGK5IK2OE6CX2AHAW45ZRCPA7LK3X2IIIM3ZJA, seg_pos={0 0}, upload=113588]: Uploading piece... Aug 4 16:10:24.051841024 (118h13m13.989499533s) [noise=true , address=104.36.82.7:4910 , piece_id=ZBYQ4SELARIGNS5Q3FBOOS3QIRVQ23VDCHMDK3TBLU5E7VREUPBA, seg_pos={0 0}, upload=113588]: Uploading piece... Aug 4 16:10:24.052106963 (118h13m13.989765462s) [noise=true , address=77.90.31.18:4805 , piece_id=OW2WLOK6KW2WUGVFEDEHMTUBSSUGGH4S5NQTYXKNOVMDXRYWHSEQ, seg_pos={0 0}, upload=113588]: Uploading piece... Aug 4 16:10:24.052435702 (118h13m13.990094201s) [noise=true , address=77.90.31.21:4877 , piece_id=GNPOUGIJ42Y47VOZ5EMFVIDEFAY5B3KQQPEZ3LS7XYBDTXBZCLKQ, seg_pos={0 0}, upload=113588]: Uploading piece... Aug 4 16:10:24.052526446 (118h13m13.990184945s) [noise=true , address=45.91.116.12:4225 , piece_id=3M2NI24IGXUHSDASHFYXDR2QPTNGEAQFT22AHIUAHVK2JVJNDQXQ, seg_pos={0 0}, upload=113588]: Uploading piece... Aug 4 16:10:24.052684418 (118h13m13.990342907s) [noise=true , address=45.91.116.10:4082 , piece_id=GKHLII3XXM72YPTIZF5ZFZESHXZQNB56E3CVZFAEXJC2WX6URD5A, seg_pos={0 0}, upload=113588]: Uploading piece... Aug 4 16:10:24.052926030 (118h13m13.990584529s) [noise=true , address=104.36.82.10:4077 , piece_id=UNAQGJCBMXKJIOAHLHZSG4RYC2UDR3B5OXAOUTI7FD5CXLFXR2UQ, seg_pos={0 0}, upload=113588]: Uploading piece... Aug 4 16:10:24.052950747 (118h13m13.990609246s) [noise=true , address=45.91.116.4:4938 , piece_id=SHP5IWA4HZTAAECBH2VRW525XCY6BVWIJOHE3U7WAD4ISJ63WUVA, seg_pos={0 0}, upload=113588]: Uploading piece... Aug 4 16:10:24.053160659 (118h13m13.990819158s) [noise=true , address=45.91.116.11:4119 , piece_id=WBFTLEUFEXKAR2U5SZLSOHPZEGV3QNWDLVJBL36XP2GEAK5L7RUA, seg_pos={0 0}, upload=113588]: Uploading piece... Aug 4 16:10:24.053383786 (118h13m13.991042285s) [noise=true , address=77.90.31.12:4589 , piece_id=BFYZ6N2AZEFVYNS4EQYPVKNLUIANMXZ7MPDSGYA5LB2XP2REAJUQ, seg_pos={0 0}, upload=113588]: Uploading piece... Aug 4 16:10:24.053410627 (118h13m13.991069126s) [noise=true , address=77.90.31.16:4735 , piece_id=XET5ROIUKXDV7MNQBYU2JPXLWHH74RD525PMYWC7VS7GZABL5P7A, seg_pos={0 0}, upload=113588]: Uploading piece... Aug 4 16:10:24.053613205 (118h13m13.991271704s) [noise=true , address=77.90.31.13:4627 , piece_id=E5LXNHOSI5URSZV7OBEWIQ75CEQWXDN2DLW4N2PUSY6MFHCNQ7YA, seg_pos={0 0}, upload=113588]: Uploading piece... Aug 4 16:10:24.053834499 (118h13m13.991492998s) [noise=true , address=104.36.82.6:4876 , piece_id=EWMXECZHZILAGIYJ5QJGYIGZSN4XP4LAD2PFY5CQHVARTAGTU5UA, seg_pos={0 0}, upload=113588]: Uploading piece... Aug 4 16:10:24.053873513 (118h13m13.991532012s) [noise=true , address=45.91.116.7:4011 , piece_id=57KLGTVQH77IOCAQPMQA66CPZPLEDKUIO4JKMUP6BOD4XSMPB5BQ, seg_pos={0 0}, upload=113588]: Uploading piece... Aug 4 16:10:24.054251246 (118h13m13.991909755s) [noise=true , address=77.90.31.10:4519 , piece_id=7HTU6S7JCLTAA7MMAYQBQP6F2PWHU3X7BU2NPHUUT2Q2MPFTLRWA, seg_pos={0 0}, upload=113588]: Uploading piece... Aug 4 16:10:24.054636593 (118h13m13.992295092s) [noise=true , address=45.91.116.8:4048 , piece_id=JGBCOCNYT4IZUJ6BCI3TEX7E46PYWQSS6HHM67YCOXQ3W5L4H5OA, seg_pos={0 0}, upload=113588]: Uploading piece... Aug 4 16:10:25.788117314 (118h13m15.725775813s) [upload=113588]: Next returned nil segment Aug 4 16:10:25.788134928 (118h13m15.725793427s) [upload=113588]: Waiting for error group managing segments... Aug 4 16:10:26.080627835 (118h13m16.018286335s) [noise=true , address=77.90.31.16:4735 , piece_id=XET5ROIUKXDV7MNQBYU2JPXLWHH74RD525PMYWC7VS7GZABL5P7A, seg_pos={0 0}, upload=113588]: Done uploading piece. err: <nil> Aug 4 16:10:26.081237151 (118h13m16.01889565s) [noise=true , address=77.90.31.21:4877 , piece_id=GNPOUGIJ42Y47VOZ5EMFVIDEFAY5B3KQQPEZ3LS7XYBDTXBZCLKQ, seg_pos={0 0}, upload=113588]: Done uploading piece. err: <nil> Aug 4 16:10:26.082642120 (118h13m16.020300619s) [noise=true , address=77.90.31.18:4805 , piece_id=OW2WLOK6KW2WUGVFEDEHMTUBSSUGGH4S5NQTYXKNOVMDXRYWHSEQ, seg_pos={0 0}, upload=113588]: Done uploading piece. err: <nil> Aug 4 16:10:26.090023893 (118h13m16.027682392s) [noise=true , address=45.91.116.11:4119 , piece_id=WBFTLEUFEXKAR2U5SZLSOHPZEGV3QNWDLVJBL36XP2GEAK5L7RUA, seg_pos={0 0}, upload=113588]: Done uploading piece. err: <nil> Aug 4 16:10:26.092155202 (118h13m16.029813701s) [noise=true , address=45.91.116.12:4225 , piece_id=3M2NI24IGXUHSDASHFYXDR2QPTNGEAQFT22AHIUAHVK2JVJNDQXQ, seg_pos={0 0}, upload=113588]: Done uploading piece. err: <nil> Aug 4 16:10:26.093301104 (118h13m16.030959593s) [noise=true , address=45.91.116.4:4938 , piece_id=SHP5IWA4HZTAAECBH2VRW525XCY6BVWIJOHE3U7WAD4ISJ63WUVA, seg_pos={0 0}, upload=113588]: Done uploading piece. err: <nil> Aug 4 16:10:26.096255166 (118h13m16.033913675s) [noise=true , address=104.36.82.6:4876 , piece_id=EWMXECZHZILAGIYJ5QJGYIGZSN4XP4LAD2PFY5CQHVARTAGTU5UA, seg_pos={0 0}, upload=113588]: Done uploading piece. err: <nil> Aug 4 16:10:26.112849236 (118h13m16.050507735s) [noise=true , address=104.36.82.16:4293 , piece_id=UUQ6WYMZMYI6PYRGK5IK2OE6CX2AHAW45ZRCPA7LK3X2IIIM3ZJA, seg_pos={0 0}, upload=113588]: Done uploading piece. err: <nil> Aug 4 16:10:26.171272489 (118h13m16.108930988s) [noise=true , address=77.90.31.12:4589 , piece_id=BFYZ6N2AZEFVYNS4EQYPVKNLUIANMXZ7MPDSGYA5LB2XP2REAJUQ, seg_pos={0 0}, upload=113588]: Done uploading piece. err: <nil> Aug 4 16:10:28.145791810 (118h13m18.083450309s) [noise=true , address=77.90.31.13:4627 , piece_id=E5LXNHOSI5URSZV7OBEWIQ75CEQWXDN2DLW4N2PUSY6MFHCNQ7YA, seg_pos={0 0}, upload=113588]: Done uploading piece. err: <nil> Aug 4 16:10:31.180407751 (118h13m21.11806625s) [noise=true , address=77.90.31.10:4519 , piece_id=7HTU6S7JCLTAA7MMAYQBQP6F2PWHU3X7BU2NPHUUT2Q2MPFTLRWA, seg_pos={0 0}, upload=113588]: Done uploading piece. err: <nil> Aug 4 16:10:53.161543551 (118h13m43.099202051s) [noise=true , address=104.36.82.7:4910 , piece_id=ZBYQ4SELARIGNS5Q3FBOOS3QIRVQ23VDCHMDK3TBLU5E7VREUPBA, seg_pos={0 0}, upload=113588]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: write tcp 188.166.225.170:52204->104.36.82.7:4910: use of closed network connection; write tcp 188.166.225.170:52204->104.36.82.7:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52204->104.36.82.7:4910: use of closed network connection Aug 4 16:10:53.161605039 (118h13m43.099263528s) [noise=true , address=104.36.82.10:4077 , piece_id=UNAQGJCBMXKJIOAHLHZSG4RYC2UDR3B5OXAOUTI7FD5CXLFXR2UQ, seg_pos={0 0}, upload=113588]: Done uploading piece. err: ecclient: upload failed (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk, address:104.36.82.10:4077): protocol: write tcp 188.166.225.170:41132->104.36.82.10:4077: use of closed network connection; write tcp 188.166.225.170:41132->104.36.82.10:4077: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41132->104.36.82.10:4077: use of closed network connection Aug 4 16:10:53.161771859 (118h13m43.099430348s) [noise=true , address=45.91.116.8:4048 , piece_id=JGBCOCNYT4IZUJ6BCI3TEX7E46PYWQSS6HHM67YCOXQ3W5L4H5OA, seg_pos={0 0}, upload=113588]: Done uploading piece. err: ecclient: upload failed (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, address:45.91.116.8:4048): protocol: write tcp 188.166.225.170:35774->45.91.116.8:4048: use of closed network connection; write tcp 188.166.225.170:35774->45.91.116.8:4048: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35774->45.91.116.8:4048: use of closed network connection Aug 4 16:10:53.161844948 (118h13m43.099503447s) [noise=true , address=45.91.116.7:4011 , piece_id=57KLGTVQH77IOCAQPMQA66CPZPLEDKUIO4JKMUP6BOD4XSMPB5BQ, seg_pos={0 0}, upload=113588]: Done uploading piece. err: ecclient: upload failed (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, address:45.91.116.7:4011): protocol: write tcp 188.166.225.170:50254->45.91.116.7:4011: use of closed network connection; write tcp 188.166.225.170:50254->45.91.116.7:4011: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50254->45.91.116.7:4011: use of closed network connection Aug 4 16:10:53.161881499 (118h13m43.099539998s) [noise=true , address=45.91.116.10:4082 , piece_id=GKHLII3XXM72YPTIZF5ZFZESHXZQNB56E3CVZFAEXJC2WX6URD5A, seg_pos={0 0}, upload=113588]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: read tcp 188.166.225.170:44088->45.91.116.10:4082: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 4 16:10:53.161905614 (118h13m43.099564103s) [seg_pos={0 0}, upload=113588]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Aug 4 16:10:53.443651470 (118h13m43.381309969s) [upload=113588]: Done uploading segments... Aug 4 16:10:53.443675736 (118h13m43.381334235s) [upload=113588]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 30.284130655s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 6.695262985s FAIL (2025-08-04 16:10:53 UTC) orbiter warning: Operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 4.36.82.36:4625 , piece_id=FCK7QGPSKJ7VDK4D7BVNF44BRPQCQS5V5ICAJHELAQHDBSQV5R3Q, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.146274823 (118h10m17.083933322s) [noise=true , address=45.91.116.7:4026 , piece_id=CB6TYC4A6DQOFFKVFPNMF5MH5MGMJHU5Z25B4QXQJI5ETDWLQJJQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.146672965 (118h10m17.084331464s) [noise=true , address=104.36.82.16:4321 , piece_id=42MZPFPEBQNQRJUHQJLZQP2UWGDV2OAXT6Z4KTWPPWLALISASN5Q, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.146726166 (118h10m17.084384665s) [noise=true , address=104.36.82.55:4206 , piece_id=KSAKYO6MALVA5REEKKJZ7MGX7YIRC2PDI5PNHFHRASPBD47V5SOQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.147136371 (118h10m17.08479488s) [noise=true , address=104.36.82.42:4823 , piece_id=66Q6G7YBCVUEAHU3Y5LHPR5MIT7H7NQWI2TWWNHKZT5FYS4DNQBA, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.147479628 (118h10m17.085138127s) [noise=true , address=193.149.164.143:4695 , piece_id=3QTYKOTBCOJ4CRM76FR2XKONKQVYTQQMPTRH3E6MYJR4ODBPY3AQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.147821592 (118h10m17.085480091s) [noise=true , address=104.36.82.62:4411 , piece_id=CIJJ4SMWKFDOIZZ5UEYIXFZHUNAZKRMM2P36IJIT5BHUX5STDZ4A, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.148134220 (118h10m17.085792719s) [noise=true , address=104.36.82.55:4224 , piece_id=NJYMQ2467DXHUEC6ELXITFJF6WWKM4XFEONRJVRF6QSJBDEMC7JA, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.148548573 (118h10m17.086207072s) [noise=true , address=104.36.82.13:4193 , piece_id=WJA72IWFD3TUA2WPGJGB6A7AHJXI4PGOK62YDMKSMDPUAVHUHW4Q, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.148889576 (118h10m17.086548085s) [noise=true , address=104.36.82.39:4728 , piece_id=ZCL6YOUXY4P6HTYUTTILZR4TLG44CDTXWLOJJWOHPFSORA2IBCBA, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.149205931 (118h10m17.08686443s) [noise=true , address=104.36.82.59:4326 , piece_id=CS7FVO3IRXKZFB2EZM2HQBVEVM4XTTFI5Y7TH3NJZGIYMREVY7UQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.149509752 (118h10m17.087168261s) [noise=true , address=104.36.82.45:4930 , piece_id=LBMDUGYMQI7PBBGEOT22S7OGNVRIYS66USSYFAZOSGI445GWKCYA, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.149866545 (118h10m17.087525044s) [noise=true , address=104.36.82.23:4474 , piece_id=PLEKQKMEQKQDW72BNLLY4JOOPFI6CPNNGT74M5EFFNDQQFWELXUQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.150193601 (118h10m17.08785211s) [noise=true , address=104.36.82.13:4212 , piece_id=ER6VIEGFSG2F7542M5P66RS3QAUQYV7BWPQ4YOX55Q3XX6DDCTGA, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.150515557 (118h10m17.088174066s) [noise=true , address=104.36.82.46:4485 , piece_id=P4FMDR5NCQOLKSXQBLE7I7BPEZJ2URAPPVNKOR53ZPROBCE4YAJQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.150935149 (118h10m17.088593648s) [noise=true , address=104.36.82.54:4181 , piece_id=AHW774W42KLBATPDVRQRKH4MQWV5YP2XEEUY6VASDMYJ2BKXCS2Q, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.151249151 (118h10m17.08890765s) [noise=true , address=104.36.82.44:4886 , piece_id=Q3BIO4GNMY64IWXC7VZ5BAYKYRJJ3GKFV7Z4IOG7SP62IUJHXN2A, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.151276363 (118h10m17.088934862s) [noise=true , address=104.36.82.9:4019 , piece_id=SBCPU6XTMB4CRRTFKDBYFQQ2XZZNE3I6AF7VVSGYC356SPQNVHUQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:28.640156348 (118h10m18.577814857s) [upload=113540]: Next returned nil segment Aug 4 16:07:28.640223306 (118h10m18.577881825s) [upload=113540]: Waiting for error group managing segments... Aug 4 16:07:29.062127352 (118h10m18.999785841s) [noise=true , address=104.36.82.16:4321 , piece_id=42MZPFPEBQNQRJUHQJLZQP2UWGDV2OAXT6Z4KTWPPWLALISASN5Q, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.080565981 (118h10m19.01822448s) [noise=true , address=104.36.82.54:4181 , piece_id=AHW774W42KLBATPDVRQRKH4MQWV5YP2XEEUY6VASDMYJ2BKXCS2Q, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.081329852 (118h10m19.018988352s) [noise=true , address=104.36.82.62:4411 , piece_id=CIJJ4SMWKFDOIZZ5UEYIXFZHUNAZKRMM2P36IJIT5BHUX5STDZ4A, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.081846201 (118h10m19.0195047s) [noise=true , address=104.36.82.55:4224 , piece_id=NJYMQ2467DXHUEC6ELXITFJF6WWKM4XFEONRJVRF6QSJBDEMC7JA, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.091017397 (118h10m19.028675896s) [noise=true , address=104.36.82.46:4485 , piece_id=P4FMDR5NCQOLKSXQBLE7I7BPEZJ2URAPPVNKOR53ZPROBCE4YAJQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.105092769 (118h10m19.042751268s) [noise=true , address=104.36.82.13:4212 , piece_id=ER6VIEGFSG2F7542M5P66RS3QAUQYV7BWPQ4YOX55Q3XX6DDCTGA, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.107141759 (118h10m19.044800248s) [noise=true , address=104.36.82.23:4474 , piece_id=PLEKQKMEQKQDW72BNLLY4JOOPFI6CPNNGT74M5EFFNDQQFWELXUQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.116502399 (118h10m19.054160898s) [noise=true , address=45.91.116.7:4026 , piece_id=CB6TYC4A6DQOFFKVFPNMF5MH5MGMJHU5Z25B4QXQJI5ETDWLQJJQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.153680253 (118h10m19.091338752s) [noise=true , address=104.36.82.9:4019 , piece_id=SBCPU6XTMB4CRRTFKDBYFQQ2XZZNE3I6AF7VVSGYC356SPQNVHUQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.210191177 (118h10m19.147849686s) [noise=true , address=104.36.82.36:4625 , piece_id=FCK7QGPSKJ7VDK4D7BVNF44BRPQCQS5V5ICAJHELAQHDBSQV5R3Q, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.219317350 (118h10m19.156975849s) [noise=true , address=104.36.82.55:4206 , piece_id=KSAKYO6MALVA5REEKKJZ7MGX7YIRC2PDI5PNHFHRASPBD47V5SOQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:31.340384557 (118h10m21.278043046s) [noise=true , address=104.36.82.59:4326 , piece_id=CS7FVO3IRXKZFB2EZM2HQBVEVM4XTTFI5Y7TH3NJZGIYMREVY7UQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:32.464238230 (118h10m22.401896729s) [noise=true , address=104.36.82.13:4193 , piece_id=WJA72IWFD3TUA2WPGJGB6A7AHJXI4PGOK62YDMKSMDPUAVHUHW4Q, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:08:26.041033880 (118h11m15.978692379s) [noise=true , address=193.149.164.143:4695 , piece_id=3QTYKOTBCOJ4CRM76FR2XKONKQVYTQQMPTRH3E6MYJR4ODBPY3AQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: ecclient: upload failed (node:12TZfgz24gnZr4UryzEKw6gu94njnmCvafX5X64MKFSJqaUWSVh, address:193.149.164.143:4695): protocol: write tcp 188.166.225.170:43298->193.149.164.143:4695: use of closed network connection; write tcp 188.166.225.170:43298->193.149.164.143:4695: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43298->193.149.164.143:4695: use of closed network connection Aug 4 16:08:26.041079005 (118h11m15.978737504s) [noise=true , address=104.36.82.44:4886 , piece_id=Q3BIO4GNMY64IWXC7VZ5BAYKYRJJ3GKFV7Z4IOG7SP62IUJHXN2A, seg_pos={0 0}, upload=113540]: Done uploading piece. err: ecclient: upload failed (node:1n2cXhqd5EWbNPTXem83jQUjYq5ZaGKBidbVCQzxYfR3rWddGR, address:104.36.82.44:4886): protocol: write tcp 188.166.225.170:38030->104.36.82.44:4886: use of closed network connection; write tcp 188.166.225.170:38030->104.36.82.44:4886: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38030->104.36.82.44:4886: use of closed network connection Aug 4 16:08:26.041128580 (118h11m15.97878711s) [noise=true , address=104.36.82.39:4728 , piece_id=ZCL6YOUXY4P6HTYUTTILZR4TLG44CDTXWLOJJWOHPFSORA2IBCBA, seg_pos={0 0}, upload=113540]: Done uploading piece. err: ecclient: upload failed (node:1YoLzpb4mMYZ8TsdxnMocekZZ1gLfTrdCM8mxTDj4tY6khZGvi, address:104.36.82.39:4728): protocol: write tcp 188.166.225.170:34326->104.36.82.39:4728: use of closed network connection; write tcp 188.166.225.170:34326->104.36.82.39:4728: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34326->104.36.82.39:4728: use of closed network connection Aug 4 16:08:26.041195678 (118h11m15.978854187s) [noise=true , address=104.36.82.45:4930 , piece_id=LBMDUGYMQI7PBBGEOT22S7OGNVRIYS66USSYFAZOSGI445GWKCYA, seg_pos={0 0}, upload=113540]: Done uploading piece. err: ecclient: upload failed (node:12qaYFSpezuLAc5zJfAYruzad6nTpYFDpog7bmKMVTA8MVm76NB, address:104.36.82.45:4930): protocol: expected piece hash; context deadline exceeded; EOF Aug 4 16:08:26.041203374 (118h11m15.978861913s) [noise=true , address=104.36.82.42:4823 , piece_id=66Q6G7YBCVUEAHU3Y5LHPR5MIT7H7NQWI2TWWNHKZT5FYS4DNQBA, seg_pos={0 0}, upload=113540]: Done uploading piece. err: ecclient: upload failed (node:1fkA5DEj9RqsR6N85z7azvujy8tPkUhB5GCqBpUmFb4vvwbd8y, address:104.36.82.42:4823): protocol: write tcp 188.166.225.170:37230->104.36.82.42:4823: use of closed network connection; write tcp 188.166.225.170:37230->104.36.82.42:4823: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37230->104.36.82.42:4823: use of closed network connection Aug 4 16:08:26.041258960 (118h11m15.978917489s) [seg_pos={0 0}, upload=113540]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Aug 4 16:08:26.274088873 (118h11m16.211747372s) [upload=113540]: Done uploading segments... Aug 4 16:08:26.274113650 (118h11m16.211772139s) [upload=113540]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 6.695285387s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........................................................XX.] duration: 8.720162348s FAIL (2025-08-04 16:10:51 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 8.72018405s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 3 out of 60 is failed (threshold=3) [X.........X.............................X...................] duration: 1.953961063s FAIL (2025-08-04 16:10:40 UTC) orbiter warning: Operation: DownloadObject uplink: object not found ("orbiter_us_0/1K") Duration: 1.95398611s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 7.774145196s FAIL (2025-08-04 16:10:36 UTC) orbiter warning: Operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 4.36.82.36:4625 , piece_id=FCK7QGPSKJ7VDK4D7BVNF44BRPQCQS5V5ICAJHELAQHDBSQV5R3Q, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.146274823 (118h10m17.083933322s) [noise=true , address=45.91.116.7:4026 , piece_id=CB6TYC4A6DQOFFKVFPNMF5MH5MGMJHU5Z25B4QXQJI5ETDWLQJJQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.146672965 (118h10m17.084331464s) [noise=true , address=104.36.82.16:4321 , piece_id=42MZPFPEBQNQRJUHQJLZQP2UWGDV2OAXT6Z4KTWPPWLALISASN5Q, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.146726166 (118h10m17.084384665s) [noise=true , address=104.36.82.55:4206 , piece_id=KSAKYO6MALVA5REEKKJZ7MGX7YIRC2PDI5PNHFHRASPBD47V5SOQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.147136371 (118h10m17.08479488s) [noise=true , address=104.36.82.42:4823 , piece_id=66Q6G7YBCVUEAHU3Y5LHPR5MIT7H7NQWI2TWWNHKZT5FYS4DNQBA, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.147479628 (118h10m17.085138127s) [noise=true , address=193.149.164.143:4695 , piece_id=3QTYKOTBCOJ4CRM76FR2XKONKQVYTQQMPTRH3E6MYJR4ODBPY3AQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.147821592 (118h10m17.085480091s) [noise=true , address=104.36.82.62:4411 , piece_id=CIJJ4SMWKFDOIZZ5UEYIXFZHUNAZKRMM2P36IJIT5BHUX5STDZ4A, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.148134220 (118h10m17.085792719s) [noise=true , address=104.36.82.55:4224 , piece_id=NJYMQ2467DXHUEC6ELXITFJF6WWKM4XFEONRJVRF6QSJBDEMC7JA, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.148548573 (118h10m17.086207072s) [noise=true , address=104.36.82.13:4193 , piece_id=WJA72IWFD3TUA2WPGJGB6A7AHJXI4PGOK62YDMKSMDPUAVHUHW4Q, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.148889576 (118h10m17.086548085s) [noise=true , address=104.36.82.39:4728 , piece_id=ZCL6YOUXY4P6HTYUTTILZR4TLG44CDTXWLOJJWOHPFSORA2IBCBA, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.149205931 (118h10m17.08686443s) [noise=true , address=104.36.82.59:4326 , piece_id=CS7FVO3IRXKZFB2EZM2HQBVEVM4XTTFI5Y7TH3NJZGIYMREVY7UQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.149509752 (118h10m17.087168261s) [noise=true , address=104.36.82.45:4930 , piece_id=LBMDUGYMQI7PBBGEOT22S7OGNVRIYS66USSYFAZOSGI445GWKCYA, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.149866545 (118h10m17.087525044s) [noise=true , address=104.36.82.23:4474 , piece_id=PLEKQKMEQKQDW72BNLLY4JOOPFI6CPNNGT74M5EFFNDQQFWELXUQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.150193601 (118h10m17.08785211s) [noise=true , address=104.36.82.13:4212 , piece_id=ER6VIEGFSG2F7542M5P66RS3QAUQYV7BWPQ4YOX55Q3XX6DDCTGA, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.150515557 (118h10m17.088174066s) [noise=true , address=104.36.82.46:4485 , piece_id=P4FMDR5NCQOLKSXQBLE7I7BPEZJ2URAPPVNKOR53ZPROBCE4YAJQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.150935149 (118h10m17.088593648s) [noise=true , address=104.36.82.54:4181 , piece_id=AHW774W42KLBATPDVRQRKH4MQWV5YP2XEEUY6VASDMYJ2BKXCS2Q, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.151249151 (118h10m17.08890765s) [noise=true , address=104.36.82.44:4886 , piece_id=Q3BIO4GNMY64IWXC7VZ5BAYKYRJJ3GKFV7Z4IOG7SP62IUJHXN2A, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.151276363 (118h10m17.088934862s) [noise=true , address=104.36.82.9:4019 , piece_id=SBCPU6XTMB4CRRTFKDBYFQQ2XZZNE3I6AF7VVSGYC356SPQNVHUQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:28.640156348 (118h10m18.577814857s) [upload=113540]: Next returned nil segment Aug 4 16:07:28.640223306 (118h10m18.577881825s) [upload=113540]: Waiting for error group managing segments... Aug 4 16:07:29.062127352 (118h10m18.999785841s) [noise=true , address=104.36.82.16:4321 , piece_id=42MZPFPEBQNQRJUHQJLZQP2UWGDV2OAXT6Z4KTWPPWLALISASN5Q, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.080565981 (118h10m19.01822448s) [noise=true , address=104.36.82.54:4181 , piece_id=AHW774W42KLBATPDVRQRKH4MQWV5YP2XEEUY6VASDMYJ2BKXCS2Q, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.081329852 (118h10m19.018988352s) [noise=true , address=104.36.82.62:4411 , piece_id=CIJJ4SMWKFDOIZZ5UEYIXFZHUNAZKRMM2P36IJIT5BHUX5STDZ4A, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.081846201 (118h10m19.0195047s) [noise=true , address=104.36.82.55:4224 , piece_id=NJYMQ2467DXHUEC6ELXITFJF6WWKM4XFEONRJVRF6QSJBDEMC7JA, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.091017397 (118h10m19.028675896s) [noise=true , address=104.36.82.46:4485 , piece_id=P4FMDR5NCQOLKSXQBLE7I7BPEZJ2URAPPVNKOR53ZPROBCE4YAJQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.105092769 (118h10m19.042751268s) [noise=true , address=104.36.82.13:4212 , piece_id=ER6VIEGFSG2F7542M5P66RS3QAUQYV7BWPQ4YOX55Q3XX6DDCTGA, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.107141759 (118h10m19.044800248s) [noise=true , address=104.36.82.23:4474 , piece_id=PLEKQKMEQKQDW72BNLLY4JOOPFI6CPNNGT74M5EFFNDQQFWELXUQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.116502399 (118h10m19.054160898s) [noise=true , address=45.91.116.7:4026 , piece_id=CB6TYC4A6DQOFFKVFPNMF5MH5MGMJHU5Z25B4QXQJI5ETDWLQJJQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.153680253 (118h10m19.091338752s) [noise=true , address=104.36.82.9:4019 , piece_id=SBCPU6XTMB4CRRTFKDBYFQQ2XZZNE3I6AF7VVSGYC356SPQNVHUQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.210191177 (118h10m19.147849686s) [noise=true , address=104.36.82.36:4625 , piece_id=FCK7QGPSKJ7VDK4D7BVNF44BRPQCQS5V5ICAJHELAQHDBSQV5R3Q, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.219317350 (118h10m19.156975849s) [noise=true , address=104.36.82.55:4206 , piece_id=KSAKYO6MALVA5REEKKJZ7MGX7YIRC2PDI5PNHFHRASPBD47V5SOQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:31.340384557 (118h10m21.278043046s) [noise=true , address=104.36.82.59:4326 , piece_id=CS7FVO3IRXKZFB2EZM2HQBVEVM4XTTFI5Y7TH3NJZGIYMREVY7UQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:32.464238230 (118h10m22.401896729s) [noise=true , address=104.36.82.13:4193 , piece_id=WJA72IWFD3TUA2WPGJGB6A7AHJXI4PGOK62YDMKSMDPUAVHUHW4Q, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:08:26.041033880 (118h11m15.978692379s) [noise=true , address=193.149.164.143:4695 , piece_id=3QTYKOTBCOJ4CRM76FR2XKONKQVYTQQMPTRH3E6MYJR4ODBPY3AQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: ecclient: upload failed (node:12TZfgz24gnZr4UryzEKw6gu94njnmCvafX5X64MKFSJqaUWSVh, address:193.149.164.143:4695): protocol: write tcp 188.166.225.170:43298->193.149.164.143:4695: use of closed network connection; write tcp 188.166.225.170:43298->193.149.164.143:4695: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43298->193.149.164.143:4695: use of closed network connection Aug 4 16:08:26.041079005 (118h11m15.978737504s) [noise=true , address=104.36.82.44:4886 , piece_id=Q3BIO4GNMY64IWXC7VZ5BAYKYRJJ3GKFV7Z4IOG7SP62IUJHXN2A, seg_pos={0 0}, upload=113540]: Done uploading piece. err: ecclient: upload failed (node:1n2cXhqd5EWbNPTXem83jQUjYq5ZaGKBidbVCQzxYfR3rWddGR, address:104.36.82.44:4886): protocol: write tcp 188.166.225.170:38030->104.36.82.44:4886: use of closed network connection; write tcp 188.166.225.170:38030->104.36.82.44:4886: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38030->104.36.82.44:4886: use of closed network connection Aug 4 16:08:26.041128580 (118h11m15.97878711s) [noise=true , address=104.36.82.39:4728 , piece_id=ZCL6YOUXY4P6HTYUTTILZR4TLG44CDTXWLOJJWOHPFSORA2IBCBA, seg_pos={0 0}, upload=113540]: Done uploading piece. err: ecclient: upload failed (node:1YoLzpb4mMYZ8TsdxnMocekZZ1gLfTrdCM8mxTDj4tY6khZGvi, address:104.36.82.39:4728): protocol: write tcp 188.166.225.170:34326->104.36.82.39:4728: use of closed network connection; write tcp 188.166.225.170:34326->104.36.82.39:4728: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34326->104.36.82.39:4728: use of closed network connection Aug 4 16:08:26.041195678 (118h11m15.978854187s) [noise=true , address=104.36.82.45:4930 , piece_id=LBMDUGYMQI7PBBGEOT22S7OGNVRIYS66USSYFAZOSGI445GWKCYA, seg_pos={0 0}, upload=113540]: Done uploading piece. err: ecclient: upload failed (node:12qaYFSpezuLAc5zJfAYruzad6nTpYFDpog7bmKMVTA8MVm76NB, address:104.36.82.45:4930): protocol: expected piece hash; context deadline exceeded; EOF Aug 4 16:08:26.041203374 (118h11m15.978861913s) [noise=true , address=104.36.82.42:4823 , piece_id=66Q6G7YBCVUEAHU3Y5LHPR5MIT7H7NQWI2TWWNHKZT5FYS4DNQBA, seg_pos={0 0}, upload=113540]: Done uploading piece. err: ecclient: upload failed (node:1fkA5DEj9RqsR6N85z7azvujy8tPkUhB5GCqBpUmFb4vvwbd8y, address:104.36.82.42:4823): protocol: write tcp 188.166.225.170:37230->104.36.82.42:4823: use of closed network connection; write tcp 188.166.225.170:37230->104.36.82.42:4823: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37230->104.36.82.42:4823: use of closed network connection Aug 4 16:08:26.041258960 (118h11m15.978917489s) [seg_pos={0 0}, upload=113540]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Aug 4 16:08:26.274088873 (118h11m16.211747372s) [upload=113540]: Done uploading segments... Aug 4 16:08:26.274113650 (118h11m16.211772139s) [upload=113540]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 7.774175904s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........................................................XX] duration: 30.00091157s FAIL (2025-08-04 16:10:36 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.000928894s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 30.236254868s FAIL (2025-08-04 16:10:09 UTC) orbiter warning: Operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== m57.434133488s) [upload=113570]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 4 16:09:07.799239239 (118h11m57.736897738s) [upload=113570]: Done uploading segments... Aug 4 16:09:07.799260039 (118h11m57.736918538s) [upload=113570]: Upload finished. err: <nil> Aug 4 16:09:40.227861559 (118h12m30.165520058s) [upload=113578]: Starting upload Aug 4 16:09:40.227908748 (118h12m30.165567247s) [upload=113578]: Done starting upload Aug 4 16:09:40.228216257 (118h12m30.165874756s) [upload=113578]: Uploading segments... Aug 4 16:09:40.228250923 (118h12m30.165909422s) [upload=113578]: Got next segment. Inline: false Aug 4 16:09:40.228274028 (118h12m30.165932527s) [upload=113578]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 4 16:09:40.228279999 (118h12m30.165938498s) [upload=113578]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 4 16:09:40.651580646 (118h12m30.589239145s) [seg_pos={0 0}, upload=113578]: Begin upload segment... Aug 4 16:09:40.651623778 (118h12m30.589282277s) [seg_pos={0 0}, upload=113578]: Done begin upload segment. Aug 4 16:09:40.651645440 (118h12m30.589303939s) [noise=true , address=77.90.31.11:4554 , piece_id=IA5UHYQI43T4FF3OCK3J4YWVQCORTP6W5L7LX3L7B3TQDPJ2QFOA, seg_pos={0 0}, upload=113578]: Uploading piece... Aug 4 16:09:40.651672652 (118h12m30.589331151s) [noise=true , address=77.90.31.16:4735 , piece_id=FZKHTGQUHQJVDHTRLK7LF5XV6LC66QBCEU5OK6HH3CIUC5O63Y7Q, seg_pos={0 0}, upload=113578]: Uploading piece... Aug 4 16:09:40.651993566 (118h12m30.589652065s) [noise=true , address=45.91.116.10:4081 , piece_id=BQBIHEHG4XJE22JFCE3N7TXRVVNTPVZYP7QJN2B3JOVVUSNK6VDQ, seg_pos={0 0}, upload=113578]: Uploading piece... Aug 4 16:09:40.652288791 (118h12m30.58994729s) [noise=true , address=104.36.82.6:4874 , piece_id=EXRM6KRLHXSRHD2WQKHYA342FPN2JMVEAW35BY3EYFKLKK2U235A, seg_pos={0 0}, upload=113578]: Uploading piece... Aug 4 16:09:40.652531635 (118h12m30.590190134s) [noise=true , address=77.90.31.15:4698 , piece_id=OKZSZ7DIG4MUUALUOGRHFD3IG6H4SPAZXN5R4N4YMQKOMOAVFGRA, seg_pos={0 0}, upload=113578]: Uploading piece... Aug 4 16:09:40.652771985 (118h12m30.590430484s) [noise=true , address=104.36.82.7:4912 , piece_id=47B7U2ZBVBMHFLVGCVCWS3OEEVWYN4VWAC37KAWN2DN5WKBYH2BA, seg_pos={0 0}, upload=113578]: Uploading piece... Aug 4 16:09:40.653005232 (118h12m30.590663731s) [noise=true , address=77.90.31.16:4733 , piece_id=ZCEO2QVC465IKFSMJPYANREPDRXTNNFE2O5GFMDGEPO6MHH2EHSQ, seg_pos={0 0}, upload=113578]: Uploading piece... Aug 4 16:09:40.653203852 (118h12m30.590862351s) [noise=true , address=104.36.82.7:4909 , piece_id=B6JVXXE3CBHHKJJL2FOT5RC6JW3GLUD6J2GSOBPSHKTCJS64G5XQ, seg_pos={0 0}, upload=113578]: Uploading piece... Aug 4 16:09:40.653391862 (118h12m30.591050361s) [noise=true , address=45.91.116.8:4047 , piece_id=ZK3GOZHUZQQ7IHBLU3ZXG6CREPLSM7Q2HKVJK34Z3WRROKAQBYTA, seg_pos={0 0}, upload=113578]: Uploading piece... Aug 4 16:09:40.653588618 (118h12m30.591247107s) [noise=true , address=45.91.116.11:4119 , piece_id=ZGZIAT4DJARF75W46CYJWPEBKUIHJSS4EU3ELCW7AWNQ7A7FOJ2Q, seg_pos={0 0}, upload=113578]: Uploading piece... Aug 4 16:09:40.653779714 (118h12m30.591438213s) [noise=true , address=45.91.116.6:4973 , piece_id=PPGQGZLZMUZ4DKRGQ7HHKOGDEKOMX7DHPMLRIFCF2UJH6OTKJCGQ, seg_pos={0 0}, upload=113578]: Uploading piece... Aug 4 16:09:40.654039631 (118h12m30.59169813s) [noise=true , address=45.91.116.11:4117 , piece_id=TCSD7X4P2JWON6PSR72YPTD24VNKMO3X5M3VOGDTBPWH2VJQLOKQ, seg_pos={0 0}, upload=113578]: Uploading piece... Aug 4 16:09:40.654224074 (118h12m30.591882563s) [noise=true , address=104.36.82.8:4970 , piece_id=PYA5UT4JVNUFOGOSBV57WF4JNIZ5GAR77UB3OOFIFFJY74FKJDBQ, seg_pos={0 0}, upload=113578]: Uploading piece... Aug 4 16:09:40.654404740 (118h12m30.592063239s) [noise=true , address=45.91.116.8:4048 , piece_id=CRNGR22XNQYXYNPUV72WOOXBTLJACNU6SLLP3ELSCZKFLFMWBYBQ, seg_pos={0 0}, upload=113578]: Uploading piece... Aug 4 16:09:40.654591116 (118h12m30.592249615s) [noise=true , address=77.90.31.13:4627 , piece_id=2FAOQDWCPVG3I4MDO4SEBPCK3YZHBGZSKYFHELADMQDSJL3QW4RQ, seg_pos={0 0}, upload=113578]: Uploading piece... Aug 4 16:09:40.654852085 (118h12m30.592510585s) [noise=true , address=45.91.116.5:4299 , piece_id=VD6BGTBEODNV4R5WZ67QHZKU3D3WHMN45Q5OP2MRIJR5QWDGQGPA, seg_pos={0 0}, upload=113578]: Uploading piece... Aug 4 16:09:42.381778453 (118h12m32.319436952s) [upload=113578]: Next returned nil segment Aug 4 16:09:42.381813750 (118h12m32.319472249s) [upload=113578]: Waiting for error group managing segments... Aug 4 16:09:42.665288313 (118h12m32.602946812s) [noise=true , address=45.91.116.11:4117 , piece_id=TCSD7X4P2JWON6PSR72YPTD24VNKMO3X5M3VOGDTBPWH2VJQLOKQ, seg_pos={0 0}, upload=113578]: Done uploading piece. err: <nil> Aug 4 16:09:42.665964958 (118h12m32.603623487s) [noise=true , address=77.90.31.16:4735 , piece_id=FZKHTGQUHQJVDHTRLK7LF5XV6LC66QBCEU5OK6HH3CIUC5O63Y7Q, seg_pos={0 0}, upload=113578]: Done uploading piece. err: <nil> Aug 4 16:09:42.666683042 (118h12m32.604341541s) [noise=true , address=77.90.31.16:4733 , piece_id=ZCEO2QVC465IKFSMJPYANREPDRXTNNFE2O5GFMDGEPO6MHH2EHSQ, seg_pos={0 0}, upload=113578]: Done uploading piece. err: <nil> Aug 4 16:09:42.670020628 (118h12m32.607679127s) [noise=true , address=45.91.116.11:4119 , piece_id=ZGZIAT4DJARF75W46CYJWPEBKUIHJSS4EU3ELCW7AWNQ7A7FOJ2Q, seg_pos={0 0}, upload=113578]: Done uploading piece. err: <nil> Aug 4 16:09:42.672427002 (118h12m32.610085501s) [noise=true , address=45.91.116.6:4973 , piece_id=PPGQGZLZMUZ4DKRGQ7HHKOGDEKOMX7DHPMLRIFCF2UJH6OTKJCGQ, seg_pos={0 0}, upload=113578]: Done uploading piece. err: <nil> Aug 4 16:09:42.675200489 (118h12m32.612858988s) [noise=true , address=77.90.31.15:4698 , piece_id=OKZSZ7DIG4MUUALUOGRHFD3IG6H4SPAZXN5R4N4YMQKOMOAVFGRA, seg_pos={0 0}, upload=113578]: Done uploading piece. err: <nil> Aug 4 16:09:42.677419585 (118h12m32.615078084s) [noise=true , address=77.90.31.11:4554 , piece_id=IA5UHYQI43T4FF3OCK3J4YWVQCORTP6W5L7LX3L7B3TQDPJ2QFOA, seg_pos={0 0}, upload=113578]: Done uploading piece. err: <nil> Aug 4 16:09:42.700504134 (118h12m32.638162633s) [noise=true , address=104.36.82.6:4874 , piece_id=EXRM6KRLHXSRHD2WQKHYA342FPN2JMVEAW35BY3EYFKLKK2U235A, seg_pos={0 0}, upload=113578]: Done uploading piece. err: <nil> Aug 4 16:09:42.831717108 (118h12m32.769375607s) [noise=true , address=45.91.116.5:4299 , piece_id=VD6BGTBEODNV4R5WZ67QHZKU3D3WHMN45Q5OP2MRIJR5QWDGQGPA, seg_pos={0 0}, upload=113578]: Done uploading piece. err: <nil> Aug 4 16:09:42.833599399 (118h12m32.771257898s) [noise=true , address=77.90.31.13:4627 , piece_id=2FAOQDWCPVG3I4MDO4SEBPCK3YZHBGZSKYFHELADMQDSJL3QW4RQ, seg_pos={0 0}, upload=113578]: Done uploading piece. err: <nil> Aug 4 16:09:44.823422538 (118h12m34.761081037s) [noise=true , address=104.36.82.7:4912 , piece_id=47B7U2ZBVBMHFLVGCVCWS3OEEVWYN4VWAC37KAWN2DN5WKBYH2BA, seg_pos={0 0}, upload=113578]: Done uploading piece. err: <nil> Aug 4 16:10:09.571455993 (118h12m59.509114492s) [noise=true , address=104.36.82.8:4970 , piece_id=PYA5UT4JVNUFOGOSBV57WF4JNIZ5GAR77UB3OOFIFFJY74FKJDBQ, seg_pos={0 0}, upload=113578]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:104.36.82.8:4970): protocol: write tcp 188.166.225.170:45128->104.36.82.8:4970: use of closed network connection; write tcp 188.166.225.170:45128->104.36.82.8:4970: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45128->104.36.82.8:4970: use of closed network connection Aug 4 16:10:09.571559852 (118h12m59.509218351s) [noise=true , address=45.91.116.10:4081 , piece_id=BQBIHEHG4XJE22JFCE3N7TXRVVNTPVZYP7QJN2B3JOVVUSNK6VDQ, seg_pos={0 0}, upload=113578]: Done uploading piece. err: ecclient: upload failed (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG, address:45.91.116.10:4081): protocol: write tcp 188.166.225.170:40310->45.91.116.10:4081: use of closed network connection; write tcp 188.166.225.170:40310->45.91.116.10:4081: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40310->45.91.116.10:4081: use of closed network connection Aug 4 16:10:09.571764674 (118h12m59.509423173s) [noise=true , address=45.91.116.8:4048 , piece_id=CRNGR22XNQYXYNPUV72WOOXBTLJACNU6SLLP3ELSCZKFLFMWBYBQ, seg_pos={0 0}, upload=113578]: Done uploading piece. err: ecclient: upload failed (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, address:45.91.116.8:4048): protocol: read tcp 188.166.225.170:45132->45.91.116.8:4048: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 4 16:10:09.571926243 (118h12m59.509584742s) [noise=true , address=104.36.82.7:4909 , piece_id=B6JVXXE3CBHHKJJL2FOT5RC6JW3GLUD6J2GSOBPSHKTCJS64G5XQ, seg_pos={0 0}, upload=113578]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:104.36.82.7:4909): protocol: read tcp 188.166.225.170:56722->104.36.82.7:4909: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 4 16:10:09.572025102 (118h12m59.509683601s) [noise=true , address=45.91.116.8:4047 , piece_id=ZK3GOZHUZQQ7IHBLU3ZXG6CREPLSM7Q2HKVJK34Z3WRROKAQBYTA, seg_pos={0 0}, upload=113578]: Done uploading piece. err: ecclient: upload failed (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, address:45.91.116.8:4047): protocol: expected piece hash; context deadline exceeded; EOF Aug 4 16:10:09.572039179 (118h12m59.509697679s) [seg_pos={0 0}, upload=113578]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Aug 4 16:10:09.805762911 (118h12m59.74342142s) [upload=113578]: Done uploading segments... Aug 4 16:10:09.805856250 (118h12m59.743514749s) [upload=113578]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration:* 30.236275286s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 3 out of 60 is failed (threshold=3) [.X.........X.............................X..................] duration: 1.888011215s FAIL (2025-08-04 16:09:58 UTC) orbiter warning: Operation: DownloadObject uplink: object not found ("orbiter_us_0/1K") Duration: 1.888037475s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 11.748083512s FAIL (2025-08-04 16:09:38 UTC) orbiter warning: Operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 4.36.82.36:4625 , piece_id=FCK7QGPSKJ7VDK4D7BVNF44BRPQCQS5V5ICAJHELAQHDBSQV5R3Q, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.146274823 (118h10m17.083933322s) [noise=true , address=45.91.116.7:4026 , piece_id=CB6TYC4A6DQOFFKVFPNMF5MH5MGMJHU5Z25B4QXQJI5ETDWLQJJQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.146672965 (118h10m17.084331464s) [noise=true , address=104.36.82.16:4321 , piece_id=42MZPFPEBQNQRJUHQJLZQP2UWGDV2OAXT6Z4KTWPPWLALISASN5Q, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.146726166 (118h10m17.084384665s) [noise=true , address=104.36.82.55:4206 , piece_id=KSAKYO6MALVA5REEKKJZ7MGX7YIRC2PDI5PNHFHRASPBD47V5SOQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.147136371 (118h10m17.08479488s) [noise=true , address=104.36.82.42:4823 , piece_id=66Q6G7YBCVUEAHU3Y5LHPR5MIT7H7NQWI2TWWNHKZT5FYS4DNQBA, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.147479628 (118h10m17.085138127s) [noise=true , address=193.149.164.143:4695 , piece_id=3QTYKOTBCOJ4CRM76FR2XKONKQVYTQQMPTRH3E6MYJR4ODBPY3AQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.147821592 (118h10m17.085480091s) [noise=true , address=104.36.82.62:4411 , piece_id=CIJJ4SMWKFDOIZZ5UEYIXFZHUNAZKRMM2P36IJIT5BHUX5STDZ4A, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.148134220 (118h10m17.085792719s) [noise=true , address=104.36.82.55:4224 , piece_id=NJYMQ2467DXHUEC6ELXITFJF6WWKM4XFEONRJVRF6QSJBDEMC7JA, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.148548573 (118h10m17.086207072s) [noise=true , address=104.36.82.13:4193 , piece_id=WJA72IWFD3TUA2WPGJGB6A7AHJXI4PGOK62YDMKSMDPUAVHUHW4Q, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.148889576 (118h10m17.086548085s) [noise=true , address=104.36.82.39:4728 , piece_id=ZCL6YOUXY4P6HTYUTTILZR4TLG44CDTXWLOJJWOHPFSORA2IBCBA, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.149205931 (118h10m17.08686443s) [noise=true , address=104.36.82.59:4326 , piece_id=CS7FVO3IRXKZFB2EZM2HQBVEVM4XTTFI5Y7TH3NJZGIYMREVY7UQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.149509752 (118h10m17.087168261s) [noise=true , address=104.36.82.45:4930 , piece_id=LBMDUGYMQI7PBBGEOT22S7OGNVRIYS66USSYFAZOSGI445GWKCYA, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.149866545 (118h10m17.087525044s) [noise=true , address=104.36.82.23:4474 , piece_id=PLEKQKMEQKQDW72BNLLY4JOOPFI6CPNNGT74M5EFFNDQQFWELXUQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.150193601 (118h10m17.08785211s) [noise=true , address=104.36.82.13:4212 , piece_id=ER6VIEGFSG2F7542M5P66RS3QAUQYV7BWPQ4YOX55Q3XX6DDCTGA, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.150515557 (118h10m17.088174066s) [noise=true , address=104.36.82.46:4485 , piece_id=P4FMDR5NCQOLKSXQBLE7I7BPEZJ2URAPPVNKOR53ZPROBCE4YAJQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.150935149 (118h10m17.088593648s) [noise=true , address=104.36.82.54:4181 , piece_id=AHW774W42KLBATPDVRQRKH4MQWV5YP2XEEUY6VASDMYJ2BKXCS2Q, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.151249151 (118h10m17.08890765s) [noise=true , address=104.36.82.44:4886 , piece_id=Q3BIO4GNMY64IWXC7VZ5BAYKYRJJ3GKFV7Z4IOG7SP62IUJHXN2A, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.151276363 (118h10m17.088934862s) [noise=true , address=104.36.82.9:4019 , piece_id=SBCPU6XTMB4CRRTFKDBYFQQ2XZZNE3I6AF7VVSGYC356SPQNVHUQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:28.640156348 (118h10m18.577814857s) [upload=113540]: Next returned nil segment Aug 4 16:07:28.640223306 (118h10m18.577881825s) [upload=113540]: Waiting for error group managing segments... Aug 4 16:07:29.062127352 (118h10m18.999785841s) [noise=true , address=104.36.82.16:4321 , piece_id=42MZPFPEBQNQRJUHQJLZQP2UWGDV2OAXT6Z4KTWPPWLALISASN5Q, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.080565981 (118h10m19.01822448s) [noise=true , address=104.36.82.54:4181 , piece_id=AHW774W42KLBATPDVRQRKH4MQWV5YP2XEEUY6VASDMYJ2BKXCS2Q, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.081329852 (118h10m19.018988352s) [noise=true , address=104.36.82.62:4411 , piece_id=CIJJ4SMWKFDOIZZ5UEYIXFZHUNAZKRMM2P36IJIT5BHUX5STDZ4A, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.081846201 (118h10m19.0195047s) [noise=true , address=104.36.82.55:4224 , piece_id=NJYMQ2467DXHUEC6ELXITFJF6WWKM4XFEONRJVRF6QSJBDEMC7JA, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.091017397 (118h10m19.028675896s) [noise=true , address=104.36.82.46:4485 , piece_id=P4FMDR5NCQOLKSXQBLE7I7BPEZJ2URAPPVNKOR53ZPROBCE4YAJQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.105092769 (118h10m19.042751268s) [noise=true , address=104.36.82.13:4212 , piece_id=ER6VIEGFSG2F7542M5P66RS3QAUQYV7BWPQ4YOX55Q3XX6DDCTGA, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.107141759 (118h10m19.044800248s) [noise=true , address=104.36.82.23:4474 , piece_id=PLEKQKMEQKQDW72BNLLY4JOOPFI6CPNNGT74M5EFFNDQQFWELXUQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.116502399 (118h10m19.054160898s) [noise=true , address=45.91.116.7:4026 , piece_id=CB6TYC4A6DQOFFKVFPNMF5MH5MGMJHU5Z25B4QXQJI5ETDWLQJJQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.153680253 (118h10m19.091338752s) [noise=true , address=104.36.82.9:4019 , piece_id=SBCPU6XTMB4CRRTFKDBYFQQ2XZZNE3I6AF7VVSGYC356SPQNVHUQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.210191177 (118h10m19.147849686s) [noise=true , address=104.36.82.36:4625 , piece_id=FCK7QGPSKJ7VDK4D7BVNF44BRPQCQS5V5ICAJHELAQHDBSQV5R3Q, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.219317350 (118h10m19.156975849s) [noise=true , address=104.36.82.55:4206 , piece_id=KSAKYO6MALVA5REEKKJZ7MGX7YIRC2PDI5PNHFHRASPBD47V5SOQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:31.340384557 (118h10m21.278043046s) [noise=true , address=104.36.82.59:4326 , piece_id=CS7FVO3IRXKZFB2EZM2HQBVEVM4XTTFI5Y7TH3NJZGIYMREVY7UQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:32.464238230 (118h10m22.401896729s) [noise=true , address=104.36.82.13:4193 , piece_id=WJA72IWFD3TUA2WPGJGB6A7AHJXI4PGOK62YDMKSMDPUAVHUHW4Q, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:08:26.041033880 (118h11m15.978692379s) [noise=true , address=193.149.164.143:4695 , piece_id=3QTYKOTBCOJ4CRM76FR2XKONKQVYTQQMPTRH3E6MYJR4ODBPY3AQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: ecclient: upload failed (node:12TZfgz24gnZr4UryzEKw6gu94njnmCvafX5X64MKFSJqaUWSVh, address:193.149.164.143:4695): protocol: write tcp 188.166.225.170:43298->193.149.164.143:4695: use of closed network connection; write tcp 188.166.225.170:43298->193.149.164.143:4695: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43298->193.149.164.143:4695: use of closed network connection Aug 4 16:08:26.041079005 (118h11m15.978737504s) [noise=true , address=104.36.82.44:4886 , piece_id=Q3BIO4GNMY64IWXC7VZ5BAYKYRJJ3GKFV7Z4IOG7SP62IUJHXN2A, seg_pos={0 0}, upload=113540]: Done uploading piece. err: ecclient: upload failed (node:1n2cXhqd5EWbNPTXem83jQUjYq5ZaGKBidbVCQzxYfR3rWddGR, address:104.36.82.44:4886): protocol: write tcp 188.166.225.170:38030->104.36.82.44:4886: use of closed network connection; write tcp 188.166.225.170:38030->104.36.82.44:4886: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38030->104.36.82.44:4886: use of closed network connection Aug 4 16:08:26.041128580 (118h11m15.97878711s) [noise=true , address=104.36.82.39:4728 , piece_id=ZCL6YOUXY4P6HTYUTTILZR4TLG44CDTXWLOJJWOHPFSORA2IBCBA, seg_pos={0 0}, upload=113540]: Done uploading piece. err: ecclient: upload failed (node:1YoLzpb4mMYZ8TsdxnMocekZZ1gLfTrdCM8mxTDj4tY6khZGvi, address:104.36.82.39:4728): protocol: write tcp 188.166.225.170:34326->104.36.82.39:4728: use of closed network connection; write tcp 188.166.225.170:34326->104.36.82.39:4728: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34326->104.36.82.39:4728: use of closed network connection Aug 4 16:08:26.041195678 (118h11m15.978854187s) [noise=true , address=104.36.82.45:4930 , piece_id=LBMDUGYMQI7PBBGEOT22S7OGNVRIYS66USSYFAZOSGI445GWKCYA, seg_pos={0 0}, upload=113540]: Done uploading piece. err: ecclient: upload failed (node:12qaYFSpezuLAc5zJfAYruzad6nTpYFDpog7bmKMVTA8MVm76NB, address:104.36.82.45:4930): protocol: expected piece hash; context deadline exceeded; EOF Aug 4 16:08:26.041203374 (118h11m15.978861913s) [noise=true , address=104.36.82.42:4823 , piece_id=66Q6G7YBCVUEAHU3Y5LHPR5MIT7H7NQWI2TWWNHKZT5FYS4DNQBA, seg_pos={0 0}, upload=113540]: Done uploading piece. err: ecclient: upload failed (node:1fkA5DEj9RqsR6N85z7azvujy8tPkUhB5GCqBpUmFb4vvwbd8y, address:104.36.82.42:4823): protocol: write tcp 188.166.225.170:37230->104.36.82.42:4823: use of closed network connection; write tcp 188.166.225.170:37230->104.36.82.42:4823: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37230->104.36.82.42:4823: use of closed network connection Aug 4 16:08:26.041258960 (118h11m15.978917489s) [seg_pos={0 0}, upload=113540]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Aug 4 16:08:26.274088873 (118h11m16.211747372s) [upload=113540]: Done uploading segments... Aug 4 16:08:26.274113650 (118h11m16.211772139s) [upload=113540]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 11.74810306s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 30.000513084s FAIL (2025-08-04 16:09:34 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.000536789s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 3 out of 60 is failed (threshold=3) [..X.........X.............................X.................] duration: 1.834902929s FAIL (2025-08-04 16:09:12 UTC) orbiter warning: Operation: DownloadObject uplink: object not found ("orbiter_us_0/1K") Duration: 1.834934189s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 13.055456009s FAIL (2025-08-04 16:09:02 UTC) orbiter warning: Operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 4.36.82.36:4625 , piece_id=FCK7QGPSKJ7VDK4D7BVNF44BRPQCQS5V5ICAJHELAQHDBSQV5R3Q, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.146274823 (118h10m17.083933322s) [noise=true , address=45.91.116.7:4026 , piece_id=CB6TYC4A6DQOFFKVFPNMF5MH5MGMJHU5Z25B4QXQJI5ETDWLQJJQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.146672965 (118h10m17.084331464s) [noise=true , address=104.36.82.16:4321 , piece_id=42MZPFPEBQNQRJUHQJLZQP2UWGDV2OAXT6Z4KTWPPWLALISASN5Q, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.146726166 (118h10m17.084384665s) [noise=true , address=104.36.82.55:4206 , piece_id=KSAKYO6MALVA5REEKKJZ7MGX7YIRC2PDI5PNHFHRASPBD47V5SOQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.147136371 (118h10m17.08479488s) [noise=true , address=104.36.82.42:4823 , piece_id=66Q6G7YBCVUEAHU3Y5LHPR5MIT7H7NQWI2TWWNHKZT5FYS4DNQBA, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.147479628 (118h10m17.085138127s) [noise=true , address=193.149.164.143:4695 , piece_id=3QTYKOTBCOJ4CRM76FR2XKONKQVYTQQMPTRH3E6MYJR4ODBPY3AQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.147821592 (118h10m17.085480091s) [noise=true , address=104.36.82.62:4411 , piece_id=CIJJ4SMWKFDOIZZ5UEYIXFZHUNAZKRMM2P36IJIT5BHUX5STDZ4A, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.148134220 (118h10m17.085792719s) [noise=true , address=104.36.82.55:4224 , piece_id=NJYMQ2467DXHUEC6ELXITFJF6WWKM4XFEONRJVRF6QSJBDEMC7JA, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.148548573 (118h10m17.086207072s) [noise=true , address=104.36.82.13:4193 , piece_id=WJA72IWFD3TUA2WPGJGB6A7AHJXI4PGOK62YDMKSMDPUAVHUHW4Q, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.148889576 (118h10m17.086548085s) [noise=true , address=104.36.82.39:4728 , piece_id=ZCL6YOUXY4P6HTYUTTILZR4TLG44CDTXWLOJJWOHPFSORA2IBCBA, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.149205931 (118h10m17.08686443s) [noise=true , address=104.36.82.59:4326 , piece_id=CS7FVO3IRXKZFB2EZM2HQBVEVM4XTTFI5Y7TH3NJZGIYMREVY7UQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.149509752 (118h10m17.087168261s) [noise=true , address=104.36.82.45:4930 , piece_id=LBMDUGYMQI7PBBGEOT22S7OGNVRIYS66USSYFAZOSGI445GWKCYA, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.149866545 (118h10m17.087525044s) [noise=true , address=104.36.82.23:4474 , piece_id=PLEKQKMEQKQDW72BNLLY4JOOPFI6CPNNGT74M5EFFNDQQFWELXUQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.150193601 (118h10m17.08785211s) [noise=true , address=104.36.82.13:4212 , piece_id=ER6VIEGFSG2F7542M5P66RS3QAUQYV7BWPQ4YOX55Q3XX6DDCTGA, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.150515557 (118h10m17.088174066s) [noise=true , address=104.36.82.46:4485 , piece_id=P4FMDR5NCQOLKSXQBLE7I7BPEZJ2URAPPVNKOR53ZPROBCE4YAJQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.150935149 (118h10m17.088593648s) [noise=true , address=104.36.82.54:4181 , piece_id=AHW774W42KLBATPDVRQRKH4MQWV5YP2XEEUY6VASDMYJ2BKXCS2Q, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.151249151 (118h10m17.08890765s) [noise=true , address=104.36.82.44:4886 , piece_id=Q3BIO4GNMY64IWXC7VZ5BAYKYRJJ3GKFV7Z4IOG7SP62IUJHXN2A, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.151276363 (118h10m17.088934862s) [noise=true , address=104.36.82.9:4019 , piece_id=SBCPU6XTMB4CRRTFKDBYFQQ2XZZNE3I6AF7VVSGYC356SPQNVHUQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:28.640156348 (118h10m18.577814857s) [upload=113540]: Next returned nil segment Aug 4 16:07:28.640223306 (118h10m18.577881825s) [upload=113540]: Waiting for error group managing segments... Aug 4 16:07:29.062127352 (118h10m18.999785841s) [noise=true , address=104.36.82.16:4321 , piece_id=42MZPFPEBQNQRJUHQJLZQP2UWGDV2OAXT6Z4KTWPPWLALISASN5Q, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.080565981 (118h10m19.01822448s) [noise=true , address=104.36.82.54:4181 , piece_id=AHW774W42KLBATPDVRQRKH4MQWV5YP2XEEUY6VASDMYJ2BKXCS2Q, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.081329852 (118h10m19.018988352s) [noise=true , address=104.36.82.62:4411 , piece_id=CIJJ4SMWKFDOIZZ5UEYIXFZHUNAZKRMM2P36IJIT5BHUX5STDZ4A, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.081846201 (118h10m19.0195047s) [noise=true , address=104.36.82.55:4224 , piece_id=NJYMQ2467DXHUEC6ELXITFJF6WWKM4XFEONRJVRF6QSJBDEMC7JA, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.091017397 (118h10m19.028675896s) [noise=true , address=104.36.82.46:4485 , piece_id=P4FMDR5NCQOLKSXQBLE7I7BPEZJ2URAPPVNKOR53ZPROBCE4YAJQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.105092769 (118h10m19.042751268s) [noise=true , address=104.36.82.13:4212 , piece_id=ER6VIEGFSG2F7542M5P66RS3QAUQYV7BWPQ4YOX55Q3XX6DDCTGA, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.107141759 (118h10m19.044800248s) [noise=true , address=104.36.82.23:4474 , piece_id=PLEKQKMEQKQDW72BNLLY4JOOPFI6CPNNGT74M5EFFNDQQFWELXUQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.116502399 (118h10m19.054160898s) [noise=true , address=45.91.116.7:4026 , piece_id=CB6TYC4A6DQOFFKVFPNMF5MH5MGMJHU5Z25B4QXQJI5ETDWLQJJQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.153680253 (118h10m19.091338752s) [noise=true , address=104.36.82.9:4019 , piece_id=SBCPU6XTMB4CRRTFKDBYFQQ2XZZNE3I6AF7VVSGYC356SPQNVHUQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.210191177 (118h10m19.147849686s) [noise=true , address=104.36.82.36:4625 , piece_id=FCK7QGPSKJ7VDK4D7BVNF44BRPQCQS5V5ICAJHELAQHDBSQV5R3Q, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.219317350 (118h10m19.156975849s) [noise=true , address=104.36.82.55:4206 , piece_id=KSAKYO6MALVA5REEKKJZ7MGX7YIRC2PDI5PNHFHRASPBD47V5SOQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:31.340384557 (118h10m21.278043046s) [noise=true , address=104.36.82.59:4326 , piece_id=CS7FVO3IRXKZFB2EZM2HQBVEVM4XTTFI5Y7TH3NJZGIYMREVY7UQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:32.464238230 (118h10m22.401896729s) [noise=true , address=104.36.82.13:4193 , piece_id=WJA72IWFD3TUA2WPGJGB6A7AHJXI4PGOK62YDMKSMDPUAVHUHW4Q, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:08:26.041033880 (118h11m15.978692379s) [noise=true , address=193.149.164.143:4695 , piece_id=3QTYKOTBCOJ4CRM76FR2XKONKQVYTQQMPTRH3E6MYJR4ODBPY3AQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: ecclient: upload failed (node:12TZfgz24gnZr4UryzEKw6gu94njnmCvafX5X64MKFSJqaUWSVh, address:193.149.164.143:4695): protocol: write tcp 188.166.225.170:43298->193.149.164.143:4695: use of closed network connection; write tcp 188.166.225.170:43298->193.149.164.143:4695: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43298->193.149.164.143:4695: use of closed network connection Aug 4 16:08:26.041079005 (118h11m15.978737504s) [noise=true , address=104.36.82.44:4886 , piece_id=Q3BIO4GNMY64IWXC7VZ5BAYKYRJJ3GKFV7Z4IOG7SP62IUJHXN2A, seg_pos={0 0}, upload=113540]: Done uploading piece. err: ecclient: upload failed (node:1n2cXhqd5EWbNPTXem83jQUjYq5ZaGKBidbVCQzxYfR3rWddGR, address:104.36.82.44:4886): protocol: write tcp 188.166.225.170:38030->104.36.82.44:4886: use of closed network connection; write tcp 188.166.225.170:38030->104.36.82.44:4886: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38030->104.36.82.44:4886: use of closed network connection Aug 4 16:08:26.041128580 (118h11m15.97878711s) [noise=true , address=104.36.82.39:4728 , piece_id=ZCL6YOUXY4P6HTYUTTILZR4TLG44CDTXWLOJJWOHPFSORA2IBCBA, seg_pos={0 0}, upload=113540]: Done uploading piece. err: ecclient: upload failed (node:1YoLzpb4mMYZ8TsdxnMocekZZ1gLfTrdCM8mxTDj4tY6khZGvi, address:104.36.82.39:4728): protocol: write tcp 188.166.225.170:34326->104.36.82.39:4728: use of closed network connection; write tcp 188.166.225.170:34326->104.36.82.39:4728: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34326->104.36.82.39:4728: use of closed network connection Aug 4 16:08:26.041195678 (118h11m15.978854187s) [noise=true , address=104.36.82.45:4930 , piece_id=LBMDUGYMQI7PBBGEOT22S7OGNVRIYS66USSYFAZOSGI445GWKCYA, seg_pos={0 0}, upload=113540]: Done uploading piece. err: ecclient: upload failed (node:12qaYFSpezuLAc5zJfAYruzad6nTpYFDpog7bmKMVTA8MVm76NB, address:104.36.82.45:4930): protocol: expected piece hash; context deadline exceeded; EOF Aug 4 16:08:26.041203374 (118h11m15.978861913s) [noise=true , address=104.36.82.42:4823 , piece_id=66Q6G7YBCVUEAHU3Y5LHPR5MIT7H7NQWI2TWWNHKZT5FYS4DNQBA, seg_pos={0 0}, upload=113540]: Done uploading piece. err: ecclient: upload failed (node:1fkA5DEj9RqsR6N85z7azvujy8tPkUhB5GCqBpUmFb4vvwbd8y, address:104.36.82.42:4823): protocol: write tcp 188.166.225.170:37230->104.36.82.42:4823: use of closed network connection; write tcp 188.166.225.170:37230->104.36.82.42:4823: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37230->104.36.82.42:4823: use of closed network connection Aug 4 16:08:26.041258960 (118h11m15.978917489s) [seg_pos={0 0}, upload=113540]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Aug 4 16:08:26.274088873 (118h11m16.211747372s) [upload=113540]: Done uploading segments... Aug 4 16:08:26.274113650 (118h11m16.211772139s) [upload=113540]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 13.05548294s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 3 out of 60 is failed (threshold=3) [...X.........X.............................X................] duration: 2.05832258s FAIL (2025-08-04 16:08:56 UTC) orbiter warning: Operation: DownloadObject uplink: object not found ("orbiter_us_0/1K") Duration: 2.058352928s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 11.116035519s FAIL (2025-08-04 16:08:37 UTC) orbiter warning: Operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 4.36.82.36:4625 , piece_id=FCK7QGPSKJ7VDK4D7BVNF44BRPQCQS5V5ICAJHELAQHDBSQV5R3Q, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.146274823 (118h10m17.083933322s) [noise=true , address=45.91.116.7:4026 , piece_id=CB6TYC4A6DQOFFKVFPNMF5MH5MGMJHU5Z25B4QXQJI5ETDWLQJJQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.146672965 (118h10m17.084331464s) [noise=true , address=104.36.82.16:4321 , piece_id=42MZPFPEBQNQRJUHQJLZQP2UWGDV2OAXT6Z4KTWPPWLALISASN5Q, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.146726166 (118h10m17.084384665s) [noise=true , address=104.36.82.55:4206 , piece_id=KSAKYO6MALVA5REEKKJZ7MGX7YIRC2PDI5PNHFHRASPBD47V5SOQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.147136371 (118h10m17.08479488s) [noise=true , address=104.36.82.42:4823 , piece_id=66Q6G7YBCVUEAHU3Y5LHPR5MIT7H7NQWI2TWWNHKZT5FYS4DNQBA, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.147479628 (118h10m17.085138127s) [noise=true , address=193.149.164.143:4695 , piece_id=3QTYKOTBCOJ4CRM76FR2XKONKQVYTQQMPTRH3E6MYJR4ODBPY3AQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.147821592 (118h10m17.085480091s) [noise=true , address=104.36.82.62:4411 , piece_id=CIJJ4SMWKFDOIZZ5UEYIXFZHUNAZKRMM2P36IJIT5BHUX5STDZ4A, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.148134220 (118h10m17.085792719s) [noise=true , address=104.36.82.55:4224 , piece_id=NJYMQ2467DXHUEC6ELXITFJF6WWKM4XFEONRJVRF6QSJBDEMC7JA, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.148548573 (118h10m17.086207072s) [noise=true , address=104.36.82.13:4193 , piece_id=WJA72IWFD3TUA2WPGJGB6A7AHJXI4PGOK62YDMKSMDPUAVHUHW4Q, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.148889576 (118h10m17.086548085s) [noise=true , address=104.36.82.39:4728 , piece_id=ZCL6YOUXY4P6HTYUTTILZR4TLG44CDTXWLOJJWOHPFSORA2IBCBA, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.149205931 (118h10m17.08686443s) [noise=true , address=104.36.82.59:4326 , piece_id=CS7FVO3IRXKZFB2EZM2HQBVEVM4XTTFI5Y7TH3NJZGIYMREVY7UQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.149509752 (118h10m17.087168261s) [noise=true , address=104.36.82.45:4930 , piece_id=LBMDUGYMQI7PBBGEOT22S7OGNVRIYS66USSYFAZOSGI445GWKCYA, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.149866545 (118h10m17.087525044s) [noise=true , address=104.36.82.23:4474 , piece_id=PLEKQKMEQKQDW72BNLLY4JOOPFI6CPNNGT74M5EFFNDQQFWELXUQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.150193601 (118h10m17.08785211s) [noise=true , address=104.36.82.13:4212 , piece_id=ER6VIEGFSG2F7542M5P66RS3QAUQYV7BWPQ4YOX55Q3XX6DDCTGA, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.150515557 (118h10m17.088174066s) [noise=true , address=104.36.82.46:4485 , piece_id=P4FMDR5NCQOLKSXQBLE7I7BPEZJ2URAPPVNKOR53ZPROBCE4YAJQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.150935149 (118h10m17.088593648s) [noise=true , address=104.36.82.54:4181 , piece_id=AHW774W42KLBATPDVRQRKH4MQWV5YP2XEEUY6VASDMYJ2BKXCS2Q, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.151249151 (118h10m17.08890765s) [noise=true , address=104.36.82.44:4886 , piece_id=Q3BIO4GNMY64IWXC7VZ5BAYKYRJJ3GKFV7Z4IOG7SP62IUJHXN2A, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.151276363 (118h10m17.088934862s) [noise=true , address=104.36.82.9:4019 , piece_id=SBCPU6XTMB4CRRTFKDBYFQQ2XZZNE3I6AF7VVSGYC356SPQNVHUQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:28.640156348 (118h10m18.577814857s) [upload=113540]: Next returned nil segment Aug 4 16:07:28.640223306 (118h10m18.577881825s) [upload=113540]: Waiting for error group managing segments... Aug 4 16:07:29.062127352 (118h10m18.999785841s) [noise=true , address=104.36.82.16:4321 , piece_id=42MZPFPEBQNQRJUHQJLZQP2UWGDV2OAXT6Z4KTWPPWLALISASN5Q, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.080565981 (118h10m19.01822448s) [noise=true , address=104.36.82.54:4181 , piece_id=AHW774W42KLBATPDVRQRKH4MQWV5YP2XEEUY6VASDMYJ2BKXCS2Q, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.081329852 (118h10m19.018988352s) [noise=true , address=104.36.82.62:4411 , piece_id=CIJJ4SMWKFDOIZZ5UEYIXFZHUNAZKRMM2P36IJIT5BHUX5STDZ4A, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.081846201 (118h10m19.0195047s) [noise=true , address=104.36.82.55:4224 , piece_id=NJYMQ2467DXHUEC6ELXITFJF6WWKM4XFEONRJVRF6QSJBDEMC7JA, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.091017397 (118h10m19.028675896s) [noise=true , address=104.36.82.46:4485 , piece_id=P4FMDR5NCQOLKSXQBLE7I7BPEZJ2URAPPVNKOR53ZPROBCE4YAJQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.105092769 (118h10m19.042751268s) [noise=true , address=104.36.82.13:4212 , piece_id=ER6VIEGFSG2F7542M5P66RS3QAUQYV7BWPQ4YOX55Q3XX6DDCTGA, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.107141759 (118h10m19.044800248s) [noise=true , address=104.36.82.23:4474 , piece_id=PLEKQKMEQKQDW72BNLLY4JOOPFI6CPNNGT74M5EFFNDQQFWELXUQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.116502399 (118h10m19.054160898s) [noise=true , address=45.91.116.7:4026 , piece_id=CB6TYC4A6DQOFFKVFPNMF5MH5MGMJHU5Z25B4QXQJI5ETDWLQJJQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.153680253 (118h10m19.091338752s) [noise=true , address=104.36.82.9:4019 , piece_id=SBCPU6XTMB4CRRTFKDBYFQQ2XZZNE3I6AF7VVSGYC356SPQNVHUQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.210191177 (118h10m19.147849686s) [noise=true , address=104.36.82.36:4625 , piece_id=FCK7QGPSKJ7VDK4D7BVNF44BRPQCQS5V5ICAJHELAQHDBSQV5R3Q, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.219317350 (118h10m19.156975849s) [noise=true , address=104.36.82.55:4206 , piece_id=KSAKYO6MALVA5REEKKJZ7MGX7YIRC2PDI5PNHFHRASPBD47V5SOQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:31.340384557 (118h10m21.278043046s) [noise=true , address=104.36.82.59:4326 , piece_id=CS7FVO3IRXKZFB2EZM2HQBVEVM4XTTFI5Y7TH3NJZGIYMREVY7UQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:32.464238230 (118h10m22.401896729s) [noise=true , address=104.36.82.13:4193 , piece_id=WJA72IWFD3TUA2WPGJGB6A7AHJXI4PGOK62YDMKSMDPUAVHUHW4Q, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:08:26.041033880 (118h11m15.978692379s) [noise=true , address=193.149.164.143:4695 , piece_id=3QTYKOTBCOJ4CRM76FR2XKONKQVYTQQMPTRH3E6MYJR4ODBPY3AQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: ecclient: upload failed (node:12TZfgz24gnZr4UryzEKw6gu94njnmCvafX5X64MKFSJqaUWSVh, address:193.149.164.143:4695): protocol: write tcp 188.166.225.170:43298->193.149.164.143:4695: use of closed network connection; write tcp 188.166.225.170:43298->193.149.164.143:4695: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43298->193.149.164.143:4695: use of closed network connection Aug 4 16:08:26.041079005 (118h11m15.978737504s) [noise=true , address=104.36.82.44:4886 , piece_id=Q3BIO4GNMY64IWXC7VZ5BAYKYRJJ3GKFV7Z4IOG7SP62IUJHXN2A, seg_pos={0 0}, upload=113540]: Done uploading piece. err: ecclient: upload failed (node:1n2cXhqd5EWbNPTXem83jQUjYq5ZaGKBidbVCQzxYfR3rWddGR, address:104.36.82.44:4886): protocol: write tcp 188.166.225.170:38030->104.36.82.44:4886: use of closed network connection; write tcp 188.166.225.170:38030->104.36.82.44:4886: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38030->104.36.82.44:4886: use of closed network connection Aug 4 16:08:26.041128580 (118h11m15.97878711s) [noise=true , address=104.36.82.39:4728 , piece_id=ZCL6YOUXY4P6HTYUTTILZR4TLG44CDTXWLOJJWOHPFSORA2IBCBA, seg_pos={0 0}, upload=113540]: Done uploading piece. err: ecclient: upload failed (node:1YoLzpb4mMYZ8TsdxnMocekZZ1gLfTrdCM8mxTDj4tY6khZGvi, address:104.36.82.39:4728): protocol: write tcp 188.166.225.170:34326->104.36.82.39:4728: use of closed network connection; write tcp 188.166.225.170:34326->104.36.82.39:4728: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34326->104.36.82.39:4728: use of closed network connection Aug 4 16:08:26.041195678 (118h11m15.978854187s) [noise=true , address=104.36.82.45:4930 , piece_id=LBMDUGYMQI7PBBGEOT22S7OGNVRIYS66USSYFAZOSGI445GWKCYA, seg_pos={0 0}, upload=113540]: Done uploading piece. err: ecclient: upload failed (node:12qaYFSpezuLAc5zJfAYruzad6nTpYFDpog7bmKMVTA8MVm76NB, address:104.36.82.45:4930): protocol: expected piece hash; context deadline exceeded; EOF Aug 4 16:08:26.041203374 (118h11m15.978861913s) [noise=true , address=104.36.82.42:4823 , piece_id=66Q6G7YBCVUEAHU3Y5LHPR5MIT7H7NQWI2TWWNHKZT5FYS4DNQBA, seg_pos={0 0}, upload=113540]: Done uploading piece. err: ecclient: upload failed (node:1fkA5DEj9RqsR6N85z7azvujy8tPkUhB5GCqBpUmFb4vvwbd8y, address:104.36.82.42:4823): protocol: write tcp 188.166.225.170:37230->104.36.82.42:4823: use of closed network connection; write tcp 188.166.225.170:37230->104.36.82.42:4823: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37230->104.36.82.42:4823: use of closed network connection Aug 4 16:08:26.041258960 (118h11m15.978917489s) [seg_pos={0 0}, upload=113540]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Aug 4 16:08:26.274088873 (118h11m16.211747372s) [upload=113540]: Done uploading segments... Aug 4 16:08:26.274113650 (118h11m16.211772139s) [upload=113540]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 11.116048493s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 3 out of 60 is failed (threshold=3) [....X.........X.............................X...............] duration: 1.966948892s FAIL (2025-08-04 16:08:37 UTC) orbiter warning: Operation: DownloadObject uplink: object not found ("orbiter_us_0/1K") Duration: 1.966976024s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 1m0.234129696s FAIL (2025-08-04 16:08:26 UTC) orbiter warning: Operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 4.36.82.36:4625 , piece_id=FCK7QGPSKJ7VDK4D7BVNF44BRPQCQS5V5ICAJHELAQHDBSQV5R3Q, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.146274823 (118h10m17.083933322s) [noise=true , address=45.91.116.7:4026 , piece_id=CB6TYC4A6DQOFFKVFPNMF5MH5MGMJHU5Z25B4QXQJI5ETDWLQJJQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.146672965 (118h10m17.084331464s) [noise=true , address=104.36.82.16:4321 , piece_id=42MZPFPEBQNQRJUHQJLZQP2UWGDV2OAXT6Z4KTWPPWLALISASN5Q, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.146726166 (118h10m17.084384665s) [noise=true , address=104.36.82.55:4206 , piece_id=KSAKYO6MALVA5REEKKJZ7MGX7YIRC2PDI5PNHFHRASPBD47V5SOQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.147136371 (118h10m17.08479488s) [noise=true , address=104.36.82.42:4823 , piece_id=66Q6G7YBCVUEAHU3Y5LHPR5MIT7H7NQWI2TWWNHKZT5FYS4DNQBA, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.147479628 (118h10m17.085138127s) [noise=true , address=193.149.164.143:4695 , piece_id=3QTYKOTBCOJ4CRM76FR2XKONKQVYTQQMPTRH3E6MYJR4ODBPY3AQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.147821592 (118h10m17.085480091s) [noise=true , address=104.36.82.62:4411 , piece_id=CIJJ4SMWKFDOIZZ5UEYIXFZHUNAZKRMM2P36IJIT5BHUX5STDZ4A, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.148134220 (118h10m17.085792719s) [noise=true , address=104.36.82.55:4224 , piece_id=NJYMQ2467DXHUEC6ELXITFJF6WWKM4XFEONRJVRF6QSJBDEMC7JA, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.148548573 (118h10m17.086207072s) [noise=true , address=104.36.82.13:4193 , piece_id=WJA72IWFD3TUA2WPGJGB6A7AHJXI4PGOK62YDMKSMDPUAVHUHW4Q, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.148889576 (118h10m17.086548085s) [noise=true , address=104.36.82.39:4728 , piece_id=ZCL6YOUXY4P6HTYUTTILZR4TLG44CDTXWLOJJWOHPFSORA2IBCBA, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.149205931 (118h10m17.08686443s) [noise=true , address=104.36.82.59:4326 , piece_id=CS7FVO3IRXKZFB2EZM2HQBVEVM4XTTFI5Y7TH3NJZGIYMREVY7UQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.149509752 (118h10m17.087168261s) [noise=true , address=104.36.82.45:4930 , piece_id=LBMDUGYMQI7PBBGEOT22S7OGNVRIYS66USSYFAZOSGI445GWKCYA, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.149866545 (118h10m17.087525044s) [noise=true , address=104.36.82.23:4474 , piece_id=PLEKQKMEQKQDW72BNLLY4JOOPFI6CPNNGT74M5EFFNDQQFWELXUQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.150193601 (118h10m17.08785211s) [noise=true , address=104.36.82.13:4212 , piece_id=ER6VIEGFSG2F7542M5P66RS3QAUQYV7BWPQ4YOX55Q3XX6DDCTGA, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.150515557 (118h10m17.088174066s) [noise=true , address=104.36.82.46:4485 , piece_id=P4FMDR5NCQOLKSXQBLE7I7BPEZJ2URAPPVNKOR53ZPROBCE4YAJQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.150935149 (118h10m17.088593648s) [noise=true , address=104.36.82.54:4181 , piece_id=AHW774W42KLBATPDVRQRKH4MQWV5YP2XEEUY6VASDMYJ2BKXCS2Q, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.151249151 (118h10m17.08890765s) [noise=true , address=104.36.82.44:4886 , piece_id=Q3BIO4GNMY64IWXC7VZ5BAYKYRJJ3GKFV7Z4IOG7SP62IUJHXN2A, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:27.151276363 (118h10m17.088934862s) [noise=true , address=104.36.82.9:4019 , piece_id=SBCPU6XTMB4CRRTFKDBYFQQ2XZZNE3I6AF7VVSGYC356SPQNVHUQ, seg_pos={0 0}, upload=113540]: Uploading piece... Aug 4 16:07:28.640156348 (118h10m18.577814857s) [upload=113540]: Next returned nil segment Aug 4 16:07:28.640223306 (118h10m18.577881825s) [upload=113540]: Waiting for error group managing segments... Aug 4 16:07:29.062127352 (118h10m18.999785841s) [noise=true , address=104.36.82.16:4321 , piece_id=42MZPFPEBQNQRJUHQJLZQP2UWGDV2OAXT6Z4KTWPPWLALISASN5Q, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.080565981 (118h10m19.01822448s) [noise=true , address=104.36.82.54:4181 , piece_id=AHW774W42KLBATPDVRQRKH4MQWV5YP2XEEUY6VASDMYJ2BKXCS2Q, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.081329852 (118h10m19.018988352s) [noise=true , address=104.36.82.62:4411 , piece_id=CIJJ4SMWKFDOIZZ5UEYIXFZHUNAZKRMM2P36IJIT5BHUX5STDZ4A, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.081846201 (118h10m19.0195047s) [noise=true , address=104.36.82.55:4224 , piece_id=NJYMQ2467DXHUEC6ELXITFJF6WWKM4XFEONRJVRF6QSJBDEMC7JA, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.091017397 (118h10m19.028675896s) [noise=true , address=104.36.82.46:4485 , piece_id=P4FMDR5NCQOLKSXQBLE7I7BPEZJ2URAPPVNKOR53ZPROBCE4YAJQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.105092769 (118h10m19.042751268s) [noise=true , address=104.36.82.13:4212 , piece_id=ER6VIEGFSG2F7542M5P66RS3QAUQYV7BWPQ4YOX55Q3XX6DDCTGA, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.107141759 (118h10m19.044800248s) [noise=true , address=104.36.82.23:4474 , piece_id=PLEKQKMEQKQDW72BNLLY4JOOPFI6CPNNGT74M5EFFNDQQFWELXUQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.116502399 (118h10m19.054160898s) [noise=true , address=45.91.116.7:4026 , piece_id=CB6TYC4A6DQOFFKVFPNMF5MH5MGMJHU5Z25B4QXQJI5ETDWLQJJQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.153680253 (118h10m19.091338752s) [noise=true , address=104.36.82.9:4019 , piece_id=SBCPU6XTMB4CRRTFKDBYFQQ2XZZNE3I6AF7VVSGYC356SPQNVHUQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.210191177 (118h10m19.147849686s) [noise=true , address=104.36.82.36:4625 , piece_id=FCK7QGPSKJ7VDK4D7BVNF44BRPQCQS5V5ICAJHELAQHDBSQV5R3Q, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:29.219317350 (118h10m19.156975849s) [noise=true , address=104.36.82.55:4206 , piece_id=KSAKYO6MALVA5REEKKJZ7MGX7YIRC2PDI5PNHFHRASPBD47V5SOQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:31.340384557 (118h10m21.278043046s) [noise=true , address=104.36.82.59:4326 , piece_id=CS7FVO3IRXKZFB2EZM2HQBVEVM4XTTFI5Y7TH3NJZGIYMREVY7UQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:07:32.464238230 (118h10m22.401896729s) [noise=true , address=104.36.82.13:4193 , piece_id=WJA72IWFD3TUA2WPGJGB6A7AHJXI4PGOK62YDMKSMDPUAVHUHW4Q, seg_pos={0 0}, upload=113540]: Done uploading piece. err: <nil> Aug 4 16:08:26.041033880 (118h11m15.978692379s) [noise=true , address=193.149.164.143:4695 , piece_id=3QTYKOTBCOJ4CRM76FR2XKONKQVYTQQMPTRH3E6MYJR4ODBPY3AQ, seg_pos={0 0}, upload=113540]: Done uploading piece. err: ecclient: upload failed (node:12TZfgz24gnZr4UryzEKw6gu94njnmCvafX5X64MKFSJqaUWSVh, address:193.149.164.143:4695): protocol: write tcp 188.166.225.170:43298->193.149.164.143:4695: use of closed network connection; write tcp 188.166.225.170:43298->193.149.164.143:4695: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43298->193.149.164.143:4695: use of closed network connection Aug 4 16:08:26.041079005 (118h11m15.978737504s) [noise=true , address=104.36.82.44:4886 , piece_id=Q3BIO4GNMY64IWXC7VZ5BAYKYRJJ3GKFV7Z4IOG7SP62IUJHXN2A, seg_pos={0 0}, upload=113540]: Done uploading piece. err: ecclient: upload failed (node:1n2cXhqd5EWbNPTXem83jQUjYq5ZaGKBidbVCQzxYfR3rWddGR, address:104.36.82.44:4886): protocol: write tcp 188.166.225.170:38030->104.36.82.44:4886: use of closed network connection; write tcp 188.166.225.170:38030->104.36.82.44:4886: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38030->104.36.82.44:4886: use of closed network connection Aug 4 16:08:26.041128580 (118h11m15.97878711s) [noise=true , address=104.36.82.39:4728 , piece_id=ZCL6YOUXY4P6HTYUTTILZR4TLG44CDTXWLOJJWOHPFSORA2IBCBA, seg_pos={0 0}, upload=113540]: Done uploading piece. err: ecclient: upload failed (node:1YoLzpb4mMYZ8TsdxnMocekZZ1gLfTrdCM8mxTDj4tY6khZGvi, address:104.36.82.39:4728): protocol: write tcp 188.166.225.170:34326->104.36.82.39:4728: use of closed network connection; write tcp 188.166.225.170:34326->104.36.82.39:4728: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34326->104.36.82.39:4728: use of closed network connection Aug 4 16:08:26.041195678 (118h11m15.978854187s) [noise=true , address=104.36.82.45:4930 , piece_id=LBMDUGYMQI7PBBGEOT22S7OGNVRIYS66USSYFAZOSGI445GWKCYA, seg_pos={0 0}, upload=113540]: Done uploading piece. err: ecclient: upload failed (node:12qaYFSpezuLAc5zJfAYruzad6nTpYFDpog7bmKMVTA8MVm76NB, address:104.36.82.45:4930): protocol: expected piece hash; context deadline exceeded; EOF Aug 4 16:08:26.041203374 (118h11m15.978861913s) [noise=true , address=104.36.82.42:4823 , piece_id=66Q6G7YBCVUEAHU3Y5LHPR5MIT7H7NQWI2TWWNHKZT5FYS4DNQBA, seg_pos={0 0}, upload=113540]: Done uploading piece. err: ecclient: upload failed (node:1fkA5DEj9RqsR6N85z7azvujy8tPkUhB5GCqBpUmFb4vvwbd8y, address:104.36.82.42:4823): protocol: write tcp 188.166.225.170:37230->104.36.82.42:4823: use of closed network connection; write tcp 188.166.225.170:37230->104.36.82.42:4823: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37230->104.36.82.42:4823: use of closed network connection Aug 4 16:08:26.041258960 (118h11m15.978917489s) [seg_pos={0 0}, upload=113540]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Aug 4 16:08:26.274088873 (118h11m16.211747372s) [upload=113540]: Done uploading segments... Aug 4 16:08:26.274113650 (118h11m16.211772139s) [upload=113540]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1m0.234151298s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)