Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [................XXX.........................................] duration: 6.025249706s FAIL (2025-08-17 04:52:27 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== g_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.761278730 (202h56m21.192358861s) [noise=true , address=104.36.82.34:4558 , piece_id=XGFNEBBVZX5UWJN7DALGM2EDG26POCTSBBAUU7DLYQKS2DNSBNXA, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.761661883 (202h56m21.192742014s) [noise=true , address=104.36.82.35:4594 , piece_id=6NUSPNYGSSJOB2CNPIW6DI45MPZARLWDBXH3CWMP4LXMDB2CXWZQ, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.761716548 (202h56m21.192796679s) [noise=true , address=45.91.116.8:4067 , piece_id=EZAI7ZDUBOOWXYJNTB7ZPOUSBUUG2FKV77QVUOZMNM6TUJSKGYWA, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.761972016 (202h56m21.193052147s) [noise=true , address=104.36.82.58:4283 , piece_id=LJKEH2KDSOFPVFPB5UWGVLOHQVZQOY33MYSHFLP4ZWGPKZ4KC64A, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.762254878 (202h56m21.193335009s) [noise=true , address=79.127.201.215:4345 , piece_id=JMSKNIZ7QPGBXIEP5RA3S74XAU5NJ2MUUXSFFIAK377YFGO646YQ, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.762523682 (202h56m21.193603803s) [noise=true , address=45.91.116.13:4181 , piece_id=7VN6RQGOTBHQO6HXM5IFTU32PWNVPH6BJZMQNETJEXOYPR2Y7OCQ, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.762864294 (202h56m21.193944415s) [noise=true , address=45.91.116.14:4203 , piece_id=5LXGQ4JZD2KLW3OMWY5QYRE4MKQNKKSAIMESODKDDT2W33XD45DQ, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.763179537 (202h56m21.194259668s) [noise=true , address=77.90.31.18:4815 , piece_id=G6PINAFFA6DZRRMKU2XK2LYFOES5WG7PTJ5L2ZPUQHMA4JNSRZRQ, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.763478499 (202h56m21.19455864s) [noise=true , address=77.90.31.11:4562 , piece_id=PUL6TAVIIU7732IOASNH5EHQMNCK7DWHJMUJVXRNK74T4GUQDGTA, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.763757683 (202h56m21.194837814s) [noise=true , address=104.36.82.5:4855 , piece_id=3FCDYA6CRYOOU3MDFJPEC4YQNICFVCN2A47RIYPSR4ZX5TIXZX3A, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.763988195 (202h56m21.195068326s) [noise=true , address=45.91.116.5:4308 , piece_id=F52M3VMHRQWR5LIEEPMVKVBFMOVG76UNNGQTFKSF6ZBD7VEOFWVA, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.764273320 (202h56m21.195353451s) [noise=true , address=66.23.193.175:4559 , piece_id=46X42PJIPA6DVC7BG7WPKMPMC44ZBUROEAB5C22M5INXKBJC4KVQ, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.764555100 (202h56m21.195635221s) [noise=true , address=77.90.31.15:4732 , piece_id=4AU7UGKXN4TIPSIRR7IMXSF6WHELO6UTOX35N5X7JHOQQXWWJF2A, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.764757416 (202h56m21.195837547s) [noise=true , address=104.36.82.39:4753 , piece_id=R7BHT5RYLSCDZLTDDRXTSJFSRPLM5PNBYC6Q2G33C5SYIUCNVCLQ, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.765868993 (202h56m21.196949124s) [noise=true , address=104.36.82.50:4047 , piece_id=AKDACFQEPBAFGYKTWBL5RELLZL3L4IJHJT73NS7T7QPWK2C2TDQA, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.767152809 (202h56m21.19823294s) [noise=true , address=104.36.82.19:4436 , piece_id=KVDJTKON4LFVJSAY7XSJ2OQ4ZH67JLMDE2O4EHUWSMZ3EECWSOXA, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.768577886 (202h56m21.199658017s) [noise=true , address=45.91.116.10:4095 , piece_id=JQ2UARQSNKBGHHHBJ6PR45ETR4EYMCXPDEQHEGWFYH52LRJC57UQ, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:27.934309011 (202h56m22.365389142s) [upload=211101]: Next returned nil segment Aug 17 04:22:27.934342545 (202h56m22.365422666s) [upload=211101]: Waiting for error group managing segments... Aug 17 04:22:28.231985083 (202h56m22.663065214s) [noise=true , address=66.23.193.175:4559 , piece_id=46X42PJIPA6DVC7BG7WPKMPMC44ZBUROEAB5C22M5INXKBJC4KVQ, seg_pos={0 0}, upload=211101]: Done uploading piece. err: <nil> Aug 17 04:22:28.670701256 (202h56m23.101781427s) [noise=true , address=45.91.116.10:4095 , piece_id=JQ2UARQSNKBGHHHBJ6PR45ETR4EYMCXPDEQHEGWFYH52LRJC57UQ, seg_pos={0 0}, upload=211101]: Done uploading piece. err: <nil> Aug 17 04:22:28.671952431 (202h56m23.103032562s) [noise=true , address=45.91.116.5:4308 , piece_id=F52M3VMHRQWR5LIEEPMVKVBFMOVG76UNNGQTFKSF6ZBD7VEOFWVA, seg_pos={0 0}, upload=211101]: Done uploading piece. err: <nil> Aug 17 04:22:28.698468278 (202h56m23.129548409s) [noise=true , address=45.91.116.8:4067 , piece_id=EZAI7ZDUBOOWXYJNTB7ZPOUSBUUG2FKV77QVUOZMNM6TUJSKGYWA, seg_pos={0 0}, upload=211101]: Done uploading piece. err: <nil> Aug 17 04:22:28.720171187 (202h56m23.151251318s) [noise=true , address=45.91.116.13:4181 , piece_id=7VN6RQGOTBHQO6HXM5IFTU32PWNVPH6BJZMQNETJEXOYPR2Y7OCQ, seg_pos={0 0}, upload=211101]: Done uploading piece. err: <nil> Aug 17 04:22:28.731591429 (202h56m23.16267156s) [noise=true , address=45.91.116.14:4203 , piece_id=5LXGQ4JZD2KLW3OMWY5QYRE4MKQNKKSAIMESODKDDT2W33XD45DQ, seg_pos={0 0}, upload=211101]: Done uploading piece. err: <nil> Aug 17 04:22:28.798148986 (202h56m23.229229117s) [noise=true , address=77.90.31.18:4815 , piece_id=G6PINAFFA6DZRRMKU2XK2LYFOES5WG7PTJ5L2ZPUQHMA4JNSRZRQ, seg_pos={0 0}, upload=211101]: Done uploading piece. err: <nil> Aug 17 04:22:28.804849037 (202h56m23.235929168s) [noise=true , address=79.127.201.215:4345 , piece_id=JMSKNIZ7QPGBXIEP5RA3S74XAU5NJ2MUUXSFFIAK377YFGO646YQ, seg_pos={0 0}, upload=211101]: Done uploading piece. err: <nil> Aug 17 04:22:28.833066822 (202h56m23.264147013s) [noise=true , address=77.90.31.15:4732 , piece_id=4AU7UGKXN4TIPSIRR7IMXSF6WHELO6UTOX35N5X7JHOQQXWWJF2A, seg_pos={0 0}, upload=211101]: Done uploading piece. err: <nil> Aug 17 04:22:28.846684710 (202h56m23.277764831s) [noise=true , address=77.90.31.11:4562 , piece_id=PUL6TAVIIU7732IOASNH5EHQMNCK7DWHJMUJVXRNK74T4GUQDGTA, seg_pos={0 0}, upload=211101]: Done uploading piece. err: <nil> Aug 17 04:23:25.582810947 (202h57m20.013891078s) [noise=true , address=104.36.82.34:4558 , piece_id=XGFNEBBVZX5UWJN7DALGM2EDG26POCTSBBAUU7DLYQKS2DNSBNXA, seg_pos={0 0}, upload=211101]: Done uploading piece. err: ecclient: upload failed (node:12TU4zvu38mF1fNSergCtMj2cK9eu57Rdfpt5MVk3tc4MZF6dNr, address:104.36.82.34:4558): protocol: rpc: dial tcp 104.36.82.34:4558: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:25.582856475 (202h57m20.013936606s) [noise=true , address=104.36.82.35:4594 , piece_id=6NUSPNYGSSJOB2CNPIW6DI45MPZARLWDBXH3CWMP4LXMDB2CXWZQ, seg_pos={0 0}, upload=211101]: Done uploading piece. err: ecclient: upload failed (node:12D9nh4XFmsjjFcZEMjaL4uw4Whsa8nfMrAsYPu6NVpsxgb3jvf, address:104.36.82.35:4594): protocol: rpc: dial tcp 104.36.82.35:4594: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:25.582973238 (202h57m20.014053369s) [noise=true , address=104.36.82.19:4436 , piece_id=KVDJTKON4LFVJSAY7XSJ2OQ4ZH67JLMDE2O4EHUWSMZ3EECWSOXA, seg_pos={0 0}, upload=211101]: Done uploading piece. err: ecclient: upload failed (node:1WvyEw8XevCiQ8yRgzyMo7WTVzpEJ158ErKfAeV4dSTBkS1oRF, address:104.36.82.19:4436): protocol: rpc: dial tcp 104.36.82.19:4436: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:25.583033393 (202h57m20.014113524s) [noise=true , address=104.36.82.58:4283 , piece_id=LJKEH2KDSOFPVFPB5UWGVLOHQVZQOY33MYSHFLP4ZWGPKZ4KC64A, seg_pos={0 0}, upload=211101]: Done uploading piece. err: ecclient: upload failed (node:1p9jkPtQnmbCXBmw6HSHDWpGaKWqfg87m6hHXy6WDu2giAFnbQ, address:104.36.82.58:4283): protocol: rpc: dial tcp 104.36.82.58:4283: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:25.583069241 (202h57m20.014149362s) [noise=true , address=104.36.82.50:4047 , piece_id=AKDACFQEPBAFGYKTWBL5RELLZL3L4IJHJT73NS7T7QPWK2C2TDQA, seg_pos={0 0}, upload=211101]: Done uploading piece. err: ecclient: upload failed (node:126Z6tvp49dLLdj3T5snJc4tBVGzcX5UBaGafFN3PkhF92vxpfs, address:104.36.82.50:4047): protocol: rpc: dial tcp 104.36.82.50:4047: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:25.583076716 (202h57m20.014156837s) [noise=true , address=104.36.82.39:4753 , piece_id=R7BHT5RYLSCDZLTDDRXTSJFSRPLM5PNBYC6Q2G33C5SYIUCNVCLQ, seg_pos={0 0}, upload=211101]: Done uploading piece. err: ecclient: upload failed (node:1zzW2z6noGQT6xdSmwSQBNYwip6gZjnmqZg7qdb6BP3nHyCLFz, address:104.36.82.39:4753): protocol: rpc: dial tcp 104.36.82.39:4753: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:25.583107535 (202h57m20.014187666s) [noise=true , address=104.36.82.23:4474 , piece_id=RITPHITVW655UQETT6UKS7BZO2S3F4P2MJIO4REOUKYYP4E2CDVQ, seg_pos={0 0}, upload=211101]: Done uploading piece. err: ecclient: upload failed (node:1Fpshye6yGAo3yf71eVpxrCdvoRjpkGX9WEX8KCWYbvuhy5i9j, address:104.36.82.23:4474): protocol: write tcp 188.166.225.170:36642->104.36.82.23:4474: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:25.583165355 (202h57m20.014245487s) [noise=true , address=104.36.82.5:4855 , piece_id=3FCDYA6CRYOOU3MDFJPEC4YQNICFVCN2A47RIYPSR4ZX5TIXZX3A, seg_pos={0 0}, upload=211101]: Done uploading piece. err: ecclient: upload failed (node:1KDNuE7YY8smhDjotushQvY3y9NtLxigbJELHD4Edxy8Qi4BWh, address:104.36.82.5:4855): protocol: write tcp 188.166.225.170:47052->104.36.82.5:4855: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:25.583194000 (202h57m20.014274131s) [seg_pos={0 0}, upload=211101]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Aug 17 04:23:25.810738626 (202h57m20.241818757s) [upload=211101]: Done uploading segments... Aug 17 04:23:25.810761991 (202h57m20.241842122s) [upload=211101]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 6.025400415s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.....................XX.....................................] duration: 6.452382078s FAIL (2025-08-17 04:52:27 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.452404001s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [...............XXXX.........................................] duration: 6.27038628s FAIL (2025-08-17 04:52:19 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.270411478s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 10 out of 60 is failed (threshold=3) [.........X..........XXXX.X.X.......X...X.................X..] duration: 5.567038385s FAIL (2025-08-17 04:52:11 UTC) orbiter warning: operation: ListObjects ======ERROR====== unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/4M ======LOGS====== Aug 17 04:51:09.056938814 (203h25m3.488018935s) [upload=211594]: Starting upload Aug 17 04:51:09.057017635 (203h25m3.488097766s) [upload=211594]: Done starting upload Aug 17 04:51:09.057190195 (203h25m3.488270326s) [upload=211594]: Uploading segments... Aug 17 04:51:09.057308371 (203h25m3.488388502s) [upload=211594]: Got next segment. Inline: false Aug 17 04:51:09.057342866 (203h25m3.488422997s) [upload=211594]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 17 04:51:09.057347857 (203h25m3.488427978s) [upload=211594]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 17 04:51:09.504198039 (203h25m3.93527816s) [seg_pos={0 0}, upload=211594]: Begin upload segment... Aug 17 04:51:09.504246441 (203h25m3.935326572s) [seg_pos={0 0}, upload=211594]: Done begin upload segment. Aug 17 04:51:09.504264686 (203h25m3.935344817s) [noise=true , address=104.36.82.2:4748 , piece_id=T6MOSFTUKYGVT2YSCEGXMSDXLULB6NI4RGGGB3SSARO7ALNZJN5Q, seg_pos={0 0}, upload=211594]: Uploading piece... Aug 17 04:51:09.504308099 (203h25m3.93538823s) [noise=true , address=104.36.82.19:4436 , piece_id=VN6NIXXOT6KUQE3JU6X35TPYNCI5K4KOSBLVXMOKSP6N2SYRFM6Q, seg_pos={0 0}, upload=211594]: Uploading piece... Aug 17 04:51:09.504588896 (203h25m3.935669017s) [noise=true , address=104.36.82.58:4295 , piece_id=3LNVXIW5IU2BQZRH6QAVV7A2PJ3L3YSGNOXBMIYOJMFBVHJZUPYA, seg_pos={0 0}, upload=211594]: Uploading piece... Aug 17 04:51:09.504650284 (203h25m3.935730405s) [noise=true , address=64.49.13.169:4706 , piece_id=Y4SS3PCBRVCQS5N5N34Y6PGUJ5FPWOCHHO5KMU2PJ5CUXWTHW7RQ, seg_pos={0 0}, upload=211594]: Uploading piece... Aug 17 04:51:09.504844445 (203h25m3.935924566s) [noise=true , address=45.91.116.12:4228 , piece_id=X5VGNN2WVGEW3E4MPUO5MQWZ7L2JMB345IQUUTBI4DYW5HESK33A, seg_pos={0 0}, upload=211594]: Uploading piece... Aug 17 04:51:09.504957571 (203h25m3.936037702s) [noise=true , address=104.36.82.23:4475 , piece_id=TYHG2VUDQ4RRMFLUOLV3U73G7JXA6ZRC757TD34Q2MZTJNRMUIZA, seg_pos={0 0}, upload=211594]: Uploading piece... Aug 17 04:51:09.505108992 (203h25m3.936189123s) [noise=true , address=104.36.82.59:4331 , piece_id=SPNJSZRY7LNIS7ZCHJLOOWKVZU7AP3CODFM73CGOLQZU7HUCT2IQ, seg_pos={0 0}, upload=211594]: Uploading piece... Aug 17 04:51:09.505263607 (203h25m3.936343738s) [noise=true , address=66.23.193.175:4553 , piece_id=BXVNQMDXCQPEUSLIZ24VI22F7FOOJN2MEGJU6UI2SDQIZJSFQJ2A, seg_pos={0 0}, upload=211594]: Uploading piece... Aug 17 04:51:09.505341626 (203h25m3.936421757s) [noise=true , address=77.90.31.10:4523 , piece_id=22C56K4AOQCL5N6WR7QECVAAXHOS3OUNMGWCRFHP5GKSTHIT7OLA, seg_pos={0 0}, upload=211594]: Uploading piece... Aug 17 04:51:09.505565765 (203h25m3.936645896s) [noise=true , address=104.36.82.4:4836 , piece_id=JJWCMFYVQUU6WGEVBEJS4JPZQNNZERI5BVTBBAQQUZV3WD63NGIA, seg_pos={0 0}, upload=211594]: Uploading piece... Aug 17 04:51:09.505597326 (203h25m3.936677457s) [noise=true , address=77.90.31.11:4563 , piece_id=QDC72JPWTU4LH5URAU44ECPV72L4NQPWGDDOHHVYSGNJTFS7C6XQ, seg_pos={0 0}, upload=211594]: Uploading piece... Aug 17 04:51:09.505853546 (203h25m3.936933677s) [noise=true , address=104.36.82.18:4396 , piece_id=3CORBTMNXLB5AG6ZGAPLPBGGFEQ4SSMJHRGXPESWTJD5YPYVUKFA, seg_pos={0 0}, upload=211594]: Uploading piece... Aug 17 04:51:09.505894053 (203h25m3.936974184s) [noise=true , address=104.36.82.44:4896 , piece_id=2DQKKZFJX4N6W5UOOYX4QWSZ3GAFBMQIIDCKU3N3N5MLM4Q7T2AQ, seg_pos={0 0}, upload=211594]: Uploading piece... Aug 17 04:51:09.506090339 (203h25m3.937170471s) [noise=true , address=77.90.31.13:4658 , piece_id=XONXAMV3VAHHSHNIDMAIMMD6KOCKLYJQAUTNVMSUXB6ZCWP5THDA, seg_pos={0 0}, upload=211594]: Uploading piece... Aug 17 04:51:09.506165653 (203h25m3.937245784s) [noise=true , address=104.36.82.5:4850 , piece_id=6ZYI2JS5HMQKTUM2XR4ZTSJMC3MHVILV4YOQ7ANA5PX3FDJXJ2DA, seg_pos={0 0}, upload=211594]: Uploading piece... Aug 17 04:51:09.506275653 (203h25m3.937355774s) [noise=true , address=45.91.116.14:4199 , piece_id=HOTL5QUTJLBBZNF6A4WPRKMQXB72I3TUVSDYUVWMU7G4ZIXOIK3A, seg_pos={0 0}, upload=211594]: Uploading piece... Aug 17 04:51:09.506396365 (203h25m3.937476496s) [noise=true , address=104.36.82.34:4565 , piece_id=LUXKT7N4OUT4UJA345KTE2T3QUD3LEGLUQKQTDZADYBOGE2Z5XNQ, seg_pos={0 0}, upload=211594]: Uploading piece... Aug 17 04:51:09.506467330 (203h25m3.937547461s) [noise=true , address=104.36.82.60:4354 , piece_id=M3INHRZ2XHGLNFSI7OHKHKNCVTSOWTSCJJCGHTTYCAJGFLDWLL7A, seg_pos={0 0}, upload=211594]: Uploading piece... Aug 17 04:51:10.617749206 (203h25m5.048829337s) [upload=211594]: Next returned nil segment Aug 17 04:51:10.617765637 (203h25m5.048845758s) [upload=211594]: Waiting for error group managing segments... Aug 17 04:51:10.899081476 (203h25m5.330161607s) [noise=true , address=64.49.13.169:4706 , piece_id=Y4SS3PCBRVCQS5N5N34Y6PGUJ5FPWOCHHO5KMU2PJ5CUXWTHW7RQ, seg_pos={0 0}, upload=211594]: Done uploading piece. err: <nil> Aug 17 04:51:10.959773754 (203h25m5.390853875s) [noise=true , address=66.23.193.175:4553 , piece_id=BXVNQMDXCQPEUSLIZ24VI22F7FOOJN2MEGJU6UI2SDQIZJSFQJ2A, seg_pos={0 0}, upload=211594]: Done uploading piece. err: <nil> Aug 17 04:51:11.387426163 (203h25m5.818506294s) [noise=true , address=77.90.31.11:4563 , piece_id=QDC72JPWTU4LH5URAU44ECPV72L4NQPWGDDOHHVYSGNJTFS7C6XQ, seg_pos={0 0}, upload=211594]: Done uploading piece. err: <nil> Aug 17 04:51:11.392712709 (203h25m5.82379284s) [noise=true , address=45.91.116.14:4199 , piece_id=HOTL5QUTJLBBZNF6A4WPRKMQXB72I3TUVSDYUVWMU7G4ZIXOIK3A, seg_pos={0 0}, upload=211594]: Done uploading piece. err: <nil> Aug 17 04:51:11.399286516 (203h25m5.830366637s) [noise=true , address=104.36.82.4:4836 , piece_id=JJWCMFYVQUU6WGEVBEJS4JPZQNNZERI5BVTBBAQQUZV3WD63NGIA, seg_pos={0 0}, upload=211594]: Done uploading piece. err: <nil> Aug 17 04:51:11.412750859 (203h25m5.84383099s) [noise=true , address=104.36.82.23:4475 , piece_id=TYHG2VUDQ4RRMFLUOLV3U73G7JXA6ZRC757TD34Q2MZTJNRMUIZA, seg_pos={0 0}, upload=211594]: Done uploading piece. err: <nil> Aug 17 04:51:11.439625902 (203h25m5.870706033s) [noise=true , address=77.90.31.10:4523 , piece_id=22C56K4AOQCL5N6WR7QECVAAXHOS3OUNMGWCRFHP5GKSTHIT7OLA, seg_pos={0 0}, upload=211594]: Done uploading piece. err: <nil> Aug 17 04:51:11.465577498 (203h25m5.896657629s) [noise=true , address=104.36.82.34:4565 , piece_id=LUXKT7N4OUT4UJA345KTE2T3QUD3LEGLUQKQTDZADYBOGE2Z5XNQ, seg_pos={0 0}, upload=211594]: Done uploading piece. err: <nil> Aug 17 04:51:11.469847087 (203h25m5.900927218s) [noise=true , address=104.36.82.60:4354 , piece_id=M3INHRZ2XHGLNFSI7OHKHKNCVTSOWTSCJJCGHTTYCAJGFLDWLL7A, seg_pos={0 0}, upload=211594]: Done uploading piece. err: <nil> Aug 17 04:51:11.470697434 (203h25m5.901777555s) [noise=true , address=104.36.82.5:4850 , piece_id=6ZYI2JS5HMQKTUM2XR4ZTSJMC3MHVILV4YOQ7ANA5PX3FDJXJ2DA, seg_pos={0 0}, upload=211594]: Done uploading piece. err: <nil> Aug 17 04:51:11.470949846 (203h25m5.902029978s) [noise=true , address=104.36.82.19:4436 , piece_id=VN6NIXXOT6KUQE3JU6X35TPYNCI5K4KOSBLVXMOKSP6N2SYRFM6Q, seg_pos={0 0}, upload=211594]: Done uploading piece. err: <nil> Aug 17 04:51:11.486560086 (203h25m5.917640217s) [noise=true , address=104.36.82.2:4748 , piece_id=T6MOSFTUKYGVT2YSCEGXMSDXLULB6NI4RGGGB3SSARO7ALNZJN5Q, seg_pos={0 0}, upload=211594]: Done uploading piece. err: <nil> Aug 17 04:51:11.498376395 (203h25m5.929456526s) [noise=true , address=104.36.82.44:4896 , piece_id=2DQKKZFJX4N6W5UOOYX4QWSZ3GAFBMQIIDCKU3N3N5MLM4Q7T2AQ, seg_pos={0 0}, upload=211594]: Done uploading piece. err: <nil> Aug 17 04:51:11.512939881 (203h25m5.944020012s) [noise=true , address=45.91.116.12:4228 , piece_id=X5VGNN2WVGEW3E4MPUO5MQWZ7L2JMB345IQUUTBI4DYW5HESK33A, seg_pos={0 0}, upload=211594]: Done uploading piece. err: <nil> Aug 17 04:51:11.512969688 (203h25m5.944049819s) [seg_pos={0 0}, upload=211594]: Segment reached optimal threshold of 14 pieces. Aug 17 04:51:11.513156405 (203h25m5.944236536s) [noise=true , address=77.90.31.13:4658 , piece_id=XONXAMV3VAHHSHNIDMAIMMD6KOCKLYJQAUTNVMSUXB6ZCWP5THDA, seg_pos={0 0}, upload=211594]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129aongkQE9HMNz8vsNNA5SvKbhsHqdt9UrtSTPJJpP9mkFsboX): protocol: expected piece hash; context canceled; EOF Aug 17 04:51:11.513220778 (203h25m5.944300909s) [noise=true , address=104.36.82.58:4295 , piece_id=3LNVXIW5IU2BQZRH6QAVV7A2PJ3L3YSGNOXBMIYOJMFBVHJZUPYA, seg_pos={0 0}, upload=211594]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1oLa34qwPBAx3CTvYYj7GN3oxxWGJ9dAYXfke1eUftZc6xzfqg): protocol: expected piece hash; context canceled; EOF Aug 17 04:51:11.513269471 (203h25m5.944349603s) [noise=true , address=104.36.82.59:4331 , piece_id=SPNJSZRY7LNIS7ZCHJLOOWKVZU7AP3CODFM73CGOLQZU7HUCT2IQ, seg_pos={0 0}, upload=211594]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EYU9Z3iNdzeVtxWGQ8F4F7ZT6FbjaFzfr4QQpUDWUpFTrWUYU): protocol: expected piece hash; context canceled; EOF Aug 17 04:51:11.513316901 (203h25m5.944397033s) [noise=true , address=104.36.82.18:4396 , piece_id=3CORBTMNXLB5AG6ZGAPLPBGGFEQ4SSMJHRGXPESWTJD5YPYVUKFA, seg_pos={0 0}, upload=211594]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1k5gzJBGWhpARcfiynburgfTqJ2t7sVNs1mAkUfToMG5saxPqx): protocol: expected piece hash; context canceled; EOF Aug 17 04:51:11.513333754 (203h25m5.944413885s) [seg_pos={0 0}, upload=211594]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 17 04:51:11.513406654 (203h25m5.944486775s) [upload=211594]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 17 04:51:11.513424287 (203h25m5.944504418s) [upload=211594]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 17 04:51:11.809871930 (203h25m6.240952061s) [upload=211594]: Done uploading segments... Aug 17 04:51:11.809892119 (203h25m6.24097225s) [upload=211594]: Upload finished. err: <nil> Duration: 5.567072379s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [.................XXX........................................] duration: 5.535167864s FAIL (2025-08-17 04:52:04 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== g_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.761278730 (202h56m21.192358861s) [noise=true , address=104.36.82.34:4558 , piece_id=XGFNEBBVZX5UWJN7DALGM2EDG26POCTSBBAUU7DLYQKS2DNSBNXA, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.761661883 (202h56m21.192742014s) [noise=true , address=104.36.82.35:4594 , piece_id=6NUSPNYGSSJOB2CNPIW6DI45MPZARLWDBXH3CWMP4LXMDB2CXWZQ, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.761716548 (202h56m21.192796679s) [noise=true , address=45.91.116.8:4067 , piece_id=EZAI7ZDUBOOWXYJNTB7ZPOUSBUUG2FKV77QVUOZMNM6TUJSKGYWA, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.761972016 (202h56m21.193052147s) [noise=true , address=104.36.82.58:4283 , piece_id=LJKEH2KDSOFPVFPB5UWGVLOHQVZQOY33MYSHFLP4ZWGPKZ4KC64A, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.762254878 (202h56m21.193335009s) [noise=true , address=79.127.201.215:4345 , piece_id=JMSKNIZ7QPGBXIEP5RA3S74XAU5NJ2MUUXSFFIAK377YFGO646YQ, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.762523682 (202h56m21.193603803s) [noise=true , address=45.91.116.13:4181 , piece_id=7VN6RQGOTBHQO6HXM5IFTU32PWNVPH6BJZMQNETJEXOYPR2Y7OCQ, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.762864294 (202h56m21.193944415s) [noise=true , address=45.91.116.14:4203 , piece_id=5LXGQ4JZD2KLW3OMWY5QYRE4MKQNKKSAIMESODKDDT2W33XD45DQ, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.763179537 (202h56m21.194259668s) [noise=true , address=77.90.31.18:4815 , piece_id=G6PINAFFA6DZRRMKU2XK2LYFOES5WG7PTJ5L2ZPUQHMA4JNSRZRQ, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.763478499 (202h56m21.19455864s) [noise=true , address=77.90.31.11:4562 , piece_id=PUL6TAVIIU7732IOASNH5EHQMNCK7DWHJMUJVXRNK74T4GUQDGTA, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.763757683 (202h56m21.194837814s) [noise=true , address=104.36.82.5:4855 , piece_id=3FCDYA6CRYOOU3MDFJPEC4YQNICFVCN2A47RIYPSR4ZX5TIXZX3A, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.763988195 (202h56m21.195068326s) [noise=true , address=45.91.116.5:4308 , piece_id=F52M3VMHRQWR5LIEEPMVKVBFMOVG76UNNGQTFKSF6ZBD7VEOFWVA, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.764273320 (202h56m21.195353451s) [noise=true , address=66.23.193.175:4559 , piece_id=46X42PJIPA6DVC7BG7WPKMPMC44ZBUROEAB5C22M5INXKBJC4KVQ, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.764555100 (202h56m21.195635221s) [noise=true , address=77.90.31.15:4732 , piece_id=4AU7UGKXN4TIPSIRR7IMXSF6WHELO6UTOX35N5X7JHOQQXWWJF2A, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.764757416 (202h56m21.195837547s) [noise=true , address=104.36.82.39:4753 , piece_id=R7BHT5RYLSCDZLTDDRXTSJFSRPLM5PNBYC6Q2G33C5SYIUCNVCLQ, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.765868993 (202h56m21.196949124s) [noise=true , address=104.36.82.50:4047 , piece_id=AKDACFQEPBAFGYKTWBL5RELLZL3L4IJHJT73NS7T7QPWK2C2TDQA, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.767152809 (202h56m21.19823294s) [noise=true , address=104.36.82.19:4436 , piece_id=KVDJTKON4LFVJSAY7XSJ2OQ4ZH67JLMDE2O4EHUWSMZ3EECWSOXA, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.768577886 (202h56m21.199658017s) [noise=true , address=45.91.116.10:4095 , piece_id=JQ2UARQSNKBGHHHBJ6PR45ETR4EYMCXPDEQHEGWFYH52LRJC57UQ, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:27.934309011 (202h56m22.365389142s) [upload=211101]: Next returned nil segment Aug 17 04:22:27.934342545 (202h56m22.365422666s) [upload=211101]: Waiting for error group managing segments... Aug 17 04:22:28.231985083 (202h56m22.663065214s) [noise=true , address=66.23.193.175:4559 , piece_id=46X42PJIPA6DVC7BG7WPKMPMC44ZBUROEAB5C22M5INXKBJC4KVQ, seg_pos={0 0}, upload=211101]: Done uploading piece. err: <nil> Aug 17 04:22:28.670701256 (202h56m23.101781427s) [noise=true , address=45.91.116.10:4095 , piece_id=JQ2UARQSNKBGHHHBJ6PR45ETR4EYMCXPDEQHEGWFYH52LRJC57UQ, seg_pos={0 0}, upload=211101]: Done uploading piece. err: <nil> Aug 17 04:22:28.671952431 (202h56m23.103032562s) [noise=true , address=45.91.116.5:4308 , piece_id=F52M3VMHRQWR5LIEEPMVKVBFMOVG76UNNGQTFKSF6ZBD7VEOFWVA, seg_pos={0 0}, upload=211101]: Done uploading piece. err: <nil> Aug 17 04:22:28.698468278 (202h56m23.129548409s) [noise=true , address=45.91.116.8:4067 , piece_id=EZAI7ZDUBOOWXYJNTB7ZPOUSBUUG2FKV77QVUOZMNM6TUJSKGYWA, seg_pos={0 0}, upload=211101]: Done uploading piece. err: <nil> Aug 17 04:22:28.720171187 (202h56m23.151251318s) [noise=true , address=45.91.116.13:4181 , piece_id=7VN6RQGOTBHQO6HXM5IFTU32PWNVPH6BJZMQNETJEXOYPR2Y7OCQ, seg_pos={0 0}, upload=211101]: Done uploading piece. err: <nil> Aug 17 04:22:28.731591429 (202h56m23.16267156s) [noise=true , address=45.91.116.14:4203 , piece_id=5LXGQ4JZD2KLW3OMWY5QYRE4MKQNKKSAIMESODKDDT2W33XD45DQ, seg_pos={0 0}, upload=211101]: Done uploading piece. err: <nil> Aug 17 04:22:28.798148986 (202h56m23.229229117s) [noise=true , address=77.90.31.18:4815 , piece_id=G6PINAFFA6DZRRMKU2XK2LYFOES5WG7PTJ5L2ZPUQHMA4JNSRZRQ, seg_pos={0 0}, upload=211101]: Done uploading piece. err: <nil> Aug 17 04:22:28.804849037 (202h56m23.235929168s) [noise=true , address=79.127.201.215:4345 , piece_id=JMSKNIZ7QPGBXIEP5RA3S74XAU5NJ2MUUXSFFIAK377YFGO646YQ, seg_pos={0 0}, upload=211101]: Done uploading piece. err: <nil> Aug 17 04:22:28.833066822 (202h56m23.264147013s) [noise=true , address=77.90.31.15:4732 , piece_id=4AU7UGKXN4TIPSIRR7IMXSF6WHELO6UTOX35N5X7JHOQQXWWJF2A, seg_pos={0 0}, upload=211101]: Done uploading piece. err: <nil> Aug 17 04:22:28.846684710 (202h56m23.277764831s) [noise=true , address=77.90.31.11:4562 , piece_id=PUL6TAVIIU7732IOASNH5EHQMNCK7DWHJMUJVXRNK74T4GUQDGTA, seg_pos={0 0}, upload=211101]: Done uploading piece. err: <nil> Aug 17 04:23:25.582810947 (202h57m20.013891078s) [noise=true , address=104.36.82.34:4558 , piece_id=XGFNEBBVZX5UWJN7DALGM2EDG26POCTSBBAUU7DLYQKS2DNSBNXA, seg_pos={0 0}, upload=211101]: Done uploading piece. err: ecclient: upload failed (node:12TU4zvu38mF1fNSergCtMj2cK9eu57Rdfpt5MVk3tc4MZF6dNr, address:104.36.82.34:4558): protocol: rpc: dial tcp 104.36.82.34:4558: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:25.582856475 (202h57m20.013936606s) [noise=true , address=104.36.82.35:4594 , piece_id=6NUSPNYGSSJOB2CNPIW6DI45MPZARLWDBXH3CWMP4LXMDB2CXWZQ, seg_pos={0 0}, upload=211101]: Done uploading piece. err: ecclient: upload failed (node:12D9nh4XFmsjjFcZEMjaL4uw4Whsa8nfMrAsYPu6NVpsxgb3jvf, address:104.36.82.35:4594): protocol: rpc: dial tcp 104.36.82.35:4594: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:25.582973238 (202h57m20.014053369s) [noise=true , address=104.36.82.19:4436 , piece_id=KVDJTKON4LFVJSAY7XSJ2OQ4ZH67JLMDE2O4EHUWSMZ3EECWSOXA, seg_pos={0 0}, upload=211101]: Done uploading piece. err: ecclient: upload failed (node:1WvyEw8XevCiQ8yRgzyMo7WTVzpEJ158ErKfAeV4dSTBkS1oRF, address:104.36.82.19:4436): protocol: rpc: dial tcp 104.36.82.19:4436: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:25.583033393 (202h57m20.014113524s) [noise=true , address=104.36.82.58:4283 , piece_id=LJKEH2KDSOFPVFPB5UWGVLOHQVZQOY33MYSHFLP4ZWGPKZ4KC64A, seg_pos={0 0}, upload=211101]: Done uploading piece. err: ecclient: upload failed (node:1p9jkPtQnmbCXBmw6HSHDWpGaKWqfg87m6hHXy6WDu2giAFnbQ, address:104.36.82.58:4283): protocol: rpc: dial tcp 104.36.82.58:4283: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:25.583069241 (202h57m20.014149362s) [noise=true , address=104.36.82.50:4047 , piece_id=AKDACFQEPBAFGYKTWBL5RELLZL3L4IJHJT73NS7T7QPWK2C2TDQA, seg_pos={0 0}, upload=211101]: Done uploading piece. err: ecclient: upload failed (node:126Z6tvp49dLLdj3T5snJc4tBVGzcX5UBaGafFN3PkhF92vxpfs, address:104.36.82.50:4047): protocol: rpc: dial tcp 104.36.82.50:4047: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:25.583076716 (202h57m20.014156837s) [noise=true , address=104.36.82.39:4753 , piece_id=R7BHT5RYLSCDZLTDDRXTSJFSRPLM5PNBYC6Q2G33C5SYIUCNVCLQ, seg_pos={0 0}, upload=211101]: Done uploading piece. err: ecclient: upload failed (node:1zzW2z6noGQT6xdSmwSQBNYwip6gZjnmqZg7qdb6BP3nHyCLFz, address:104.36.82.39:4753): protocol: rpc: dial tcp 104.36.82.39:4753: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:25.583107535 (202h57m20.014187666s) [noise=true , address=104.36.82.23:4474 , piece_id=RITPHITVW655UQETT6UKS7BZO2S3F4P2MJIO4REOUKYYP4E2CDVQ, seg_pos={0 0}, upload=211101]: Done uploading piece. err: ecclient: upload failed (node:1Fpshye6yGAo3yf71eVpxrCdvoRjpkGX9WEX8KCWYbvuhy5i9j, address:104.36.82.23:4474): protocol: write tcp 188.166.225.170:36642->104.36.82.23:4474: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:25.583165355 (202h57m20.014245487s) [noise=true , address=104.36.82.5:4855 , piece_id=3FCDYA6CRYOOU3MDFJPEC4YQNICFVCN2A47RIYPSR4ZX5TIXZX3A, seg_pos={0 0}, upload=211101]: Done uploading piece. err: ecclient: upload failed (node:1KDNuE7YY8smhDjotushQvY3y9NtLxigbJELHD4Edxy8Qi4BWh, address:104.36.82.5:4855): protocol: write tcp 188.166.225.170:47052->104.36.82.5:4855: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:25.583194000 (202h57m20.014274131s) [seg_pos={0 0}, upload=211101]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Aug 17 04:23:25.810738626 (202h57m20.241818757s) [upload=211101]: Done uploading segments... Aug 17 04:23:25.810761991 (202h57m20.241842122s) [upload=211101]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 5.535213812s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [.....................................X.....X................] duration: 2.107601449s FAIL (2025-08-17 04:52:01 UTC) orbiter warning: operation: ListObjects ======ERROR====== unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/1K ======LOGS====== Aug 17 04:41:26.040732864 (203h15m20.471812985s) [upload=211420]: Starting upload Aug 17 04:41:26.040780384 (203h15m20.471860515s) [upload=211420]: Done starting upload Aug 17 04:41:26.040828978 (203h15m20.471909109s) [upload=211420]: Uploading segments... Aug 17 04:41:26.040863473 (203h15m20.471943604s) [upload=211420]: Got next segment. Inline: true Aug 17 04:41:26.040867290 (203h15m20.471947422s) [upload=211420]: Waiting for error group managing segments... Aug 17 04:41:26.040897880 (203h15m20.471978011s) [upload=211420]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 17 04:41:26.040901597 (203h15m20.471981728s) [upload=211420]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 17 04:41:26.040904041 (203h15m20.471984162s) [upload=211420]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 17 04:41:26.462140724 (203h15m20.893220855s) [upload=211420]: Done uploading segments... Aug 17 04:41:26.462159310 (203h15m20.893239431s) [upload=211420]: Upload finished. err: <nil> Duration:* 2.107637798s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [......................XX....................................] duration: 6.689135421s FAIL (2025-08-17 04:51:55 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.689156462s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [................XXXX........................................] duration: 6.176503115s FAIL (2025-08-17 04:51:53 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.176531109s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 10 out of 60 is failed (threshold=3) [..........X..........XXXX.X.X.......X...X.................X.] duration: 5.660855373s FAIL (2025-08-17 04:51:43 UTC) orbiter warning: operation: ListObjects ======ERROR====== unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/4M ======LOGS====== Aug 17 04:51:09.056938814 (203h25m3.488018935s) [upload=211594]: Starting upload Aug 17 04:51:09.057017635 (203h25m3.488097766s) [upload=211594]: Done starting upload Aug 17 04:51:09.057190195 (203h25m3.488270326s) [upload=211594]: Uploading segments... Aug 17 04:51:09.057308371 (203h25m3.488388502s) [upload=211594]: Got next segment. Inline: false Aug 17 04:51:09.057342866 (203h25m3.488422997s) [upload=211594]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 17 04:51:09.057347857 (203h25m3.488427978s) [upload=211594]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 17 04:51:09.504198039 (203h25m3.93527816s) [seg_pos={0 0}, upload=211594]: Begin upload segment... Aug 17 04:51:09.504246441 (203h25m3.935326572s) [seg_pos={0 0}, upload=211594]: Done begin upload segment. Aug 17 04:51:09.504264686 (203h25m3.935344817s) [noise=true , address=104.36.82.2:4748 , piece_id=T6MOSFTUKYGVT2YSCEGXMSDXLULB6NI4RGGGB3SSARO7ALNZJN5Q, seg_pos={0 0}, upload=211594]: Uploading piece... Aug 17 04:51:09.504308099 (203h25m3.93538823s) [noise=true , address=104.36.82.19:4436 , piece_id=VN6NIXXOT6KUQE3JU6X35TPYNCI5K4KOSBLVXMOKSP6N2SYRFM6Q, seg_pos={0 0}, upload=211594]: Uploading piece... Aug 17 04:51:09.504588896 (203h25m3.935669017s) [noise=true , address=104.36.82.58:4295 , piece_id=3LNVXIW5IU2BQZRH6QAVV7A2PJ3L3YSGNOXBMIYOJMFBVHJZUPYA, seg_pos={0 0}, upload=211594]: Uploading piece... Aug 17 04:51:09.504650284 (203h25m3.935730405s) [noise=true , address=64.49.13.169:4706 , piece_id=Y4SS3PCBRVCQS5N5N34Y6PGUJ5FPWOCHHO5KMU2PJ5CUXWTHW7RQ, seg_pos={0 0}, upload=211594]: Uploading piece... Aug 17 04:51:09.504844445 (203h25m3.935924566s) [noise=true , address=45.91.116.12:4228 , piece_id=X5VGNN2WVGEW3E4MPUO5MQWZ7L2JMB345IQUUTBI4DYW5HESK33A, seg_pos={0 0}, upload=211594]: Uploading piece... Aug 17 04:51:09.504957571 (203h25m3.936037702s) [noise=true , address=104.36.82.23:4475 , piece_id=TYHG2VUDQ4RRMFLUOLV3U73G7JXA6ZRC757TD34Q2MZTJNRMUIZA, seg_pos={0 0}, upload=211594]: Uploading piece... Aug 17 04:51:09.505108992 (203h25m3.936189123s) [noise=true , address=104.36.82.59:4331 , piece_id=SPNJSZRY7LNIS7ZCHJLOOWKVZU7AP3CODFM73CGOLQZU7HUCT2IQ, seg_pos={0 0}, upload=211594]: Uploading piece... Aug 17 04:51:09.505263607 (203h25m3.936343738s) [noise=true , address=66.23.193.175:4553 , piece_id=BXVNQMDXCQPEUSLIZ24VI22F7FOOJN2MEGJU6UI2SDQIZJSFQJ2A, seg_pos={0 0}, upload=211594]: Uploading piece... Aug 17 04:51:09.505341626 (203h25m3.936421757s) [noise=true , address=77.90.31.10:4523 , piece_id=22C56K4AOQCL5N6WR7QECVAAXHOS3OUNMGWCRFHP5GKSTHIT7OLA, seg_pos={0 0}, upload=211594]: Uploading piece... Aug 17 04:51:09.505565765 (203h25m3.936645896s) [noise=true , address=104.36.82.4:4836 , piece_id=JJWCMFYVQUU6WGEVBEJS4JPZQNNZERI5BVTBBAQQUZV3WD63NGIA, seg_pos={0 0}, upload=211594]: Uploading piece... Aug 17 04:51:09.505597326 (203h25m3.936677457s) [noise=true , address=77.90.31.11:4563 , piece_id=QDC72JPWTU4LH5URAU44ECPV72L4NQPWGDDOHHVYSGNJTFS7C6XQ, seg_pos={0 0}, upload=211594]: Uploading piece... Aug 17 04:51:09.505853546 (203h25m3.936933677s) [noise=true , address=104.36.82.18:4396 , piece_id=3CORBTMNXLB5AG6ZGAPLPBGGFEQ4SSMJHRGXPESWTJD5YPYVUKFA, seg_pos={0 0}, upload=211594]: Uploading piece... Aug 17 04:51:09.505894053 (203h25m3.936974184s) [noise=true , address=104.36.82.44:4896 , piece_id=2DQKKZFJX4N6W5UOOYX4QWSZ3GAFBMQIIDCKU3N3N5MLM4Q7T2AQ, seg_pos={0 0}, upload=211594]: Uploading piece... Aug 17 04:51:09.506090339 (203h25m3.937170471s) [noise=true , address=77.90.31.13:4658 , piece_id=XONXAMV3VAHHSHNIDMAIMMD6KOCKLYJQAUTNVMSUXB6ZCWP5THDA, seg_pos={0 0}, upload=211594]: Uploading piece... Aug 17 04:51:09.506165653 (203h25m3.937245784s) [noise=true , address=104.36.82.5:4850 , piece_id=6ZYI2JS5HMQKTUM2XR4ZTSJMC3MHVILV4YOQ7ANA5PX3FDJXJ2DA, seg_pos={0 0}, upload=211594]: Uploading piece... Aug 17 04:51:09.506275653 (203h25m3.937355774s) [noise=true , address=45.91.116.14:4199 , piece_id=HOTL5QUTJLBBZNF6A4WPRKMQXB72I3TUVSDYUVWMU7G4ZIXOIK3A, seg_pos={0 0}, upload=211594]: Uploading piece... Aug 17 04:51:09.506396365 (203h25m3.937476496s) [noise=true , address=104.36.82.34:4565 , piece_id=LUXKT7N4OUT4UJA345KTE2T3QUD3LEGLUQKQTDZADYBOGE2Z5XNQ, seg_pos={0 0}, upload=211594]: Uploading piece... Aug 17 04:51:09.506467330 (203h25m3.937547461s) [noise=true , address=104.36.82.60:4354 , piece_id=M3INHRZ2XHGLNFSI7OHKHKNCVTSOWTSCJJCGHTTYCAJGFLDWLL7A, seg_pos={0 0}, upload=211594]: Uploading piece... Aug 17 04:51:10.617749206 (203h25m5.048829337s) [upload=211594]: Next returned nil segment Aug 17 04:51:10.617765637 (203h25m5.048845758s) [upload=211594]: Waiting for error group managing segments... Aug 17 04:51:10.899081476 (203h25m5.330161607s) [noise=true , address=64.49.13.169:4706 , piece_id=Y4SS3PCBRVCQS5N5N34Y6PGUJ5FPWOCHHO5KMU2PJ5CUXWTHW7RQ, seg_pos={0 0}, upload=211594]: Done uploading piece. err: <nil> Aug 17 04:51:10.959773754 (203h25m5.390853875s) [noise=true , address=66.23.193.175:4553 , piece_id=BXVNQMDXCQPEUSLIZ24VI22F7FOOJN2MEGJU6UI2SDQIZJSFQJ2A, seg_pos={0 0}, upload=211594]: Done uploading piece. err: <nil> Aug 17 04:51:11.387426163 (203h25m5.818506294s) [noise=true , address=77.90.31.11:4563 , piece_id=QDC72JPWTU4LH5URAU44ECPV72L4NQPWGDDOHHVYSGNJTFS7C6XQ, seg_pos={0 0}, upload=211594]: Done uploading piece. err: <nil> Aug 17 04:51:11.392712709 (203h25m5.82379284s) [noise=true , address=45.91.116.14:4199 , piece_id=HOTL5QUTJLBBZNF6A4WPRKMQXB72I3TUVSDYUVWMU7G4ZIXOIK3A, seg_pos={0 0}, upload=211594]: Done uploading piece. err: <nil> Aug 17 04:51:11.399286516 (203h25m5.830366637s) [noise=true , address=104.36.82.4:4836 , piece_id=JJWCMFYVQUU6WGEVBEJS4JPZQNNZERI5BVTBBAQQUZV3WD63NGIA, seg_pos={0 0}, upload=211594]: Done uploading piece. err: <nil> Aug 17 04:51:11.412750859 (203h25m5.84383099s) [noise=true , address=104.36.82.23:4475 , piece_id=TYHG2VUDQ4RRMFLUOLV3U73G7JXA6ZRC757TD34Q2MZTJNRMUIZA, seg_pos={0 0}, upload=211594]: Done uploading piece. err: <nil> Aug 17 04:51:11.439625902 (203h25m5.870706033s) [noise=true , address=77.90.31.10:4523 , piece_id=22C56K4AOQCL5N6WR7QECVAAXHOS3OUNMGWCRFHP5GKSTHIT7OLA, seg_pos={0 0}, upload=211594]: Done uploading piece. err: <nil> Aug 17 04:51:11.465577498 (203h25m5.896657629s) [noise=true , address=104.36.82.34:4565 , piece_id=LUXKT7N4OUT4UJA345KTE2T3QUD3LEGLUQKQTDZADYBOGE2Z5XNQ, seg_pos={0 0}, upload=211594]: Done uploading piece. err: <nil> Aug 17 04:51:11.469847087 (203h25m5.900927218s) [noise=true , address=104.36.82.60:4354 , piece_id=M3INHRZ2XHGLNFSI7OHKHKNCVTSOWTSCJJCGHTTYCAJGFLDWLL7A, seg_pos={0 0}, upload=211594]: Done uploading piece. err: <nil> Aug 17 04:51:11.470697434 (203h25m5.901777555s) [noise=true , address=104.36.82.5:4850 , piece_id=6ZYI2JS5HMQKTUM2XR4ZTSJMC3MHVILV4YOQ7ANA5PX3FDJXJ2DA, seg_pos={0 0}, upload=211594]: Done uploading piece. err: <nil> Aug 17 04:51:11.470949846 (203h25m5.902029978s) [noise=true , address=104.36.82.19:4436 , piece_id=VN6NIXXOT6KUQE3JU6X35TPYNCI5K4KOSBLVXMOKSP6N2SYRFM6Q, seg_pos={0 0}, upload=211594]: Done uploading piece. err: <nil> Aug 17 04:51:11.486560086 (203h25m5.917640217s) [noise=true , address=104.36.82.2:4748 , piece_id=T6MOSFTUKYGVT2YSCEGXMSDXLULB6NI4RGGGB3SSARO7ALNZJN5Q, seg_pos={0 0}, upload=211594]: Done uploading piece. err: <nil> Aug 17 04:51:11.498376395 (203h25m5.929456526s) [noise=true , address=104.36.82.44:4896 , piece_id=2DQKKZFJX4N6W5UOOYX4QWSZ3GAFBMQIIDCKU3N3N5MLM4Q7T2AQ, seg_pos={0 0}, upload=211594]: Done uploading piece. err: <nil> Aug 17 04:51:11.512939881 (203h25m5.944020012s) [noise=true , address=45.91.116.12:4228 , piece_id=X5VGNN2WVGEW3E4MPUO5MQWZ7L2JMB345IQUUTBI4DYW5HESK33A, seg_pos={0 0}, upload=211594]: Done uploading piece. err: <nil> Aug 17 04:51:11.512969688 (203h25m5.944049819s) [seg_pos={0 0}, upload=211594]: Segment reached optimal threshold of 14 pieces. Aug 17 04:51:11.513156405 (203h25m5.944236536s) [noise=true , address=77.90.31.13:4658 , piece_id=XONXAMV3VAHHSHNIDMAIMMD6KOCKLYJQAUTNVMSUXB6ZCWP5THDA, seg_pos={0 0}, upload=211594]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129aongkQE9HMNz8vsNNA5SvKbhsHqdt9UrtSTPJJpP9mkFsboX): protocol: expected piece hash; context canceled; EOF Aug 17 04:51:11.513220778 (203h25m5.944300909s) [noise=true , address=104.36.82.58:4295 , piece_id=3LNVXIW5IU2BQZRH6QAVV7A2PJ3L3YSGNOXBMIYOJMFBVHJZUPYA, seg_pos={0 0}, upload=211594]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1oLa34qwPBAx3CTvYYj7GN3oxxWGJ9dAYXfke1eUftZc6xzfqg): protocol: expected piece hash; context canceled; EOF Aug 17 04:51:11.513269471 (203h25m5.944349603s) [noise=true , address=104.36.82.59:4331 , piece_id=SPNJSZRY7LNIS7ZCHJLOOWKVZU7AP3CODFM73CGOLQZU7HUCT2IQ, seg_pos={0 0}, upload=211594]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EYU9Z3iNdzeVtxWGQ8F4F7ZT6FbjaFzfr4QQpUDWUpFTrWUYU): protocol: expected piece hash; context canceled; EOF Aug 17 04:51:11.513316901 (203h25m5.944397033s) [noise=true , address=104.36.82.18:4396 , piece_id=3CORBTMNXLB5AG6ZGAPLPBGGFEQ4SSMJHRGXPESWTJD5YPYVUKFA, seg_pos={0 0}, upload=211594]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1k5gzJBGWhpARcfiynburgfTqJ2t7sVNs1mAkUfToMG5saxPqx): protocol: expected piece hash; context canceled; EOF Aug 17 04:51:11.513333754 (203h25m5.944413885s) [seg_pos={0 0}, upload=211594]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 17 04:51:11.513406654 (203h25m5.944486775s) [upload=211594]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 17 04:51:11.513424287 (203h25m5.944504418s) [upload=211594]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 17 04:51:11.809871930 (203h25m6.240952061s) [upload=211594]: Done uploading segments... Aug 17 04:51:11.809892119 (203h25m6.24097225s) [upload=211594]: Upload finished. err: <nil> Duration: 5.660876483s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [......................................X.....X...............] duration: 1.82687158s FAIL (2025-08-17 04:51:43 UTC) orbiter warning: operation: ListObjects ======ERROR====== unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/1K ======LOGS====== Aug 17 04:41:26.040732864 (203h15m20.471812985s) [upload=211420]: Starting upload Aug 17 04:41:26.040780384 (203h15m20.471860515s) [upload=211420]: Done starting upload Aug 17 04:41:26.040828978 (203h15m20.471909109s) [upload=211420]: Uploading segments... Aug 17 04:41:26.040863473 (203h15m20.471943604s) [upload=211420]: Got next segment. Inline: true Aug 17 04:41:26.040867290 (203h15m20.471947422s) [upload=211420]: Waiting for error group managing segments... Aug 17 04:41:26.040897880 (203h15m20.471978011s) [upload=211420]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 17 04:41:26.040901597 (203h15m20.471981728s) [upload=211420]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 17 04:41:26.040904041 (203h15m20.471984162s) [upload=211420]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 17 04:41:26.462140724 (203h15m20.893220855s) [upload=211420]: Done uploading segments... Aug 17 04:41:26.462159310 (203h15m20.893239431s) [upload=211420]: Upload finished. err: <nil> Duration:* 1.82689769s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [................XXXX........................................] duration: 5.684215709s FAIL (2025-08-17 04:51:29 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== Aug 17 04:23:03.873697274 (202h56m58.304777405s) [upload=211112]: Starting upload Aug 17 04:23:03.873756368 (202h56m58.304836499s) [upload=211112]: Done starting upload Aug 17 04:23:03.873839848 (202h56m58.304919989s) [upload=211112]: Uploading segments... Aug 17 04:23:03.873950850 (202h56m58.305030981s) [upload=211112]: Got next segment. Inline: false Aug 17 04:23:03.873976509 (202h56m58.30505664s) [upload=211112]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 17 04:23:03.873980757 (202h56m58.305060888s) [upload=211112]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 17 04:23:04.113727713 (202h56m58.544807844s) [seg_pos={0 0}, upload=211112]: Begin upload segment... Aug 17 04:23:04.113774051 (202h56m58.544854182s) [seg_pos={0 0}, upload=211112]: Done begin upload segment. Aug 17 04:23:04.113820890 (202h56m58.544901021s) [noise=true , address=45.91.116.13:4155 , piece_id=CO3R5XZGH3NJADAFDILPNZEKKFJSRPB4JOFBBCSYYQAD3KNRABNA, seg_pos={0 0}, upload=211112]: Uploading piece... Aug 17 04:23:04.113845989 (202h56m58.54492612s) [noise=true , address=45.91.116.7:4009 , piece_id=IVTDHCTNJEEO3QATGN25YH5Y2MSEQUWJIFPXSJWINUEXDHQAGELA, seg_pos={0 0}, upload=211112]: Uploading piece... Aug 17 04:23:04.114195037 (202h56m58.545275168s) [noise=true , address=45.91.116.4:4938 , piece_id=X5MDNBZYRYYOXDEAMGIOD5CXX7PGAMTFRMROOVTR6NIAYQ5VMXGA, seg_pos={0 0}, upload=211112]: Uploading piece... Aug 17 04:23:04.114281412 (202h56m58.545361543s) [noise=true , address=45.91.116.9:4261 , piece_id=PNOEBFIPLWM4B2QLGJUVEBTT23J5AVRFTWTTKLHFUJ5MNJTSM2JA, seg_pos={0 0}, upload=211112]: Uploading piece... Aug 17 04:23:04.114482115 (202h56m58.545562237s) [noise=true , address=77.90.31.16:4735 , piece_id=G5YGF7LXNPBU2TDRYPAFPVYQSTWUGG7Y6YM2Q6RWMCZY5RVNM4BA, seg_pos={0 0}, upload=211112]: Uploading piece... Aug 17 04:23:04.114570385 (202h56m58.545650516s) [noise=true , address=104.36.82.9:4005 , piece_id=FAR4B5KAALBOACKANEFKGECI2ZVXSHBNYDEIWVGDU2RNBNQXLMXA, seg_pos={0 0}, upload=211112]: Uploading piece... Aug 17 04:23:04.114757392 (202h56m58.545837523s) [noise=true , address=45.91.116.9:4263 , piece_id=VT734KJZ7T3CZ3W3P4M4ZCYKOLLTPP3ILFRWL26Z7K4LWBDZV4FA, seg_pos={0 0}, upload=211112]: Uploading piece... Aug 17 04:23:04.114874727 (202h56m58.545954858s) [noise=true , address=104.36.82.12:4151 , piece_id=2GGYMOWZ2FTWDNWNQRKSAH7UVJ4R6MLRGC6OHNVV3KTVIIMO4QIQ, seg_pos={0 0}, upload=211112]: Uploading piece... Aug 17 04:23:04.115030715 (202h56m58.546110846s) [noise=true , address=45.91.116.12:4225 , piece_id=EUPPM5NRIU5MCZBER3VGZBZEMZZ65Q2MOY2DXKFBIPFJ53M5TJHQ, seg_pos={0 0}, upload=211112]: Uploading piece... Aug 17 04:23:04.115194288 (202h56m58.546274419s) [noise=true , address=104.36.82.4:4804 , piece_id=MSAHN2DERVG2HL7KZMKIWYUQYO6LRIXGUNWJFXJDDGOWZT54FPVA, seg_pos={0 0}, upload=211112]: Uploading piece... Aug 17 04:23:04.115307967 (202h56m58.546388098s) [noise=true , address=104.36.82.5:4838 , piece_id=IYA7ERDAAFYUGC4666CBEX5HZEXZVFPES6JUYRZROY25OAPGVEQQ, seg_pos={0 0}, upload=211112]: Uploading piece... Aug 17 04:23:04.115535692 (202h56m58.546615823s) [noise=true , address=45.91.116.12:4227 , piece_id=XZR3AWJG57RUB4TN7SUWNIWWD2XGKCMDXSWSM74WS3ATHS32EBJQ, seg_pos={0 0}, upload=211112]: Uploading piece... Aug 17 04:23:04.115754130 (202h56m58.546834261s) [noise=true , address=45.91.116.7:4010 , piece_id=HNJU4MPWT67V26J5XDCTHFEOOJL2HS3VAOHB4EIPQVF4D2NNRFMA, seg_pos={0 0}, upload=211112]: Uploading piece... Aug 17 04:23:04.116079182 (202h56m58.547159323s) [noise=true , address=45.91.116.14:4189 , piece_id=2P3GOZDEMJGBAK2F6SY7NJNQUTS5ZYM4B4IMQFAA5TYFARD4DVCQ, seg_pos={0 0}, upload=211112]: Uploading piece... Aug 17 04:23:04.116350181 (202h56m58.547430312s) [noise=true , address=104.36.82.15:4257 , piece_id=Q5GLP6TS7N7CM4DO3LWDCDR6O76OFZ6IVQJC5VQWMA5XLNWZ2E4A, seg_pos={0 0}, upload=211112]: Uploading piece... Aug 17 04:23:04.116554471 (202h56m58.547634602s) [noise=true , address=104.36.82.8:4969 , piece_id=3D3YAZSP7CD6IFZGDKPNKIJPWZ2OJFFVJCSMTO42VNJEFVU3ORHA, seg_pos={0 0}, upload=211112]: Uploading piece... Aug 17 04:23:05.868765063 (202h57m0.299845195s) [upload=211112]: Next returned nil segment Aug 17 04:23:05.868801043 (202h57m0.299881164s) [upload=211112]: Waiting for error group managing segments... Aug 17 04:23:06.147430529 (202h57m0.57851066s) [noise=true , address=45.91.116.12:4227 , piece_id=XZR3AWJG57RUB4TN7SUWNIWWD2XGKCMDXSWSM74WS3ATHS32EBJQ, seg_pos={0 0}, upload=211112]: Done uploading piece. err: <nil> Aug 17 04:23:06.182400314 (202h57m0.613480445s) [noise=true , address=45.91.116.12:4225 , piece_id=EUPPM5NRIU5MCZBER3VGZBZEMZZ65Q2MOY2DXKFBIPFJ53M5TJHQ, seg_pos={0 0}, upload=211112]: Done uploading piece. err: <nil> Aug 17 04:23:06.191910532 (202h57m0.622990663s) [noise=true , address=45.91.116.4:4938 , piece_id=X5MDNBZYRYYOXDEAMGIOD5CXX7PGAMTFRMROOVTR6NIAYQ5VMXGA, seg_pos={0 0}, upload=211112]: Done uploading piece. err: <nil> Aug 17 04:23:06.197391631 (202h57m0.628471782s) [noise=true , address=45.91.116.9:4261 , piece_id=PNOEBFIPLWM4B2QLGJUVEBTT23J5AVRFTWTTKLHFUJ5MNJTSM2JA, seg_pos={0 0}, upload=211112]: Done uploading piece. err: <nil> Aug 17 04:23:06.198069487 (202h57m0.629149618s) [noise=true , address=45.91.116.9:4263 , piece_id=VT734KJZ7T3CZ3W3P4M4ZCYKOLLTPP3ILFRWL26Z7K4LWBDZV4FA, seg_pos={0 0}, upload=211112]: Done uploading piece. err: <nil> Aug 17 04:23:06.200597014 (202h57m0.631677145s) [noise=true , address=45.91.116.7:4009 , piece_id=IVTDHCTNJEEO3QATGN25YH5Y2MSEQUWJIFPXSJWINUEXDHQAGELA, seg_pos={0 0}, upload=211112]: Done uploading piece. err: <nil> Aug 17 04:23:06.288053251 (202h57m0.719133372s) [noise=true , address=45.91.116.7:4010 , piece_id=HNJU4MPWT67V26J5XDCTHFEOOJL2HS3VAOHB4EIPQVF4D2NNRFMA, seg_pos={0 0}, upload=211112]: Done uploading piece. err: <nil> Aug 17 04:23:06.298810435 (202h57m0.729890566s) [noise=true , address=45.91.116.14:4189 , piece_id=2P3GOZDEMJGBAK2F6SY7NJNQUTS5ZYM4B4IMQFAA5TYFARD4DVCQ, seg_pos={0 0}, upload=211112]: Done uploading piece. err: <nil> Aug 17 04:23:06.301049520 (202h57m0.732129651s) [noise=true , address=45.91.116.13:4155 , piece_id=CO3R5XZGH3NJADAFDILPNZEKKFJSRPB4JOFBBCSYYQAD3KNRABNA, seg_pos={0 0}, upload=211112]: Done uploading piece. err: <nil> Aug 17 04:23:06.402776985 (202h57m0.833857116s) [noise=true , address=77.90.31.16:4735 , piece_id=G5YGF7LXNPBU2TDRYPAFPVYQSTWUGG7Y6YM2Q6RWMCZY5RVNM4BA, seg_pos={0 0}, upload=211112]: Done uploading piece. err: <nil> Aug 17 04:23:33.223003412 (202h57m27.654083543s) [noise=true , address=104.36.82.5:4838 , piece_id=IYA7ERDAAFYUGC4666CBEX5HZEXZVFPES6JUYRZROY25OAPGVEQQ, seg_pos={0 0}, upload=211112]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:104.36.82.5:4838): protocol: write tcp 188.166.225.170:42258->104.36.82.5:4838: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:33.223050702 (202h57m27.654130823s) [noise=true , address=104.36.82.8:4969 , piece_id=3D3YAZSP7CD6IFZGDKPNKIJPWZ2OJFFVJCSMTO42VNJEFVU3ORHA, seg_pos={0 0}, upload=211112]: Done uploading piece. err: ecclient: upload failed (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, address:104.36.82.8:4969): protocol: write tcp 188.166.225.170:52308->104.36.82.8:4969: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:33.223090619 (202h57m27.65417075s) [noise=true , address=104.36.82.15:4257 , piece_id=Q5GLP6TS7N7CM4DO3LWDCDR6O76OFZ6IVQJC5VQWMA5XLNWZ2E4A, seg_pos={0 0}, upload=211112]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:104.36.82.15:4257): protocol: write tcp 188.166.225.170:53078->104.36.82.15:4257: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:33.223104976 (202h57m27.654185097s) [noise=true , address=104.36.82.4:4804 , piece_id=MSAHN2DERVG2HL7KZMKIWYUQYO6LRIXGUNWJFXJDDGOWZT54FPVA, seg_pos={0 0}, upload=211112]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:104.36.82.4:4804): protocol: write tcp 188.166.225.170:56176->104.36.82.4:4804: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:33.223149581 (202h57m27.654229712s) [noise=true , address=104.36.82.12:4151 , piece_id=2GGYMOWZ2FTWDNWNQRKSAH7UVJ4R6MLRGC6OHNVV3KTVIIMO4QIQ, seg_pos={0 0}, upload=211112]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:104.36.82.12:4151): protocol: write tcp 188.166.225.170:40582->104.36.82.12:4151: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:33.223157536 (202h57m27.654237667s) [noise=true , address=104.36.82.9:4005 , piece_id=FAR4B5KAALBOACKANEFKGECI2ZVXSHBNYDEIWVGDU2RNBNQXLMXA, seg_pos={0 0}, upload=211112]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:104.36.82.9:4005): protocol: write tcp 188.166.225.170:32954->104.36.82.9:4005: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:33.223177415 (202h57m27.654257536s) [seg_pos={0 0}, upload=211112]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Aug 17 04:23:33.448862924 (202h57m27.879943045s) [upload=211112]: Done uploading segments... Aug 17 04:23:33.448886830 (202h57m27.879966961s) [upload=211112]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 5.684243343s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 10 out of 60 is failed (threshold=3) [...........X..........XXXX.X.X.......X...X.................X] duration: 3.850209721s FAIL (2025-08-17 04:51:12 UTC) orbiter warning: operation: ListObjects ======ERROR====== unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/4M ======LOGS====== Aug 17 04:51:09.056938814 (203h25m3.488018935s) [upload=211594]: Starting upload Aug 17 04:51:09.057017635 (203h25m3.488097766s) [upload=211594]: Done starting upload Aug 17 04:51:09.057190195 (203h25m3.488270326s) [upload=211594]: Uploading segments... Aug 17 04:51:09.057308371 (203h25m3.488388502s) [upload=211594]: Got next segment. Inline: false Aug 17 04:51:09.057342866 (203h25m3.488422997s) [upload=211594]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 17 04:51:09.057347857 (203h25m3.488427978s) [upload=211594]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 17 04:51:09.504198039 (203h25m3.93527816s) [seg_pos={0 0}, upload=211594]: Begin upload segment... Aug 17 04:51:09.504246441 (203h25m3.935326572s) [seg_pos={0 0}, upload=211594]: Done begin upload segment. Aug 17 04:51:09.504264686 (203h25m3.935344817s) [noise=true , address=104.36.82.2:4748 , piece_id=T6MOSFTUKYGVT2YSCEGXMSDXLULB6NI4RGGGB3SSARO7ALNZJN5Q, seg_pos={0 0}, upload=211594]: Uploading piece... Aug 17 04:51:09.504308099 (203h25m3.93538823s) [noise=true , address=104.36.82.19:4436 , piece_id=VN6NIXXOT6KUQE3JU6X35TPYNCI5K4KOSBLVXMOKSP6N2SYRFM6Q, seg_pos={0 0}, upload=211594]: Uploading piece... Aug 17 04:51:09.504588896 (203h25m3.935669017s) [noise=true , address=104.36.82.58:4295 , piece_id=3LNVXIW5IU2BQZRH6QAVV7A2PJ3L3YSGNOXBMIYOJMFBVHJZUPYA, seg_pos={0 0}, upload=211594]: Uploading piece... Aug 17 04:51:09.504650284 (203h25m3.935730405s) [noise=true , address=64.49.13.169:4706 , piece_id=Y4SS3PCBRVCQS5N5N34Y6PGUJ5FPWOCHHO5KMU2PJ5CUXWTHW7RQ, seg_pos={0 0}, upload=211594]: Uploading piece... Aug 17 04:51:09.504844445 (203h25m3.935924566s) [noise=true , address=45.91.116.12:4228 , piece_id=X5VGNN2WVGEW3E4MPUO5MQWZ7L2JMB345IQUUTBI4DYW5HESK33A, seg_pos={0 0}, upload=211594]: Uploading piece... Aug 17 04:51:09.504957571 (203h25m3.936037702s) [noise=true , address=104.36.82.23:4475 , piece_id=TYHG2VUDQ4RRMFLUOLV3U73G7JXA6ZRC757TD34Q2MZTJNRMUIZA, seg_pos={0 0}, upload=211594]: Uploading piece... Aug 17 04:51:09.505108992 (203h25m3.936189123s) [noise=true , address=104.36.82.59:4331 , piece_id=SPNJSZRY7LNIS7ZCHJLOOWKVZU7AP3CODFM73CGOLQZU7HUCT2IQ, seg_pos={0 0}, upload=211594]: Uploading piece... Aug 17 04:51:09.505263607 (203h25m3.936343738s) [noise=true , address=66.23.193.175:4553 , piece_id=BXVNQMDXCQPEUSLIZ24VI22F7FOOJN2MEGJU6UI2SDQIZJSFQJ2A, seg_pos={0 0}, upload=211594]: Uploading piece... Aug 17 04:51:09.505341626 (203h25m3.936421757s) [noise=true , address=77.90.31.10:4523 , piece_id=22C56K4AOQCL5N6WR7QECVAAXHOS3OUNMGWCRFHP5GKSTHIT7OLA, seg_pos={0 0}, upload=211594]: Uploading piece... Aug 17 04:51:09.505565765 (203h25m3.936645896s) [noise=true , address=104.36.82.4:4836 , piece_id=JJWCMFYVQUU6WGEVBEJS4JPZQNNZERI5BVTBBAQQUZV3WD63NGIA, seg_pos={0 0}, upload=211594]: Uploading piece... Aug 17 04:51:09.505597326 (203h25m3.936677457s) [noise=true , address=77.90.31.11:4563 , piece_id=QDC72JPWTU4LH5URAU44ECPV72L4NQPWGDDOHHVYSGNJTFS7C6XQ, seg_pos={0 0}, upload=211594]: Uploading piece... Aug 17 04:51:09.505853546 (203h25m3.936933677s) [noise=true , address=104.36.82.18:4396 , piece_id=3CORBTMNXLB5AG6ZGAPLPBGGFEQ4SSMJHRGXPESWTJD5YPYVUKFA, seg_pos={0 0}, upload=211594]: Uploading piece... Aug 17 04:51:09.505894053 (203h25m3.936974184s) [noise=true , address=104.36.82.44:4896 , piece_id=2DQKKZFJX4N6W5UOOYX4QWSZ3GAFBMQIIDCKU3N3N5MLM4Q7T2AQ, seg_pos={0 0}, upload=211594]: Uploading piece... Aug 17 04:51:09.506090339 (203h25m3.937170471s) [noise=true , address=77.90.31.13:4658 , piece_id=XONXAMV3VAHHSHNIDMAIMMD6KOCKLYJQAUTNVMSUXB6ZCWP5THDA, seg_pos={0 0}, upload=211594]: Uploading piece... Aug 17 04:51:09.506165653 (203h25m3.937245784s) [noise=true , address=104.36.82.5:4850 , piece_id=6ZYI2JS5HMQKTUM2XR4ZTSJMC3MHVILV4YOQ7ANA5PX3FDJXJ2DA, seg_pos={0 0}, upload=211594]: Uploading piece... Aug 17 04:51:09.506275653 (203h25m3.937355774s) [noise=true , address=45.91.116.14:4199 , piece_id=HOTL5QUTJLBBZNF6A4WPRKMQXB72I3TUVSDYUVWMU7G4ZIXOIK3A, seg_pos={0 0}, upload=211594]: Uploading piece... Aug 17 04:51:09.506396365 (203h25m3.937476496s) [noise=true , address=104.36.82.34:4565 , piece_id=LUXKT7N4OUT4UJA345KTE2T3QUD3LEGLUQKQTDZADYBOGE2Z5XNQ, seg_pos={0 0}, upload=211594]: Uploading piece... Aug 17 04:51:09.506467330 (203h25m3.937547461s) [noise=true , address=104.36.82.60:4354 , piece_id=M3INHRZ2XHGLNFSI7OHKHKNCVTSOWTSCJJCGHTTYCAJGFLDWLL7A, seg_pos={0 0}, upload=211594]: Uploading piece... Aug 17 04:51:10.617749206 (203h25m5.048829337s) [upload=211594]: Next returned nil segment Aug 17 04:51:10.617765637 (203h25m5.048845758s) [upload=211594]: Waiting for error group managing segments... Aug 17 04:51:10.899081476 (203h25m5.330161607s) [noise=true , address=64.49.13.169:4706 , piece_id=Y4SS3PCBRVCQS5N5N34Y6PGUJ5FPWOCHHO5KMU2PJ5CUXWTHW7RQ, seg_pos={0 0}, upload=211594]: Done uploading piece. err: <nil> Aug 17 04:51:10.959773754 (203h25m5.390853875s) [noise=true , address=66.23.193.175:4553 , piece_id=BXVNQMDXCQPEUSLIZ24VI22F7FOOJN2MEGJU6UI2SDQIZJSFQJ2A, seg_pos={0 0}, upload=211594]: Done uploading piece. err: <nil> Aug 17 04:51:11.387426163 (203h25m5.818506294s) [noise=true , address=77.90.31.11:4563 , piece_id=QDC72JPWTU4LH5URAU44ECPV72L4NQPWGDDOHHVYSGNJTFS7C6XQ, seg_pos={0 0}, upload=211594]: Done uploading piece. err: <nil> Aug 17 04:51:11.392712709 (203h25m5.82379284s) [noise=true , address=45.91.116.14:4199 , piece_id=HOTL5QUTJLBBZNF6A4WPRKMQXB72I3TUVSDYUVWMU7G4ZIXOIK3A, seg_pos={0 0}, upload=211594]: Done uploading piece. err: <nil> Aug 17 04:51:11.399286516 (203h25m5.830366637s) [noise=true , address=104.36.82.4:4836 , piece_id=JJWCMFYVQUU6WGEVBEJS4JPZQNNZERI5BVTBBAQQUZV3WD63NGIA, seg_pos={0 0}, upload=211594]: Done uploading piece. err: <nil> Aug 17 04:51:11.412750859 (203h25m5.84383099s) [noise=true , address=104.36.82.23:4475 , piece_id=TYHG2VUDQ4RRMFLUOLV3U73G7JXA6ZRC757TD34Q2MZTJNRMUIZA, seg_pos={0 0}, upload=211594]: Done uploading piece. err: <nil> Aug 17 04:51:11.439625902 (203h25m5.870706033s) [noise=true , address=77.90.31.10:4523 , piece_id=22C56K4AOQCL5N6WR7QECVAAXHOS3OUNMGWCRFHP5GKSTHIT7OLA, seg_pos={0 0}, upload=211594]: Done uploading piece. err: <nil> Aug 17 04:51:11.465577498 (203h25m5.896657629s) [noise=true , address=104.36.82.34:4565 , piece_id=LUXKT7N4OUT4UJA345KTE2T3QUD3LEGLUQKQTDZADYBOGE2Z5XNQ, seg_pos={0 0}, upload=211594]: Done uploading piece. err: <nil> Aug 17 04:51:11.469847087 (203h25m5.900927218s) [noise=true , address=104.36.82.60:4354 , piece_id=M3INHRZ2XHGLNFSI7OHKHKNCVTSOWTSCJJCGHTTYCAJGFLDWLL7A, seg_pos={0 0}, upload=211594]: Done uploading piece. err: <nil> Aug 17 04:51:11.470697434 (203h25m5.901777555s) [noise=true , address=104.36.82.5:4850 , piece_id=6ZYI2JS5HMQKTUM2XR4ZTSJMC3MHVILV4YOQ7ANA5PX3FDJXJ2DA, seg_pos={0 0}, upload=211594]: Done uploading piece. err: <nil> Aug 17 04:51:11.470949846 (203h25m5.902029978s) [noise=true , address=104.36.82.19:4436 , piece_id=VN6NIXXOT6KUQE3JU6X35TPYNCI5K4KOSBLVXMOKSP6N2SYRFM6Q, seg_pos={0 0}, upload=211594]: Done uploading piece. err: <nil> Aug 17 04:51:11.486560086 (203h25m5.917640217s) [noise=true , address=104.36.82.2:4748 , piece_id=T6MOSFTUKYGVT2YSCEGXMSDXLULB6NI4RGGGB3SSARO7ALNZJN5Q, seg_pos={0 0}, upload=211594]: Done uploading piece. err: <nil> Aug 17 04:51:11.498376395 (203h25m5.929456526s) [noise=true , address=104.36.82.44:4896 , piece_id=2DQKKZFJX4N6W5UOOYX4QWSZ3GAFBMQIIDCKU3N3N5MLM4Q7T2AQ, seg_pos={0 0}, upload=211594]: Done uploading piece. err: <nil> Aug 17 04:51:11.512939881 (203h25m5.944020012s) [noise=true , address=45.91.116.12:4228 , piece_id=X5VGNN2WVGEW3E4MPUO5MQWZ7L2JMB345IQUUTBI4DYW5HESK33A, seg_pos={0 0}, upload=211594]: Done uploading piece. err: <nil> Aug 17 04:51:11.512969688 (203h25m5.944049819s) [seg_pos={0 0}, upload=211594]: Segment reached optimal threshold of 14 pieces. Aug 17 04:51:11.513156405 (203h25m5.944236536s) [noise=true , address=77.90.31.13:4658 , piece_id=XONXAMV3VAHHSHNIDMAIMMD6KOCKLYJQAUTNVMSUXB6ZCWP5THDA, seg_pos={0 0}, upload=211594]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129aongkQE9HMNz8vsNNA5SvKbhsHqdt9UrtSTPJJpP9mkFsboX): protocol: expected piece hash; context canceled; EOF Aug 17 04:51:11.513220778 (203h25m5.944300909s) [noise=true , address=104.36.82.58:4295 , piece_id=3LNVXIW5IU2BQZRH6QAVV7A2PJ3L3YSGNOXBMIYOJMFBVHJZUPYA, seg_pos={0 0}, upload=211594]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1oLa34qwPBAx3CTvYYj7GN3oxxWGJ9dAYXfke1eUftZc6xzfqg): protocol: expected piece hash; context canceled; EOF Aug 17 04:51:11.513269471 (203h25m5.944349603s) [noise=true , address=104.36.82.59:4331 , piece_id=SPNJSZRY7LNIS7ZCHJLOOWKVZU7AP3CODFM73CGOLQZU7HUCT2IQ, seg_pos={0 0}, upload=211594]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EYU9Z3iNdzeVtxWGQ8F4F7ZT6FbjaFzfr4QQpUDWUpFTrWUYU): protocol: expected piece hash; context canceled; EOF Aug 17 04:51:11.513316901 (203h25m5.944397033s) [noise=true , address=104.36.82.18:4396 , piece_id=3CORBTMNXLB5AG6ZGAPLPBGGFEQ4SSMJHRGXPESWTJD5YPYVUKFA, seg_pos={0 0}, upload=211594]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1k5gzJBGWhpARcfiynburgfTqJ2t7sVNs1mAkUfToMG5saxPqx): protocol: expected piece hash; context canceled; EOF Aug 17 04:51:11.513333754 (203h25m5.944413885s) [seg_pos={0 0}, upload=211594]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 17 04:51:11.513406654 (203h25m5.944486775s) [upload=211594]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 17 04:51:11.513424287 (203h25m5.944504418s) [upload=211594]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 17 04:51:11.809871930 (203h25m6.240952061s) [upload=211594]: Done uploading segments... Aug 17 04:51:11.809892119 (203h25m6.24097225s) [upload=211594]: Upload finished. err: <nil> Duration: 3.850241592s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [..................XXX.......................................] duration: 5.60143098s FAIL (2025-08-17 04:51:04 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== g_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.761278730 (202h56m21.192358861s) [noise=true , address=104.36.82.34:4558 , piece_id=XGFNEBBVZX5UWJN7DALGM2EDG26POCTSBBAUU7DLYQKS2DNSBNXA, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.761661883 (202h56m21.192742014s) [noise=true , address=104.36.82.35:4594 , piece_id=6NUSPNYGSSJOB2CNPIW6DI45MPZARLWDBXH3CWMP4LXMDB2CXWZQ, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.761716548 (202h56m21.192796679s) [noise=true , address=45.91.116.8:4067 , piece_id=EZAI7ZDUBOOWXYJNTB7ZPOUSBUUG2FKV77QVUOZMNM6TUJSKGYWA, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.761972016 (202h56m21.193052147s) [noise=true , address=104.36.82.58:4283 , piece_id=LJKEH2KDSOFPVFPB5UWGVLOHQVZQOY33MYSHFLP4ZWGPKZ4KC64A, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.762254878 (202h56m21.193335009s) [noise=true , address=79.127.201.215:4345 , piece_id=JMSKNIZ7QPGBXIEP5RA3S74XAU5NJ2MUUXSFFIAK377YFGO646YQ, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.762523682 (202h56m21.193603803s) [noise=true , address=45.91.116.13:4181 , piece_id=7VN6RQGOTBHQO6HXM5IFTU32PWNVPH6BJZMQNETJEXOYPR2Y7OCQ, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.762864294 (202h56m21.193944415s) [noise=true , address=45.91.116.14:4203 , piece_id=5LXGQ4JZD2KLW3OMWY5QYRE4MKQNKKSAIMESODKDDT2W33XD45DQ, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.763179537 (202h56m21.194259668s) [noise=true , address=77.90.31.18:4815 , piece_id=G6PINAFFA6DZRRMKU2XK2LYFOES5WG7PTJ5L2ZPUQHMA4JNSRZRQ, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.763478499 (202h56m21.19455864s) [noise=true , address=77.90.31.11:4562 , piece_id=PUL6TAVIIU7732IOASNH5EHQMNCK7DWHJMUJVXRNK74T4GUQDGTA, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.763757683 (202h56m21.194837814s) [noise=true , address=104.36.82.5:4855 , piece_id=3FCDYA6CRYOOU3MDFJPEC4YQNICFVCN2A47RIYPSR4ZX5TIXZX3A, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.763988195 (202h56m21.195068326s) [noise=true , address=45.91.116.5:4308 , piece_id=F52M3VMHRQWR5LIEEPMVKVBFMOVG76UNNGQTFKSF6ZBD7VEOFWVA, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.764273320 (202h56m21.195353451s) [noise=true , address=66.23.193.175:4559 , piece_id=46X42PJIPA6DVC7BG7WPKMPMC44ZBUROEAB5C22M5INXKBJC4KVQ, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.764555100 (202h56m21.195635221s) [noise=true , address=77.90.31.15:4732 , piece_id=4AU7UGKXN4TIPSIRR7IMXSF6WHELO6UTOX35N5X7JHOQQXWWJF2A, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.764757416 (202h56m21.195837547s) [noise=true , address=104.36.82.39:4753 , piece_id=R7BHT5RYLSCDZLTDDRXTSJFSRPLM5PNBYC6Q2G33C5SYIUCNVCLQ, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.765868993 (202h56m21.196949124s) [noise=true , address=104.36.82.50:4047 , piece_id=AKDACFQEPBAFGYKTWBL5RELLZL3L4IJHJT73NS7T7QPWK2C2TDQA, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.767152809 (202h56m21.19823294s) [noise=true , address=104.36.82.19:4436 , piece_id=KVDJTKON4LFVJSAY7XSJ2OQ4ZH67JLMDE2O4EHUWSMZ3EECWSOXA, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.768577886 (202h56m21.199658017s) [noise=true , address=45.91.116.10:4095 , piece_id=JQ2UARQSNKBGHHHBJ6PR45ETR4EYMCXPDEQHEGWFYH52LRJC57UQ, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:27.934309011 (202h56m22.365389142s) [upload=211101]: Next returned nil segment Aug 17 04:22:27.934342545 (202h56m22.365422666s) [upload=211101]: Waiting for error group managing segments... Aug 17 04:22:28.231985083 (202h56m22.663065214s) [noise=true , address=66.23.193.175:4559 , piece_id=46X42PJIPA6DVC7BG7WPKMPMC44ZBUROEAB5C22M5INXKBJC4KVQ, seg_pos={0 0}, upload=211101]: Done uploading piece. err: <nil> Aug 17 04:22:28.670701256 (202h56m23.101781427s) [noise=true , address=45.91.116.10:4095 , piece_id=JQ2UARQSNKBGHHHBJ6PR45ETR4EYMCXPDEQHEGWFYH52LRJC57UQ, seg_pos={0 0}, upload=211101]: Done uploading piece. err: <nil> Aug 17 04:22:28.671952431 (202h56m23.103032562s) [noise=true , address=45.91.116.5:4308 , piece_id=F52M3VMHRQWR5LIEEPMVKVBFMOVG76UNNGQTFKSF6ZBD7VEOFWVA, seg_pos={0 0}, upload=211101]: Done uploading piece. err: <nil> Aug 17 04:22:28.698468278 (202h56m23.129548409s) [noise=true , address=45.91.116.8:4067 , piece_id=EZAI7ZDUBOOWXYJNTB7ZPOUSBUUG2FKV77QVUOZMNM6TUJSKGYWA, seg_pos={0 0}, upload=211101]: Done uploading piece. err: <nil> Aug 17 04:22:28.720171187 (202h56m23.151251318s) [noise=true , address=45.91.116.13:4181 , piece_id=7VN6RQGOTBHQO6HXM5IFTU32PWNVPH6BJZMQNETJEXOYPR2Y7OCQ, seg_pos={0 0}, upload=211101]: Done uploading piece. err: <nil> Aug 17 04:22:28.731591429 (202h56m23.16267156s) [noise=true , address=45.91.116.14:4203 , piece_id=5LXGQ4JZD2KLW3OMWY5QYRE4MKQNKKSAIMESODKDDT2W33XD45DQ, seg_pos={0 0}, upload=211101]: Done uploading piece. err: <nil> Aug 17 04:22:28.798148986 (202h56m23.229229117s) [noise=true , address=77.90.31.18:4815 , piece_id=G6PINAFFA6DZRRMKU2XK2LYFOES5WG7PTJ5L2ZPUQHMA4JNSRZRQ, seg_pos={0 0}, upload=211101]: Done uploading piece. err: <nil> Aug 17 04:22:28.804849037 (202h56m23.235929168s) [noise=true , address=79.127.201.215:4345 , piece_id=JMSKNIZ7QPGBXIEP5RA3S74XAU5NJ2MUUXSFFIAK377YFGO646YQ, seg_pos={0 0}, upload=211101]: Done uploading piece. err: <nil> Aug 17 04:22:28.833066822 (202h56m23.264147013s) [noise=true , address=77.90.31.15:4732 , piece_id=4AU7UGKXN4TIPSIRR7IMXSF6WHELO6UTOX35N5X7JHOQQXWWJF2A, seg_pos={0 0}, upload=211101]: Done uploading piece. err: <nil> Aug 17 04:22:28.846684710 (202h56m23.277764831s) [noise=true , address=77.90.31.11:4562 , piece_id=PUL6TAVIIU7732IOASNH5EHQMNCK7DWHJMUJVXRNK74T4GUQDGTA, seg_pos={0 0}, upload=211101]: Done uploading piece. err: <nil> Aug 17 04:23:25.582810947 (202h57m20.013891078s) [noise=true , address=104.36.82.34:4558 , piece_id=XGFNEBBVZX5UWJN7DALGM2EDG26POCTSBBAUU7DLYQKS2DNSBNXA, seg_pos={0 0}, upload=211101]: Done uploading piece. err: ecclient: upload failed (node:12TU4zvu38mF1fNSergCtMj2cK9eu57Rdfpt5MVk3tc4MZF6dNr, address:104.36.82.34:4558): protocol: rpc: dial tcp 104.36.82.34:4558: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:25.582856475 (202h57m20.013936606s) [noise=true , address=104.36.82.35:4594 , piece_id=6NUSPNYGSSJOB2CNPIW6DI45MPZARLWDBXH3CWMP4LXMDB2CXWZQ, seg_pos={0 0}, upload=211101]: Done uploading piece. err: ecclient: upload failed (node:12D9nh4XFmsjjFcZEMjaL4uw4Whsa8nfMrAsYPu6NVpsxgb3jvf, address:104.36.82.35:4594): protocol: rpc: dial tcp 104.36.82.35:4594: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:25.582973238 (202h57m20.014053369s) [noise=true , address=104.36.82.19:4436 , piece_id=KVDJTKON4LFVJSAY7XSJ2OQ4ZH67JLMDE2O4EHUWSMZ3EECWSOXA, seg_pos={0 0}, upload=211101]: Done uploading piece. err: ecclient: upload failed (node:1WvyEw8XevCiQ8yRgzyMo7WTVzpEJ158ErKfAeV4dSTBkS1oRF, address:104.36.82.19:4436): protocol: rpc: dial tcp 104.36.82.19:4436: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:25.583033393 (202h57m20.014113524s) [noise=true , address=104.36.82.58:4283 , piece_id=LJKEH2KDSOFPVFPB5UWGVLOHQVZQOY33MYSHFLP4ZWGPKZ4KC64A, seg_pos={0 0}, upload=211101]: Done uploading piece. err: ecclient: upload failed (node:1p9jkPtQnmbCXBmw6HSHDWpGaKWqfg87m6hHXy6WDu2giAFnbQ, address:104.36.82.58:4283): protocol: rpc: dial tcp 104.36.82.58:4283: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:25.583069241 (202h57m20.014149362s) [noise=true , address=104.36.82.50:4047 , piece_id=AKDACFQEPBAFGYKTWBL5RELLZL3L4IJHJT73NS7T7QPWK2C2TDQA, seg_pos={0 0}, upload=211101]: Done uploading piece. err: ecclient: upload failed (node:126Z6tvp49dLLdj3T5snJc4tBVGzcX5UBaGafFN3PkhF92vxpfs, address:104.36.82.50:4047): protocol: rpc: dial tcp 104.36.82.50:4047: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:25.583076716 (202h57m20.014156837s) [noise=true , address=104.36.82.39:4753 , piece_id=R7BHT5RYLSCDZLTDDRXTSJFSRPLM5PNBYC6Q2G33C5SYIUCNVCLQ, seg_pos={0 0}, upload=211101]: Done uploading piece. err: ecclient: upload failed (node:1zzW2z6noGQT6xdSmwSQBNYwip6gZjnmqZg7qdb6BP3nHyCLFz, address:104.36.82.39:4753): protocol: rpc: dial tcp 104.36.82.39:4753: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:25.583107535 (202h57m20.014187666s) [noise=true , address=104.36.82.23:4474 , piece_id=RITPHITVW655UQETT6UKS7BZO2S3F4P2MJIO4REOUKYYP4E2CDVQ, seg_pos={0 0}, upload=211101]: Done uploading piece. err: ecclient: upload failed (node:1Fpshye6yGAo3yf71eVpxrCdvoRjpkGX9WEX8KCWYbvuhy5i9j, address:104.36.82.23:4474): protocol: write tcp 188.166.225.170:36642->104.36.82.23:4474: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:25.583165355 (202h57m20.014245487s) [noise=true , address=104.36.82.5:4855 , piece_id=3FCDYA6CRYOOU3MDFJPEC4YQNICFVCN2A47RIYPSR4ZX5TIXZX3A, seg_pos={0 0}, upload=211101]: Done uploading piece. err: ecclient: upload failed (node:1KDNuE7YY8smhDjotushQvY3y9NtLxigbJELHD4Edxy8Qi4BWh, address:104.36.82.5:4855): protocol: write tcp 188.166.225.170:47052->104.36.82.5:4855: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:25.583194000 (202h57m20.014274131s) [seg_pos={0 0}, upload=211101]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Aug 17 04:23:25.810738626 (202h57m20.241818757s) [upload=211101]: Done uploading segments... Aug 17 04:23:25.810761991 (202h57m20.241842122s) [upload=211101]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 5.6014725s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 9 out of 60 is failed (threshold=3) [............X..........XXXX.X.X.......X...X.................] duration: 5.814132263s FAIL (2025-08-17 04:50:55 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/4M") ======LOGS====== Aug 17 04:37:54.182428668 (203h11m48.613508799s) [upload=211361]: Starting upload Aug 17 04:37:54.182474656 (203h11m48.613554777s) [upload=211361]: Done starting upload Aug 17 04:37:54.182665000 (203h11m48.613745131s) [upload=211361]: Uploading segments... Aug 17 04:37:54.182728421 (203h11m48.613808552s) [upload=211361]: Got next segment. Inline: false Aug 17 04:37:54.182758048 (203h11m48.613838179s) [upload=211361]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 17 04:37:54.182762306 (203h11m48.613842437s) [upload=211361]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 17 04:37:54.737160222 (203h11m49.168240353s) [seg_pos={0 0}, upload=211361]: Begin upload segment... Aug 17 04:37:54.737203494 (203h11m49.168283625s) [seg_pos={0 0}, upload=211361]: Done begin upload segment. Aug 17 04:37:54.737224945 (203h11m49.168305076s) [noise=true , address=77.90.31.19:4863 , piece_id=PYERWDVDOTOW2SVDUO63ODTEWMWUGH5OMV2PE7RJTIDVFTF2ZRVQ, seg_pos={0 0}, upload=211361]: Uploading piece... Aug 17 04:37:54.737311591 (203h11m49.168391722s) [noise=true , address=79.127.220.101:4068 , piece_id=6MFJ22JL437NTOH25E3LNH3Q2LQNKKMUT5Q43F2XNIB7346LGFCQ, seg_pos={0 0}, upload=211361]: Uploading piece... Aug 17 04:37:54.737638917 (203h11m49.168719048s) [noise=true , address=77.90.31.12:4601 , piece_id=JC5VIGLKCWTCDHQYWEL6QSRZW3MJZFRFQKRJH54OK7LPUQH7WQTQ, seg_pos={0 0}, upload=211361]: Uploading piece... Aug 17 04:37:54.737939432 (203h11m49.169019563s) [noise=true , address=66.23.193.175:4562 , piece_id=2RZ2A5KBLZBLEG6UECY2GTNF5THR3XJZMS34ULG5SQNWN4O3XNMA, seg_pos={0 0}, upload=211361]: Uploading piece... Aug 17 04:37:54.737952067 (203h11m49.169032198s) [noise=true , address=77.90.31.16:4761 , piece_id=44YO7GYP4RNRWHPP7LP2C4FJBGMWOLEDEC2NN7TAWU3YFHAST3XQ, seg_pos={0 0}, upload=211361]: Uploading piece... Aug 17 04:37:54.738229999 (203h11m49.16931013s) [noise=true , address=104.36.82.36:4648 , piece_id=GTHXHUS6GR26G6UHHMB35MCSEI7I5UJ6K65O76HRJI4WVPIHD7RQ, seg_pos={0 0}, upload=211361]: Uploading piece... Aug 17 04:37:54.738324038 (203h11m49.169404169s) [noise=true , address=66.23.193.175:4563 , piece_id=I65W2GP5RTRZUIT7Y6NSEAL7XC726PP3BYMCWS6UMZEOWAP5CP2A, seg_pos={0 0}, upload=211361]: Uploading piece... Aug 17 04:37:54.738511367 (203h11m49.169591488s) [noise=true , address=104.36.82.4:4833 , piece_id=AMRE5SIA7EJLAP55GFYILJYZJL5624E534TPWFUEQILHG7CCFDXQ, seg_pos={0 0}, upload=211361]: Uploading piece... Aug 17 04:37:54.738655813 (203h11m49.169735954s) [noise=true , address=104.36.82.5:4844 , piece_id=PULFFEXZLSMOYEFGRK6L2POF7ERJQT75J23JT3CTXKZ3LMARQZHQ, seg_pos={0 0}, upload=211361]: Uploading piece... Aug 17 04:37:54.738747769 (203h11m49.1698279s) [noise=true , address=77.90.31.10:4532 , piece_id=5IBCLJNM6LUBCIOPSAERY5HAXV66CR4OKI45RBGOHALTRFYGLMWQ, seg_pos={0 0}, upload=211361]: Uploading piece... Aug 17 04:37:54.738971367 (203h11m49.170051498s) [noise=true , address=104.36.82.60:4360 , piece_id=W4OJU6NRV2HKPC7JDTISAPVT77KIC3JHVJR5KHYY225FLH4PZBBA, seg_pos={0 0}, upload=211361]: Uploading piece... Aug 17 04:37:54.738975705 (203h11m49.170055836s) [noise=true , address=77.90.31.18:4830 , piece_id=QNQF4JAEHOFWRTL6XWKL42XSKA2DMO6Z4CZO5NF337NNSHCMYYBA, seg_pos={0 0}, upload=211361]: Uploading piece... Aug 17 04:37:54.739194995 (203h11m49.170275126s) [noise=true , address=104.36.82.23:4468 , piece_id=NC2BFOASSHA4ZG46LZONWF23DNN3ZZBA6SFO3Q44KD4TCOH4AIQQ, seg_pos={0 0}, upload=211361]: Uploading piece... Aug 17 04:37:54.739278324 (203h11m49.170358445s) [noise=true , address=104.36.82.39:4747 , piece_id=SF2P47ED5FEOCMJ6R6AXP2NTI5LYYALSDLHINNLDVMUMZIE4YLHQ, seg_pos={0 0}, upload=211361]: Uploading piece... Aug 17 04:37:54.739421147 (203h11m49.170501278s) [noise=true , address=104.36.82.3:4800 , piece_id=EKC43HOMYNG77PDESU3QV5OL44HWYQRAN5RUNIB7IMC6NVXQWFVA, seg_pos={0 0}, upload=211361]: Uploading piece... Aug 17 04:37:54.739561055 (203h11m49.170641186s) [noise=true , address=77.90.31.13:4634 , piece_id=ZXX5E54TSYUG557OGOCBQMBKS35TUG6OPDOIG3TTS3OYGTRA24XA, seg_pos={0 0}, upload=211361]: Uploading piece... Aug 17 04:37:54.739643824 (203h11m49.170723955s) [noise=true , address=104.36.82.58:4286 , piece_id=JDBFFYOZPYJ5FXJQIVEBB5IWSBGT6LCBUZF5O25OKWFRXROPHS3Q, seg_pos={0 0}, upload=211361]: Uploading piece... Aug 17 04:37:54.739852683 (203h11m49.170932814s) [noise=true , address=104.36.82.44:4876 , piece_id=H53T7WMGX4CDVQNCYOG6NOYFVRHD5Z35LNF37FVKSWYIVJS2EFZA, seg_pos={0 0}, upload=211361]: Uploading piece... Aug 17 04:37:55.861243533 (203h11m50.292323664s) [upload=211361]: Next returned nil segment Aug 17 04:37:55.861256768 (203h11m50.292336899s) [upload=211361]: Waiting for error group managing segments... Aug 17 04:37:56.165335502 (203h11m50.596415633s) [noise=true , address=66.23.193.175:4562 , piece_id=2RZ2A5KBLZBLEG6UECY2GTNF5THR3XJZMS34ULG5SQNWN4O3XNMA, seg_pos={0 0}, upload=211361]: Done uploading piece. err: <nil> Aug 17 04:37:56.263548583 (203h11m50.694628715s) [noise=true , address=66.23.193.175:4563 , piece_id=I65W2GP5RTRZUIT7Y6NSEAL7XC726PP3BYMCWS6UMZEOWAP5CP2A, seg_pos={0 0}, upload=211361]: Done uploading piece. err: <nil> Aug 17 04:37:56.569392606 (203h11m51.000472737s) [noise=true , address=79.127.220.101:4068 , piece_id=6MFJ22JL437NTOH25E3LNH3Q2LQNKKMUT5Q43F2XNIB7346LGFCQ, seg_pos={0 0}, upload=211361]: Done uploading piece. err: <nil> Aug 17 04:37:56.620901052 (203h11m51.051981183s) [noise=true , address=77.90.31.12:4601 , piece_id=JC5VIGLKCWTCDHQYWEL6QSRZW3MJZFRFQKRJH54OK7LPUQH7WQTQ, seg_pos={0 0}, upload=211361]: Done uploading piece. err: <nil> Aug 17 04:37:56.639713969 (203h11m51.0707941s) [noise=true , address=77.90.31.16:4761 , piece_id=44YO7GYP4RNRWHPP7LP2C4FJBGMWOLEDEC2NN7TAWU3YFHAST3XQ, seg_pos={0 0}, upload=211361]: Done uploading piece. err: <nil> Aug 17 04:37:56.649068307 (203h11m51.080148438s) [noise=true , address=104.36.82.4:4833 , piece_id=AMRE5SIA7EJLAP55GFYILJYZJL5624E534TPWFUEQILHG7CCFDXQ, seg_pos={0 0}, upload=211361]: Done uploading piece. err: <nil> Aug 17 04:37:56.701677831 (203h11m51.132757962s) [noise=true , address=104.36.82.3:4800 , piece_id=EKC43HOMYNG77PDESU3QV5OL44HWYQRAN5RUNIB7IMC6NVXQWFVA, seg_pos={0 0}, upload=211361]: Done uploading piece. err: <nil> Aug 17 04:37:56.719713991 (203h11m51.150794132s) [noise=true , address=104.36.82.23:4468 , piece_id=NC2BFOASSHA4ZG46LZONWF23DNN3ZZBA6SFO3Q44KD4TCOH4AIQQ, seg_pos={0 0}, upload=211361]: Done uploading piece. err: <nil> Aug 17 04:37:56.730632311 (203h11m51.161712432s) [noise=true , address=104.36.82.39:4747 , piece_id=SF2P47ED5FEOCMJ6R6AXP2NTI5LYYALSDLHINNLDVMUMZIE4YLHQ, seg_pos={0 0}, upload=211361]: Done uploading piece. err: <nil> Aug 17 04:37:56.761149974 (203h11m51.192230105s) [noise=true , address=104.36.82.60:4360 , piece_id=W4OJU6NRV2HKPC7JDTISAPVT77KIC3JHVJR5KHYY225FLH4PZBBA, seg_pos={0 0}, upload=211361]: Done uploading piece. err: <nil> Aug 17 04:37:56.775738719 (203h11m51.20681885s) [noise=true , address=77.90.31.18:4830 , piece_id=QNQF4JAEHOFWRTL6XWKL42XSKA2DMO6Z4CZO5NF337NNSHCMYYBA, seg_pos={0 0}, upload=211361]: Done uploading piece. err: <nil> Aug 17 04:37:56.779316274 (203h11m51.210396405s) [noise=true , address=77.90.31.13:4634 , piece_id=ZXX5E54TSYUG557OGOCBQMBKS35TUG6OPDOIG3TTS3OYGTRA24XA, seg_pos={0 0}, upload=211361]: Done uploading piece. err: <nil> Aug 17 04:37:56.802147800 (203h11m51.233227961s) [noise=true , address=104.36.82.58:4286 , piece_id=JDBFFYOZPYJ5FXJQIVEBB5IWSBGT6LCBUZF5O25OKWFRXROPHS3Q, seg_pos={0 0}, upload=211361]: Done uploading piece. err: <nil> Aug 17 04:37:56.832614886 (203h11m51.263695017s) [noise=true , address=104.36.82.44:4876 , piece_id=H53T7WMGX4CDVQNCYOG6NOYFVRHD5Z35LNF37FVKSWYIVJS2EFZA, seg_pos={0 0}, upload=211361]: Done uploading piece. err: <nil> Aug 17 04:37:56.832646346 (203h11m51.263726467s) [seg_pos={0 0}, upload=211361]: Segment reached optimal threshold of 14 pieces. Aug 17 04:37:56.832765443 (203h11m51.263845574s) [noise=true , address=77.90.31.10:4532 , piece_id=5IBCLJNM6LUBCIOPSAERY5HAXV66CR4OKI45RBGOHALTRFYGLMWQ, seg_pos={0 0}, upload=211361]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1T6kkKvTNR5MZZijjr1vMU6g75YkoRpzMvGSgQCqRnfCm9JULe): protocol: expected piece hash; context canceled; EOF Aug 17 04:37:56.832872889 (203h11m51.26395302s) [noise=true , address=104.36.82.36:4648 , piece_id=GTHXHUS6GR26G6UHHMB35MCSEI7I5UJ6K65O76HRJI4WVPIHD7RQ, seg_pos={0 0}, upload=211361]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XHNZDY7xKZ2bss1rSdjbGP3ugSwGRwRzBSWHfXgdKKz794wMq): protocol: expected piece hash; context canceled; EOF Aug 17 04:37:56.832892156 (203h11m51.263972287s) [noise=true , address=77.90.31.19:4863 , piece_id=PYERWDVDOTOW2SVDUO63ODTEWMWUGH5OMV2PE7RJTIDVFTF2ZRVQ, seg_pos={0 0}, upload=211361]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124x1cbCPJRK8YkgSP6UzHymoLu1E1znnfg5WBiD89kSDLi8JeL): protocol: expected piece hash; context canceled; EOF Aug 17 04:37:56.832974564 (203h11m51.264054695s) [noise=true , address=104.36.82.5:4844 , piece_id=PULFFEXZLSMOYEFGRK6L2POF7ERJQT75J23JT3CTXKZ3LMARQZHQ, seg_pos={0 0}, upload=211361]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12iovpNdbidkiv9mEHaAPjBubGmoqr4sUJWm8Cb64LzS6wuwLWa): protocol: write tcp 188.166.225.170:59826->104.36.82.5:4844: use of closed network connection; write tcp 188.166.225.170:59826->104.36.82.5:4844: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59826->104.36.82.5:4844: use of closed network connection Aug 17 04:37:56.833025241 (203h11m51.264105372s) [seg_pos={0 0}, upload=211361]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 17 04:37:56.833116014 (203h11m51.264196135s) [upload=211361]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 17 04:37:56.833120332 (203h11m51.264200463s) [upload=211361]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 17 04:37:58.261163702 (203h11m52.692243834s) [upload=211361]: Done uploading segments... Aug 17 04:37:58.261181997 (203h11m52.692262118s) [upload=211361]: Upload finished. err: <nil> Duration: 5.814156622s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.................XXXX.......................................] duration: 6.338183256s FAIL (2025-08-17 04:50:46 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.338202822s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.......................XX...................................] duration: 6.558847441s FAIL (2025-08-17 04:50:44 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.558862079s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.................XXXX.......................................] duration: 5.765210116s FAIL (2025-08-17 04:50:41 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== Aug 17 04:23:03.873697274 (202h56m58.304777405s) [upload=211112]: Starting upload Aug 17 04:23:03.873756368 (202h56m58.304836499s) [upload=211112]: Done starting upload Aug 17 04:23:03.873839848 (202h56m58.304919989s) [upload=211112]: Uploading segments... Aug 17 04:23:03.873950850 (202h56m58.305030981s) [upload=211112]: Got next segment. Inline: false Aug 17 04:23:03.873976509 (202h56m58.30505664s) [upload=211112]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 17 04:23:03.873980757 (202h56m58.305060888s) [upload=211112]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 17 04:23:04.113727713 (202h56m58.544807844s) [seg_pos={0 0}, upload=211112]: Begin upload segment... Aug 17 04:23:04.113774051 (202h56m58.544854182s) [seg_pos={0 0}, upload=211112]: Done begin upload segment. Aug 17 04:23:04.113820890 (202h56m58.544901021s) [noise=true , address=45.91.116.13:4155 , piece_id=CO3R5XZGH3NJADAFDILPNZEKKFJSRPB4JOFBBCSYYQAD3KNRABNA, seg_pos={0 0}, upload=211112]: Uploading piece... Aug 17 04:23:04.113845989 (202h56m58.54492612s) [noise=true , address=45.91.116.7:4009 , piece_id=IVTDHCTNJEEO3QATGN25YH5Y2MSEQUWJIFPXSJWINUEXDHQAGELA, seg_pos={0 0}, upload=211112]: Uploading piece... Aug 17 04:23:04.114195037 (202h56m58.545275168s) [noise=true , address=45.91.116.4:4938 , piece_id=X5MDNBZYRYYOXDEAMGIOD5CXX7PGAMTFRMROOVTR6NIAYQ5VMXGA, seg_pos={0 0}, upload=211112]: Uploading piece... Aug 17 04:23:04.114281412 (202h56m58.545361543s) [noise=true , address=45.91.116.9:4261 , piece_id=PNOEBFIPLWM4B2QLGJUVEBTT23J5AVRFTWTTKLHFUJ5MNJTSM2JA, seg_pos={0 0}, upload=211112]: Uploading piece... Aug 17 04:23:04.114482115 (202h56m58.545562237s) [noise=true , address=77.90.31.16:4735 , piece_id=G5YGF7LXNPBU2TDRYPAFPVYQSTWUGG7Y6YM2Q6RWMCZY5RVNM4BA, seg_pos={0 0}, upload=211112]: Uploading piece... Aug 17 04:23:04.114570385 (202h56m58.545650516s) [noise=true , address=104.36.82.9:4005 , piece_id=FAR4B5KAALBOACKANEFKGECI2ZVXSHBNYDEIWVGDU2RNBNQXLMXA, seg_pos={0 0}, upload=211112]: Uploading piece... Aug 17 04:23:04.114757392 (202h56m58.545837523s) [noise=true , address=45.91.116.9:4263 , piece_id=VT734KJZ7T3CZ3W3P4M4ZCYKOLLTPP3ILFRWL26Z7K4LWBDZV4FA, seg_pos={0 0}, upload=211112]: Uploading piece... Aug 17 04:23:04.114874727 (202h56m58.545954858s) [noise=true , address=104.36.82.12:4151 , piece_id=2GGYMOWZ2FTWDNWNQRKSAH7UVJ4R6MLRGC6OHNVV3KTVIIMO4QIQ, seg_pos={0 0}, upload=211112]: Uploading piece... Aug 17 04:23:04.115030715 (202h56m58.546110846s) [noise=true , address=45.91.116.12:4225 , piece_id=EUPPM5NRIU5MCZBER3VGZBZEMZZ65Q2MOY2DXKFBIPFJ53M5TJHQ, seg_pos={0 0}, upload=211112]: Uploading piece... Aug 17 04:23:04.115194288 (202h56m58.546274419s) [noise=true , address=104.36.82.4:4804 , piece_id=MSAHN2DERVG2HL7KZMKIWYUQYO6LRIXGUNWJFXJDDGOWZT54FPVA, seg_pos={0 0}, upload=211112]: Uploading piece... Aug 17 04:23:04.115307967 (202h56m58.546388098s) [noise=true , address=104.36.82.5:4838 , piece_id=IYA7ERDAAFYUGC4666CBEX5HZEXZVFPES6JUYRZROY25OAPGVEQQ, seg_pos={0 0}, upload=211112]: Uploading piece... Aug 17 04:23:04.115535692 (202h56m58.546615823s) [noise=true , address=45.91.116.12:4227 , piece_id=XZR3AWJG57RUB4TN7SUWNIWWD2XGKCMDXSWSM74WS3ATHS32EBJQ, seg_pos={0 0}, upload=211112]: Uploading piece... Aug 17 04:23:04.115754130 (202h56m58.546834261s) [noise=true , address=45.91.116.7:4010 , piece_id=HNJU4MPWT67V26J5XDCTHFEOOJL2HS3VAOHB4EIPQVF4D2NNRFMA, seg_pos={0 0}, upload=211112]: Uploading piece... Aug 17 04:23:04.116079182 (202h56m58.547159323s) [noise=true , address=45.91.116.14:4189 , piece_id=2P3GOZDEMJGBAK2F6SY7NJNQUTS5ZYM4B4IMQFAA5TYFARD4DVCQ, seg_pos={0 0}, upload=211112]: Uploading piece... Aug 17 04:23:04.116350181 (202h56m58.547430312s) [noise=true , address=104.36.82.15:4257 , piece_id=Q5GLP6TS7N7CM4DO3LWDCDR6O76OFZ6IVQJC5VQWMA5XLNWZ2E4A, seg_pos={0 0}, upload=211112]: Uploading piece... Aug 17 04:23:04.116554471 (202h56m58.547634602s) [noise=true , address=104.36.82.8:4969 , piece_id=3D3YAZSP7CD6IFZGDKPNKIJPWZ2OJFFVJCSMTO42VNJEFVU3ORHA, seg_pos={0 0}, upload=211112]: Uploading piece... Aug 17 04:23:05.868765063 (202h57m0.299845195s) [upload=211112]: Next returned nil segment Aug 17 04:23:05.868801043 (202h57m0.299881164s) [upload=211112]: Waiting for error group managing segments... Aug 17 04:23:06.147430529 (202h57m0.57851066s) [noise=true , address=45.91.116.12:4227 , piece_id=XZR3AWJG57RUB4TN7SUWNIWWD2XGKCMDXSWSM74WS3ATHS32EBJQ, seg_pos={0 0}, upload=211112]: Done uploading piece. err: <nil> Aug 17 04:23:06.182400314 (202h57m0.613480445s) [noise=true , address=45.91.116.12:4225 , piece_id=EUPPM5NRIU5MCZBER3VGZBZEMZZ65Q2MOY2DXKFBIPFJ53M5TJHQ, seg_pos={0 0}, upload=211112]: Done uploading piece. err: <nil> Aug 17 04:23:06.191910532 (202h57m0.622990663s) [noise=true , address=45.91.116.4:4938 , piece_id=X5MDNBZYRYYOXDEAMGIOD5CXX7PGAMTFRMROOVTR6NIAYQ5VMXGA, seg_pos={0 0}, upload=211112]: Done uploading piece. err: <nil> Aug 17 04:23:06.197391631 (202h57m0.628471782s) [noise=true , address=45.91.116.9:4261 , piece_id=PNOEBFIPLWM4B2QLGJUVEBTT23J5AVRFTWTTKLHFUJ5MNJTSM2JA, seg_pos={0 0}, upload=211112]: Done uploading piece. err: <nil> Aug 17 04:23:06.198069487 (202h57m0.629149618s) [noise=true , address=45.91.116.9:4263 , piece_id=VT734KJZ7T3CZ3W3P4M4ZCYKOLLTPP3ILFRWL26Z7K4LWBDZV4FA, seg_pos={0 0}, upload=211112]: Done uploading piece. err: <nil> Aug 17 04:23:06.200597014 (202h57m0.631677145s) [noise=true , address=45.91.116.7:4009 , piece_id=IVTDHCTNJEEO3QATGN25YH5Y2MSEQUWJIFPXSJWINUEXDHQAGELA, seg_pos={0 0}, upload=211112]: Done uploading piece. err: <nil> Aug 17 04:23:06.288053251 (202h57m0.719133372s) [noise=true , address=45.91.116.7:4010 , piece_id=HNJU4MPWT67V26J5XDCTHFEOOJL2HS3VAOHB4EIPQVF4D2NNRFMA, seg_pos={0 0}, upload=211112]: Done uploading piece. err: <nil> Aug 17 04:23:06.298810435 (202h57m0.729890566s) [noise=true , address=45.91.116.14:4189 , piece_id=2P3GOZDEMJGBAK2F6SY7NJNQUTS5ZYM4B4IMQFAA5TYFARD4DVCQ, seg_pos={0 0}, upload=211112]: Done uploading piece. err: <nil> Aug 17 04:23:06.301049520 (202h57m0.732129651s) [noise=true , address=45.91.116.13:4155 , piece_id=CO3R5XZGH3NJADAFDILPNZEKKFJSRPB4JOFBBCSYYQAD3KNRABNA, seg_pos={0 0}, upload=211112]: Done uploading piece. err: <nil> Aug 17 04:23:06.402776985 (202h57m0.833857116s) [noise=true , address=77.90.31.16:4735 , piece_id=G5YGF7LXNPBU2TDRYPAFPVYQSTWUGG7Y6YM2Q6RWMCZY5RVNM4BA, seg_pos={0 0}, upload=211112]: Done uploading piece. err: <nil> Aug 17 04:23:33.223003412 (202h57m27.654083543s) [noise=true , address=104.36.82.5:4838 , piece_id=IYA7ERDAAFYUGC4666CBEX5HZEXZVFPES6JUYRZROY25OAPGVEQQ, seg_pos={0 0}, upload=211112]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:104.36.82.5:4838): protocol: write tcp 188.166.225.170:42258->104.36.82.5:4838: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:33.223050702 (202h57m27.654130823s) [noise=true , address=104.36.82.8:4969 , piece_id=3D3YAZSP7CD6IFZGDKPNKIJPWZ2OJFFVJCSMTO42VNJEFVU3ORHA, seg_pos={0 0}, upload=211112]: Done uploading piece. err: ecclient: upload failed (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, address:104.36.82.8:4969): protocol: write tcp 188.166.225.170:52308->104.36.82.8:4969: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:33.223090619 (202h57m27.65417075s) [noise=true , address=104.36.82.15:4257 , piece_id=Q5GLP6TS7N7CM4DO3LWDCDR6O76OFZ6IVQJC5VQWMA5XLNWZ2E4A, seg_pos={0 0}, upload=211112]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:104.36.82.15:4257): protocol: write tcp 188.166.225.170:53078->104.36.82.15:4257: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:33.223104976 (202h57m27.654185097s) [noise=true , address=104.36.82.4:4804 , piece_id=MSAHN2DERVG2HL7KZMKIWYUQYO6LRIXGUNWJFXJDDGOWZT54FPVA, seg_pos={0 0}, upload=211112]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:104.36.82.4:4804): protocol: write tcp 188.166.225.170:56176->104.36.82.4:4804: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:33.223149581 (202h57m27.654229712s) [noise=true , address=104.36.82.12:4151 , piece_id=2GGYMOWZ2FTWDNWNQRKSAH7UVJ4R6MLRGC6OHNVV3KTVIIMO4QIQ, seg_pos={0 0}, upload=211112]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:104.36.82.12:4151): protocol: write tcp 188.166.225.170:40582->104.36.82.12:4151: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:33.223157536 (202h57m27.654237667s) [noise=true , address=104.36.82.9:4005 , piece_id=FAR4B5KAALBOACKANEFKGECI2ZVXSHBNYDEIWVGDU2RNBNQXLMXA, seg_pos={0 0}, upload=211112]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:104.36.82.9:4005): protocol: write tcp 188.166.225.170:32954->104.36.82.9:4005: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:33.223177415 (202h57m27.654257536s) [seg_pos={0 0}, upload=211112]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Aug 17 04:23:33.448862924 (202h57m27.879943045s) [upload=211112]: Done uploading segments... Aug 17 04:23:33.448886830 (202h57m27.879966961s) [upload=211112]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 5.765231847s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [.......................................X.....X..............] duration: 1.873209891s FAIL (2025-08-17 04:50:28 UTC) orbiter warning: operation: ListObjects ======ERROR====== unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/1K ======LOGS====== Aug 17 04:41:26.040732864 (203h15m20.471812985s) [upload=211420]: Starting upload Aug 17 04:41:26.040780384 (203h15m20.471860515s) [upload=211420]: Done starting upload Aug 17 04:41:26.040828978 (203h15m20.471909109s) [upload=211420]: Uploading segments... Aug 17 04:41:26.040863473 (203h15m20.471943604s) [upload=211420]: Got next segment. Inline: true Aug 17 04:41:26.040867290 (203h15m20.471947422s) [upload=211420]: Waiting for error group managing segments... Aug 17 04:41:26.040897880 (203h15m20.471978011s) [upload=211420]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 17 04:41:26.040901597 (203h15m20.471981728s) [upload=211420]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 17 04:41:26.040904041 (203h15m20.471984162s) [upload=211420]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 17 04:41:26.462140724 (203h15m20.893220855s) [upload=211420]: Done uploading segments... Aug 17 04:41:26.462159310 (203h15m20.893239431s) [upload=211420]: Upload finished. err: <nil> Duration:* 1.873234116s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [..................XXXX......................................] duration: 6.266592915s FAIL (2025-08-17 04:50:25 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.266612492s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [..................XXXX......................................] duration: 6.035528233s FAIL (2025-08-17 04:50:15 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== Aug 17 04:23:03.873697274 (202h56m58.304777405s) [upload=211112]: Starting upload Aug 17 04:23:03.873756368 (202h56m58.304836499s) [upload=211112]: Done starting upload Aug 17 04:23:03.873839848 (202h56m58.304919989s) [upload=211112]: Uploading segments... Aug 17 04:23:03.873950850 (202h56m58.305030981s) [upload=211112]: Got next segment. Inline: false Aug 17 04:23:03.873976509 (202h56m58.30505664s) [upload=211112]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 17 04:23:03.873980757 (202h56m58.305060888s) [upload=211112]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 17 04:23:04.113727713 (202h56m58.544807844s) [seg_pos={0 0}, upload=211112]: Begin upload segment... Aug 17 04:23:04.113774051 (202h56m58.544854182s) [seg_pos={0 0}, upload=211112]: Done begin upload segment. Aug 17 04:23:04.113820890 (202h56m58.544901021s) [noise=true , address=45.91.116.13:4155 , piece_id=CO3R5XZGH3NJADAFDILPNZEKKFJSRPB4JOFBBCSYYQAD3KNRABNA, seg_pos={0 0}, upload=211112]: Uploading piece... Aug 17 04:23:04.113845989 (202h56m58.54492612s) [noise=true , address=45.91.116.7:4009 , piece_id=IVTDHCTNJEEO3QATGN25YH5Y2MSEQUWJIFPXSJWINUEXDHQAGELA, seg_pos={0 0}, upload=211112]: Uploading piece... Aug 17 04:23:04.114195037 (202h56m58.545275168s) [noise=true , address=45.91.116.4:4938 , piece_id=X5MDNBZYRYYOXDEAMGIOD5CXX7PGAMTFRMROOVTR6NIAYQ5VMXGA, seg_pos={0 0}, upload=211112]: Uploading piece... Aug 17 04:23:04.114281412 (202h56m58.545361543s) [noise=true , address=45.91.116.9:4261 , piece_id=PNOEBFIPLWM4B2QLGJUVEBTT23J5AVRFTWTTKLHFUJ5MNJTSM2JA, seg_pos={0 0}, upload=211112]: Uploading piece... Aug 17 04:23:04.114482115 (202h56m58.545562237s) [noise=true , address=77.90.31.16:4735 , piece_id=G5YGF7LXNPBU2TDRYPAFPVYQSTWUGG7Y6YM2Q6RWMCZY5RVNM4BA, seg_pos={0 0}, upload=211112]: Uploading piece... Aug 17 04:23:04.114570385 (202h56m58.545650516s) [noise=true , address=104.36.82.9:4005 , piece_id=FAR4B5KAALBOACKANEFKGECI2ZVXSHBNYDEIWVGDU2RNBNQXLMXA, seg_pos={0 0}, upload=211112]: Uploading piece... Aug 17 04:23:04.114757392 (202h56m58.545837523s) [noise=true , address=45.91.116.9:4263 , piece_id=VT734KJZ7T3CZ3W3P4M4ZCYKOLLTPP3ILFRWL26Z7K4LWBDZV4FA, seg_pos={0 0}, upload=211112]: Uploading piece... Aug 17 04:23:04.114874727 (202h56m58.545954858s) [noise=true , address=104.36.82.12:4151 , piece_id=2GGYMOWZ2FTWDNWNQRKSAH7UVJ4R6MLRGC6OHNVV3KTVIIMO4QIQ, seg_pos={0 0}, upload=211112]: Uploading piece... Aug 17 04:23:04.115030715 (202h56m58.546110846s) [noise=true , address=45.91.116.12:4225 , piece_id=EUPPM5NRIU5MCZBER3VGZBZEMZZ65Q2MOY2DXKFBIPFJ53M5TJHQ, seg_pos={0 0}, upload=211112]: Uploading piece... Aug 17 04:23:04.115194288 (202h56m58.546274419s) [noise=true , address=104.36.82.4:4804 , piece_id=MSAHN2DERVG2HL7KZMKIWYUQYO6LRIXGUNWJFXJDDGOWZT54FPVA, seg_pos={0 0}, upload=211112]: Uploading piece... Aug 17 04:23:04.115307967 (202h56m58.546388098s) [noise=true , address=104.36.82.5:4838 , piece_id=IYA7ERDAAFYUGC4666CBEX5HZEXZVFPES6JUYRZROY25OAPGVEQQ, seg_pos={0 0}, upload=211112]: Uploading piece... Aug 17 04:23:04.115535692 (202h56m58.546615823s) [noise=true , address=45.91.116.12:4227 , piece_id=XZR3AWJG57RUB4TN7SUWNIWWD2XGKCMDXSWSM74WS3ATHS32EBJQ, seg_pos={0 0}, upload=211112]: Uploading piece... Aug 17 04:23:04.115754130 (202h56m58.546834261s) [noise=true , address=45.91.116.7:4010 , piece_id=HNJU4MPWT67V26J5XDCTHFEOOJL2HS3VAOHB4EIPQVF4D2NNRFMA, seg_pos={0 0}, upload=211112]: Uploading piece... Aug 17 04:23:04.116079182 (202h56m58.547159323s) [noise=true , address=45.91.116.14:4189 , piece_id=2P3GOZDEMJGBAK2F6SY7NJNQUTS5ZYM4B4IMQFAA5TYFARD4DVCQ, seg_pos={0 0}, upload=211112]: Uploading piece... Aug 17 04:23:04.116350181 (202h56m58.547430312s) [noise=true , address=104.36.82.15:4257 , piece_id=Q5GLP6TS7N7CM4DO3LWDCDR6O76OFZ6IVQJC5VQWMA5XLNWZ2E4A, seg_pos={0 0}, upload=211112]: Uploading piece... Aug 17 04:23:04.116554471 (202h56m58.547634602s) [noise=true , address=104.36.82.8:4969 , piece_id=3D3YAZSP7CD6IFZGDKPNKIJPWZ2OJFFVJCSMTO42VNJEFVU3ORHA, seg_pos={0 0}, upload=211112]: Uploading piece... Aug 17 04:23:05.868765063 (202h57m0.299845195s) [upload=211112]: Next returned nil segment Aug 17 04:23:05.868801043 (202h57m0.299881164s) [upload=211112]: Waiting for error group managing segments... Aug 17 04:23:06.147430529 (202h57m0.57851066s) [noise=true , address=45.91.116.12:4227 , piece_id=XZR3AWJG57RUB4TN7SUWNIWWD2XGKCMDXSWSM74WS3ATHS32EBJQ, seg_pos={0 0}, upload=211112]: Done uploading piece. err: <nil> Aug 17 04:23:06.182400314 (202h57m0.613480445s) [noise=true , address=45.91.116.12:4225 , piece_id=EUPPM5NRIU5MCZBER3VGZBZEMZZ65Q2MOY2DXKFBIPFJ53M5TJHQ, seg_pos={0 0}, upload=211112]: Done uploading piece. err: <nil> Aug 17 04:23:06.191910532 (202h57m0.622990663s) [noise=true , address=45.91.116.4:4938 , piece_id=X5MDNBZYRYYOXDEAMGIOD5CXX7PGAMTFRMROOVTR6NIAYQ5VMXGA, seg_pos={0 0}, upload=211112]: Done uploading piece. err: <nil> Aug 17 04:23:06.197391631 (202h57m0.628471782s) [noise=true , address=45.91.116.9:4261 , piece_id=PNOEBFIPLWM4B2QLGJUVEBTT23J5AVRFTWTTKLHFUJ5MNJTSM2JA, seg_pos={0 0}, upload=211112]: Done uploading piece. err: <nil> Aug 17 04:23:06.198069487 (202h57m0.629149618s) [noise=true , address=45.91.116.9:4263 , piece_id=VT734KJZ7T3CZ3W3P4M4ZCYKOLLTPP3ILFRWL26Z7K4LWBDZV4FA, seg_pos={0 0}, upload=211112]: Done uploading piece. err: <nil> Aug 17 04:23:06.200597014 (202h57m0.631677145s) [noise=true , address=45.91.116.7:4009 , piece_id=IVTDHCTNJEEO3QATGN25YH5Y2MSEQUWJIFPXSJWINUEXDHQAGELA, seg_pos={0 0}, upload=211112]: Done uploading piece. err: <nil> Aug 17 04:23:06.288053251 (202h57m0.719133372s) [noise=true , address=45.91.116.7:4010 , piece_id=HNJU4MPWT67V26J5XDCTHFEOOJL2HS3VAOHB4EIPQVF4D2NNRFMA, seg_pos={0 0}, upload=211112]: Done uploading piece. err: <nil> Aug 17 04:23:06.298810435 (202h57m0.729890566s) [noise=true , address=45.91.116.14:4189 , piece_id=2P3GOZDEMJGBAK2F6SY7NJNQUTS5ZYM4B4IMQFAA5TYFARD4DVCQ, seg_pos={0 0}, upload=211112]: Done uploading piece. err: <nil> Aug 17 04:23:06.301049520 (202h57m0.732129651s) [noise=true , address=45.91.116.13:4155 , piece_id=CO3R5XZGH3NJADAFDILPNZEKKFJSRPB4JOFBBCSYYQAD3KNRABNA, seg_pos={0 0}, upload=211112]: Done uploading piece. err: <nil> Aug 17 04:23:06.402776985 (202h57m0.833857116s) [noise=true , address=77.90.31.16:4735 , piece_id=G5YGF7LXNPBU2TDRYPAFPVYQSTWUGG7Y6YM2Q6RWMCZY5RVNM4BA, seg_pos={0 0}, upload=211112]: Done uploading piece. err: <nil> Aug 17 04:23:33.223003412 (202h57m27.654083543s) [noise=true , address=104.36.82.5:4838 , piece_id=IYA7ERDAAFYUGC4666CBEX5HZEXZVFPES6JUYRZROY25OAPGVEQQ, seg_pos={0 0}, upload=211112]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:104.36.82.5:4838): protocol: write tcp 188.166.225.170:42258->104.36.82.5:4838: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:33.223050702 (202h57m27.654130823s) [noise=true , address=104.36.82.8:4969 , piece_id=3D3YAZSP7CD6IFZGDKPNKIJPWZ2OJFFVJCSMTO42VNJEFVU3ORHA, seg_pos={0 0}, upload=211112]: Done uploading piece. err: ecclient: upload failed (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, address:104.36.82.8:4969): protocol: write tcp 188.166.225.170:52308->104.36.82.8:4969: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:33.223090619 (202h57m27.65417075s) [noise=true , address=104.36.82.15:4257 , piece_id=Q5GLP6TS7N7CM4DO3LWDCDR6O76OFZ6IVQJC5VQWMA5XLNWZ2E4A, seg_pos={0 0}, upload=211112]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:104.36.82.15:4257): protocol: write tcp 188.166.225.170:53078->104.36.82.15:4257: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:33.223104976 (202h57m27.654185097s) [noise=true , address=104.36.82.4:4804 , piece_id=MSAHN2DERVG2HL7KZMKIWYUQYO6LRIXGUNWJFXJDDGOWZT54FPVA, seg_pos={0 0}, upload=211112]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:104.36.82.4:4804): protocol: write tcp 188.166.225.170:56176->104.36.82.4:4804: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:33.223149581 (202h57m27.654229712s) [noise=true , address=104.36.82.12:4151 , piece_id=2GGYMOWZ2FTWDNWNQRKSAH7UVJ4R6MLRGC6OHNVV3KTVIIMO4QIQ, seg_pos={0 0}, upload=211112]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:104.36.82.12:4151): protocol: write tcp 188.166.225.170:40582->104.36.82.12:4151: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:33.223157536 (202h57m27.654237667s) [noise=true , address=104.36.82.9:4005 , piece_id=FAR4B5KAALBOACKANEFKGECI2ZVXSHBNYDEIWVGDU2RNBNQXLMXA, seg_pos={0 0}, upload=211112]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:104.36.82.9:4005): protocol: write tcp 188.166.225.170:32954->104.36.82.9:4005: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:33.223177415 (202h57m27.654257536s) [seg_pos={0 0}, upload=211112]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Aug 17 04:23:33.448862924 (202h57m27.879943045s) [upload=211112]: Done uploading segments... Aug 17 04:23:33.448886830 (202h57m27.879966961s) [upload=211112]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 6.03555752s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [...................XXX......................................] duration: 5.793620116s FAIL (2025-08-17 04:50:12 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== g_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.761278730 (202h56m21.192358861s) [noise=true , address=104.36.82.34:4558 , piece_id=XGFNEBBVZX5UWJN7DALGM2EDG26POCTSBBAUU7DLYQKS2DNSBNXA, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.761661883 (202h56m21.192742014s) [noise=true , address=104.36.82.35:4594 , piece_id=6NUSPNYGSSJOB2CNPIW6DI45MPZARLWDBXH3CWMP4LXMDB2CXWZQ, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.761716548 (202h56m21.192796679s) [noise=true , address=45.91.116.8:4067 , piece_id=EZAI7ZDUBOOWXYJNTB7ZPOUSBUUG2FKV77QVUOZMNM6TUJSKGYWA, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.761972016 (202h56m21.193052147s) [noise=true , address=104.36.82.58:4283 , piece_id=LJKEH2KDSOFPVFPB5UWGVLOHQVZQOY33MYSHFLP4ZWGPKZ4KC64A, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.762254878 (202h56m21.193335009s) [noise=true , address=79.127.201.215:4345 , piece_id=JMSKNIZ7QPGBXIEP5RA3S74XAU5NJ2MUUXSFFIAK377YFGO646YQ, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.762523682 (202h56m21.193603803s) [noise=true , address=45.91.116.13:4181 , piece_id=7VN6RQGOTBHQO6HXM5IFTU32PWNVPH6BJZMQNETJEXOYPR2Y7OCQ, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.762864294 (202h56m21.193944415s) [noise=true , address=45.91.116.14:4203 , piece_id=5LXGQ4JZD2KLW3OMWY5QYRE4MKQNKKSAIMESODKDDT2W33XD45DQ, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.763179537 (202h56m21.194259668s) [noise=true , address=77.90.31.18:4815 , piece_id=G6PINAFFA6DZRRMKU2XK2LYFOES5WG7PTJ5L2ZPUQHMA4JNSRZRQ, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.763478499 (202h56m21.19455864s) [noise=true , address=77.90.31.11:4562 , piece_id=PUL6TAVIIU7732IOASNH5EHQMNCK7DWHJMUJVXRNK74T4GUQDGTA, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.763757683 (202h56m21.194837814s) [noise=true , address=104.36.82.5:4855 , piece_id=3FCDYA6CRYOOU3MDFJPEC4YQNICFVCN2A47RIYPSR4ZX5TIXZX3A, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.763988195 (202h56m21.195068326s) [noise=true , address=45.91.116.5:4308 , piece_id=F52M3VMHRQWR5LIEEPMVKVBFMOVG76UNNGQTFKSF6ZBD7VEOFWVA, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.764273320 (202h56m21.195353451s) [noise=true , address=66.23.193.175:4559 , piece_id=46X42PJIPA6DVC7BG7WPKMPMC44ZBUROEAB5C22M5INXKBJC4KVQ, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.764555100 (202h56m21.195635221s) [noise=true , address=77.90.31.15:4732 , piece_id=4AU7UGKXN4TIPSIRR7IMXSF6WHELO6UTOX35N5X7JHOQQXWWJF2A, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.764757416 (202h56m21.195837547s) [noise=true , address=104.36.82.39:4753 , piece_id=R7BHT5RYLSCDZLTDDRXTSJFSRPLM5PNBYC6Q2G33C5SYIUCNVCLQ, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.765868993 (202h56m21.196949124s) [noise=true , address=104.36.82.50:4047 , piece_id=AKDACFQEPBAFGYKTWBL5RELLZL3L4IJHJT73NS7T7QPWK2C2TDQA, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.767152809 (202h56m21.19823294s) [noise=true , address=104.36.82.19:4436 , piece_id=KVDJTKON4LFVJSAY7XSJ2OQ4ZH67JLMDE2O4EHUWSMZ3EECWSOXA, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.768577886 (202h56m21.199658017s) [noise=true , address=45.91.116.10:4095 , piece_id=JQ2UARQSNKBGHHHBJ6PR45ETR4EYMCXPDEQHEGWFYH52LRJC57UQ, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:27.934309011 (202h56m22.365389142s) [upload=211101]: Next returned nil segment Aug 17 04:22:27.934342545 (202h56m22.365422666s) [upload=211101]: Waiting for error group managing segments... Aug 17 04:22:28.231985083 (202h56m22.663065214s) [noise=true , address=66.23.193.175:4559 , piece_id=46X42PJIPA6DVC7BG7WPKMPMC44ZBUROEAB5C22M5INXKBJC4KVQ, seg_pos={0 0}, upload=211101]: Done uploading piece. err: <nil> Aug 17 04:22:28.670701256 (202h56m23.101781427s) [noise=true , address=45.91.116.10:4095 , piece_id=JQ2UARQSNKBGHHHBJ6PR45ETR4EYMCXPDEQHEGWFYH52LRJC57UQ, seg_pos={0 0}, upload=211101]: Done uploading piece. err: <nil> Aug 17 04:22:28.671952431 (202h56m23.103032562s) [noise=true , address=45.91.116.5:4308 , piece_id=F52M3VMHRQWR5LIEEPMVKVBFMOVG76UNNGQTFKSF6ZBD7VEOFWVA, seg_pos={0 0}, upload=211101]: Done uploading piece. err: <nil> Aug 17 04:22:28.698468278 (202h56m23.129548409s) [noise=true , address=45.91.116.8:4067 , piece_id=EZAI7ZDUBOOWXYJNTB7ZPOUSBUUG2FKV77QVUOZMNM6TUJSKGYWA, seg_pos={0 0}, upload=211101]: Done uploading piece. err: <nil> Aug 17 04:22:28.720171187 (202h56m23.151251318s) [noise=true , address=45.91.116.13:4181 , piece_id=7VN6RQGOTBHQO6HXM5IFTU32PWNVPH6BJZMQNETJEXOYPR2Y7OCQ, seg_pos={0 0}, upload=211101]: Done uploading piece. err: <nil> Aug 17 04:22:28.731591429 (202h56m23.16267156s) [noise=true , address=45.91.116.14:4203 , piece_id=5LXGQ4JZD2KLW3OMWY5QYRE4MKQNKKSAIMESODKDDT2W33XD45DQ, seg_pos={0 0}, upload=211101]: Done uploading piece. err: <nil> Aug 17 04:22:28.798148986 (202h56m23.229229117s) [noise=true , address=77.90.31.18:4815 , piece_id=G6PINAFFA6DZRRMKU2XK2LYFOES5WG7PTJ5L2ZPUQHMA4JNSRZRQ, seg_pos={0 0}, upload=211101]: Done uploading piece. err: <nil> Aug 17 04:22:28.804849037 (202h56m23.235929168s) [noise=true , address=79.127.201.215:4345 , piece_id=JMSKNIZ7QPGBXIEP5RA3S74XAU5NJ2MUUXSFFIAK377YFGO646YQ, seg_pos={0 0}, upload=211101]: Done uploading piece. err: <nil> Aug 17 04:22:28.833066822 (202h56m23.264147013s) [noise=true , address=77.90.31.15:4732 , piece_id=4AU7UGKXN4TIPSIRR7IMXSF6WHELO6UTOX35N5X7JHOQQXWWJF2A, seg_pos={0 0}, upload=211101]: Done uploading piece. err: <nil> Aug 17 04:22:28.846684710 (202h56m23.277764831s) [noise=true , address=77.90.31.11:4562 , piece_id=PUL6TAVIIU7732IOASNH5EHQMNCK7DWHJMUJVXRNK74T4GUQDGTA, seg_pos={0 0}, upload=211101]: Done uploading piece. err: <nil> Aug 17 04:23:25.582810947 (202h57m20.013891078s) [noise=true , address=104.36.82.34:4558 , piece_id=XGFNEBBVZX5UWJN7DALGM2EDG26POCTSBBAUU7DLYQKS2DNSBNXA, seg_pos={0 0}, upload=211101]: Done uploading piece. err: ecclient: upload failed (node:12TU4zvu38mF1fNSergCtMj2cK9eu57Rdfpt5MVk3tc4MZF6dNr, address:104.36.82.34:4558): protocol: rpc: dial tcp 104.36.82.34:4558: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:25.582856475 (202h57m20.013936606s) [noise=true , address=104.36.82.35:4594 , piece_id=6NUSPNYGSSJOB2CNPIW6DI45MPZARLWDBXH3CWMP4LXMDB2CXWZQ, seg_pos={0 0}, upload=211101]: Done uploading piece. err: ecclient: upload failed (node:12D9nh4XFmsjjFcZEMjaL4uw4Whsa8nfMrAsYPu6NVpsxgb3jvf, address:104.36.82.35:4594): protocol: rpc: dial tcp 104.36.82.35:4594: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:25.582973238 (202h57m20.014053369s) [noise=true , address=104.36.82.19:4436 , piece_id=KVDJTKON4LFVJSAY7XSJ2OQ4ZH67JLMDE2O4EHUWSMZ3EECWSOXA, seg_pos={0 0}, upload=211101]: Done uploading piece. err: ecclient: upload failed (node:1WvyEw8XevCiQ8yRgzyMo7WTVzpEJ158ErKfAeV4dSTBkS1oRF, address:104.36.82.19:4436): protocol: rpc: dial tcp 104.36.82.19:4436: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:25.583033393 (202h57m20.014113524s) [noise=true , address=104.36.82.58:4283 , piece_id=LJKEH2KDSOFPVFPB5UWGVLOHQVZQOY33MYSHFLP4ZWGPKZ4KC64A, seg_pos={0 0}, upload=211101]: Done uploading piece. err: ecclient: upload failed (node:1p9jkPtQnmbCXBmw6HSHDWpGaKWqfg87m6hHXy6WDu2giAFnbQ, address:104.36.82.58:4283): protocol: rpc: dial tcp 104.36.82.58:4283: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:25.583069241 (202h57m20.014149362s) [noise=true , address=104.36.82.50:4047 , piece_id=AKDACFQEPBAFGYKTWBL5RELLZL3L4IJHJT73NS7T7QPWK2C2TDQA, seg_pos={0 0}, upload=211101]: Done uploading piece. err: ecclient: upload failed (node:126Z6tvp49dLLdj3T5snJc4tBVGzcX5UBaGafFN3PkhF92vxpfs, address:104.36.82.50:4047): protocol: rpc: dial tcp 104.36.82.50:4047: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:25.583076716 (202h57m20.014156837s) [noise=true , address=104.36.82.39:4753 , piece_id=R7BHT5RYLSCDZLTDDRXTSJFSRPLM5PNBYC6Q2G33C5SYIUCNVCLQ, seg_pos={0 0}, upload=211101]: Done uploading piece. err: ecclient: upload failed (node:1zzW2z6noGQT6xdSmwSQBNYwip6gZjnmqZg7qdb6BP3nHyCLFz, address:104.36.82.39:4753): protocol: rpc: dial tcp 104.36.82.39:4753: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:25.583107535 (202h57m20.014187666s) [noise=true , address=104.36.82.23:4474 , piece_id=RITPHITVW655UQETT6UKS7BZO2S3F4P2MJIO4REOUKYYP4E2CDVQ, seg_pos={0 0}, upload=211101]: Done uploading piece. err: ecclient: upload failed (node:1Fpshye6yGAo3yf71eVpxrCdvoRjpkGX9WEX8KCWYbvuhy5i9j, address:104.36.82.23:4474): protocol: write tcp 188.166.225.170:36642->104.36.82.23:4474: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:25.583165355 (202h57m20.014245487s) [noise=true , address=104.36.82.5:4855 , piece_id=3FCDYA6CRYOOU3MDFJPEC4YQNICFVCN2A47RIYPSR4ZX5TIXZX3A, seg_pos={0 0}, upload=211101]: Done uploading piece. err: ecclient: upload failed (node:1KDNuE7YY8smhDjotushQvY3y9NtLxigbJELHD4Edxy8Qi4BWh, address:104.36.82.5:4855): protocol: write tcp 188.166.225.170:47052->104.36.82.5:4855: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:25.583194000 (202h57m20.014274131s) [seg_pos={0 0}, upload=211101]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Aug 17 04:23:25.810738626 (202h57m20.241818757s) [upload=211101]: Done uploading segments... Aug 17 04:23:25.810761991 (202h57m20.241842122s) [upload=211101]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 5.793676954s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [...................XXXX.....................................] duration: 6.167486193s FAIL (2025-08-17 04:50:09 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.167510017s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 9 out of 60 is failed (threshold=3) [.............X..........XXXX.X.X.......X...X................] duration: 5.66212222s FAIL (2025-08-17 04:49:48 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/4M") ======LOGS====== Aug 17 04:37:54.182428668 (203h11m48.613508799s) [upload=211361]: Starting upload Aug 17 04:37:54.182474656 (203h11m48.613554777s) [upload=211361]: Done starting upload Aug 17 04:37:54.182665000 (203h11m48.613745131s) [upload=211361]: Uploading segments... Aug 17 04:37:54.182728421 (203h11m48.613808552s) [upload=211361]: Got next segment. Inline: false Aug 17 04:37:54.182758048 (203h11m48.613838179s) [upload=211361]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 17 04:37:54.182762306 (203h11m48.613842437s) [upload=211361]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 17 04:37:54.737160222 (203h11m49.168240353s) [seg_pos={0 0}, upload=211361]: Begin upload segment... Aug 17 04:37:54.737203494 (203h11m49.168283625s) [seg_pos={0 0}, upload=211361]: Done begin upload segment. Aug 17 04:37:54.737224945 (203h11m49.168305076s) [noise=true , address=77.90.31.19:4863 , piece_id=PYERWDVDOTOW2SVDUO63ODTEWMWUGH5OMV2PE7RJTIDVFTF2ZRVQ, seg_pos={0 0}, upload=211361]: Uploading piece... Aug 17 04:37:54.737311591 (203h11m49.168391722s) [noise=true , address=79.127.220.101:4068 , piece_id=6MFJ22JL437NTOH25E3LNH3Q2LQNKKMUT5Q43F2XNIB7346LGFCQ, seg_pos={0 0}, upload=211361]: Uploading piece... Aug 17 04:37:54.737638917 (203h11m49.168719048s) [noise=true , address=77.90.31.12:4601 , piece_id=JC5VIGLKCWTCDHQYWEL6QSRZW3MJZFRFQKRJH54OK7LPUQH7WQTQ, seg_pos={0 0}, upload=211361]: Uploading piece... Aug 17 04:37:54.737939432 (203h11m49.169019563s) [noise=true , address=66.23.193.175:4562 , piece_id=2RZ2A5KBLZBLEG6UECY2GTNF5THR3XJZMS34ULG5SQNWN4O3XNMA, seg_pos={0 0}, upload=211361]: Uploading piece... Aug 17 04:37:54.737952067 (203h11m49.169032198s) [noise=true , address=77.90.31.16:4761 , piece_id=44YO7GYP4RNRWHPP7LP2C4FJBGMWOLEDEC2NN7TAWU3YFHAST3XQ, seg_pos={0 0}, upload=211361]: Uploading piece... Aug 17 04:37:54.738229999 (203h11m49.16931013s) [noise=true , address=104.36.82.36:4648 , piece_id=GTHXHUS6GR26G6UHHMB35MCSEI7I5UJ6K65O76HRJI4WVPIHD7RQ, seg_pos={0 0}, upload=211361]: Uploading piece... Aug 17 04:37:54.738324038 (203h11m49.169404169s) [noise=true , address=66.23.193.175:4563 , piece_id=I65W2GP5RTRZUIT7Y6NSEAL7XC726PP3BYMCWS6UMZEOWAP5CP2A, seg_pos={0 0}, upload=211361]: Uploading piece... Aug 17 04:37:54.738511367 (203h11m49.169591488s) [noise=true , address=104.36.82.4:4833 , piece_id=AMRE5SIA7EJLAP55GFYILJYZJL5624E534TPWFUEQILHG7CCFDXQ, seg_pos={0 0}, upload=211361]: Uploading piece... Aug 17 04:37:54.738655813 (203h11m49.169735954s) [noise=true , address=104.36.82.5:4844 , piece_id=PULFFEXZLSMOYEFGRK6L2POF7ERJQT75J23JT3CTXKZ3LMARQZHQ, seg_pos={0 0}, upload=211361]: Uploading piece... Aug 17 04:37:54.738747769 (203h11m49.1698279s) [noise=true , address=77.90.31.10:4532 , piece_id=5IBCLJNM6LUBCIOPSAERY5HAXV66CR4OKI45RBGOHALTRFYGLMWQ, seg_pos={0 0}, upload=211361]: Uploading piece... Aug 17 04:37:54.738971367 (203h11m49.170051498s) [noise=true , address=104.36.82.60:4360 , piece_id=W4OJU6NRV2HKPC7JDTISAPVT77KIC3JHVJR5KHYY225FLH4PZBBA, seg_pos={0 0}, upload=211361]: Uploading piece... Aug 17 04:37:54.738975705 (203h11m49.170055836s) [noise=true , address=77.90.31.18:4830 , piece_id=QNQF4JAEHOFWRTL6XWKL42XSKA2DMO6Z4CZO5NF337NNSHCMYYBA, seg_pos={0 0}, upload=211361]: Uploading piece... Aug 17 04:37:54.739194995 (203h11m49.170275126s) [noise=true , address=104.36.82.23:4468 , piece_id=NC2BFOASSHA4ZG46LZONWF23DNN3ZZBA6SFO3Q44KD4TCOH4AIQQ, seg_pos={0 0}, upload=211361]: Uploading piece... Aug 17 04:37:54.739278324 (203h11m49.170358445s) [noise=true , address=104.36.82.39:4747 , piece_id=SF2P47ED5FEOCMJ6R6AXP2NTI5LYYALSDLHINNLDVMUMZIE4YLHQ, seg_pos={0 0}, upload=211361]: Uploading piece... Aug 17 04:37:54.739421147 (203h11m49.170501278s) [noise=true , address=104.36.82.3:4800 , piece_id=EKC43HOMYNG77PDESU3QV5OL44HWYQRAN5RUNIB7IMC6NVXQWFVA, seg_pos={0 0}, upload=211361]: Uploading piece... Aug 17 04:37:54.739561055 (203h11m49.170641186s) [noise=true , address=77.90.31.13:4634 , piece_id=ZXX5E54TSYUG557OGOCBQMBKS35TUG6OPDOIG3TTS3OYGTRA24XA, seg_pos={0 0}, upload=211361]: Uploading piece... Aug 17 04:37:54.739643824 (203h11m49.170723955s) [noise=true , address=104.36.82.58:4286 , piece_id=JDBFFYOZPYJ5FXJQIVEBB5IWSBGT6LCBUZF5O25OKWFRXROPHS3Q, seg_pos={0 0}, upload=211361]: Uploading piece... Aug 17 04:37:54.739852683 (203h11m49.170932814s) [noise=true , address=104.36.82.44:4876 , piece_id=H53T7WMGX4CDVQNCYOG6NOYFVRHD5Z35LNF37FVKSWYIVJS2EFZA, seg_pos={0 0}, upload=211361]: Uploading piece... Aug 17 04:37:55.861243533 (203h11m50.292323664s) [upload=211361]: Next returned nil segment Aug 17 04:37:55.861256768 (203h11m50.292336899s) [upload=211361]: Waiting for error group managing segments... Aug 17 04:37:56.165335502 (203h11m50.596415633s) [noise=true , address=66.23.193.175:4562 , piece_id=2RZ2A5KBLZBLEG6UECY2GTNF5THR3XJZMS34ULG5SQNWN4O3XNMA, seg_pos={0 0}, upload=211361]: Done uploading piece. err: <nil> Aug 17 04:37:56.263548583 (203h11m50.694628715s) [noise=true , address=66.23.193.175:4563 , piece_id=I65W2GP5RTRZUIT7Y6NSEAL7XC726PP3BYMCWS6UMZEOWAP5CP2A, seg_pos={0 0}, upload=211361]: Done uploading piece. err: <nil> Aug 17 04:37:56.569392606 (203h11m51.000472737s) [noise=true , address=79.127.220.101:4068 , piece_id=6MFJ22JL437NTOH25E3LNH3Q2LQNKKMUT5Q43F2XNIB7346LGFCQ, seg_pos={0 0}, upload=211361]: Done uploading piece. err: <nil> Aug 17 04:37:56.620901052 (203h11m51.051981183s) [noise=true , address=77.90.31.12:4601 , piece_id=JC5VIGLKCWTCDHQYWEL6QSRZW3MJZFRFQKRJH54OK7LPUQH7WQTQ, seg_pos={0 0}, upload=211361]: Done uploading piece. err: <nil> Aug 17 04:37:56.639713969 (203h11m51.0707941s) [noise=true , address=77.90.31.16:4761 , piece_id=44YO7GYP4RNRWHPP7LP2C4FJBGMWOLEDEC2NN7TAWU3YFHAST3XQ, seg_pos={0 0}, upload=211361]: Done uploading piece. err: <nil> Aug 17 04:37:56.649068307 (203h11m51.080148438s) [noise=true , address=104.36.82.4:4833 , piece_id=AMRE5SIA7EJLAP55GFYILJYZJL5624E534TPWFUEQILHG7CCFDXQ, seg_pos={0 0}, upload=211361]: Done uploading piece. err: <nil> Aug 17 04:37:56.701677831 (203h11m51.132757962s) [noise=true , address=104.36.82.3:4800 , piece_id=EKC43HOMYNG77PDESU3QV5OL44HWYQRAN5RUNIB7IMC6NVXQWFVA, seg_pos={0 0}, upload=211361]: Done uploading piece. err: <nil> Aug 17 04:37:56.719713991 (203h11m51.150794132s) [noise=true , address=104.36.82.23:4468 , piece_id=NC2BFOASSHA4ZG46LZONWF23DNN3ZZBA6SFO3Q44KD4TCOH4AIQQ, seg_pos={0 0}, upload=211361]: Done uploading piece. err: <nil> Aug 17 04:37:56.730632311 (203h11m51.161712432s) [noise=true , address=104.36.82.39:4747 , piece_id=SF2P47ED5FEOCMJ6R6AXP2NTI5LYYALSDLHINNLDVMUMZIE4YLHQ, seg_pos={0 0}, upload=211361]: Done uploading piece. err: <nil> Aug 17 04:37:56.761149974 (203h11m51.192230105s) [noise=true , address=104.36.82.60:4360 , piece_id=W4OJU6NRV2HKPC7JDTISAPVT77KIC3JHVJR5KHYY225FLH4PZBBA, seg_pos={0 0}, upload=211361]: Done uploading piece. err: <nil> Aug 17 04:37:56.775738719 (203h11m51.20681885s) [noise=true , address=77.90.31.18:4830 , piece_id=QNQF4JAEHOFWRTL6XWKL42XSKA2DMO6Z4CZO5NF337NNSHCMYYBA, seg_pos={0 0}, upload=211361]: Done uploading piece. err: <nil> Aug 17 04:37:56.779316274 (203h11m51.210396405s) [noise=true , address=77.90.31.13:4634 , piece_id=ZXX5E54TSYUG557OGOCBQMBKS35TUG6OPDOIG3TTS3OYGTRA24XA, seg_pos={0 0}, upload=211361]: Done uploading piece. err: <nil> Aug 17 04:37:56.802147800 (203h11m51.233227961s) [noise=true , address=104.36.82.58:4286 , piece_id=JDBFFYOZPYJ5FXJQIVEBB5IWSBGT6LCBUZF5O25OKWFRXROPHS3Q, seg_pos={0 0}, upload=211361]: Done uploading piece. err: <nil> Aug 17 04:37:56.832614886 (203h11m51.263695017s) [noise=true , address=104.36.82.44:4876 , piece_id=H53T7WMGX4CDVQNCYOG6NOYFVRHD5Z35LNF37FVKSWYIVJS2EFZA, seg_pos={0 0}, upload=211361]: Done uploading piece. err: <nil> Aug 17 04:37:56.832646346 (203h11m51.263726467s) [seg_pos={0 0}, upload=211361]: Segment reached optimal threshold of 14 pieces. Aug 17 04:37:56.832765443 (203h11m51.263845574s) [noise=true , address=77.90.31.10:4532 , piece_id=5IBCLJNM6LUBCIOPSAERY5HAXV66CR4OKI45RBGOHALTRFYGLMWQ, seg_pos={0 0}, upload=211361]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1T6kkKvTNR5MZZijjr1vMU6g75YkoRpzMvGSgQCqRnfCm9JULe): protocol: expected piece hash; context canceled; EOF Aug 17 04:37:56.832872889 (203h11m51.26395302s) [noise=true , address=104.36.82.36:4648 , piece_id=GTHXHUS6GR26G6UHHMB35MCSEI7I5UJ6K65O76HRJI4WVPIHD7RQ, seg_pos={0 0}, upload=211361]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XHNZDY7xKZ2bss1rSdjbGP3ugSwGRwRzBSWHfXgdKKz794wMq): protocol: expected piece hash; context canceled; EOF Aug 17 04:37:56.832892156 (203h11m51.263972287s) [noise=true , address=77.90.31.19:4863 , piece_id=PYERWDVDOTOW2SVDUO63ODTEWMWUGH5OMV2PE7RJTIDVFTF2ZRVQ, seg_pos={0 0}, upload=211361]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124x1cbCPJRK8YkgSP6UzHymoLu1E1znnfg5WBiD89kSDLi8JeL): protocol: expected piece hash; context canceled; EOF Aug 17 04:37:56.832974564 (203h11m51.264054695s) [noise=true , address=104.36.82.5:4844 , piece_id=PULFFEXZLSMOYEFGRK6L2POF7ERJQT75J23JT3CTXKZ3LMARQZHQ, seg_pos={0 0}, upload=211361]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12iovpNdbidkiv9mEHaAPjBubGmoqr4sUJWm8Cb64LzS6wuwLWa): protocol: write tcp 188.166.225.170:59826->104.36.82.5:4844: use of closed network connection; write tcp 188.166.225.170:59826->104.36.82.5:4844: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59826->104.36.82.5:4844: use of closed network connection Aug 17 04:37:56.833025241 (203h11m51.264105372s) [seg_pos={0 0}, upload=211361]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 17 04:37:56.833116014 (203h11m51.264196135s) [upload=211361]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 17 04:37:56.833120332 (203h11m51.264200463s) [upload=211361]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 17 04:37:58.261163702 (203h11m52.692243834s) [upload=211361]: Done uploading segments... Aug 17 04:37:58.261181997 (203h11m52.692262118s) [upload=211361]: Upload finished. err: <nil> Duration: 5.662150472s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [...................XXXX.....................................] duration: 5.862486023s FAIL (2025-08-17 04:49:47 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== Aug 17 04:23:03.873697274 (202h56m58.304777405s) [upload=211112]: Starting upload Aug 17 04:23:03.873756368 (202h56m58.304836499s) [upload=211112]: Done starting upload Aug 17 04:23:03.873839848 (202h56m58.304919989s) [upload=211112]: Uploading segments... Aug 17 04:23:03.873950850 (202h56m58.305030981s) [upload=211112]: Got next segment. Inline: false Aug 17 04:23:03.873976509 (202h56m58.30505664s) [upload=211112]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 17 04:23:03.873980757 (202h56m58.305060888s) [upload=211112]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 17 04:23:04.113727713 (202h56m58.544807844s) [seg_pos={0 0}, upload=211112]: Begin upload segment... Aug 17 04:23:04.113774051 (202h56m58.544854182s) [seg_pos={0 0}, upload=211112]: Done begin upload segment. Aug 17 04:23:04.113820890 (202h56m58.544901021s) [noise=true , address=45.91.116.13:4155 , piece_id=CO3R5XZGH3NJADAFDILPNZEKKFJSRPB4JOFBBCSYYQAD3KNRABNA, seg_pos={0 0}, upload=211112]: Uploading piece... Aug 17 04:23:04.113845989 (202h56m58.54492612s) [noise=true , address=45.91.116.7:4009 , piece_id=IVTDHCTNJEEO3QATGN25YH5Y2MSEQUWJIFPXSJWINUEXDHQAGELA, seg_pos={0 0}, upload=211112]: Uploading piece... Aug 17 04:23:04.114195037 (202h56m58.545275168s) [noise=true , address=45.91.116.4:4938 , piece_id=X5MDNBZYRYYOXDEAMGIOD5CXX7PGAMTFRMROOVTR6NIAYQ5VMXGA, seg_pos={0 0}, upload=211112]: Uploading piece... Aug 17 04:23:04.114281412 (202h56m58.545361543s) [noise=true , address=45.91.116.9:4261 , piece_id=PNOEBFIPLWM4B2QLGJUVEBTT23J5AVRFTWTTKLHFUJ5MNJTSM2JA, seg_pos={0 0}, upload=211112]: Uploading piece... Aug 17 04:23:04.114482115 (202h56m58.545562237s) [noise=true , address=77.90.31.16:4735 , piece_id=G5YGF7LXNPBU2TDRYPAFPVYQSTWUGG7Y6YM2Q6RWMCZY5RVNM4BA, seg_pos={0 0}, upload=211112]: Uploading piece... Aug 17 04:23:04.114570385 (202h56m58.545650516s) [noise=true , address=104.36.82.9:4005 , piece_id=FAR4B5KAALBOACKANEFKGECI2ZVXSHBNYDEIWVGDU2RNBNQXLMXA, seg_pos={0 0}, upload=211112]: Uploading piece... Aug 17 04:23:04.114757392 (202h56m58.545837523s) [noise=true , address=45.91.116.9:4263 , piece_id=VT734KJZ7T3CZ3W3P4M4ZCYKOLLTPP3ILFRWL26Z7K4LWBDZV4FA, seg_pos={0 0}, upload=211112]: Uploading piece... Aug 17 04:23:04.114874727 (202h56m58.545954858s) [noise=true , address=104.36.82.12:4151 , piece_id=2GGYMOWZ2FTWDNWNQRKSAH7UVJ4R6MLRGC6OHNVV3KTVIIMO4QIQ, seg_pos={0 0}, upload=211112]: Uploading piece... Aug 17 04:23:04.115030715 (202h56m58.546110846s) [noise=true , address=45.91.116.12:4225 , piece_id=EUPPM5NRIU5MCZBER3VGZBZEMZZ65Q2MOY2DXKFBIPFJ53M5TJHQ, seg_pos={0 0}, upload=211112]: Uploading piece... Aug 17 04:23:04.115194288 (202h56m58.546274419s) [noise=true , address=104.36.82.4:4804 , piece_id=MSAHN2DERVG2HL7KZMKIWYUQYO6LRIXGUNWJFXJDDGOWZT54FPVA, seg_pos={0 0}, upload=211112]: Uploading piece... Aug 17 04:23:04.115307967 (202h56m58.546388098s) [noise=true , address=104.36.82.5:4838 , piece_id=IYA7ERDAAFYUGC4666CBEX5HZEXZVFPES6JUYRZROY25OAPGVEQQ, seg_pos={0 0}, upload=211112]: Uploading piece... Aug 17 04:23:04.115535692 (202h56m58.546615823s) [noise=true , address=45.91.116.12:4227 , piece_id=XZR3AWJG57RUB4TN7SUWNIWWD2XGKCMDXSWSM74WS3ATHS32EBJQ, seg_pos={0 0}, upload=211112]: Uploading piece... Aug 17 04:23:04.115754130 (202h56m58.546834261s) [noise=true , address=45.91.116.7:4010 , piece_id=HNJU4MPWT67V26J5XDCTHFEOOJL2HS3VAOHB4EIPQVF4D2NNRFMA, seg_pos={0 0}, upload=211112]: Uploading piece... Aug 17 04:23:04.116079182 (202h56m58.547159323s) [noise=true , address=45.91.116.14:4189 , piece_id=2P3GOZDEMJGBAK2F6SY7NJNQUTS5ZYM4B4IMQFAA5TYFARD4DVCQ, seg_pos={0 0}, upload=211112]: Uploading piece... Aug 17 04:23:04.116350181 (202h56m58.547430312s) [noise=true , address=104.36.82.15:4257 , piece_id=Q5GLP6TS7N7CM4DO3LWDCDR6O76OFZ6IVQJC5VQWMA5XLNWZ2E4A, seg_pos={0 0}, upload=211112]: Uploading piece... Aug 17 04:23:04.116554471 (202h56m58.547634602s) [noise=true , address=104.36.82.8:4969 , piece_id=3D3YAZSP7CD6IFZGDKPNKIJPWZ2OJFFVJCSMTO42VNJEFVU3ORHA, seg_pos={0 0}, upload=211112]: Uploading piece... Aug 17 04:23:05.868765063 (202h57m0.299845195s) [upload=211112]: Next returned nil segment Aug 17 04:23:05.868801043 (202h57m0.299881164s) [upload=211112]: Waiting for error group managing segments... Aug 17 04:23:06.147430529 (202h57m0.57851066s) [noise=true , address=45.91.116.12:4227 , piece_id=XZR3AWJG57RUB4TN7SUWNIWWD2XGKCMDXSWSM74WS3ATHS32EBJQ, seg_pos={0 0}, upload=211112]: Done uploading piece. err: <nil> Aug 17 04:23:06.182400314 (202h57m0.613480445s) [noise=true , address=45.91.116.12:4225 , piece_id=EUPPM5NRIU5MCZBER3VGZBZEMZZ65Q2MOY2DXKFBIPFJ53M5TJHQ, seg_pos={0 0}, upload=211112]: Done uploading piece. err: <nil> Aug 17 04:23:06.191910532 (202h57m0.622990663s) [noise=true , address=45.91.116.4:4938 , piece_id=X5MDNBZYRYYOXDEAMGIOD5CXX7PGAMTFRMROOVTR6NIAYQ5VMXGA, seg_pos={0 0}, upload=211112]: Done uploading piece. err: <nil> Aug 17 04:23:06.197391631 (202h57m0.628471782s) [noise=true , address=45.91.116.9:4261 , piece_id=PNOEBFIPLWM4B2QLGJUVEBTT23J5AVRFTWTTKLHFUJ5MNJTSM2JA, seg_pos={0 0}, upload=211112]: Done uploading piece. err: <nil> Aug 17 04:23:06.198069487 (202h57m0.629149618s) [noise=true , address=45.91.116.9:4263 , piece_id=VT734KJZ7T3CZ3W3P4M4ZCYKOLLTPP3ILFRWL26Z7K4LWBDZV4FA, seg_pos={0 0}, upload=211112]: Done uploading piece. err: <nil> Aug 17 04:23:06.200597014 (202h57m0.631677145s) [noise=true , address=45.91.116.7:4009 , piece_id=IVTDHCTNJEEO3QATGN25YH5Y2MSEQUWJIFPXSJWINUEXDHQAGELA, seg_pos={0 0}, upload=211112]: Done uploading piece. err: <nil> Aug 17 04:23:06.288053251 (202h57m0.719133372s) [noise=true , address=45.91.116.7:4010 , piece_id=HNJU4MPWT67V26J5XDCTHFEOOJL2HS3VAOHB4EIPQVF4D2NNRFMA, seg_pos={0 0}, upload=211112]: Done uploading piece. err: <nil> Aug 17 04:23:06.298810435 (202h57m0.729890566s) [noise=true , address=45.91.116.14:4189 , piece_id=2P3GOZDEMJGBAK2F6SY7NJNQUTS5ZYM4B4IMQFAA5TYFARD4DVCQ, seg_pos={0 0}, upload=211112]: Done uploading piece. err: <nil> Aug 17 04:23:06.301049520 (202h57m0.732129651s) [noise=true , address=45.91.116.13:4155 , piece_id=CO3R5XZGH3NJADAFDILPNZEKKFJSRPB4JOFBBCSYYQAD3KNRABNA, seg_pos={0 0}, upload=211112]: Done uploading piece. err: <nil> Aug 17 04:23:06.402776985 (202h57m0.833857116s) [noise=true , address=77.90.31.16:4735 , piece_id=G5YGF7LXNPBU2TDRYPAFPVYQSTWUGG7Y6YM2Q6RWMCZY5RVNM4BA, seg_pos={0 0}, upload=211112]: Done uploading piece. err: <nil> Aug 17 04:23:33.223003412 (202h57m27.654083543s) [noise=true , address=104.36.82.5:4838 , piece_id=IYA7ERDAAFYUGC4666CBEX5HZEXZVFPES6JUYRZROY25OAPGVEQQ, seg_pos={0 0}, upload=211112]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:104.36.82.5:4838): protocol: write tcp 188.166.225.170:42258->104.36.82.5:4838: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:33.223050702 (202h57m27.654130823s) [noise=true , address=104.36.82.8:4969 , piece_id=3D3YAZSP7CD6IFZGDKPNKIJPWZ2OJFFVJCSMTO42VNJEFVU3ORHA, seg_pos={0 0}, upload=211112]: Done uploading piece. err: ecclient: upload failed (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, address:104.36.82.8:4969): protocol: write tcp 188.166.225.170:52308->104.36.82.8:4969: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:33.223090619 (202h57m27.65417075s) [noise=true , address=104.36.82.15:4257 , piece_id=Q5GLP6TS7N7CM4DO3LWDCDR6O76OFZ6IVQJC5VQWMA5XLNWZ2E4A, seg_pos={0 0}, upload=211112]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:104.36.82.15:4257): protocol: write tcp 188.166.225.170:53078->104.36.82.15:4257: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:33.223104976 (202h57m27.654185097s) [noise=true , address=104.36.82.4:4804 , piece_id=MSAHN2DERVG2HL7KZMKIWYUQYO6LRIXGUNWJFXJDDGOWZT54FPVA, seg_pos={0 0}, upload=211112]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:104.36.82.4:4804): protocol: write tcp 188.166.225.170:56176->104.36.82.4:4804: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:33.223149581 (202h57m27.654229712s) [noise=true , address=104.36.82.12:4151 , piece_id=2GGYMOWZ2FTWDNWNQRKSAH7UVJ4R6MLRGC6OHNVV3KTVIIMO4QIQ, seg_pos={0 0}, upload=211112]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:104.36.82.12:4151): protocol: write tcp 188.166.225.170:40582->104.36.82.12:4151: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:33.223157536 (202h57m27.654237667s) [noise=true , address=104.36.82.9:4005 , piece_id=FAR4B5KAALBOACKANEFKGECI2ZVXSHBNYDEIWVGDU2RNBNQXLMXA, seg_pos={0 0}, upload=211112]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:104.36.82.9:4005): protocol: write tcp 188.166.225.170:32954->104.36.82.9:4005: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:33.223177415 (202h57m27.654257536s) [seg_pos={0 0}, upload=211112]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Aug 17 04:23:33.448862924 (202h57m27.879943045s) [upload=211112]: Done uploading segments... Aug 17 04:23:33.448886830 (202h57m27.879966961s) [upload=211112]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 5.862516292s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [....................XXXX....................................] duration: 6.098171584s FAIL (2025-08-17 04:49:45 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.098188226s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [........................XX..................................] duration: 6.778445642s FAIL (2025-08-17 04:49:40 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.778487173s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [....................XXX.....................................] duration: 5.682682257s FAIL (2025-08-17 04:49:38 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== g_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.761278730 (202h56m21.192358861s) [noise=true , address=104.36.82.34:4558 , piece_id=XGFNEBBVZX5UWJN7DALGM2EDG26POCTSBBAUU7DLYQKS2DNSBNXA, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.761661883 (202h56m21.192742014s) [noise=true , address=104.36.82.35:4594 , piece_id=6NUSPNYGSSJOB2CNPIW6DI45MPZARLWDBXH3CWMP4LXMDB2CXWZQ, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.761716548 (202h56m21.192796679s) [noise=true , address=45.91.116.8:4067 , piece_id=EZAI7ZDUBOOWXYJNTB7ZPOUSBUUG2FKV77QVUOZMNM6TUJSKGYWA, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.761972016 (202h56m21.193052147s) [noise=true , address=104.36.82.58:4283 , piece_id=LJKEH2KDSOFPVFPB5UWGVLOHQVZQOY33MYSHFLP4ZWGPKZ4KC64A, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.762254878 (202h56m21.193335009s) [noise=true , address=79.127.201.215:4345 , piece_id=JMSKNIZ7QPGBXIEP5RA3S74XAU5NJ2MUUXSFFIAK377YFGO646YQ, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.762523682 (202h56m21.193603803s) [noise=true , address=45.91.116.13:4181 , piece_id=7VN6RQGOTBHQO6HXM5IFTU32PWNVPH6BJZMQNETJEXOYPR2Y7OCQ, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.762864294 (202h56m21.193944415s) [noise=true , address=45.91.116.14:4203 , piece_id=5LXGQ4JZD2KLW3OMWY5QYRE4MKQNKKSAIMESODKDDT2W33XD45DQ, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.763179537 (202h56m21.194259668s) [noise=true , address=77.90.31.18:4815 , piece_id=G6PINAFFA6DZRRMKU2XK2LYFOES5WG7PTJ5L2ZPUQHMA4JNSRZRQ, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.763478499 (202h56m21.19455864s) [noise=true , address=77.90.31.11:4562 , piece_id=PUL6TAVIIU7732IOASNH5EHQMNCK7DWHJMUJVXRNK74T4GUQDGTA, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.763757683 (202h56m21.194837814s) [noise=true , address=104.36.82.5:4855 , piece_id=3FCDYA6CRYOOU3MDFJPEC4YQNICFVCN2A47RIYPSR4ZX5TIXZX3A, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.763988195 (202h56m21.195068326s) [noise=true , address=45.91.116.5:4308 , piece_id=F52M3VMHRQWR5LIEEPMVKVBFMOVG76UNNGQTFKSF6ZBD7VEOFWVA, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.764273320 (202h56m21.195353451s) [noise=true , address=66.23.193.175:4559 , piece_id=46X42PJIPA6DVC7BG7WPKMPMC44ZBUROEAB5C22M5INXKBJC4KVQ, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.764555100 (202h56m21.195635221s) [noise=true , address=77.90.31.15:4732 , piece_id=4AU7UGKXN4TIPSIRR7IMXSF6WHELO6UTOX35N5X7JHOQQXWWJF2A, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.764757416 (202h56m21.195837547s) [noise=true , address=104.36.82.39:4753 , piece_id=R7BHT5RYLSCDZLTDDRXTSJFSRPLM5PNBYC6Q2G33C5SYIUCNVCLQ, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.765868993 (202h56m21.196949124s) [noise=true , address=104.36.82.50:4047 , piece_id=AKDACFQEPBAFGYKTWBL5RELLZL3L4IJHJT73NS7T7QPWK2C2TDQA, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.767152809 (202h56m21.19823294s) [noise=true , address=104.36.82.19:4436 , piece_id=KVDJTKON4LFVJSAY7XSJ2OQ4ZH67JLMDE2O4EHUWSMZ3EECWSOXA, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.768577886 (202h56m21.199658017s) [noise=true , address=45.91.116.10:4095 , piece_id=JQ2UARQSNKBGHHHBJ6PR45ETR4EYMCXPDEQHEGWFYH52LRJC57UQ, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:27.934309011 (202h56m22.365389142s) [upload=211101]: Next returned nil segment Aug 17 04:22:27.934342545 (202h56m22.365422666s) [upload=211101]: Waiting for error group managing segments... Aug 17 04:22:28.231985083 (202h56m22.663065214s) [noise=true , address=66.23.193.175:4559 , piece_id=46X42PJIPA6DVC7BG7WPKMPMC44ZBUROEAB5C22M5INXKBJC4KVQ, seg_pos={0 0}, upload=211101]: Done uploading piece. err: <nil> Aug 17 04:22:28.670701256 (202h56m23.101781427s) [noise=true , address=45.91.116.10:4095 , piece_id=JQ2UARQSNKBGHHHBJ6PR45ETR4EYMCXPDEQHEGWFYH52LRJC57UQ, seg_pos={0 0}, upload=211101]: Done uploading piece. err: <nil> Aug 17 04:22:28.671952431 (202h56m23.103032562s) [noise=true , address=45.91.116.5:4308 , piece_id=F52M3VMHRQWR5LIEEPMVKVBFMOVG76UNNGQTFKSF6ZBD7VEOFWVA, seg_pos={0 0}, upload=211101]: Done uploading piece. err: <nil> Aug 17 04:22:28.698468278 (202h56m23.129548409s) [noise=true , address=45.91.116.8:4067 , piece_id=EZAI7ZDUBOOWXYJNTB7ZPOUSBUUG2FKV77QVUOZMNM6TUJSKGYWA, seg_pos={0 0}, upload=211101]: Done uploading piece. err: <nil> Aug 17 04:22:28.720171187 (202h56m23.151251318s) [noise=true , address=45.91.116.13:4181 , piece_id=7VN6RQGOTBHQO6HXM5IFTU32PWNVPH6BJZMQNETJEXOYPR2Y7OCQ, seg_pos={0 0}, upload=211101]: Done uploading piece. err: <nil> Aug 17 04:22:28.731591429 (202h56m23.16267156s) [noise=true , address=45.91.116.14:4203 , piece_id=5LXGQ4JZD2KLW3OMWY5QYRE4MKQNKKSAIMESODKDDT2W33XD45DQ, seg_pos={0 0}, upload=211101]: Done uploading piece. err: <nil> Aug 17 04:22:28.798148986 (202h56m23.229229117s) [noise=true , address=77.90.31.18:4815 , piece_id=G6PINAFFA6DZRRMKU2XK2LYFOES5WG7PTJ5L2ZPUQHMA4JNSRZRQ, seg_pos={0 0}, upload=211101]: Done uploading piece. err: <nil> Aug 17 04:22:28.804849037 (202h56m23.235929168s) [noise=true , address=79.127.201.215:4345 , piece_id=JMSKNIZ7QPGBXIEP5RA3S74XAU5NJ2MUUXSFFIAK377YFGO646YQ, seg_pos={0 0}, upload=211101]: Done uploading piece. err: <nil> Aug 17 04:22:28.833066822 (202h56m23.264147013s) [noise=true , address=77.90.31.15:4732 , piece_id=4AU7UGKXN4TIPSIRR7IMXSF6WHELO6UTOX35N5X7JHOQQXWWJF2A, seg_pos={0 0}, upload=211101]: Done uploading piece. err: <nil> Aug 17 04:22:28.846684710 (202h56m23.277764831s) [noise=true , address=77.90.31.11:4562 , piece_id=PUL6TAVIIU7732IOASNH5EHQMNCK7DWHJMUJVXRNK74T4GUQDGTA, seg_pos={0 0}, upload=211101]: Done uploading piece. err: <nil> Aug 17 04:23:25.582810947 (202h57m20.013891078s) [noise=true , address=104.36.82.34:4558 , piece_id=XGFNEBBVZX5UWJN7DALGM2EDG26POCTSBBAUU7DLYQKS2DNSBNXA, seg_pos={0 0}, upload=211101]: Done uploading piece. err: ecclient: upload failed (node:12TU4zvu38mF1fNSergCtMj2cK9eu57Rdfpt5MVk3tc4MZF6dNr, address:104.36.82.34:4558): protocol: rpc: dial tcp 104.36.82.34:4558: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:25.582856475 (202h57m20.013936606s) [noise=true , address=104.36.82.35:4594 , piece_id=6NUSPNYGSSJOB2CNPIW6DI45MPZARLWDBXH3CWMP4LXMDB2CXWZQ, seg_pos={0 0}, upload=211101]: Done uploading piece. err: ecclient: upload failed (node:12D9nh4XFmsjjFcZEMjaL4uw4Whsa8nfMrAsYPu6NVpsxgb3jvf, address:104.36.82.35:4594): protocol: rpc: dial tcp 104.36.82.35:4594: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:25.582973238 (202h57m20.014053369s) [noise=true , address=104.36.82.19:4436 , piece_id=KVDJTKON4LFVJSAY7XSJ2OQ4ZH67JLMDE2O4EHUWSMZ3EECWSOXA, seg_pos={0 0}, upload=211101]: Done uploading piece. err: ecclient: upload failed (node:1WvyEw8XevCiQ8yRgzyMo7WTVzpEJ158ErKfAeV4dSTBkS1oRF, address:104.36.82.19:4436): protocol: rpc: dial tcp 104.36.82.19:4436: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:25.583033393 (202h57m20.014113524s) [noise=true , address=104.36.82.58:4283 , piece_id=LJKEH2KDSOFPVFPB5UWGVLOHQVZQOY33MYSHFLP4ZWGPKZ4KC64A, seg_pos={0 0}, upload=211101]: Done uploading piece. err: ecclient: upload failed (node:1p9jkPtQnmbCXBmw6HSHDWpGaKWqfg87m6hHXy6WDu2giAFnbQ, address:104.36.82.58:4283): protocol: rpc: dial tcp 104.36.82.58:4283: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:25.583069241 (202h57m20.014149362s) [noise=true , address=104.36.82.50:4047 , piece_id=AKDACFQEPBAFGYKTWBL5RELLZL3L4IJHJT73NS7T7QPWK2C2TDQA, seg_pos={0 0}, upload=211101]: Done uploading piece. err: ecclient: upload failed (node:126Z6tvp49dLLdj3T5snJc4tBVGzcX5UBaGafFN3PkhF92vxpfs, address:104.36.82.50:4047): protocol: rpc: dial tcp 104.36.82.50:4047: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:25.583076716 (202h57m20.014156837s) [noise=true , address=104.36.82.39:4753 , piece_id=R7BHT5RYLSCDZLTDDRXTSJFSRPLM5PNBYC6Q2G33C5SYIUCNVCLQ, seg_pos={0 0}, upload=211101]: Done uploading piece. err: ecclient: upload failed (node:1zzW2z6noGQT6xdSmwSQBNYwip6gZjnmqZg7qdb6BP3nHyCLFz, address:104.36.82.39:4753): protocol: rpc: dial tcp 104.36.82.39:4753: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:25.583107535 (202h57m20.014187666s) [noise=true , address=104.36.82.23:4474 , piece_id=RITPHITVW655UQETT6UKS7BZO2S3F4P2MJIO4REOUKYYP4E2CDVQ, seg_pos={0 0}, upload=211101]: Done uploading piece. err: ecclient: upload failed (node:1Fpshye6yGAo3yf71eVpxrCdvoRjpkGX9WEX8KCWYbvuhy5i9j, address:104.36.82.23:4474): protocol: write tcp 188.166.225.170:36642->104.36.82.23:4474: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:25.583165355 (202h57m20.014245487s) [noise=true , address=104.36.82.5:4855 , piece_id=3FCDYA6CRYOOU3MDFJPEC4YQNICFVCN2A47RIYPSR4ZX5TIXZX3A, seg_pos={0 0}, upload=211101]: Done uploading piece. err: ecclient: upload failed (node:1KDNuE7YY8smhDjotushQvY3y9NtLxigbJELHD4Edxy8Qi4BWh, address:104.36.82.5:4855): protocol: write tcp 188.166.225.170:47052->104.36.82.5:4855: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:25.583194000 (202h57m20.014274131s) [seg_pos={0 0}, upload=211101]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Aug 17 04:23:25.810738626 (202h57m20.241818757s) [upload=211101]: Done uploading segments... Aug 17 04:23:25.810761991 (202h57m20.241842122s) [upload=211101]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 5.682703227s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [........................................X.....X.............] duration: 1.844734128s FAIL (2025-08-17 04:49:25 UTC) orbiter warning: operation: ListObjects ======ERROR====== unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/1K ======LOGS====== Aug 17 04:41:26.040732864 (203h15m20.471812985s) [upload=211420]: Starting upload Aug 17 04:41:26.040780384 (203h15m20.471860515s) [upload=211420]: Done starting upload Aug 17 04:41:26.040828978 (203h15m20.471909109s) [upload=211420]: Uploading segments... Aug 17 04:41:26.040863473 (203h15m20.471943604s) [upload=211420]: Got next segment. Inline: true Aug 17 04:41:26.040867290 (203h15m20.471947422s) [upload=211420]: Waiting for error group managing segments... Aug 17 04:41:26.040897880 (203h15m20.471978011s) [upload=211420]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 17 04:41:26.040901597 (203h15m20.471981728s) [upload=211420]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 17 04:41:26.040904041 (203h15m20.471984162s) [upload=211420]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 17 04:41:26.462140724 (203h15m20.893220855s) [upload=211420]: Done uploading segments... Aug 17 04:41:26.462159310 (203h15m20.893239431s) [upload=211420]: Upload finished. err: <nil> Duration:* 1.844764978s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 9 out of 60 is failed (threshold=3) [..............X..........XXXX.X.X.......X...X...............] duration: 5.784845254s FAIL (2025-08-17 04:49:20 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/4M") ======LOGS====== Aug 17 04:37:54.182428668 (203h11m48.613508799s) [upload=211361]: Starting upload Aug 17 04:37:54.182474656 (203h11m48.613554777s) [upload=211361]: Done starting upload Aug 17 04:37:54.182665000 (203h11m48.613745131s) [upload=211361]: Uploading segments... Aug 17 04:37:54.182728421 (203h11m48.613808552s) [upload=211361]: Got next segment. Inline: false Aug 17 04:37:54.182758048 (203h11m48.613838179s) [upload=211361]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 17 04:37:54.182762306 (203h11m48.613842437s) [upload=211361]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 17 04:37:54.737160222 (203h11m49.168240353s) [seg_pos={0 0}, upload=211361]: Begin upload segment... Aug 17 04:37:54.737203494 (203h11m49.168283625s) [seg_pos={0 0}, upload=211361]: Done begin upload segment. Aug 17 04:37:54.737224945 (203h11m49.168305076s) [noise=true , address=77.90.31.19:4863 , piece_id=PYERWDVDOTOW2SVDUO63ODTEWMWUGH5OMV2PE7RJTIDVFTF2ZRVQ, seg_pos={0 0}, upload=211361]: Uploading piece... Aug 17 04:37:54.737311591 (203h11m49.168391722s) [noise=true , address=79.127.220.101:4068 , piece_id=6MFJ22JL437NTOH25E3LNH3Q2LQNKKMUT5Q43F2XNIB7346LGFCQ, seg_pos={0 0}, upload=211361]: Uploading piece... Aug 17 04:37:54.737638917 (203h11m49.168719048s) [noise=true , address=77.90.31.12:4601 , piece_id=JC5VIGLKCWTCDHQYWEL6QSRZW3MJZFRFQKRJH54OK7LPUQH7WQTQ, seg_pos={0 0}, upload=211361]: Uploading piece... Aug 17 04:37:54.737939432 (203h11m49.169019563s) [noise=true , address=66.23.193.175:4562 , piece_id=2RZ2A5KBLZBLEG6UECY2GTNF5THR3XJZMS34ULG5SQNWN4O3XNMA, seg_pos={0 0}, upload=211361]: Uploading piece... Aug 17 04:37:54.737952067 (203h11m49.169032198s) [noise=true , address=77.90.31.16:4761 , piece_id=44YO7GYP4RNRWHPP7LP2C4FJBGMWOLEDEC2NN7TAWU3YFHAST3XQ, seg_pos={0 0}, upload=211361]: Uploading piece... Aug 17 04:37:54.738229999 (203h11m49.16931013s) [noise=true , address=104.36.82.36:4648 , piece_id=GTHXHUS6GR26G6UHHMB35MCSEI7I5UJ6K65O76HRJI4WVPIHD7RQ, seg_pos={0 0}, upload=211361]: Uploading piece... Aug 17 04:37:54.738324038 (203h11m49.169404169s) [noise=true , address=66.23.193.175:4563 , piece_id=I65W2GP5RTRZUIT7Y6NSEAL7XC726PP3BYMCWS6UMZEOWAP5CP2A, seg_pos={0 0}, upload=211361]: Uploading piece... Aug 17 04:37:54.738511367 (203h11m49.169591488s) [noise=true , address=104.36.82.4:4833 , piece_id=AMRE5SIA7EJLAP55GFYILJYZJL5624E534TPWFUEQILHG7CCFDXQ, seg_pos={0 0}, upload=211361]: Uploading piece... Aug 17 04:37:54.738655813 (203h11m49.169735954s) [noise=true , address=104.36.82.5:4844 , piece_id=PULFFEXZLSMOYEFGRK6L2POF7ERJQT75J23JT3CTXKZ3LMARQZHQ, seg_pos={0 0}, upload=211361]: Uploading piece... Aug 17 04:37:54.738747769 (203h11m49.1698279s) [noise=true , address=77.90.31.10:4532 , piece_id=5IBCLJNM6LUBCIOPSAERY5HAXV66CR4OKI45RBGOHALTRFYGLMWQ, seg_pos={0 0}, upload=211361]: Uploading piece... Aug 17 04:37:54.738971367 (203h11m49.170051498s) [noise=true , address=104.36.82.60:4360 , piece_id=W4OJU6NRV2HKPC7JDTISAPVT77KIC3JHVJR5KHYY225FLH4PZBBA, seg_pos={0 0}, upload=211361]: Uploading piece... Aug 17 04:37:54.738975705 (203h11m49.170055836s) [noise=true , address=77.90.31.18:4830 , piece_id=QNQF4JAEHOFWRTL6XWKL42XSKA2DMO6Z4CZO5NF337NNSHCMYYBA, seg_pos={0 0}, upload=211361]: Uploading piece... Aug 17 04:37:54.739194995 (203h11m49.170275126s) [noise=true , address=104.36.82.23:4468 , piece_id=NC2BFOASSHA4ZG46LZONWF23DNN3ZZBA6SFO3Q44KD4TCOH4AIQQ, seg_pos={0 0}, upload=211361]: Uploading piece... Aug 17 04:37:54.739278324 (203h11m49.170358445s) [noise=true , address=104.36.82.39:4747 , piece_id=SF2P47ED5FEOCMJ6R6AXP2NTI5LYYALSDLHINNLDVMUMZIE4YLHQ, seg_pos={0 0}, upload=211361]: Uploading piece... Aug 17 04:37:54.739421147 (203h11m49.170501278s) [noise=true , address=104.36.82.3:4800 , piece_id=EKC43HOMYNG77PDESU3QV5OL44HWYQRAN5RUNIB7IMC6NVXQWFVA, seg_pos={0 0}, upload=211361]: Uploading piece... Aug 17 04:37:54.739561055 (203h11m49.170641186s) [noise=true , address=77.90.31.13:4634 , piece_id=ZXX5E54TSYUG557OGOCBQMBKS35TUG6OPDOIG3TTS3OYGTRA24XA, seg_pos={0 0}, upload=211361]: Uploading piece... Aug 17 04:37:54.739643824 (203h11m49.170723955s) [noise=true , address=104.36.82.58:4286 , piece_id=JDBFFYOZPYJ5FXJQIVEBB5IWSBGT6LCBUZF5O25OKWFRXROPHS3Q, seg_pos={0 0}, upload=211361]: Uploading piece... Aug 17 04:37:54.739852683 (203h11m49.170932814s) [noise=true , address=104.36.82.44:4876 , piece_id=H53T7WMGX4CDVQNCYOG6NOYFVRHD5Z35LNF37FVKSWYIVJS2EFZA, seg_pos={0 0}, upload=211361]: Uploading piece... Aug 17 04:37:55.861243533 (203h11m50.292323664s) [upload=211361]: Next returned nil segment Aug 17 04:37:55.861256768 (203h11m50.292336899s) [upload=211361]: Waiting for error group managing segments... Aug 17 04:37:56.165335502 (203h11m50.596415633s) [noise=true , address=66.23.193.175:4562 , piece_id=2RZ2A5KBLZBLEG6UECY2GTNF5THR3XJZMS34ULG5SQNWN4O3XNMA, seg_pos={0 0}, upload=211361]: Done uploading piece. err: <nil> Aug 17 04:37:56.263548583 (203h11m50.694628715s) [noise=true , address=66.23.193.175:4563 , piece_id=I65W2GP5RTRZUIT7Y6NSEAL7XC726PP3BYMCWS6UMZEOWAP5CP2A, seg_pos={0 0}, upload=211361]: Done uploading piece. err: <nil> Aug 17 04:37:56.569392606 (203h11m51.000472737s) [noise=true , address=79.127.220.101:4068 , piece_id=6MFJ22JL437NTOH25E3LNH3Q2LQNKKMUT5Q43F2XNIB7346LGFCQ, seg_pos={0 0}, upload=211361]: Done uploading piece. err: <nil> Aug 17 04:37:56.620901052 (203h11m51.051981183s) [noise=true , address=77.90.31.12:4601 , piece_id=JC5VIGLKCWTCDHQYWEL6QSRZW3MJZFRFQKRJH54OK7LPUQH7WQTQ, seg_pos={0 0}, upload=211361]: Done uploading piece. err: <nil> Aug 17 04:37:56.639713969 (203h11m51.0707941s) [noise=true , address=77.90.31.16:4761 , piece_id=44YO7GYP4RNRWHPP7LP2C4FJBGMWOLEDEC2NN7TAWU3YFHAST3XQ, seg_pos={0 0}, upload=211361]: Done uploading piece. err: <nil> Aug 17 04:37:56.649068307 (203h11m51.080148438s) [noise=true , address=104.36.82.4:4833 , piece_id=AMRE5SIA7EJLAP55GFYILJYZJL5624E534TPWFUEQILHG7CCFDXQ, seg_pos={0 0}, upload=211361]: Done uploading piece. err: <nil> Aug 17 04:37:56.701677831 (203h11m51.132757962s) [noise=true , address=104.36.82.3:4800 , piece_id=EKC43HOMYNG77PDESU3QV5OL44HWYQRAN5RUNIB7IMC6NVXQWFVA, seg_pos={0 0}, upload=211361]: Done uploading piece. err: <nil> Aug 17 04:37:56.719713991 (203h11m51.150794132s) [noise=true , address=104.36.82.23:4468 , piece_id=NC2BFOASSHA4ZG46LZONWF23DNN3ZZBA6SFO3Q44KD4TCOH4AIQQ, seg_pos={0 0}, upload=211361]: Done uploading piece. err: <nil> Aug 17 04:37:56.730632311 (203h11m51.161712432s) [noise=true , address=104.36.82.39:4747 , piece_id=SF2P47ED5FEOCMJ6R6AXP2NTI5LYYALSDLHINNLDVMUMZIE4YLHQ, seg_pos={0 0}, upload=211361]: Done uploading piece. err: <nil> Aug 17 04:37:56.761149974 (203h11m51.192230105s) [noise=true , address=104.36.82.60:4360 , piece_id=W4OJU6NRV2HKPC7JDTISAPVT77KIC3JHVJR5KHYY225FLH4PZBBA, seg_pos={0 0}, upload=211361]: Done uploading piece. err: <nil> Aug 17 04:37:56.775738719 (203h11m51.20681885s) [noise=true , address=77.90.31.18:4830 , piece_id=QNQF4JAEHOFWRTL6XWKL42XSKA2DMO6Z4CZO5NF337NNSHCMYYBA, seg_pos={0 0}, upload=211361]: Done uploading piece. err: <nil> Aug 17 04:37:56.779316274 (203h11m51.210396405s) [noise=true , address=77.90.31.13:4634 , piece_id=ZXX5E54TSYUG557OGOCBQMBKS35TUG6OPDOIG3TTS3OYGTRA24XA, seg_pos={0 0}, upload=211361]: Done uploading piece. err: <nil> Aug 17 04:37:56.802147800 (203h11m51.233227961s) [noise=true , address=104.36.82.58:4286 , piece_id=JDBFFYOZPYJ5FXJQIVEBB5IWSBGT6LCBUZF5O25OKWFRXROPHS3Q, seg_pos={0 0}, upload=211361]: Done uploading piece. err: <nil> Aug 17 04:37:56.832614886 (203h11m51.263695017s) [noise=true , address=104.36.82.44:4876 , piece_id=H53T7WMGX4CDVQNCYOG6NOYFVRHD5Z35LNF37FVKSWYIVJS2EFZA, seg_pos={0 0}, upload=211361]: Done uploading piece. err: <nil> Aug 17 04:37:56.832646346 (203h11m51.263726467s) [seg_pos={0 0}, upload=211361]: Segment reached optimal threshold of 14 pieces. Aug 17 04:37:56.832765443 (203h11m51.263845574s) [noise=true , address=77.90.31.10:4532 , piece_id=5IBCLJNM6LUBCIOPSAERY5HAXV66CR4OKI45RBGOHALTRFYGLMWQ, seg_pos={0 0}, upload=211361]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1T6kkKvTNR5MZZijjr1vMU6g75YkoRpzMvGSgQCqRnfCm9JULe): protocol: expected piece hash; context canceled; EOF Aug 17 04:37:56.832872889 (203h11m51.26395302s) [noise=true , address=104.36.82.36:4648 , piece_id=GTHXHUS6GR26G6UHHMB35MCSEI7I5UJ6K65O76HRJI4WVPIHD7RQ, seg_pos={0 0}, upload=211361]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XHNZDY7xKZ2bss1rSdjbGP3ugSwGRwRzBSWHfXgdKKz794wMq): protocol: expected piece hash; context canceled; EOF Aug 17 04:37:56.832892156 (203h11m51.263972287s) [noise=true , address=77.90.31.19:4863 , piece_id=PYERWDVDOTOW2SVDUO63ODTEWMWUGH5OMV2PE7RJTIDVFTF2ZRVQ, seg_pos={0 0}, upload=211361]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124x1cbCPJRK8YkgSP6UzHymoLu1E1znnfg5WBiD89kSDLi8JeL): protocol: expected piece hash; context canceled; EOF Aug 17 04:37:56.832974564 (203h11m51.264054695s) [noise=true , address=104.36.82.5:4844 , piece_id=PULFFEXZLSMOYEFGRK6L2POF7ERJQT75J23JT3CTXKZ3LMARQZHQ, seg_pos={0 0}, upload=211361]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12iovpNdbidkiv9mEHaAPjBubGmoqr4sUJWm8Cb64LzS6wuwLWa): protocol: write tcp 188.166.225.170:59826->104.36.82.5:4844: use of closed network connection; write tcp 188.166.225.170:59826->104.36.82.5:4844: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59826->104.36.82.5:4844: use of closed network connection Aug 17 04:37:56.833025241 (203h11m51.264105372s) [seg_pos={0 0}, upload=211361]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 17 04:37:56.833116014 (203h11m51.264196135s) [upload=211361]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 17 04:37:56.833120332 (203h11m51.264200463s) [upload=211361]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 17 04:37:58.261163702 (203h11m52.692243834s) [upload=211361]: Done uploading segments... Aug 17 04:37:58.261181997 (203h11m52.692262118s) [upload=211361]: Upload finished. err: <nil> Duration: 5.784871235s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [....................XXXX....................................] duration: 5.664447107s FAIL (2025-08-17 04:49:11 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== Aug 17 04:23:03.873697274 (202h56m58.304777405s) [upload=211112]: Starting upload Aug 17 04:23:03.873756368 (202h56m58.304836499s) [upload=211112]: Done starting upload Aug 17 04:23:03.873839848 (202h56m58.304919989s) [upload=211112]: Uploading segments... Aug 17 04:23:03.873950850 (202h56m58.305030981s) [upload=211112]: Got next segment. Inline: false Aug 17 04:23:03.873976509 (202h56m58.30505664s) [upload=211112]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 17 04:23:03.873980757 (202h56m58.305060888s) [upload=211112]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 17 04:23:04.113727713 (202h56m58.544807844s) [seg_pos={0 0}, upload=211112]: Begin upload segment... Aug 17 04:23:04.113774051 (202h56m58.544854182s) [seg_pos={0 0}, upload=211112]: Done begin upload segment. Aug 17 04:23:04.113820890 (202h56m58.544901021s) [noise=true , address=45.91.116.13:4155 , piece_id=CO3R5XZGH3NJADAFDILPNZEKKFJSRPB4JOFBBCSYYQAD3KNRABNA, seg_pos={0 0}, upload=211112]: Uploading piece... Aug 17 04:23:04.113845989 (202h56m58.54492612s) [noise=true , address=45.91.116.7:4009 , piece_id=IVTDHCTNJEEO3QATGN25YH5Y2MSEQUWJIFPXSJWINUEXDHQAGELA, seg_pos={0 0}, upload=211112]: Uploading piece... Aug 17 04:23:04.114195037 (202h56m58.545275168s) [noise=true , address=45.91.116.4:4938 , piece_id=X5MDNBZYRYYOXDEAMGIOD5CXX7PGAMTFRMROOVTR6NIAYQ5VMXGA, seg_pos={0 0}, upload=211112]: Uploading piece... Aug 17 04:23:04.114281412 (202h56m58.545361543s) [noise=true , address=45.91.116.9:4261 , piece_id=PNOEBFIPLWM4B2QLGJUVEBTT23J5AVRFTWTTKLHFUJ5MNJTSM2JA, seg_pos={0 0}, upload=211112]: Uploading piece... Aug 17 04:23:04.114482115 (202h56m58.545562237s) [noise=true , address=77.90.31.16:4735 , piece_id=G5YGF7LXNPBU2TDRYPAFPVYQSTWUGG7Y6YM2Q6RWMCZY5RVNM4BA, seg_pos={0 0}, upload=211112]: Uploading piece... Aug 17 04:23:04.114570385 (202h56m58.545650516s) [noise=true , address=104.36.82.9:4005 , piece_id=FAR4B5KAALBOACKANEFKGECI2ZVXSHBNYDEIWVGDU2RNBNQXLMXA, seg_pos={0 0}, upload=211112]: Uploading piece... Aug 17 04:23:04.114757392 (202h56m58.545837523s) [noise=true , address=45.91.116.9:4263 , piece_id=VT734KJZ7T3CZ3W3P4M4ZCYKOLLTPP3ILFRWL26Z7K4LWBDZV4FA, seg_pos={0 0}, upload=211112]: Uploading piece... Aug 17 04:23:04.114874727 (202h56m58.545954858s) [noise=true , address=104.36.82.12:4151 , piece_id=2GGYMOWZ2FTWDNWNQRKSAH7UVJ4R6MLRGC6OHNVV3KTVIIMO4QIQ, seg_pos={0 0}, upload=211112]: Uploading piece... Aug 17 04:23:04.115030715 (202h56m58.546110846s) [noise=true , address=45.91.116.12:4225 , piece_id=EUPPM5NRIU5MCZBER3VGZBZEMZZ65Q2MOY2DXKFBIPFJ53M5TJHQ, seg_pos={0 0}, upload=211112]: Uploading piece... Aug 17 04:23:04.115194288 (202h56m58.546274419s) [noise=true , address=104.36.82.4:4804 , piece_id=MSAHN2DERVG2HL7KZMKIWYUQYO6LRIXGUNWJFXJDDGOWZT54FPVA, seg_pos={0 0}, upload=211112]: Uploading piece... Aug 17 04:23:04.115307967 (202h56m58.546388098s) [noise=true , address=104.36.82.5:4838 , piece_id=IYA7ERDAAFYUGC4666CBEX5HZEXZVFPES6JUYRZROY25OAPGVEQQ, seg_pos={0 0}, upload=211112]: Uploading piece... Aug 17 04:23:04.115535692 (202h56m58.546615823s) [noise=true , address=45.91.116.12:4227 , piece_id=XZR3AWJG57RUB4TN7SUWNIWWD2XGKCMDXSWSM74WS3ATHS32EBJQ, seg_pos={0 0}, upload=211112]: Uploading piece... Aug 17 04:23:04.115754130 (202h56m58.546834261s) [noise=true , address=45.91.116.7:4010 , piece_id=HNJU4MPWT67V26J5XDCTHFEOOJL2HS3VAOHB4EIPQVF4D2NNRFMA, seg_pos={0 0}, upload=211112]: Uploading piece... Aug 17 04:23:04.116079182 (202h56m58.547159323s) [noise=true , address=45.91.116.14:4189 , piece_id=2P3GOZDEMJGBAK2F6SY7NJNQUTS5ZYM4B4IMQFAA5TYFARD4DVCQ, seg_pos={0 0}, upload=211112]: Uploading piece... Aug 17 04:23:04.116350181 (202h56m58.547430312s) [noise=true , address=104.36.82.15:4257 , piece_id=Q5GLP6TS7N7CM4DO3LWDCDR6O76OFZ6IVQJC5VQWMA5XLNWZ2E4A, seg_pos={0 0}, upload=211112]: Uploading piece... Aug 17 04:23:04.116554471 (202h56m58.547634602s) [noise=true , address=104.36.82.8:4969 , piece_id=3D3YAZSP7CD6IFZGDKPNKIJPWZ2OJFFVJCSMTO42VNJEFVU3ORHA, seg_pos={0 0}, upload=211112]: Uploading piece... Aug 17 04:23:05.868765063 (202h57m0.299845195s) [upload=211112]: Next returned nil segment Aug 17 04:23:05.868801043 (202h57m0.299881164s) [upload=211112]: Waiting for error group managing segments... Aug 17 04:23:06.147430529 (202h57m0.57851066s) [noise=true , address=45.91.116.12:4227 , piece_id=XZR3AWJG57RUB4TN7SUWNIWWD2XGKCMDXSWSM74WS3ATHS32EBJQ, seg_pos={0 0}, upload=211112]: Done uploading piece. err: <nil> Aug 17 04:23:06.182400314 (202h57m0.613480445s) [noise=true , address=45.91.116.12:4225 , piece_id=EUPPM5NRIU5MCZBER3VGZBZEMZZ65Q2MOY2DXKFBIPFJ53M5TJHQ, seg_pos={0 0}, upload=211112]: Done uploading piece. err: <nil> Aug 17 04:23:06.191910532 (202h57m0.622990663s) [noise=true , address=45.91.116.4:4938 , piece_id=X5MDNBZYRYYOXDEAMGIOD5CXX7PGAMTFRMROOVTR6NIAYQ5VMXGA, seg_pos={0 0}, upload=211112]: Done uploading piece. err: <nil> Aug 17 04:23:06.197391631 (202h57m0.628471782s) [noise=true , address=45.91.116.9:4261 , piece_id=PNOEBFIPLWM4B2QLGJUVEBTT23J5AVRFTWTTKLHFUJ5MNJTSM2JA, seg_pos={0 0}, upload=211112]: Done uploading piece. err: <nil> Aug 17 04:23:06.198069487 (202h57m0.629149618s) [noise=true , address=45.91.116.9:4263 , piece_id=VT734KJZ7T3CZ3W3P4M4ZCYKOLLTPP3ILFRWL26Z7K4LWBDZV4FA, seg_pos={0 0}, upload=211112]: Done uploading piece. err: <nil> Aug 17 04:23:06.200597014 (202h57m0.631677145s) [noise=true , address=45.91.116.7:4009 , piece_id=IVTDHCTNJEEO3QATGN25YH5Y2MSEQUWJIFPXSJWINUEXDHQAGELA, seg_pos={0 0}, upload=211112]: Done uploading piece. err: <nil> Aug 17 04:23:06.288053251 (202h57m0.719133372s) [noise=true , address=45.91.116.7:4010 , piece_id=HNJU4MPWT67V26J5XDCTHFEOOJL2HS3VAOHB4EIPQVF4D2NNRFMA, seg_pos={0 0}, upload=211112]: Done uploading piece. err: <nil> Aug 17 04:23:06.298810435 (202h57m0.729890566s) [noise=true , address=45.91.116.14:4189 , piece_id=2P3GOZDEMJGBAK2F6SY7NJNQUTS5ZYM4B4IMQFAA5TYFARD4DVCQ, seg_pos={0 0}, upload=211112]: Done uploading piece. err: <nil> Aug 17 04:23:06.301049520 (202h57m0.732129651s) [noise=true , address=45.91.116.13:4155 , piece_id=CO3R5XZGH3NJADAFDILPNZEKKFJSRPB4JOFBBCSYYQAD3KNRABNA, seg_pos={0 0}, upload=211112]: Done uploading piece. err: <nil> Aug 17 04:23:06.402776985 (202h57m0.833857116s) [noise=true , address=77.90.31.16:4735 , piece_id=G5YGF7LXNPBU2TDRYPAFPVYQSTWUGG7Y6YM2Q6RWMCZY5RVNM4BA, seg_pos={0 0}, upload=211112]: Done uploading piece. err: <nil> Aug 17 04:23:33.223003412 (202h57m27.654083543s) [noise=true , address=104.36.82.5:4838 , piece_id=IYA7ERDAAFYUGC4666CBEX5HZEXZVFPES6JUYRZROY25OAPGVEQQ, seg_pos={0 0}, upload=211112]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:104.36.82.5:4838): protocol: write tcp 188.166.225.170:42258->104.36.82.5:4838: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:33.223050702 (202h57m27.654130823s) [noise=true , address=104.36.82.8:4969 , piece_id=3D3YAZSP7CD6IFZGDKPNKIJPWZ2OJFFVJCSMTO42VNJEFVU3ORHA, seg_pos={0 0}, upload=211112]: Done uploading piece. err: ecclient: upload failed (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, address:104.36.82.8:4969): protocol: write tcp 188.166.225.170:52308->104.36.82.8:4969: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:33.223090619 (202h57m27.65417075s) [noise=true , address=104.36.82.15:4257 , piece_id=Q5GLP6TS7N7CM4DO3LWDCDR6O76OFZ6IVQJC5VQWMA5XLNWZ2E4A, seg_pos={0 0}, upload=211112]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:104.36.82.15:4257): protocol: write tcp 188.166.225.170:53078->104.36.82.15:4257: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:33.223104976 (202h57m27.654185097s) [noise=true , address=104.36.82.4:4804 , piece_id=MSAHN2DERVG2HL7KZMKIWYUQYO6LRIXGUNWJFXJDDGOWZT54FPVA, seg_pos={0 0}, upload=211112]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:104.36.82.4:4804): protocol: write tcp 188.166.225.170:56176->104.36.82.4:4804: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:33.223149581 (202h57m27.654229712s) [noise=true , address=104.36.82.12:4151 , piece_id=2GGYMOWZ2FTWDNWNQRKSAH7UVJ4R6MLRGC6OHNVV3KTVIIMO4QIQ, seg_pos={0 0}, upload=211112]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:104.36.82.12:4151): protocol: write tcp 188.166.225.170:40582->104.36.82.12:4151: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:33.223157536 (202h57m27.654237667s) [noise=true , address=104.36.82.9:4005 , piece_id=FAR4B5KAALBOACKANEFKGECI2ZVXSHBNYDEIWVGDU2RNBNQXLMXA, seg_pos={0 0}, upload=211112]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:104.36.82.9:4005): protocol: write tcp 188.166.225.170:32954->104.36.82.9:4005: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:33.223177415 (202h57m27.654257536s) [seg_pos={0 0}, upload=211112]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Aug 17 04:23:33.448862924 (202h57m27.879943045s) [upload=211112]: Done uploading segments... Aug 17 04:23:33.448886830 (202h57m27.879966961s) [upload=211112]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 5.664472195s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [.........................................X.....X............] duration: 2.026232913s FAIL (2025-08-17 04:49:06 UTC) orbiter warning: operation: ListObjects ======ERROR====== unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/1K ======LOGS====== Aug 17 04:41:26.040732864 (203h15m20.471812985s) [upload=211420]: Starting upload Aug 17 04:41:26.040780384 (203h15m20.471860515s) [upload=211420]: Done starting upload Aug 17 04:41:26.040828978 (203h15m20.471909109s) [upload=211420]: Uploading segments... Aug 17 04:41:26.040863473 (203h15m20.471943604s) [upload=211420]: Got next segment. Inline: true Aug 17 04:41:26.040867290 (203h15m20.471947422s) [upload=211420]: Waiting for error group managing segments... Aug 17 04:41:26.040897880 (203h15m20.471978011s) [upload=211420]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 17 04:41:26.040901597 (203h15m20.471981728s) [upload=211420]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 17 04:41:26.040904041 (203h15m20.471984162s) [upload=211420]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 17 04:41:26.462140724 (203h15m20.893220855s) [upload=211420]: Done uploading segments... Aug 17 04:41:26.462159310 (203h15m20.893239431s) [upload=211420]: Upload finished. err: <nil> Duration:* 2.026260487s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.........................XX.................................] duration: 6.684424703s FAIL (2025-08-17 04:48:57 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.684628122s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.....................XXXX...................................] duration: 5.795940886s FAIL (2025-08-17 04:48:43 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== Aug 17 04:23:03.873697274 (202h56m58.304777405s) [upload=211112]: Starting upload Aug 17 04:23:03.873756368 (202h56m58.304836499s) [upload=211112]: Done starting upload Aug 17 04:23:03.873839848 (202h56m58.304919989s) [upload=211112]: Uploading segments... Aug 17 04:23:03.873950850 (202h56m58.305030981s) [upload=211112]: Got next segment. Inline: false Aug 17 04:23:03.873976509 (202h56m58.30505664s) [upload=211112]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 17 04:23:03.873980757 (202h56m58.305060888s) [upload=211112]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 17 04:23:04.113727713 (202h56m58.544807844s) [seg_pos={0 0}, upload=211112]: Begin upload segment... Aug 17 04:23:04.113774051 (202h56m58.544854182s) [seg_pos={0 0}, upload=211112]: Done begin upload segment. Aug 17 04:23:04.113820890 (202h56m58.544901021s) [noise=true , address=45.91.116.13:4155 , piece_id=CO3R5XZGH3NJADAFDILPNZEKKFJSRPB4JOFBBCSYYQAD3KNRABNA, seg_pos={0 0}, upload=211112]: Uploading piece... Aug 17 04:23:04.113845989 (202h56m58.54492612s) [noise=true , address=45.91.116.7:4009 , piece_id=IVTDHCTNJEEO3QATGN25YH5Y2MSEQUWJIFPXSJWINUEXDHQAGELA, seg_pos={0 0}, upload=211112]: Uploading piece... Aug 17 04:23:04.114195037 (202h56m58.545275168s) [noise=true , address=45.91.116.4:4938 , piece_id=X5MDNBZYRYYOXDEAMGIOD5CXX7PGAMTFRMROOVTR6NIAYQ5VMXGA, seg_pos={0 0}, upload=211112]: Uploading piece... Aug 17 04:23:04.114281412 (202h56m58.545361543s) [noise=true , address=45.91.116.9:4261 , piece_id=PNOEBFIPLWM4B2QLGJUVEBTT23J5AVRFTWTTKLHFUJ5MNJTSM2JA, seg_pos={0 0}, upload=211112]: Uploading piece... Aug 17 04:23:04.114482115 (202h56m58.545562237s) [noise=true , address=77.90.31.16:4735 , piece_id=G5YGF7LXNPBU2TDRYPAFPVYQSTWUGG7Y6YM2Q6RWMCZY5RVNM4BA, seg_pos={0 0}, upload=211112]: Uploading piece... Aug 17 04:23:04.114570385 (202h56m58.545650516s) [noise=true , address=104.36.82.9:4005 , piece_id=FAR4B5KAALBOACKANEFKGECI2ZVXSHBNYDEIWVGDU2RNBNQXLMXA, seg_pos={0 0}, upload=211112]: Uploading piece... Aug 17 04:23:04.114757392 (202h56m58.545837523s) [noise=true , address=45.91.116.9:4263 , piece_id=VT734KJZ7T3CZ3W3P4M4ZCYKOLLTPP3ILFRWL26Z7K4LWBDZV4FA, seg_pos={0 0}, upload=211112]: Uploading piece... Aug 17 04:23:04.114874727 (202h56m58.545954858s) [noise=true , address=104.36.82.12:4151 , piece_id=2GGYMOWZ2FTWDNWNQRKSAH7UVJ4R6MLRGC6OHNVV3KTVIIMO4QIQ, seg_pos={0 0}, upload=211112]: Uploading piece... Aug 17 04:23:04.115030715 (202h56m58.546110846s) [noise=true , address=45.91.116.12:4225 , piece_id=EUPPM5NRIU5MCZBER3VGZBZEMZZ65Q2MOY2DXKFBIPFJ53M5TJHQ, seg_pos={0 0}, upload=211112]: Uploading piece... Aug 17 04:23:04.115194288 (202h56m58.546274419s) [noise=true , address=104.36.82.4:4804 , piece_id=MSAHN2DERVG2HL7KZMKIWYUQYO6LRIXGUNWJFXJDDGOWZT54FPVA, seg_pos={0 0}, upload=211112]: Uploading piece... Aug 17 04:23:04.115307967 (202h56m58.546388098s) [noise=true , address=104.36.82.5:4838 , piece_id=IYA7ERDAAFYUGC4666CBEX5HZEXZVFPES6JUYRZROY25OAPGVEQQ, seg_pos={0 0}, upload=211112]: Uploading piece... Aug 17 04:23:04.115535692 (202h56m58.546615823s) [noise=true , address=45.91.116.12:4227 , piece_id=XZR3AWJG57RUB4TN7SUWNIWWD2XGKCMDXSWSM74WS3ATHS32EBJQ, seg_pos={0 0}, upload=211112]: Uploading piece... Aug 17 04:23:04.115754130 (202h56m58.546834261s) [noise=true , address=45.91.116.7:4010 , piece_id=HNJU4MPWT67V26J5XDCTHFEOOJL2HS3VAOHB4EIPQVF4D2NNRFMA, seg_pos={0 0}, upload=211112]: Uploading piece... Aug 17 04:23:04.116079182 (202h56m58.547159323s) [noise=true , address=45.91.116.14:4189 , piece_id=2P3GOZDEMJGBAK2F6SY7NJNQUTS5ZYM4B4IMQFAA5TYFARD4DVCQ, seg_pos={0 0}, upload=211112]: Uploading piece... Aug 17 04:23:04.116350181 (202h56m58.547430312s) [noise=true , address=104.36.82.15:4257 , piece_id=Q5GLP6TS7N7CM4DO3LWDCDR6O76OFZ6IVQJC5VQWMA5XLNWZ2E4A, seg_pos={0 0}, upload=211112]: Uploading piece... Aug 17 04:23:04.116554471 (202h56m58.547634602s) [noise=true , address=104.36.82.8:4969 , piece_id=3D3YAZSP7CD6IFZGDKPNKIJPWZ2OJFFVJCSMTO42VNJEFVU3ORHA, seg_pos={0 0}, upload=211112]: Uploading piece... Aug 17 04:23:05.868765063 (202h57m0.299845195s) [upload=211112]: Next returned nil segment Aug 17 04:23:05.868801043 (202h57m0.299881164s) [upload=211112]: Waiting for error group managing segments... Aug 17 04:23:06.147430529 (202h57m0.57851066s) [noise=true , address=45.91.116.12:4227 , piece_id=XZR3AWJG57RUB4TN7SUWNIWWD2XGKCMDXSWSM74WS3ATHS32EBJQ, seg_pos={0 0}, upload=211112]: Done uploading piece. err: <nil> Aug 17 04:23:06.182400314 (202h57m0.613480445s) [noise=true , address=45.91.116.12:4225 , piece_id=EUPPM5NRIU5MCZBER3VGZBZEMZZ65Q2MOY2DXKFBIPFJ53M5TJHQ, seg_pos={0 0}, upload=211112]: Done uploading piece. err: <nil> Aug 17 04:23:06.191910532 (202h57m0.622990663s) [noise=true , address=45.91.116.4:4938 , piece_id=X5MDNBZYRYYOXDEAMGIOD5CXX7PGAMTFRMROOVTR6NIAYQ5VMXGA, seg_pos={0 0}, upload=211112]: Done uploading piece. err: <nil> Aug 17 04:23:06.197391631 (202h57m0.628471782s) [noise=true , address=45.91.116.9:4261 , piece_id=PNOEBFIPLWM4B2QLGJUVEBTT23J5AVRFTWTTKLHFUJ5MNJTSM2JA, seg_pos={0 0}, upload=211112]: Done uploading piece. err: <nil> Aug 17 04:23:06.198069487 (202h57m0.629149618s) [noise=true , address=45.91.116.9:4263 , piece_id=VT734KJZ7T3CZ3W3P4M4ZCYKOLLTPP3ILFRWL26Z7K4LWBDZV4FA, seg_pos={0 0}, upload=211112]: Done uploading piece. err: <nil> Aug 17 04:23:06.200597014 (202h57m0.631677145s) [noise=true , address=45.91.116.7:4009 , piece_id=IVTDHCTNJEEO3QATGN25YH5Y2MSEQUWJIFPXSJWINUEXDHQAGELA, seg_pos={0 0}, upload=211112]: Done uploading piece. err: <nil> Aug 17 04:23:06.288053251 (202h57m0.719133372s) [noise=true , address=45.91.116.7:4010 , piece_id=HNJU4MPWT67V26J5XDCTHFEOOJL2HS3VAOHB4EIPQVF4D2NNRFMA, seg_pos={0 0}, upload=211112]: Done uploading piece. err: <nil> Aug 17 04:23:06.298810435 (202h57m0.729890566s) [noise=true , address=45.91.116.14:4189 , piece_id=2P3GOZDEMJGBAK2F6SY7NJNQUTS5ZYM4B4IMQFAA5TYFARD4DVCQ, seg_pos={0 0}, upload=211112]: Done uploading piece. err: <nil> Aug 17 04:23:06.301049520 (202h57m0.732129651s) [noise=true , address=45.91.116.13:4155 , piece_id=CO3R5XZGH3NJADAFDILPNZEKKFJSRPB4JOFBBCSYYQAD3KNRABNA, seg_pos={0 0}, upload=211112]: Done uploading piece. err: <nil> Aug 17 04:23:06.402776985 (202h57m0.833857116s) [noise=true , address=77.90.31.16:4735 , piece_id=G5YGF7LXNPBU2TDRYPAFPVYQSTWUGG7Y6YM2Q6RWMCZY5RVNM4BA, seg_pos={0 0}, upload=211112]: Done uploading piece. err: <nil> Aug 17 04:23:33.223003412 (202h57m27.654083543s) [noise=true , address=104.36.82.5:4838 , piece_id=IYA7ERDAAFYUGC4666CBEX5HZEXZVFPES6JUYRZROY25OAPGVEQQ, seg_pos={0 0}, upload=211112]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:104.36.82.5:4838): protocol: write tcp 188.166.225.170:42258->104.36.82.5:4838: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:33.223050702 (202h57m27.654130823s) [noise=true , address=104.36.82.8:4969 , piece_id=3D3YAZSP7CD6IFZGDKPNKIJPWZ2OJFFVJCSMTO42VNJEFVU3ORHA, seg_pos={0 0}, upload=211112]: Done uploading piece. err: ecclient: upload failed (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, address:104.36.82.8:4969): protocol: write tcp 188.166.225.170:52308->104.36.82.8:4969: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:33.223090619 (202h57m27.65417075s) [noise=true , address=104.36.82.15:4257 , piece_id=Q5GLP6TS7N7CM4DO3LWDCDR6O76OFZ6IVQJC5VQWMA5XLNWZ2E4A, seg_pos={0 0}, upload=211112]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:104.36.82.15:4257): protocol: write tcp 188.166.225.170:53078->104.36.82.15:4257: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:33.223104976 (202h57m27.654185097s) [noise=true , address=104.36.82.4:4804 , piece_id=MSAHN2DERVG2HL7KZMKIWYUQYO6LRIXGUNWJFXJDDGOWZT54FPVA, seg_pos={0 0}, upload=211112]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:104.36.82.4:4804): protocol: write tcp 188.166.225.170:56176->104.36.82.4:4804: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:33.223149581 (202h57m27.654229712s) [noise=true , address=104.36.82.12:4151 , piece_id=2GGYMOWZ2FTWDNWNQRKSAH7UVJ4R6MLRGC6OHNVV3KTVIIMO4QIQ, seg_pos={0 0}, upload=211112]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:104.36.82.12:4151): protocol: write tcp 188.166.225.170:40582->104.36.82.12:4151: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:33.223157536 (202h57m27.654237667s) [noise=true , address=104.36.82.9:4005 , piece_id=FAR4B5KAALBOACKANEFKGECI2ZVXSHBNYDEIWVGDU2RNBNQXLMXA, seg_pos={0 0}, upload=211112]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:104.36.82.9:4005): protocol: write tcp 188.166.225.170:32954->104.36.82.9:4005: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:33.223177415 (202h57m27.654257536s) [seg_pos={0 0}, upload=211112]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Aug 17 04:23:33.448862924 (202h57m27.879943045s) [upload=211112]: Done uploading segments... Aug 17 04:23:33.448886830 (202h57m27.879966961s) [upload=211112]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 5.79596395s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [.....................XXX....................................] duration: 5.756282674s FAIL (2025-08-17 04:48:39 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== g_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.761278730 (202h56m21.192358861s) [noise=true , address=104.36.82.34:4558 , piece_id=XGFNEBBVZX5UWJN7DALGM2EDG26POCTSBBAUU7DLYQKS2DNSBNXA, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.761661883 (202h56m21.192742014s) [noise=true , address=104.36.82.35:4594 , piece_id=6NUSPNYGSSJOB2CNPIW6DI45MPZARLWDBXH3CWMP4LXMDB2CXWZQ, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.761716548 (202h56m21.192796679s) [noise=true , address=45.91.116.8:4067 , piece_id=EZAI7ZDUBOOWXYJNTB7ZPOUSBUUG2FKV77QVUOZMNM6TUJSKGYWA, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.761972016 (202h56m21.193052147s) [noise=true , address=104.36.82.58:4283 , piece_id=LJKEH2KDSOFPVFPB5UWGVLOHQVZQOY33MYSHFLP4ZWGPKZ4KC64A, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.762254878 (202h56m21.193335009s) [noise=true , address=79.127.201.215:4345 , piece_id=JMSKNIZ7QPGBXIEP5RA3S74XAU5NJ2MUUXSFFIAK377YFGO646YQ, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.762523682 (202h56m21.193603803s) [noise=true , address=45.91.116.13:4181 , piece_id=7VN6RQGOTBHQO6HXM5IFTU32PWNVPH6BJZMQNETJEXOYPR2Y7OCQ, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.762864294 (202h56m21.193944415s) [noise=true , address=45.91.116.14:4203 , piece_id=5LXGQ4JZD2KLW3OMWY5QYRE4MKQNKKSAIMESODKDDT2W33XD45DQ, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.763179537 (202h56m21.194259668s) [noise=true , address=77.90.31.18:4815 , piece_id=G6PINAFFA6DZRRMKU2XK2LYFOES5WG7PTJ5L2ZPUQHMA4JNSRZRQ, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.763478499 (202h56m21.19455864s) [noise=true , address=77.90.31.11:4562 , piece_id=PUL6TAVIIU7732IOASNH5EHQMNCK7DWHJMUJVXRNK74T4GUQDGTA, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.763757683 (202h56m21.194837814s) [noise=true , address=104.36.82.5:4855 , piece_id=3FCDYA6CRYOOU3MDFJPEC4YQNICFVCN2A47RIYPSR4ZX5TIXZX3A, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.763988195 (202h56m21.195068326s) [noise=true , address=45.91.116.5:4308 , piece_id=F52M3VMHRQWR5LIEEPMVKVBFMOVG76UNNGQTFKSF6ZBD7VEOFWVA, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.764273320 (202h56m21.195353451s) [noise=true , address=66.23.193.175:4559 , piece_id=46X42PJIPA6DVC7BG7WPKMPMC44ZBUROEAB5C22M5INXKBJC4KVQ, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.764555100 (202h56m21.195635221s) [noise=true , address=77.90.31.15:4732 , piece_id=4AU7UGKXN4TIPSIRR7IMXSF6WHELO6UTOX35N5X7JHOQQXWWJF2A, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.764757416 (202h56m21.195837547s) [noise=true , address=104.36.82.39:4753 , piece_id=R7BHT5RYLSCDZLTDDRXTSJFSRPLM5PNBYC6Q2G33C5SYIUCNVCLQ, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.765868993 (202h56m21.196949124s) [noise=true , address=104.36.82.50:4047 , piece_id=AKDACFQEPBAFGYKTWBL5RELLZL3L4IJHJT73NS7T7QPWK2C2TDQA, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.767152809 (202h56m21.19823294s) [noise=true , address=104.36.82.19:4436 , piece_id=KVDJTKON4LFVJSAY7XSJ2OQ4ZH67JLMDE2O4EHUWSMZ3EECWSOXA, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.768577886 (202h56m21.199658017s) [noise=true , address=45.91.116.10:4095 , piece_id=JQ2UARQSNKBGHHHBJ6PR45ETR4EYMCXPDEQHEGWFYH52LRJC57UQ, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:27.934309011 (202h56m22.365389142s) [upload=211101]: Next returned nil segment Aug 17 04:22:27.934342545 (202h56m22.365422666s) [upload=211101]: Waiting for error group managing segments... Aug 17 04:22:28.231985083 (202h56m22.663065214s) [noise=true , address=66.23.193.175:4559 , piece_id=46X42PJIPA6DVC7BG7WPKMPMC44ZBUROEAB5C22M5INXKBJC4KVQ, seg_pos={0 0}, upload=211101]: Done uploading piece. err: <nil> Aug 17 04:22:28.670701256 (202h56m23.101781427s) [noise=true , address=45.91.116.10:4095 , piece_id=JQ2UARQSNKBGHHHBJ6PR45ETR4EYMCXPDEQHEGWFYH52LRJC57UQ, seg_pos={0 0}, upload=211101]: Done uploading piece. err: <nil> Aug 17 04:22:28.671952431 (202h56m23.103032562s) [noise=true , address=45.91.116.5:4308 , piece_id=F52M3VMHRQWR5LIEEPMVKVBFMOVG76UNNGQTFKSF6ZBD7VEOFWVA, seg_pos={0 0}, upload=211101]: Done uploading piece. err: <nil> Aug 17 04:22:28.698468278 (202h56m23.129548409s) [noise=true , address=45.91.116.8:4067 , piece_id=EZAI7ZDUBOOWXYJNTB7ZPOUSBUUG2FKV77QVUOZMNM6TUJSKGYWA, seg_pos={0 0}, upload=211101]: Done uploading piece. err: <nil> Aug 17 04:22:28.720171187 (202h56m23.151251318s) [noise=true , address=45.91.116.13:4181 , piece_id=7VN6RQGOTBHQO6HXM5IFTU32PWNVPH6BJZMQNETJEXOYPR2Y7OCQ, seg_pos={0 0}, upload=211101]: Done uploading piece. err: <nil> Aug 17 04:22:28.731591429 (202h56m23.16267156s) [noise=true , address=45.91.116.14:4203 , piece_id=5LXGQ4JZD2KLW3OMWY5QYRE4MKQNKKSAIMESODKDDT2W33XD45DQ, seg_pos={0 0}, upload=211101]: Done uploading piece. err: <nil> Aug 17 04:22:28.798148986 (202h56m23.229229117s) [noise=true , address=77.90.31.18:4815 , piece_id=G6PINAFFA6DZRRMKU2XK2LYFOES5WG7PTJ5L2ZPUQHMA4JNSRZRQ, seg_pos={0 0}, upload=211101]: Done uploading piece. err: <nil> Aug 17 04:22:28.804849037 (202h56m23.235929168s) [noise=true , address=79.127.201.215:4345 , piece_id=JMSKNIZ7QPGBXIEP5RA3S74XAU5NJ2MUUXSFFIAK377YFGO646YQ, seg_pos={0 0}, upload=211101]: Done uploading piece. err: <nil> Aug 17 04:22:28.833066822 (202h56m23.264147013s) [noise=true , address=77.90.31.15:4732 , piece_id=4AU7UGKXN4TIPSIRR7IMXSF6WHELO6UTOX35N5X7JHOQQXWWJF2A, seg_pos={0 0}, upload=211101]: Done uploading piece. err: <nil> Aug 17 04:22:28.846684710 (202h56m23.277764831s) [noise=true , address=77.90.31.11:4562 , piece_id=PUL6TAVIIU7732IOASNH5EHQMNCK7DWHJMUJVXRNK74T4GUQDGTA, seg_pos={0 0}, upload=211101]: Done uploading piece. err: <nil> Aug 17 04:23:25.582810947 (202h57m20.013891078s) [noise=true , address=104.36.82.34:4558 , piece_id=XGFNEBBVZX5UWJN7DALGM2EDG26POCTSBBAUU7DLYQKS2DNSBNXA, seg_pos={0 0}, upload=211101]: Done uploading piece. err: ecclient: upload failed (node:12TU4zvu38mF1fNSergCtMj2cK9eu57Rdfpt5MVk3tc4MZF6dNr, address:104.36.82.34:4558): protocol: rpc: dial tcp 104.36.82.34:4558: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:25.582856475 (202h57m20.013936606s) [noise=true , address=104.36.82.35:4594 , piece_id=6NUSPNYGSSJOB2CNPIW6DI45MPZARLWDBXH3CWMP4LXMDB2CXWZQ, seg_pos={0 0}, upload=211101]: Done uploading piece. err: ecclient: upload failed (node:12D9nh4XFmsjjFcZEMjaL4uw4Whsa8nfMrAsYPu6NVpsxgb3jvf, address:104.36.82.35:4594): protocol: rpc: dial tcp 104.36.82.35:4594: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:25.582973238 (202h57m20.014053369s) [noise=true , address=104.36.82.19:4436 , piece_id=KVDJTKON4LFVJSAY7XSJ2OQ4ZH67JLMDE2O4EHUWSMZ3EECWSOXA, seg_pos={0 0}, upload=211101]: Done uploading piece. err: ecclient: upload failed (node:1WvyEw8XevCiQ8yRgzyMo7WTVzpEJ158ErKfAeV4dSTBkS1oRF, address:104.36.82.19:4436): protocol: rpc: dial tcp 104.36.82.19:4436: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:25.583033393 (202h57m20.014113524s) [noise=true , address=104.36.82.58:4283 , piece_id=LJKEH2KDSOFPVFPB5UWGVLOHQVZQOY33MYSHFLP4ZWGPKZ4KC64A, seg_pos={0 0}, upload=211101]: Done uploading piece. err: ecclient: upload failed (node:1p9jkPtQnmbCXBmw6HSHDWpGaKWqfg87m6hHXy6WDu2giAFnbQ, address:104.36.82.58:4283): protocol: rpc: dial tcp 104.36.82.58:4283: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:25.583069241 (202h57m20.014149362s) [noise=true , address=104.36.82.50:4047 , piece_id=AKDACFQEPBAFGYKTWBL5RELLZL3L4IJHJT73NS7T7QPWK2C2TDQA, seg_pos={0 0}, upload=211101]: Done uploading piece. err: ecclient: upload failed (node:126Z6tvp49dLLdj3T5snJc4tBVGzcX5UBaGafFN3PkhF92vxpfs, address:104.36.82.50:4047): protocol: rpc: dial tcp 104.36.82.50:4047: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:25.583076716 (202h57m20.014156837s) [noise=true , address=104.36.82.39:4753 , piece_id=R7BHT5RYLSCDZLTDDRXTSJFSRPLM5PNBYC6Q2G33C5SYIUCNVCLQ, seg_pos={0 0}, upload=211101]: Done uploading piece. err: ecclient: upload failed (node:1zzW2z6noGQT6xdSmwSQBNYwip6gZjnmqZg7qdb6BP3nHyCLFz, address:104.36.82.39:4753): protocol: rpc: dial tcp 104.36.82.39:4753: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:25.583107535 (202h57m20.014187666s) [noise=true , address=104.36.82.23:4474 , piece_id=RITPHITVW655UQETT6UKS7BZO2S3F4P2MJIO4REOUKYYP4E2CDVQ, seg_pos={0 0}, upload=211101]: Done uploading piece. err: ecclient: upload failed (node:1Fpshye6yGAo3yf71eVpxrCdvoRjpkGX9WEX8KCWYbvuhy5i9j, address:104.36.82.23:4474): protocol: write tcp 188.166.225.170:36642->104.36.82.23:4474: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:25.583165355 (202h57m20.014245487s) [noise=true , address=104.36.82.5:4855 , piece_id=3FCDYA6CRYOOU3MDFJPEC4YQNICFVCN2A47RIYPSR4ZX5TIXZX3A, seg_pos={0 0}, upload=211101]: Done uploading piece. err: ecclient: upload failed (node:1KDNuE7YY8smhDjotushQvY3y9NtLxigbJELHD4Edxy8Qi4BWh, address:104.36.82.5:4855): protocol: write tcp 188.166.225.170:47052->104.36.82.5:4855: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:25.583194000 (202h57m20.014274131s) [seg_pos={0 0}, upload=211101]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Aug 17 04:23:25.810738626 (202h57m20.241818757s) [upload=211101]: Done uploading segments... Aug 17 04:23:25.810761991 (202h57m20.241842122s) [upload=211101]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 5.756307241s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 9 out of 60 is failed (threshold=3) [...............X..........XXXX.X.X.......X...X..............] duration: 5.777134413s FAIL (2025-08-17 04:48:39 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/4M") ======LOGS====== Aug 17 04:37:54.182428668 (203h11m48.613508799s) [upload=211361]: Starting upload Aug 17 04:37:54.182474656 (203h11m48.613554777s) [upload=211361]: Done starting upload Aug 17 04:37:54.182665000 (203h11m48.613745131s) [upload=211361]: Uploading segments... Aug 17 04:37:54.182728421 (203h11m48.613808552s) [upload=211361]: Got next segment. Inline: false Aug 17 04:37:54.182758048 (203h11m48.613838179s) [upload=211361]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 17 04:37:54.182762306 (203h11m48.613842437s) [upload=211361]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 17 04:37:54.737160222 (203h11m49.168240353s) [seg_pos={0 0}, upload=211361]: Begin upload segment... Aug 17 04:37:54.737203494 (203h11m49.168283625s) [seg_pos={0 0}, upload=211361]: Done begin upload segment. Aug 17 04:37:54.737224945 (203h11m49.168305076s) [noise=true , address=77.90.31.19:4863 , piece_id=PYERWDVDOTOW2SVDUO63ODTEWMWUGH5OMV2PE7RJTIDVFTF2ZRVQ, seg_pos={0 0}, upload=211361]: Uploading piece... Aug 17 04:37:54.737311591 (203h11m49.168391722s) [noise=true , address=79.127.220.101:4068 , piece_id=6MFJ22JL437NTOH25E3LNH3Q2LQNKKMUT5Q43F2XNIB7346LGFCQ, seg_pos={0 0}, upload=211361]: Uploading piece... Aug 17 04:37:54.737638917 (203h11m49.168719048s) [noise=true , address=77.90.31.12:4601 , piece_id=JC5VIGLKCWTCDHQYWEL6QSRZW3MJZFRFQKRJH54OK7LPUQH7WQTQ, seg_pos={0 0}, upload=211361]: Uploading piece... Aug 17 04:37:54.737939432 (203h11m49.169019563s) [noise=true , address=66.23.193.175:4562 , piece_id=2RZ2A5KBLZBLEG6UECY2GTNF5THR3XJZMS34ULG5SQNWN4O3XNMA, seg_pos={0 0}, upload=211361]: Uploading piece... Aug 17 04:37:54.737952067 (203h11m49.169032198s) [noise=true , address=77.90.31.16:4761 , piece_id=44YO7GYP4RNRWHPP7LP2C4FJBGMWOLEDEC2NN7TAWU3YFHAST3XQ, seg_pos={0 0}, upload=211361]: Uploading piece... Aug 17 04:37:54.738229999 (203h11m49.16931013s) [noise=true , address=104.36.82.36:4648 , piece_id=GTHXHUS6GR26G6UHHMB35MCSEI7I5UJ6K65O76HRJI4WVPIHD7RQ, seg_pos={0 0}, upload=211361]: Uploading piece... Aug 17 04:37:54.738324038 (203h11m49.169404169s) [noise=true , address=66.23.193.175:4563 , piece_id=I65W2GP5RTRZUIT7Y6NSEAL7XC726PP3BYMCWS6UMZEOWAP5CP2A, seg_pos={0 0}, upload=211361]: Uploading piece... Aug 17 04:37:54.738511367 (203h11m49.169591488s) [noise=true , address=104.36.82.4:4833 , piece_id=AMRE5SIA7EJLAP55GFYILJYZJL5624E534TPWFUEQILHG7CCFDXQ, seg_pos={0 0}, upload=211361]: Uploading piece... Aug 17 04:37:54.738655813 (203h11m49.169735954s) [noise=true , address=104.36.82.5:4844 , piece_id=PULFFEXZLSMOYEFGRK6L2POF7ERJQT75J23JT3CTXKZ3LMARQZHQ, seg_pos={0 0}, upload=211361]: Uploading piece... Aug 17 04:37:54.738747769 (203h11m49.1698279s) [noise=true , address=77.90.31.10:4532 , piece_id=5IBCLJNM6LUBCIOPSAERY5HAXV66CR4OKI45RBGOHALTRFYGLMWQ, seg_pos={0 0}, upload=211361]: Uploading piece... Aug 17 04:37:54.738971367 (203h11m49.170051498s) [noise=true , address=104.36.82.60:4360 , piece_id=W4OJU6NRV2HKPC7JDTISAPVT77KIC3JHVJR5KHYY225FLH4PZBBA, seg_pos={0 0}, upload=211361]: Uploading piece... Aug 17 04:37:54.738975705 (203h11m49.170055836s) [noise=true , address=77.90.31.18:4830 , piece_id=QNQF4JAEHOFWRTL6XWKL42XSKA2DMO6Z4CZO5NF337NNSHCMYYBA, seg_pos={0 0}, upload=211361]: Uploading piece... Aug 17 04:37:54.739194995 (203h11m49.170275126s) [noise=true , address=104.36.82.23:4468 , piece_id=NC2BFOASSHA4ZG46LZONWF23DNN3ZZBA6SFO3Q44KD4TCOH4AIQQ, seg_pos={0 0}, upload=211361]: Uploading piece... Aug 17 04:37:54.739278324 (203h11m49.170358445s) [noise=true , address=104.36.82.39:4747 , piece_id=SF2P47ED5FEOCMJ6R6AXP2NTI5LYYALSDLHINNLDVMUMZIE4YLHQ, seg_pos={0 0}, upload=211361]: Uploading piece... Aug 17 04:37:54.739421147 (203h11m49.170501278s) [noise=true , address=104.36.82.3:4800 , piece_id=EKC43HOMYNG77PDESU3QV5OL44HWYQRAN5RUNIB7IMC6NVXQWFVA, seg_pos={0 0}, upload=211361]: Uploading piece... Aug 17 04:37:54.739561055 (203h11m49.170641186s) [noise=true , address=77.90.31.13:4634 , piece_id=ZXX5E54TSYUG557OGOCBQMBKS35TUG6OPDOIG3TTS3OYGTRA24XA, seg_pos={0 0}, upload=211361]: Uploading piece... Aug 17 04:37:54.739643824 (203h11m49.170723955s) [noise=true , address=104.36.82.58:4286 , piece_id=JDBFFYOZPYJ5FXJQIVEBB5IWSBGT6LCBUZF5O25OKWFRXROPHS3Q, seg_pos={0 0}, upload=211361]: Uploading piece... Aug 17 04:37:54.739852683 (203h11m49.170932814s) [noise=true , address=104.36.82.44:4876 , piece_id=H53T7WMGX4CDVQNCYOG6NOYFVRHD5Z35LNF37FVKSWYIVJS2EFZA, seg_pos={0 0}, upload=211361]: Uploading piece... Aug 17 04:37:55.861243533 (203h11m50.292323664s) [upload=211361]: Next returned nil segment Aug 17 04:37:55.861256768 (203h11m50.292336899s) [upload=211361]: Waiting for error group managing segments... Aug 17 04:37:56.165335502 (203h11m50.596415633s) [noise=true , address=66.23.193.175:4562 , piece_id=2RZ2A5KBLZBLEG6UECY2GTNF5THR3XJZMS34ULG5SQNWN4O3XNMA, seg_pos={0 0}, upload=211361]: Done uploading piece. err: <nil> Aug 17 04:37:56.263548583 (203h11m50.694628715s) [noise=true , address=66.23.193.175:4563 , piece_id=I65W2GP5RTRZUIT7Y6NSEAL7XC726PP3BYMCWS6UMZEOWAP5CP2A, seg_pos={0 0}, upload=211361]: Done uploading piece. err: <nil> Aug 17 04:37:56.569392606 (203h11m51.000472737s) [noise=true , address=79.127.220.101:4068 , piece_id=6MFJ22JL437NTOH25E3LNH3Q2LQNKKMUT5Q43F2XNIB7346LGFCQ, seg_pos={0 0}, upload=211361]: Done uploading piece. err: <nil> Aug 17 04:37:56.620901052 (203h11m51.051981183s) [noise=true , address=77.90.31.12:4601 , piece_id=JC5VIGLKCWTCDHQYWEL6QSRZW3MJZFRFQKRJH54OK7LPUQH7WQTQ, seg_pos={0 0}, upload=211361]: Done uploading piece. err: <nil> Aug 17 04:37:56.639713969 (203h11m51.0707941s) [noise=true , address=77.90.31.16:4761 , piece_id=44YO7GYP4RNRWHPP7LP2C4FJBGMWOLEDEC2NN7TAWU3YFHAST3XQ, seg_pos={0 0}, upload=211361]: Done uploading piece. err: <nil> Aug 17 04:37:56.649068307 (203h11m51.080148438s) [noise=true , address=104.36.82.4:4833 , piece_id=AMRE5SIA7EJLAP55GFYILJYZJL5624E534TPWFUEQILHG7CCFDXQ, seg_pos={0 0}, upload=211361]: Done uploading piece. err: <nil> Aug 17 04:37:56.701677831 (203h11m51.132757962s) [noise=true , address=104.36.82.3:4800 , piece_id=EKC43HOMYNG77PDESU3QV5OL44HWYQRAN5RUNIB7IMC6NVXQWFVA, seg_pos={0 0}, upload=211361]: Done uploading piece. err: <nil> Aug 17 04:37:56.719713991 (203h11m51.150794132s) [noise=true , address=104.36.82.23:4468 , piece_id=NC2BFOASSHA4ZG46LZONWF23DNN3ZZBA6SFO3Q44KD4TCOH4AIQQ, seg_pos={0 0}, upload=211361]: Done uploading piece. err: <nil> Aug 17 04:37:56.730632311 (203h11m51.161712432s) [noise=true , address=104.36.82.39:4747 , piece_id=SF2P47ED5FEOCMJ6R6AXP2NTI5LYYALSDLHINNLDVMUMZIE4YLHQ, seg_pos={0 0}, upload=211361]: Done uploading piece. err: <nil> Aug 17 04:37:56.761149974 (203h11m51.192230105s) [noise=true , address=104.36.82.60:4360 , piece_id=W4OJU6NRV2HKPC7JDTISAPVT77KIC3JHVJR5KHYY225FLH4PZBBA, seg_pos={0 0}, upload=211361]: Done uploading piece. err: <nil> Aug 17 04:37:56.775738719 (203h11m51.20681885s) [noise=true , address=77.90.31.18:4830 , piece_id=QNQF4JAEHOFWRTL6XWKL42XSKA2DMO6Z4CZO5NF337NNSHCMYYBA, seg_pos={0 0}, upload=211361]: Done uploading piece. err: <nil> Aug 17 04:37:56.779316274 (203h11m51.210396405s) [noise=true , address=77.90.31.13:4634 , piece_id=ZXX5E54TSYUG557OGOCBQMBKS35TUG6OPDOIG3TTS3OYGTRA24XA, seg_pos={0 0}, upload=211361]: Done uploading piece. err: <nil> Aug 17 04:37:56.802147800 (203h11m51.233227961s) [noise=true , address=104.36.82.58:4286 , piece_id=JDBFFYOZPYJ5FXJQIVEBB5IWSBGT6LCBUZF5O25OKWFRXROPHS3Q, seg_pos={0 0}, upload=211361]: Done uploading piece. err: <nil> Aug 17 04:37:56.832614886 (203h11m51.263695017s) [noise=true , address=104.36.82.44:4876 , piece_id=H53T7WMGX4CDVQNCYOG6NOYFVRHD5Z35LNF37FVKSWYIVJS2EFZA, seg_pos={0 0}, upload=211361]: Done uploading piece. err: <nil> Aug 17 04:37:56.832646346 (203h11m51.263726467s) [seg_pos={0 0}, upload=211361]: Segment reached optimal threshold of 14 pieces. Aug 17 04:37:56.832765443 (203h11m51.263845574s) [noise=true , address=77.90.31.10:4532 , piece_id=5IBCLJNM6LUBCIOPSAERY5HAXV66CR4OKI45RBGOHALTRFYGLMWQ, seg_pos={0 0}, upload=211361]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1T6kkKvTNR5MZZijjr1vMU6g75YkoRpzMvGSgQCqRnfCm9JULe): protocol: expected piece hash; context canceled; EOF Aug 17 04:37:56.832872889 (203h11m51.26395302s) [noise=true , address=104.36.82.36:4648 , piece_id=GTHXHUS6GR26G6UHHMB35MCSEI7I5UJ6K65O76HRJI4WVPIHD7RQ, seg_pos={0 0}, upload=211361]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XHNZDY7xKZ2bss1rSdjbGP3ugSwGRwRzBSWHfXgdKKz794wMq): protocol: expected piece hash; context canceled; EOF Aug 17 04:37:56.832892156 (203h11m51.263972287s) [noise=true , address=77.90.31.19:4863 , piece_id=PYERWDVDOTOW2SVDUO63ODTEWMWUGH5OMV2PE7RJTIDVFTF2ZRVQ, seg_pos={0 0}, upload=211361]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124x1cbCPJRK8YkgSP6UzHymoLu1E1znnfg5WBiD89kSDLi8JeL): protocol: expected piece hash; context canceled; EOF Aug 17 04:37:56.832974564 (203h11m51.264054695s) [noise=true , address=104.36.82.5:4844 , piece_id=PULFFEXZLSMOYEFGRK6L2POF7ERJQT75J23JT3CTXKZ3LMARQZHQ, seg_pos={0 0}, upload=211361]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12iovpNdbidkiv9mEHaAPjBubGmoqr4sUJWm8Cb64LzS6wuwLWa): protocol: write tcp 188.166.225.170:59826->104.36.82.5:4844: use of closed network connection; write tcp 188.166.225.170:59826->104.36.82.5:4844: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59826->104.36.82.5:4844: use of closed network connection Aug 17 04:37:56.833025241 (203h11m51.264105372s) [seg_pos={0 0}, upload=211361]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 17 04:37:56.833116014 (203h11m51.264196135s) [upload=211361]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 17 04:37:56.833120332 (203h11m51.264200463s) [upload=211361]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 17 04:37:58.261163702 (203h11m52.692243834s) [upload=211361]: Done uploading segments... Aug 17 04:37:58.261181997 (203h11m52.692262118s) [upload=211361]: Upload finished. err: <nil> Duration: 5.77715983s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.....................XXXX...................................] duration: 6.487789089s FAIL (2025-08-17 04:48:34 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.487807104s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [......................XXXX..................................] duration: 6.358473486s FAIL (2025-08-17 04:48:17 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.358491141s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [......................XXX...................................] duration: 5.677970531s FAIL (2025-08-17 04:48:13 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== g_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.761278730 (202h56m21.192358861s) [noise=true , address=104.36.82.34:4558 , piece_id=XGFNEBBVZX5UWJN7DALGM2EDG26POCTSBBAUU7DLYQKS2DNSBNXA, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.761661883 (202h56m21.192742014s) [noise=true , address=104.36.82.35:4594 , piece_id=6NUSPNYGSSJOB2CNPIW6DI45MPZARLWDBXH3CWMP4LXMDB2CXWZQ, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.761716548 (202h56m21.192796679s) [noise=true , address=45.91.116.8:4067 , piece_id=EZAI7ZDUBOOWXYJNTB7ZPOUSBUUG2FKV77QVUOZMNM6TUJSKGYWA, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.761972016 (202h56m21.193052147s) [noise=true , address=104.36.82.58:4283 , piece_id=LJKEH2KDSOFPVFPB5UWGVLOHQVZQOY33MYSHFLP4ZWGPKZ4KC64A, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.762254878 (202h56m21.193335009s) [noise=true , address=79.127.201.215:4345 , piece_id=JMSKNIZ7QPGBXIEP5RA3S74XAU5NJ2MUUXSFFIAK377YFGO646YQ, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.762523682 (202h56m21.193603803s) [noise=true , address=45.91.116.13:4181 , piece_id=7VN6RQGOTBHQO6HXM5IFTU32PWNVPH6BJZMQNETJEXOYPR2Y7OCQ, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.762864294 (202h56m21.193944415s) [noise=true , address=45.91.116.14:4203 , piece_id=5LXGQ4JZD2KLW3OMWY5QYRE4MKQNKKSAIMESODKDDT2W33XD45DQ, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.763179537 (202h56m21.194259668s) [noise=true , address=77.90.31.18:4815 , piece_id=G6PINAFFA6DZRRMKU2XK2LYFOES5WG7PTJ5L2ZPUQHMA4JNSRZRQ, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.763478499 (202h56m21.19455864s) [noise=true , address=77.90.31.11:4562 , piece_id=PUL6TAVIIU7732IOASNH5EHQMNCK7DWHJMUJVXRNK74T4GUQDGTA, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.763757683 (202h56m21.194837814s) [noise=true , address=104.36.82.5:4855 , piece_id=3FCDYA6CRYOOU3MDFJPEC4YQNICFVCN2A47RIYPSR4ZX5TIXZX3A, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.763988195 (202h56m21.195068326s) [noise=true , address=45.91.116.5:4308 , piece_id=F52M3VMHRQWR5LIEEPMVKVBFMOVG76UNNGQTFKSF6ZBD7VEOFWVA, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.764273320 (202h56m21.195353451s) [noise=true , address=66.23.193.175:4559 , piece_id=46X42PJIPA6DVC7BG7WPKMPMC44ZBUROEAB5C22M5INXKBJC4KVQ, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.764555100 (202h56m21.195635221s) [noise=true , address=77.90.31.15:4732 , piece_id=4AU7UGKXN4TIPSIRR7IMXSF6WHELO6UTOX35N5X7JHOQQXWWJF2A, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.764757416 (202h56m21.195837547s) [noise=true , address=104.36.82.39:4753 , piece_id=R7BHT5RYLSCDZLTDDRXTSJFSRPLM5PNBYC6Q2G33C5SYIUCNVCLQ, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.765868993 (202h56m21.196949124s) [noise=true , address=104.36.82.50:4047 , piece_id=AKDACFQEPBAFGYKTWBL5RELLZL3L4IJHJT73NS7T7QPWK2C2TDQA, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.767152809 (202h56m21.19823294s) [noise=true , address=104.36.82.19:4436 , piece_id=KVDJTKON4LFVJSAY7XSJ2OQ4ZH67JLMDE2O4EHUWSMZ3EECWSOXA, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.768577886 (202h56m21.199658017s) [noise=true , address=45.91.116.10:4095 , piece_id=JQ2UARQSNKBGHHHBJ6PR45ETR4EYMCXPDEQHEGWFYH52LRJC57UQ, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:27.934309011 (202h56m22.365389142s) [upload=211101]: Next returned nil segment Aug 17 04:22:27.934342545 (202h56m22.365422666s) [upload=211101]: Waiting for error group managing segments... Aug 17 04:22:28.231985083 (202h56m22.663065214s) [noise=true , address=66.23.193.175:4559 , piece_id=46X42PJIPA6DVC7BG7WPKMPMC44ZBUROEAB5C22M5INXKBJC4KVQ, seg_pos={0 0}, upload=211101]: Done uploading piece. err: <nil> Aug 17 04:22:28.670701256 (202h56m23.101781427s) [noise=true , address=45.91.116.10:4095 , piece_id=JQ2UARQSNKBGHHHBJ6PR45ETR4EYMCXPDEQHEGWFYH52LRJC57UQ, seg_pos={0 0}, upload=211101]: Done uploading piece. err: <nil> Aug 17 04:22:28.671952431 (202h56m23.103032562s) [noise=true , address=45.91.116.5:4308 , piece_id=F52M3VMHRQWR5LIEEPMVKVBFMOVG76UNNGQTFKSF6ZBD7VEOFWVA, seg_pos={0 0}, upload=211101]: Done uploading piece. err: <nil> Aug 17 04:22:28.698468278 (202h56m23.129548409s) [noise=true , address=45.91.116.8:4067 , piece_id=EZAI7ZDUBOOWXYJNTB7ZPOUSBUUG2FKV77QVUOZMNM6TUJSKGYWA, seg_pos={0 0}, upload=211101]: Done uploading piece. err: <nil> Aug 17 04:22:28.720171187 (202h56m23.151251318s) [noise=true , address=45.91.116.13:4181 , piece_id=7VN6RQGOTBHQO6HXM5IFTU32PWNVPH6BJZMQNETJEXOYPR2Y7OCQ, seg_pos={0 0}, upload=211101]: Done uploading piece. err: <nil> Aug 17 04:22:28.731591429 (202h56m23.16267156s) [noise=true , address=45.91.116.14:4203 , piece_id=5LXGQ4JZD2KLW3OMWY5QYRE4MKQNKKSAIMESODKDDT2W33XD45DQ, seg_pos={0 0}, upload=211101]: Done uploading piece. err: <nil> Aug 17 04:22:28.798148986 (202h56m23.229229117s) [noise=true , address=77.90.31.18:4815 , piece_id=G6PINAFFA6DZRRMKU2XK2LYFOES5WG7PTJ5L2ZPUQHMA4JNSRZRQ, seg_pos={0 0}, upload=211101]: Done uploading piece. err: <nil> Aug 17 04:22:28.804849037 (202h56m23.235929168s) [noise=true , address=79.127.201.215:4345 , piece_id=JMSKNIZ7QPGBXIEP5RA3S74XAU5NJ2MUUXSFFIAK377YFGO646YQ, seg_pos={0 0}, upload=211101]: Done uploading piece. err: <nil> Aug 17 04:22:28.833066822 (202h56m23.264147013s) [noise=true , address=77.90.31.15:4732 , piece_id=4AU7UGKXN4TIPSIRR7IMXSF6WHELO6UTOX35N5X7JHOQQXWWJF2A, seg_pos={0 0}, upload=211101]: Done uploading piece. err: <nil> Aug 17 04:22:28.846684710 (202h56m23.277764831s) [noise=true , address=77.90.31.11:4562 , piece_id=PUL6TAVIIU7732IOASNH5EHQMNCK7DWHJMUJVXRNK74T4GUQDGTA, seg_pos={0 0}, upload=211101]: Done uploading piece. err: <nil> Aug 17 04:23:25.582810947 (202h57m20.013891078s) [noise=true , address=104.36.82.34:4558 , piece_id=XGFNEBBVZX5UWJN7DALGM2EDG26POCTSBBAUU7DLYQKS2DNSBNXA, seg_pos={0 0}, upload=211101]: Done uploading piece. err: ecclient: upload failed (node:12TU4zvu38mF1fNSergCtMj2cK9eu57Rdfpt5MVk3tc4MZF6dNr, address:104.36.82.34:4558): protocol: rpc: dial tcp 104.36.82.34:4558: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:25.582856475 (202h57m20.013936606s) [noise=true , address=104.36.82.35:4594 , piece_id=6NUSPNYGSSJOB2CNPIW6DI45MPZARLWDBXH3CWMP4LXMDB2CXWZQ, seg_pos={0 0}, upload=211101]: Done uploading piece. err: ecclient: upload failed (node:12D9nh4XFmsjjFcZEMjaL4uw4Whsa8nfMrAsYPu6NVpsxgb3jvf, address:104.36.82.35:4594): protocol: rpc: dial tcp 104.36.82.35:4594: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:25.582973238 (202h57m20.014053369s) [noise=true , address=104.36.82.19:4436 , piece_id=KVDJTKON4LFVJSAY7XSJ2OQ4ZH67JLMDE2O4EHUWSMZ3EECWSOXA, seg_pos={0 0}, upload=211101]: Done uploading piece. err: ecclient: upload failed (node:1WvyEw8XevCiQ8yRgzyMo7WTVzpEJ158ErKfAeV4dSTBkS1oRF, address:104.36.82.19:4436): protocol: rpc: dial tcp 104.36.82.19:4436: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:25.583033393 (202h57m20.014113524s) [noise=true , address=104.36.82.58:4283 , piece_id=LJKEH2KDSOFPVFPB5UWGVLOHQVZQOY33MYSHFLP4ZWGPKZ4KC64A, seg_pos={0 0}, upload=211101]: Done uploading piece. err: ecclient: upload failed (node:1p9jkPtQnmbCXBmw6HSHDWpGaKWqfg87m6hHXy6WDu2giAFnbQ, address:104.36.82.58:4283): protocol: rpc: dial tcp 104.36.82.58:4283: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:25.583069241 (202h57m20.014149362s) [noise=true , address=104.36.82.50:4047 , piece_id=AKDACFQEPBAFGYKTWBL5RELLZL3L4IJHJT73NS7T7QPWK2C2TDQA, seg_pos={0 0}, upload=211101]: Done uploading piece. err: ecclient: upload failed (node:126Z6tvp49dLLdj3T5snJc4tBVGzcX5UBaGafFN3PkhF92vxpfs, address:104.36.82.50:4047): protocol: rpc: dial tcp 104.36.82.50:4047: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:25.583076716 (202h57m20.014156837s) [noise=true , address=104.36.82.39:4753 , piece_id=R7BHT5RYLSCDZLTDDRXTSJFSRPLM5PNBYC6Q2G33C5SYIUCNVCLQ, seg_pos={0 0}, upload=211101]: Done uploading piece. err: ecclient: upload failed (node:1zzW2z6noGQT6xdSmwSQBNYwip6gZjnmqZg7qdb6BP3nHyCLFz, address:104.36.82.39:4753): protocol: rpc: dial tcp 104.36.82.39:4753: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:25.583107535 (202h57m20.014187666s) [noise=true , address=104.36.82.23:4474 , piece_id=RITPHITVW655UQETT6UKS7BZO2S3F4P2MJIO4REOUKYYP4E2CDVQ, seg_pos={0 0}, upload=211101]: Done uploading piece. err: ecclient: upload failed (node:1Fpshye6yGAo3yf71eVpxrCdvoRjpkGX9WEX8KCWYbvuhy5i9j, address:104.36.82.23:4474): protocol: write tcp 188.166.225.170:36642->104.36.82.23:4474: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:25.583165355 (202h57m20.014245487s) [noise=true , address=104.36.82.5:4855 , piece_id=3FCDYA6CRYOOU3MDFJPEC4YQNICFVCN2A47RIYPSR4ZX5TIXZX3A, seg_pos={0 0}, upload=211101]: Done uploading piece. err: ecclient: upload failed (node:1KDNuE7YY8smhDjotushQvY3y9NtLxigbJELHD4Edxy8Qi4BWh, address:104.36.82.5:4855): protocol: write tcp 188.166.225.170:47052->104.36.82.5:4855: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:25.583194000 (202h57m20.014274131s) [seg_pos={0 0}, upload=211101]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Aug 17 04:23:25.810738626 (202h57m20.241818757s) [upload=211101]: Done uploading segments... Aug 17 04:23:25.810761991 (202h57m20.241842122s) [upload=211101]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 5.677996421s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [..........................................X.....X...........] duration: 1.924283854s FAIL (2025-08-17 04:48:02 UTC) orbiter warning: operation: ListObjects ======ERROR====== unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/1K ======LOGS====== Aug 17 04:41:26.040732864 (203h15m20.471812985s) [upload=211420]: Starting upload Aug 17 04:41:26.040780384 (203h15m20.471860515s) [upload=211420]: Done starting upload Aug 17 04:41:26.040828978 (203h15m20.471909109s) [upload=211420]: Uploading segments... Aug 17 04:41:26.040863473 (203h15m20.471943604s) [upload=211420]: Got next segment. Inline: true Aug 17 04:41:26.040867290 (203h15m20.471947422s) [upload=211420]: Waiting for error group managing segments... Aug 17 04:41:26.040897880 (203h15m20.471978011s) [upload=211420]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 17 04:41:26.040901597 (203h15m20.471981728s) [upload=211420]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 17 04:41:26.040904041 (203h15m20.471984162s) [upload=211420]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 17 04:41:26.462140724 (203h15m20.893220855s) [upload=211420]: Done uploading segments... Aug 17 04:41:26.462159310 (203h15m20.893239431s) [upload=211420]: Upload finished. err: <nil> Duration:* 1.924317439s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.......................XXXX.................................] duration: 5.669627057s FAIL (2025-08-17 04:47:54 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.669651355s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 9 out of 60 is failed (threshold=3) [................X..........XXXX.X.X.......X...X.............] duration: 6.126944802s FAIL (2025-08-17 04:47:47 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/4M") ======LOGS====== Aug 17 04:37:54.182428668 (203h11m48.613508799s) [upload=211361]: Starting upload Aug 17 04:37:54.182474656 (203h11m48.613554777s) [upload=211361]: Done starting upload Aug 17 04:37:54.182665000 (203h11m48.613745131s) [upload=211361]: Uploading segments... Aug 17 04:37:54.182728421 (203h11m48.613808552s) [upload=211361]: Got next segment. Inline: false Aug 17 04:37:54.182758048 (203h11m48.613838179s) [upload=211361]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 17 04:37:54.182762306 (203h11m48.613842437s) [upload=211361]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 17 04:37:54.737160222 (203h11m49.168240353s) [seg_pos={0 0}, upload=211361]: Begin upload segment... Aug 17 04:37:54.737203494 (203h11m49.168283625s) [seg_pos={0 0}, upload=211361]: Done begin upload segment. Aug 17 04:37:54.737224945 (203h11m49.168305076s) [noise=true , address=77.90.31.19:4863 , piece_id=PYERWDVDOTOW2SVDUO63ODTEWMWUGH5OMV2PE7RJTIDVFTF2ZRVQ, seg_pos={0 0}, upload=211361]: Uploading piece... Aug 17 04:37:54.737311591 (203h11m49.168391722s) [noise=true , address=79.127.220.101:4068 , piece_id=6MFJ22JL437NTOH25E3LNH3Q2LQNKKMUT5Q43F2XNIB7346LGFCQ, seg_pos={0 0}, upload=211361]: Uploading piece... Aug 17 04:37:54.737638917 (203h11m49.168719048s) [noise=true , address=77.90.31.12:4601 , piece_id=JC5VIGLKCWTCDHQYWEL6QSRZW3MJZFRFQKRJH54OK7LPUQH7WQTQ, seg_pos={0 0}, upload=211361]: Uploading piece... Aug 17 04:37:54.737939432 (203h11m49.169019563s) [noise=true , address=66.23.193.175:4562 , piece_id=2RZ2A5KBLZBLEG6UECY2GTNF5THR3XJZMS34ULG5SQNWN4O3XNMA, seg_pos={0 0}, upload=211361]: Uploading piece... Aug 17 04:37:54.737952067 (203h11m49.169032198s) [noise=true , address=77.90.31.16:4761 , piece_id=44YO7GYP4RNRWHPP7LP2C4FJBGMWOLEDEC2NN7TAWU3YFHAST3XQ, seg_pos={0 0}, upload=211361]: Uploading piece... Aug 17 04:37:54.738229999 (203h11m49.16931013s) [noise=true , address=104.36.82.36:4648 , piece_id=GTHXHUS6GR26G6UHHMB35MCSEI7I5UJ6K65O76HRJI4WVPIHD7RQ, seg_pos={0 0}, upload=211361]: Uploading piece... Aug 17 04:37:54.738324038 (203h11m49.169404169s) [noise=true , address=66.23.193.175:4563 , piece_id=I65W2GP5RTRZUIT7Y6NSEAL7XC726PP3BYMCWS6UMZEOWAP5CP2A, seg_pos={0 0}, upload=211361]: Uploading piece... Aug 17 04:37:54.738511367 (203h11m49.169591488s) [noise=true , address=104.36.82.4:4833 , piece_id=AMRE5SIA7EJLAP55GFYILJYZJL5624E534TPWFUEQILHG7CCFDXQ, seg_pos={0 0}, upload=211361]: Uploading piece... Aug 17 04:37:54.738655813 (203h11m49.169735954s) [noise=true , address=104.36.82.5:4844 , piece_id=PULFFEXZLSMOYEFGRK6L2POF7ERJQT75J23JT3CTXKZ3LMARQZHQ, seg_pos={0 0}, upload=211361]: Uploading piece... Aug 17 04:37:54.738747769 (203h11m49.1698279s) [noise=true , address=77.90.31.10:4532 , piece_id=5IBCLJNM6LUBCIOPSAERY5HAXV66CR4OKI45RBGOHALTRFYGLMWQ, seg_pos={0 0}, upload=211361]: Uploading piece... Aug 17 04:37:54.738971367 (203h11m49.170051498s) [noise=true , address=104.36.82.60:4360 , piece_id=W4OJU6NRV2HKPC7JDTISAPVT77KIC3JHVJR5KHYY225FLH4PZBBA, seg_pos={0 0}, upload=211361]: Uploading piece... Aug 17 04:37:54.738975705 (203h11m49.170055836s) [noise=true , address=77.90.31.18:4830 , piece_id=QNQF4JAEHOFWRTL6XWKL42XSKA2DMO6Z4CZO5NF337NNSHCMYYBA, seg_pos={0 0}, upload=211361]: Uploading piece... Aug 17 04:37:54.739194995 (203h11m49.170275126s) [noise=true , address=104.36.82.23:4468 , piece_id=NC2BFOASSHA4ZG46LZONWF23DNN3ZZBA6SFO3Q44KD4TCOH4AIQQ, seg_pos={0 0}, upload=211361]: Uploading piece... Aug 17 04:37:54.739278324 (203h11m49.170358445s) [noise=true , address=104.36.82.39:4747 , piece_id=SF2P47ED5FEOCMJ6R6AXP2NTI5LYYALSDLHINNLDVMUMZIE4YLHQ, seg_pos={0 0}, upload=211361]: Uploading piece... Aug 17 04:37:54.739421147 (203h11m49.170501278s) [noise=true , address=104.36.82.3:4800 , piece_id=EKC43HOMYNG77PDESU3QV5OL44HWYQRAN5RUNIB7IMC6NVXQWFVA, seg_pos={0 0}, upload=211361]: Uploading piece... Aug 17 04:37:54.739561055 (203h11m49.170641186s) [noise=true , address=77.90.31.13:4634 , piece_id=ZXX5E54TSYUG557OGOCBQMBKS35TUG6OPDOIG3TTS3OYGTRA24XA, seg_pos={0 0}, upload=211361]: Uploading piece... Aug 17 04:37:54.739643824 (203h11m49.170723955s) [noise=true , address=104.36.82.58:4286 , piece_id=JDBFFYOZPYJ5FXJQIVEBB5IWSBGT6LCBUZF5O25OKWFRXROPHS3Q, seg_pos={0 0}, upload=211361]: Uploading piece... Aug 17 04:37:54.739852683 (203h11m49.170932814s) [noise=true , address=104.36.82.44:4876 , piece_id=H53T7WMGX4CDVQNCYOG6NOYFVRHD5Z35LNF37FVKSWYIVJS2EFZA, seg_pos={0 0}, upload=211361]: Uploading piece... Aug 17 04:37:55.861243533 (203h11m50.292323664s) [upload=211361]: Next returned nil segment Aug 17 04:37:55.861256768 (203h11m50.292336899s) [upload=211361]: Waiting for error group managing segments... Aug 17 04:37:56.165335502 (203h11m50.596415633s) [noise=true , address=66.23.193.175:4562 , piece_id=2RZ2A5KBLZBLEG6UECY2GTNF5THR3XJZMS34ULG5SQNWN4O3XNMA, seg_pos={0 0}, upload=211361]: Done uploading piece. err: <nil> Aug 17 04:37:56.263548583 (203h11m50.694628715s) [noise=true , address=66.23.193.175:4563 , piece_id=I65W2GP5RTRZUIT7Y6NSEAL7XC726PP3BYMCWS6UMZEOWAP5CP2A, seg_pos={0 0}, upload=211361]: Done uploading piece. err: <nil> Aug 17 04:37:56.569392606 (203h11m51.000472737s) [noise=true , address=79.127.220.101:4068 , piece_id=6MFJ22JL437NTOH25E3LNH3Q2LQNKKMUT5Q43F2XNIB7346LGFCQ, seg_pos={0 0}, upload=211361]: Done uploading piece. err: <nil> Aug 17 04:37:56.620901052 (203h11m51.051981183s) [noise=true , address=77.90.31.12:4601 , piece_id=JC5VIGLKCWTCDHQYWEL6QSRZW3MJZFRFQKRJH54OK7LPUQH7WQTQ, seg_pos={0 0}, upload=211361]: Done uploading piece. err: <nil> Aug 17 04:37:56.639713969 (203h11m51.0707941s) [noise=true , address=77.90.31.16:4761 , piece_id=44YO7GYP4RNRWHPP7LP2C4FJBGMWOLEDEC2NN7TAWU3YFHAST3XQ, seg_pos={0 0}, upload=211361]: Done uploading piece. err: <nil> Aug 17 04:37:56.649068307 (203h11m51.080148438s) [noise=true , address=104.36.82.4:4833 , piece_id=AMRE5SIA7EJLAP55GFYILJYZJL5624E534TPWFUEQILHG7CCFDXQ, seg_pos={0 0}, upload=211361]: Done uploading piece. err: <nil> Aug 17 04:37:56.701677831 (203h11m51.132757962s) [noise=true , address=104.36.82.3:4800 , piece_id=EKC43HOMYNG77PDESU3QV5OL44HWYQRAN5RUNIB7IMC6NVXQWFVA, seg_pos={0 0}, upload=211361]: Done uploading piece. err: <nil> Aug 17 04:37:56.719713991 (203h11m51.150794132s) [noise=true , address=104.36.82.23:4468 , piece_id=NC2BFOASSHA4ZG46LZONWF23DNN3ZZBA6SFO3Q44KD4TCOH4AIQQ, seg_pos={0 0}, upload=211361]: Done uploading piece. err: <nil> Aug 17 04:37:56.730632311 (203h11m51.161712432s) [noise=true , address=104.36.82.39:4747 , piece_id=SF2P47ED5FEOCMJ6R6AXP2NTI5LYYALSDLHINNLDVMUMZIE4YLHQ, seg_pos={0 0}, upload=211361]: Done uploading piece. err: <nil> Aug 17 04:37:56.761149974 (203h11m51.192230105s) [noise=true , address=104.36.82.60:4360 , piece_id=W4OJU6NRV2HKPC7JDTISAPVT77KIC3JHVJR5KHYY225FLH4PZBBA, seg_pos={0 0}, upload=211361]: Done uploading piece. err: <nil> Aug 17 04:37:56.775738719 (203h11m51.20681885s) [noise=true , address=77.90.31.18:4830 , piece_id=QNQF4JAEHOFWRTL6XWKL42XSKA2DMO6Z4CZO5NF337NNSHCMYYBA, seg_pos={0 0}, upload=211361]: Done uploading piece. err: <nil> Aug 17 04:37:56.779316274 (203h11m51.210396405s) [noise=true , address=77.90.31.13:4634 , piece_id=ZXX5E54TSYUG557OGOCBQMBKS35TUG6OPDOIG3TTS3OYGTRA24XA, seg_pos={0 0}, upload=211361]: Done uploading piece. err: <nil> Aug 17 04:37:56.802147800 (203h11m51.233227961s) [noise=true , address=104.36.82.58:4286 , piece_id=JDBFFYOZPYJ5FXJQIVEBB5IWSBGT6LCBUZF5O25OKWFRXROPHS3Q, seg_pos={0 0}, upload=211361]: Done uploading piece. err: <nil> Aug 17 04:37:56.832614886 (203h11m51.263695017s) [noise=true , address=104.36.82.44:4876 , piece_id=H53T7WMGX4CDVQNCYOG6NOYFVRHD5Z35LNF37FVKSWYIVJS2EFZA, seg_pos={0 0}, upload=211361]: Done uploading piece. err: <nil> Aug 17 04:37:56.832646346 (203h11m51.263726467s) [seg_pos={0 0}, upload=211361]: Segment reached optimal threshold of 14 pieces. Aug 17 04:37:56.832765443 (203h11m51.263845574s) [noise=true , address=77.90.31.10:4532 , piece_id=5IBCLJNM6LUBCIOPSAERY5HAXV66CR4OKI45RBGOHALTRFYGLMWQ, seg_pos={0 0}, upload=211361]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1T6kkKvTNR5MZZijjr1vMU6g75YkoRpzMvGSgQCqRnfCm9JULe): protocol: expected piece hash; context canceled; EOF Aug 17 04:37:56.832872889 (203h11m51.26395302s) [noise=true , address=104.36.82.36:4648 , piece_id=GTHXHUS6GR26G6UHHMB35MCSEI7I5UJ6K65O76HRJI4WVPIHD7RQ, seg_pos={0 0}, upload=211361]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XHNZDY7xKZ2bss1rSdjbGP3ugSwGRwRzBSWHfXgdKKz794wMq): protocol: expected piece hash; context canceled; EOF Aug 17 04:37:56.832892156 (203h11m51.263972287s) [noise=true , address=77.90.31.19:4863 , piece_id=PYERWDVDOTOW2SVDUO63ODTEWMWUGH5OMV2PE7RJTIDVFTF2ZRVQ, seg_pos={0 0}, upload=211361]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124x1cbCPJRK8YkgSP6UzHymoLu1E1znnfg5WBiD89kSDLi8JeL): protocol: expected piece hash; context canceled; EOF Aug 17 04:37:56.832974564 (203h11m51.264054695s) [noise=true , address=104.36.82.5:4844 , piece_id=PULFFEXZLSMOYEFGRK6L2POF7ERJQT75J23JT3CTXKZ3LMARQZHQ, seg_pos={0 0}, upload=211361]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12iovpNdbidkiv9mEHaAPjBubGmoqr4sUJWm8Cb64LzS6wuwLWa): protocol: write tcp 188.166.225.170:59826->104.36.82.5:4844: use of closed network connection; write tcp 188.166.225.170:59826->104.36.82.5:4844: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59826->104.36.82.5:4844: use of closed network connection Aug 17 04:37:56.833025241 (203h11m51.264105372s) [seg_pos={0 0}, upload=211361]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 17 04:37:56.833116014 (203h11m51.264196135s) [upload=211361]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 17 04:37:56.833120332 (203h11m51.264200463s) [upload=211361]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 17 04:37:58.261163702 (203h11m52.692243834s) [upload=211361]: Done uploading segments... Aug 17 04:37:58.261181997 (203h11m52.692262118s) [upload=211361]: Upload finished. err: <nil> Duration: 6.126971442s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [...........................................X.....X..........] duration: 1.834541222s FAIL (2025-08-17 04:47:44 UTC) orbiter warning: operation: ListObjects ======ERROR====== unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/1K ======LOGS====== Aug 17 04:41:26.040732864 (203h15m20.471812985s) [upload=211420]: Starting upload Aug 17 04:41:26.040780384 (203h15m20.471860515s) [upload=211420]: Done starting upload Aug 17 04:41:26.040828978 (203h15m20.471909109s) [upload=211420]: Uploading segments... Aug 17 04:41:26.040863473 (203h15m20.471943604s) [upload=211420]: Got next segment. Inline: true Aug 17 04:41:26.040867290 (203h15m20.471947422s) [upload=211420]: Waiting for error group managing segments... Aug 17 04:41:26.040897880 (203h15m20.471978011s) [upload=211420]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 17 04:41:26.040901597 (203h15m20.471981728s) [upload=211420]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 17 04:41:26.040904041 (203h15m20.471984162s) [upload=211420]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 17 04:41:26.462140724 (203h15m20.893220855s) [upload=211420]: Done uploading segments... Aug 17 04:41:26.462159310 (203h15m20.893239431s) [upload=211420]: Upload finished. err: <nil> Duration:* 1.834562503s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [..........................XX................................] duration: 6.613577495s FAIL (2025-08-17 04:47:43 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.613599768s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [......................XXXX..................................] duration: 5.717570685s FAIL (2025-08-17 04:47:42 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== Aug 17 04:23:03.873697274 (202h56m58.304777405s) [upload=211112]: Starting upload Aug 17 04:23:03.873756368 (202h56m58.304836499s) [upload=211112]: Done starting upload Aug 17 04:23:03.873839848 (202h56m58.304919989s) [upload=211112]: Uploading segments... Aug 17 04:23:03.873950850 (202h56m58.305030981s) [upload=211112]: Got next segment. Inline: false Aug 17 04:23:03.873976509 (202h56m58.30505664s) [upload=211112]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 17 04:23:03.873980757 (202h56m58.305060888s) [upload=211112]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 17 04:23:04.113727713 (202h56m58.544807844s) [seg_pos={0 0}, upload=211112]: Begin upload segment... Aug 17 04:23:04.113774051 (202h56m58.544854182s) [seg_pos={0 0}, upload=211112]: Done begin upload segment. Aug 17 04:23:04.113820890 (202h56m58.544901021s) [noise=true , address=45.91.116.13:4155 , piece_id=CO3R5XZGH3NJADAFDILPNZEKKFJSRPB4JOFBBCSYYQAD3KNRABNA, seg_pos={0 0}, upload=211112]: Uploading piece... Aug 17 04:23:04.113845989 (202h56m58.54492612s) [noise=true , address=45.91.116.7:4009 , piece_id=IVTDHCTNJEEO3QATGN25YH5Y2MSEQUWJIFPXSJWINUEXDHQAGELA, seg_pos={0 0}, upload=211112]: Uploading piece... Aug 17 04:23:04.114195037 (202h56m58.545275168s) [noise=true , address=45.91.116.4:4938 , piece_id=X5MDNBZYRYYOXDEAMGIOD5CXX7PGAMTFRMROOVTR6NIAYQ5VMXGA, seg_pos={0 0}, upload=211112]: Uploading piece... Aug 17 04:23:04.114281412 (202h56m58.545361543s) [noise=true , address=45.91.116.9:4261 , piece_id=PNOEBFIPLWM4B2QLGJUVEBTT23J5AVRFTWTTKLHFUJ5MNJTSM2JA, seg_pos={0 0}, upload=211112]: Uploading piece... Aug 17 04:23:04.114482115 (202h56m58.545562237s) [noise=true , address=77.90.31.16:4735 , piece_id=G5YGF7LXNPBU2TDRYPAFPVYQSTWUGG7Y6YM2Q6RWMCZY5RVNM4BA, seg_pos={0 0}, upload=211112]: Uploading piece... Aug 17 04:23:04.114570385 (202h56m58.545650516s) [noise=true , address=104.36.82.9:4005 , piece_id=FAR4B5KAALBOACKANEFKGECI2ZVXSHBNYDEIWVGDU2RNBNQXLMXA, seg_pos={0 0}, upload=211112]: Uploading piece... Aug 17 04:23:04.114757392 (202h56m58.545837523s) [noise=true , address=45.91.116.9:4263 , piece_id=VT734KJZ7T3CZ3W3P4M4ZCYKOLLTPP3ILFRWL26Z7K4LWBDZV4FA, seg_pos={0 0}, upload=211112]: Uploading piece... Aug 17 04:23:04.114874727 (202h56m58.545954858s) [noise=true , address=104.36.82.12:4151 , piece_id=2GGYMOWZ2FTWDNWNQRKSAH7UVJ4R6MLRGC6OHNVV3KTVIIMO4QIQ, seg_pos={0 0}, upload=211112]: Uploading piece... Aug 17 04:23:04.115030715 (202h56m58.546110846s) [noise=true , address=45.91.116.12:4225 , piece_id=EUPPM5NRIU5MCZBER3VGZBZEMZZ65Q2MOY2DXKFBIPFJ53M5TJHQ, seg_pos={0 0}, upload=211112]: Uploading piece... Aug 17 04:23:04.115194288 (202h56m58.546274419s) [noise=true , address=104.36.82.4:4804 , piece_id=MSAHN2DERVG2HL7KZMKIWYUQYO6LRIXGUNWJFXJDDGOWZT54FPVA, seg_pos={0 0}, upload=211112]: Uploading piece... Aug 17 04:23:04.115307967 (202h56m58.546388098s) [noise=true , address=104.36.82.5:4838 , piece_id=IYA7ERDAAFYUGC4666CBEX5HZEXZVFPES6JUYRZROY25OAPGVEQQ, seg_pos={0 0}, upload=211112]: Uploading piece... Aug 17 04:23:04.115535692 (202h56m58.546615823s) [noise=true , address=45.91.116.12:4227 , piece_id=XZR3AWJG57RUB4TN7SUWNIWWD2XGKCMDXSWSM74WS3ATHS32EBJQ, seg_pos={0 0}, upload=211112]: Uploading piece... Aug 17 04:23:04.115754130 (202h56m58.546834261s) [noise=true , address=45.91.116.7:4010 , piece_id=HNJU4MPWT67V26J5XDCTHFEOOJL2HS3VAOHB4EIPQVF4D2NNRFMA, seg_pos={0 0}, upload=211112]: Uploading piece... Aug 17 04:23:04.116079182 (202h56m58.547159323s) [noise=true , address=45.91.116.14:4189 , piece_id=2P3GOZDEMJGBAK2F6SY7NJNQUTS5ZYM4B4IMQFAA5TYFARD4DVCQ, seg_pos={0 0}, upload=211112]: Uploading piece... Aug 17 04:23:04.116350181 (202h56m58.547430312s) [noise=true , address=104.36.82.15:4257 , piece_id=Q5GLP6TS7N7CM4DO3LWDCDR6O76OFZ6IVQJC5VQWMA5XLNWZ2E4A, seg_pos={0 0}, upload=211112]: Uploading piece... Aug 17 04:23:04.116554471 (202h56m58.547634602s) [noise=true , address=104.36.82.8:4969 , piece_id=3D3YAZSP7CD6IFZGDKPNKIJPWZ2OJFFVJCSMTO42VNJEFVU3ORHA, seg_pos={0 0}, upload=211112]: Uploading piece... Aug 17 04:23:05.868765063 (202h57m0.299845195s) [upload=211112]: Next returned nil segment Aug 17 04:23:05.868801043 (202h57m0.299881164s) [upload=211112]: Waiting for error group managing segments... Aug 17 04:23:06.147430529 (202h57m0.57851066s) [noise=true , address=45.91.116.12:4227 , piece_id=XZR3AWJG57RUB4TN7SUWNIWWD2XGKCMDXSWSM74WS3ATHS32EBJQ, seg_pos={0 0}, upload=211112]: Done uploading piece. err: <nil> Aug 17 04:23:06.182400314 (202h57m0.613480445s) [noise=true , address=45.91.116.12:4225 , piece_id=EUPPM5NRIU5MCZBER3VGZBZEMZZ65Q2MOY2DXKFBIPFJ53M5TJHQ, seg_pos={0 0}, upload=211112]: Done uploading piece. err: <nil> Aug 17 04:23:06.191910532 (202h57m0.622990663s) [noise=true , address=45.91.116.4:4938 , piece_id=X5MDNBZYRYYOXDEAMGIOD5CXX7PGAMTFRMROOVTR6NIAYQ5VMXGA, seg_pos={0 0}, upload=211112]: Done uploading piece. err: <nil> Aug 17 04:23:06.197391631 (202h57m0.628471782s) [noise=true , address=45.91.116.9:4261 , piece_id=PNOEBFIPLWM4B2QLGJUVEBTT23J5AVRFTWTTKLHFUJ5MNJTSM2JA, seg_pos={0 0}, upload=211112]: Done uploading piece. err: <nil> Aug 17 04:23:06.198069487 (202h57m0.629149618s) [noise=true , address=45.91.116.9:4263 , piece_id=VT734KJZ7T3CZ3W3P4M4ZCYKOLLTPP3ILFRWL26Z7K4LWBDZV4FA, seg_pos={0 0}, upload=211112]: Done uploading piece. err: <nil> Aug 17 04:23:06.200597014 (202h57m0.631677145s) [noise=true , address=45.91.116.7:4009 , piece_id=IVTDHCTNJEEO3QATGN25YH5Y2MSEQUWJIFPXSJWINUEXDHQAGELA, seg_pos={0 0}, upload=211112]: Done uploading piece. err: <nil> Aug 17 04:23:06.288053251 (202h57m0.719133372s) [noise=true , address=45.91.116.7:4010 , piece_id=HNJU4MPWT67V26J5XDCTHFEOOJL2HS3VAOHB4EIPQVF4D2NNRFMA, seg_pos={0 0}, upload=211112]: Done uploading piece. err: <nil> Aug 17 04:23:06.298810435 (202h57m0.729890566s) [noise=true , address=45.91.116.14:4189 , piece_id=2P3GOZDEMJGBAK2F6SY7NJNQUTS5ZYM4B4IMQFAA5TYFARD4DVCQ, seg_pos={0 0}, upload=211112]: Done uploading piece. err: <nil> Aug 17 04:23:06.301049520 (202h57m0.732129651s) [noise=true , address=45.91.116.13:4155 , piece_id=CO3R5XZGH3NJADAFDILPNZEKKFJSRPB4JOFBBCSYYQAD3KNRABNA, seg_pos={0 0}, upload=211112]: Done uploading piece. err: <nil> Aug 17 04:23:06.402776985 (202h57m0.833857116s) [noise=true , address=77.90.31.16:4735 , piece_id=G5YGF7LXNPBU2TDRYPAFPVYQSTWUGG7Y6YM2Q6RWMCZY5RVNM4BA, seg_pos={0 0}, upload=211112]: Done uploading piece. err: <nil> Aug 17 04:23:33.223003412 (202h57m27.654083543s) [noise=true , address=104.36.82.5:4838 , piece_id=IYA7ERDAAFYUGC4666CBEX5HZEXZVFPES6JUYRZROY25OAPGVEQQ, seg_pos={0 0}, upload=211112]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:104.36.82.5:4838): protocol: write tcp 188.166.225.170:42258->104.36.82.5:4838: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:33.223050702 (202h57m27.654130823s) [noise=true , address=104.36.82.8:4969 , piece_id=3D3YAZSP7CD6IFZGDKPNKIJPWZ2OJFFVJCSMTO42VNJEFVU3ORHA, seg_pos={0 0}, upload=211112]: Done uploading piece. err: ecclient: upload failed (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, address:104.36.82.8:4969): protocol: write tcp 188.166.225.170:52308->104.36.82.8:4969: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:33.223090619 (202h57m27.65417075s) [noise=true , address=104.36.82.15:4257 , piece_id=Q5GLP6TS7N7CM4DO3LWDCDR6O76OFZ6IVQJC5VQWMA5XLNWZ2E4A, seg_pos={0 0}, upload=211112]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:104.36.82.15:4257): protocol: write tcp 188.166.225.170:53078->104.36.82.15:4257: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:33.223104976 (202h57m27.654185097s) [noise=true , address=104.36.82.4:4804 , piece_id=MSAHN2DERVG2HL7KZMKIWYUQYO6LRIXGUNWJFXJDDGOWZT54FPVA, seg_pos={0 0}, upload=211112]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:104.36.82.4:4804): protocol: write tcp 188.166.225.170:56176->104.36.82.4:4804: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:33.223149581 (202h57m27.654229712s) [noise=true , address=104.36.82.12:4151 , piece_id=2GGYMOWZ2FTWDNWNQRKSAH7UVJ4R6MLRGC6OHNVV3KTVIIMO4QIQ, seg_pos={0 0}, upload=211112]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:104.36.82.12:4151): protocol: write tcp 188.166.225.170:40582->104.36.82.12:4151: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:33.223157536 (202h57m27.654237667s) [noise=true , address=104.36.82.9:4005 , piece_id=FAR4B5KAALBOACKANEFKGECI2ZVXSHBNYDEIWVGDU2RNBNQXLMXA, seg_pos={0 0}, upload=211112]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:104.36.82.9:4005): protocol: write tcp 188.166.225.170:32954->104.36.82.9:4005: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:33.223177415 (202h57m27.654257536s) [seg_pos={0 0}, upload=211112]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Aug 17 04:23:33.448862924 (202h57m27.879943045s) [upload=211112]: Done uploading segments... Aug 17 04:23:33.448886830 (202h57m27.879966961s) [upload=211112]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 5.717595633s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [............................................X.....X.........] duration: 1.847516441s FAIL (2025-08-17 04:47:24 UTC) orbiter warning: operation: ListObjects ======ERROR====== unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/1K ======LOGS====== Aug 17 04:41:26.040732864 (203h15m20.471812985s) [upload=211420]: Starting upload Aug 17 04:41:26.040780384 (203h15m20.471860515s) [upload=211420]: Done starting upload Aug 17 04:41:26.040828978 (203h15m20.471909109s) [upload=211420]: Uploading segments... Aug 17 04:41:26.040863473 (203h15m20.471943604s) [upload=211420]: Got next segment. Inline: true Aug 17 04:41:26.040867290 (203h15m20.471947422s) [upload=211420]: Waiting for error group managing segments... Aug 17 04:41:26.040897880 (203h15m20.471978011s) [upload=211420]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 17 04:41:26.040901597 (203h15m20.471981728s) [upload=211420]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 17 04:41:26.040904041 (203h15m20.471984162s) [upload=211420]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 17 04:41:26.462140724 (203h15m20.893220855s) [upload=211420]: Done uploading segments... Aug 17 04:41:26.462159310 (203h15m20.893239431s) [upload=211420]: Upload finished. err: <nil> Duration:* 1.847540547s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.......................XXXX.................................] duration: 6.165148071s FAIL (2025-08-17 04:47:10 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== Aug 17 04:23:03.873697274 (202h56m58.304777405s) [upload=211112]: Starting upload Aug 17 04:23:03.873756368 (202h56m58.304836499s) [upload=211112]: Done starting upload Aug 17 04:23:03.873839848 (202h56m58.304919989s) [upload=211112]: Uploading segments... Aug 17 04:23:03.873950850 (202h56m58.305030981s) [upload=211112]: Got next segment. Inline: false Aug 17 04:23:03.873976509 (202h56m58.30505664s) [upload=211112]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 17 04:23:03.873980757 (202h56m58.305060888s) [upload=211112]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 17 04:23:04.113727713 (202h56m58.544807844s) [seg_pos={0 0}, upload=211112]: Begin upload segment... Aug 17 04:23:04.113774051 (202h56m58.544854182s) [seg_pos={0 0}, upload=211112]: Done begin upload segment. Aug 17 04:23:04.113820890 (202h56m58.544901021s) [noise=true , address=45.91.116.13:4155 , piece_id=CO3R5XZGH3NJADAFDILPNZEKKFJSRPB4JOFBBCSYYQAD3KNRABNA, seg_pos={0 0}, upload=211112]: Uploading piece... Aug 17 04:23:04.113845989 (202h56m58.54492612s) [noise=true , address=45.91.116.7:4009 , piece_id=IVTDHCTNJEEO3QATGN25YH5Y2MSEQUWJIFPXSJWINUEXDHQAGELA, seg_pos={0 0}, upload=211112]: Uploading piece... Aug 17 04:23:04.114195037 (202h56m58.545275168s) [noise=true , address=45.91.116.4:4938 , piece_id=X5MDNBZYRYYOXDEAMGIOD5CXX7PGAMTFRMROOVTR6NIAYQ5VMXGA, seg_pos={0 0}, upload=211112]: Uploading piece... Aug 17 04:23:04.114281412 (202h56m58.545361543s) [noise=true , address=45.91.116.9:4261 , piece_id=PNOEBFIPLWM4B2QLGJUVEBTT23J5AVRFTWTTKLHFUJ5MNJTSM2JA, seg_pos={0 0}, upload=211112]: Uploading piece... Aug 17 04:23:04.114482115 (202h56m58.545562237s) [noise=true , address=77.90.31.16:4735 , piece_id=G5YGF7LXNPBU2TDRYPAFPVYQSTWUGG7Y6YM2Q6RWMCZY5RVNM4BA, seg_pos={0 0}, upload=211112]: Uploading piece... Aug 17 04:23:04.114570385 (202h56m58.545650516s) [noise=true , address=104.36.82.9:4005 , piece_id=FAR4B5KAALBOACKANEFKGECI2ZVXSHBNYDEIWVGDU2RNBNQXLMXA, seg_pos={0 0}, upload=211112]: Uploading piece... Aug 17 04:23:04.114757392 (202h56m58.545837523s) [noise=true , address=45.91.116.9:4263 , piece_id=VT734KJZ7T3CZ3W3P4M4ZCYKOLLTPP3ILFRWL26Z7K4LWBDZV4FA, seg_pos={0 0}, upload=211112]: Uploading piece... Aug 17 04:23:04.114874727 (202h56m58.545954858s) [noise=true , address=104.36.82.12:4151 , piece_id=2GGYMOWZ2FTWDNWNQRKSAH7UVJ4R6MLRGC6OHNVV3KTVIIMO4QIQ, seg_pos={0 0}, upload=211112]: Uploading piece... Aug 17 04:23:04.115030715 (202h56m58.546110846s) [noise=true , address=45.91.116.12:4225 , piece_id=EUPPM5NRIU5MCZBER3VGZBZEMZZ65Q2MOY2DXKFBIPFJ53M5TJHQ, seg_pos={0 0}, upload=211112]: Uploading piece... Aug 17 04:23:04.115194288 (202h56m58.546274419s) [noise=true , address=104.36.82.4:4804 , piece_id=MSAHN2DERVG2HL7KZMKIWYUQYO6LRIXGUNWJFXJDDGOWZT54FPVA, seg_pos={0 0}, upload=211112]: Uploading piece... Aug 17 04:23:04.115307967 (202h56m58.546388098s) [noise=true , address=104.36.82.5:4838 , piece_id=IYA7ERDAAFYUGC4666CBEX5HZEXZVFPES6JUYRZROY25OAPGVEQQ, seg_pos={0 0}, upload=211112]: Uploading piece... Aug 17 04:23:04.115535692 (202h56m58.546615823s) [noise=true , address=45.91.116.12:4227 , piece_id=XZR3AWJG57RUB4TN7SUWNIWWD2XGKCMDXSWSM74WS3ATHS32EBJQ, seg_pos={0 0}, upload=211112]: Uploading piece... Aug 17 04:23:04.115754130 (202h56m58.546834261s) [noise=true , address=45.91.116.7:4010 , piece_id=HNJU4MPWT67V26J5XDCTHFEOOJL2HS3VAOHB4EIPQVF4D2NNRFMA, seg_pos={0 0}, upload=211112]: Uploading piece... Aug 17 04:23:04.116079182 (202h56m58.547159323s) [noise=true , address=45.91.116.14:4189 , piece_id=2P3GOZDEMJGBAK2F6SY7NJNQUTS5ZYM4B4IMQFAA5TYFARD4DVCQ, seg_pos={0 0}, upload=211112]: Uploading piece... Aug 17 04:23:04.116350181 (202h56m58.547430312s) [noise=true , address=104.36.82.15:4257 , piece_id=Q5GLP6TS7N7CM4DO3LWDCDR6O76OFZ6IVQJC5VQWMA5XLNWZ2E4A, seg_pos={0 0}, upload=211112]: Uploading piece... Aug 17 04:23:04.116554471 (202h56m58.547634602s) [noise=true , address=104.36.82.8:4969 , piece_id=3D3YAZSP7CD6IFZGDKPNKIJPWZ2OJFFVJCSMTO42VNJEFVU3ORHA, seg_pos={0 0}, upload=211112]: Uploading piece... Aug 17 04:23:05.868765063 (202h57m0.299845195s) [upload=211112]: Next returned nil segment Aug 17 04:23:05.868801043 (202h57m0.299881164s) [upload=211112]: Waiting for error group managing segments... Aug 17 04:23:06.147430529 (202h57m0.57851066s) [noise=true , address=45.91.116.12:4227 , piece_id=XZR3AWJG57RUB4TN7SUWNIWWD2XGKCMDXSWSM74WS3ATHS32EBJQ, seg_pos={0 0}, upload=211112]: Done uploading piece. err: <nil> Aug 17 04:23:06.182400314 (202h57m0.613480445s) [noise=true , address=45.91.116.12:4225 , piece_id=EUPPM5NRIU5MCZBER3VGZBZEMZZ65Q2MOY2DXKFBIPFJ53M5TJHQ, seg_pos={0 0}, upload=211112]: Done uploading piece. err: <nil> Aug 17 04:23:06.191910532 (202h57m0.622990663s) [noise=true , address=45.91.116.4:4938 , piece_id=X5MDNBZYRYYOXDEAMGIOD5CXX7PGAMTFRMROOVTR6NIAYQ5VMXGA, seg_pos={0 0}, upload=211112]: Done uploading piece. err: <nil> Aug 17 04:23:06.197391631 (202h57m0.628471782s) [noise=true , address=45.91.116.9:4261 , piece_id=PNOEBFIPLWM4B2QLGJUVEBTT23J5AVRFTWTTKLHFUJ5MNJTSM2JA, seg_pos={0 0}, upload=211112]: Done uploading piece. err: <nil> Aug 17 04:23:06.198069487 (202h57m0.629149618s) [noise=true , address=45.91.116.9:4263 , piece_id=VT734KJZ7T3CZ3W3P4M4ZCYKOLLTPP3ILFRWL26Z7K4LWBDZV4FA, seg_pos={0 0}, upload=211112]: Done uploading piece. err: <nil> Aug 17 04:23:06.200597014 (202h57m0.631677145s) [noise=true , address=45.91.116.7:4009 , piece_id=IVTDHCTNJEEO3QATGN25YH5Y2MSEQUWJIFPXSJWINUEXDHQAGELA, seg_pos={0 0}, upload=211112]: Done uploading piece. err: <nil> Aug 17 04:23:06.288053251 (202h57m0.719133372s) [noise=true , address=45.91.116.7:4010 , piece_id=HNJU4MPWT67V26J5XDCTHFEOOJL2HS3VAOHB4EIPQVF4D2NNRFMA, seg_pos={0 0}, upload=211112]: Done uploading piece. err: <nil> Aug 17 04:23:06.298810435 (202h57m0.729890566s) [noise=true , address=45.91.116.14:4189 , piece_id=2P3GOZDEMJGBAK2F6SY7NJNQUTS5ZYM4B4IMQFAA5TYFARD4DVCQ, seg_pos={0 0}, upload=211112]: Done uploading piece. err: <nil> Aug 17 04:23:06.301049520 (202h57m0.732129651s) [noise=true , address=45.91.116.13:4155 , piece_id=CO3R5XZGH3NJADAFDILPNZEKKFJSRPB4JOFBBCSYYQAD3KNRABNA, seg_pos={0 0}, upload=211112]: Done uploading piece. err: <nil> Aug 17 04:23:06.402776985 (202h57m0.833857116s) [noise=true , address=77.90.31.16:4735 , piece_id=G5YGF7LXNPBU2TDRYPAFPVYQSTWUGG7Y6YM2Q6RWMCZY5RVNM4BA, seg_pos={0 0}, upload=211112]: Done uploading piece. err: <nil> Aug 17 04:23:33.223003412 (202h57m27.654083543s) [noise=true , address=104.36.82.5:4838 , piece_id=IYA7ERDAAFYUGC4666CBEX5HZEXZVFPES6JUYRZROY25OAPGVEQQ, seg_pos={0 0}, upload=211112]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:104.36.82.5:4838): protocol: write tcp 188.166.225.170:42258->104.36.82.5:4838: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:33.223050702 (202h57m27.654130823s) [noise=true , address=104.36.82.8:4969 , piece_id=3D3YAZSP7CD6IFZGDKPNKIJPWZ2OJFFVJCSMTO42VNJEFVU3ORHA, seg_pos={0 0}, upload=211112]: Done uploading piece. err: ecclient: upload failed (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, address:104.36.82.8:4969): protocol: write tcp 188.166.225.170:52308->104.36.82.8:4969: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:33.223090619 (202h57m27.65417075s) [noise=true , address=104.36.82.15:4257 , piece_id=Q5GLP6TS7N7CM4DO3LWDCDR6O76OFZ6IVQJC5VQWMA5XLNWZ2E4A, seg_pos={0 0}, upload=211112]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:104.36.82.15:4257): protocol: write tcp 188.166.225.170:53078->104.36.82.15:4257: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:33.223104976 (202h57m27.654185097s) [noise=true , address=104.36.82.4:4804 , piece_id=MSAHN2DERVG2HL7KZMKIWYUQYO6LRIXGUNWJFXJDDGOWZT54FPVA, seg_pos={0 0}, upload=211112]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:104.36.82.4:4804): protocol: write tcp 188.166.225.170:56176->104.36.82.4:4804: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:33.223149581 (202h57m27.654229712s) [noise=true , address=104.36.82.12:4151 , piece_id=2GGYMOWZ2FTWDNWNQRKSAH7UVJ4R6MLRGC6OHNVV3KTVIIMO4QIQ, seg_pos={0 0}, upload=211112]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:104.36.82.12:4151): protocol: write tcp 188.166.225.170:40582->104.36.82.12:4151: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:33.223157536 (202h57m27.654237667s) [noise=true , address=104.36.82.9:4005 , piece_id=FAR4B5KAALBOACKANEFKGECI2ZVXSHBNYDEIWVGDU2RNBNQXLMXA, seg_pos={0 0}, upload=211112]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:104.36.82.9:4005): protocol: write tcp 188.166.225.170:32954->104.36.82.9:4005: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:33.223177415 (202h57m27.654257536s) [seg_pos={0 0}, upload=211112]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Aug 17 04:23:33.448862924 (202h57m27.879943045s) [upload=211112]: Done uploading segments... Aug 17 04:23:33.448886830 (202h57m27.879966961s) [upload=211112]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 6.165175762s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [........................XXXX................................] duration: 5.979671498s FAIL (2025-08-17 04:47:09 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.979701494s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [.......................XXX..................................] duration: 5.769979681s FAIL (2025-08-17 04:47:05 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== g_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.761278730 (202h56m21.192358861s) [noise=true , address=104.36.82.34:4558 , piece_id=XGFNEBBVZX5UWJN7DALGM2EDG26POCTSBBAUU7DLYQKS2DNSBNXA, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.761661883 (202h56m21.192742014s) [noise=true , address=104.36.82.35:4594 , piece_id=6NUSPNYGSSJOB2CNPIW6DI45MPZARLWDBXH3CWMP4LXMDB2CXWZQ, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.761716548 (202h56m21.192796679s) [noise=true , address=45.91.116.8:4067 , piece_id=EZAI7ZDUBOOWXYJNTB7ZPOUSBUUG2FKV77QVUOZMNM6TUJSKGYWA, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.761972016 (202h56m21.193052147s) [noise=true , address=104.36.82.58:4283 , piece_id=LJKEH2KDSOFPVFPB5UWGVLOHQVZQOY33MYSHFLP4ZWGPKZ4KC64A, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.762254878 (202h56m21.193335009s) [noise=true , address=79.127.201.215:4345 , piece_id=JMSKNIZ7QPGBXIEP5RA3S74XAU5NJ2MUUXSFFIAK377YFGO646YQ, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.762523682 (202h56m21.193603803s) [noise=true , address=45.91.116.13:4181 , piece_id=7VN6RQGOTBHQO6HXM5IFTU32PWNVPH6BJZMQNETJEXOYPR2Y7OCQ, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.762864294 (202h56m21.193944415s) [noise=true , address=45.91.116.14:4203 , piece_id=5LXGQ4JZD2KLW3OMWY5QYRE4MKQNKKSAIMESODKDDT2W33XD45DQ, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.763179537 (202h56m21.194259668s) [noise=true , address=77.90.31.18:4815 , piece_id=G6PINAFFA6DZRRMKU2XK2LYFOES5WG7PTJ5L2ZPUQHMA4JNSRZRQ, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.763478499 (202h56m21.19455864s) [noise=true , address=77.90.31.11:4562 , piece_id=PUL6TAVIIU7732IOASNH5EHQMNCK7DWHJMUJVXRNK74T4GUQDGTA, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.763757683 (202h56m21.194837814s) [noise=true , address=104.36.82.5:4855 , piece_id=3FCDYA6CRYOOU3MDFJPEC4YQNICFVCN2A47RIYPSR4ZX5TIXZX3A, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.763988195 (202h56m21.195068326s) [noise=true , address=45.91.116.5:4308 , piece_id=F52M3VMHRQWR5LIEEPMVKVBFMOVG76UNNGQTFKSF6ZBD7VEOFWVA, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.764273320 (202h56m21.195353451s) [noise=true , address=66.23.193.175:4559 , piece_id=46X42PJIPA6DVC7BG7WPKMPMC44ZBUROEAB5C22M5INXKBJC4KVQ, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.764555100 (202h56m21.195635221s) [noise=true , address=77.90.31.15:4732 , piece_id=4AU7UGKXN4TIPSIRR7IMXSF6WHELO6UTOX35N5X7JHOQQXWWJF2A, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.764757416 (202h56m21.195837547s) [noise=true , address=104.36.82.39:4753 , piece_id=R7BHT5RYLSCDZLTDDRXTSJFSRPLM5PNBYC6Q2G33C5SYIUCNVCLQ, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.765868993 (202h56m21.196949124s) [noise=true , address=104.36.82.50:4047 , piece_id=AKDACFQEPBAFGYKTWBL5RELLZL3L4IJHJT73NS7T7QPWK2C2TDQA, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.767152809 (202h56m21.19823294s) [noise=true , address=104.36.82.19:4436 , piece_id=KVDJTKON4LFVJSAY7XSJ2OQ4ZH67JLMDE2O4EHUWSMZ3EECWSOXA, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:26.768577886 (202h56m21.199658017s) [noise=true , address=45.91.116.10:4095 , piece_id=JQ2UARQSNKBGHHHBJ6PR45ETR4EYMCXPDEQHEGWFYH52LRJC57UQ, seg_pos={0 0}, upload=211101]: Uploading piece... Aug 17 04:22:27.934309011 (202h56m22.365389142s) [upload=211101]: Next returned nil segment Aug 17 04:22:27.934342545 (202h56m22.365422666s) [upload=211101]: Waiting for error group managing segments... Aug 17 04:22:28.231985083 (202h56m22.663065214s) [noise=true , address=66.23.193.175:4559 , piece_id=46X42PJIPA6DVC7BG7WPKMPMC44ZBUROEAB5C22M5INXKBJC4KVQ, seg_pos={0 0}, upload=211101]: Done uploading piece. err: <nil> Aug 17 04:22:28.670701256 (202h56m23.101781427s) [noise=true , address=45.91.116.10:4095 , piece_id=JQ2UARQSNKBGHHHBJ6PR45ETR4EYMCXPDEQHEGWFYH52LRJC57UQ, seg_pos={0 0}, upload=211101]: Done uploading piece. err: <nil> Aug 17 04:22:28.671952431 (202h56m23.103032562s) [noise=true , address=45.91.116.5:4308 , piece_id=F52M3VMHRQWR5LIEEPMVKVBFMOVG76UNNGQTFKSF6ZBD7VEOFWVA, seg_pos={0 0}, upload=211101]: Done uploading piece. err: <nil> Aug 17 04:22:28.698468278 (202h56m23.129548409s) [noise=true , address=45.91.116.8:4067 , piece_id=EZAI7ZDUBOOWXYJNTB7ZPOUSBUUG2FKV77QVUOZMNM6TUJSKGYWA, seg_pos={0 0}, upload=211101]: Done uploading piece. err: <nil> Aug 17 04:22:28.720171187 (202h56m23.151251318s) [noise=true , address=45.91.116.13:4181 , piece_id=7VN6RQGOTBHQO6HXM5IFTU32PWNVPH6BJZMQNETJEXOYPR2Y7OCQ, seg_pos={0 0}, upload=211101]: Done uploading piece. err: <nil> Aug 17 04:22:28.731591429 (202h56m23.16267156s) [noise=true , address=45.91.116.14:4203 , piece_id=5LXGQ4JZD2KLW3OMWY5QYRE4MKQNKKSAIMESODKDDT2W33XD45DQ, seg_pos={0 0}, upload=211101]: Done uploading piece. err: <nil> Aug 17 04:22:28.798148986 (202h56m23.229229117s) [noise=true , address=77.90.31.18:4815 , piece_id=G6PINAFFA6DZRRMKU2XK2LYFOES5WG7PTJ5L2ZPUQHMA4JNSRZRQ, seg_pos={0 0}, upload=211101]: Done uploading piece. err: <nil> Aug 17 04:22:28.804849037 (202h56m23.235929168s) [noise=true , address=79.127.201.215:4345 , piece_id=JMSKNIZ7QPGBXIEP5RA3S74XAU5NJ2MUUXSFFIAK377YFGO646YQ, seg_pos={0 0}, upload=211101]: Done uploading piece. err: <nil> Aug 17 04:22:28.833066822 (202h56m23.264147013s) [noise=true , address=77.90.31.15:4732 , piece_id=4AU7UGKXN4TIPSIRR7IMXSF6WHELO6UTOX35N5X7JHOQQXWWJF2A, seg_pos={0 0}, upload=211101]: Done uploading piece. err: <nil> Aug 17 04:22:28.846684710 (202h56m23.277764831s) [noise=true , address=77.90.31.11:4562 , piece_id=PUL6TAVIIU7732IOASNH5EHQMNCK7DWHJMUJVXRNK74T4GUQDGTA, seg_pos={0 0}, upload=211101]: Done uploading piece. err: <nil> Aug 17 04:23:25.582810947 (202h57m20.013891078s) [noise=true , address=104.36.82.34:4558 , piece_id=XGFNEBBVZX5UWJN7DALGM2EDG26POCTSBBAUU7DLYQKS2DNSBNXA, seg_pos={0 0}, upload=211101]: Done uploading piece. err: ecclient: upload failed (node:12TU4zvu38mF1fNSergCtMj2cK9eu57Rdfpt5MVk3tc4MZF6dNr, address:104.36.82.34:4558): protocol: rpc: dial tcp 104.36.82.34:4558: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:25.582856475 (202h57m20.013936606s) [noise=true , address=104.36.82.35:4594 , piece_id=6NUSPNYGSSJOB2CNPIW6DI45MPZARLWDBXH3CWMP4LXMDB2CXWZQ, seg_pos={0 0}, upload=211101]: Done uploading piece. err: ecclient: upload failed (node:12D9nh4XFmsjjFcZEMjaL4uw4Whsa8nfMrAsYPu6NVpsxgb3jvf, address:104.36.82.35:4594): protocol: rpc: dial tcp 104.36.82.35:4594: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:25.582973238 (202h57m20.014053369s) [noise=true , address=104.36.82.19:4436 , piece_id=KVDJTKON4LFVJSAY7XSJ2OQ4ZH67JLMDE2O4EHUWSMZ3EECWSOXA, seg_pos={0 0}, upload=211101]: Done uploading piece. err: ecclient: upload failed (node:1WvyEw8XevCiQ8yRgzyMo7WTVzpEJ158ErKfAeV4dSTBkS1oRF, address:104.36.82.19:4436): protocol: rpc: dial tcp 104.36.82.19:4436: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:25.583033393 (202h57m20.014113524s) [noise=true , address=104.36.82.58:4283 , piece_id=LJKEH2KDSOFPVFPB5UWGVLOHQVZQOY33MYSHFLP4ZWGPKZ4KC64A, seg_pos={0 0}, upload=211101]: Done uploading piece. err: ecclient: upload failed (node:1p9jkPtQnmbCXBmw6HSHDWpGaKWqfg87m6hHXy6WDu2giAFnbQ, address:104.36.82.58:4283): protocol: rpc: dial tcp 104.36.82.58:4283: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:25.583069241 (202h57m20.014149362s) [noise=true , address=104.36.82.50:4047 , piece_id=AKDACFQEPBAFGYKTWBL5RELLZL3L4IJHJT73NS7T7QPWK2C2TDQA, seg_pos={0 0}, upload=211101]: Done uploading piece. err: ecclient: upload failed (node:126Z6tvp49dLLdj3T5snJc4tBVGzcX5UBaGafFN3PkhF92vxpfs, address:104.36.82.50:4047): protocol: rpc: dial tcp 104.36.82.50:4047: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:25.583076716 (202h57m20.014156837s) [noise=true , address=104.36.82.39:4753 , piece_id=R7BHT5RYLSCDZLTDDRXTSJFSRPLM5PNBYC6Q2G33C5SYIUCNVCLQ, seg_pos={0 0}, upload=211101]: Done uploading piece. err: ecclient: upload failed (node:1zzW2z6noGQT6xdSmwSQBNYwip6gZjnmqZg7qdb6BP3nHyCLFz, address:104.36.82.39:4753): protocol: rpc: dial tcp 104.36.82.39:4753: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:25.583107535 (202h57m20.014187666s) [noise=true , address=104.36.82.23:4474 , piece_id=RITPHITVW655UQETT6UKS7BZO2S3F4P2MJIO4REOUKYYP4E2CDVQ, seg_pos={0 0}, upload=211101]: Done uploading piece. err: ecclient: upload failed (node:1Fpshye6yGAo3yf71eVpxrCdvoRjpkGX9WEX8KCWYbvuhy5i9j, address:104.36.82.23:4474): protocol: write tcp 188.166.225.170:36642->104.36.82.23:4474: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:25.583165355 (202h57m20.014245487s) [noise=true , address=104.36.82.5:4855 , piece_id=3FCDYA6CRYOOU3MDFJPEC4YQNICFVCN2A47RIYPSR4ZX5TIXZX3A, seg_pos={0 0}, upload=211101]: Done uploading piece. err: ecclient: upload failed (node:1KDNuE7YY8smhDjotushQvY3y9NtLxigbJELHD4Edxy8Qi4BWh, address:104.36.82.5:4855): protocol: write tcp 188.166.225.170:47052->104.36.82.5:4855: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:25.583194000 (202h57m20.014274131s) [seg_pos={0 0}, upload=211101]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Aug 17 04:23:25.810738626 (202h57m20.241818757s) [upload=211101]: Done uploading segments... Aug 17 04:23:25.810761991 (202h57m20.241842122s) [upload=211101]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 5.770006904s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 9 out of 60 is failed (threshold=3) [.................X..........XXXX.X.X.......X...X............] duration: 5.777166493s FAIL (2025-08-17 04:46:54 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/4M") ======LOGS====== Aug 17 04:37:54.182428668 (203h11m48.613508799s) [upload=211361]: Starting upload Aug 17 04:37:54.182474656 (203h11m48.613554777s) [upload=211361]: Done starting upload Aug 17 04:37:54.182665000 (203h11m48.613745131s) [upload=211361]: Uploading segments... Aug 17 04:37:54.182728421 (203h11m48.613808552s) [upload=211361]: Got next segment. Inline: false Aug 17 04:37:54.182758048 (203h11m48.613838179s) [upload=211361]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 17 04:37:54.182762306 (203h11m48.613842437s) [upload=211361]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 17 04:37:54.737160222 (203h11m49.168240353s) [seg_pos={0 0}, upload=211361]: Begin upload segment... Aug 17 04:37:54.737203494 (203h11m49.168283625s) [seg_pos={0 0}, upload=211361]: Done begin upload segment. Aug 17 04:37:54.737224945 (203h11m49.168305076s) [noise=true , address=77.90.31.19:4863 , piece_id=PYERWDVDOTOW2SVDUO63ODTEWMWUGH5OMV2PE7RJTIDVFTF2ZRVQ, seg_pos={0 0}, upload=211361]: Uploading piece... Aug 17 04:37:54.737311591 (203h11m49.168391722s) [noise=true , address=79.127.220.101:4068 , piece_id=6MFJ22JL437NTOH25E3LNH3Q2LQNKKMUT5Q43F2XNIB7346LGFCQ, seg_pos={0 0}, upload=211361]: Uploading piece... Aug 17 04:37:54.737638917 (203h11m49.168719048s) [noise=true , address=77.90.31.12:4601 , piece_id=JC5VIGLKCWTCDHQYWEL6QSRZW3MJZFRFQKRJH54OK7LPUQH7WQTQ, seg_pos={0 0}, upload=211361]: Uploading piece... Aug 17 04:37:54.737939432 (203h11m49.169019563s) [noise=true , address=66.23.193.175:4562 , piece_id=2RZ2A5KBLZBLEG6UECY2GTNF5THR3XJZMS34ULG5SQNWN4O3XNMA, seg_pos={0 0}, upload=211361]: Uploading piece... Aug 17 04:37:54.737952067 (203h11m49.169032198s) [noise=true , address=77.90.31.16:4761 , piece_id=44YO7GYP4RNRWHPP7LP2C4FJBGMWOLEDEC2NN7TAWU3YFHAST3XQ, seg_pos={0 0}, upload=211361]: Uploading piece... Aug 17 04:37:54.738229999 (203h11m49.16931013s) [noise=true , address=104.36.82.36:4648 , piece_id=GTHXHUS6GR26G6UHHMB35MCSEI7I5UJ6K65O76HRJI4WVPIHD7RQ, seg_pos={0 0}, upload=211361]: Uploading piece... Aug 17 04:37:54.738324038 (203h11m49.169404169s) [noise=true , address=66.23.193.175:4563 , piece_id=I65W2GP5RTRZUIT7Y6NSEAL7XC726PP3BYMCWS6UMZEOWAP5CP2A, seg_pos={0 0}, upload=211361]: Uploading piece... Aug 17 04:37:54.738511367 (203h11m49.169591488s) [noise=true , address=104.36.82.4:4833 , piece_id=AMRE5SIA7EJLAP55GFYILJYZJL5624E534TPWFUEQILHG7CCFDXQ, seg_pos={0 0}, upload=211361]: Uploading piece... Aug 17 04:37:54.738655813 (203h11m49.169735954s) [noise=true , address=104.36.82.5:4844 , piece_id=PULFFEXZLSMOYEFGRK6L2POF7ERJQT75J23JT3CTXKZ3LMARQZHQ, seg_pos={0 0}, upload=211361]: Uploading piece... Aug 17 04:37:54.738747769 (203h11m49.1698279s) [noise=true , address=77.90.31.10:4532 , piece_id=5IBCLJNM6LUBCIOPSAERY5HAXV66CR4OKI45RBGOHALTRFYGLMWQ, seg_pos={0 0}, upload=211361]: Uploading piece... Aug 17 04:37:54.738971367 (203h11m49.170051498s) [noise=true , address=104.36.82.60:4360 , piece_id=W4OJU6NRV2HKPC7JDTISAPVT77KIC3JHVJR5KHYY225FLH4PZBBA, seg_pos={0 0}, upload=211361]: Uploading piece... Aug 17 04:37:54.738975705 (203h11m49.170055836s) [noise=true , address=77.90.31.18:4830 , piece_id=QNQF4JAEHOFWRTL6XWKL42XSKA2DMO6Z4CZO5NF337NNSHCMYYBA, seg_pos={0 0}, upload=211361]: Uploading piece... Aug 17 04:37:54.739194995 (203h11m49.170275126s) [noise=true , address=104.36.82.23:4468 , piece_id=NC2BFOASSHA4ZG46LZONWF23DNN3ZZBA6SFO3Q44KD4TCOH4AIQQ, seg_pos={0 0}, upload=211361]: Uploading piece... Aug 17 04:37:54.739278324 (203h11m49.170358445s) [noise=true , address=104.36.82.39:4747 , piece_id=SF2P47ED5FEOCMJ6R6AXP2NTI5LYYALSDLHINNLDVMUMZIE4YLHQ, seg_pos={0 0}, upload=211361]: Uploading piece... Aug 17 04:37:54.739421147 (203h11m49.170501278s) [noise=true , address=104.36.82.3:4800 , piece_id=EKC43HOMYNG77PDESU3QV5OL44HWYQRAN5RUNIB7IMC6NVXQWFVA, seg_pos={0 0}, upload=211361]: Uploading piece... Aug 17 04:37:54.739561055 (203h11m49.170641186s) [noise=true , address=77.90.31.13:4634 , piece_id=ZXX5E54TSYUG557OGOCBQMBKS35TUG6OPDOIG3TTS3OYGTRA24XA, seg_pos={0 0}, upload=211361]: Uploading piece... Aug 17 04:37:54.739643824 (203h11m49.170723955s) [noise=true , address=104.36.82.58:4286 , piece_id=JDBFFYOZPYJ5FXJQIVEBB5IWSBGT6LCBUZF5O25OKWFRXROPHS3Q, seg_pos={0 0}, upload=211361]: Uploading piece... Aug 17 04:37:54.739852683 (203h11m49.170932814s) [noise=true , address=104.36.82.44:4876 , piece_id=H53T7WMGX4CDVQNCYOG6NOYFVRHD5Z35LNF37FVKSWYIVJS2EFZA, seg_pos={0 0}, upload=211361]: Uploading piece... Aug 17 04:37:55.861243533 (203h11m50.292323664s) [upload=211361]: Next returned nil segment Aug 17 04:37:55.861256768 (203h11m50.292336899s) [upload=211361]: Waiting for error group managing segments... Aug 17 04:37:56.165335502 (203h11m50.596415633s) [noise=true , address=66.23.193.175:4562 , piece_id=2RZ2A5KBLZBLEG6UECY2GTNF5THR3XJZMS34ULG5SQNWN4O3XNMA, seg_pos={0 0}, upload=211361]: Done uploading piece. err: <nil> Aug 17 04:37:56.263548583 (203h11m50.694628715s) [noise=true , address=66.23.193.175:4563 , piece_id=I65W2GP5RTRZUIT7Y6NSEAL7XC726PP3BYMCWS6UMZEOWAP5CP2A, seg_pos={0 0}, upload=211361]: Done uploading piece. err: <nil> Aug 17 04:37:56.569392606 (203h11m51.000472737s) [noise=true , address=79.127.220.101:4068 , piece_id=6MFJ22JL437NTOH25E3LNH3Q2LQNKKMUT5Q43F2XNIB7346LGFCQ, seg_pos={0 0}, upload=211361]: Done uploading piece. err: <nil> Aug 17 04:37:56.620901052 (203h11m51.051981183s) [noise=true , address=77.90.31.12:4601 , piece_id=JC5VIGLKCWTCDHQYWEL6QSRZW3MJZFRFQKRJH54OK7LPUQH7WQTQ, seg_pos={0 0}, upload=211361]: Done uploading piece. err: <nil> Aug 17 04:37:56.639713969 (203h11m51.0707941s) [noise=true , address=77.90.31.16:4761 , piece_id=44YO7GYP4RNRWHPP7LP2C4FJBGMWOLEDEC2NN7TAWU3YFHAST3XQ, seg_pos={0 0}, upload=211361]: Done uploading piece. err: <nil> Aug 17 04:37:56.649068307 (203h11m51.080148438s) [noise=true , address=104.36.82.4:4833 , piece_id=AMRE5SIA7EJLAP55GFYILJYZJL5624E534TPWFUEQILHG7CCFDXQ, seg_pos={0 0}, upload=211361]: Done uploading piece. err: <nil> Aug 17 04:37:56.701677831 (203h11m51.132757962s) [noise=true , address=104.36.82.3:4800 , piece_id=EKC43HOMYNG77PDESU3QV5OL44HWYQRAN5RUNIB7IMC6NVXQWFVA, seg_pos={0 0}, upload=211361]: Done uploading piece. err: <nil> Aug 17 04:37:56.719713991 (203h11m51.150794132s) [noise=true , address=104.36.82.23:4468 , piece_id=NC2BFOASSHA4ZG46LZONWF23DNN3ZZBA6SFO3Q44KD4TCOH4AIQQ, seg_pos={0 0}, upload=211361]: Done uploading piece. err: <nil> Aug 17 04:37:56.730632311 (203h11m51.161712432s) [noise=true , address=104.36.82.39:4747 , piece_id=SF2P47ED5FEOCMJ6R6AXP2NTI5LYYALSDLHINNLDVMUMZIE4YLHQ, seg_pos={0 0}, upload=211361]: Done uploading piece. err: <nil> Aug 17 04:37:56.761149974 (203h11m51.192230105s) [noise=true , address=104.36.82.60:4360 , piece_id=W4OJU6NRV2HKPC7JDTISAPVT77KIC3JHVJR5KHYY225FLH4PZBBA, seg_pos={0 0}, upload=211361]: Done uploading piece. err: <nil> Aug 17 04:37:56.775738719 (203h11m51.20681885s) [noise=true , address=77.90.31.18:4830 , piece_id=QNQF4JAEHOFWRTL6XWKL42XSKA2DMO6Z4CZO5NF337NNSHCMYYBA, seg_pos={0 0}, upload=211361]: Done uploading piece. err: <nil> Aug 17 04:37:56.779316274 (203h11m51.210396405s) [noise=true , address=77.90.31.13:4634 , piece_id=ZXX5E54TSYUG557OGOCBQMBKS35TUG6OPDOIG3TTS3OYGTRA24XA, seg_pos={0 0}, upload=211361]: Done uploading piece. err: <nil> Aug 17 04:37:56.802147800 (203h11m51.233227961s) [noise=true , address=104.36.82.58:4286 , piece_id=JDBFFYOZPYJ5FXJQIVEBB5IWSBGT6LCBUZF5O25OKWFRXROPHS3Q, seg_pos={0 0}, upload=211361]: Done uploading piece. err: <nil> Aug 17 04:37:56.832614886 (203h11m51.263695017s) [noise=true , address=104.36.82.44:4876 , piece_id=H53T7WMGX4CDVQNCYOG6NOYFVRHD5Z35LNF37FVKSWYIVJS2EFZA, seg_pos={0 0}, upload=211361]: Done uploading piece. err: <nil> Aug 17 04:37:56.832646346 (203h11m51.263726467s) [seg_pos={0 0}, upload=211361]: Segment reached optimal threshold of 14 pieces. Aug 17 04:37:56.832765443 (203h11m51.263845574s) [noise=true , address=77.90.31.10:4532 , piece_id=5IBCLJNM6LUBCIOPSAERY5HAXV66CR4OKI45RBGOHALTRFYGLMWQ, seg_pos={0 0}, upload=211361]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1T6kkKvTNR5MZZijjr1vMU6g75YkoRpzMvGSgQCqRnfCm9JULe): protocol: expected piece hash; context canceled; EOF Aug 17 04:37:56.832872889 (203h11m51.26395302s) [noise=true , address=104.36.82.36:4648 , piece_id=GTHXHUS6GR26G6UHHMB35MCSEI7I5UJ6K65O76HRJI4WVPIHD7RQ, seg_pos={0 0}, upload=211361]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XHNZDY7xKZ2bss1rSdjbGP3ugSwGRwRzBSWHfXgdKKz794wMq): protocol: expected piece hash; context canceled; EOF Aug 17 04:37:56.832892156 (203h11m51.263972287s) [noise=true , address=77.90.31.19:4863 , piece_id=PYERWDVDOTOW2SVDUO63ODTEWMWUGH5OMV2PE7RJTIDVFTF2ZRVQ, seg_pos={0 0}, upload=211361]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124x1cbCPJRK8YkgSP6UzHymoLu1E1znnfg5WBiD89kSDLi8JeL): protocol: expected piece hash; context canceled; EOF Aug 17 04:37:56.832974564 (203h11m51.264054695s) [noise=true , address=104.36.82.5:4844 , piece_id=PULFFEXZLSMOYEFGRK6L2POF7ERJQT75J23JT3CTXKZ3LMARQZHQ, seg_pos={0 0}, upload=211361]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12iovpNdbidkiv9mEHaAPjBubGmoqr4sUJWm8Cb64LzS6wuwLWa): protocol: write tcp 188.166.225.170:59826->104.36.82.5:4844: use of closed network connection; write tcp 188.166.225.170:59826->104.36.82.5:4844: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59826->104.36.82.5:4844: use of closed network connection Aug 17 04:37:56.833025241 (203h11m51.264105372s) [seg_pos={0 0}, upload=211361]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 17 04:37:56.833116014 (203h11m51.264196135s) [upload=211361]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 17 04:37:56.833120332 (203h11m51.264200463s) [upload=211361]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 17 04:37:58.261163702 (203h11m52.692243834s) [upload=211361]: Done uploading segments... Aug 17 04:37:58.261181997 (203h11m52.692262118s) [upload=211361]: Upload finished. err: <nil> Duration: 5.777195578s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [........................XXXX................................] duration: 5.834242622s FAIL (2025-08-17 04:46:53 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== Aug 17 04:23:03.873697274 (202h56m58.304777405s) [upload=211112]: Starting upload Aug 17 04:23:03.873756368 (202h56m58.304836499s) [upload=211112]: Done starting upload Aug 17 04:23:03.873839848 (202h56m58.304919989s) [upload=211112]: Uploading segments... Aug 17 04:23:03.873950850 (202h56m58.305030981s) [upload=211112]: Got next segment. Inline: false Aug 17 04:23:03.873976509 (202h56m58.30505664s) [upload=211112]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 17 04:23:03.873980757 (202h56m58.305060888s) [upload=211112]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 17 04:23:04.113727713 (202h56m58.544807844s) [seg_pos={0 0}, upload=211112]: Begin upload segment... Aug 17 04:23:04.113774051 (202h56m58.544854182s) [seg_pos={0 0}, upload=211112]: Done begin upload segment. Aug 17 04:23:04.113820890 (202h56m58.544901021s) [noise=true , address=45.91.116.13:4155 , piece_id=CO3R5XZGH3NJADAFDILPNZEKKFJSRPB4JOFBBCSYYQAD3KNRABNA, seg_pos={0 0}, upload=211112]: Uploading piece... Aug 17 04:23:04.113845989 (202h56m58.54492612s) [noise=true , address=45.91.116.7:4009 , piece_id=IVTDHCTNJEEO3QATGN25YH5Y2MSEQUWJIFPXSJWINUEXDHQAGELA, seg_pos={0 0}, upload=211112]: Uploading piece... Aug 17 04:23:04.114195037 (202h56m58.545275168s) [noise=true , address=45.91.116.4:4938 , piece_id=X5MDNBZYRYYOXDEAMGIOD5CXX7PGAMTFRMROOVTR6NIAYQ5VMXGA, seg_pos={0 0}, upload=211112]: Uploading piece... Aug 17 04:23:04.114281412 (202h56m58.545361543s) [noise=true , address=45.91.116.9:4261 , piece_id=PNOEBFIPLWM4B2QLGJUVEBTT23J5AVRFTWTTKLHFUJ5MNJTSM2JA, seg_pos={0 0}, upload=211112]: Uploading piece... Aug 17 04:23:04.114482115 (202h56m58.545562237s) [noise=true , address=77.90.31.16:4735 , piece_id=G5YGF7LXNPBU2TDRYPAFPVYQSTWUGG7Y6YM2Q6RWMCZY5RVNM4BA, seg_pos={0 0}, upload=211112]: Uploading piece... Aug 17 04:23:04.114570385 (202h56m58.545650516s) [noise=true , address=104.36.82.9:4005 , piece_id=FAR4B5KAALBOACKANEFKGECI2ZVXSHBNYDEIWVGDU2RNBNQXLMXA, seg_pos={0 0}, upload=211112]: Uploading piece... Aug 17 04:23:04.114757392 (202h56m58.545837523s) [noise=true , address=45.91.116.9:4263 , piece_id=VT734KJZ7T3CZ3W3P4M4ZCYKOLLTPP3ILFRWL26Z7K4LWBDZV4FA, seg_pos={0 0}, upload=211112]: Uploading piece... Aug 17 04:23:04.114874727 (202h56m58.545954858s) [noise=true , address=104.36.82.12:4151 , piece_id=2GGYMOWZ2FTWDNWNQRKSAH7UVJ4R6MLRGC6OHNVV3KTVIIMO4QIQ, seg_pos={0 0}, upload=211112]: Uploading piece... Aug 17 04:23:04.115030715 (202h56m58.546110846s) [noise=true , address=45.91.116.12:4225 , piece_id=EUPPM5NRIU5MCZBER3VGZBZEMZZ65Q2MOY2DXKFBIPFJ53M5TJHQ, seg_pos={0 0}, upload=211112]: Uploading piece... Aug 17 04:23:04.115194288 (202h56m58.546274419s) [noise=true , address=104.36.82.4:4804 , piece_id=MSAHN2DERVG2HL7KZMKIWYUQYO6LRIXGUNWJFXJDDGOWZT54FPVA, seg_pos={0 0}, upload=211112]: Uploading piece... Aug 17 04:23:04.115307967 (202h56m58.546388098s) [noise=true , address=104.36.82.5:4838 , piece_id=IYA7ERDAAFYUGC4666CBEX5HZEXZVFPES6JUYRZROY25OAPGVEQQ, seg_pos={0 0}, upload=211112]: Uploading piece... Aug 17 04:23:04.115535692 (202h56m58.546615823s) [noise=true , address=45.91.116.12:4227 , piece_id=XZR3AWJG57RUB4TN7SUWNIWWD2XGKCMDXSWSM74WS3ATHS32EBJQ, seg_pos={0 0}, upload=211112]: Uploading piece... Aug 17 04:23:04.115754130 (202h56m58.546834261s) [noise=true , address=45.91.116.7:4010 , piece_id=HNJU4MPWT67V26J5XDCTHFEOOJL2HS3VAOHB4EIPQVF4D2NNRFMA, seg_pos={0 0}, upload=211112]: Uploading piece... Aug 17 04:23:04.116079182 (202h56m58.547159323s) [noise=true , address=45.91.116.14:4189 , piece_id=2P3GOZDEMJGBAK2F6SY7NJNQUTS5ZYM4B4IMQFAA5TYFARD4DVCQ, seg_pos={0 0}, upload=211112]: Uploading piece... Aug 17 04:23:04.116350181 (202h56m58.547430312s) [noise=true , address=104.36.82.15:4257 , piece_id=Q5GLP6TS7N7CM4DO3LWDCDR6O76OFZ6IVQJC5VQWMA5XLNWZ2E4A, seg_pos={0 0}, upload=211112]: Uploading piece... Aug 17 04:23:04.116554471 (202h56m58.547634602s) [noise=true , address=104.36.82.8:4969 , piece_id=3D3YAZSP7CD6IFZGDKPNKIJPWZ2OJFFVJCSMTO42VNJEFVU3ORHA, seg_pos={0 0}, upload=211112]: Uploading piece... Aug 17 04:23:05.868765063 (202h57m0.299845195s) [upload=211112]: Next returned nil segment Aug 17 04:23:05.868801043 (202h57m0.299881164s) [upload=211112]: Waiting for error group managing segments... Aug 17 04:23:06.147430529 (202h57m0.57851066s) [noise=true , address=45.91.116.12:4227 , piece_id=XZR3AWJG57RUB4TN7SUWNIWWD2XGKCMDXSWSM74WS3ATHS32EBJQ, seg_pos={0 0}, upload=211112]: Done uploading piece. err: <nil> Aug 17 04:23:06.182400314 (202h57m0.613480445s) [noise=true , address=45.91.116.12:4225 , piece_id=EUPPM5NRIU5MCZBER3VGZBZEMZZ65Q2MOY2DXKFBIPFJ53M5TJHQ, seg_pos={0 0}, upload=211112]: Done uploading piece. err: <nil> Aug 17 04:23:06.191910532 (202h57m0.622990663s) [noise=true , address=45.91.116.4:4938 , piece_id=X5MDNBZYRYYOXDEAMGIOD5CXX7PGAMTFRMROOVTR6NIAYQ5VMXGA, seg_pos={0 0}, upload=211112]: Done uploading piece. err: <nil> Aug 17 04:23:06.197391631 (202h57m0.628471782s) [noise=true , address=45.91.116.9:4261 , piece_id=PNOEBFIPLWM4B2QLGJUVEBTT23J5AVRFTWTTKLHFUJ5MNJTSM2JA, seg_pos={0 0}, upload=211112]: Done uploading piece. err: <nil> Aug 17 04:23:06.198069487 (202h57m0.629149618s) [noise=true , address=45.91.116.9:4263 , piece_id=VT734KJZ7T3CZ3W3P4M4ZCYKOLLTPP3ILFRWL26Z7K4LWBDZV4FA, seg_pos={0 0}, upload=211112]: Done uploading piece. err: <nil> Aug 17 04:23:06.200597014 (202h57m0.631677145s) [noise=true , address=45.91.116.7:4009 , piece_id=IVTDHCTNJEEO3QATGN25YH5Y2MSEQUWJIFPXSJWINUEXDHQAGELA, seg_pos={0 0}, upload=211112]: Done uploading piece. err: <nil> Aug 17 04:23:06.288053251 (202h57m0.719133372s) [noise=true , address=45.91.116.7:4010 , piece_id=HNJU4MPWT67V26J5XDCTHFEOOJL2HS3VAOHB4EIPQVF4D2NNRFMA, seg_pos={0 0}, upload=211112]: Done uploading piece. err: <nil> Aug 17 04:23:06.298810435 (202h57m0.729890566s) [noise=true , address=45.91.116.14:4189 , piece_id=2P3GOZDEMJGBAK2F6SY7NJNQUTS5ZYM4B4IMQFAA5TYFARD4DVCQ, seg_pos={0 0}, upload=211112]: Done uploading piece. err: <nil> Aug 17 04:23:06.301049520 (202h57m0.732129651s) [noise=true , address=45.91.116.13:4155 , piece_id=CO3R5XZGH3NJADAFDILPNZEKKFJSRPB4JOFBBCSYYQAD3KNRABNA, seg_pos={0 0}, upload=211112]: Done uploading piece. err: <nil> Aug 17 04:23:06.402776985 (202h57m0.833857116s) [noise=true , address=77.90.31.16:4735 , piece_id=G5YGF7LXNPBU2TDRYPAFPVYQSTWUGG7Y6YM2Q6RWMCZY5RVNM4BA, seg_pos={0 0}, upload=211112]: Done uploading piece. err: <nil> Aug 17 04:23:33.223003412 (202h57m27.654083543s) [noise=true , address=104.36.82.5:4838 , piece_id=IYA7ERDAAFYUGC4666CBEX5HZEXZVFPES6JUYRZROY25OAPGVEQQ, seg_pos={0 0}, upload=211112]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:104.36.82.5:4838): protocol: write tcp 188.166.225.170:42258->104.36.82.5:4838: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:33.223050702 (202h57m27.654130823s) [noise=true , address=104.36.82.8:4969 , piece_id=3D3YAZSP7CD6IFZGDKPNKIJPWZ2OJFFVJCSMTO42VNJEFVU3ORHA, seg_pos={0 0}, upload=211112]: Done uploading piece. err: ecclient: upload failed (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, address:104.36.82.8:4969): protocol: write tcp 188.166.225.170:52308->104.36.82.8:4969: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:33.223090619 (202h57m27.65417075s) [noise=true , address=104.36.82.15:4257 , piece_id=Q5GLP6TS7N7CM4DO3LWDCDR6O76OFZ6IVQJC5VQWMA5XLNWZ2E4A, seg_pos={0 0}, upload=211112]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:104.36.82.15:4257): protocol: write tcp 188.166.225.170:53078->104.36.82.15:4257: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:33.223104976 (202h57m27.654185097s) [noise=true , address=104.36.82.4:4804 , piece_id=MSAHN2DERVG2HL7KZMKIWYUQYO6LRIXGUNWJFXJDDGOWZT54FPVA, seg_pos={0 0}, upload=211112]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:104.36.82.4:4804): protocol: write tcp 188.166.225.170:56176->104.36.82.4:4804: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:33.223149581 (202h57m27.654229712s) [noise=true , address=104.36.82.12:4151 , piece_id=2GGYMOWZ2FTWDNWNQRKSAH7UVJ4R6MLRGC6OHNVV3KTVIIMO4QIQ, seg_pos={0 0}, upload=211112]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:104.36.82.12:4151): protocol: write tcp 188.166.225.170:40582->104.36.82.12:4151: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:33.223157536 (202h57m27.654237667s) [noise=true , address=104.36.82.9:4005 , piece_id=FAR4B5KAALBOACKANEFKGECI2ZVXSHBNYDEIWVGDU2RNBNQXLMXA, seg_pos={0 0}, upload=211112]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:104.36.82.9:4005): protocol: write tcp 188.166.225.170:32954->104.36.82.9:4005: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:33.223177415 (202h57m27.654257536s) [seg_pos={0 0}, upload=211112]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Aug 17 04:23:33.448862924 (202h57m27.879943045s) [upload=211112]: Done uploading segments... Aug 17 04:23:33.448886830 (202h57m27.879966961s) [upload=211112]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 5.83426815s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)