Orbiter History

checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [........X.....X.............................................] duration: 5.887304465s FAIL (2025-08-19 17:57:06 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.887327238s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 6.134322332s FAIL (2025-08-19 17:57:03 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Got next segment. Inline: false Aug 19 17:23:57.290467622 (263h57m51.721547763s) [upload=274428]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 17:23:57.290471560 (263h57m51.721551681s) [upload=274428]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 19 17:23:57.743765742 (263h57m52.174845863s) [seg_pos={0 0}, upload=274428]: Begin upload segment... Aug 19 17:23:57.743818412 (263h57m52.174898543s) [seg_pos={0 0}, upload=274428]: Done begin upload segment. Aug 19 17:23:57.743842609 (263h57m52.17492274s) [noise=true , address=77.90.31.16:4750 , piece_id=XXJ4VHMZ4NVVT2WX322JS7MW3FHQ7UZJW4VLR4KG6FW4A4RHVN6A, seg_pos={0 0}, upload=274428]: Uploading piece... Aug 19 17:23:57.743875672 (263h57m52.174955803s) [noise=true , address=45.91.116.14:4198 , piece_id=NSLSBMDS2ZJKN62N4NGN7HRHXAUIS4WL4AGHODZZBD2CDYTXHVUA, seg_pos={0 0}, upload=274428]: Uploading piece... Aug 19 17:23:57.744218498 (263h57m52.175298629s) [noise=true , address=77.90.31.14:4667 , piece_id=BF53QUGTGVD23KCIIAA5NWUCWKWUXXSHKCAAWSDZLAORPHXQEMQQ, seg_pos={0 0}, upload=274428]: Uploading piece... Aug 19 17:23:57.744456373 (263h57m52.175536504s) [noise=true , address=77.90.31.21:4883 , piece_id=SVFJCG4T5OCDMVKBE7A33XPDIQUH6NESE4GIVTETRZRBX4N3MJDQ, seg_pos={0 0}, upload=274428]: Uploading piece... Aug 19 17:23:57.744523051 (263h57m52.175603182s) [noise=true , address=104.36.82.60:4365 , piece_id=7BXP5MWOH2B3H74CPVWOQN2NEGMXUYECIVOEHRRKINQ544WKAAHA, seg_pos={0 0}, upload=274428]: Uploading piece... Aug 19 17:23:57.744762870 (263h57m52.175843001s) [noise=true , address=77.90.31.19:4851 , piece_id=POYQI453UA5767REISB5BNE3MYV6IOYZNIK357PYGH5DM66EOBIA, seg_pos={0 0}, upload=274428]: Uploading piece... Aug 19 17:23:57.744993531 (263h57m52.176073662s) [noise=true , address=104.36.82.44:4871 , piece_id=WQGOUIWA7NVK5SUTIWX6E32ZKW62Y2KGK7E3HB5YW2CKDUR42BPQ, seg_pos={0 0}, upload=274428]: Uploading piece... Aug 19 17:23:57.745229373 (263h57m52.176309494s) [noise=true , address=79.127.205.234:4587 , piece_id=ALBAMUJVGA4RIJKGCJNJT3BT6RBD5QWU4C33RJGCEP3XIQQLW6QQ, seg_pos={0 0}, upload=274428]: Uploading piece... Aug 19 17:23:57.745473950 (263h57m52.176554081s) [noise=true , address=104.36.82.3:4771 , piece_id=2RZZG2MORQZR3ROAADIWSK46JDEZSNYBGASCDWZTKWWU72JFHRSQ, seg_pos={0 0}, upload=274428]: Uploading piece... Aug 19 17:23:57.745746030 (263h57m52.176826161s) [noise=true , address=104.36.82.34:4558 , piece_id=AJNHQTDQSMJ3R2ZKVQUCBU5XOMWNF66ID2M3RVPYDL52F5LGTPFQ, seg_pos={0 0}, upload=274428]: Uploading piece... Aug 19 17:23:57.745999426 (263h57m52.177079547s) [noise=true , address=77.90.31.18:4814 , piece_id=VUIFMMFA5PCHIKNNX7PA7RHVXIJHK3UWH4SI34R6XP5MEYTAMS6A, seg_pos={0 0}, upload=274428]: Uploading piece... Aug 19 17:23:57.746774839 (263h57m52.17785497s) [noise=true , address=104.36.82.35:4610 , piece_id=DM7HADQAK5T67ZDQR3VGUUGLHCGBLC5KJYYFF7QQNRKVE2SA5CHA, seg_pos={0 0}, upload=274428]: Uploading piece... Aug 19 17:23:57.747503714 (263h57m52.178583836s) [noise=true , address=64.49.13.137:4496 , piece_id=DW5YMZIQJZIA53K22HGK2ICRT5IVVP2ZMIVYM46BWQWAYPLNUC2Q, seg_pos={0 0}, upload=274428]: Uploading piece... Aug 19 17:23:57.747727743 (263h57m52.178807874s) [noise=true , address=77.90.31.13:4640 , piece_id=NKLBPZJDAIAVAYH3QRDCPS6VWHJBFZZWY3HCTSW2E4II77YC3BYA, seg_pos={0 0}, upload=274428]: Uploading piece... Aug 19 17:23:57.747948565 (263h57m52.179028696s) [noise=true , address=104.36.82.55:4214 , piece_id=64P34K2YEWCWZFGWSR4AQ5SRALT5CSZCB4F6FHPKDP3ZFJIOJC2Q, seg_pos={0 0}, upload=274428]: Uploading piece... Aug 19 17:23:57.748174007 (263h57m52.179254138s) [noise=true , address=104.36.82.23:4448 , piece_id=SCMNBDCYA7PUN2VP6JN37YM5LNY2GBGQHJKYSSWUWMPTZZ7LXZUQ, seg_pos={0 0}, upload=274428]: Uploading piece... Aug 19 17:23:57.748433112 (263h57m52.179513233s) [noise=true , address=104.36.82.5:4854 , piece_id=FC4DIKMBI5OPLDQCVA6P3MA7AQZV23QBIX3UAKLWJJMY724TNYHQ, seg_pos={0 0}, upload=274428]: Uploading piece... Aug 19 17:23:57.748647252 (263h57m52.179727383s) [noise=true , address=104.36.82.62:4442 , piece_id=VV5BBYB6UI4O3B4ESOBJX44I5RJBRWSLJUFR6H3GKEUZWNONTPMA, seg_pos={0 0}, upload=274428]: Uploading piece... Aug 19 17:23:58.869276532 (263h57m53.300356663s) [upload=274428]: Next returned nil segment Aug 19 17:23:58.869291511 (263h57m53.300371642s) [upload=274428]: Waiting for error group managing segments... Aug 19 17:23:59.195579562 (263h57m53.626659683s) [noise=true , address=79.127.205.234:4587 , piece_id=ALBAMUJVGA4RIJKGCJNJT3BT6RBD5QWU4C33RJGCEP3XIQQLW6QQ, seg_pos={0 0}, upload=274428]: Done uploading piece. err: <nil> Aug 19 17:23:59.239543682 (263h57m53.670623813s) [noise=true , address=64.49.13.137:4496 , piece_id=DW5YMZIQJZIA53K22HGK2ICRT5IVVP2ZMIVYM46BWQWAYPLNUC2Q, seg_pos={0 0}, upload=274428]: Done uploading piece. err: <nil> Aug 19 17:23:59.654494638 (263h57m54.085574769s) [noise=true , address=104.36.82.35:4610 , piece_id=DM7HADQAK5T67ZDQR3VGUUGLHCGBLC5KJYYFF7QQNRKVE2SA5CHA, seg_pos={0 0}, upload=274428]: Done uploading piece. err: <nil> Aug 19 17:23:59.700507838 (263h57m54.131587959s) [noise=true , address=104.36.82.34:4558 , piece_id=AJNHQTDQSMJ3R2ZKVQUCBU5XOMWNF66ID2M3RVPYDL52F5LGTPFQ, seg_pos={0 0}, upload=274428]: Done uploading piece. err: <nil> Aug 19 17:23:59.757508140 (263h57m54.188588272s) [noise=true , address=104.36.82.5:4854 , piece_id=FC4DIKMBI5OPLDQCVA6P3MA7AQZV23QBIX3UAKLWJJMY724TNYHQ, seg_pos={0 0}, upload=274428]: Done uploading piece. err: <nil> Aug 19 17:23:59.765247029 (263h57m54.19632716s) [noise=true , address=104.36.82.62:4442 , piece_id=VV5BBYB6UI4O3B4ESOBJX44I5RJBRWSLJUFR6H3GKEUZWNONTPMA, seg_pos={0 0}, upload=274428]: Done uploading piece. err: <nil> Aug 19 17:23:59.774367630 (263h57m54.205447761s) [noise=true , address=104.36.82.23:4448 , piece_id=SCMNBDCYA7PUN2VP6JN37YM5LNY2GBGQHJKYSSWUWMPTZZ7LXZUQ, seg_pos={0 0}, upload=274428]: Done uploading piece. err: <nil> Aug 19 17:23:59.827970201 (263h57m54.259050332s) [noise=true , address=104.36.82.55:4214 , piece_id=64P34K2YEWCWZFGWSR4AQ5SRALT5CSZCB4F6FHPKDP3ZFJIOJC2Q, seg_pos={0 0}, upload=274428]: Done uploading piece. err: <nil> Aug 19 17:23:59.851412395 (263h57m54.282492526s) [noise=true , address=104.36.82.44:4871 , piece_id=WQGOUIWA7NVK5SUTIWX6E32ZKW62Y2KGK7E3HB5YW2CKDUR42BPQ, seg_pos={0 0}, upload=274428]: Done uploading piece. err: <nil> Aug 19 17:23:59.946562012 (263h57m54.377642143s) [noise=true , address=104.36.82.3:4771 , piece_id=2RZZG2MORQZR3ROAADIWSK46JDEZSNYBGASCDWZTKWWU72JFHRSQ, seg_pos={0 0}, upload=274428]: Done uploading piece. err: <nil> Aug 19 17:23:59.979671513 (263h57m54.410751634s) [noise=true , address=104.36.82.60:4365 , piece_id=7BXP5MWOH2B3H74CPVWOQN2NEGMXUYECIVOEHRRKINQ544WKAAHA, seg_pos={0 0}, upload=274428]: Done uploading piece. err: <nil> Aug 19 17:24:55.494274881 (263h58m49.925355012s) [noise=true , address=77.90.31.18:4814 , piece_id=VUIFMMFA5PCHIKNNX7PA7RHVXIJHK3UWH4SI34R6XP5MEYTAMS6A, seg_pos={0 0}, upload=274428]: Done uploading piece. err: <nil> Aug 19 17:24:55.842835997 (263h58m50.273916128s) [noise=true , address=77.90.31.21:4883 , piece_id=SVFJCG4T5OCDMVKBE7A33XPDIQUH6NESE4GIVTETRZRBX4N3MJDQ, seg_pos={0 0}, upload=274428]: Done uploading piece. err: <nil> Aug 19 17:24:56.634889530 (263h58m51.065969661s) [noise=true , address=77.90.31.19:4851 , piece_id=POYQI453UA5767REISB5BNE3MYV6IOYZNIK357PYGH5DM66EOBIA, seg_pos={0 0}, upload=274428]: Done uploading piece. err: ecclient: upload failed (node:1NEV2ndHw4fohVUNsfrzwojd8S11TmMST6Sz59GVnXS8vDGhWk, address:77.90.31.19:4851): protocol: rpc: dial tcp 77.90.31.19:4851: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:56.635097748 (263h58m51.066177879s) [noise=true , address=45.91.116.14:4198 , piece_id=NSLSBMDS2ZJKN62N4NGN7HRHXAUIS4WL4AGHODZZBD2CDYTXHVUA, seg_pos={0 0}, upload=274428]: Done uploading piece. err: ecclient: upload failed (node:1xxfUrCm5PgSLpge1wYiH2EcE7kzzJ38P9rfeCYmqXTbTisioi, address:45.91.116.14:4198): protocol: rpc: dial tcp 45.91.116.14:4198: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:56.635158895 (263h58m51.066239036s) [noise=true , address=77.90.31.16:4750 , piece_id=XXJ4VHMZ4NVVT2WX322JS7MW3FHQ7UZJW4VLR4KG6FW4A4RHVN6A, seg_pos={0 0}, upload=274428]: Done uploading piece. err: ecclient: upload failed (node:12TGd8NnWdekDtHzBQHmJPgTtHjXC6g8fi6RoZYFQJVHDa5hdyE, address:77.90.31.16:4750): protocol: read tcp 188.166.225.170:54198->77.90.31.16:4750: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:56.635224120 (263h58m51.066304281s) [noise=true , address=77.90.31.14:4667 , piece_id=BF53QUGTGVD23KCIIAA5NWUCWKWUXXSHKCAAWSDZLAORPHXQEMQQ, seg_pos={0 0}, upload=274428]: Done uploading piece. err: ecclient: upload failed (node:12R5gJFhsmocYPNZf4yNnYqJwcwAKt9yiHmmqGMk3btzVjkFV4S, address:77.90.31.14:4667): protocol: read tcp 188.166.225.170:46984->77.90.31.14:4667: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:56.635280057 (263h58m51.066360198s) [noise=true , address=77.90.31.13:4640 , piece_id=NKLBPZJDAIAVAYH3QRDCPS6VWHJBFZZWY3HCTSW2E4II77YC3BYA, seg_pos={0 0}, upload=274428]: Done uploading piece. err: ecclient: upload failed (node:1XjJpLBEZm5daHcBUmUyC7QpLVhu9Vm9ex9RYcYJWEuLGk527F, address:77.90.31.13:4640): protocol: read tcp 188.166.225.170:48656->77.90.31.13:4640: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:56.635314653 (263h58m51.066394794s) [seg_pos={0 0}, upload=274428]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Aug 19 17:24:56.862515362 (263h58m51.293595493s) [upload=274428]: Done uploading segments... Aug 19 17:24:56.862536042 (263h58m51.293616163s) [upload=274428]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 6.134378007s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........X...X.............................................] duration: 5.841820455s FAIL (2025-08-19 17:56:56 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 23:59.057943072 (263h57m53.489023273s) [noise=true , address=45.91.116.10:4083 , piece_id=V3H4T2WN2MBVALSIHZNOQHBE75WQRNYLIE3Z7DFWUUHBPYDT7IFA, seg_pos={0 0}, upload=274429]: Uploading piece... Aug 19 17:23:59.057955436 (263h57m53.489035557s) [noise=true , address=45.91.116.5:4298 , piece_id=HMF6DXO5KFMLK67WVA7BAWLQFLC2BCVPJ7ZT255FMRCCEI3MCBNA, seg_pos={0 0}, upload=274429]: Uploading piece... Aug 19 17:23:59.058557689 (263h57m53.48963782s) [noise=true , address=104.36.82.8:4969 , piece_id=3P655Z5OUVZ4VRESACEBKYR3KUFYY56NYPWX7ZVOHL3XI2YQ5TCQ, seg_pos={0 0}, upload=274429]: Uploading piece... Aug 19 17:23:59.059170771 (263h57m53.490250902s) [noise=true , address=45.91.116.7:4009 , piece_id=NSZYKL6FHDLDM463ZXKEQGX4ZY7JPJXENH63PPDV76XMGCBV7ZRQ, seg_pos={0 0}, upload=274429]: Uploading piece... Aug 19 17:23:59.059468281 (263h57m53.490548412s) [noise=true , address=77.90.31.15:4699 , piece_id=MFF7PNMDLHYEH7PY46MGGOJSY4XPHRDPGFWJABBO4JDTJJJJ3CTQ, seg_pos={0 0}, upload=274429]: Uploading piece... Aug 19 17:23:59.059498208 (263h57m53.490578339s) [noise=true , address=104.36.82.2:4729 , piece_id=BNX6UNYZ7VHQ4MKAH65DGSHULWRRXDSCMKOE7WFZO6T2V3ELBJ7Q, seg_pos={0 0}, upload=274429]: Uploading piece... Aug 19 17:23:59.059814182 (263h57m53.490894323s) [noise=true , address=104.36.82.6:4874 , piece_id=QMGNWSGMEV762Y2XEDK2LIAY3JMNLD3UOVFQ6HYP7ZMLLVAVO7ZQ, seg_pos={0 0}, upload=274429]: Uploading piece... Aug 19 17:23:59.059839941 (263h57m53.490920072s) [noise=true , address=45.91.116.12:4225 , piece_id=PCJYR2VLFH23UPQEABUYESR3MHKDHIVTJWDIWVYIL6JAFGGXXXFQ, seg_pos={0 0}, upload=274429]: Uploading piece... Aug 19 17:23:59.059875480 (263h57m53.490955611s) [noise=true , address=104.36.82.10:4079 , piece_id=S3BSVJWRPP2PDGLG7FKFROUKAM3GTBDL2ALAJPAZQQQZJVRMMZ3Q, seg_pos={0 0}, upload=274429]: Uploading piece... Aug 19 17:23:59.060109147 (263h57m53.491189278s) [noise=true , address=104.36.82.2:4732 , piece_id=NLTK4572JAQHZQRDD2DOOIW7VBO7NK4DI4RUZRSXQ3GR72EWV3KQ, seg_pos={0 0}, upload=274429]: Uploading piece... Aug 19 17:23:59.060142190 (263h57m53.491222321s) [noise=true , address=45.91.116.9:4261 , piece_id=J5IMFL3JW4YKDLI6OEY2POEB4LJR2P3BIFAD4YIF3E4JTUXIXY3A, seg_pos={0 0}, upload=274429]: Uploading piece... Aug 19 17:23:59.060348274 (263h57m53.491428405s) [noise=true , address=104.36.82.15:4259 , piece_id=RM7JL7JKCMFM4CNXVVZQ2QJTOEPQEQHGLW3GYXWUD6LAQ2ZJ3P5Q, seg_pos={0 0}, upload=274429]: Uploading piece... Aug 19 17:23:59.060544830 (263h57m53.491624961s) [noise=true , address=45.91.116.14:4191 , piece_id=PJ33GP36S76VCI5RVR26PBFZ6HZUUZYVWNGPJITRLGDZI6LWCJIQ, seg_pos={0 0}, upload=274429]: Uploading piece... Aug 19 17:23:59.060752227 (263h57m53.491832358s) [noise=true , address=45.91.116.11:4118 , piece_id=VD2HK2QU63UVJFRASQOIRFZRWZN2HAUSAZJL2VZKYALGSV67SP2A, seg_pos={0 0}, upload=274429]: Uploading piece... Aug 19 17:23:59.060967990 (263h57m53.492048121s) [noise=true , address=77.90.31.11:4555 , piece_id=GTU6PGCHI4WOFGK2B3FP2Y22VUDBWLQKKEMZBT6I4F47OKK4RPNQ, seg_pos={0 0}, upload=274429]: Uploading piece... Aug 19 17:23:59.061294044 (263h57m53.492374175s) [noise=true , address=45.91.116.8:4048 , piece_id=M2CBILQKSVSQ7VY4AWQQM5GHF2KYBSXH73YJGAEJZAHOR7FOBWEQ, seg_pos={0 0}, upload=274429]: Uploading piece... Aug 19 17:24:00.872920400 (263h57m55.304000531s) [upload=274429]: Next returned nil segment Aug 19 17:24:00.872945548 (263h57m55.304025679s) [upload=274429]: Waiting for error group managing segments... Aug 19 17:24:01.156214888 (263h57m55.587295019s) [noise=true , address=104.36.82.6:4874 , piece_id=QMGNWSGMEV762Y2XEDK2LIAY3JMNLD3UOVFQ6HYP7ZMLLVAVO7ZQ, seg_pos={0 0}, upload=274429]: Done uploading piece. err: <nil> Aug 19 17:24:01.181164926 (263h57m55.612245057s) [noise=true , address=104.36.82.2:4732 , piece_id=NLTK4572JAQHZQRDD2DOOIW7VBO7NK4DI4RUZRSXQ3GR72EWV3KQ, seg_pos={0 0}, upload=274429]: Done uploading piece. err: <nil> Aug 19 17:24:01.222047199 (263h57m55.653127331s) [noise=true , address=104.36.82.2:4729 , piece_id=BNX6UNYZ7VHQ4MKAH65DGSHULWRRXDSCMKOE7WFZO6T2V3ELBJ7Q, seg_pos={0 0}, upload=274429]: Done uploading piece. err: <nil> Aug 19 17:24:01.250473409 (263h57m55.68155354s) [noise=true , address=104.36.82.15:4259 , piece_id=RM7JL7JKCMFM4CNXVVZQ2QJTOEPQEQHGLW3GYXWUD6LAQ2ZJ3P5Q, seg_pos={0 0}, upload=274429]: Done uploading piece. err: <nil> Aug 19 17:24:01.259222299 (263h57m55.69030242s) [noise=true , address=104.36.82.8:4969 , piece_id=3P655Z5OUVZ4VRESACEBKYR3KUFYY56NYPWX7ZVOHL3XI2YQ5TCQ, seg_pos={0 0}, upload=274429]: Done uploading piece. err: <nil> Aug 19 17:24:01.261735818 (263h57m55.692815949s) [noise=true , address=104.36.82.10:4079 , piece_id=S3BSVJWRPP2PDGLG7FKFROUKAM3GTBDL2ALAJPAZQQQZJVRMMZ3Q, seg_pos={0 0}, upload=274429]: Done uploading piece. err: <nil> Aug 19 17:24:28.183964515 (263h58m22.615044646s) [noise=true , address=45.91.116.14:4191 , piece_id=PJ33GP36S76VCI5RVR26PBFZ6HZUUZYVWNGPJITRLGDZI6LWCJIQ, seg_pos={0 0}, upload=274429]: Done uploading piece. err: ecclient: upload failed (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, address:45.91.116.14:4191): protocol: read tcp 188.166.225.170:37646->45.91.116.14:4191: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:28.184013127 (263h58m22.615093258s) [noise=true , address=45.91.116.9:4261 , piece_id=J5IMFL3JW4YKDLI6OEY2POEB4LJR2P3BIFAD4YIF3E4JTUXIXY3A, seg_pos={0 0}, upload=274429]: Done uploading piece. err: ecclient: upload failed (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, address:45.91.116.9:4261): protocol: rpc: dial tcp 45.91.116.9:4261: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:28.184026694 (263h58m22.615106825s) [noise=true , address=45.91.116.8:4048 , piece_id=M2CBILQKSVSQ7VY4AWQQM5GHF2KYBSXH73YJGAEJZAHOR7FOBWEQ, seg_pos={0 0}, upload=274429]: Done uploading piece. err: ecclient: upload failed (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, address:45.91.116.8:4048): protocol: read tcp 188.166.225.170:45604->45.91.116.8:4048: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:28.184070568 (263h58m22.615150689s) [noise=true , address=45.91.116.5:4298 , piece_id=HMF6DXO5KFMLK67WVA7BAWLQFLC2BCVPJ7ZT255FMRCCEI3MCBNA, seg_pos={0 0}, upload=274429]: Done uploading piece. err: ecclient: upload failed (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, address:45.91.116.5:4298): protocol: read tcp 188.166.225.170:46970->45.91.116.5:4298: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:28.184103010 (263h58m22.615183141s) [noise=true , address=45.91.116.10:4083 , piece_id=V3H4T2WN2MBVALSIHZNOQHBE75WQRNYLIE3Z7DFWUUHBPYDT7IFA, seg_pos={0 0}, upload=274429]: Done uploading piece. err: ecclient: upload failed (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, address:45.91.116.10:4083): protocol: read tcp 188.166.225.170:41632->45.91.116.10:4083: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:28.184139489 (263h58m22.61521962s) [noise=true , address=45.91.116.7:4009 , piece_id=NSZYKL6FHDLDM463ZXKEQGX4ZY7JPJXENH63PPDV76XMGCBV7ZRQ, seg_pos={0 0}, upload=274429]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: read tcp 188.166.225.170:45756->45.91.116.7:4009: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:28.184171009 (263h58m22.615251141s) [noise=true , address=77.90.31.11:4555 , piece_id=GTU6PGCHI4WOFGK2B3FP2Y22VUDBWLQKKEMZBT6I4F47OKK4RPNQ, seg_pos={0 0}, upload=274429]: Done uploading piece. err: ecclient: upload failed (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, address:77.90.31.11:4555): protocol: read tcp 188.166.225.170:37438->77.90.31.11:4555: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:28.184197480 (263h58m22.615277611s) [noise=true , address=77.90.31.15:4699 , piece_id=MFF7PNMDLHYEH7PY46MGGOJSY4XPHRDPGFWJABBO4JDTJJJJ3CTQ, seg_pos={0 0}, upload=274429]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: read tcp 188.166.225.170:36788->77.90.31.15:4699: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:28.184218410 (263h58m22.615298541s) [noise=true , address=45.91.116.12:4225 , piece_id=PCJYR2VLFH23UPQEABUYESR3MHKDHIVTJWDIWVYIL6JAFGGXXXFQ, seg_pos={0 0}, upload=274429]: Done uploading piece. err: ecclient: upload failed (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, address:45.91.116.12:4225): protocol: read tcp 188.166.225.170:60636->45.91.116.12:4225: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:28.184249710 (263h58m22.615329841s) [noise=true , address=45.91.116.11:4118 , piece_id=VD2HK2QU63UVJFRASQOIRFZRWZN2HAUSAZJL2VZKYALGSV67SP2A, seg_pos={0 0}, upload=274429]: Done uploading piece. err: ecclient: upload failed (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, address:45.91.116.11:4118): protocol: read tcp 188.166.225.170:48006->45.91.116.11:4118: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:28.184274337 (263h58m22.615354468s) [seg_pos={0 0}, upload=274429]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Aug 19 17:24:28.412483516 (263h58m22.843563647s) [upload=274429]: Done uploading segments... Aug 19 17:24:28.412520628 (263h58m22.843600759s) [upload=274429]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 5.841851766s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 5.656256034s FAIL (2025-08-19 17:56:35 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Got next segment. Inline: false Aug 19 17:23:57.290467622 (263h57m51.721547763s) [upload=274428]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 17:23:57.290471560 (263h57m51.721551681s) [upload=274428]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 19 17:23:57.743765742 (263h57m52.174845863s) [seg_pos={0 0}, upload=274428]: Begin upload segment... Aug 19 17:23:57.743818412 (263h57m52.174898543s) [seg_pos={0 0}, upload=274428]: Done begin upload segment. Aug 19 17:23:57.743842609 (263h57m52.17492274s) [noise=true , address=77.90.31.16:4750 , piece_id=XXJ4VHMZ4NVVT2WX322JS7MW3FHQ7UZJW4VLR4KG6FW4A4RHVN6A, seg_pos={0 0}, upload=274428]: Uploading piece... Aug 19 17:23:57.743875672 (263h57m52.174955803s) [noise=true , address=45.91.116.14:4198 , piece_id=NSLSBMDS2ZJKN62N4NGN7HRHXAUIS4WL4AGHODZZBD2CDYTXHVUA, seg_pos={0 0}, upload=274428]: Uploading piece... Aug 19 17:23:57.744218498 (263h57m52.175298629s) [noise=true , address=77.90.31.14:4667 , piece_id=BF53QUGTGVD23KCIIAA5NWUCWKWUXXSHKCAAWSDZLAORPHXQEMQQ, seg_pos={0 0}, upload=274428]: Uploading piece... Aug 19 17:23:57.744456373 (263h57m52.175536504s) [noise=true , address=77.90.31.21:4883 , piece_id=SVFJCG4T5OCDMVKBE7A33XPDIQUH6NESE4GIVTETRZRBX4N3MJDQ, seg_pos={0 0}, upload=274428]: Uploading piece... Aug 19 17:23:57.744523051 (263h57m52.175603182s) [noise=true , address=104.36.82.60:4365 , piece_id=7BXP5MWOH2B3H74CPVWOQN2NEGMXUYECIVOEHRRKINQ544WKAAHA, seg_pos={0 0}, upload=274428]: Uploading piece... Aug 19 17:23:57.744762870 (263h57m52.175843001s) [noise=true , address=77.90.31.19:4851 , piece_id=POYQI453UA5767REISB5BNE3MYV6IOYZNIK357PYGH5DM66EOBIA, seg_pos={0 0}, upload=274428]: Uploading piece... Aug 19 17:23:57.744993531 (263h57m52.176073662s) [noise=true , address=104.36.82.44:4871 , piece_id=WQGOUIWA7NVK5SUTIWX6E32ZKW62Y2KGK7E3HB5YW2CKDUR42BPQ, seg_pos={0 0}, upload=274428]: Uploading piece... Aug 19 17:23:57.745229373 (263h57m52.176309494s) [noise=true , address=79.127.205.234:4587 , piece_id=ALBAMUJVGA4RIJKGCJNJT3BT6RBD5QWU4C33RJGCEP3XIQQLW6QQ, seg_pos={0 0}, upload=274428]: Uploading piece... Aug 19 17:23:57.745473950 (263h57m52.176554081s) [noise=true , address=104.36.82.3:4771 , piece_id=2RZZG2MORQZR3ROAADIWSK46JDEZSNYBGASCDWZTKWWU72JFHRSQ, seg_pos={0 0}, upload=274428]: Uploading piece... Aug 19 17:23:57.745746030 (263h57m52.176826161s) [noise=true , address=104.36.82.34:4558 , piece_id=AJNHQTDQSMJ3R2ZKVQUCBU5XOMWNF66ID2M3RVPYDL52F5LGTPFQ, seg_pos={0 0}, upload=274428]: Uploading piece... Aug 19 17:23:57.745999426 (263h57m52.177079547s) [noise=true , address=77.90.31.18:4814 , piece_id=VUIFMMFA5PCHIKNNX7PA7RHVXIJHK3UWH4SI34R6XP5MEYTAMS6A, seg_pos={0 0}, upload=274428]: Uploading piece... Aug 19 17:23:57.746774839 (263h57m52.17785497s) [noise=true , address=104.36.82.35:4610 , piece_id=DM7HADQAK5T67ZDQR3VGUUGLHCGBLC5KJYYFF7QQNRKVE2SA5CHA, seg_pos={0 0}, upload=274428]: Uploading piece... Aug 19 17:23:57.747503714 (263h57m52.178583836s) [noise=true , address=64.49.13.137:4496 , piece_id=DW5YMZIQJZIA53K22HGK2ICRT5IVVP2ZMIVYM46BWQWAYPLNUC2Q, seg_pos={0 0}, upload=274428]: Uploading piece... Aug 19 17:23:57.747727743 (263h57m52.178807874s) [noise=true , address=77.90.31.13:4640 , piece_id=NKLBPZJDAIAVAYH3QRDCPS6VWHJBFZZWY3HCTSW2E4II77YC3BYA, seg_pos={0 0}, upload=274428]: Uploading piece... Aug 19 17:23:57.747948565 (263h57m52.179028696s) [noise=true , address=104.36.82.55:4214 , piece_id=64P34K2YEWCWZFGWSR4AQ5SRALT5CSZCB4F6FHPKDP3ZFJIOJC2Q, seg_pos={0 0}, upload=274428]: Uploading piece... Aug 19 17:23:57.748174007 (263h57m52.179254138s) [noise=true , address=104.36.82.23:4448 , piece_id=SCMNBDCYA7PUN2VP6JN37YM5LNY2GBGQHJKYSSWUWMPTZZ7LXZUQ, seg_pos={0 0}, upload=274428]: Uploading piece... Aug 19 17:23:57.748433112 (263h57m52.179513233s) [noise=true , address=104.36.82.5:4854 , piece_id=FC4DIKMBI5OPLDQCVA6P3MA7AQZV23QBIX3UAKLWJJMY724TNYHQ, seg_pos={0 0}, upload=274428]: Uploading piece... Aug 19 17:23:57.748647252 (263h57m52.179727383s) [noise=true , address=104.36.82.62:4442 , piece_id=VV5BBYB6UI4O3B4ESOBJX44I5RJBRWSLJUFR6H3GKEUZWNONTPMA, seg_pos={0 0}, upload=274428]: Uploading piece... Aug 19 17:23:58.869276532 (263h57m53.300356663s) [upload=274428]: Next returned nil segment Aug 19 17:23:58.869291511 (263h57m53.300371642s) [upload=274428]: Waiting for error group managing segments... Aug 19 17:23:59.195579562 (263h57m53.626659683s) [noise=true , address=79.127.205.234:4587 , piece_id=ALBAMUJVGA4RIJKGCJNJT3BT6RBD5QWU4C33RJGCEP3XIQQLW6QQ, seg_pos={0 0}, upload=274428]: Done uploading piece. err: <nil> Aug 19 17:23:59.239543682 (263h57m53.670623813s) [noise=true , address=64.49.13.137:4496 , piece_id=DW5YMZIQJZIA53K22HGK2ICRT5IVVP2ZMIVYM46BWQWAYPLNUC2Q, seg_pos={0 0}, upload=274428]: Done uploading piece. err: <nil> Aug 19 17:23:59.654494638 (263h57m54.085574769s) [noise=true , address=104.36.82.35:4610 , piece_id=DM7HADQAK5T67ZDQR3VGUUGLHCGBLC5KJYYFF7QQNRKVE2SA5CHA, seg_pos={0 0}, upload=274428]: Done uploading piece. err: <nil> Aug 19 17:23:59.700507838 (263h57m54.131587959s) [noise=true , address=104.36.82.34:4558 , piece_id=AJNHQTDQSMJ3R2ZKVQUCBU5XOMWNF66ID2M3RVPYDL52F5LGTPFQ, seg_pos={0 0}, upload=274428]: Done uploading piece. err: <nil> Aug 19 17:23:59.757508140 (263h57m54.188588272s) [noise=true , address=104.36.82.5:4854 , piece_id=FC4DIKMBI5OPLDQCVA6P3MA7AQZV23QBIX3UAKLWJJMY724TNYHQ, seg_pos={0 0}, upload=274428]: Done uploading piece. err: <nil> Aug 19 17:23:59.765247029 (263h57m54.19632716s) [noise=true , address=104.36.82.62:4442 , piece_id=VV5BBYB6UI4O3B4ESOBJX44I5RJBRWSLJUFR6H3GKEUZWNONTPMA, seg_pos={0 0}, upload=274428]: Done uploading piece. err: <nil> Aug 19 17:23:59.774367630 (263h57m54.205447761s) [noise=true , address=104.36.82.23:4448 , piece_id=SCMNBDCYA7PUN2VP6JN37YM5LNY2GBGQHJKYSSWUWMPTZZ7LXZUQ, seg_pos={0 0}, upload=274428]: Done uploading piece. err: <nil> Aug 19 17:23:59.827970201 (263h57m54.259050332s) [noise=true , address=104.36.82.55:4214 , piece_id=64P34K2YEWCWZFGWSR4AQ5SRALT5CSZCB4F6FHPKDP3ZFJIOJC2Q, seg_pos={0 0}, upload=274428]: Done uploading piece. err: <nil> Aug 19 17:23:59.851412395 (263h57m54.282492526s) [noise=true , address=104.36.82.44:4871 , piece_id=WQGOUIWA7NVK5SUTIWX6E32ZKW62Y2KGK7E3HB5YW2CKDUR42BPQ, seg_pos={0 0}, upload=274428]: Done uploading piece. err: <nil> Aug 19 17:23:59.946562012 (263h57m54.377642143s) [noise=true , address=104.36.82.3:4771 , piece_id=2RZZG2MORQZR3ROAADIWSK46JDEZSNYBGASCDWZTKWWU72JFHRSQ, seg_pos={0 0}, upload=274428]: Done uploading piece. err: <nil> Aug 19 17:23:59.979671513 (263h57m54.410751634s) [noise=true , address=104.36.82.60:4365 , piece_id=7BXP5MWOH2B3H74CPVWOQN2NEGMXUYECIVOEHRRKINQ544WKAAHA, seg_pos={0 0}, upload=274428]: Done uploading piece. err: <nil> Aug 19 17:24:55.494274881 (263h58m49.925355012s) [noise=true , address=77.90.31.18:4814 , piece_id=VUIFMMFA5PCHIKNNX7PA7RHVXIJHK3UWH4SI34R6XP5MEYTAMS6A, seg_pos={0 0}, upload=274428]: Done uploading piece. err: <nil> Aug 19 17:24:55.842835997 (263h58m50.273916128s) [noise=true , address=77.90.31.21:4883 , piece_id=SVFJCG4T5OCDMVKBE7A33XPDIQUH6NESE4GIVTETRZRBX4N3MJDQ, seg_pos={0 0}, upload=274428]: Done uploading piece. err: <nil> Aug 19 17:24:56.634889530 (263h58m51.065969661s) [noise=true , address=77.90.31.19:4851 , piece_id=POYQI453UA5767REISB5BNE3MYV6IOYZNIK357PYGH5DM66EOBIA, seg_pos={0 0}, upload=274428]: Done uploading piece. err: ecclient: upload failed (node:1NEV2ndHw4fohVUNsfrzwojd8S11TmMST6Sz59GVnXS8vDGhWk, address:77.90.31.19:4851): protocol: rpc: dial tcp 77.90.31.19:4851: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:56.635097748 (263h58m51.066177879s) [noise=true , address=45.91.116.14:4198 , piece_id=NSLSBMDS2ZJKN62N4NGN7HRHXAUIS4WL4AGHODZZBD2CDYTXHVUA, seg_pos={0 0}, upload=274428]: Done uploading piece. err: ecclient: upload failed (node:1xxfUrCm5PgSLpge1wYiH2EcE7kzzJ38P9rfeCYmqXTbTisioi, address:45.91.116.14:4198): protocol: rpc: dial tcp 45.91.116.14:4198: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:56.635158895 (263h58m51.066239036s) [noise=true , address=77.90.31.16:4750 , piece_id=XXJ4VHMZ4NVVT2WX322JS7MW3FHQ7UZJW4VLR4KG6FW4A4RHVN6A, seg_pos={0 0}, upload=274428]: Done uploading piece. err: ecclient: upload failed (node:12TGd8NnWdekDtHzBQHmJPgTtHjXC6g8fi6RoZYFQJVHDa5hdyE, address:77.90.31.16:4750): protocol: read tcp 188.166.225.170:54198->77.90.31.16:4750: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:56.635224120 (263h58m51.066304281s) [noise=true , address=77.90.31.14:4667 , piece_id=BF53QUGTGVD23KCIIAA5NWUCWKWUXXSHKCAAWSDZLAORPHXQEMQQ, seg_pos={0 0}, upload=274428]: Done uploading piece. err: ecclient: upload failed (node:12R5gJFhsmocYPNZf4yNnYqJwcwAKt9yiHmmqGMk3btzVjkFV4S, address:77.90.31.14:4667): protocol: read tcp 188.166.225.170:46984->77.90.31.14:4667: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:56.635280057 (263h58m51.066360198s) [noise=true , address=77.90.31.13:4640 , piece_id=NKLBPZJDAIAVAYH3QRDCPS6VWHJBFZZWY3HCTSW2E4II77YC3BYA, seg_pos={0 0}, upload=274428]: Done uploading piece. err: ecclient: upload failed (node:1XjJpLBEZm5daHcBUmUyC7QpLVhu9Vm9ex9RYcYJWEuLGk527F, address:77.90.31.13:4640): protocol: read tcp 188.166.225.170:48656->77.90.31.13:4640: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:56.635314653 (263h58m51.066394794s) [seg_pos={0 0}, upload=274428]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Aug 19 17:24:56.862515362 (263h58m51.293595493s) [upload=274428]: Done uploading segments... Aug 19 17:24:56.862536042 (263h58m51.293616163s) [upload=274428]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.656279759s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........X.....X............................................] duration: 6.111841236s FAIL (2025-08-19 17:56:30 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.112044865s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........X.....X...........................................] duration: 6.279199252s FAIL (2025-08-19 17:56:08 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.279220542s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [..............................X...........................X.] duration: 1.999230139s FAIL (2025-08-19 17:56:00 UTC) orbiter warning: operation: ListObjects ======ERROR====== unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/1K ======LOGS====== Aug 19 17:55:08.636689315 (264h29m3.067769446s) [upload=274973]: Starting upload Aug 19 17:55:08.636740844 (264h29m3.067820975s) [upload=274973]: Done starting upload Aug 19 17:55:08.636829314 (264h29m3.067909445s) [upload=274973]: Uploading segments... Aug 19 17:55:08.636904547 (264h29m3.067984678s) [upload=274973]: Got next segment. Inline: true Aug 19 17:55:08.636912002 (264h29m3.067992133s) [upload=274973]: Waiting for error group managing segments... Aug 19 17:55:08.636956797 (264h29m3.068036928s) [upload=274973]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 17:55:08.636961596 (264h29m3.068041727s) [upload=274973]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 19 17:55:08.636964062 (264h29m3.068044193s) [upload=274973]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 17:55:08.938231099 (264h29m3.36931123s) [upload=274973]: Done uploading segments... Aug 19 17:55:08.938277699 (264h29m3.36935785s) [upload=274973]: Upload finished. err: <nil> Duration:* 1.99925683s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...........X...X............................................] duration: 5.940638055s FAIL (2025-08-19 17:55:53 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 23:59.057943072 (263h57m53.489023273s) [noise=true , address=45.91.116.10:4083 , piece_id=V3H4T2WN2MBVALSIHZNOQHBE75WQRNYLIE3Z7DFWUUHBPYDT7IFA, seg_pos={0 0}, upload=274429]: Uploading piece... Aug 19 17:23:59.057955436 (263h57m53.489035557s) [noise=true , address=45.91.116.5:4298 , piece_id=HMF6DXO5KFMLK67WVA7BAWLQFLC2BCVPJ7ZT255FMRCCEI3MCBNA, seg_pos={0 0}, upload=274429]: Uploading piece... Aug 19 17:23:59.058557689 (263h57m53.48963782s) [noise=true , address=104.36.82.8:4969 , piece_id=3P655Z5OUVZ4VRESACEBKYR3KUFYY56NYPWX7ZVOHL3XI2YQ5TCQ, seg_pos={0 0}, upload=274429]: Uploading piece... Aug 19 17:23:59.059170771 (263h57m53.490250902s) [noise=true , address=45.91.116.7:4009 , piece_id=NSZYKL6FHDLDM463ZXKEQGX4ZY7JPJXENH63PPDV76XMGCBV7ZRQ, seg_pos={0 0}, upload=274429]: Uploading piece... Aug 19 17:23:59.059468281 (263h57m53.490548412s) [noise=true , address=77.90.31.15:4699 , piece_id=MFF7PNMDLHYEH7PY46MGGOJSY4XPHRDPGFWJABBO4JDTJJJJ3CTQ, seg_pos={0 0}, upload=274429]: Uploading piece... Aug 19 17:23:59.059498208 (263h57m53.490578339s) [noise=true , address=104.36.82.2:4729 , piece_id=BNX6UNYZ7VHQ4MKAH65DGSHULWRRXDSCMKOE7WFZO6T2V3ELBJ7Q, seg_pos={0 0}, upload=274429]: Uploading piece... Aug 19 17:23:59.059814182 (263h57m53.490894323s) [noise=true , address=104.36.82.6:4874 , piece_id=QMGNWSGMEV762Y2XEDK2LIAY3JMNLD3UOVFQ6HYP7ZMLLVAVO7ZQ, seg_pos={0 0}, upload=274429]: Uploading piece... Aug 19 17:23:59.059839941 (263h57m53.490920072s) [noise=true , address=45.91.116.12:4225 , piece_id=PCJYR2VLFH23UPQEABUYESR3MHKDHIVTJWDIWVYIL6JAFGGXXXFQ, seg_pos={0 0}, upload=274429]: Uploading piece... Aug 19 17:23:59.059875480 (263h57m53.490955611s) [noise=true , address=104.36.82.10:4079 , piece_id=S3BSVJWRPP2PDGLG7FKFROUKAM3GTBDL2ALAJPAZQQQZJVRMMZ3Q, seg_pos={0 0}, upload=274429]: Uploading piece... Aug 19 17:23:59.060109147 (263h57m53.491189278s) [noise=true , address=104.36.82.2:4732 , piece_id=NLTK4572JAQHZQRDD2DOOIW7VBO7NK4DI4RUZRSXQ3GR72EWV3KQ, seg_pos={0 0}, upload=274429]: Uploading piece... Aug 19 17:23:59.060142190 (263h57m53.491222321s) [noise=true , address=45.91.116.9:4261 , piece_id=J5IMFL3JW4YKDLI6OEY2POEB4LJR2P3BIFAD4YIF3E4JTUXIXY3A, seg_pos={0 0}, upload=274429]: Uploading piece... Aug 19 17:23:59.060348274 (263h57m53.491428405s) [noise=true , address=104.36.82.15:4259 , piece_id=RM7JL7JKCMFM4CNXVVZQ2QJTOEPQEQHGLW3GYXWUD6LAQ2ZJ3P5Q, seg_pos={0 0}, upload=274429]: Uploading piece... Aug 19 17:23:59.060544830 (263h57m53.491624961s) [noise=true , address=45.91.116.14:4191 , piece_id=PJ33GP36S76VCI5RVR26PBFZ6HZUUZYVWNGPJITRLGDZI6LWCJIQ, seg_pos={0 0}, upload=274429]: Uploading piece... Aug 19 17:23:59.060752227 (263h57m53.491832358s) [noise=true , address=45.91.116.11:4118 , piece_id=VD2HK2QU63UVJFRASQOIRFZRWZN2HAUSAZJL2VZKYALGSV67SP2A, seg_pos={0 0}, upload=274429]: Uploading piece... Aug 19 17:23:59.060967990 (263h57m53.492048121s) [noise=true , address=77.90.31.11:4555 , piece_id=GTU6PGCHI4WOFGK2B3FP2Y22VUDBWLQKKEMZBT6I4F47OKK4RPNQ, seg_pos={0 0}, upload=274429]: Uploading piece... Aug 19 17:23:59.061294044 (263h57m53.492374175s) [noise=true , address=45.91.116.8:4048 , piece_id=M2CBILQKSVSQ7VY4AWQQM5GHF2KYBSXH73YJGAEJZAHOR7FOBWEQ, seg_pos={0 0}, upload=274429]: Uploading piece... Aug 19 17:24:00.872920400 (263h57m55.304000531s) [upload=274429]: Next returned nil segment Aug 19 17:24:00.872945548 (263h57m55.304025679s) [upload=274429]: Waiting for error group managing segments... Aug 19 17:24:01.156214888 (263h57m55.587295019s) [noise=true , address=104.36.82.6:4874 , piece_id=QMGNWSGMEV762Y2XEDK2LIAY3JMNLD3UOVFQ6HYP7ZMLLVAVO7ZQ, seg_pos={0 0}, upload=274429]: Done uploading piece. err: <nil> Aug 19 17:24:01.181164926 (263h57m55.612245057s) [noise=true , address=104.36.82.2:4732 , piece_id=NLTK4572JAQHZQRDD2DOOIW7VBO7NK4DI4RUZRSXQ3GR72EWV3KQ, seg_pos={0 0}, upload=274429]: Done uploading piece. err: <nil> Aug 19 17:24:01.222047199 (263h57m55.653127331s) [noise=true , address=104.36.82.2:4729 , piece_id=BNX6UNYZ7VHQ4MKAH65DGSHULWRRXDSCMKOE7WFZO6T2V3ELBJ7Q, seg_pos={0 0}, upload=274429]: Done uploading piece. err: <nil> Aug 19 17:24:01.250473409 (263h57m55.68155354s) [noise=true , address=104.36.82.15:4259 , piece_id=RM7JL7JKCMFM4CNXVVZQ2QJTOEPQEQHGLW3GYXWUD6LAQ2ZJ3P5Q, seg_pos={0 0}, upload=274429]: Done uploading piece. err: <nil> Aug 19 17:24:01.259222299 (263h57m55.69030242s) [noise=true , address=104.36.82.8:4969 , piece_id=3P655Z5OUVZ4VRESACEBKYR3KUFYY56NYPWX7ZVOHL3XI2YQ5TCQ, seg_pos={0 0}, upload=274429]: Done uploading piece. err: <nil> Aug 19 17:24:01.261735818 (263h57m55.692815949s) [noise=true , address=104.36.82.10:4079 , piece_id=S3BSVJWRPP2PDGLG7FKFROUKAM3GTBDL2ALAJPAZQQQZJVRMMZ3Q, seg_pos={0 0}, upload=274429]: Done uploading piece. err: <nil> Aug 19 17:24:28.183964515 (263h58m22.615044646s) [noise=true , address=45.91.116.14:4191 , piece_id=PJ33GP36S76VCI5RVR26PBFZ6HZUUZYVWNGPJITRLGDZI6LWCJIQ, seg_pos={0 0}, upload=274429]: Done uploading piece. err: ecclient: upload failed (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, address:45.91.116.14:4191): protocol: read tcp 188.166.225.170:37646->45.91.116.14:4191: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:28.184013127 (263h58m22.615093258s) [noise=true , address=45.91.116.9:4261 , piece_id=J5IMFL3JW4YKDLI6OEY2POEB4LJR2P3BIFAD4YIF3E4JTUXIXY3A, seg_pos={0 0}, upload=274429]: Done uploading piece. err: ecclient: upload failed (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, address:45.91.116.9:4261): protocol: rpc: dial tcp 45.91.116.9:4261: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:28.184026694 (263h58m22.615106825s) [noise=true , address=45.91.116.8:4048 , piece_id=M2CBILQKSVSQ7VY4AWQQM5GHF2KYBSXH73YJGAEJZAHOR7FOBWEQ, seg_pos={0 0}, upload=274429]: Done uploading piece. err: ecclient: upload failed (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, address:45.91.116.8:4048): protocol: read tcp 188.166.225.170:45604->45.91.116.8:4048: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:28.184070568 (263h58m22.615150689s) [noise=true , address=45.91.116.5:4298 , piece_id=HMF6DXO5KFMLK67WVA7BAWLQFLC2BCVPJ7ZT255FMRCCEI3MCBNA, seg_pos={0 0}, upload=274429]: Done uploading piece. err: ecclient: upload failed (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, address:45.91.116.5:4298): protocol: read tcp 188.166.225.170:46970->45.91.116.5:4298: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:28.184103010 (263h58m22.615183141s) [noise=true , address=45.91.116.10:4083 , piece_id=V3H4T2WN2MBVALSIHZNOQHBE75WQRNYLIE3Z7DFWUUHBPYDT7IFA, seg_pos={0 0}, upload=274429]: Done uploading piece. err: ecclient: upload failed (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, address:45.91.116.10:4083): protocol: read tcp 188.166.225.170:41632->45.91.116.10:4083: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:28.184139489 (263h58m22.61521962s) [noise=true , address=45.91.116.7:4009 , piece_id=NSZYKL6FHDLDM463ZXKEQGX4ZY7JPJXENH63PPDV76XMGCBV7ZRQ, seg_pos={0 0}, upload=274429]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: read tcp 188.166.225.170:45756->45.91.116.7:4009: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:28.184171009 (263h58m22.615251141s) [noise=true , address=77.90.31.11:4555 , piece_id=GTU6PGCHI4WOFGK2B3FP2Y22VUDBWLQKKEMZBT6I4F47OKK4RPNQ, seg_pos={0 0}, upload=274429]: Done uploading piece. err: ecclient: upload failed (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, address:77.90.31.11:4555): protocol: read tcp 188.166.225.170:37438->77.90.31.11:4555: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:28.184197480 (263h58m22.615277611s) [noise=true , address=77.90.31.15:4699 , piece_id=MFF7PNMDLHYEH7PY46MGGOJSY4XPHRDPGFWJABBO4JDTJJJJ3CTQ, seg_pos={0 0}, upload=274429]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: read tcp 188.166.225.170:36788->77.90.31.15:4699: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:28.184218410 (263h58m22.615298541s) [noise=true , address=45.91.116.12:4225 , piece_id=PCJYR2VLFH23UPQEABUYESR3MHKDHIVTJWDIWVYIL6JAFGGXXXFQ, seg_pos={0 0}, upload=274429]: Done uploading piece. err: ecclient: upload failed (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, address:45.91.116.12:4225): protocol: read tcp 188.166.225.170:60636->45.91.116.12:4225: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:28.184249710 (263h58m22.615329841s) [noise=true , address=45.91.116.11:4118 , piece_id=VD2HK2QU63UVJFRASQOIRFZRWZN2HAUSAZJL2VZKYALGSV67SP2A, seg_pos={0 0}, upload=274429]: Done uploading piece. err: ecclient: upload failed (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, address:45.91.116.11:4118): protocol: read tcp 188.166.225.170:48006->45.91.116.11:4118: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:28.184274337 (263h58m22.615354468s) [seg_pos={0 0}, upload=274429]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Aug 19 17:24:28.412483516 (263h58m22.843563647s) [upload=274429]: Done uploading segments... Aug 19 17:24:28.412520628 (263h58m22.843600759s) [upload=274429]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 5.940671248s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 6 out of 60 is failed (threshold=3) [.........X......X.X........X....XX..........................] duration: 5.939305804s FAIL (2025-08-19 17:55:52 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/4M") ======LOGS====== Aug 19 17:38:09.246414094 (264h12m3.677494255s) [upload=274674]: Starting upload Aug 19 17:38:09.246486392 (264h12m3.677566533s) [upload=274674]: Done starting upload Aug 19 17:38:09.246601452 (264h12m3.677681583s) [upload=274674]: Uploading segments... Aug 19 17:38:09.246691885 (264h12m3.677772016s) [upload=274674]: Got next segment. Inline: false Aug 19 17:38:09.246724627 (264h12m3.677804758s) [upload=274674]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 17:38:09.246730909 (264h12m3.67781104s) [upload=274674]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 19 17:38:09.693242556 (264h12m4.124322687s) [seg_pos={0 0}, upload=274674]: Begin upload segment... Aug 19 17:38:09.693302230 (264h12m4.124382361s) [seg_pos={0 0}, upload=274674]: Done begin upload segment. Aug 19 17:38:09.693343529 (264h12m4.12442367s) [noise=true , address=77.90.31.18:4825 , piece_id=UQLJVRVUBBN5LFBIAQL7YQ4IRAWI3SBVBHUT4IIFBROAUHXHVZ3A, seg_pos={0 0}, upload=274674]: Uploading piece... Aug 19 17:38:09.693350092 (264h12m4.124430213s) [noise=true , address=45.91.116.14:4222 , piece_id=DXHEK43V7USJLYKPF5PKEHLJPVP4OPH2H3TQVGUBMK5MVRWQHXJQ, seg_pos={0 0}, upload=274674]: Uploading piece... Aug 19 17:38:09.693395940 (264h12m4.124476081s) [noise=true , address=45.91.116.11:4140 , piece_id=KQNAZLV4XPDYYLJQUMI2UL3GVLA7QTEZ6TU3HDYFYB7LPRBSRQFA, seg_pos={0 0}, upload=274674]: Uploading piece... Aug 19 17:38:09.693960980 (264h12m4.125041131s) [noise=true , address=79.127.201.215:4343 , piece_id=MIATMAVPWKBJPVYH3XDYXQGKW7VWKRULLP6UV2ZNKJZ7YDVJJEBQ, seg_pos={0 0}, upload=274674]: Uploading piece... Aug 19 17:38:09.694358652 (264h12m4.125438783s) [noise=true , address=104.36.82.5:4852 , piece_id=UALXJPWV34HHWPC6GNCOFNP2RDC3KF6ZY4FUHLL3MWR2UZDCXNCQ, seg_pos={0 0}, upload=274674]: Uploading piece... Aug 19 17:38:09.694689475 (264h12m4.125769606s) [noise=true , address=104.36.82.58:4269 , piece_id=CTQSY5PQFUNXUJR2JXSI7O2EHMYTOAHUSHGTMVTI42E45RYFIGJA, seg_pos={0 0}, upload=274674]: Uploading piece... Aug 19 17:38:09.695011601 (264h12m4.126091732s) [noise=true , address=104.36.82.60:4364 , piece_id=A5BGK2CC62VYRH27SR2MONG2GPSZ5WONCHAF42LFPXOTM2W7XIJA, seg_pos={0 0}, upload=274674]: Uploading piece... Aug 19 17:38:09.695441633 (264h12m4.126521764s) [noise=true , address=104.36.82.9:4028 , piece_id=4LUYCAISMQC3SGDLJZSDFDJET4R54U3N4B6ZAJ35UDXB5CT3GV2Q, seg_pos={0 0}, upload=274674]: Uploading piece... Aug 19 17:38:09.695845597 (264h12m4.126925728s) [noise=true , address=77.90.31.15:4701 , piece_id=CCTT764Z3HFZFT4PE7YOOBVVGDASLFYVFGVXGWADBLDLURBOKGGQ, seg_pos={0 0}, upload=274674]: Uploading piece... Aug 19 17:38:09.696191328 (264h12m4.127271459s) [noise=true , address=45.91.116.5:4302 , piece_id=D2ZYYIFLA7GFPEII4X75HVJGL5DOM6BZLQFXL7HYVXYLNCL4CDQQ, seg_pos={0 0}, upload=274674]: Uploading piece... Aug 19 17:38:09.696539624 (264h12m4.127619755s) [noise=true , address=77.90.31.14:4689 , piece_id=6FO4WN5BXNW6UFGW2ALS5YBYBP5FZCAZBLQZXAULFCN65NHH7N6A, seg_pos={0 0}, upload=274674]: Uploading piece... Aug 19 17:38:09.696809421 (264h12m4.127889552s) [noise=true , address=79.127.201.214:4299 , piece_id=C4JSI4OPEZPPHADT6QTVJGEXQNVSHEK42KNYQBV3S2OH7Z5Z7HHA, seg_pos={0 0}, upload=274674]: Uploading piece... Aug 19 17:38:09.697172445 (264h12m4.128252576s) [noise=true , address=104.36.82.43:4855 , piece_id=3H2PUTGFVC2PR224NBHGRB7GLWBOL2MNXXWJ7JF23HYVF53TEWFQ, seg_pos={0 0}, upload=274674]: Uploading piece... Aug 19 17:38:09.697485064 (264h12m4.128565205s) [noise=true , address=77.90.31.12:4619 , piece_id=XMWKT6JBVLOPTRUIPNEK4GXVCH5OOVO5WYWDGWMXGA2VVVD74CBA, seg_pos={0 0}, upload=274674]: Uploading piece... Aug 19 17:38:09.697774858 (264h12m4.128854989s) [noise=true , address=104.36.82.38:4714 , piece_id=SZ42LIQE4LXTBCZY4BTXWS3EL27UYIQS3AAXGWNLH65TUZWNUYSQ, seg_pos={0 0}, upload=274674]: Uploading piece... Aug 19 17:38:09.698096103 (264h12m4.129176234s) [noise=true , address=104.36.82.4:4817 , piece_id=JNKPBDLUZUWI22ZLGTCD3BZJBAUX7TZVAGXJVFZOEJPK2Z3PMADA, seg_pos={0 0}, upload=274674]: Uploading piece... Aug 19 17:38:09.698352945 (264h12m4.129433076s) [noise=true , address=45.91.116.13:4183 , piece_id=OMUE6JFSTRIF64VYM5XCHNL7U26SEXP3IGWDBHP5G2E6YJQETEUQ, seg_pos={0 0}, upload=274674]: Uploading piece... Aug 19 17:38:09.698876106 (264h12m4.129956227s) [noise=true , address=104.36.82.6:4879 , piece_id=JKK5HMPYTMR7PNWHVPBDHPH3K6H7VM7AA4K4X4WXQ5LN57VRBUFA, seg_pos={0 0}, upload=274674]: Uploading piece... Aug 19 17:38:11.192631202 (264h12m5.623711333s) [upload=274674]: Next returned nil segment Aug 19 17:38:11.192648606 (264h12m5.623728737s) [upload=274674]: Waiting for error group managing segments... Aug 19 17:38:11.591537039 (264h12m6.02261717s) [noise=true , address=45.91.116.14:4222 , piece_id=DXHEK43V7USJLYKPF5PKEHLJPVP4OPH2H3TQVGUBMK5MVRWQHXJQ, seg_pos={0 0}, upload=274674]: Done uploading piece. err: <nil> Aug 19 17:38:11.606093581 (264h12m6.037173712s) [noise=true , address=77.90.31.14:4689 , piece_id=6FO4WN5BXNW6UFGW2ALS5YBYBP5FZCAZBLQZXAULFCN65NHH7N6A, seg_pos={0 0}, upload=274674]: Done uploading piece. err: <nil> Aug 19 17:38:11.610339916 (264h12m6.041420047s) [noise=true , address=77.90.31.15:4701 , piece_id=CCTT764Z3HFZFT4PE7YOOBVVGDASLFYVFGVXGWADBLDLURBOKGGQ, seg_pos={0 0}, upload=274674]: Done uploading piece. err: <nil> Aug 19 17:38:11.630055040 (264h12m6.061135171s) [noise=true , address=79.127.201.215:4343 , piece_id=MIATMAVPWKBJPVYH3XDYXQGKW7VWKRULLP6UV2ZNKJZ7YDVJJEBQ, seg_pos={0 0}, upload=274674]: Done uploading piece. err: <nil> Aug 19 17:38:11.678703675 (264h12m6.109783797s) [noise=true , address=104.36.82.9:4028 , piece_id=4LUYCAISMQC3SGDLJZSDFDJET4R54U3N4B6ZAJ35UDXB5CT3GV2Q, seg_pos={0 0}, upload=274674]: Done uploading piece. err: <nil> Aug 19 17:38:11.693543009 (264h12m6.12462314s) [noise=true , address=104.36.82.60:4364 , piece_id=A5BGK2CC62VYRH27SR2MONG2GPSZ5WONCHAF42LFPXOTM2W7XIJA, seg_pos={0 0}, upload=274674]: Done uploading piece. err: <nil> Aug 19 17:38:11.693568448 (264h12m6.124648579s) [noise=true , address=45.91.116.13:4183 , piece_id=OMUE6JFSTRIF64VYM5XCHNL7U26SEXP3IGWDBHP5G2E6YJQETEUQ, seg_pos={0 0}, upload=274674]: Done uploading piece. err: <nil> Aug 19 17:38:11.698831207 (264h12m6.129911329s) [noise=true , address=104.36.82.5:4852 , piece_id=UALXJPWV34HHWPC6GNCOFNP2RDC3KF6ZY4FUHLL3MWR2UZDCXNCQ, seg_pos={0 0}, upload=274674]: Done uploading piece. err: <nil> Aug 19 17:38:11.698886834 (264h12m6.129966955s) [noise=true , address=77.90.31.12:4619 , piece_id=XMWKT6JBVLOPTRUIPNEK4GXVCH5OOVO5WYWDGWMXGA2VVVD74CBA, seg_pos={0 0}, upload=274674]: Done uploading piece. err: <nil> Aug 19 17:38:11.714704781 (264h12m6.145784912s) [noise=true , address=79.127.201.214:4299 , piece_id=C4JSI4OPEZPPHADT6QTVJGEXQNVSHEK42KNYQBV3S2OH7Z5Z7HHA, seg_pos={0 0}, upload=274674]: Done uploading piece. err: <nil> Aug 19 17:38:11.720319795 (264h12m6.151399956s) [noise=true , address=45.91.116.11:4140 , piece_id=KQNAZLV4XPDYYLJQUMI2UL3GVLA7QTEZ6TU3HDYFYB7LPRBSRQFA, seg_pos={0 0}, upload=274674]: Done uploading piece. err: <nil> Aug 19 17:38:11.722063401 (264h12m6.153143532s) [noise=true , address=104.36.82.6:4879 , piece_id=JKK5HMPYTMR7PNWHVPBDHPH3K6H7VM7AA4K4X4WXQ5LN57VRBUFA, seg_pos={0 0}, upload=274674]: Done uploading piece. err: <nil> Aug 19 17:38:11.726762272 (264h12m6.157842393s) [noise=true , address=104.36.82.38:4714 , piece_id=SZ42LIQE4LXTBCZY4BTXWS3EL27UYIQS3AAXGWNLH65TUZWNUYSQ, seg_pos={0 0}, upload=274674]: Done uploading piece. err: <nil> Aug 19 17:38:11.734258877 (264h12m6.165339008s) [noise=true , address=104.36.82.58:4269 , piece_id=CTQSY5PQFUNXUJR2JXSI7O2EHMYTOAHUSHGTMVTI42E45RYFIGJA, seg_pos={0 0}, upload=274674]: Done uploading piece. err: <nil> Aug 19 17:38:11.734284346 (264h12m6.165364477s) [seg_pos={0 0}, upload=274674]: Segment reached optimal threshold of 14 pieces. Aug 19 17:38:11.734416277 (264h12m6.165496408s) [noise=true , address=104.36.82.4:4817 , piece_id=JNKPBDLUZUWI22ZLGTCD3BZJBAUX7TZVAGXJVFZOEJPK2Z3PMADA, seg_pos={0 0}, upload=274674]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RSBodzJxMye65QgkyrzkWCKQ49cnfdu4haUZKW23nWpqzMcYz): protocol: expected piece hash; context canceled; EOF Aug 19 17:38:11.734486873 (264h12m6.165567004s) [noise=true , address=104.36.82.43:4855 , piece_id=3H2PUTGFVC2PR224NBHGRB7GLWBOL2MNXXWJ7JF23HYVF53TEWFQ, seg_pos={0 0}, upload=274674]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TGTRhEi2KsbrkC6ZmgGkMaJri9uhzEcmXe1J4nJjpEd1BE4ka): protocol: expected piece hash; context canceled; EOF Aug 19 17:38:11.734587926 (264h12m6.165668057s) [noise=true , address=77.90.31.18:4825 , piece_id=UQLJVRVUBBN5LFBIAQL7YQ4IRAWI3SBVBHUT4IIFBROAUHXHVZ3A, seg_pos={0 0}, upload=274674]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PqKyfFxfVKTGARbPqBcQL3Naqjdf5uR44PeLi3sJpu57dcm1x): protocol: expected piece hash; context canceled; EOF Aug 19 17:38:11.734592956 (264h12m6.165673087s) [noise=true , address=45.91.116.5:4302 , piece_id=D2ZYYIFLA7GFPEII4X75HVJGL5DOM6BZLQFXL7HYVXYLNCL4CDQQ, seg_pos={0 0}, upload=274674]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pJWhSLza8WaMuvvDwH4nae57UxQmfCdDvAkQ3v25MgPYHoZXx): protocol: expected piece hash; context canceled; EOF Aug 19 17:38:11.734645206 (264h12m6.165725337s) [seg_pos={0 0}, upload=274674]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 19 17:38:11.734756098 (264h12m6.165836239s) [upload=274674]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 19 17:38:11.734764033 (264h12m6.165844154s) [upload=274674]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 17:38:12.023665554 (264h12m6.454745685s) [upload=274674]: Done uploading segments... Aug 19 17:38:12.023686044 (264h12m6.454766195s) [upload=274674]: Upload finished. err: <nil> Duration: 5.939340492s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [............X...X...........................................] duration: 5.79794411s FAIL (2025-08-19 17:55:31 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 23:59.057943072 (263h57m53.489023273s) [noise=true , address=45.91.116.10:4083 , piece_id=V3H4T2WN2MBVALSIHZNOQHBE75WQRNYLIE3Z7DFWUUHBPYDT7IFA, seg_pos={0 0}, upload=274429]: Uploading piece... Aug 19 17:23:59.057955436 (263h57m53.489035557s) [noise=true , address=45.91.116.5:4298 , piece_id=HMF6DXO5KFMLK67WVA7BAWLQFLC2BCVPJ7ZT255FMRCCEI3MCBNA, seg_pos={0 0}, upload=274429]: Uploading piece... Aug 19 17:23:59.058557689 (263h57m53.48963782s) [noise=true , address=104.36.82.8:4969 , piece_id=3P655Z5OUVZ4VRESACEBKYR3KUFYY56NYPWX7ZVOHL3XI2YQ5TCQ, seg_pos={0 0}, upload=274429]: Uploading piece... Aug 19 17:23:59.059170771 (263h57m53.490250902s) [noise=true , address=45.91.116.7:4009 , piece_id=NSZYKL6FHDLDM463ZXKEQGX4ZY7JPJXENH63PPDV76XMGCBV7ZRQ, seg_pos={0 0}, upload=274429]: Uploading piece... Aug 19 17:23:59.059468281 (263h57m53.490548412s) [noise=true , address=77.90.31.15:4699 , piece_id=MFF7PNMDLHYEH7PY46MGGOJSY4XPHRDPGFWJABBO4JDTJJJJ3CTQ, seg_pos={0 0}, upload=274429]: Uploading piece... Aug 19 17:23:59.059498208 (263h57m53.490578339s) [noise=true , address=104.36.82.2:4729 , piece_id=BNX6UNYZ7VHQ4MKAH65DGSHULWRRXDSCMKOE7WFZO6T2V3ELBJ7Q, seg_pos={0 0}, upload=274429]: Uploading piece... Aug 19 17:23:59.059814182 (263h57m53.490894323s) [noise=true , address=104.36.82.6:4874 , piece_id=QMGNWSGMEV762Y2XEDK2LIAY3JMNLD3UOVFQ6HYP7ZMLLVAVO7ZQ, seg_pos={0 0}, upload=274429]: Uploading piece... Aug 19 17:23:59.059839941 (263h57m53.490920072s) [noise=true , address=45.91.116.12:4225 , piece_id=PCJYR2VLFH23UPQEABUYESR3MHKDHIVTJWDIWVYIL6JAFGGXXXFQ, seg_pos={0 0}, upload=274429]: Uploading piece... Aug 19 17:23:59.059875480 (263h57m53.490955611s) [noise=true , address=104.36.82.10:4079 , piece_id=S3BSVJWRPP2PDGLG7FKFROUKAM3GTBDL2ALAJPAZQQQZJVRMMZ3Q, seg_pos={0 0}, upload=274429]: Uploading piece... Aug 19 17:23:59.060109147 (263h57m53.491189278s) [noise=true , address=104.36.82.2:4732 , piece_id=NLTK4572JAQHZQRDD2DOOIW7VBO7NK4DI4RUZRSXQ3GR72EWV3KQ, seg_pos={0 0}, upload=274429]: Uploading piece... Aug 19 17:23:59.060142190 (263h57m53.491222321s) [noise=true , address=45.91.116.9:4261 , piece_id=J5IMFL3JW4YKDLI6OEY2POEB4LJR2P3BIFAD4YIF3E4JTUXIXY3A, seg_pos={0 0}, upload=274429]: Uploading piece... Aug 19 17:23:59.060348274 (263h57m53.491428405s) [noise=true , address=104.36.82.15:4259 , piece_id=RM7JL7JKCMFM4CNXVVZQ2QJTOEPQEQHGLW3GYXWUD6LAQ2ZJ3P5Q, seg_pos={0 0}, upload=274429]: Uploading piece... Aug 19 17:23:59.060544830 (263h57m53.491624961s) [noise=true , address=45.91.116.14:4191 , piece_id=PJ33GP36S76VCI5RVR26PBFZ6HZUUZYVWNGPJITRLGDZI6LWCJIQ, seg_pos={0 0}, upload=274429]: Uploading piece... Aug 19 17:23:59.060752227 (263h57m53.491832358s) [noise=true , address=45.91.116.11:4118 , piece_id=VD2HK2QU63UVJFRASQOIRFZRWZN2HAUSAZJL2VZKYALGSV67SP2A, seg_pos={0 0}, upload=274429]: Uploading piece... Aug 19 17:23:59.060967990 (263h57m53.492048121s) [noise=true , address=77.90.31.11:4555 , piece_id=GTU6PGCHI4WOFGK2B3FP2Y22VUDBWLQKKEMZBT6I4F47OKK4RPNQ, seg_pos={0 0}, upload=274429]: Uploading piece... Aug 19 17:23:59.061294044 (263h57m53.492374175s) [noise=true , address=45.91.116.8:4048 , piece_id=M2CBILQKSVSQ7VY4AWQQM5GHF2KYBSXH73YJGAEJZAHOR7FOBWEQ, seg_pos={0 0}, upload=274429]: Uploading piece... Aug 19 17:24:00.872920400 (263h57m55.304000531s) [upload=274429]: Next returned nil segment Aug 19 17:24:00.872945548 (263h57m55.304025679s) [upload=274429]: Waiting for error group managing segments... Aug 19 17:24:01.156214888 (263h57m55.587295019s) [noise=true , address=104.36.82.6:4874 , piece_id=QMGNWSGMEV762Y2XEDK2LIAY3JMNLD3UOVFQ6HYP7ZMLLVAVO7ZQ, seg_pos={0 0}, upload=274429]: Done uploading piece. err: <nil> Aug 19 17:24:01.181164926 (263h57m55.612245057s) [noise=true , address=104.36.82.2:4732 , piece_id=NLTK4572JAQHZQRDD2DOOIW7VBO7NK4DI4RUZRSXQ3GR72EWV3KQ, seg_pos={0 0}, upload=274429]: Done uploading piece. err: <nil> Aug 19 17:24:01.222047199 (263h57m55.653127331s) [noise=true , address=104.36.82.2:4729 , piece_id=BNX6UNYZ7VHQ4MKAH65DGSHULWRRXDSCMKOE7WFZO6T2V3ELBJ7Q, seg_pos={0 0}, upload=274429]: Done uploading piece. err: <nil> Aug 19 17:24:01.250473409 (263h57m55.68155354s) [noise=true , address=104.36.82.15:4259 , piece_id=RM7JL7JKCMFM4CNXVVZQ2QJTOEPQEQHGLW3GYXWUD6LAQ2ZJ3P5Q, seg_pos={0 0}, upload=274429]: Done uploading piece. err: <nil> Aug 19 17:24:01.259222299 (263h57m55.69030242s) [noise=true , address=104.36.82.8:4969 , piece_id=3P655Z5OUVZ4VRESACEBKYR3KUFYY56NYPWX7ZVOHL3XI2YQ5TCQ, seg_pos={0 0}, upload=274429]: Done uploading piece. err: <nil> Aug 19 17:24:01.261735818 (263h57m55.692815949s) [noise=true , address=104.36.82.10:4079 , piece_id=S3BSVJWRPP2PDGLG7FKFROUKAM3GTBDL2ALAJPAZQQQZJVRMMZ3Q, seg_pos={0 0}, upload=274429]: Done uploading piece. err: <nil> Aug 19 17:24:28.183964515 (263h58m22.615044646s) [noise=true , address=45.91.116.14:4191 , piece_id=PJ33GP36S76VCI5RVR26PBFZ6HZUUZYVWNGPJITRLGDZI6LWCJIQ, seg_pos={0 0}, upload=274429]: Done uploading piece. err: ecclient: upload failed (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, address:45.91.116.14:4191): protocol: read tcp 188.166.225.170:37646->45.91.116.14:4191: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:28.184013127 (263h58m22.615093258s) [noise=true , address=45.91.116.9:4261 , piece_id=J5IMFL3JW4YKDLI6OEY2POEB4LJR2P3BIFAD4YIF3E4JTUXIXY3A, seg_pos={0 0}, upload=274429]: Done uploading piece. err: ecclient: upload failed (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, address:45.91.116.9:4261): protocol: rpc: dial tcp 45.91.116.9:4261: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:28.184026694 (263h58m22.615106825s) [noise=true , address=45.91.116.8:4048 , piece_id=M2CBILQKSVSQ7VY4AWQQM5GHF2KYBSXH73YJGAEJZAHOR7FOBWEQ, seg_pos={0 0}, upload=274429]: Done uploading piece. err: ecclient: upload failed (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, address:45.91.116.8:4048): protocol: read tcp 188.166.225.170:45604->45.91.116.8:4048: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:28.184070568 (263h58m22.615150689s) [noise=true , address=45.91.116.5:4298 , piece_id=HMF6DXO5KFMLK67WVA7BAWLQFLC2BCVPJ7ZT255FMRCCEI3MCBNA, seg_pos={0 0}, upload=274429]: Done uploading piece. err: ecclient: upload failed (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, address:45.91.116.5:4298): protocol: read tcp 188.166.225.170:46970->45.91.116.5:4298: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:28.184103010 (263h58m22.615183141s) [noise=true , address=45.91.116.10:4083 , piece_id=V3H4T2WN2MBVALSIHZNOQHBE75WQRNYLIE3Z7DFWUUHBPYDT7IFA, seg_pos={0 0}, upload=274429]: Done uploading piece. err: ecclient: upload failed (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, address:45.91.116.10:4083): protocol: read tcp 188.166.225.170:41632->45.91.116.10:4083: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:28.184139489 (263h58m22.61521962s) [noise=true , address=45.91.116.7:4009 , piece_id=NSZYKL6FHDLDM463ZXKEQGX4ZY7JPJXENH63PPDV76XMGCBV7ZRQ, seg_pos={0 0}, upload=274429]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: read tcp 188.166.225.170:45756->45.91.116.7:4009: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:28.184171009 (263h58m22.615251141s) [noise=true , address=77.90.31.11:4555 , piece_id=GTU6PGCHI4WOFGK2B3FP2Y22VUDBWLQKKEMZBT6I4F47OKK4RPNQ, seg_pos={0 0}, upload=274429]: Done uploading piece. err: ecclient: upload failed (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, address:77.90.31.11:4555): protocol: read tcp 188.166.225.170:37438->77.90.31.11:4555: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:28.184197480 (263h58m22.615277611s) [noise=true , address=77.90.31.15:4699 , piece_id=MFF7PNMDLHYEH7PY46MGGOJSY4XPHRDPGFWJABBO4JDTJJJJ3CTQ, seg_pos={0 0}, upload=274429]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: read tcp 188.166.225.170:36788->77.90.31.15:4699: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:28.184218410 (263h58m22.615298541s) [noise=true , address=45.91.116.12:4225 , piece_id=PCJYR2VLFH23UPQEABUYESR3MHKDHIVTJWDIWVYIL6JAFGGXXXFQ, seg_pos={0 0}, upload=274429]: Done uploading piece. err: ecclient: upload failed (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, address:45.91.116.12:4225): protocol: read tcp 188.166.225.170:60636->45.91.116.12:4225: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:28.184249710 (263h58m22.615329841s) [noise=true , address=45.91.116.11:4118 , piece_id=VD2HK2QU63UVJFRASQOIRFZRWZN2HAUSAZJL2VZKYALGSV67SP2A, seg_pos={0 0}, upload=274429]: Done uploading piece. err: ecclient: upload failed (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, address:45.91.116.11:4118): protocol: read tcp 188.166.225.170:48006->45.91.116.11:4118: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:28.184274337 (263h58m22.615354468s) [seg_pos={0 0}, upload=274429]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Aug 19 17:24:28.412483516 (263h58m22.843563647s) [upload=274429]: Done uploading segments... Aug 19 17:24:28.412520628 (263h58m22.843600759s) [upload=274429]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 5.79796934s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 6 out of 60 is failed (threshold=3) [..........X......X.X........X....XX.........................] duration: 5.54669006s FAIL (2025-08-19 17:55:31 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/4M") ======LOGS====== Aug 19 17:38:09.246414094 (264h12m3.677494255s) [upload=274674]: Starting upload Aug 19 17:38:09.246486392 (264h12m3.677566533s) [upload=274674]: Done starting upload Aug 19 17:38:09.246601452 (264h12m3.677681583s) [upload=274674]: Uploading segments... Aug 19 17:38:09.246691885 (264h12m3.677772016s) [upload=274674]: Got next segment. Inline: false Aug 19 17:38:09.246724627 (264h12m3.677804758s) [upload=274674]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 17:38:09.246730909 (264h12m3.67781104s) [upload=274674]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 19 17:38:09.693242556 (264h12m4.124322687s) [seg_pos={0 0}, upload=274674]: Begin upload segment... Aug 19 17:38:09.693302230 (264h12m4.124382361s) [seg_pos={0 0}, upload=274674]: Done begin upload segment. Aug 19 17:38:09.693343529 (264h12m4.12442367s) [noise=true , address=77.90.31.18:4825 , piece_id=UQLJVRVUBBN5LFBIAQL7YQ4IRAWI3SBVBHUT4IIFBROAUHXHVZ3A, seg_pos={0 0}, upload=274674]: Uploading piece... Aug 19 17:38:09.693350092 (264h12m4.124430213s) [noise=true , address=45.91.116.14:4222 , piece_id=DXHEK43V7USJLYKPF5PKEHLJPVP4OPH2H3TQVGUBMK5MVRWQHXJQ, seg_pos={0 0}, upload=274674]: Uploading piece... Aug 19 17:38:09.693395940 (264h12m4.124476081s) [noise=true , address=45.91.116.11:4140 , piece_id=KQNAZLV4XPDYYLJQUMI2UL3GVLA7QTEZ6TU3HDYFYB7LPRBSRQFA, seg_pos={0 0}, upload=274674]: Uploading piece... Aug 19 17:38:09.693960980 (264h12m4.125041131s) [noise=true , address=79.127.201.215:4343 , piece_id=MIATMAVPWKBJPVYH3XDYXQGKW7VWKRULLP6UV2ZNKJZ7YDVJJEBQ, seg_pos={0 0}, upload=274674]: Uploading piece... Aug 19 17:38:09.694358652 (264h12m4.125438783s) [noise=true , address=104.36.82.5:4852 , piece_id=UALXJPWV34HHWPC6GNCOFNP2RDC3KF6ZY4FUHLL3MWR2UZDCXNCQ, seg_pos={0 0}, upload=274674]: Uploading piece... Aug 19 17:38:09.694689475 (264h12m4.125769606s) [noise=true , address=104.36.82.58:4269 , piece_id=CTQSY5PQFUNXUJR2JXSI7O2EHMYTOAHUSHGTMVTI42E45RYFIGJA, seg_pos={0 0}, upload=274674]: Uploading piece... Aug 19 17:38:09.695011601 (264h12m4.126091732s) [noise=true , address=104.36.82.60:4364 , piece_id=A5BGK2CC62VYRH27SR2MONG2GPSZ5WONCHAF42LFPXOTM2W7XIJA, seg_pos={0 0}, upload=274674]: Uploading piece... Aug 19 17:38:09.695441633 (264h12m4.126521764s) [noise=true , address=104.36.82.9:4028 , piece_id=4LUYCAISMQC3SGDLJZSDFDJET4R54U3N4B6ZAJ35UDXB5CT3GV2Q, seg_pos={0 0}, upload=274674]: Uploading piece... Aug 19 17:38:09.695845597 (264h12m4.126925728s) [noise=true , address=77.90.31.15:4701 , piece_id=CCTT764Z3HFZFT4PE7YOOBVVGDASLFYVFGVXGWADBLDLURBOKGGQ, seg_pos={0 0}, upload=274674]: Uploading piece... Aug 19 17:38:09.696191328 (264h12m4.127271459s) [noise=true , address=45.91.116.5:4302 , piece_id=D2ZYYIFLA7GFPEII4X75HVJGL5DOM6BZLQFXL7HYVXYLNCL4CDQQ, seg_pos={0 0}, upload=274674]: Uploading piece... Aug 19 17:38:09.696539624 (264h12m4.127619755s) [noise=true , address=77.90.31.14:4689 , piece_id=6FO4WN5BXNW6UFGW2ALS5YBYBP5FZCAZBLQZXAULFCN65NHH7N6A, seg_pos={0 0}, upload=274674]: Uploading piece... Aug 19 17:38:09.696809421 (264h12m4.127889552s) [noise=true , address=79.127.201.214:4299 , piece_id=C4JSI4OPEZPPHADT6QTVJGEXQNVSHEK42KNYQBV3S2OH7Z5Z7HHA, seg_pos={0 0}, upload=274674]: Uploading piece... Aug 19 17:38:09.697172445 (264h12m4.128252576s) [noise=true , address=104.36.82.43:4855 , piece_id=3H2PUTGFVC2PR224NBHGRB7GLWBOL2MNXXWJ7JF23HYVF53TEWFQ, seg_pos={0 0}, upload=274674]: Uploading piece... Aug 19 17:38:09.697485064 (264h12m4.128565205s) [noise=true , address=77.90.31.12:4619 , piece_id=XMWKT6JBVLOPTRUIPNEK4GXVCH5OOVO5WYWDGWMXGA2VVVD74CBA, seg_pos={0 0}, upload=274674]: Uploading piece... Aug 19 17:38:09.697774858 (264h12m4.128854989s) [noise=true , address=104.36.82.38:4714 , piece_id=SZ42LIQE4LXTBCZY4BTXWS3EL27UYIQS3AAXGWNLH65TUZWNUYSQ, seg_pos={0 0}, upload=274674]: Uploading piece... Aug 19 17:38:09.698096103 (264h12m4.129176234s) [noise=true , address=104.36.82.4:4817 , piece_id=JNKPBDLUZUWI22ZLGTCD3BZJBAUX7TZVAGXJVFZOEJPK2Z3PMADA, seg_pos={0 0}, upload=274674]: Uploading piece... Aug 19 17:38:09.698352945 (264h12m4.129433076s) [noise=true , address=45.91.116.13:4183 , piece_id=OMUE6JFSTRIF64VYM5XCHNL7U26SEXP3IGWDBHP5G2E6YJQETEUQ, seg_pos={0 0}, upload=274674]: Uploading piece... Aug 19 17:38:09.698876106 (264h12m4.129956227s) [noise=true , address=104.36.82.6:4879 , piece_id=JKK5HMPYTMR7PNWHVPBDHPH3K6H7VM7AA4K4X4WXQ5LN57VRBUFA, seg_pos={0 0}, upload=274674]: Uploading piece... Aug 19 17:38:11.192631202 (264h12m5.623711333s) [upload=274674]: Next returned nil segment Aug 19 17:38:11.192648606 (264h12m5.623728737s) [upload=274674]: Waiting for error group managing segments... Aug 19 17:38:11.591537039 (264h12m6.02261717s) [noise=true , address=45.91.116.14:4222 , piece_id=DXHEK43V7USJLYKPF5PKEHLJPVP4OPH2H3TQVGUBMK5MVRWQHXJQ, seg_pos={0 0}, upload=274674]: Done uploading piece. err: <nil> Aug 19 17:38:11.606093581 (264h12m6.037173712s) [noise=true , address=77.90.31.14:4689 , piece_id=6FO4WN5BXNW6UFGW2ALS5YBYBP5FZCAZBLQZXAULFCN65NHH7N6A, seg_pos={0 0}, upload=274674]: Done uploading piece. err: <nil> Aug 19 17:38:11.610339916 (264h12m6.041420047s) [noise=true , address=77.90.31.15:4701 , piece_id=CCTT764Z3HFZFT4PE7YOOBVVGDASLFYVFGVXGWADBLDLURBOKGGQ, seg_pos={0 0}, upload=274674]: Done uploading piece. err: <nil> Aug 19 17:38:11.630055040 (264h12m6.061135171s) [noise=true , address=79.127.201.215:4343 , piece_id=MIATMAVPWKBJPVYH3XDYXQGKW7VWKRULLP6UV2ZNKJZ7YDVJJEBQ, seg_pos={0 0}, upload=274674]: Done uploading piece. err: <nil> Aug 19 17:38:11.678703675 (264h12m6.109783797s) [noise=true , address=104.36.82.9:4028 , piece_id=4LUYCAISMQC3SGDLJZSDFDJET4R54U3N4B6ZAJ35UDXB5CT3GV2Q, seg_pos={0 0}, upload=274674]: Done uploading piece. err: <nil> Aug 19 17:38:11.693543009 (264h12m6.12462314s) [noise=true , address=104.36.82.60:4364 , piece_id=A5BGK2CC62VYRH27SR2MONG2GPSZ5WONCHAF42LFPXOTM2W7XIJA, seg_pos={0 0}, upload=274674]: Done uploading piece. err: <nil> Aug 19 17:38:11.693568448 (264h12m6.124648579s) [noise=true , address=45.91.116.13:4183 , piece_id=OMUE6JFSTRIF64VYM5XCHNL7U26SEXP3IGWDBHP5G2E6YJQETEUQ, seg_pos={0 0}, upload=274674]: Done uploading piece. err: <nil> Aug 19 17:38:11.698831207 (264h12m6.129911329s) [noise=true , address=104.36.82.5:4852 , piece_id=UALXJPWV34HHWPC6GNCOFNP2RDC3KF6ZY4FUHLL3MWR2UZDCXNCQ, seg_pos={0 0}, upload=274674]: Done uploading piece. err: <nil> Aug 19 17:38:11.698886834 (264h12m6.129966955s) [noise=true , address=77.90.31.12:4619 , piece_id=XMWKT6JBVLOPTRUIPNEK4GXVCH5OOVO5WYWDGWMXGA2VVVD74CBA, seg_pos={0 0}, upload=274674]: Done uploading piece. err: <nil> Aug 19 17:38:11.714704781 (264h12m6.145784912s) [noise=true , address=79.127.201.214:4299 , piece_id=C4JSI4OPEZPPHADT6QTVJGEXQNVSHEK42KNYQBV3S2OH7Z5Z7HHA, seg_pos={0 0}, upload=274674]: Done uploading piece. err: <nil> Aug 19 17:38:11.720319795 (264h12m6.151399956s) [noise=true , address=45.91.116.11:4140 , piece_id=KQNAZLV4XPDYYLJQUMI2UL3GVLA7QTEZ6TU3HDYFYB7LPRBSRQFA, seg_pos={0 0}, upload=274674]: Done uploading piece. err: <nil> Aug 19 17:38:11.722063401 (264h12m6.153143532s) [noise=true , address=104.36.82.6:4879 , piece_id=JKK5HMPYTMR7PNWHVPBDHPH3K6H7VM7AA4K4X4WXQ5LN57VRBUFA, seg_pos={0 0}, upload=274674]: Done uploading piece. err: <nil> Aug 19 17:38:11.726762272 (264h12m6.157842393s) [noise=true , address=104.36.82.38:4714 , piece_id=SZ42LIQE4LXTBCZY4BTXWS3EL27UYIQS3AAXGWNLH65TUZWNUYSQ, seg_pos={0 0}, upload=274674]: Done uploading piece. err: <nil> Aug 19 17:38:11.734258877 (264h12m6.165339008s) [noise=true , address=104.36.82.58:4269 , piece_id=CTQSY5PQFUNXUJR2JXSI7O2EHMYTOAHUSHGTMVTI42E45RYFIGJA, seg_pos={0 0}, upload=274674]: Done uploading piece. err: <nil> Aug 19 17:38:11.734284346 (264h12m6.165364477s) [seg_pos={0 0}, upload=274674]: Segment reached optimal threshold of 14 pieces. Aug 19 17:38:11.734416277 (264h12m6.165496408s) [noise=true , address=104.36.82.4:4817 , piece_id=JNKPBDLUZUWI22ZLGTCD3BZJBAUX7TZVAGXJVFZOEJPK2Z3PMADA, seg_pos={0 0}, upload=274674]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RSBodzJxMye65QgkyrzkWCKQ49cnfdu4haUZKW23nWpqzMcYz): protocol: expected piece hash; context canceled; EOF Aug 19 17:38:11.734486873 (264h12m6.165567004s) [noise=true , address=104.36.82.43:4855 , piece_id=3H2PUTGFVC2PR224NBHGRB7GLWBOL2MNXXWJ7JF23HYVF53TEWFQ, seg_pos={0 0}, upload=274674]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TGTRhEi2KsbrkC6ZmgGkMaJri9uhzEcmXe1J4nJjpEd1BE4ka): protocol: expected piece hash; context canceled; EOF Aug 19 17:38:11.734587926 (264h12m6.165668057s) [noise=true , address=77.90.31.18:4825 , piece_id=UQLJVRVUBBN5LFBIAQL7YQ4IRAWI3SBVBHUT4IIFBROAUHXHVZ3A, seg_pos={0 0}, upload=274674]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PqKyfFxfVKTGARbPqBcQL3Naqjdf5uR44PeLi3sJpu57dcm1x): protocol: expected piece hash; context canceled; EOF Aug 19 17:38:11.734592956 (264h12m6.165673087s) [noise=true , address=45.91.116.5:4302 , piece_id=D2ZYYIFLA7GFPEII4X75HVJGL5DOM6BZLQFXL7HYVXYLNCL4CDQQ, seg_pos={0 0}, upload=274674]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pJWhSLza8WaMuvvDwH4nae57UxQmfCdDvAkQ3v25MgPYHoZXx): protocol: expected piece hash; context canceled; EOF Aug 19 17:38:11.734645206 (264h12m6.165725337s) [seg_pos={0 0}, upload=274674]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 19 17:38:11.734756098 (264h12m6.165836239s) [upload=274674]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 19 17:38:11.734764033 (264h12m6.165844154s) [upload=274674]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 17:38:12.023665554 (264h12m6.454745685s) [upload=274674]: Done uploading segments... Aug 19 17:38:12.023686044 (264h12m6.454766195s) [upload=274674]: Upload finished. err: <nil> Duration: 5.546715398s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 6.065321505s FAIL (2025-08-19 17:55:28 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Got next segment. Inline: false Aug 19 17:23:57.290467622 (263h57m51.721547763s) [upload=274428]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 17:23:57.290471560 (263h57m51.721551681s) [upload=274428]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 19 17:23:57.743765742 (263h57m52.174845863s) [seg_pos={0 0}, upload=274428]: Begin upload segment... Aug 19 17:23:57.743818412 (263h57m52.174898543s) [seg_pos={0 0}, upload=274428]: Done begin upload segment. Aug 19 17:23:57.743842609 (263h57m52.17492274s) [noise=true , address=77.90.31.16:4750 , piece_id=XXJ4VHMZ4NVVT2WX322JS7MW3FHQ7UZJW4VLR4KG6FW4A4RHVN6A, seg_pos={0 0}, upload=274428]: Uploading piece... Aug 19 17:23:57.743875672 (263h57m52.174955803s) [noise=true , address=45.91.116.14:4198 , piece_id=NSLSBMDS2ZJKN62N4NGN7HRHXAUIS4WL4AGHODZZBD2CDYTXHVUA, seg_pos={0 0}, upload=274428]: Uploading piece... Aug 19 17:23:57.744218498 (263h57m52.175298629s) [noise=true , address=77.90.31.14:4667 , piece_id=BF53QUGTGVD23KCIIAA5NWUCWKWUXXSHKCAAWSDZLAORPHXQEMQQ, seg_pos={0 0}, upload=274428]: Uploading piece... Aug 19 17:23:57.744456373 (263h57m52.175536504s) [noise=true , address=77.90.31.21:4883 , piece_id=SVFJCG4T5OCDMVKBE7A33XPDIQUH6NESE4GIVTETRZRBX4N3MJDQ, seg_pos={0 0}, upload=274428]: Uploading piece... Aug 19 17:23:57.744523051 (263h57m52.175603182s) [noise=true , address=104.36.82.60:4365 , piece_id=7BXP5MWOH2B3H74CPVWOQN2NEGMXUYECIVOEHRRKINQ544WKAAHA, seg_pos={0 0}, upload=274428]: Uploading piece... Aug 19 17:23:57.744762870 (263h57m52.175843001s) [noise=true , address=77.90.31.19:4851 , piece_id=POYQI453UA5767REISB5BNE3MYV6IOYZNIK357PYGH5DM66EOBIA, seg_pos={0 0}, upload=274428]: Uploading piece... Aug 19 17:23:57.744993531 (263h57m52.176073662s) [noise=true , address=104.36.82.44:4871 , piece_id=WQGOUIWA7NVK5SUTIWX6E32ZKW62Y2KGK7E3HB5YW2CKDUR42BPQ, seg_pos={0 0}, upload=274428]: Uploading piece... Aug 19 17:23:57.745229373 (263h57m52.176309494s) [noise=true , address=79.127.205.234:4587 , piece_id=ALBAMUJVGA4RIJKGCJNJT3BT6RBD5QWU4C33RJGCEP3XIQQLW6QQ, seg_pos={0 0}, upload=274428]: Uploading piece... Aug 19 17:23:57.745473950 (263h57m52.176554081s) [noise=true , address=104.36.82.3:4771 , piece_id=2RZZG2MORQZR3ROAADIWSK46JDEZSNYBGASCDWZTKWWU72JFHRSQ, seg_pos={0 0}, upload=274428]: Uploading piece... Aug 19 17:23:57.745746030 (263h57m52.176826161s) [noise=true , address=104.36.82.34:4558 , piece_id=AJNHQTDQSMJ3R2ZKVQUCBU5XOMWNF66ID2M3RVPYDL52F5LGTPFQ, seg_pos={0 0}, upload=274428]: Uploading piece... Aug 19 17:23:57.745999426 (263h57m52.177079547s) [noise=true , address=77.90.31.18:4814 , piece_id=VUIFMMFA5PCHIKNNX7PA7RHVXIJHK3UWH4SI34R6XP5MEYTAMS6A, seg_pos={0 0}, upload=274428]: Uploading piece... Aug 19 17:23:57.746774839 (263h57m52.17785497s) [noise=true , address=104.36.82.35:4610 , piece_id=DM7HADQAK5T67ZDQR3VGUUGLHCGBLC5KJYYFF7QQNRKVE2SA5CHA, seg_pos={0 0}, upload=274428]: Uploading piece... Aug 19 17:23:57.747503714 (263h57m52.178583836s) [noise=true , address=64.49.13.137:4496 , piece_id=DW5YMZIQJZIA53K22HGK2ICRT5IVVP2ZMIVYM46BWQWAYPLNUC2Q, seg_pos={0 0}, upload=274428]: Uploading piece... Aug 19 17:23:57.747727743 (263h57m52.178807874s) [noise=true , address=77.90.31.13:4640 , piece_id=NKLBPZJDAIAVAYH3QRDCPS6VWHJBFZZWY3HCTSW2E4II77YC3BYA, seg_pos={0 0}, upload=274428]: Uploading piece... Aug 19 17:23:57.747948565 (263h57m52.179028696s) [noise=true , address=104.36.82.55:4214 , piece_id=64P34K2YEWCWZFGWSR4AQ5SRALT5CSZCB4F6FHPKDP3ZFJIOJC2Q, seg_pos={0 0}, upload=274428]: Uploading piece... Aug 19 17:23:57.748174007 (263h57m52.179254138s) [noise=true , address=104.36.82.23:4448 , piece_id=SCMNBDCYA7PUN2VP6JN37YM5LNY2GBGQHJKYSSWUWMPTZZ7LXZUQ, seg_pos={0 0}, upload=274428]: Uploading piece... Aug 19 17:23:57.748433112 (263h57m52.179513233s) [noise=true , address=104.36.82.5:4854 , piece_id=FC4DIKMBI5OPLDQCVA6P3MA7AQZV23QBIX3UAKLWJJMY724TNYHQ, seg_pos={0 0}, upload=274428]: Uploading piece... Aug 19 17:23:57.748647252 (263h57m52.179727383s) [noise=true , address=104.36.82.62:4442 , piece_id=VV5BBYB6UI4O3B4ESOBJX44I5RJBRWSLJUFR6H3GKEUZWNONTPMA, seg_pos={0 0}, upload=274428]: Uploading piece... Aug 19 17:23:58.869276532 (263h57m53.300356663s) [upload=274428]: Next returned nil segment Aug 19 17:23:58.869291511 (263h57m53.300371642s) [upload=274428]: Waiting for error group managing segments... Aug 19 17:23:59.195579562 (263h57m53.626659683s) [noise=true , address=79.127.205.234:4587 , piece_id=ALBAMUJVGA4RIJKGCJNJT3BT6RBD5QWU4C33RJGCEP3XIQQLW6QQ, seg_pos={0 0}, upload=274428]: Done uploading piece. err: <nil> Aug 19 17:23:59.239543682 (263h57m53.670623813s) [noise=true , address=64.49.13.137:4496 , piece_id=DW5YMZIQJZIA53K22HGK2ICRT5IVVP2ZMIVYM46BWQWAYPLNUC2Q, seg_pos={0 0}, upload=274428]: Done uploading piece. err: <nil> Aug 19 17:23:59.654494638 (263h57m54.085574769s) [noise=true , address=104.36.82.35:4610 , piece_id=DM7HADQAK5T67ZDQR3VGUUGLHCGBLC5KJYYFF7QQNRKVE2SA5CHA, seg_pos={0 0}, upload=274428]: Done uploading piece. err: <nil> Aug 19 17:23:59.700507838 (263h57m54.131587959s) [noise=true , address=104.36.82.34:4558 , piece_id=AJNHQTDQSMJ3R2ZKVQUCBU5XOMWNF66ID2M3RVPYDL52F5LGTPFQ, seg_pos={0 0}, upload=274428]: Done uploading piece. err: <nil> Aug 19 17:23:59.757508140 (263h57m54.188588272s) [noise=true , address=104.36.82.5:4854 , piece_id=FC4DIKMBI5OPLDQCVA6P3MA7AQZV23QBIX3UAKLWJJMY724TNYHQ, seg_pos={0 0}, upload=274428]: Done uploading piece. err: <nil> Aug 19 17:23:59.765247029 (263h57m54.19632716s) [noise=true , address=104.36.82.62:4442 , piece_id=VV5BBYB6UI4O3B4ESOBJX44I5RJBRWSLJUFR6H3GKEUZWNONTPMA, seg_pos={0 0}, upload=274428]: Done uploading piece. err: <nil> Aug 19 17:23:59.774367630 (263h57m54.205447761s) [noise=true , address=104.36.82.23:4448 , piece_id=SCMNBDCYA7PUN2VP6JN37YM5LNY2GBGQHJKYSSWUWMPTZZ7LXZUQ, seg_pos={0 0}, upload=274428]: Done uploading piece. err: <nil> Aug 19 17:23:59.827970201 (263h57m54.259050332s) [noise=true , address=104.36.82.55:4214 , piece_id=64P34K2YEWCWZFGWSR4AQ5SRALT5CSZCB4F6FHPKDP3ZFJIOJC2Q, seg_pos={0 0}, upload=274428]: Done uploading piece. err: <nil> Aug 19 17:23:59.851412395 (263h57m54.282492526s) [noise=true , address=104.36.82.44:4871 , piece_id=WQGOUIWA7NVK5SUTIWX6E32ZKW62Y2KGK7E3HB5YW2CKDUR42BPQ, seg_pos={0 0}, upload=274428]: Done uploading piece. err: <nil> Aug 19 17:23:59.946562012 (263h57m54.377642143s) [noise=true , address=104.36.82.3:4771 , piece_id=2RZZG2MORQZR3ROAADIWSK46JDEZSNYBGASCDWZTKWWU72JFHRSQ, seg_pos={0 0}, upload=274428]: Done uploading piece. err: <nil> Aug 19 17:23:59.979671513 (263h57m54.410751634s) [noise=true , address=104.36.82.60:4365 , piece_id=7BXP5MWOH2B3H74CPVWOQN2NEGMXUYECIVOEHRRKINQ544WKAAHA, seg_pos={0 0}, upload=274428]: Done uploading piece. err: <nil> Aug 19 17:24:55.494274881 (263h58m49.925355012s) [noise=true , address=77.90.31.18:4814 , piece_id=VUIFMMFA5PCHIKNNX7PA7RHVXIJHK3UWH4SI34R6XP5MEYTAMS6A, seg_pos={0 0}, upload=274428]: Done uploading piece. err: <nil> Aug 19 17:24:55.842835997 (263h58m50.273916128s) [noise=true , address=77.90.31.21:4883 , piece_id=SVFJCG4T5OCDMVKBE7A33XPDIQUH6NESE4GIVTETRZRBX4N3MJDQ, seg_pos={0 0}, upload=274428]: Done uploading piece. err: <nil> Aug 19 17:24:56.634889530 (263h58m51.065969661s) [noise=true , address=77.90.31.19:4851 , piece_id=POYQI453UA5767REISB5BNE3MYV6IOYZNIK357PYGH5DM66EOBIA, seg_pos={0 0}, upload=274428]: Done uploading piece. err: ecclient: upload failed (node:1NEV2ndHw4fohVUNsfrzwojd8S11TmMST6Sz59GVnXS8vDGhWk, address:77.90.31.19:4851): protocol: rpc: dial tcp 77.90.31.19:4851: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:56.635097748 (263h58m51.066177879s) [noise=true , address=45.91.116.14:4198 , piece_id=NSLSBMDS2ZJKN62N4NGN7HRHXAUIS4WL4AGHODZZBD2CDYTXHVUA, seg_pos={0 0}, upload=274428]: Done uploading piece. err: ecclient: upload failed (node:1xxfUrCm5PgSLpge1wYiH2EcE7kzzJ38P9rfeCYmqXTbTisioi, address:45.91.116.14:4198): protocol: rpc: dial tcp 45.91.116.14:4198: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:56.635158895 (263h58m51.066239036s) [noise=true , address=77.90.31.16:4750 , piece_id=XXJ4VHMZ4NVVT2WX322JS7MW3FHQ7UZJW4VLR4KG6FW4A4RHVN6A, seg_pos={0 0}, upload=274428]: Done uploading piece. err: ecclient: upload failed (node:12TGd8NnWdekDtHzBQHmJPgTtHjXC6g8fi6RoZYFQJVHDa5hdyE, address:77.90.31.16:4750): protocol: read tcp 188.166.225.170:54198->77.90.31.16:4750: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:56.635224120 (263h58m51.066304281s) [noise=true , address=77.90.31.14:4667 , piece_id=BF53QUGTGVD23KCIIAA5NWUCWKWUXXSHKCAAWSDZLAORPHXQEMQQ, seg_pos={0 0}, upload=274428]: Done uploading piece. err: ecclient: upload failed (node:12R5gJFhsmocYPNZf4yNnYqJwcwAKt9yiHmmqGMk3btzVjkFV4S, address:77.90.31.14:4667): protocol: read tcp 188.166.225.170:46984->77.90.31.14:4667: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:56.635280057 (263h58m51.066360198s) [noise=true , address=77.90.31.13:4640 , piece_id=NKLBPZJDAIAVAYH3QRDCPS6VWHJBFZZWY3HCTSW2E4II77YC3BYA, seg_pos={0 0}, upload=274428]: Done uploading piece. err: ecclient: upload failed (node:1XjJpLBEZm5daHcBUmUyC7QpLVhu9Vm9ex9RYcYJWEuLGk527F, address:77.90.31.13:4640): protocol: read tcp 188.166.225.170:48656->77.90.31.13:4640: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:56.635314653 (263h58m51.066394794s) [seg_pos={0 0}, upload=274428]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Aug 19 17:24:56.862515362 (263h58m51.293595493s) [upload=274428]: Done uploading segments... Aug 19 17:24:56.862536042 (263h58m51.293616163s) [upload=274428]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 6.06535029s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...........X.....X..........................................] duration: 6.086873092s FAIL (2025-08-19 17:55:17 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.086897559s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [...............................X...........................X] duration: 1.404257301s FAIL (2025-08-19 17:55:09 UTC) orbiter warning: operation: ListObjects ======ERROR====== unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/1K ======LOGS====== Aug 19 17:55:08.636689315 (264h29m3.067769446s) [upload=274973]: Starting upload Aug 19 17:55:08.636740844 (264h29m3.067820975s) [upload=274973]: Done starting upload Aug 19 17:55:08.636829314 (264h29m3.067909445s) [upload=274973]: Uploading segments... Aug 19 17:55:08.636904547 (264h29m3.067984678s) [upload=274973]: Got next segment. Inline: true Aug 19 17:55:08.636912002 (264h29m3.067992133s) [upload=274973]: Waiting for error group managing segments... Aug 19 17:55:08.636956797 (264h29m3.068036928s) [upload=274973]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 17:55:08.636961596 (264h29m3.068041727s) [upload=274973]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 19 17:55:08.636964062 (264h29m3.068044193s) [upload=274973]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 17:55:08.938231099 (264h29m3.36931123s) [upload=274973]: Done uploading segments... Aug 19 17:55:08.938277699 (264h29m3.36935785s) [upload=274973]: Upload finished. err: <nil> Duration:* 1.404285285s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 1.876543761s FAIL (2025-08-19 17:54:54 UTC) orbiter warning: operation: ListObjects ======ERROR====== uplink: metainfo: metaclient: context deadline exceeded ======LOGS====== Aug 19 17:35:36.607320794 (264h9m31.038400925s) [upload=274632]: Starting upload Aug 19 17:35:36.607367684 (264h9m31.038447815s) [upload=274632]: Done starting upload Aug 19 17:35:36.607435082 (264h9m31.038515213s) [upload=274632]: Uploading segments... Aug 19 17:35:36.607490538 (264h9m31.038570669s) [upload=274632]: Got next segment. Inline: true Aug 19 17:35:36.607495088 (264h9m31.038575219s) [upload=274632]: Waiting for error group managing segments... Aug 19 17:35:36.607537528 (264h9m31.038617659s) [upload=274632]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 17:35:36.607541656 (264h9m31.038621787s) [upload=274632]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 19 17:35:36.607544051 (264h9m31.038624182s) [upload=274632]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 17:35:36.889055260 (264h9m31.320135391s) [upload=274632]: Done uploading segments... Aug 19 17:35:36.889082943 (264h9m31.320163124s) [upload=274632]: Upload finished. err: <nil> Duration:* 1.876564459s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 6 out of 60 is failed (threshold=3) [...........X......X.X........X....XX........................] duration: 6.047583115s FAIL (2025-08-19 17:54:48 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/4M") ======LOGS====== Aug 19 17:38:09.246414094 (264h12m3.677494255s) [upload=274674]: Starting upload Aug 19 17:38:09.246486392 (264h12m3.677566533s) [upload=274674]: Done starting upload Aug 19 17:38:09.246601452 (264h12m3.677681583s) [upload=274674]: Uploading segments... Aug 19 17:38:09.246691885 (264h12m3.677772016s) [upload=274674]: Got next segment. Inline: false Aug 19 17:38:09.246724627 (264h12m3.677804758s) [upload=274674]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 17:38:09.246730909 (264h12m3.67781104s) [upload=274674]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 19 17:38:09.693242556 (264h12m4.124322687s) [seg_pos={0 0}, upload=274674]: Begin upload segment... Aug 19 17:38:09.693302230 (264h12m4.124382361s) [seg_pos={0 0}, upload=274674]: Done begin upload segment. Aug 19 17:38:09.693343529 (264h12m4.12442367s) [noise=true , address=77.90.31.18:4825 , piece_id=UQLJVRVUBBN5LFBIAQL7YQ4IRAWI3SBVBHUT4IIFBROAUHXHVZ3A, seg_pos={0 0}, upload=274674]: Uploading piece... Aug 19 17:38:09.693350092 (264h12m4.124430213s) [noise=true , address=45.91.116.14:4222 , piece_id=DXHEK43V7USJLYKPF5PKEHLJPVP4OPH2H3TQVGUBMK5MVRWQHXJQ, seg_pos={0 0}, upload=274674]: Uploading piece... Aug 19 17:38:09.693395940 (264h12m4.124476081s) [noise=true , address=45.91.116.11:4140 , piece_id=KQNAZLV4XPDYYLJQUMI2UL3GVLA7QTEZ6TU3HDYFYB7LPRBSRQFA, seg_pos={0 0}, upload=274674]: Uploading piece... Aug 19 17:38:09.693960980 (264h12m4.125041131s) [noise=true , address=79.127.201.215:4343 , piece_id=MIATMAVPWKBJPVYH3XDYXQGKW7VWKRULLP6UV2ZNKJZ7YDVJJEBQ, seg_pos={0 0}, upload=274674]: Uploading piece... Aug 19 17:38:09.694358652 (264h12m4.125438783s) [noise=true , address=104.36.82.5:4852 , piece_id=UALXJPWV34HHWPC6GNCOFNP2RDC3KF6ZY4FUHLL3MWR2UZDCXNCQ, seg_pos={0 0}, upload=274674]: Uploading piece... Aug 19 17:38:09.694689475 (264h12m4.125769606s) [noise=true , address=104.36.82.58:4269 , piece_id=CTQSY5PQFUNXUJR2JXSI7O2EHMYTOAHUSHGTMVTI42E45RYFIGJA, seg_pos={0 0}, upload=274674]: Uploading piece... Aug 19 17:38:09.695011601 (264h12m4.126091732s) [noise=true , address=104.36.82.60:4364 , piece_id=A5BGK2CC62VYRH27SR2MONG2GPSZ5WONCHAF42LFPXOTM2W7XIJA, seg_pos={0 0}, upload=274674]: Uploading piece... Aug 19 17:38:09.695441633 (264h12m4.126521764s) [noise=true , address=104.36.82.9:4028 , piece_id=4LUYCAISMQC3SGDLJZSDFDJET4R54U3N4B6ZAJ35UDXB5CT3GV2Q, seg_pos={0 0}, upload=274674]: Uploading piece... Aug 19 17:38:09.695845597 (264h12m4.126925728s) [noise=true , address=77.90.31.15:4701 , piece_id=CCTT764Z3HFZFT4PE7YOOBVVGDASLFYVFGVXGWADBLDLURBOKGGQ, seg_pos={0 0}, upload=274674]: Uploading piece... Aug 19 17:38:09.696191328 (264h12m4.127271459s) [noise=true , address=45.91.116.5:4302 , piece_id=D2ZYYIFLA7GFPEII4X75HVJGL5DOM6BZLQFXL7HYVXYLNCL4CDQQ, seg_pos={0 0}, upload=274674]: Uploading piece... Aug 19 17:38:09.696539624 (264h12m4.127619755s) [noise=true , address=77.90.31.14:4689 , piece_id=6FO4WN5BXNW6UFGW2ALS5YBYBP5FZCAZBLQZXAULFCN65NHH7N6A, seg_pos={0 0}, upload=274674]: Uploading piece... Aug 19 17:38:09.696809421 (264h12m4.127889552s) [noise=true , address=79.127.201.214:4299 , piece_id=C4JSI4OPEZPPHADT6QTVJGEXQNVSHEK42KNYQBV3S2OH7Z5Z7HHA, seg_pos={0 0}, upload=274674]: Uploading piece... Aug 19 17:38:09.697172445 (264h12m4.128252576s) [noise=true , address=104.36.82.43:4855 , piece_id=3H2PUTGFVC2PR224NBHGRB7GLWBOL2MNXXWJ7JF23HYVF53TEWFQ, seg_pos={0 0}, upload=274674]: Uploading piece... Aug 19 17:38:09.697485064 (264h12m4.128565205s) [noise=true , address=77.90.31.12:4619 , piece_id=XMWKT6JBVLOPTRUIPNEK4GXVCH5OOVO5WYWDGWMXGA2VVVD74CBA, seg_pos={0 0}, upload=274674]: Uploading piece... Aug 19 17:38:09.697774858 (264h12m4.128854989s) [noise=true , address=104.36.82.38:4714 , piece_id=SZ42LIQE4LXTBCZY4BTXWS3EL27UYIQS3AAXGWNLH65TUZWNUYSQ, seg_pos={0 0}, upload=274674]: Uploading piece... Aug 19 17:38:09.698096103 (264h12m4.129176234s) [noise=true , address=104.36.82.4:4817 , piece_id=JNKPBDLUZUWI22ZLGTCD3BZJBAUX7TZVAGXJVFZOEJPK2Z3PMADA, seg_pos={0 0}, upload=274674]: Uploading piece... Aug 19 17:38:09.698352945 (264h12m4.129433076s) [noise=true , address=45.91.116.13:4183 , piece_id=OMUE6JFSTRIF64VYM5XCHNL7U26SEXP3IGWDBHP5G2E6YJQETEUQ, seg_pos={0 0}, upload=274674]: Uploading piece... Aug 19 17:38:09.698876106 (264h12m4.129956227s) [noise=true , address=104.36.82.6:4879 , piece_id=JKK5HMPYTMR7PNWHVPBDHPH3K6H7VM7AA4K4X4WXQ5LN57VRBUFA, seg_pos={0 0}, upload=274674]: Uploading piece... Aug 19 17:38:11.192631202 (264h12m5.623711333s) [upload=274674]: Next returned nil segment Aug 19 17:38:11.192648606 (264h12m5.623728737s) [upload=274674]: Waiting for error group managing segments... Aug 19 17:38:11.591537039 (264h12m6.02261717s) [noise=true , address=45.91.116.14:4222 , piece_id=DXHEK43V7USJLYKPF5PKEHLJPVP4OPH2H3TQVGUBMK5MVRWQHXJQ, seg_pos={0 0}, upload=274674]: Done uploading piece. err: <nil> Aug 19 17:38:11.606093581 (264h12m6.037173712s) [noise=true , address=77.90.31.14:4689 , piece_id=6FO4WN5BXNW6UFGW2ALS5YBYBP5FZCAZBLQZXAULFCN65NHH7N6A, seg_pos={0 0}, upload=274674]: Done uploading piece. err: <nil> Aug 19 17:38:11.610339916 (264h12m6.041420047s) [noise=true , address=77.90.31.15:4701 , piece_id=CCTT764Z3HFZFT4PE7YOOBVVGDASLFYVFGVXGWADBLDLURBOKGGQ, seg_pos={0 0}, upload=274674]: Done uploading piece. err: <nil> Aug 19 17:38:11.630055040 (264h12m6.061135171s) [noise=true , address=79.127.201.215:4343 , piece_id=MIATMAVPWKBJPVYH3XDYXQGKW7VWKRULLP6UV2ZNKJZ7YDVJJEBQ, seg_pos={0 0}, upload=274674]: Done uploading piece. err: <nil> Aug 19 17:38:11.678703675 (264h12m6.109783797s) [noise=true , address=104.36.82.9:4028 , piece_id=4LUYCAISMQC3SGDLJZSDFDJET4R54U3N4B6ZAJ35UDXB5CT3GV2Q, seg_pos={0 0}, upload=274674]: Done uploading piece. err: <nil> Aug 19 17:38:11.693543009 (264h12m6.12462314s) [noise=true , address=104.36.82.60:4364 , piece_id=A5BGK2CC62VYRH27SR2MONG2GPSZ5WONCHAF42LFPXOTM2W7XIJA, seg_pos={0 0}, upload=274674]: Done uploading piece. err: <nil> Aug 19 17:38:11.693568448 (264h12m6.124648579s) [noise=true , address=45.91.116.13:4183 , piece_id=OMUE6JFSTRIF64VYM5XCHNL7U26SEXP3IGWDBHP5G2E6YJQETEUQ, seg_pos={0 0}, upload=274674]: Done uploading piece. err: <nil> Aug 19 17:38:11.698831207 (264h12m6.129911329s) [noise=true , address=104.36.82.5:4852 , piece_id=UALXJPWV34HHWPC6GNCOFNP2RDC3KF6ZY4FUHLL3MWR2UZDCXNCQ, seg_pos={0 0}, upload=274674]: Done uploading piece. err: <nil> Aug 19 17:38:11.698886834 (264h12m6.129966955s) [noise=true , address=77.90.31.12:4619 , piece_id=XMWKT6JBVLOPTRUIPNEK4GXVCH5OOVO5WYWDGWMXGA2VVVD74CBA, seg_pos={0 0}, upload=274674]: Done uploading piece. err: <nil> Aug 19 17:38:11.714704781 (264h12m6.145784912s) [noise=true , address=79.127.201.214:4299 , piece_id=C4JSI4OPEZPPHADT6QTVJGEXQNVSHEK42KNYQBV3S2OH7Z5Z7HHA, seg_pos={0 0}, upload=274674]: Done uploading piece. err: <nil> Aug 19 17:38:11.720319795 (264h12m6.151399956s) [noise=true , address=45.91.116.11:4140 , piece_id=KQNAZLV4XPDYYLJQUMI2UL3GVLA7QTEZ6TU3HDYFYB7LPRBSRQFA, seg_pos={0 0}, upload=274674]: Done uploading piece. err: <nil> Aug 19 17:38:11.722063401 (264h12m6.153143532s) [noise=true , address=104.36.82.6:4879 , piece_id=JKK5HMPYTMR7PNWHVPBDHPH3K6H7VM7AA4K4X4WXQ5LN57VRBUFA, seg_pos={0 0}, upload=274674]: Done uploading piece. err: <nil> Aug 19 17:38:11.726762272 (264h12m6.157842393s) [noise=true , address=104.36.82.38:4714 , piece_id=SZ42LIQE4LXTBCZY4BTXWS3EL27UYIQS3AAXGWNLH65TUZWNUYSQ, seg_pos={0 0}, upload=274674]: Done uploading piece. err: <nil> Aug 19 17:38:11.734258877 (264h12m6.165339008s) [noise=true , address=104.36.82.58:4269 , piece_id=CTQSY5PQFUNXUJR2JXSI7O2EHMYTOAHUSHGTMVTI42E45RYFIGJA, seg_pos={0 0}, upload=274674]: Done uploading piece. err: <nil> Aug 19 17:38:11.734284346 (264h12m6.165364477s) [seg_pos={0 0}, upload=274674]: Segment reached optimal threshold of 14 pieces. Aug 19 17:38:11.734416277 (264h12m6.165496408s) [noise=true , address=104.36.82.4:4817 , piece_id=JNKPBDLUZUWI22ZLGTCD3BZJBAUX7TZVAGXJVFZOEJPK2Z3PMADA, seg_pos={0 0}, upload=274674]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RSBodzJxMye65QgkyrzkWCKQ49cnfdu4haUZKW23nWpqzMcYz): protocol: expected piece hash; context canceled; EOF Aug 19 17:38:11.734486873 (264h12m6.165567004s) [noise=true , address=104.36.82.43:4855 , piece_id=3H2PUTGFVC2PR224NBHGRB7GLWBOL2MNXXWJ7JF23HYVF53TEWFQ, seg_pos={0 0}, upload=274674]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TGTRhEi2KsbrkC6ZmgGkMaJri9uhzEcmXe1J4nJjpEd1BE4ka): protocol: expected piece hash; context canceled; EOF Aug 19 17:38:11.734587926 (264h12m6.165668057s) [noise=true , address=77.90.31.18:4825 , piece_id=UQLJVRVUBBN5LFBIAQL7YQ4IRAWI3SBVBHUT4IIFBROAUHXHVZ3A, seg_pos={0 0}, upload=274674]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PqKyfFxfVKTGARbPqBcQL3Naqjdf5uR44PeLi3sJpu57dcm1x): protocol: expected piece hash; context canceled; EOF Aug 19 17:38:11.734592956 (264h12m6.165673087s) [noise=true , address=45.91.116.5:4302 , piece_id=D2ZYYIFLA7GFPEII4X75HVJGL5DOM6BZLQFXL7HYVXYLNCL4CDQQ, seg_pos={0 0}, upload=274674]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pJWhSLza8WaMuvvDwH4nae57UxQmfCdDvAkQ3v25MgPYHoZXx): protocol: expected piece hash; context canceled; EOF Aug 19 17:38:11.734645206 (264h12m6.165725337s) [seg_pos={0 0}, upload=274674]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 19 17:38:11.734756098 (264h12m6.165836239s) [upload=274674]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 19 17:38:11.734764033 (264h12m6.165844154s) [upload=274674]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 17:38:12.023665554 (264h12m6.454745685s) [upload=274674]: Done uploading segments... Aug 19 17:38:12.023686044 (264h12m6.454766195s) [upload=274674]: Upload finished. err: <nil> Duration: 6.047609585s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.............X...X..........................................] duration: 6.03416559s FAIL (2025-08-19 17:54:38 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 23:59.057943072 (263h57m53.489023273s) [noise=true , address=45.91.116.10:4083 , piece_id=V3H4T2WN2MBVALSIHZNOQHBE75WQRNYLIE3Z7DFWUUHBPYDT7IFA, seg_pos={0 0}, upload=274429]: Uploading piece... Aug 19 17:23:59.057955436 (263h57m53.489035557s) [noise=true , address=45.91.116.5:4298 , piece_id=HMF6DXO5KFMLK67WVA7BAWLQFLC2BCVPJ7ZT255FMRCCEI3MCBNA, seg_pos={0 0}, upload=274429]: Uploading piece... Aug 19 17:23:59.058557689 (263h57m53.48963782s) [noise=true , address=104.36.82.8:4969 , piece_id=3P655Z5OUVZ4VRESACEBKYR3KUFYY56NYPWX7ZVOHL3XI2YQ5TCQ, seg_pos={0 0}, upload=274429]: Uploading piece... Aug 19 17:23:59.059170771 (263h57m53.490250902s) [noise=true , address=45.91.116.7:4009 , piece_id=NSZYKL6FHDLDM463ZXKEQGX4ZY7JPJXENH63PPDV76XMGCBV7ZRQ, seg_pos={0 0}, upload=274429]: Uploading piece... Aug 19 17:23:59.059468281 (263h57m53.490548412s) [noise=true , address=77.90.31.15:4699 , piece_id=MFF7PNMDLHYEH7PY46MGGOJSY4XPHRDPGFWJABBO4JDTJJJJ3CTQ, seg_pos={0 0}, upload=274429]: Uploading piece... Aug 19 17:23:59.059498208 (263h57m53.490578339s) [noise=true , address=104.36.82.2:4729 , piece_id=BNX6UNYZ7VHQ4MKAH65DGSHULWRRXDSCMKOE7WFZO6T2V3ELBJ7Q, seg_pos={0 0}, upload=274429]: Uploading piece... Aug 19 17:23:59.059814182 (263h57m53.490894323s) [noise=true , address=104.36.82.6:4874 , piece_id=QMGNWSGMEV762Y2XEDK2LIAY3JMNLD3UOVFQ6HYP7ZMLLVAVO7ZQ, seg_pos={0 0}, upload=274429]: Uploading piece... Aug 19 17:23:59.059839941 (263h57m53.490920072s) [noise=true , address=45.91.116.12:4225 , piece_id=PCJYR2VLFH23UPQEABUYESR3MHKDHIVTJWDIWVYIL6JAFGGXXXFQ, seg_pos={0 0}, upload=274429]: Uploading piece... Aug 19 17:23:59.059875480 (263h57m53.490955611s) [noise=true , address=104.36.82.10:4079 , piece_id=S3BSVJWRPP2PDGLG7FKFROUKAM3GTBDL2ALAJPAZQQQZJVRMMZ3Q, seg_pos={0 0}, upload=274429]: Uploading piece... Aug 19 17:23:59.060109147 (263h57m53.491189278s) [noise=true , address=104.36.82.2:4732 , piece_id=NLTK4572JAQHZQRDD2DOOIW7VBO7NK4DI4RUZRSXQ3GR72EWV3KQ, seg_pos={0 0}, upload=274429]: Uploading piece... Aug 19 17:23:59.060142190 (263h57m53.491222321s) [noise=true , address=45.91.116.9:4261 , piece_id=J5IMFL3JW4YKDLI6OEY2POEB4LJR2P3BIFAD4YIF3E4JTUXIXY3A, seg_pos={0 0}, upload=274429]: Uploading piece... Aug 19 17:23:59.060348274 (263h57m53.491428405s) [noise=true , address=104.36.82.15:4259 , piece_id=RM7JL7JKCMFM4CNXVVZQ2QJTOEPQEQHGLW3GYXWUD6LAQ2ZJ3P5Q, seg_pos={0 0}, upload=274429]: Uploading piece... Aug 19 17:23:59.060544830 (263h57m53.491624961s) [noise=true , address=45.91.116.14:4191 , piece_id=PJ33GP36S76VCI5RVR26PBFZ6HZUUZYVWNGPJITRLGDZI6LWCJIQ, seg_pos={0 0}, upload=274429]: Uploading piece... Aug 19 17:23:59.060752227 (263h57m53.491832358s) [noise=true , address=45.91.116.11:4118 , piece_id=VD2HK2QU63UVJFRASQOIRFZRWZN2HAUSAZJL2VZKYALGSV67SP2A, seg_pos={0 0}, upload=274429]: Uploading piece... Aug 19 17:23:59.060967990 (263h57m53.492048121s) [noise=true , address=77.90.31.11:4555 , piece_id=GTU6PGCHI4WOFGK2B3FP2Y22VUDBWLQKKEMZBT6I4F47OKK4RPNQ, seg_pos={0 0}, upload=274429]: Uploading piece... Aug 19 17:23:59.061294044 (263h57m53.492374175s) [noise=true , address=45.91.116.8:4048 , piece_id=M2CBILQKSVSQ7VY4AWQQM5GHF2KYBSXH73YJGAEJZAHOR7FOBWEQ, seg_pos={0 0}, upload=274429]: Uploading piece... Aug 19 17:24:00.872920400 (263h57m55.304000531s) [upload=274429]: Next returned nil segment Aug 19 17:24:00.872945548 (263h57m55.304025679s) [upload=274429]: Waiting for error group managing segments... Aug 19 17:24:01.156214888 (263h57m55.587295019s) [noise=true , address=104.36.82.6:4874 , piece_id=QMGNWSGMEV762Y2XEDK2LIAY3JMNLD3UOVFQ6HYP7ZMLLVAVO7ZQ, seg_pos={0 0}, upload=274429]: Done uploading piece. err: <nil> Aug 19 17:24:01.181164926 (263h57m55.612245057s) [noise=true , address=104.36.82.2:4732 , piece_id=NLTK4572JAQHZQRDD2DOOIW7VBO7NK4DI4RUZRSXQ3GR72EWV3KQ, seg_pos={0 0}, upload=274429]: Done uploading piece. err: <nil> Aug 19 17:24:01.222047199 (263h57m55.653127331s) [noise=true , address=104.36.82.2:4729 , piece_id=BNX6UNYZ7VHQ4MKAH65DGSHULWRRXDSCMKOE7WFZO6T2V3ELBJ7Q, seg_pos={0 0}, upload=274429]: Done uploading piece. err: <nil> Aug 19 17:24:01.250473409 (263h57m55.68155354s) [noise=true , address=104.36.82.15:4259 , piece_id=RM7JL7JKCMFM4CNXVVZQ2QJTOEPQEQHGLW3GYXWUD6LAQ2ZJ3P5Q, seg_pos={0 0}, upload=274429]: Done uploading piece. err: <nil> Aug 19 17:24:01.259222299 (263h57m55.69030242s) [noise=true , address=104.36.82.8:4969 , piece_id=3P655Z5OUVZ4VRESACEBKYR3KUFYY56NYPWX7ZVOHL3XI2YQ5TCQ, seg_pos={0 0}, upload=274429]: Done uploading piece. err: <nil> Aug 19 17:24:01.261735818 (263h57m55.692815949s) [noise=true , address=104.36.82.10:4079 , piece_id=S3BSVJWRPP2PDGLG7FKFROUKAM3GTBDL2ALAJPAZQQQZJVRMMZ3Q, seg_pos={0 0}, upload=274429]: Done uploading piece. err: <nil> Aug 19 17:24:28.183964515 (263h58m22.615044646s) [noise=true , address=45.91.116.14:4191 , piece_id=PJ33GP36S76VCI5RVR26PBFZ6HZUUZYVWNGPJITRLGDZI6LWCJIQ, seg_pos={0 0}, upload=274429]: Done uploading piece. err: ecclient: upload failed (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, address:45.91.116.14:4191): protocol: read tcp 188.166.225.170:37646->45.91.116.14:4191: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:28.184013127 (263h58m22.615093258s) [noise=true , address=45.91.116.9:4261 , piece_id=J5IMFL3JW4YKDLI6OEY2POEB4LJR2P3BIFAD4YIF3E4JTUXIXY3A, seg_pos={0 0}, upload=274429]: Done uploading piece. err: ecclient: upload failed (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, address:45.91.116.9:4261): protocol: rpc: dial tcp 45.91.116.9:4261: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:28.184026694 (263h58m22.615106825s) [noise=true , address=45.91.116.8:4048 , piece_id=M2CBILQKSVSQ7VY4AWQQM5GHF2KYBSXH73YJGAEJZAHOR7FOBWEQ, seg_pos={0 0}, upload=274429]: Done uploading piece. err: ecclient: upload failed (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, address:45.91.116.8:4048): protocol: read tcp 188.166.225.170:45604->45.91.116.8:4048: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:28.184070568 (263h58m22.615150689s) [noise=true , address=45.91.116.5:4298 , piece_id=HMF6DXO5KFMLK67WVA7BAWLQFLC2BCVPJ7ZT255FMRCCEI3MCBNA, seg_pos={0 0}, upload=274429]: Done uploading piece. err: ecclient: upload failed (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, address:45.91.116.5:4298): protocol: read tcp 188.166.225.170:46970->45.91.116.5:4298: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:28.184103010 (263h58m22.615183141s) [noise=true , address=45.91.116.10:4083 , piece_id=V3H4T2WN2MBVALSIHZNOQHBE75WQRNYLIE3Z7DFWUUHBPYDT7IFA, seg_pos={0 0}, upload=274429]: Done uploading piece. err: ecclient: upload failed (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, address:45.91.116.10:4083): protocol: read tcp 188.166.225.170:41632->45.91.116.10:4083: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:28.184139489 (263h58m22.61521962s) [noise=true , address=45.91.116.7:4009 , piece_id=NSZYKL6FHDLDM463ZXKEQGX4ZY7JPJXENH63PPDV76XMGCBV7ZRQ, seg_pos={0 0}, upload=274429]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: read tcp 188.166.225.170:45756->45.91.116.7:4009: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:28.184171009 (263h58m22.615251141s) [noise=true , address=77.90.31.11:4555 , piece_id=GTU6PGCHI4WOFGK2B3FP2Y22VUDBWLQKKEMZBT6I4F47OKK4RPNQ, seg_pos={0 0}, upload=274429]: Done uploading piece. err: ecclient: upload failed (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, address:77.90.31.11:4555): protocol: read tcp 188.166.225.170:37438->77.90.31.11:4555: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:28.184197480 (263h58m22.615277611s) [noise=true , address=77.90.31.15:4699 , piece_id=MFF7PNMDLHYEH7PY46MGGOJSY4XPHRDPGFWJABBO4JDTJJJJ3CTQ, seg_pos={0 0}, upload=274429]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: read tcp 188.166.225.170:36788->77.90.31.15:4699: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:28.184218410 (263h58m22.615298541s) [noise=true , address=45.91.116.12:4225 , piece_id=PCJYR2VLFH23UPQEABUYESR3MHKDHIVTJWDIWVYIL6JAFGGXXXFQ, seg_pos={0 0}, upload=274429]: Done uploading piece. err: ecclient: upload failed (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, address:45.91.116.12:4225): protocol: read tcp 188.166.225.170:60636->45.91.116.12:4225: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:28.184249710 (263h58m22.615329841s) [noise=true , address=45.91.116.11:4118 , piece_id=VD2HK2QU63UVJFRASQOIRFZRWZN2HAUSAZJL2VZKYALGSV67SP2A, seg_pos={0 0}, upload=274429]: Done uploading piece. err: ecclient: upload failed (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, address:45.91.116.11:4118): protocol: read tcp 188.166.225.170:48006->45.91.116.11:4118: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:28.184274337 (263h58m22.615354468s) [seg_pos={0 0}, upload=274429]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Aug 19 17:24:28.412483516 (263h58m22.843563647s) [upload=274429]: Done uploading segments... Aug 19 17:24:28.412520628 (263h58m22.843600759s) [upload=274429]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 6.034194525s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [............X.....X.........................................] duration: 6.103210403s FAIL (2025-08-19 17:54:25 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.103236351s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 6 out of 60 is failed (threshold=3) [............X......X.X........X....XX.......................] duration: 5.937193623s FAIL (2025-08-19 17:54:23 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/4M") ======LOGS====== Aug 19 17:38:09.246414094 (264h12m3.677494255s) [upload=274674]: Starting upload Aug 19 17:38:09.246486392 (264h12m3.677566533s) [upload=274674]: Done starting upload Aug 19 17:38:09.246601452 (264h12m3.677681583s) [upload=274674]: Uploading segments... Aug 19 17:38:09.246691885 (264h12m3.677772016s) [upload=274674]: Got next segment. Inline: false Aug 19 17:38:09.246724627 (264h12m3.677804758s) [upload=274674]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 17:38:09.246730909 (264h12m3.67781104s) [upload=274674]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 19 17:38:09.693242556 (264h12m4.124322687s) [seg_pos={0 0}, upload=274674]: Begin upload segment... Aug 19 17:38:09.693302230 (264h12m4.124382361s) [seg_pos={0 0}, upload=274674]: Done begin upload segment. Aug 19 17:38:09.693343529 (264h12m4.12442367s) [noise=true , address=77.90.31.18:4825 , piece_id=UQLJVRVUBBN5LFBIAQL7YQ4IRAWI3SBVBHUT4IIFBROAUHXHVZ3A, seg_pos={0 0}, upload=274674]: Uploading piece... Aug 19 17:38:09.693350092 (264h12m4.124430213s) [noise=true , address=45.91.116.14:4222 , piece_id=DXHEK43V7USJLYKPF5PKEHLJPVP4OPH2H3TQVGUBMK5MVRWQHXJQ, seg_pos={0 0}, upload=274674]: Uploading piece... Aug 19 17:38:09.693395940 (264h12m4.124476081s) [noise=true , address=45.91.116.11:4140 , piece_id=KQNAZLV4XPDYYLJQUMI2UL3GVLA7QTEZ6TU3HDYFYB7LPRBSRQFA, seg_pos={0 0}, upload=274674]: Uploading piece... Aug 19 17:38:09.693960980 (264h12m4.125041131s) [noise=true , address=79.127.201.215:4343 , piece_id=MIATMAVPWKBJPVYH3XDYXQGKW7VWKRULLP6UV2ZNKJZ7YDVJJEBQ, seg_pos={0 0}, upload=274674]: Uploading piece... Aug 19 17:38:09.694358652 (264h12m4.125438783s) [noise=true , address=104.36.82.5:4852 , piece_id=UALXJPWV34HHWPC6GNCOFNP2RDC3KF6ZY4FUHLL3MWR2UZDCXNCQ, seg_pos={0 0}, upload=274674]: Uploading piece... Aug 19 17:38:09.694689475 (264h12m4.125769606s) [noise=true , address=104.36.82.58:4269 , piece_id=CTQSY5PQFUNXUJR2JXSI7O2EHMYTOAHUSHGTMVTI42E45RYFIGJA, seg_pos={0 0}, upload=274674]: Uploading piece... Aug 19 17:38:09.695011601 (264h12m4.126091732s) [noise=true , address=104.36.82.60:4364 , piece_id=A5BGK2CC62VYRH27SR2MONG2GPSZ5WONCHAF42LFPXOTM2W7XIJA, seg_pos={0 0}, upload=274674]: Uploading piece... Aug 19 17:38:09.695441633 (264h12m4.126521764s) [noise=true , address=104.36.82.9:4028 , piece_id=4LUYCAISMQC3SGDLJZSDFDJET4R54U3N4B6ZAJ35UDXB5CT3GV2Q, seg_pos={0 0}, upload=274674]: Uploading piece... Aug 19 17:38:09.695845597 (264h12m4.126925728s) [noise=true , address=77.90.31.15:4701 , piece_id=CCTT764Z3HFZFT4PE7YOOBVVGDASLFYVFGVXGWADBLDLURBOKGGQ, seg_pos={0 0}, upload=274674]: Uploading piece... Aug 19 17:38:09.696191328 (264h12m4.127271459s) [noise=true , address=45.91.116.5:4302 , piece_id=D2ZYYIFLA7GFPEII4X75HVJGL5DOM6BZLQFXL7HYVXYLNCL4CDQQ, seg_pos={0 0}, upload=274674]: Uploading piece... Aug 19 17:38:09.696539624 (264h12m4.127619755s) [noise=true , address=77.90.31.14:4689 , piece_id=6FO4WN5BXNW6UFGW2ALS5YBYBP5FZCAZBLQZXAULFCN65NHH7N6A, seg_pos={0 0}, upload=274674]: Uploading piece... Aug 19 17:38:09.696809421 (264h12m4.127889552s) [noise=true , address=79.127.201.214:4299 , piece_id=C4JSI4OPEZPPHADT6QTVJGEXQNVSHEK42KNYQBV3S2OH7Z5Z7HHA, seg_pos={0 0}, upload=274674]: Uploading piece... Aug 19 17:38:09.697172445 (264h12m4.128252576s) [noise=true , address=104.36.82.43:4855 , piece_id=3H2PUTGFVC2PR224NBHGRB7GLWBOL2MNXXWJ7JF23HYVF53TEWFQ, seg_pos={0 0}, upload=274674]: Uploading piece... Aug 19 17:38:09.697485064 (264h12m4.128565205s) [noise=true , address=77.90.31.12:4619 , piece_id=XMWKT6JBVLOPTRUIPNEK4GXVCH5OOVO5WYWDGWMXGA2VVVD74CBA, seg_pos={0 0}, upload=274674]: Uploading piece... Aug 19 17:38:09.697774858 (264h12m4.128854989s) [noise=true , address=104.36.82.38:4714 , piece_id=SZ42LIQE4LXTBCZY4BTXWS3EL27UYIQS3AAXGWNLH65TUZWNUYSQ, seg_pos={0 0}, upload=274674]: Uploading piece... Aug 19 17:38:09.698096103 (264h12m4.129176234s) [noise=true , address=104.36.82.4:4817 , piece_id=JNKPBDLUZUWI22ZLGTCD3BZJBAUX7TZVAGXJVFZOEJPK2Z3PMADA, seg_pos={0 0}, upload=274674]: Uploading piece... Aug 19 17:38:09.698352945 (264h12m4.129433076s) [noise=true , address=45.91.116.13:4183 , piece_id=OMUE6JFSTRIF64VYM5XCHNL7U26SEXP3IGWDBHP5G2E6YJQETEUQ, seg_pos={0 0}, upload=274674]: Uploading piece... Aug 19 17:38:09.698876106 (264h12m4.129956227s) [noise=true , address=104.36.82.6:4879 , piece_id=JKK5HMPYTMR7PNWHVPBDHPH3K6H7VM7AA4K4X4WXQ5LN57VRBUFA, seg_pos={0 0}, upload=274674]: Uploading piece... Aug 19 17:38:11.192631202 (264h12m5.623711333s) [upload=274674]: Next returned nil segment Aug 19 17:38:11.192648606 (264h12m5.623728737s) [upload=274674]: Waiting for error group managing segments... Aug 19 17:38:11.591537039 (264h12m6.02261717s) [noise=true , address=45.91.116.14:4222 , piece_id=DXHEK43V7USJLYKPF5PKEHLJPVP4OPH2H3TQVGUBMK5MVRWQHXJQ, seg_pos={0 0}, upload=274674]: Done uploading piece. err: <nil> Aug 19 17:38:11.606093581 (264h12m6.037173712s) [noise=true , address=77.90.31.14:4689 , piece_id=6FO4WN5BXNW6UFGW2ALS5YBYBP5FZCAZBLQZXAULFCN65NHH7N6A, seg_pos={0 0}, upload=274674]: Done uploading piece. err: <nil> Aug 19 17:38:11.610339916 (264h12m6.041420047s) [noise=true , address=77.90.31.15:4701 , piece_id=CCTT764Z3HFZFT4PE7YOOBVVGDASLFYVFGVXGWADBLDLURBOKGGQ, seg_pos={0 0}, upload=274674]: Done uploading piece. err: <nil> Aug 19 17:38:11.630055040 (264h12m6.061135171s) [noise=true , address=79.127.201.215:4343 , piece_id=MIATMAVPWKBJPVYH3XDYXQGKW7VWKRULLP6UV2ZNKJZ7YDVJJEBQ, seg_pos={0 0}, upload=274674]: Done uploading piece. err: <nil> Aug 19 17:38:11.678703675 (264h12m6.109783797s) [noise=true , address=104.36.82.9:4028 , piece_id=4LUYCAISMQC3SGDLJZSDFDJET4R54U3N4B6ZAJ35UDXB5CT3GV2Q, seg_pos={0 0}, upload=274674]: Done uploading piece. err: <nil> Aug 19 17:38:11.693543009 (264h12m6.12462314s) [noise=true , address=104.36.82.60:4364 , piece_id=A5BGK2CC62VYRH27SR2MONG2GPSZ5WONCHAF42LFPXOTM2W7XIJA, seg_pos={0 0}, upload=274674]: Done uploading piece. err: <nil> Aug 19 17:38:11.693568448 (264h12m6.124648579s) [noise=true , address=45.91.116.13:4183 , piece_id=OMUE6JFSTRIF64VYM5XCHNL7U26SEXP3IGWDBHP5G2E6YJQETEUQ, seg_pos={0 0}, upload=274674]: Done uploading piece. err: <nil> Aug 19 17:38:11.698831207 (264h12m6.129911329s) [noise=true , address=104.36.82.5:4852 , piece_id=UALXJPWV34HHWPC6GNCOFNP2RDC3KF6ZY4FUHLL3MWR2UZDCXNCQ, seg_pos={0 0}, upload=274674]: Done uploading piece. err: <nil> Aug 19 17:38:11.698886834 (264h12m6.129966955s) [noise=true , address=77.90.31.12:4619 , piece_id=XMWKT6JBVLOPTRUIPNEK4GXVCH5OOVO5WYWDGWMXGA2VVVD74CBA, seg_pos={0 0}, upload=274674]: Done uploading piece. err: <nil> Aug 19 17:38:11.714704781 (264h12m6.145784912s) [noise=true , address=79.127.201.214:4299 , piece_id=C4JSI4OPEZPPHADT6QTVJGEXQNVSHEK42KNYQBV3S2OH7Z5Z7HHA, seg_pos={0 0}, upload=274674]: Done uploading piece. err: <nil> Aug 19 17:38:11.720319795 (264h12m6.151399956s) [noise=true , address=45.91.116.11:4140 , piece_id=KQNAZLV4XPDYYLJQUMI2UL3GVLA7QTEZ6TU3HDYFYB7LPRBSRQFA, seg_pos={0 0}, upload=274674]: Done uploading piece. err: <nil> Aug 19 17:38:11.722063401 (264h12m6.153143532s) [noise=true , address=104.36.82.6:4879 , piece_id=JKK5HMPYTMR7PNWHVPBDHPH3K6H7VM7AA4K4X4WXQ5LN57VRBUFA, seg_pos={0 0}, upload=274674]: Done uploading piece. err: <nil> Aug 19 17:38:11.726762272 (264h12m6.157842393s) [noise=true , address=104.36.82.38:4714 , piece_id=SZ42LIQE4LXTBCZY4BTXWS3EL27UYIQS3AAXGWNLH65TUZWNUYSQ, seg_pos={0 0}, upload=274674]: Done uploading piece. err: <nil> Aug 19 17:38:11.734258877 (264h12m6.165339008s) [noise=true , address=104.36.82.58:4269 , piece_id=CTQSY5PQFUNXUJR2JXSI7O2EHMYTOAHUSHGTMVTI42E45RYFIGJA, seg_pos={0 0}, upload=274674]: Done uploading piece. err: <nil> Aug 19 17:38:11.734284346 (264h12m6.165364477s) [seg_pos={0 0}, upload=274674]: Segment reached optimal threshold of 14 pieces. Aug 19 17:38:11.734416277 (264h12m6.165496408s) [noise=true , address=104.36.82.4:4817 , piece_id=JNKPBDLUZUWI22ZLGTCD3BZJBAUX7TZVAGXJVFZOEJPK2Z3PMADA, seg_pos={0 0}, upload=274674]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RSBodzJxMye65QgkyrzkWCKQ49cnfdu4haUZKW23nWpqzMcYz): protocol: expected piece hash; context canceled; EOF Aug 19 17:38:11.734486873 (264h12m6.165567004s) [noise=true , address=104.36.82.43:4855 , piece_id=3H2PUTGFVC2PR224NBHGRB7GLWBOL2MNXXWJ7JF23HYVF53TEWFQ, seg_pos={0 0}, upload=274674]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TGTRhEi2KsbrkC6ZmgGkMaJri9uhzEcmXe1J4nJjpEd1BE4ka): protocol: expected piece hash; context canceled; EOF Aug 19 17:38:11.734587926 (264h12m6.165668057s) [noise=true , address=77.90.31.18:4825 , piece_id=UQLJVRVUBBN5LFBIAQL7YQ4IRAWI3SBVBHUT4IIFBROAUHXHVZ3A, seg_pos={0 0}, upload=274674]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PqKyfFxfVKTGARbPqBcQL3Naqjdf5uR44PeLi3sJpu57dcm1x): protocol: expected piece hash; context canceled; EOF Aug 19 17:38:11.734592956 (264h12m6.165673087s) [noise=true , address=45.91.116.5:4302 , piece_id=D2ZYYIFLA7GFPEII4X75HVJGL5DOM6BZLQFXL7HYVXYLNCL4CDQQ, seg_pos={0 0}, upload=274674]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pJWhSLza8WaMuvvDwH4nae57UxQmfCdDvAkQ3v25MgPYHoZXx): protocol: expected piece hash; context canceled; EOF Aug 19 17:38:11.734645206 (264h12m6.165725337s) [seg_pos={0 0}, upload=274674]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 19 17:38:11.734756098 (264h12m6.165836239s) [upload=274674]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 19 17:38:11.734764033 (264h12m6.165844154s) [upload=274674]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 17:38:12.023665554 (264h12m6.454745685s) [upload=274674]: Done uploading segments... Aug 19 17:38:12.023686044 (264h12m6.454766195s) [upload=274674]: Upload finished. err: <nil> Duration: 5.937219261s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 5.711236856s FAIL (2025-08-19 17:54:21 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Got next segment. Inline: false Aug 19 17:23:57.290467622 (263h57m51.721547763s) [upload=274428]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 17:23:57.290471560 (263h57m51.721551681s) [upload=274428]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 19 17:23:57.743765742 (263h57m52.174845863s) [seg_pos={0 0}, upload=274428]: Begin upload segment... Aug 19 17:23:57.743818412 (263h57m52.174898543s) [seg_pos={0 0}, upload=274428]: Done begin upload segment. Aug 19 17:23:57.743842609 (263h57m52.17492274s) [noise=true , address=77.90.31.16:4750 , piece_id=XXJ4VHMZ4NVVT2WX322JS7MW3FHQ7UZJW4VLR4KG6FW4A4RHVN6A, seg_pos={0 0}, upload=274428]: Uploading piece... Aug 19 17:23:57.743875672 (263h57m52.174955803s) [noise=true , address=45.91.116.14:4198 , piece_id=NSLSBMDS2ZJKN62N4NGN7HRHXAUIS4WL4AGHODZZBD2CDYTXHVUA, seg_pos={0 0}, upload=274428]: Uploading piece... Aug 19 17:23:57.744218498 (263h57m52.175298629s) [noise=true , address=77.90.31.14:4667 , piece_id=BF53QUGTGVD23KCIIAA5NWUCWKWUXXSHKCAAWSDZLAORPHXQEMQQ, seg_pos={0 0}, upload=274428]: Uploading piece... Aug 19 17:23:57.744456373 (263h57m52.175536504s) [noise=true , address=77.90.31.21:4883 , piece_id=SVFJCG4T5OCDMVKBE7A33XPDIQUH6NESE4GIVTETRZRBX4N3MJDQ, seg_pos={0 0}, upload=274428]: Uploading piece... Aug 19 17:23:57.744523051 (263h57m52.175603182s) [noise=true , address=104.36.82.60:4365 , piece_id=7BXP5MWOH2B3H74CPVWOQN2NEGMXUYECIVOEHRRKINQ544WKAAHA, seg_pos={0 0}, upload=274428]: Uploading piece... Aug 19 17:23:57.744762870 (263h57m52.175843001s) [noise=true , address=77.90.31.19:4851 , piece_id=POYQI453UA5767REISB5BNE3MYV6IOYZNIK357PYGH5DM66EOBIA, seg_pos={0 0}, upload=274428]: Uploading piece... Aug 19 17:23:57.744993531 (263h57m52.176073662s) [noise=true , address=104.36.82.44:4871 , piece_id=WQGOUIWA7NVK5SUTIWX6E32ZKW62Y2KGK7E3HB5YW2CKDUR42BPQ, seg_pos={0 0}, upload=274428]: Uploading piece... Aug 19 17:23:57.745229373 (263h57m52.176309494s) [noise=true , address=79.127.205.234:4587 , piece_id=ALBAMUJVGA4RIJKGCJNJT3BT6RBD5QWU4C33RJGCEP3XIQQLW6QQ, seg_pos={0 0}, upload=274428]: Uploading piece... Aug 19 17:23:57.745473950 (263h57m52.176554081s) [noise=true , address=104.36.82.3:4771 , piece_id=2RZZG2MORQZR3ROAADIWSK46JDEZSNYBGASCDWZTKWWU72JFHRSQ, seg_pos={0 0}, upload=274428]: Uploading piece... Aug 19 17:23:57.745746030 (263h57m52.176826161s) [noise=true , address=104.36.82.34:4558 , piece_id=AJNHQTDQSMJ3R2ZKVQUCBU5XOMWNF66ID2M3RVPYDL52F5LGTPFQ, seg_pos={0 0}, upload=274428]: Uploading piece... Aug 19 17:23:57.745999426 (263h57m52.177079547s) [noise=true , address=77.90.31.18:4814 , piece_id=VUIFMMFA5PCHIKNNX7PA7RHVXIJHK3UWH4SI34R6XP5MEYTAMS6A, seg_pos={0 0}, upload=274428]: Uploading piece... Aug 19 17:23:57.746774839 (263h57m52.17785497s) [noise=true , address=104.36.82.35:4610 , piece_id=DM7HADQAK5T67ZDQR3VGUUGLHCGBLC5KJYYFF7QQNRKVE2SA5CHA, seg_pos={0 0}, upload=274428]: Uploading piece... Aug 19 17:23:57.747503714 (263h57m52.178583836s) [noise=true , address=64.49.13.137:4496 , piece_id=DW5YMZIQJZIA53K22HGK2ICRT5IVVP2ZMIVYM46BWQWAYPLNUC2Q, seg_pos={0 0}, upload=274428]: Uploading piece... Aug 19 17:23:57.747727743 (263h57m52.178807874s) [noise=true , address=77.90.31.13:4640 , piece_id=NKLBPZJDAIAVAYH3QRDCPS6VWHJBFZZWY3HCTSW2E4II77YC3BYA, seg_pos={0 0}, upload=274428]: Uploading piece... Aug 19 17:23:57.747948565 (263h57m52.179028696s) [noise=true , address=104.36.82.55:4214 , piece_id=64P34K2YEWCWZFGWSR4AQ5SRALT5CSZCB4F6FHPKDP3ZFJIOJC2Q, seg_pos={0 0}, upload=274428]: Uploading piece... Aug 19 17:23:57.748174007 (263h57m52.179254138s) [noise=true , address=104.36.82.23:4448 , piece_id=SCMNBDCYA7PUN2VP6JN37YM5LNY2GBGQHJKYSSWUWMPTZZ7LXZUQ, seg_pos={0 0}, upload=274428]: Uploading piece... Aug 19 17:23:57.748433112 (263h57m52.179513233s) [noise=true , address=104.36.82.5:4854 , piece_id=FC4DIKMBI5OPLDQCVA6P3MA7AQZV23QBIX3UAKLWJJMY724TNYHQ, seg_pos={0 0}, upload=274428]: Uploading piece... Aug 19 17:23:57.748647252 (263h57m52.179727383s) [noise=true , address=104.36.82.62:4442 , piece_id=VV5BBYB6UI4O3B4ESOBJX44I5RJBRWSLJUFR6H3GKEUZWNONTPMA, seg_pos={0 0}, upload=274428]: Uploading piece... Aug 19 17:23:58.869276532 (263h57m53.300356663s) [upload=274428]: Next returned nil segment Aug 19 17:23:58.869291511 (263h57m53.300371642s) [upload=274428]: Waiting for error group managing segments... Aug 19 17:23:59.195579562 (263h57m53.626659683s) [noise=true , address=79.127.205.234:4587 , piece_id=ALBAMUJVGA4RIJKGCJNJT3BT6RBD5QWU4C33RJGCEP3XIQQLW6QQ, seg_pos={0 0}, upload=274428]: Done uploading piece. err: <nil> Aug 19 17:23:59.239543682 (263h57m53.670623813s) [noise=true , address=64.49.13.137:4496 , piece_id=DW5YMZIQJZIA53K22HGK2ICRT5IVVP2ZMIVYM46BWQWAYPLNUC2Q, seg_pos={0 0}, upload=274428]: Done uploading piece. err: <nil> Aug 19 17:23:59.654494638 (263h57m54.085574769s) [noise=true , address=104.36.82.35:4610 , piece_id=DM7HADQAK5T67ZDQR3VGUUGLHCGBLC5KJYYFF7QQNRKVE2SA5CHA, seg_pos={0 0}, upload=274428]: Done uploading piece. err: <nil> Aug 19 17:23:59.700507838 (263h57m54.131587959s) [noise=true , address=104.36.82.34:4558 , piece_id=AJNHQTDQSMJ3R2ZKVQUCBU5XOMWNF66ID2M3RVPYDL52F5LGTPFQ, seg_pos={0 0}, upload=274428]: Done uploading piece. err: <nil> Aug 19 17:23:59.757508140 (263h57m54.188588272s) [noise=true , address=104.36.82.5:4854 , piece_id=FC4DIKMBI5OPLDQCVA6P3MA7AQZV23QBIX3UAKLWJJMY724TNYHQ, seg_pos={0 0}, upload=274428]: Done uploading piece. err: <nil> Aug 19 17:23:59.765247029 (263h57m54.19632716s) [noise=true , address=104.36.82.62:4442 , piece_id=VV5BBYB6UI4O3B4ESOBJX44I5RJBRWSLJUFR6H3GKEUZWNONTPMA, seg_pos={0 0}, upload=274428]: Done uploading piece. err: <nil> Aug 19 17:23:59.774367630 (263h57m54.205447761s) [noise=true , address=104.36.82.23:4448 , piece_id=SCMNBDCYA7PUN2VP6JN37YM5LNY2GBGQHJKYSSWUWMPTZZ7LXZUQ, seg_pos={0 0}, upload=274428]: Done uploading piece. err: <nil> Aug 19 17:23:59.827970201 (263h57m54.259050332s) [noise=true , address=104.36.82.55:4214 , piece_id=64P34K2YEWCWZFGWSR4AQ5SRALT5CSZCB4F6FHPKDP3ZFJIOJC2Q, seg_pos={0 0}, upload=274428]: Done uploading piece. err: <nil> Aug 19 17:23:59.851412395 (263h57m54.282492526s) [noise=true , address=104.36.82.44:4871 , piece_id=WQGOUIWA7NVK5SUTIWX6E32ZKW62Y2KGK7E3HB5YW2CKDUR42BPQ, seg_pos={0 0}, upload=274428]: Done uploading piece. err: <nil> Aug 19 17:23:59.946562012 (263h57m54.377642143s) [noise=true , address=104.36.82.3:4771 , piece_id=2RZZG2MORQZR3ROAADIWSK46JDEZSNYBGASCDWZTKWWU72JFHRSQ, seg_pos={0 0}, upload=274428]: Done uploading piece. err: <nil> Aug 19 17:23:59.979671513 (263h57m54.410751634s) [noise=true , address=104.36.82.60:4365 , piece_id=7BXP5MWOH2B3H74CPVWOQN2NEGMXUYECIVOEHRRKINQ544WKAAHA, seg_pos={0 0}, upload=274428]: Done uploading piece. err: <nil> Aug 19 17:24:55.494274881 (263h58m49.925355012s) [noise=true , address=77.90.31.18:4814 , piece_id=VUIFMMFA5PCHIKNNX7PA7RHVXIJHK3UWH4SI34R6XP5MEYTAMS6A, seg_pos={0 0}, upload=274428]: Done uploading piece. err: <nil> Aug 19 17:24:55.842835997 (263h58m50.273916128s) [noise=true , address=77.90.31.21:4883 , piece_id=SVFJCG4T5OCDMVKBE7A33XPDIQUH6NESE4GIVTETRZRBX4N3MJDQ, seg_pos={0 0}, upload=274428]: Done uploading piece. err: <nil> Aug 19 17:24:56.634889530 (263h58m51.065969661s) [noise=true , address=77.90.31.19:4851 , piece_id=POYQI453UA5767REISB5BNE3MYV6IOYZNIK357PYGH5DM66EOBIA, seg_pos={0 0}, upload=274428]: Done uploading piece. err: ecclient: upload failed (node:1NEV2ndHw4fohVUNsfrzwojd8S11TmMST6Sz59GVnXS8vDGhWk, address:77.90.31.19:4851): protocol: rpc: dial tcp 77.90.31.19:4851: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:56.635097748 (263h58m51.066177879s) [noise=true , address=45.91.116.14:4198 , piece_id=NSLSBMDS2ZJKN62N4NGN7HRHXAUIS4WL4AGHODZZBD2CDYTXHVUA, seg_pos={0 0}, upload=274428]: Done uploading piece. err: ecclient: upload failed (node:1xxfUrCm5PgSLpge1wYiH2EcE7kzzJ38P9rfeCYmqXTbTisioi, address:45.91.116.14:4198): protocol: rpc: dial tcp 45.91.116.14:4198: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:56.635158895 (263h58m51.066239036s) [noise=true , address=77.90.31.16:4750 , piece_id=XXJ4VHMZ4NVVT2WX322JS7MW3FHQ7UZJW4VLR4KG6FW4A4RHVN6A, seg_pos={0 0}, upload=274428]: Done uploading piece. err: ecclient: upload failed (node:12TGd8NnWdekDtHzBQHmJPgTtHjXC6g8fi6RoZYFQJVHDa5hdyE, address:77.90.31.16:4750): protocol: read tcp 188.166.225.170:54198->77.90.31.16:4750: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:56.635224120 (263h58m51.066304281s) [noise=true , address=77.90.31.14:4667 , piece_id=BF53QUGTGVD23KCIIAA5NWUCWKWUXXSHKCAAWSDZLAORPHXQEMQQ, seg_pos={0 0}, upload=274428]: Done uploading piece. err: ecclient: upload failed (node:12R5gJFhsmocYPNZf4yNnYqJwcwAKt9yiHmmqGMk3btzVjkFV4S, address:77.90.31.14:4667): protocol: read tcp 188.166.225.170:46984->77.90.31.14:4667: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:56.635280057 (263h58m51.066360198s) [noise=true , address=77.90.31.13:4640 , piece_id=NKLBPZJDAIAVAYH3QRDCPS6VWHJBFZZWY3HCTSW2E4II77YC3BYA, seg_pos={0 0}, upload=274428]: Done uploading piece. err: ecclient: upload failed (node:1XjJpLBEZm5daHcBUmUyC7QpLVhu9Vm9ex9RYcYJWEuLGk527F, address:77.90.31.13:4640): protocol: read tcp 188.166.225.170:48656->77.90.31.13:4640: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:56.635314653 (263h58m51.066394794s) [seg_pos={0 0}, upload=274428]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Aug 19 17:24:56.862515362 (263h58m51.293595493s) [upload=274428]: Done uploading segments... Aug 19 17:24:56.862536042 (263h58m51.293616163s) [upload=274428]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.711316458s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 1.979919281s FAIL (2025-08-19 17:54:12 UTC) orbiter warning: operation: ListObjects ======ERROR====== uplink: metainfo: metaclient: context deadline exceeded ======LOGS====== Aug 19 17:35:36.607320794 (264h9m31.038400925s) [upload=274632]: Starting upload Aug 19 17:35:36.607367684 (264h9m31.038447815s) [upload=274632]: Done starting upload Aug 19 17:35:36.607435082 (264h9m31.038515213s) [upload=274632]: Uploading segments... Aug 19 17:35:36.607490538 (264h9m31.038570669s) [upload=274632]: Got next segment. Inline: true Aug 19 17:35:36.607495088 (264h9m31.038575219s) [upload=274632]: Waiting for error group managing segments... Aug 19 17:35:36.607537528 (264h9m31.038617659s) [upload=274632]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 17:35:36.607541656 (264h9m31.038621787s) [upload=274632]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 19 17:35:36.607544051 (264h9m31.038624182s) [upload=274632]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 17:35:36.889055260 (264h9m31.320135391s) [upload=274632]: Done uploading segments... Aug 19 17:35:36.889082943 (264h9m31.320163124s) [upload=274632]: Upload finished. err: <nil> Duration:* 1.979950409s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 6 out of 60 is failed (threshold=3) [.............X......X.X........X....XX......................] duration: 5.904727373s FAIL (2025-08-19 17:53:53 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/4M") ======LOGS====== Aug 19 17:38:09.246414094 (264h12m3.677494255s) [upload=274674]: Starting upload Aug 19 17:38:09.246486392 (264h12m3.677566533s) [upload=274674]: Done starting upload Aug 19 17:38:09.246601452 (264h12m3.677681583s) [upload=274674]: Uploading segments... Aug 19 17:38:09.246691885 (264h12m3.677772016s) [upload=274674]: Got next segment. Inline: false Aug 19 17:38:09.246724627 (264h12m3.677804758s) [upload=274674]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 17:38:09.246730909 (264h12m3.67781104s) [upload=274674]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 19 17:38:09.693242556 (264h12m4.124322687s) [seg_pos={0 0}, upload=274674]: Begin upload segment... Aug 19 17:38:09.693302230 (264h12m4.124382361s) [seg_pos={0 0}, upload=274674]: Done begin upload segment. Aug 19 17:38:09.693343529 (264h12m4.12442367s) [noise=true , address=77.90.31.18:4825 , piece_id=UQLJVRVUBBN5LFBIAQL7YQ4IRAWI3SBVBHUT4IIFBROAUHXHVZ3A, seg_pos={0 0}, upload=274674]: Uploading piece... Aug 19 17:38:09.693350092 (264h12m4.124430213s) [noise=true , address=45.91.116.14:4222 , piece_id=DXHEK43V7USJLYKPF5PKEHLJPVP4OPH2H3TQVGUBMK5MVRWQHXJQ, seg_pos={0 0}, upload=274674]: Uploading piece... Aug 19 17:38:09.693395940 (264h12m4.124476081s) [noise=true , address=45.91.116.11:4140 , piece_id=KQNAZLV4XPDYYLJQUMI2UL3GVLA7QTEZ6TU3HDYFYB7LPRBSRQFA, seg_pos={0 0}, upload=274674]: Uploading piece... Aug 19 17:38:09.693960980 (264h12m4.125041131s) [noise=true , address=79.127.201.215:4343 , piece_id=MIATMAVPWKBJPVYH3XDYXQGKW7VWKRULLP6UV2ZNKJZ7YDVJJEBQ, seg_pos={0 0}, upload=274674]: Uploading piece... Aug 19 17:38:09.694358652 (264h12m4.125438783s) [noise=true , address=104.36.82.5:4852 , piece_id=UALXJPWV34HHWPC6GNCOFNP2RDC3KF6ZY4FUHLL3MWR2UZDCXNCQ, seg_pos={0 0}, upload=274674]: Uploading piece... Aug 19 17:38:09.694689475 (264h12m4.125769606s) [noise=true , address=104.36.82.58:4269 , piece_id=CTQSY5PQFUNXUJR2JXSI7O2EHMYTOAHUSHGTMVTI42E45RYFIGJA, seg_pos={0 0}, upload=274674]: Uploading piece... Aug 19 17:38:09.695011601 (264h12m4.126091732s) [noise=true , address=104.36.82.60:4364 , piece_id=A5BGK2CC62VYRH27SR2MONG2GPSZ5WONCHAF42LFPXOTM2W7XIJA, seg_pos={0 0}, upload=274674]: Uploading piece... Aug 19 17:38:09.695441633 (264h12m4.126521764s) [noise=true , address=104.36.82.9:4028 , piece_id=4LUYCAISMQC3SGDLJZSDFDJET4R54U3N4B6ZAJ35UDXB5CT3GV2Q, seg_pos={0 0}, upload=274674]: Uploading piece... Aug 19 17:38:09.695845597 (264h12m4.126925728s) [noise=true , address=77.90.31.15:4701 , piece_id=CCTT764Z3HFZFT4PE7YOOBVVGDASLFYVFGVXGWADBLDLURBOKGGQ, seg_pos={0 0}, upload=274674]: Uploading piece... Aug 19 17:38:09.696191328 (264h12m4.127271459s) [noise=true , address=45.91.116.5:4302 , piece_id=D2ZYYIFLA7GFPEII4X75HVJGL5DOM6BZLQFXL7HYVXYLNCL4CDQQ, seg_pos={0 0}, upload=274674]: Uploading piece... Aug 19 17:38:09.696539624 (264h12m4.127619755s) [noise=true , address=77.90.31.14:4689 , piece_id=6FO4WN5BXNW6UFGW2ALS5YBYBP5FZCAZBLQZXAULFCN65NHH7N6A, seg_pos={0 0}, upload=274674]: Uploading piece... Aug 19 17:38:09.696809421 (264h12m4.127889552s) [noise=true , address=79.127.201.214:4299 , piece_id=C4JSI4OPEZPPHADT6QTVJGEXQNVSHEK42KNYQBV3S2OH7Z5Z7HHA, seg_pos={0 0}, upload=274674]: Uploading piece... Aug 19 17:38:09.697172445 (264h12m4.128252576s) [noise=true , address=104.36.82.43:4855 , piece_id=3H2PUTGFVC2PR224NBHGRB7GLWBOL2MNXXWJ7JF23HYVF53TEWFQ, seg_pos={0 0}, upload=274674]: Uploading piece... Aug 19 17:38:09.697485064 (264h12m4.128565205s) [noise=true , address=77.90.31.12:4619 , piece_id=XMWKT6JBVLOPTRUIPNEK4GXVCH5OOVO5WYWDGWMXGA2VVVD74CBA, seg_pos={0 0}, upload=274674]: Uploading piece... Aug 19 17:38:09.697774858 (264h12m4.128854989s) [noise=true , address=104.36.82.38:4714 , piece_id=SZ42LIQE4LXTBCZY4BTXWS3EL27UYIQS3AAXGWNLH65TUZWNUYSQ, seg_pos={0 0}, upload=274674]: Uploading piece... Aug 19 17:38:09.698096103 (264h12m4.129176234s) [noise=true , address=104.36.82.4:4817 , piece_id=JNKPBDLUZUWI22ZLGTCD3BZJBAUX7TZVAGXJVFZOEJPK2Z3PMADA, seg_pos={0 0}, upload=274674]: Uploading piece... Aug 19 17:38:09.698352945 (264h12m4.129433076s) [noise=true , address=45.91.116.13:4183 , piece_id=OMUE6JFSTRIF64VYM5XCHNL7U26SEXP3IGWDBHP5G2E6YJQETEUQ, seg_pos={0 0}, upload=274674]: Uploading piece... Aug 19 17:38:09.698876106 (264h12m4.129956227s) [noise=true , address=104.36.82.6:4879 , piece_id=JKK5HMPYTMR7PNWHVPBDHPH3K6H7VM7AA4K4X4WXQ5LN57VRBUFA, seg_pos={0 0}, upload=274674]: Uploading piece... Aug 19 17:38:11.192631202 (264h12m5.623711333s) [upload=274674]: Next returned nil segment Aug 19 17:38:11.192648606 (264h12m5.623728737s) [upload=274674]: Waiting for error group managing segments... Aug 19 17:38:11.591537039 (264h12m6.02261717s) [noise=true , address=45.91.116.14:4222 , piece_id=DXHEK43V7USJLYKPF5PKEHLJPVP4OPH2H3TQVGUBMK5MVRWQHXJQ, seg_pos={0 0}, upload=274674]: Done uploading piece. err: <nil> Aug 19 17:38:11.606093581 (264h12m6.037173712s) [noise=true , address=77.90.31.14:4689 , piece_id=6FO4WN5BXNW6UFGW2ALS5YBYBP5FZCAZBLQZXAULFCN65NHH7N6A, seg_pos={0 0}, upload=274674]: Done uploading piece. err: <nil> Aug 19 17:38:11.610339916 (264h12m6.041420047s) [noise=true , address=77.90.31.15:4701 , piece_id=CCTT764Z3HFZFT4PE7YOOBVVGDASLFYVFGVXGWADBLDLURBOKGGQ, seg_pos={0 0}, upload=274674]: Done uploading piece. err: <nil> Aug 19 17:38:11.630055040 (264h12m6.061135171s) [noise=true , address=79.127.201.215:4343 , piece_id=MIATMAVPWKBJPVYH3XDYXQGKW7VWKRULLP6UV2ZNKJZ7YDVJJEBQ, seg_pos={0 0}, upload=274674]: Done uploading piece. err: <nil> Aug 19 17:38:11.678703675 (264h12m6.109783797s) [noise=true , address=104.36.82.9:4028 , piece_id=4LUYCAISMQC3SGDLJZSDFDJET4R54U3N4B6ZAJ35UDXB5CT3GV2Q, seg_pos={0 0}, upload=274674]: Done uploading piece. err: <nil> Aug 19 17:38:11.693543009 (264h12m6.12462314s) [noise=true , address=104.36.82.60:4364 , piece_id=A5BGK2CC62VYRH27SR2MONG2GPSZ5WONCHAF42LFPXOTM2W7XIJA, seg_pos={0 0}, upload=274674]: Done uploading piece. err: <nil> Aug 19 17:38:11.693568448 (264h12m6.124648579s) [noise=true , address=45.91.116.13:4183 , piece_id=OMUE6JFSTRIF64VYM5XCHNL7U26SEXP3IGWDBHP5G2E6YJQETEUQ, seg_pos={0 0}, upload=274674]: Done uploading piece. err: <nil> Aug 19 17:38:11.698831207 (264h12m6.129911329s) [noise=true , address=104.36.82.5:4852 , piece_id=UALXJPWV34HHWPC6GNCOFNP2RDC3KF6ZY4FUHLL3MWR2UZDCXNCQ, seg_pos={0 0}, upload=274674]: Done uploading piece. err: <nil> Aug 19 17:38:11.698886834 (264h12m6.129966955s) [noise=true , address=77.90.31.12:4619 , piece_id=XMWKT6JBVLOPTRUIPNEK4GXVCH5OOVO5WYWDGWMXGA2VVVD74CBA, seg_pos={0 0}, upload=274674]: Done uploading piece. err: <nil> Aug 19 17:38:11.714704781 (264h12m6.145784912s) [noise=true , address=79.127.201.214:4299 , piece_id=C4JSI4OPEZPPHADT6QTVJGEXQNVSHEK42KNYQBV3S2OH7Z5Z7HHA, seg_pos={0 0}, upload=274674]: Done uploading piece. err: <nil> Aug 19 17:38:11.720319795 (264h12m6.151399956s) [noise=true , address=45.91.116.11:4140 , piece_id=KQNAZLV4XPDYYLJQUMI2UL3GVLA7QTEZ6TU3HDYFYB7LPRBSRQFA, seg_pos={0 0}, upload=274674]: Done uploading piece. err: <nil> Aug 19 17:38:11.722063401 (264h12m6.153143532s) [noise=true , address=104.36.82.6:4879 , piece_id=JKK5HMPYTMR7PNWHVPBDHPH3K6H7VM7AA4K4X4WXQ5LN57VRBUFA, seg_pos={0 0}, upload=274674]: Done uploading piece. err: <nil> Aug 19 17:38:11.726762272 (264h12m6.157842393s) [noise=true , address=104.36.82.38:4714 , piece_id=SZ42LIQE4LXTBCZY4BTXWS3EL27UYIQS3AAXGWNLH65TUZWNUYSQ, seg_pos={0 0}, upload=274674]: Done uploading piece. err: <nil> Aug 19 17:38:11.734258877 (264h12m6.165339008s) [noise=true , address=104.36.82.58:4269 , piece_id=CTQSY5PQFUNXUJR2JXSI7O2EHMYTOAHUSHGTMVTI42E45RYFIGJA, seg_pos={0 0}, upload=274674]: Done uploading piece. err: <nil> Aug 19 17:38:11.734284346 (264h12m6.165364477s) [seg_pos={0 0}, upload=274674]: Segment reached optimal threshold of 14 pieces. Aug 19 17:38:11.734416277 (264h12m6.165496408s) [noise=true , address=104.36.82.4:4817 , piece_id=JNKPBDLUZUWI22ZLGTCD3BZJBAUX7TZVAGXJVFZOEJPK2Z3PMADA, seg_pos={0 0}, upload=274674]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RSBodzJxMye65QgkyrzkWCKQ49cnfdu4haUZKW23nWpqzMcYz): protocol: expected piece hash; context canceled; EOF Aug 19 17:38:11.734486873 (264h12m6.165567004s) [noise=true , address=104.36.82.43:4855 , piece_id=3H2PUTGFVC2PR224NBHGRB7GLWBOL2MNXXWJ7JF23HYVF53TEWFQ, seg_pos={0 0}, upload=274674]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TGTRhEi2KsbrkC6ZmgGkMaJri9uhzEcmXe1J4nJjpEd1BE4ka): protocol: expected piece hash; context canceled; EOF Aug 19 17:38:11.734587926 (264h12m6.165668057s) [noise=true , address=77.90.31.18:4825 , piece_id=UQLJVRVUBBN5LFBIAQL7YQ4IRAWI3SBVBHUT4IIFBROAUHXHVZ3A, seg_pos={0 0}, upload=274674]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PqKyfFxfVKTGARbPqBcQL3Naqjdf5uR44PeLi3sJpu57dcm1x): protocol: expected piece hash; context canceled; EOF Aug 19 17:38:11.734592956 (264h12m6.165673087s) [noise=true , address=45.91.116.5:4302 , piece_id=D2ZYYIFLA7GFPEII4X75HVJGL5DOM6BZLQFXL7HYVXYLNCL4CDQQ, seg_pos={0 0}, upload=274674]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pJWhSLza8WaMuvvDwH4nae57UxQmfCdDvAkQ3v25MgPYHoZXx): protocol: expected piece hash; context canceled; EOF Aug 19 17:38:11.734645206 (264h12m6.165725337s) [seg_pos={0 0}, upload=274674]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 19 17:38:11.734756098 (264h12m6.165836239s) [upload=274674]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 19 17:38:11.734764033 (264h12m6.165844154s) [upload=274674]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 17:38:12.023665554 (264h12m6.454745685s) [upload=274674]: Done uploading segments... Aug 19 17:38:12.023686044 (264h12m6.454766195s) [upload=274674]: Upload finished. err: <nil> Duration: 5.904803119s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..............X...X.........................................] duration: 5.745166025s FAIL (2025-08-19 17:53:25 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 23:59.057943072 (263h57m53.489023273s) [noise=true , address=45.91.116.10:4083 , piece_id=V3H4T2WN2MBVALSIHZNOQHBE75WQRNYLIE3Z7DFWUUHBPYDT7IFA, seg_pos={0 0}, upload=274429]: Uploading piece... Aug 19 17:23:59.057955436 (263h57m53.489035557s) [noise=true , address=45.91.116.5:4298 , piece_id=HMF6DXO5KFMLK67WVA7BAWLQFLC2BCVPJ7ZT255FMRCCEI3MCBNA, seg_pos={0 0}, upload=274429]: Uploading piece... Aug 19 17:23:59.058557689 (263h57m53.48963782s) [noise=true , address=104.36.82.8:4969 , piece_id=3P655Z5OUVZ4VRESACEBKYR3KUFYY56NYPWX7ZVOHL3XI2YQ5TCQ, seg_pos={0 0}, upload=274429]: Uploading piece... Aug 19 17:23:59.059170771 (263h57m53.490250902s) [noise=true , address=45.91.116.7:4009 , piece_id=NSZYKL6FHDLDM463ZXKEQGX4ZY7JPJXENH63PPDV76XMGCBV7ZRQ, seg_pos={0 0}, upload=274429]: Uploading piece... Aug 19 17:23:59.059468281 (263h57m53.490548412s) [noise=true , address=77.90.31.15:4699 , piece_id=MFF7PNMDLHYEH7PY46MGGOJSY4XPHRDPGFWJABBO4JDTJJJJ3CTQ, seg_pos={0 0}, upload=274429]: Uploading piece... Aug 19 17:23:59.059498208 (263h57m53.490578339s) [noise=true , address=104.36.82.2:4729 , piece_id=BNX6UNYZ7VHQ4MKAH65DGSHULWRRXDSCMKOE7WFZO6T2V3ELBJ7Q, seg_pos={0 0}, upload=274429]: Uploading piece... Aug 19 17:23:59.059814182 (263h57m53.490894323s) [noise=true , address=104.36.82.6:4874 , piece_id=QMGNWSGMEV762Y2XEDK2LIAY3JMNLD3UOVFQ6HYP7ZMLLVAVO7ZQ, seg_pos={0 0}, upload=274429]: Uploading piece... Aug 19 17:23:59.059839941 (263h57m53.490920072s) [noise=true , address=45.91.116.12:4225 , piece_id=PCJYR2VLFH23UPQEABUYESR3MHKDHIVTJWDIWVYIL6JAFGGXXXFQ, seg_pos={0 0}, upload=274429]: Uploading piece... Aug 19 17:23:59.059875480 (263h57m53.490955611s) [noise=true , address=104.36.82.10:4079 , piece_id=S3BSVJWRPP2PDGLG7FKFROUKAM3GTBDL2ALAJPAZQQQZJVRMMZ3Q, seg_pos={0 0}, upload=274429]: Uploading piece... Aug 19 17:23:59.060109147 (263h57m53.491189278s) [noise=true , address=104.36.82.2:4732 , piece_id=NLTK4572JAQHZQRDD2DOOIW7VBO7NK4DI4RUZRSXQ3GR72EWV3KQ, seg_pos={0 0}, upload=274429]: Uploading piece... Aug 19 17:23:59.060142190 (263h57m53.491222321s) [noise=true , address=45.91.116.9:4261 , piece_id=J5IMFL3JW4YKDLI6OEY2POEB4LJR2P3BIFAD4YIF3E4JTUXIXY3A, seg_pos={0 0}, upload=274429]: Uploading piece... Aug 19 17:23:59.060348274 (263h57m53.491428405s) [noise=true , address=104.36.82.15:4259 , piece_id=RM7JL7JKCMFM4CNXVVZQ2QJTOEPQEQHGLW3GYXWUD6LAQ2ZJ3P5Q, seg_pos={0 0}, upload=274429]: Uploading piece... Aug 19 17:23:59.060544830 (263h57m53.491624961s) [noise=true , address=45.91.116.14:4191 , piece_id=PJ33GP36S76VCI5RVR26PBFZ6HZUUZYVWNGPJITRLGDZI6LWCJIQ, seg_pos={0 0}, upload=274429]: Uploading piece... Aug 19 17:23:59.060752227 (263h57m53.491832358s) [noise=true , address=45.91.116.11:4118 , piece_id=VD2HK2QU63UVJFRASQOIRFZRWZN2HAUSAZJL2VZKYALGSV67SP2A, seg_pos={0 0}, upload=274429]: Uploading piece... Aug 19 17:23:59.060967990 (263h57m53.492048121s) [noise=true , address=77.90.31.11:4555 , piece_id=GTU6PGCHI4WOFGK2B3FP2Y22VUDBWLQKKEMZBT6I4F47OKK4RPNQ, seg_pos={0 0}, upload=274429]: Uploading piece... Aug 19 17:23:59.061294044 (263h57m53.492374175s) [noise=true , address=45.91.116.8:4048 , piece_id=M2CBILQKSVSQ7VY4AWQQM5GHF2KYBSXH73YJGAEJZAHOR7FOBWEQ, seg_pos={0 0}, upload=274429]: Uploading piece... Aug 19 17:24:00.872920400 (263h57m55.304000531s) [upload=274429]: Next returned nil segment Aug 19 17:24:00.872945548 (263h57m55.304025679s) [upload=274429]: Waiting for error group managing segments... Aug 19 17:24:01.156214888 (263h57m55.587295019s) [noise=true , address=104.36.82.6:4874 , piece_id=QMGNWSGMEV762Y2XEDK2LIAY3JMNLD3UOVFQ6HYP7ZMLLVAVO7ZQ, seg_pos={0 0}, upload=274429]: Done uploading piece. err: <nil> Aug 19 17:24:01.181164926 (263h57m55.612245057s) [noise=true , address=104.36.82.2:4732 , piece_id=NLTK4572JAQHZQRDD2DOOIW7VBO7NK4DI4RUZRSXQ3GR72EWV3KQ, seg_pos={0 0}, upload=274429]: Done uploading piece. err: <nil> Aug 19 17:24:01.222047199 (263h57m55.653127331s) [noise=true , address=104.36.82.2:4729 , piece_id=BNX6UNYZ7VHQ4MKAH65DGSHULWRRXDSCMKOE7WFZO6T2V3ELBJ7Q, seg_pos={0 0}, upload=274429]: Done uploading piece. err: <nil> Aug 19 17:24:01.250473409 (263h57m55.68155354s) [noise=true , address=104.36.82.15:4259 , piece_id=RM7JL7JKCMFM4CNXVVZQ2QJTOEPQEQHGLW3GYXWUD6LAQ2ZJ3P5Q, seg_pos={0 0}, upload=274429]: Done uploading piece. err: <nil> Aug 19 17:24:01.259222299 (263h57m55.69030242s) [noise=true , address=104.36.82.8:4969 , piece_id=3P655Z5OUVZ4VRESACEBKYR3KUFYY56NYPWX7ZVOHL3XI2YQ5TCQ, seg_pos={0 0}, upload=274429]: Done uploading piece. err: <nil> Aug 19 17:24:01.261735818 (263h57m55.692815949s) [noise=true , address=104.36.82.10:4079 , piece_id=S3BSVJWRPP2PDGLG7FKFROUKAM3GTBDL2ALAJPAZQQQZJVRMMZ3Q, seg_pos={0 0}, upload=274429]: Done uploading piece. err: <nil> Aug 19 17:24:28.183964515 (263h58m22.615044646s) [noise=true , address=45.91.116.14:4191 , piece_id=PJ33GP36S76VCI5RVR26PBFZ6HZUUZYVWNGPJITRLGDZI6LWCJIQ, seg_pos={0 0}, upload=274429]: Done uploading piece. err: ecclient: upload failed (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, address:45.91.116.14:4191): protocol: read tcp 188.166.225.170:37646->45.91.116.14:4191: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:28.184013127 (263h58m22.615093258s) [noise=true , address=45.91.116.9:4261 , piece_id=J5IMFL3JW4YKDLI6OEY2POEB4LJR2P3BIFAD4YIF3E4JTUXIXY3A, seg_pos={0 0}, upload=274429]: Done uploading piece. err: ecclient: upload failed (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, address:45.91.116.9:4261): protocol: rpc: dial tcp 45.91.116.9:4261: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:28.184026694 (263h58m22.615106825s) [noise=true , address=45.91.116.8:4048 , piece_id=M2CBILQKSVSQ7VY4AWQQM5GHF2KYBSXH73YJGAEJZAHOR7FOBWEQ, seg_pos={0 0}, upload=274429]: Done uploading piece. err: ecclient: upload failed (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, address:45.91.116.8:4048): protocol: read tcp 188.166.225.170:45604->45.91.116.8:4048: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:28.184070568 (263h58m22.615150689s) [noise=true , address=45.91.116.5:4298 , piece_id=HMF6DXO5KFMLK67WVA7BAWLQFLC2BCVPJ7ZT255FMRCCEI3MCBNA, seg_pos={0 0}, upload=274429]: Done uploading piece. err: ecclient: upload failed (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, address:45.91.116.5:4298): protocol: read tcp 188.166.225.170:46970->45.91.116.5:4298: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:28.184103010 (263h58m22.615183141s) [noise=true , address=45.91.116.10:4083 , piece_id=V3H4T2WN2MBVALSIHZNOQHBE75WQRNYLIE3Z7DFWUUHBPYDT7IFA, seg_pos={0 0}, upload=274429]: Done uploading piece. err: ecclient: upload failed (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, address:45.91.116.10:4083): protocol: read tcp 188.166.225.170:41632->45.91.116.10:4083: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:28.184139489 (263h58m22.61521962s) [noise=true , address=45.91.116.7:4009 , piece_id=NSZYKL6FHDLDM463ZXKEQGX4ZY7JPJXENH63PPDV76XMGCBV7ZRQ, seg_pos={0 0}, upload=274429]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: read tcp 188.166.225.170:45756->45.91.116.7:4009: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:28.184171009 (263h58m22.615251141s) [noise=true , address=77.90.31.11:4555 , piece_id=GTU6PGCHI4WOFGK2B3FP2Y22VUDBWLQKKEMZBT6I4F47OKK4RPNQ, seg_pos={0 0}, upload=274429]: Done uploading piece. err: ecclient: upload failed (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, address:77.90.31.11:4555): protocol: read tcp 188.166.225.170:37438->77.90.31.11:4555: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:28.184197480 (263h58m22.615277611s) [noise=true , address=77.90.31.15:4699 , piece_id=MFF7PNMDLHYEH7PY46MGGOJSY4XPHRDPGFWJABBO4JDTJJJJ3CTQ, seg_pos={0 0}, upload=274429]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: read tcp 188.166.225.170:36788->77.90.31.15:4699: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:28.184218410 (263h58m22.615298541s) [noise=true , address=45.91.116.12:4225 , piece_id=PCJYR2VLFH23UPQEABUYESR3MHKDHIVTJWDIWVYIL6JAFGGXXXFQ, seg_pos={0 0}, upload=274429]: Done uploading piece. err: ecclient: upload failed (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, address:45.91.116.12:4225): protocol: read tcp 188.166.225.170:60636->45.91.116.12:4225: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:28.184249710 (263h58m22.615329841s) [noise=true , address=45.91.116.11:4118 , piece_id=VD2HK2QU63UVJFRASQOIRFZRWZN2HAUSAZJL2VZKYALGSV67SP2A, seg_pos={0 0}, upload=274429]: Done uploading piece. err: ecclient: upload failed (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, address:45.91.116.11:4118): protocol: read tcp 188.166.225.170:48006->45.91.116.11:4118: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:28.184274337 (263h58m22.615354468s) [seg_pos={0 0}, upload=274429]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Aug 19 17:24:28.412483516 (263h58m22.843563647s) [upload=274429]: Done uploading segments... Aug 19 17:24:28.412520628 (263h58m22.843600759s) [upload=274429]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 5.745193357s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 1.871159777s FAIL (2025-08-19 17:53:19 UTC) orbiter warning: operation: ListObjects ======ERROR====== uplink: metainfo: metaclient: context deadline exceeded ======LOGS====== Aug 19 17:35:36.607320794 (264h9m31.038400925s) [upload=274632]: Starting upload Aug 19 17:35:36.607367684 (264h9m31.038447815s) [upload=274632]: Done starting upload Aug 19 17:35:36.607435082 (264h9m31.038515213s) [upload=274632]: Uploading segments... Aug 19 17:35:36.607490538 (264h9m31.038570669s) [upload=274632]: Got next segment. Inline: true Aug 19 17:35:36.607495088 (264h9m31.038575219s) [upload=274632]: Waiting for error group managing segments... Aug 19 17:35:36.607537528 (264h9m31.038617659s) [upload=274632]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 17:35:36.607541656 (264h9m31.038621787s) [upload=274632]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 19 17:35:36.607544051 (264h9m31.038624182s) [upload=274632]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 17:35:36.889055260 (264h9m31.320135391s) [upload=274632]: Done uploading segments... Aug 19 17:35:36.889082943 (264h9m31.320163124s) [upload=274632]: Upload finished. err: <nil> Duration:* 1.871189675s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.............X.....X........................................] duration: 5.882660078s FAIL (2025-08-19 17:53:12 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.882705215s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 5.879593489s FAIL (2025-08-19 17:53:11 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Got next segment. Inline: false Aug 19 17:23:57.290467622 (263h57m51.721547763s) [upload=274428]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 17:23:57.290471560 (263h57m51.721551681s) [upload=274428]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 19 17:23:57.743765742 (263h57m52.174845863s) [seg_pos={0 0}, upload=274428]: Begin upload segment... Aug 19 17:23:57.743818412 (263h57m52.174898543s) [seg_pos={0 0}, upload=274428]: Done begin upload segment. Aug 19 17:23:57.743842609 (263h57m52.17492274s) [noise=true , address=77.90.31.16:4750 , piece_id=XXJ4VHMZ4NVVT2WX322JS7MW3FHQ7UZJW4VLR4KG6FW4A4RHVN6A, seg_pos={0 0}, upload=274428]: Uploading piece... Aug 19 17:23:57.743875672 (263h57m52.174955803s) [noise=true , address=45.91.116.14:4198 , piece_id=NSLSBMDS2ZJKN62N4NGN7HRHXAUIS4WL4AGHODZZBD2CDYTXHVUA, seg_pos={0 0}, upload=274428]: Uploading piece... Aug 19 17:23:57.744218498 (263h57m52.175298629s) [noise=true , address=77.90.31.14:4667 , piece_id=BF53QUGTGVD23KCIIAA5NWUCWKWUXXSHKCAAWSDZLAORPHXQEMQQ, seg_pos={0 0}, upload=274428]: Uploading piece... Aug 19 17:23:57.744456373 (263h57m52.175536504s) [noise=true , address=77.90.31.21:4883 , piece_id=SVFJCG4T5OCDMVKBE7A33XPDIQUH6NESE4GIVTETRZRBX4N3MJDQ, seg_pos={0 0}, upload=274428]: Uploading piece... Aug 19 17:23:57.744523051 (263h57m52.175603182s) [noise=true , address=104.36.82.60:4365 , piece_id=7BXP5MWOH2B3H74CPVWOQN2NEGMXUYECIVOEHRRKINQ544WKAAHA, seg_pos={0 0}, upload=274428]: Uploading piece... Aug 19 17:23:57.744762870 (263h57m52.175843001s) [noise=true , address=77.90.31.19:4851 , piece_id=POYQI453UA5767REISB5BNE3MYV6IOYZNIK357PYGH5DM66EOBIA, seg_pos={0 0}, upload=274428]: Uploading piece... Aug 19 17:23:57.744993531 (263h57m52.176073662s) [noise=true , address=104.36.82.44:4871 , piece_id=WQGOUIWA7NVK5SUTIWX6E32ZKW62Y2KGK7E3HB5YW2CKDUR42BPQ, seg_pos={0 0}, upload=274428]: Uploading piece... Aug 19 17:23:57.745229373 (263h57m52.176309494s) [noise=true , address=79.127.205.234:4587 , piece_id=ALBAMUJVGA4RIJKGCJNJT3BT6RBD5QWU4C33RJGCEP3XIQQLW6QQ, seg_pos={0 0}, upload=274428]: Uploading piece... Aug 19 17:23:57.745473950 (263h57m52.176554081s) [noise=true , address=104.36.82.3:4771 , piece_id=2RZZG2MORQZR3ROAADIWSK46JDEZSNYBGASCDWZTKWWU72JFHRSQ, seg_pos={0 0}, upload=274428]: Uploading piece... Aug 19 17:23:57.745746030 (263h57m52.176826161s) [noise=true , address=104.36.82.34:4558 , piece_id=AJNHQTDQSMJ3R2ZKVQUCBU5XOMWNF66ID2M3RVPYDL52F5LGTPFQ, seg_pos={0 0}, upload=274428]: Uploading piece... Aug 19 17:23:57.745999426 (263h57m52.177079547s) [noise=true , address=77.90.31.18:4814 , piece_id=VUIFMMFA5PCHIKNNX7PA7RHVXIJHK3UWH4SI34R6XP5MEYTAMS6A, seg_pos={0 0}, upload=274428]: Uploading piece... Aug 19 17:23:57.746774839 (263h57m52.17785497s) [noise=true , address=104.36.82.35:4610 , piece_id=DM7HADQAK5T67ZDQR3VGUUGLHCGBLC5KJYYFF7QQNRKVE2SA5CHA, seg_pos={0 0}, upload=274428]: Uploading piece... Aug 19 17:23:57.747503714 (263h57m52.178583836s) [noise=true , address=64.49.13.137:4496 , piece_id=DW5YMZIQJZIA53K22HGK2ICRT5IVVP2ZMIVYM46BWQWAYPLNUC2Q, seg_pos={0 0}, upload=274428]: Uploading piece... Aug 19 17:23:57.747727743 (263h57m52.178807874s) [noise=true , address=77.90.31.13:4640 , piece_id=NKLBPZJDAIAVAYH3QRDCPS6VWHJBFZZWY3HCTSW2E4II77YC3BYA, seg_pos={0 0}, upload=274428]: Uploading piece... Aug 19 17:23:57.747948565 (263h57m52.179028696s) [noise=true , address=104.36.82.55:4214 , piece_id=64P34K2YEWCWZFGWSR4AQ5SRALT5CSZCB4F6FHPKDP3ZFJIOJC2Q, seg_pos={0 0}, upload=274428]: Uploading piece... Aug 19 17:23:57.748174007 (263h57m52.179254138s) [noise=true , address=104.36.82.23:4448 , piece_id=SCMNBDCYA7PUN2VP6JN37YM5LNY2GBGQHJKYSSWUWMPTZZ7LXZUQ, seg_pos={0 0}, upload=274428]: Uploading piece... Aug 19 17:23:57.748433112 (263h57m52.179513233s) [noise=true , address=104.36.82.5:4854 , piece_id=FC4DIKMBI5OPLDQCVA6P3MA7AQZV23QBIX3UAKLWJJMY724TNYHQ, seg_pos={0 0}, upload=274428]: Uploading piece... Aug 19 17:23:57.748647252 (263h57m52.179727383s) [noise=true , address=104.36.82.62:4442 , piece_id=VV5BBYB6UI4O3B4ESOBJX44I5RJBRWSLJUFR6H3GKEUZWNONTPMA, seg_pos={0 0}, upload=274428]: Uploading piece... Aug 19 17:23:58.869276532 (263h57m53.300356663s) [upload=274428]: Next returned nil segment Aug 19 17:23:58.869291511 (263h57m53.300371642s) [upload=274428]: Waiting for error group managing segments... Aug 19 17:23:59.195579562 (263h57m53.626659683s) [noise=true , address=79.127.205.234:4587 , piece_id=ALBAMUJVGA4RIJKGCJNJT3BT6RBD5QWU4C33RJGCEP3XIQQLW6QQ, seg_pos={0 0}, upload=274428]: Done uploading piece. err: <nil> Aug 19 17:23:59.239543682 (263h57m53.670623813s) [noise=true , address=64.49.13.137:4496 , piece_id=DW5YMZIQJZIA53K22HGK2ICRT5IVVP2ZMIVYM46BWQWAYPLNUC2Q, seg_pos={0 0}, upload=274428]: Done uploading piece. err: <nil> Aug 19 17:23:59.654494638 (263h57m54.085574769s) [noise=true , address=104.36.82.35:4610 , piece_id=DM7HADQAK5T67ZDQR3VGUUGLHCGBLC5KJYYFF7QQNRKVE2SA5CHA, seg_pos={0 0}, upload=274428]: Done uploading piece. err: <nil> Aug 19 17:23:59.700507838 (263h57m54.131587959s) [noise=true , address=104.36.82.34:4558 , piece_id=AJNHQTDQSMJ3R2ZKVQUCBU5XOMWNF66ID2M3RVPYDL52F5LGTPFQ, seg_pos={0 0}, upload=274428]: Done uploading piece. err: <nil> Aug 19 17:23:59.757508140 (263h57m54.188588272s) [noise=true , address=104.36.82.5:4854 , piece_id=FC4DIKMBI5OPLDQCVA6P3MA7AQZV23QBIX3UAKLWJJMY724TNYHQ, seg_pos={0 0}, upload=274428]: Done uploading piece. err: <nil> Aug 19 17:23:59.765247029 (263h57m54.19632716s) [noise=true , address=104.36.82.62:4442 , piece_id=VV5BBYB6UI4O3B4ESOBJX44I5RJBRWSLJUFR6H3GKEUZWNONTPMA, seg_pos={0 0}, upload=274428]: Done uploading piece. err: <nil> Aug 19 17:23:59.774367630 (263h57m54.205447761s) [noise=true , address=104.36.82.23:4448 , piece_id=SCMNBDCYA7PUN2VP6JN37YM5LNY2GBGQHJKYSSWUWMPTZZ7LXZUQ, seg_pos={0 0}, upload=274428]: Done uploading piece. err: <nil> Aug 19 17:23:59.827970201 (263h57m54.259050332s) [noise=true , address=104.36.82.55:4214 , piece_id=64P34K2YEWCWZFGWSR4AQ5SRALT5CSZCB4F6FHPKDP3ZFJIOJC2Q, seg_pos={0 0}, upload=274428]: Done uploading piece. err: <nil> Aug 19 17:23:59.851412395 (263h57m54.282492526s) [noise=true , address=104.36.82.44:4871 , piece_id=WQGOUIWA7NVK5SUTIWX6E32ZKW62Y2KGK7E3HB5YW2CKDUR42BPQ, seg_pos={0 0}, upload=274428]: Done uploading piece. err: <nil> Aug 19 17:23:59.946562012 (263h57m54.377642143s) [noise=true , address=104.36.82.3:4771 , piece_id=2RZZG2MORQZR3ROAADIWSK46JDEZSNYBGASCDWZTKWWU72JFHRSQ, seg_pos={0 0}, upload=274428]: Done uploading piece. err: <nil> Aug 19 17:23:59.979671513 (263h57m54.410751634s) [noise=true , address=104.36.82.60:4365 , piece_id=7BXP5MWOH2B3H74CPVWOQN2NEGMXUYECIVOEHRRKINQ544WKAAHA, seg_pos={0 0}, upload=274428]: Done uploading piece. err: <nil> Aug 19 17:24:55.494274881 (263h58m49.925355012s) [noise=true , address=77.90.31.18:4814 , piece_id=VUIFMMFA5PCHIKNNX7PA7RHVXIJHK3UWH4SI34R6XP5MEYTAMS6A, seg_pos={0 0}, upload=274428]: Done uploading piece. err: <nil> Aug 19 17:24:55.842835997 (263h58m50.273916128s) [noise=true , address=77.90.31.21:4883 , piece_id=SVFJCG4T5OCDMVKBE7A33XPDIQUH6NESE4GIVTETRZRBX4N3MJDQ, seg_pos={0 0}, upload=274428]: Done uploading piece. err: <nil> Aug 19 17:24:56.634889530 (263h58m51.065969661s) [noise=true , address=77.90.31.19:4851 , piece_id=POYQI453UA5767REISB5BNE3MYV6IOYZNIK357PYGH5DM66EOBIA, seg_pos={0 0}, upload=274428]: Done uploading piece. err: ecclient: upload failed (node:1NEV2ndHw4fohVUNsfrzwojd8S11TmMST6Sz59GVnXS8vDGhWk, address:77.90.31.19:4851): protocol: rpc: dial tcp 77.90.31.19:4851: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:56.635097748 (263h58m51.066177879s) [noise=true , address=45.91.116.14:4198 , piece_id=NSLSBMDS2ZJKN62N4NGN7HRHXAUIS4WL4AGHODZZBD2CDYTXHVUA, seg_pos={0 0}, upload=274428]: Done uploading piece. err: ecclient: upload failed (node:1xxfUrCm5PgSLpge1wYiH2EcE7kzzJ38P9rfeCYmqXTbTisioi, address:45.91.116.14:4198): protocol: rpc: dial tcp 45.91.116.14:4198: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:56.635158895 (263h58m51.066239036s) [noise=true , address=77.90.31.16:4750 , piece_id=XXJ4VHMZ4NVVT2WX322JS7MW3FHQ7UZJW4VLR4KG6FW4A4RHVN6A, seg_pos={0 0}, upload=274428]: Done uploading piece. err: ecclient: upload failed (node:12TGd8NnWdekDtHzBQHmJPgTtHjXC6g8fi6RoZYFQJVHDa5hdyE, address:77.90.31.16:4750): protocol: read tcp 188.166.225.170:54198->77.90.31.16:4750: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:56.635224120 (263h58m51.066304281s) [noise=true , address=77.90.31.14:4667 , piece_id=BF53QUGTGVD23KCIIAA5NWUCWKWUXXSHKCAAWSDZLAORPHXQEMQQ, seg_pos={0 0}, upload=274428]: Done uploading piece. err: ecclient: upload failed (node:12R5gJFhsmocYPNZf4yNnYqJwcwAKt9yiHmmqGMk3btzVjkFV4S, address:77.90.31.14:4667): protocol: read tcp 188.166.225.170:46984->77.90.31.14:4667: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:56.635280057 (263h58m51.066360198s) [noise=true , address=77.90.31.13:4640 , piece_id=NKLBPZJDAIAVAYH3QRDCPS6VWHJBFZZWY3HCTSW2E4II77YC3BYA, seg_pos={0 0}, upload=274428]: Done uploading piece. err: ecclient: upload failed (node:1XjJpLBEZm5daHcBUmUyC7QpLVhu9Vm9ex9RYcYJWEuLGk527F, address:77.90.31.13:4640): protocol: read tcp 188.166.225.170:48656->77.90.31.13:4640: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:56.635314653 (263h58m51.066394794s) [seg_pos={0 0}, upload=274428]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Aug 19 17:24:56.862515362 (263h58m51.293595493s) [upload=274428]: Done uploading segments... Aug 19 17:24:56.862536042 (263h58m51.293616163s) [upload=274428]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.879620029s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...............X...X........................................] duration: 5.921694757s FAIL (2025-08-19 17:53:03 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 23:59.057943072 (263h57m53.489023273s) [noise=true , address=45.91.116.10:4083 , piece_id=V3H4T2WN2MBVALSIHZNOQHBE75WQRNYLIE3Z7DFWUUHBPYDT7IFA, seg_pos={0 0}, upload=274429]: Uploading piece... Aug 19 17:23:59.057955436 (263h57m53.489035557s) [noise=true , address=45.91.116.5:4298 , piece_id=HMF6DXO5KFMLK67WVA7BAWLQFLC2BCVPJ7ZT255FMRCCEI3MCBNA, seg_pos={0 0}, upload=274429]: Uploading piece... Aug 19 17:23:59.058557689 (263h57m53.48963782s) [noise=true , address=104.36.82.8:4969 , piece_id=3P655Z5OUVZ4VRESACEBKYR3KUFYY56NYPWX7ZVOHL3XI2YQ5TCQ, seg_pos={0 0}, upload=274429]: Uploading piece... Aug 19 17:23:59.059170771 (263h57m53.490250902s) [noise=true , address=45.91.116.7:4009 , piece_id=NSZYKL6FHDLDM463ZXKEQGX4ZY7JPJXENH63PPDV76XMGCBV7ZRQ, seg_pos={0 0}, upload=274429]: Uploading piece... Aug 19 17:23:59.059468281 (263h57m53.490548412s) [noise=true , address=77.90.31.15:4699 , piece_id=MFF7PNMDLHYEH7PY46MGGOJSY4XPHRDPGFWJABBO4JDTJJJJ3CTQ, seg_pos={0 0}, upload=274429]: Uploading piece... Aug 19 17:23:59.059498208 (263h57m53.490578339s) [noise=true , address=104.36.82.2:4729 , piece_id=BNX6UNYZ7VHQ4MKAH65DGSHULWRRXDSCMKOE7WFZO6T2V3ELBJ7Q, seg_pos={0 0}, upload=274429]: Uploading piece... Aug 19 17:23:59.059814182 (263h57m53.490894323s) [noise=true , address=104.36.82.6:4874 , piece_id=QMGNWSGMEV762Y2XEDK2LIAY3JMNLD3UOVFQ6HYP7ZMLLVAVO7ZQ, seg_pos={0 0}, upload=274429]: Uploading piece... Aug 19 17:23:59.059839941 (263h57m53.490920072s) [noise=true , address=45.91.116.12:4225 , piece_id=PCJYR2VLFH23UPQEABUYESR3MHKDHIVTJWDIWVYIL6JAFGGXXXFQ, seg_pos={0 0}, upload=274429]: Uploading piece... Aug 19 17:23:59.059875480 (263h57m53.490955611s) [noise=true , address=104.36.82.10:4079 , piece_id=S3BSVJWRPP2PDGLG7FKFROUKAM3GTBDL2ALAJPAZQQQZJVRMMZ3Q, seg_pos={0 0}, upload=274429]: Uploading piece... Aug 19 17:23:59.060109147 (263h57m53.491189278s) [noise=true , address=104.36.82.2:4732 , piece_id=NLTK4572JAQHZQRDD2DOOIW7VBO7NK4DI4RUZRSXQ3GR72EWV3KQ, seg_pos={0 0}, upload=274429]: Uploading piece... Aug 19 17:23:59.060142190 (263h57m53.491222321s) [noise=true , address=45.91.116.9:4261 , piece_id=J5IMFL3JW4YKDLI6OEY2POEB4LJR2P3BIFAD4YIF3E4JTUXIXY3A, seg_pos={0 0}, upload=274429]: Uploading piece... Aug 19 17:23:59.060348274 (263h57m53.491428405s) [noise=true , address=104.36.82.15:4259 , piece_id=RM7JL7JKCMFM4CNXVVZQ2QJTOEPQEQHGLW3GYXWUD6LAQ2ZJ3P5Q, seg_pos={0 0}, upload=274429]: Uploading piece... Aug 19 17:23:59.060544830 (263h57m53.491624961s) [noise=true , address=45.91.116.14:4191 , piece_id=PJ33GP36S76VCI5RVR26PBFZ6HZUUZYVWNGPJITRLGDZI6LWCJIQ, seg_pos={0 0}, upload=274429]: Uploading piece... Aug 19 17:23:59.060752227 (263h57m53.491832358s) [noise=true , address=45.91.116.11:4118 , piece_id=VD2HK2QU63UVJFRASQOIRFZRWZN2HAUSAZJL2VZKYALGSV67SP2A, seg_pos={0 0}, upload=274429]: Uploading piece... Aug 19 17:23:59.060967990 (263h57m53.492048121s) [noise=true , address=77.90.31.11:4555 , piece_id=GTU6PGCHI4WOFGK2B3FP2Y22VUDBWLQKKEMZBT6I4F47OKK4RPNQ, seg_pos={0 0}, upload=274429]: Uploading piece... Aug 19 17:23:59.061294044 (263h57m53.492374175s) [noise=true , address=45.91.116.8:4048 , piece_id=M2CBILQKSVSQ7VY4AWQQM5GHF2KYBSXH73YJGAEJZAHOR7FOBWEQ, seg_pos={0 0}, upload=274429]: Uploading piece... Aug 19 17:24:00.872920400 (263h57m55.304000531s) [upload=274429]: Next returned nil segment Aug 19 17:24:00.872945548 (263h57m55.304025679s) [upload=274429]: Waiting for error group managing segments... Aug 19 17:24:01.156214888 (263h57m55.587295019s) [noise=true , address=104.36.82.6:4874 , piece_id=QMGNWSGMEV762Y2XEDK2LIAY3JMNLD3UOVFQ6HYP7ZMLLVAVO7ZQ, seg_pos={0 0}, upload=274429]: Done uploading piece. err: <nil> Aug 19 17:24:01.181164926 (263h57m55.612245057s) [noise=true , address=104.36.82.2:4732 , piece_id=NLTK4572JAQHZQRDD2DOOIW7VBO7NK4DI4RUZRSXQ3GR72EWV3KQ, seg_pos={0 0}, upload=274429]: Done uploading piece. err: <nil> Aug 19 17:24:01.222047199 (263h57m55.653127331s) [noise=true , address=104.36.82.2:4729 , piece_id=BNX6UNYZ7VHQ4MKAH65DGSHULWRRXDSCMKOE7WFZO6T2V3ELBJ7Q, seg_pos={0 0}, upload=274429]: Done uploading piece. err: <nil> Aug 19 17:24:01.250473409 (263h57m55.68155354s) [noise=true , address=104.36.82.15:4259 , piece_id=RM7JL7JKCMFM4CNXVVZQ2QJTOEPQEQHGLW3GYXWUD6LAQ2ZJ3P5Q, seg_pos={0 0}, upload=274429]: Done uploading piece. err: <nil> Aug 19 17:24:01.259222299 (263h57m55.69030242s) [noise=true , address=104.36.82.8:4969 , piece_id=3P655Z5OUVZ4VRESACEBKYR3KUFYY56NYPWX7ZVOHL3XI2YQ5TCQ, seg_pos={0 0}, upload=274429]: Done uploading piece. err: <nil> Aug 19 17:24:01.261735818 (263h57m55.692815949s) [noise=true , address=104.36.82.10:4079 , piece_id=S3BSVJWRPP2PDGLG7FKFROUKAM3GTBDL2ALAJPAZQQQZJVRMMZ3Q, seg_pos={0 0}, upload=274429]: Done uploading piece. err: <nil> Aug 19 17:24:28.183964515 (263h58m22.615044646s) [noise=true , address=45.91.116.14:4191 , piece_id=PJ33GP36S76VCI5RVR26PBFZ6HZUUZYVWNGPJITRLGDZI6LWCJIQ, seg_pos={0 0}, upload=274429]: Done uploading piece. err: ecclient: upload failed (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, address:45.91.116.14:4191): protocol: read tcp 188.166.225.170:37646->45.91.116.14:4191: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:28.184013127 (263h58m22.615093258s) [noise=true , address=45.91.116.9:4261 , piece_id=J5IMFL3JW4YKDLI6OEY2POEB4LJR2P3BIFAD4YIF3E4JTUXIXY3A, seg_pos={0 0}, upload=274429]: Done uploading piece. err: ecclient: upload failed (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, address:45.91.116.9:4261): protocol: rpc: dial tcp 45.91.116.9:4261: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:28.184026694 (263h58m22.615106825s) [noise=true , address=45.91.116.8:4048 , piece_id=M2CBILQKSVSQ7VY4AWQQM5GHF2KYBSXH73YJGAEJZAHOR7FOBWEQ, seg_pos={0 0}, upload=274429]: Done uploading piece. err: ecclient: upload failed (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, address:45.91.116.8:4048): protocol: read tcp 188.166.225.170:45604->45.91.116.8:4048: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:28.184070568 (263h58m22.615150689s) [noise=true , address=45.91.116.5:4298 , piece_id=HMF6DXO5KFMLK67WVA7BAWLQFLC2BCVPJ7ZT255FMRCCEI3MCBNA, seg_pos={0 0}, upload=274429]: Done uploading piece. err: ecclient: upload failed (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, address:45.91.116.5:4298): protocol: read tcp 188.166.225.170:46970->45.91.116.5:4298: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:28.184103010 (263h58m22.615183141s) [noise=true , address=45.91.116.10:4083 , piece_id=V3H4T2WN2MBVALSIHZNOQHBE75WQRNYLIE3Z7DFWUUHBPYDT7IFA, seg_pos={0 0}, upload=274429]: Done uploading piece. err: ecclient: upload failed (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, address:45.91.116.10:4083): protocol: read tcp 188.166.225.170:41632->45.91.116.10:4083: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:28.184139489 (263h58m22.61521962s) [noise=true , address=45.91.116.7:4009 , piece_id=NSZYKL6FHDLDM463ZXKEQGX4ZY7JPJXENH63PPDV76XMGCBV7ZRQ, seg_pos={0 0}, upload=274429]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: read tcp 188.166.225.170:45756->45.91.116.7:4009: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:28.184171009 (263h58m22.615251141s) [noise=true , address=77.90.31.11:4555 , piece_id=GTU6PGCHI4WOFGK2B3FP2Y22VUDBWLQKKEMZBT6I4F47OKK4RPNQ, seg_pos={0 0}, upload=274429]: Done uploading piece. err: ecclient: upload failed (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, address:77.90.31.11:4555): protocol: read tcp 188.166.225.170:37438->77.90.31.11:4555: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:28.184197480 (263h58m22.615277611s) [noise=true , address=77.90.31.15:4699 , piece_id=MFF7PNMDLHYEH7PY46MGGOJSY4XPHRDPGFWJABBO4JDTJJJJ3CTQ, seg_pos={0 0}, upload=274429]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: read tcp 188.166.225.170:36788->77.90.31.15:4699: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:28.184218410 (263h58m22.615298541s) [noise=true , address=45.91.116.12:4225 , piece_id=PCJYR2VLFH23UPQEABUYESR3MHKDHIVTJWDIWVYIL6JAFGGXXXFQ, seg_pos={0 0}, upload=274429]: Done uploading piece. err: ecclient: upload failed (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, address:45.91.116.12:4225): protocol: read tcp 188.166.225.170:60636->45.91.116.12:4225: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:28.184249710 (263h58m22.615329841s) [noise=true , address=45.91.116.11:4118 , piece_id=VD2HK2QU63UVJFRASQOIRFZRWZN2HAUSAZJL2VZKYALGSV67SP2A, seg_pos={0 0}, upload=274429]: Done uploading piece. err: ecclient: upload failed (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, address:45.91.116.11:4118): protocol: read tcp 188.166.225.170:48006->45.91.116.11:4118: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:28.184274337 (263h58m22.615354468s) [seg_pos={0 0}, upload=274429]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Aug 19 17:24:28.412483516 (263h58m22.843563647s) [upload=274429]: Done uploading segments... Aug 19 17:24:28.412520628 (263h58m22.843600759s) [upload=274429]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 5.921716699s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 7 out of 60 is failed (threshold=3) [X.............X......X.X........X....XX.....................] duration: 5.657668776s FAIL (2025-08-19 17:52:59 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/4M") ======LOGS====== Aug 19 17:38:09.246414094 (264h12m3.677494255s) [upload=274674]: Starting upload Aug 19 17:38:09.246486392 (264h12m3.677566533s) [upload=274674]: Done starting upload Aug 19 17:38:09.246601452 (264h12m3.677681583s) [upload=274674]: Uploading segments... Aug 19 17:38:09.246691885 (264h12m3.677772016s) [upload=274674]: Got next segment. Inline: false Aug 19 17:38:09.246724627 (264h12m3.677804758s) [upload=274674]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 17:38:09.246730909 (264h12m3.67781104s) [upload=274674]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 19 17:38:09.693242556 (264h12m4.124322687s) [seg_pos={0 0}, upload=274674]: Begin upload segment... Aug 19 17:38:09.693302230 (264h12m4.124382361s) [seg_pos={0 0}, upload=274674]: Done begin upload segment. Aug 19 17:38:09.693343529 (264h12m4.12442367s) [noise=true , address=77.90.31.18:4825 , piece_id=UQLJVRVUBBN5LFBIAQL7YQ4IRAWI3SBVBHUT4IIFBROAUHXHVZ3A, seg_pos={0 0}, upload=274674]: Uploading piece... Aug 19 17:38:09.693350092 (264h12m4.124430213s) [noise=true , address=45.91.116.14:4222 , piece_id=DXHEK43V7USJLYKPF5PKEHLJPVP4OPH2H3TQVGUBMK5MVRWQHXJQ, seg_pos={0 0}, upload=274674]: Uploading piece... Aug 19 17:38:09.693395940 (264h12m4.124476081s) [noise=true , address=45.91.116.11:4140 , piece_id=KQNAZLV4XPDYYLJQUMI2UL3GVLA7QTEZ6TU3HDYFYB7LPRBSRQFA, seg_pos={0 0}, upload=274674]: Uploading piece... Aug 19 17:38:09.693960980 (264h12m4.125041131s) [noise=true , address=79.127.201.215:4343 , piece_id=MIATMAVPWKBJPVYH3XDYXQGKW7VWKRULLP6UV2ZNKJZ7YDVJJEBQ, seg_pos={0 0}, upload=274674]: Uploading piece... Aug 19 17:38:09.694358652 (264h12m4.125438783s) [noise=true , address=104.36.82.5:4852 , piece_id=UALXJPWV34HHWPC6GNCOFNP2RDC3KF6ZY4FUHLL3MWR2UZDCXNCQ, seg_pos={0 0}, upload=274674]: Uploading piece... Aug 19 17:38:09.694689475 (264h12m4.125769606s) [noise=true , address=104.36.82.58:4269 , piece_id=CTQSY5PQFUNXUJR2JXSI7O2EHMYTOAHUSHGTMVTI42E45RYFIGJA, seg_pos={0 0}, upload=274674]: Uploading piece... Aug 19 17:38:09.695011601 (264h12m4.126091732s) [noise=true , address=104.36.82.60:4364 , piece_id=A5BGK2CC62VYRH27SR2MONG2GPSZ5WONCHAF42LFPXOTM2W7XIJA, seg_pos={0 0}, upload=274674]: Uploading piece... Aug 19 17:38:09.695441633 (264h12m4.126521764s) [noise=true , address=104.36.82.9:4028 , piece_id=4LUYCAISMQC3SGDLJZSDFDJET4R54U3N4B6ZAJ35UDXB5CT3GV2Q, seg_pos={0 0}, upload=274674]: Uploading piece... Aug 19 17:38:09.695845597 (264h12m4.126925728s) [noise=true , address=77.90.31.15:4701 , piece_id=CCTT764Z3HFZFT4PE7YOOBVVGDASLFYVFGVXGWADBLDLURBOKGGQ, seg_pos={0 0}, upload=274674]: Uploading piece... Aug 19 17:38:09.696191328 (264h12m4.127271459s) [noise=true , address=45.91.116.5:4302 , piece_id=D2ZYYIFLA7GFPEII4X75HVJGL5DOM6BZLQFXL7HYVXYLNCL4CDQQ, seg_pos={0 0}, upload=274674]: Uploading piece... Aug 19 17:38:09.696539624 (264h12m4.127619755s) [noise=true , address=77.90.31.14:4689 , piece_id=6FO4WN5BXNW6UFGW2ALS5YBYBP5FZCAZBLQZXAULFCN65NHH7N6A, seg_pos={0 0}, upload=274674]: Uploading piece... Aug 19 17:38:09.696809421 (264h12m4.127889552s) [noise=true , address=79.127.201.214:4299 , piece_id=C4JSI4OPEZPPHADT6QTVJGEXQNVSHEK42KNYQBV3S2OH7Z5Z7HHA, seg_pos={0 0}, upload=274674]: Uploading piece... Aug 19 17:38:09.697172445 (264h12m4.128252576s) [noise=true , address=104.36.82.43:4855 , piece_id=3H2PUTGFVC2PR224NBHGRB7GLWBOL2MNXXWJ7JF23HYVF53TEWFQ, seg_pos={0 0}, upload=274674]: Uploading piece... Aug 19 17:38:09.697485064 (264h12m4.128565205s) [noise=true , address=77.90.31.12:4619 , piece_id=XMWKT6JBVLOPTRUIPNEK4GXVCH5OOVO5WYWDGWMXGA2VVVD74CBA, seg_pos={0 0}, upload=274674]: Uploading piece... Aug 19 17:38:09.697774858 (264h12m4.128854989s) [noise=true , address=104.36.82.38:4714 , piece_id=SZ42LIQE4LXTBCZY4BTXWS3EL27UYIQS3AAXGWNLH65TUZWNUYSQ, seg_pos={0 0}, upload=274674]: Uploading piece... Aug 19 17:38:09.698096103 (264h12m4.129176234s) [noise=true , address=104.36.82.4:4817 , piece_id=JNKPBDLUZUWI22ZLGTCD3BZJBAUX7TZVAGXJVFZOEJPK2Z3PMADA, seg_pos={0 0}, upload=274674]: Uploading piece... Aug 19 17:38:09.698352945 (264h12m4.129433076s) [noise=true , address=45.91.116.13:4183 , piece_id=OMUE6JFSTRIF64VYM5XCHNL7U26SEXP3IGWDBHP5G2E6YJQETEUQ, seg_pos={0 0}, upload=274674]: Uploading piece... Aug 19 17:38:09.698876106 (264h12m4.129956227s) [noise=true , address=104.36.82.6:4879 , piece_id=JKK5HMPYTMR7PNWHVPBDHPH3K6H7VM7AA4K4X4WXQ5LN57VRBUFA, seg_pos={0 0}, upload=274674]: Uploading piece... Aug 19 17:38:11.192631202 (264h12m5.623711333s) [upload=274674]: Next returned nil segment Aug 19 17:38:11.192648606 (264h12m5.623728737s) [upload=274674]: Waiting for error group managing segments... Aug 19 17:38:11.591537039 (264h12m6.02261717s) [noise=true , address=45.91.116.14:4222 , piece_id=DXHEK43V7USJLYKPF5PKEHLJPVP4OPH2H3TQVGUBMK5MVRWQHXJQ, seg_pos={0 0}, upload=274674]: Done uploading piece. err: <nil> Aug 19 17:38:11.606093581 (264h12m6.037173712s) [noise=true , address=77.90.31.14:4689 , piece_id=6FO4WN5BXNW6UFGW2ALS5YBYBP5FZCAZBLQZXAULFCN65NHH7N6A, seg_pos={0 0}, upload=274674]: Done uploading piece. err: <nil> Aug 19 17:38:11.610339916 (264h12m6.041420047s) [noise=true , address=77.90.31.15:4701 , piece_id=CCTT764Z3HFZFT4PE7YOOBVVGDASLFYVFGVXGWADBLDLURBOKGGQ, seg_pos={0 0}, upload=274674]: Done uploading piece. err: <nil> Aug 19 17:38:11.630055040 (264h12m6.061135171s) [noise=true , address=79.127.201.215:4343 , piece_id=MIATMAVPWKBJPVYH3XDYXQGKW7VWKRULLP6UV2ZNKJZ7YDVJJEBQ, seg_pos={0 0}, upload=274674]: Done uploading piece. err: <nil> Aug 19 17:38:11.678703675 (264h12m6.109783797s) [noise=true , address=104.36.82.9:4028 , piece_id=4LUYCAISMQC3SGDLJZSDFDJET4R54U3N4B6ZAJ35UDXB5CT3GV2Q, seg_pos={0 0}, upload=274674]: Done uploading piece. err: <nil> Aug 19 17:38:11.693543009 (264h12m6.12462314s) [noise=true , address=104.36.82.60:4364 , piece_id=A5BGK2CC62VYRH27SR2MONG2GPSZ5WONCHAF42LFPXOTM2W7XIJA, seg_pos={0 0}, upload=274674]: Done uploading piece. err: <nil> Aug 19 17:38:11.693568448 (264h12m6.124648579s) [noise=true , address=45.91.116.13:4183 , piece_id=OMUE6JFSTRIF64VYM5XCHNL7U26SEXP3IGWDBHP5G2E6YJQETEUQ, seg_pos={0 0}, upload=274674]: Done uploading piece. err: <nil> Aug 19 17:38:11.698831207 (264h12m6.129911329s) [noise=true , address=104.36.82.5:4852 , piece_id=UALXJPWV34HHWPC6GNCOFNP2RDC3KF6ZY4FUHLL3MWR2UZDCXNCQ, seg_pos={0 0}, upload=274674]: Done uploading piece. err: <nil> Aug 19 17:38:11.698886834 (264h12m6.129966955s) [noise=true , address=77.90.31.12:4619 , piece_id=XMWKT6JBVLOPTRUIPNEK4GXVCH5OOVO5WYWDGWMXGA2VVVD74CBA, seg_pos={0 0}, upload=274674]: Done uploading piece. err: <nil> Aug 19 17:38:11.714704781 (264h12m6.145784912s) [noise=true , address=79.127.201.214:4299 , piece_id=C4JSI4OPEZPPHADT6QTVJGEXQNVSHEK42KNYQBV3S2OH7Z5Z7HHA, seg_pos={0 0}, upload=274674]: Done uploading piece. err: <nil> Aug 19 17:38:11.720319795 (264h12m6.151399956s) [noise=true , address=45.91.116.11:4140 , piece_id=KQNAZLV4XPDYYLJQUMI2UL3GVLA7QTEZ6TU3HDYFYB7LPRBSRQFA, seg_pos={0 0}, upload=274674]: Done uploading piece. err: <nil> Aug 19 17:38:11.722063401 (264h12m6.153143532s) [noise=true , address=104.36.82.6:4879 , piece_id=JKK5HMPYTMR7PNWHVPBDHPH3K6H7VM7AA4K4X4WXQ5LN57VRBUFA, seg_pos={0 0}, upload=274674]: Done uploading piece. err: <nil> Aug 19 17:38:11.726762272 (264h12m6.157842393s) [noise=true , address=104.36.82.38:4714 , piece_id=SZ42LIQE4LXTBCZY4BTXWS3EL27UYIQS3AAXGWNLH65TUZWNUYSQ, seg_pos={0 0}, upload=274674]: Done uploading piece. err: <nil> Aug 19 17:38:11.734258877 (264h12m6.165339008s) [noise=true , address=104.36.82.58:4269 , piece_id=CTQSY5PQFUNXUJR2JXSI7O2EHMYTOAHUSHGTMVTI42E45RYFIGJA, seg_pos={0 0}, upload=274674]: Done uploading piece. err: <nil> Aug 19 17:38:11.734284346 (264h12m6.165364477s) [seg_pos={0 0}, upload=274674]: Segment reached optimal threshold of 14 pieces. Aug 19 17:38:11.734416277 (264h12m6.165496408s) [noise=true , address=104.36.82.4:4817 , piece_id=JNKPBDLUZUWI22ZLGTCD3BZJBAUX7TZVAGXJVFZOEJPK2Z3PMADA, seg_pos={0 0}, upload=274674]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RSBodzJxMye65QgkyrzkWCKQ49cnfdu4haUZKW23nWpqzMcYz): protocol: expected piece hash; context canceled; EOF Aug 19 17:38:11.734486873 (264h12m6.165567004s) [noise=true , address=104.36.82.43:4855 , piece_id=3H2PUTGFVC2PR224NBHGRB7GLWBOL2MNXXWJ7JF23HYVF53TEWFQ, seg_pos={0 0}, upload=274674]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TGTRhEi2KsbrkC6ZmgGkMaJri9uhzEcmXe1J4nJjpEd1BE4ka): protocol: expected piece hash; context canceled; EOF Aug 19 17:38:11.734587926 (264h12m6.165668057s) [noise=true , address=77.90.31.18:4825 , piece_id=UQLJVRVUBBN5LFBIAQL7YQ4IRAWI3SBVBHUT4IIFBROAUHXHVZ3A, seg_pos={0 0}, upload=274674]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PqKyfFxfVKTGARbPqBcQL3Naqjdf5uR44PeLi3sJpu57dcm1x): protocol: expected piece hash; context canceled; EOF Aug 19 17:38:11.734592956 (264h12m6.165673087s) [noise=true , address=45.91.116.5:4302 , piece_id=D2ZYYIFLA7GFPEII4X75HVJGL5DOM6BZLQFXL7HYVXYLNCL4CDQQ, seg_pos={0 0}, upload=274674]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pJWhSLza8WaMuvvDwH4nae57UxQmfCdDvAkQ3v25MgPYHoZXx): protocol: expected piece hash; context canceled; EOF Aug 19 17:38:11.734645206 (264h12m6.165725337s) [seg_pos={0 0}, upload=274674]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 19 17:38:11.734756098 (264h12m6.165836239s) [upload=274674]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 19 17:38:11.734764033 (264h12m6.165844154s) [upload=274674]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 17:38:12.023665554 (264h12m6.454745685s) [upload=274674]: Done uploading segments... Aug 19 17:38:12.023686044 (264h12m6.454766195s) [upload=274674]: Upload finished. err: <nil> Duration: 5.65771201s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 2.026258896s FAIL (2025-08-19 17:52:39 UTC) orbiter warning: operation: ListObjects ======ERROR====== uplink: metainfo: metaclient: context deadline exceeded ======LOGS====== Aug 19 17:35:36.607320794 (264h9m31.038400925s) [upload=274632]: Starting upload Aug 19 17:35:36.607367684 (264h9m31.038447815s) [upload=274632]: Done starting upload Aug 19 17:35:36.607435082 (264h9m31.038515213s) [upload=274632]: Uploading segments... Aug 19 17:35:36.607490538 (264h9m31.038570669s) [upload=274632]: Got next segment. Inline: true Aug 19 17:35:36.607495088 (264h9m31.038575219s) [upload=274632]: Waiting for error group managing segments... Aug 19 17:35:36.607537528 (264h9m31.038617659s) [upload=274632]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 17:35:36.607541656 (264h9m31.038621787s) [upload=274632]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 19 17:35:36.607544051 (264h9m31.038624182s) [upload=274632]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 17:35:36.889055260 (264h9m31.320135391s) [upload=274632]: Done uploading segments... Aug 19 17:35:36.889082943 (264h9m31.320163124s) [upload=274632]: Upload finished. err: <nil> Duration:* 2.026285529s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [................X...X.......................................] duration: 5.802429443s FAIL (2025-08-19 17:52:29 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 23:59.057943072 (263h57m53.489023273s) [noise=true , address=45.91.116.10:4083 , piece_id=V3H4T2WN2MBVALSIHZNOQHBE75WQRNYLIE3Z7DFWUUHBPYDT7IFA, seg_pos={0 0}, upload=274429]: Uploading piece... Aug 19 17:23:59.057955436 (263h57m53.489035557s) [noise=true , address=45.91.116.5:4298 , piece_id=HMF6DXO5KFMLK67WVA7BAWLQFLC2BCVPJ7ZT255FMRCCEI3MCBNA, seg_pos={0 0}, upload=274429]: Uploading piece... Aug 19 17:23:59.058557689 (263h57m53.48963782s) [noise=true , address=104.36.82.8:4969 , piece_id=3P655Z5OUVZ4VRESACEBKYR3KUFYY56NYPWX7ZVOHL3XI2YQ5TCQ, seg_pos={0 0}, upload=274429]: Uploading piece... Aug 19 17:23:59.059170771 (263h57m53.490250902s) [noise=true , address=45.91.116.7:4009 , piece_id=NSZYKL6FHDLDM463ZXKEQGX4ZY7JPJXENH63PPDV76XMGCBV7ZRQ, seg_pos={0 0}, upload=274429]: Uploading piece... Aug 19 17:23:59.059468281 (263h57m53.490548412s) [noise=true , address=77.90.31.15:4699 , piece_id=MFF7PNMDLHYEH7PY46MGGOJSY4XPHRDPGFWJABBO4JDTJJJJ3CTQ, seg_pos={0 0}, upload=274429]: Uploading piece... Aug 19 17:23:59.059498208 (263h57m53.490578339s) [noise=true , address=104.36.82.2:4729 , piece_id=BNX6UNYZ7VHQ4MKAH65DGSHULWRRXDSCMKOE7WFZO6T2V3ELBJ7Q, seg_pos={0 0}, upload=274429]: Uploading piece... Aug 19 17:23:59.059814182 (263h57m53.490894323s) [noise=true , address=104.36.82.6:4874 , piece_id=QMGNWSGMEV762Y2XEDK2LIAY3JMNLD3UOVFQ6HYP7ZMLLVAVO7ZQ, seg_pos={0 0}, upload=274429]: Uploading piece... Aug 19 17:23:59.059839941 (263h57m53.490920072s) [noise=true , address=45.91.116.12:4225 , piece_id=PCJYR2VLFH23UPQEABUYESR3MHKDHIVTJWDIWVYIL6JAFGGXXXFQ, seg_pos={0 0}, upload=274429]: Uploading piece... Aug 19 17:23:59.059875480 (263h57m53.490955611s) [noise=true , address=104.36.82.10:4079 , piece_id=S3BSVJWRPP2PDGLG7FKFROUKAM3GTBDL2ALAJPAZQQQZJVRMMZ3Q, seg_pos={0 0}, upload=274429]: Uploading piece... Aug 19 17:23:59.060109147 (263h57m53.491189278s) [noise=true , address=104.36.82.2:4732 , piece_id=NLTK4572JAQHZQRDD2DOOIW7VBO7NK4DI4RUZRSXQ3GR72EWV3KQ, seg_pos={0 0}, upload=274429]: Uploading piece... Aug 19 17:23:59.060142190 (263h57m53.491222321s) [noise=true , address=45.91.116.9:4261 , piece_id=J5IMFL3JW4YKDLI6OEY2POEB4LJR2P3BIFAD4YIF3E4JTUXIXY3A, seg_pos={0 0}, upload=274429]: Uploading piece... Aug 19 17:23:59.060348274 (263h57m53.491428405s) [noise=true , address=104.36.82.15:4259 , piece_id=RM7JL7JKCMFM4CNXVVZQ2QJTOEPQEQHGLW3GYXWUD6LAQ2ZJ3P5Q, seg_pos={0 0}, upload=274429]: Uploading piece... Aug 19 17:23:59.060544830 (263h57m53.491624961s) [noise=true , address=45.91.116.14:4191 , piece_id=PJ33GP36S76VCI5RVR26PBFZ6HZUUZYVWNGPJITRLGDZI6LWCJIQ, seg_pos={0 0}, upload=274429]: Uploading piece... Aug 19 17:23:59.060752227 (263h57m53.491832358s) [noise=true , address=45.91.116.11:4118 , piece_id=VD2HK2QU63UVJFRASQOIRFZRWZN2HAUSAZJL2VZKYALGSV67SP2A, seg_pos={0 0}, upload=274429]: Uploading piece... Aug 19 17:23:59.060967990 (263h57m53.492048121s) [noise=true , address=77.90.31.11:4555 , piece_id=GTU6PGCHI4WOFGK2B3FP2Y22VUDBWLQKKEMZBT6I4F47OKK4RPNQ, seg_pos={0 0}, upload=274429]: Uploading piece... Aug 19 17:23:59.061294044 (263h57m53.492374175s) [noise=true , address=45.91.116.8:4048 , piece_id=M2CBILQKSVSQ7VY4AWQQM5GHF2KYBSXH73YJGAEJZAHOR7FOBWEQ, seg_pos={0 0}, upload=274429]: Uploading piece... Aug 19 17:24:00.872920400 (263h57m55.304000531s) [upload=274429]: Next returned nil segment Aug 19 17:24:00.872945548 (263h57m55.304025679s) [upload=274429]: Waiting for error group managing segments... Aug 19 17:24:01.156214888 (263h57m55.587295019s) [noise=true , address=104.36.82.6:4874 , piece_id=QMGNWSGMEV762Y2XEDK2LIAY3JMNLD3UOVFQ6HYP7ZMLLVAVO7ZQ, seg_pos={0 0}, upload=274429]: Done uploading piece. err: <nil> Aug 19 17:24:01.181164926 (263h57m55.612245057s) [noise=true , address=104.36.82.2:4732 , piece_id=NLTK4572JAQHZQRDD2DOOIW7VBO7NK4DI4RUZRSXQ3GR72EWV3KQ, seg_pos={0 0}, upload=274429]: Done uploading piece. err: <nil> Aug 19 17:24:01.222047199 (263h57m55.653127331s) [noise=true , address=104.36.82.2:4729 , piece_id=BNX6UNYZ7VHQ4MKAH65DGSHULWRRXDSCMKOE7WFZO6T2V3ELBJ7Q, seg_pos={0 0}, upload=274429]: Done uploading piece. err: <nil> Aug 19 17:24:01.250473409 (263h57m55.68155354s) [noise=true , address=104.36.82.15:4259 , piece_id=RM7JL7JKCMFM4CNXVVZQ2QJTOEPQEQHGLW3GYXWUD6LAQ2ZJ3P5Q, seg_pos={0 0}, upload=274429]: Done uploading piece. err: <nil> Aug 19 17:24:01.259222299 (263h57m55.69030242s) [noise=true , address=104.36.82.8:4969 , piece_id=3P655Z5OUVZ4VRESACEBKYR3KUFYY56NYPWX7ZVOHL3XI2YQ5TCQ, seg_pos={0 0}, upload=274429]: Done uploading piece. err: <nil> Aug 19 17:24:01.261735818 (263h57m55.692815949s) [noise=true , address=104.36.82.10:4079 , piece_id=S3BSVJWRPP2PDGLG7FKFROUKAM3GTBDL2ALAJPAZQQQZJVRMMZ3Q, seg_pos={0 0}, upload=274429]: Done uploading piece. err: <nil> Aug 19 17:24:28.183964515 (263h58m22.615044646s) [noise=true , address=45.91.116.14:4191 , piece_id=PJ33GP36S76VCI5RVR26PBFZ6HZUUZYVWNGPJITRLGDZI6LWCJIQ, seg_pos={0 0}, upload=274429]: Done uploading piece. err: ecclient: upload failed (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, address:45.91.116.14:4191): protocol: read tcp 188.166.225.170:37646->45.91.116.14:4191: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:28.184013127 (263h58m22.615093258s) [noise=true , address=45.91.116.9:4261 , piece_id=J5IMFL3JW4YKDLI6OEY2POEB4LJR2P3BIFAD4YIF3E4JTUXIXY3A, seg_pos={0 0}, upload=274429]: Done uploading piece. err: ecclient: upload failed (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, address:45.91.116.9:4261): protocol: rpc: dial tcp 45.91.116.9:4261: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:28.184026694 (263h58m22.615106825s) [noise=true , address=45.91.116.8:4048 , piece_id=M2CBILQKSVSQ7VY4AWQQM5GHF2KYBSXH73YJGAEJZAHOR7FOBWEQ, seg_pos={0 0}, upload=274429]: Done uploading piece. err: ecclient: upload failed (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, address:45.91.116.8:4048): protocol: read tcp 188.166.225.170:45604->45.91.116.8:4048: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:28.184070568 (263h58m22.615150689s) [noise=true , address=45.91.116.5:4298 , piece_id=HMF6DXO5KFMLK67WVA7BAWLQFLC2BCVPJ7ZT255FMRCCEI3MCBNA, seg_pos={0 0}, upload=274429]: Done uploading piece. err: ecclient: upload failed (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, address:45.91.116.5:4298): protocol: read tcp 188.166.225.170:46970->45.91.116.5:4298: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:28.184103010 (263h58m22.615183141s) [noise=true , address=45.91.116.10:4083 , piece_id=V3H4T2WN2MBVALSIHZNOQHBE75WQRNYLIE3Z7DFWUUHBPYDT7IFA, seg_pos={0 0}, upload=274429]: Done uploading piece. err: ecclient: upload failed (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, address:45.91.116.10:4083): protocol: read tcp 188.166.225.170:41632->45.91.116.10:4083: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:28.184139489 (263h58m22.61521962s) [noise=true , address=45.91.116.7:4009 , piece_id=NSZYKL6FHDLDM463ZXKEQGX4ZY7JPJXENH63PPDV76XMGCBV7ZRQ, seg_pos={0 0}, upload=274429]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: read tcp 188.166.225.170:45756->45.91.116.7:4009: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:28.184171009 (263h58m22.615251141s) [noise=true , address=77.90.31.11:4555 , piece_id=GTU6PGCHI4WOFGK2B3FP2Y22VUDBWLQKKEMZBT6I4F47OKK4RPNQ, seg_pos={0 0}, upload=274429]: Done uploading piece. err: ecclient: upload failed (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, address:77.90.31.11:4555): protocol: read tcp 188.166.225.170:37438->77.90.31.11:4555: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:28.184197480 (263h58m22.615277611s) [noise=true , address=77.90.31.15:4699 , piece_id=MFF7PNMDLHYEH7PY46MGGOJSY4XPHRDPGFWJABBO4JDTJJJJ3CTQ, seg_pos={0 0}, upload=274429]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: read tcp 188.166.225.170:36788->77.90.31.15:4699: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:28.184218410 (263h58m22.615298541s) [noise=true , address=45.91.116.12:4225 , piece_id=PCJYR2VLFH23UPQEABUYESR3MHKDHIVTJWDIWVYIL6JAFGGXXXFQ, seg_pos={0 0}, upload=274429]: Done uploading piece. err: ecclient: upload failed (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, address:45.91.116.12:4225): protocol: read tcp 188.166.225.170:60636->45.91.116.12:4225: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:28.184249710 (263h58m22.615329841s) [noise=true , address=45.91.116.11:4118 , piece_id=VD2HK2QU63UVJFRASQOIRFZRWZN2HAUSAZJL2VZKYALGSV67SP2A, seg_pos={0 0}, upload=274429]: Done uploading piece. err: ecclient: upload failed (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, address:45.91.116.11:4118): protocol: read tcp 188.166.225.170:48006->45.91.116.11:4118: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:28.184274337 (263h58m22.615354468s) [seg_pos={0 0}, upload=274429]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Aug 19 17:24:28.412483516 (263h58m22.843563647s) [upload=274429]: Done uploading segments... Aug 19 17:24:28.412520628 (263h58m22.843600759s) [upload=274429]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 5.802457908s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 7 out of 60 is failed (threshold=3) [.X.............X......X.X........X....XX....................] duration: 5.714482767s FAIL (2025-08-19 17:52:15 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/4M") ======LOGS====== Aug 19 17:38:09.246414094 (264h12m3.677494255s) [upload=274674]: Starting upload Aug 19 17:38:09.246486392 (264h12m3.677566533s) [upload=274674]: Done starting upload Aug 19 17:38:09.246601452 (264h12m3.677681583s) [upload=274674]: Uploading segments... Aug 19 17:38:09.246691885 (264h12m3.677772016s) [upload=274674]: Got next segment. Inline: false Aug 19 17:38:09.246724627 (264h12m3.677804758s) [upload=274674]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 17:38:09.246730909 (264h12m3.67781104s) [upload=274674]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 19 17:38:09.693242556 (264h12m4.124322687s) [seg_pos={0 0}, upload=274674]: Begin upload segment... Aug 19 17:38:09.693302230 (264h12m4.124382361s) [seg_pos={0 0}, upload=274674]: Done begin upload segment. Aug 19 17:38:09.693343529 (264h12m4.12442367s) [noise=true , address=77.90.31.18:4825 , piece_id=UQLJVRVUBBN5LFBIAQL7YQ4IRAWI3SBVBHUT4IIFBROAUHXHVZ3A, seg_pos={0 0}, upload=274674]: Uploading piece... Aug 19 17:38:09.693350092 (264h12m4.124430213s) [noise=true , address=45.91.116.14:4222 , piece_id=DXHEK43V7USJLYKPF5PKEHLJPVP4OPH2H3TQVGUBMK5MVRWQHXJQ, seg_pos={0 0}, upload=274674]: Uploading piece... Aug 19 17:38:09.693395940 (264h12m4.124476081s) [noise=true , address=45.91.116.11:4140 , piece_id=KQNAZLV4XPDYYLJQUMI2UL3GVLA7QTEZ6TU3HDYFYB7LPRBSRQFA, seg_pos={0 0}, upload=274674]: Uploading piece... Aug 19 17:38:09.693960980 (264h12m4.125041131s) [noise=true , address=79.127.201.215:4343 , piece_id=MIATMAVPWKBJPVYH3XDYXQGKW7VWKRULLP6UV2ZNKJZ7YDVJJEBQ, seg_pos={0 0}, upload=274674]: Uploading piece... Aug 19 17:38:09.694358652 (264h12m4.125438783s) [noise=true , address=104.36.82.5:4852 , piece_id=UALXJPWV34HHWPC6GNCOFNP2RDC3KF6ZY4FUHLL3MWR2UZDCXNCQ, seg_pos={0 0}, upload=274674]: Uploading piece... Aug 19 17:38:09.694689475 (264h12m4.125769606s) [noise=true , address=104.36.82.58:4269 , piece_id=CTQSY5PQFUNXUJR2JXSI7O2EHMYTOAHUSHGTMVTI42E45RYFIGJA, seg_pos={0 0}, upload=274674]: Uploading piece... Aug 19 17:38:09.695011601 (264h12m4.126091732s) [noise=true , address=104.36.82.60:4364 , piece_id=A5BGK2CC62VYRH27SR2MONG2GPSZ5WONCHAF42LFPXOTM2W7XIJA, seg_pos={0 0}, upload=274674]: Uploading piece... Aug 19 17:38:09.695441633 (264h12m4.126521764s) [noise=true , address=104.36.82.9:4028 , piece_id=4LUYCAISMQC3SGDLJZSDFDJET4R54U3N4B6ZAJ35UDXB5CT3GV2Q, seg_pos={0 0}, upload=274674]: Uploading piece... Aug 19 17:38:09.695845597 (264h12m4.126925728s) [noise=true , address=77.90.31.15:4701 , piece_id=CCTT764Z3HFZFT4PE7YOOBVVGDASLFYVFGVXGWADBLDLURBOKGGQ, seg_pos={0 0}, upload=274674]: Uploading piece... Aug 19 17:38:09.696191328 (264h12m4.127271459s) [noise=true , address=45.91.116.5:4302 , piece_id=D2ZYYIFLA7GFPEII4X75HVJGL5DOM6BZLQFXL7HYVXYLNCL4CDQQ, seg_pos={0 0}, upload=274674]: Uploading piece... Aug 19 17:38:09.696539624 (264h12m4.127619755s) [noise=true , address=77.90.31.14:4689 , piece_id=6FO4WN5BXNW6UFGW2ALS5YBYBP5FZCAZBLQZXAULFCN65NHH7N6A, seg_pos={0 0}, upload=274674]: Uploading piece... Aug 19 17:38:09.696809421 (264h12m4.127889552s) [noise=true , address=79.127.201.214:4299 , piece_id=C4JSI4OPEZPPHADT6QTVJGEXQNVSHEK42KNYQBV3S2OH7Z5Z7HHA, seg_pos={0 0}, upload=274674]: Uploading piece... Aug 19 17:38:09.697172445 (264h12m4.128252576s) [noise=true , address=104.36.82.43:4855 , piece_id=3H2PUTGFVC2PR224NBHGRB7GLWBOL2MNXXWJ7JF23HYVF53TEWFQ, seg_pos={0 0}, upload=274674]: Uploading piece... Aug 19 17:38:09.697485064 (264h12m4.128565205s) [noise=true , address=77.90.31.12:4619 , piece_id=XMWKT6JBVLOPTRUIPNEK4GXVCH5OOVO5WYWDGWMXGA2VVVD74CBA, seg_pos={0 0}, upload=274674]: Uploading piece... Aug 19 17:38:09.697774858 (264h12m4.128854989s) [noise=true , address=104.36.82.38:4714 , piece_id=SZ42LIQE4LXTBCZY4BTXWS3EL27UYIQS3AAXGWNLH65TUZWNUYSQ, seg_pos={0 0}, upload=274674]: Uploading piece... Aug 19 17:38:09.698096103 (264h12m4.129176234s) [noise=true , address=104.36.82.4:4817 , piece_id=JNKPBDLUZUWI22ZLGTCD3BZJBAUX7TZVAGXJVFZOEJPK2Z3PMADA, seg_pos={0 0}, upload=274674]: Uploading piece... Aug 19 17:38:09.698352945 (264h12m4.129433076s) [noise=true , address=45.91.116.13:4183 , piece_id=OMUE6JFSTRIF64VYM5XCHNL7U26SEXP3IGWDBHP5G2E6YJQETEUQ, seg_pos={0 0}, upload=274674]: Uploading piece... Aug 19 17:38:09.698876106 (264h12m4.129956227s) [noise=true , address=104.36.82.6:4879 , piece_id=JKK5HMPYTMR7PNWHVPBDHPH3K6H7VM7AA4K4X4WXQ5LN57VRBUFA, seg_pos={0 0}, upload=274674]: Uploading piece... Aug 19 17:38:11.192631202 (264h12m5.623711333s) [upload=274674]: Next returned nil segment Aug 19 17:38:11.192648606 (264h12m5.623728737s) [upload=274674]: Waiting for error group managing segments... Aug 19 17:38:11.591537039 (264h12m6.02261717s) [noise=true , address=45.91.116.14:4222 , piece_id=DXHEK43V7USJLYKPF5PKEHLJPVP4OPH2H3TQVGUBMK5MVRWQHXJQ, seg_pos={0 0}, upload=274674]: Done uploading piece. err: <nil> Aug 19 17:38:11.606093581 (264h12m6.037173712s) [noise=true , address=77.90.31.14:4689 , piece_id=6FO4WN5BXNW6UFGW2ALS5YBYBP5FZCAZBLQZXAULFCN65NHH7N6A, seg_pos={0 0}, upload=274674]: Done uploading piece. err: <nil> Aug 19 17:38:11.610339916 (264h12m6.041420047s) [noise=true , address=77.90.31.15:4701 , piece_id=CCTT764Z3HFZFT4PE7YOOBVVGDASLFYVFGVXGWADBLDLURBOKGGQ, seg_pos={0 0}, upload=274674]: Done uploading piece. err: <nil> Aug 19 17:38:11.630055040 (264h12m6.061135171s) [noise=true , address=79.127.201.215:4343 , piece_id=MIATMAVPWKBJPVYH3XDYXQGKW7VWKRULLP6UV2ZNKJZ7YDVJJEBQ, seg_pos={0 0}, upload=274674]: Done uploading piece. err: <nil> Aug 19 17:38:11.678703675 (264h12m6.109783797s) [noise=true , address=104.36.82.9:4028 , piece_id=4LUYCAISMQC3SGDLJZSDFDJET4R54U3N4B6ZAJ35UDXB5CT3GV2Q, seg_pos={0 0}, upload=274674]: Done uploading piece. err: <nil> Aug 19 17:38:11.693543009 (264h12m6.12462314s) [noise=true , address=104.36.82.60:4364 , piece_id=A5BGK2CC62VYRH27SR2MONG2GPSZ5WONCHAF42LFPXOTM2W7XIJA, seg_pos={0 0}, upload=274674]: Done uploading piece. err: <nil> Aug 19 17:38:11.693568448 (264h12m6.124648579s) [noise=true , address=45.91.116.13:4183 , piece_id=OMUE6JFSTRIF64VYM5XCHNL7U26SEXP3IGWDBHP5G2E6YJQETEUQ, seg_pos={0 0}, upload=274674]: Done uploading piece. err: <nil> Aug 19 17:38:11.698831207 (264h12m6.129911329s) [noise=true , address=104.36.82.5:4852 , piece_id=UALXJPWV34HHWPC6GNCOFNP2RDC3KF6ZY4FUHLL3MWR2UZDCXNCQ, seg_pos={0 0}, upload=274674]: Done uploading piece. err: <nil> Aug 19 17:38:11.698886834 (264h12m6.129966955s) [noise=true , address=77.90.31.12:4619 , piece_id=XMWKT6JBVLOPTRUIPNEK4GXVCH5OOVO5WYWDGWMXGA2VVVD74CBA, seg_pos={0 0}, upload=274674]: Done uploading piece. err: <nil> Aug 19 17:38:11.714704781 (264h12m6.145784912s) [noise=true , address=79.127.201.214:4299 , piece_id=C4JSI4OPEZPPHADT6QTVJGEXQNVSHEK42KNYQBV3S2OH7Z5Z7HHA, seg_pos={0 0}, upload=274674]: Done uploading piece. err: <nil> Aug 19 17:38:11.720319795 (264h12m6.151399956s) [noise=true , address=45.91.116.11:4140 , piece_id=KQNAZLV4XPDYYLJQUMI2UL3GVLA7QTEZ6TU3HDYFYB7LPRBSRQFA, seg_pos={0 0}, upload=274674]: Done uploading piece. err: <nil> Aug 19 17:38:11.722063401 (264h12m6.153143532s) [noise=true , address=104.36.82.6:4879 , piece_id=JKK5HMPYTMR7PNWHVPBDHPH3K6H7VM7AA4K4X4WXQ5LN57VRBUFA, seg_pos={0 0}, upload=274674]: Done uploading piece. err: <nil> Aug 19 17:38:11.726762272 (264h12m6.157842393s) [noise=true , address=104.36.82.38:4714 , piece_id=SZ42LIQE4LXTBCZY4BTXWS3EL27UYIQS3AAXGWNLH65TUZWNUYSQ, seg_pos={0 0}, upload=274674]: Done uploading piece. err: <nil> Aug 19 17:38:11.734258877 (264h12m6.165339008s) [noise=true , address=104.36.82.58:4269 , piece_id=CTQSY5PQFUNXUJR2JXSI7O2EHMYTOAHUSHGTMVTI42E45RYFIGJA, seg_pos={0 0}, upload=274674]: Done uploading piece. err: <nil> Aug 19 17:38:11.734284346 (264h12m6.165364477s) [seg_pos={0 0}, upload=274674]: Segment reached optimal threshold of 14 pieces. Aug 19 17:38:11.734416277 (264h12m6.165496408s) [noise=true , address=104.36.82.4:4817 , piece_id=JNKPBDLUZUWI22ZLGTCD3BZJBAUX7TZVAGXJVFZOEJPK2Z3PMADA, seg_pos={0 0}, upload=274674]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RSBodzJxMye65QgkyrzkWCKQ49cnfdu4haUZKW23nWpqzMcYz): protocol: expected piece hash; context canceled; EOF Aug 19 17:38:11.734486873 (264h12m6.165567004s) [noise=true , address=104.36.82.43:4855 , piece_id=3H2PUTGFVC2PR224NBHGRB7GLWBOL2MNXXWJ7JF23HYVF53TEWFQ, seg_pos={0 0}, upload=274674]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TGTRhEi2KsbrkC6ZmgGkMaJri9uhzEcmXe1J4nJjpEd1BE4ka): protocol: expected piece hash; context canceled; EOF Aug 19 17:38:11.734587926 (264h12m6.165668057s) [noise=true , address=77.90.31.18:4825 , piece_id=UQLJVRVUBBN5LFBIAQL7YQ4IRAWI3SBVBHUT4IIFBROAUHXHVZ3A, seg_pos={0 0}, upload=274674]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PqKyfFxfVKTGARbPqBcQL3Naqjdf5uR44PeLi3sJpu57dcm1x): protocol: expected piece hash; context canceled; EOF Aug 19 17:38:11.734592956 (264h12m6.165673087s) [noise=true , address=45.91.116.5:4302 , piece_id=D2ZYYIFLA7GFPEII4X75HVJGL5DOM6BZLQFXL7HYVXYLNCL4CDQQ, seg_pos={0 0}, upload=274674]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pJWhSLza8WaMuvvDwH4nae57UxQmfCdDvAkQ3v25MgPYHoZXx): protocol: expected piece hash; context canceled; EOF Aug 19 17:38:11.734645206 (264h12m6.165725337s) [seg_pos={0 0}, upload=274674]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 19 17:38:11.734756098 (264h12m6.165836239s) [upload=274674]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 19 17:38:11.734764033 (264h12m6.165844154s) [upload=274674]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 17:38:12.023665554 (264h12m6.454745685s) [upload=274674]: Done uploading segments... Aug 19 17:38:12.023686044 (264h12m6.454766195s) [upload=274674]: Upload finished. err: <nil> Duration: 5.714510801s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 5.685262094s FAIL (2025-08-19 17:52:09 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Got next segment. Inline: false Aug 19 17:23:57.290467622 (263h57m51.721547763s) [upload=274428]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 17:23:57.290471560 (263h57m51.721551681s) [upload=274428]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 19 17:23:57.743765742 (263h57m52.174845863s) [seg_pos={0 0}, upload=274428]: Begin upload segment... Aug 19 17:23:57.743818412 (263h57m52.174898543s) [seg_pos={0 0}, upload=274428]: Done begin upload segment. Aug 19 17:23:57.743842609 (263h57m52.17492274s) [noise=true , address=77.90.31.16:4750 , piece_id=XXJ4VHMZ4NVVT2WX322JS7MW3FHQ7UZJW4VLR4KG6FW4A4RHVN6A, seg_pos={0 0}, upload=274428]: Uploading piece... Aug 19 17:23:57.743875672 (263h57m52.174955803s) [noise=true , address=45.91.116.14:4198 , piece_id=NSLSBMDS2ZJKN62N4NGN7HRHXAUIS4WL4AGHODZZBD2CDYTXHVUA, seg_pos={0 0}, upload=274428]: Uploading piece... Aug 19 17:23:57.744218498 (263h57m52.175298629s) [noise=true , address=77.90.31.14:4667 , piece_id=BF53QUGTGVD23KCIIAA5NWUCWKWUXXSHKCAAWSDZLAORPHXQEMQQ, seg_pos={0 0}, upload=274428]: Uploading piece... Aug 19 17:23:57.744456373 (263h57m52.175536504s) [noise=true , address=77.90.31.21:4883 , piece_id=SVFJCG4T5OCDMVKBE7A33XPDIQUH6NESE4GIVTETRZRBX4N3MJDQ, seg_pos={0 0}, upload=274428]: Uploading piece... Aug 19 17:23:57.744523051 (263h57m52.175603182s) [noise=true , address=104.36.82.60:4365 , piece_id=7BXP5MWOH2B3H74CPVWOQN2NEGMXUYECIVOEHRRKINQ544WKAAHA, seg_pos={0 0}, upload=274428]: Uploading piece... Aug 19 17:23:57.744762870 (263h57m52.175843001s) [noise=true , address=77.90.31.19:4851 , piece_id=POYQI453UA5767REISB5BNE3MYV6IOYZNIK357PYGH5DM66EOBIA, seg_pos={0 0}, upload=274428]: Uploading piece... Aug 19 17:23:57.744993531 (263h57m52.176073662s) [noise=true , address=104.36.82.44:4871 , piece_id=WQGOUIWA7NVK5SUTIWX6E32ZKW62Y2KGK7E3HB5YW2CKDUR42BPQ, seg_pos={0 0}, upload=274428]: Uploading piece... Aug 19 17:23:57.745229373 (263h57m52.176309494s) [noise=true , address=79.127.205.234:4587 , piece_id=ALBAMUJVGA4RIJKGCJNJT3BT6RBD5QWU4C33RJGCEP3XIQQLW6QQ, seg_pos={0 0}, upload=274428]: Uploading piece... Aug 19 17:23:57.745473950 (263h57m52.176554081s) [noise=true , address=104.36.82.3:4771 , piece_id=2RZZG2MORQZR3ROAADIWSK46JDEZSNYBGASCDWZTKWWU72JFHRSQ, seg_pos={0 0}, upload=274428]: Uploading piece... Aug 19 17:23:57.745746030 (263h57m52.176826161s) [noise=true , address=104.36.82.34:4558 , piece_id=AJNHQTDQSMJ3R2ZKVQUCBU5XOMWNF66ID2M3RVPYDL52F5LGTPFQ, seg_pos={0 0}, upload=274428]: Uploading piece... Aug 19 17:23:57.745999426 (263h57m52.177079547s) [noise=true , address=77.90.31.18:4814 , piece_id=VUIFMMFA5PCHIKNNX7PA7RHVXIJHK3UWH4SI34R6XP5MEYTAMS6A, seg_pos={0 0}, upload=274428]: Uploading piece... Aug 19 17:23:57.746774839 (263h57m52.17785497s) [noise=true , address=104.36.82.35:4610 , piece_id=DM7HADQAK5T67ZDQR3VGUUGLHCGBLC5KJYYFF7QQNRKVE2SA5CHA, seg_pos={0 0}, upload=274428]: Uploading piece... Aug 19 17:23:57.747503714 (263h57m52.178583836s) [noise=true , address=64.49.13.137:4496 , piece_id=DW5YMZIQJZIA53K22HGK2ICRT5IVVP2ZMIVYM46BWQWAYPLNUC2Q, seg_pos={0 0}, upload=274428]: Uploading piece... Aug 19 17:23:57.747727743 (263h57m52.178807874s) [noise=true , address=77.90.31.13:4640 , piece_id=NKLBPZJDAIAVAYH3QRDCPS6VWHJBFZZWY3HCTSW2E4II77YC3BYA, seg_pos={0 0}, upload=274428]: Uploading piece... Aug 19 17:23:57.747948565 (263h57m52.179028696s) [noise=true , address=104.36.82.55:4214 , piece_id=64P34K2YEWCWZFGWSR4AQ5SRALT5CSZCB4F6FHPKDP3ZFJIOJC2Q, seg_pos={0 0}, upload=274428]: Uploading piece... Aug 19 17:23:57.748174007 (263h57m52.179254138s) [noise=true , address=104.36.82.23:4448 , piece_id=SCMNBDCYA7PUN2VP6JN37YM5LNY2GBGQHJKYSSWUWMPTZZ7LXZUQ, seg_pos={0 0}, upload=274428]: Uploading piece... Aug 19 17:23:57.748433112 (263h57m52.179513233s) [noise=true , address=104.36.82.5:4854 , piece_id=FC4DIKMBI5OPLDQCVA6P3MA7AQZV23QBIX3UAKLWJJMY724TNYHQ, seg_pos={0 0}, upload=274428]: Uploading piece... Aug 19 17:23:57.748647252 (263h57m52.179727383s) [noise=true , address=104.36.82.62:4442 , piece_id=VV5BBYB6UI4O3B4ESOBJX44I5RJBRWSLJUFR6H3GKEUZWNONTPMA, seg_pos={0 0}, upload=274428]: Uploading piece... Aug 19 17:23:58.869276532 (263h57m53.300356663s) [upload=274428]: Next returned nil segment Aug 19 17:23:58.869291511 (263h57m53.300371642s) [upload=274428]: Waiting for error group managing segments... Aug 19 17:23:59.195579562 (263h57m53.626659683s) [noise=true , address=79.127.205.234:4587 , piece_id=ALBAMUJVGA4RIJKGCJNJT3BT6RBD5QWU4C33RJGCEP3XIQQLW6QQ, seg_pos={0 0}, upload=274428]: Done uploading piece. err: <nil> Aug 19 17:23:59.239543682 (263h57m53.670623813s) [noise=true , address=64.49.13.137:4496 , piece_id=DW5YMZIQJZIA53K22HGK2ICRT5IVVP2ZMIVYM46BWQWAYPLNUC2Q, seg_pos={0 0}, upload=274428]: Done uploading piece. err: <nil> Aug 19 17:23:59.654494638 (263h57m54.085574769s) [noise=true , address=104.36.82.35:4610 , piece_id=DM7HADQAK5T67ZDQR3VGUUGLHCGBLC5KJYYFF7QQNRKVE2SA5CHA, seg_pos={0 0}, upload=274428]: Done uploading piece. err: <nil> Aug 19 17:23:59.700507838 (263h57m54.131587959s) [noise=true , address=104.36.82.34:4558 , piece_id=AJNHQTDQSMJ3R2ZKVQUCBU5XOMWNF66ID2M3RVPYDL52F5LGTPFQ, seg_pos={0 0}, upload=274428]: Done uploading piece. err: <nil> Aug 19 17:23:59.757508140 (263h57m54.188588272s) [noise=true , address=104.36.82.5:4854 , piece_id=FC4DIKMBI5OPLDQCVA6P3MA7AQZV23QBIX3UAKLWJJMY724TNYHQ, seg_pos={0 0}, upload=274428]: Done uploading piece. err: <nil> Aug 19 17:23:59.765247029 (263h57m54.19632716s) [noise=true , address=104.36.82.62:4442 , piece_id=VV5BBYB6UI4O3B4ESOBJX44I5RJBRWSLJUFR6H3GKEUZWNONTPMA, seg_pos={0 0}, upload=274428]: Done uploading piece. err: <nil> Aug 19 17:23:59.774367630 (263h57m54.205447761s) [noise=true , address=104.36.82.23:4448 , piece_id=SCMNBDCYA7PUN2VP6JN37YM5LNY2GBGQHJKYSSWUWMPTZZ7LXZUQ, seg_pos={0 0}, upload=274428]: Done uploading piece. err: <nil> Aug 19 17:23:59.827970201 (263h57m54.259050332s) [noise=true , address=104.36.82.55:4214 , piece_id=64P34K2YEWCWZFGWSR4AQ5SRALT5CSZCB4F6FHPKDP3ZFJIOJC2Q, seg_pos={0 0}, upload=274428]: Done uploading piece. err: <nil> Aug 19 17:23:59.851412395 (263h57m54.282492526s) [noise=true , address=104.36.82.44:4871 , piece_id=WQGOUIWA7NVK5SUTIWX6E32ZKW62Y2KGK7E3HB5YW2CKDUR42BPQ, seg_pos={0 0}, upload=274428]: Done uploading piece. err: <nil> Aug 19 17:23:59.946562012 (263h57m54.377642143s) [noise=true , address=104.36.82.3:4771 , piece_id=2RZZG2MORQZR3ROAADIWSK46JDEZSNYBGASCDWZTKWWU72JFHRSQ, seg_pos={0 0}, upload=274428]: Done uploading piece. err: <nil> Aug 19 17:23:59.979671513 (263h57m54.410751634s) [noise=true , address=104.36.82.60:4365 , piece_id=7BXP5MWOH2B3H74CPVWOQN2NEGMXUYECIVOEHRRKINQ544WKAAHA, seg_pos={0 0}, upload=274428]: Done uploading piece. err: <nil> Aug 19 17:24:55.494274881 (263h58m49.925355012s) [noise=true , address=77.90.31.18:4814 , piece_id=VUIFMMFA5PCHIKNNX7PA7RHVXIJHK3UWH4SI34R6XP5MEYTAMS6A, seg_pos={0 0}, upload=274428]: Done uploading piece. err: <nil> Aug 19 17:24:55.842835997 (263h58m50.273916128s) [noise=true , address=77.90.31.21:4883 , piece_id=SVFJCG4T5OCDMVKBE7A33XPDIQUH6NESE4GIVTETRZRBX4N3MJDQ, seg_pos={0 0}, upload=274428]: Done uploading piece. err: <nil> Aug 19 17:24:56.634889530 (263h58m51.065969661s) [noise=true , address=77.90.31.19:4851 , piece_id=POYQI453UA5767REISB5BNE3MYV6IOYZNIK357PYGH5DM66EOBIA, seg_pos={0 0}, upload=274428]: Done uploading piece. err: ecclient: upload failed (node:1NEV2ndHw4fohVUNsfrzwojd8S11TmMST6Sz59GVnXS8vDGhWk, address:77.90.31.19:4851): protocol: rpc: dial tcp 77.90.31.19:4851: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:56.635097748 (263h58m51.066177879s) [noise=true , address=45.91.116.14:4198 , piece_id=NSLSBMDS2ZJKN62N4NGN7HRHXAUIS4WL4AGHODZZBD2CDYTXHVUA, seg_pos={0 0}, upload=274428]: Done uploading piece. err: ecclient: upload failed (node:1xxfUrCm5PgSLpge1wYiH2EcE7kzzJ38P9rfeCYmqXTbTisioi, address:45.91.116.14:4198): protocol: rpc: dial tcp 45.91.116.14:4198: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:56.635158895 (263h58m51.066239036s) [noise=true , address=77.90.31.16:4750 , piece_id=XXJ4VHMZ4NVVT2WX322JS7MW3FHQ7UZJW4VLR4KG6FW4A4RHVN6A, seg_pos={0 0}, upload=274428]: Done uploading piece. err: ecclient: upload failed (node:12TGd8NnWdekDtHzBQHmJPgTtHjXC6g8fi6RoZYFQJVHDa5hdyE, address:77.90.31.16:4750): protocol: read tcp 188.166.225.170:54198->77.90.31.16:4750: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:56.635224120 (263h58m51.066304281s) [noise=true , address=77.90.31.14:4667 , piece_id=BF53QUGTGVD23KCIIAA5NWUCWKWUXXSHKCAAWSDZLAORPHXQEMQQ, seg_pos={0 0}, upload=274428]: Done uploading piece. err: ecclient: upload failed (node:12R5gJFhsmocYPNZf4yNnYqJwcwAKt9yiHmmqGMk3btzVjkFV4S, address:77.90.31.14:4667): protocol: read tcp 188.166.225.170:46984->77.90.31.14:4667: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:56.635280057 (263h58m51.066360198s) [noise=true , address=77.90.31.13:4640 , piece_id=NKLBPZJDAIAVAYH3QRDCPS6VWHJBFZZWY3HCTSW2E4II77YC3BYA, seg_pos={0 0}, upload=274428]: Done uploading piece. err: ecclient: upload failed (node:1XjJpLBEZm5daHcBUmUyC7QpLVhu9Vm9ex9RYcYJWEuLGk527F, address:77.90.31.13:4640): protocol: read tcp 188.166.225.170:48656->77.90.31.13:4640: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:56.635314653 (263h58m51.066394794s) [seg_pos={0 0}, upload=274428]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Aug 19 17:24:56.862515362 (263h58m51.293595493s) [upload=274428]: Done uploading segments... Aug 19 17:24:56.862536042 (263h58m51.293616163s) [upload=274428]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.685290068s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..............X.....X.......................................] duration: 6.202584634s FAIL (2025-08-19 17:52:09 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.202606987s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 2.041409537s FAIL (2025-08-19 17:52:01 UTC) orbiter warning: operation: ListObjects ======ERROR====== uplink: metainfo: metaclient: context deadline exceeded ======LOGS====== Aug 19 17:35:36.607320794 (264h9m31.038400925s) [upload=274632]: Starting upload Aug 19 17:35:36.607367684 (264h9m31.038447815s) [upload=274632]: Done starting upload Aug 19 17:35:36.607435082 (264h9m31.038515213s) [upload=274632]: Uploading segments... Aug 19 17:35:36.607490538 (264h9m31.038570669s) [upload=274632]: Got next segment. Inline: true Aug 19 17:35:36.607495088 (264h9m31.038575219s) [upload=274632]: Waiting for error group managing segments... Aug 19 17:35:36.607537528 (264h9m31.038617659s) [upload=274632]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 17:35:36.607541656 (264h9m31.038621787s) [upload=274632]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 19 17:35:36.607544051 (264h9m31.038624182s) [upload=274632]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 17:35:36.889055260 (264h9m31.320135391s) [upload=274632]: Done uploading segments... Aug 19 17:35:36.889082943 (264h9m31.320163124s) [upload=274632]: Upload finished. err: <nil> Duration:* 2.041441728s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...............X.....X......................................] duration: 6.225633127s FAIL (2025-08-19 17:51:28 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.225652535s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 7 out of 60 is failed (threshold=3) [..X.............X......X.X........X....XX...................] duration: 6.09618476s FAIL (2025-08-19 17:51:27 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/4M") ======LOGS====== Aug 19 17:38:09.246414094 (264h12m3.677494255s) [upload=274674]: Starting upload Aug 19 17:38:09.246486392 (264h12m3.677566533s) [upload=274674]: Done starting upload Aug 19 17:38:09.246601452 (264h12m3.677681583s) [upload=274674]: Uploading segments... Aug 19 17:38:09.246691885 (264h12m3.677772016s) [upload=274674]: Got next segment. Inline: false Aug 19 17:38:09.246724627 (264h12m3.677804758s) [upload=274674]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 17:38:09.246730909 (264h12m3.67781104s) [upload=274674]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 19 17:38:09.693242556 (264h12m4.124322687s) [seg_pos={0 0}, upload=274674]: Begin upload segment... Aug 19 17:38:09.693302230 (264h12m4.124382361s) [seg_pos={0 0}, upload=274674]: Done begin upload segment. Aug 19 17:38:09.693343529 (264h12m4.12442367s) [noise=true , address=77.90.31.18:4825 , piece_id=UQLJVRVUBBN5LFBIAQL7YQ4IRAWI3SBVBHUT4IIFBROAUHXHVZ3A, seg_pos={0 0}, upload=274674]: Uploading piece... Aug 19 17:38:09.693350092 (264h12m4.124430213s) [noise=true , address=45.91.116.14:4222 , piece_id=DXHEK43V7USJLYKPF5PKEHLJPVP4OPH2H3TQVGUBMK5MVRWQHXJQ, seg_pos={0 0}, upload=274674]: Uploading piece... Aug 19 17:38:09.693395940 (264h12m4.124476081s) [noise=true , address=45.91.116.11:4140 , piece_id=KQNAZLV4XPDYYLJQUMI2UL3GVLA7QTEZ6TU3HDYFYB7LPRBSRQFA, seg_pos={0 0}, upload=274674]: Uploading piece... Aug 19 17:38:09.693960980 (264h12m4.125041131s) [noise=true , address=79.127.201.215:4343 , piece_id=MIATMAVPWKBJPVYH3XDYXQGKW7VWKRULLP6UV2ZNKJZ7YDVJJEBQ, seg_pos={0 0}, upload=274674]: Uploading piece... Aug 19 17:38:09.694358652 (264h12m4.125438783s) [noise=true , address=104.36.82.5:4852 , piece_id=UALXJPWV34HHWPC6GNCOFNP2RDC3KF6ZY4FUHLL3MWR2UZDCXNCQ, seg_pos={0 0}, upload=274674]: Uploading piece... Aug 19 17:38:09.694689475 (264h12m4.125769606s) [noise=true , address=104.36.82.58:4269 , piece_id=CTQSY5PQFUNXUJR2JXSI7O2EHMYTOAHUSHGTMVTI42E45RYFIGJA, seg_pos={0 0}, upload=274674]: Uploading piece... Aug 19 17:38:09.695011601 (264h12m4.126091732s) [noise=true , address=104.36.82.60:4364 , piece_id=A5BGK2CC62VYRH27SR2MONG2GPSZ5WONCHAF42LFPXOTM2W7XIJA, seg_pos={0 0}, upload=274674]: Uploading piece... Aug 19 17:38:09.695441633 (264h12m4.126521764s) [noise=true , address=104.36.82.9:4028 , piece_id=4LUYCAISMQC3SGDLJZSDFDJET4R54U3N4B6ZAJ35UDXB5CT3GV2Q, seg_pos={0 0}, upload=274674]: Uploading piece... Aug 19 17:38:09.695845597 (264h12m4.126925728s) [noise=true , address=77.90.31.15:4701 , piece_id=CCTT764Z3HFZFT4PE7YOOBVVGDASLFYVFGVXGWADBLDLURBOKGGQ, seg_pos={0 0}, upload=274674]: Uploading piece... Aug 19 17:38:09.696191328 (264h12m4.127271459s) [noise=true , address=45.91.116.5:4302 , piece_id=D2ZYYIFLA7GFPEII4X75HVJGL5DOM6BZLQFXL7HYVXYLNCL4CDQQ, seg_pos={0 0}, upload=274674]: Uploading piece... Aug 19 17:38:09.696539624 (264h12m4.127619755s) [noise=true , address=77.90.31.14:4689 , piece_id=6FO4WN5BXNW6UFGW2ALS5YBYBP5FZCAZBLQZXAULFCN65NHH7N6A, seg_pos={0 0}, upload=274674]: Uploading piece... Aug 19 17:38:09.696809421 (264h12m4.127889552s) [noise=true , address=79.127.201.214:4299 , piece_id=C4JSI4OPEZPPHADT6QTVJGEXQNVSHEK42KNYQBV3S2OH7Z5Z7HHA, seg_pos={0 0}, upload=274674]: Uploading piece... Aug 19 17:38:09.697172445 (264h12m4.128252576s) [noise=true , address=104.36.82.43:4855 , piece_id=3H2PUTGFVC2PR224NBHGRB7GLWBOL2MNXXWJ7JF23HYVF53TEWFQ, seg_pos={0 0}, upload=274674]: Uploading piece... Aug 19 17:38:09.697485064 (264h12m4.128565205s) [noise=true , address=77.90.31.12:4619 , piece_id=XMWKT6JBVLOPTRUIPNEK4GXVCH5OOVO5WYWDGWMXGA2VVVD74CBA, seg_pos={0 0}, upload=274674]: Uploading piece... Aug 19 17:38:09.697774858 (264h12m4.128854989s) [noise=true , address=104.36.82.38:4714 , piece_id=SZ42LIQE4LXTBCZY4BTXWS3EL27UYIQS3AAXGWNLH65TUZWNUYSQ, seg_pos={0 0}, upload=274674]: Uploading piece... Aug 19 17:38:09.698096103 (264h12m4.129176234s) [noise=true , address=104.36.82.4:4817 , piece_id=JNKPBDLUZUWI22ZLGTCD3BZJBAUX7TZVAGXJVFZOEJPK2Z3PMADA, seg_pos={0 0}, upload=274674]: Uploading piece... Aug 19 17:38:09.698352945 (264h12m4.129433076s) [noise=true , address=45.91.116.13:4183 , piece_id=OMUE6JFSTRIF64VYM5XCHNL7U26SEXP3IGWDBHP5G2E6YJQETEUQ, seg_pos={0 0}, upload=274674]: Uploading piece... Aug 19 17:38:09.698876106 (264h12m4.129956227s) [noise=true , address=104.36.82.6:4879 , piece_id=JKK5HMPYTMR7PNWHVPBDHPH3K6H7VM7AA4K4X4WXQ5LN57VRBUFA, seg_pos={0 0}, upload=274674]: Uploading piece... Aug 19 17:38:11.192631202 (264h12m5.623711333s) [upload=274674]: Next returned nil segment Aug 19 17:38:11.192648606 (264h12m5.623728737s) [upload=274674]: Waiting for error group managing segments... Aug 19 17:38:11.591537039 (264h12m6.02261717s) [noise=true , address=45.91.116.14:4222 , piece_id=DXHEK43V7USJLYKPF5PKEHLJPVP4OPH2H3TQVGUBMK5MVRWQHXJQ, seg_pos={0 0}, upload=274674]: Done uploading piece. err: <nil> Aug 19 17:38:11.606093581 (264h12m6.037173712s) [noise=true , address=77.90.31.14:4689 , piece_id=6FO4WN5BXNW6UFGW2ALS5YBYBP5FZCAZBLQZXAULFCN65NHH7N6A, seg_pos={0 0}, upload=274674]: Done uploading piece. err: <nil> Aug 19 17:38:11.610339916 (264h12m6.041420047s) [noise=true , address=77.90.31.15:4701 , piece_id=CCTT764Z3HFZFT4PE7YOOBVVGDASLFYVFGVXGWADBLDLURBOKGGQ, seg_pos={0 0}, upload=274674]: Done uploading piece. err: <nil> Aug 19 17:38:11.630055040 (264h12m6.061135171s) [noise=true , address=79.127.201.215:4343 , piece_id=MIATMAVPWKBJPVYH3XDYXQGKW7VWKRULLP6UV2ZNKJZ7YDVJJEBQ, seg_pos={0 0}, upload=274674]: Done uploading piece. err: <nil> Aug 19 17:38:11.678703675 (264h12m6.109783797s) [noise=true , address=104.36.82.9:4028 , piece_id=4LUYCAISMQC3SGDLJZSDFDJET4R54U3N4B6ZAJ35UDXB5CT3GV2Q, seg_pos={0 0}, upload=274674]: Done uploading piece. err: <nil> Aug 19 17:38:11.693543009 (264h12m6.12462314s) [noise=true , address=104.36.82.60:4364 , piece_id=A5BGK2CC62VYRH27SR2MONG2GPSZ5WONCHAF42LFPXOTM2W7XIJA, seg_pos={0 0}, upload=274674]: Done uploading piece. err: <nil> Aug 19 17:38:11.693568448 (264h12m6.124648579s) [noise=true , address=45.91.116.13:4183 , piece_id=OMUE6JFSTRIF64VYM5XCHNL7U26SEXP3IGWDBHP5G2E6YJQETEUQ, seg_pos={0 0}, upload=274674]: Done uploading piece. err: <nil> Aug 19 17:38:11.698831207 (264h12m6.129911329s) [noise=true , address=104.36.82.5:4852 , piece_id=UALXJPWV34HHWPC6GNCOFNP2RDC3KF6ZY4FUHLL3MWR2UZDCXNCQ, seg_pos={0 0}, upload=274674]: Done uploading piece. err: <nil> Aug 19 17:38:11.698886834 (264h12m6.129966955s) [noise=true , address=77.90.31.12:4619 , piece_id=XMWKT6JBVLOPTRUIPNEK4GXVCH5OOVO5WYWDGWMXGA2VVVD74CBA, seg_pos={0 0}, upload=274674]: Done uploading piece. err: <nil> Aug 19 17:38:11.714704781 (264h12m6.145784912s) [noise=true , address=79.127.201.214:4299 , piece_id=C4JSI4OPEZPPHADT6QTVJGEXQNVSHEK42KNYQBV3S2OH7Z5Z7HHA, seg_pos={0 0}, upload=274674]: Done uploading piece. err: <nil> Aug 19 17:38:11.720319795 (264h12m6.151399956s) [noise=true , address=45.91.116.11:4140 , piece_id=KQNAZLV4XPDYYLJQUMI2UL3GVLA7QTEZ6TU3HDYFYB7LPRBSRQFA, seg_pos={0 0}, upload=274674]: Done uploading piece. err: <nil> Aug 19 17:38:11.722063401 (264h12m6.153143532s) [noise=true , address=104.36.82.6:4879 , piece_id=JKK5HMPYTMR7PNWHVPBDHPH3K6H7VM7AA4K4X4WXQ5LN57VRBUFA, seg_pos={0 0}, upload=274674]: Done uploading piece. err: <nil> Aug 19 17:38:11.726762272 (264h12m6.157842393s) [noise=true , address=104.36.82.38:4714 , piece_id=SZ42LIQE4LXTBCZY4BTXWS3EL27UYIQS3AAXGWNLH65TUZWNUYSQ, seg_pos={0 0}, upload=274674]: Done uploading piece. err: <nil> Aug 19 17:38:11.734258877 (264h12m6.165339008s) [noise=true , address=104.36.82.58:4269 , piece_id=CTQSY5PQFUNXUJR2JXSI7O2EHMYTOAHUSHGTMVTI42E45RYFIGJA, seg_pos={0 0}, upload=274674]: Done uploading piece. err: <nil> Aug 19 17:38:11.734284346 (264h12m6.165364477s) [seg_pos={0 0}, upload=274674]: Segment reached optimal threshold of 14 pieces. Aug 19 17:38:11.734416277 (264h12m6.165496408s) [noise=true , address=104.36.82.4:4817 , piece_id=JNKPBDLUZUWI22ZLGTCD3BZJBAUX7TZVAGXJVFZOEJPK2Z3PMADA, seg_pos={0 0}, upload=274674]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RSBodzJxMye65QgkyrzkWCKQ49cnfdu4haUZKW23nWpqzMcYz): protocol: expected piece hash; context canceled; EOF Aug 19 17:38:11.734486873 (264h12m6.165567004s) [noise=true , address=104.36.82.43:4855 , piece_id=3H2PUTGFVC2PR224NBHGRB7GLWBOL2MNXXWJ7JF23HYVF53TEWFQ, seg_pos={0 0}, upload=274674]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TGTRhEi2KsbrkC6ZmgGkMaJri9uhzEcmXe1J4nJjpEd1BE4ka): protocol: expected piece hash; context canceled; EOF Aug 19 17:38:11.734587926 (264h12m6.165668057s) [noise=true , address=77.90.31.18:4825 , piece_id=UQLJVRVUBBN5LFBIAQL7YQ4IRAWI3SBVBHUT4IIFBROAUHXHVZ3A, seg_pos={0 0}, upload=274674]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PqKyfFxfVKTGARbPqBcQL3Naqjdf5uR44PeLi3sJpu57dcm1x): protocol: expected piece hash; context canceled; EOF Aug 19 17:38:11.734592956 (264h12m6.165673087s) [noise=true , address=45.91.116.5:4302 , piece_id=D2ZYYIFLA7GFPEII4X75HVJGL5DOM6BZLQFXL7HYVXYLNCL4CDQQ, seg_pos={0 0}, upload=274674]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pJWhSLza8WaMuvvDwH4nae57UxQmfCdDvAkQ3v25MgPYHoZXx): protocol: expected piece hash; context canceled; EOF Aug 19 17:38:11.734645206 (264h12m6.165725337s) [seg_pos={0 0}, upload=274674]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 19 17:38:11.734756098 (264h12m6.165836239s) [upload=274674]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 19 17:38:11.734764033 (264h12m6.165844154s) [upload=274674]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 17:38:12.023665554 (264h12m6.454745685s) [upload=274674]: Done uploading segments... Aug 19 17:38:12.023686044 (264h12m6.454766195s) [upload=274674]: Upload finished. err: <nil> Duration: 6.096203295s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.................X...X......................................] duration: 6.456488028s FAIL (2025-08-19 17:51:22 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 23:59.057943072 (263h57m53.489023273s) [noise=true , address=45.91.116.10:4083 , piece_id=V3H4T2WN2MBVALSIHZNOQHBE75WQRNYLIE3Z7DFWUUHBPYDT7IFA, seg_pos={0 0}, upload=274429]: Uploading piece... Aug 19 17:23:59.057955436 (263h57m53.489035557s) [noise=true , address=45.91.116.5:4298 , piece_id=HMF6DXO5KFMLK67WVA7BAWLQFLC2BCVPJ7ZT255FMRCCEI3MCBNA, seg_pos={0 0}, upload=274429]: Uploading piece... Aug 19 17:23:59.058557689 (263h57m53.48963782s) [noise=true , address=104.36.82.8:4969 , piece_id=3P655Z5OUVZ4VRESACEBKYR3KUFYY56NYPWX7ZVOHL3XI2YQ5TCQ, seg_pos={0 0}, upload=274429]: Uploading piece... Aug 19 17:23:59.059170771 (263h57m53.490250902s) [noise=true , address=45.91.116.7:4009 , piece_id=NSZYKL6FHDLDM463ZXKEQGX4ZY7JPJXENH63PPDV76XMGCBV7ZRQ, seg_pos={0 0}, upload=274429]: Uploading piece... Aug 19 17:23:59.059468281 (263h57m53.490548412s) [noise=true , address=77.90.31.15:4699 , piece_id=MFF7PNMDLHYEH7PY46MGGOJSY4XPHRDPGFWJABBO4JDTJJJJ3CTQ, seg_pos={0 0}, upload=274429]: Uploading piece... Aug 19 17:23:59.059498208 (263h57m53.490578339s) [noise=true , address=104.36.82.2:4729 , piece_id=BNX6UNYZ7VHQ4MKAH65DGSHULWRRXDSCMKOE7WFZO6T2V3ELBJ7Q, seg_pos={0 0}, upload=274429]: Uploading piece... Aug 19 17:23:59.059814182 (263h57m53.490894323s) [noise=true , address=104.36.82.6:4874 , piece_id=QMGNWSGMEV762Y2XEDK2LIAY3JMNLD3UOVFQ6HYP7ZMLLVAVO7ZQ, seg_pos={0 0}, upload=274429]: Uploading piece... Aug 19 17:23:59.059839941 (263h57m53.490920072s) [noise=true , address=45.91.116.12:4225 , piece_id=PCJYR2VLFH23UPQEABUYESR3MHKDHIVTJWDIWVYIL6JAFGGXXXFQ, seg_pos={0 0}, upload=274429]: Uploading piece... Aug 19 17:23:59.059875480 (263h57m53.490955611s) [noise=true , address=104.36.82.10:4079 , piece_id=S3BSVJWRPP2PDGLG7FKFROUKAM3GTBDL2ALAJPAZQQQZJVRMMZ3Q, seg_pos={0 0}, upload=274429]: Uploading piece... Aug 19 17:23:59.060109147 (263h57m53.491189278s) [noise=true , address=104.36.82.2:4732 , piece_id=NLTK4572JAQHZQRDD2DOOIW7VBO7NK4DI4RUZRSXQ3GR72EWV3KQ, seg_pos={0 0}, upload=274429]: Uploading piece... Aug 19 17:23:59.060142190 (263h57m53.491222321s) [noise=true , address=45.91.116.9:4261 , piece_id=J5IMFL3JW4YKDLI6OEY2POEB4LJR2P3BIFAD4YIF3E4JTUXIXY3A, seg_pos={0 0}, upload=274429]: Uploading piece... Aug 19 17:23:59.060348274 (263h57m53.491428405s) [noise=true , address=104.36.82.15:4259 , piece_id=RM7JL7JKCMFM4CNXVVZQ2QJTOEPQEQHGLW3GYXWUD6LAQ2ZJ3P5Q, seg_pos={0 0}, upload=274429]: Uploading piece... Aug 19 17:23:59.060544830 (263h57m53.491624961s) [noise=true , address=45.91.116.14:4191 , piece_id=PJ33GP36S76VCI5RVR26PBFZ6HZUUZYVWNGPJITRLGDZI6LWCJIQ, seg_pos={0 0}, upload=274429]: Uploading piece... Aug 19 17:23:59.060752227 (263h57m53.491832358s) [noise=true , address=45.91.116.11:4118 , piece_id=VD2HK2QU63UVJFRASQOIRFZRWZN2HAUSAZJL2VZKYALGSV67SP2A, seg_pos={0 0}, upload=274429]: Uploading piece... Aug 19 17:23:59.060967990 (263h57m53.492048121s) [noise=true , address=77.90.31.11:4555 , piece_id=GTU6PGCHI4WOFGK2B3FP2Y22VUDBWLQKKEMZBT6I4F47OKK4RPNQ, seg_pos={0 0}, upload=274429]: Uploading piece... Aug 19 17:23:59.061294044 (263h57m53.492374175s) [noise=true , address=45.91.116.8:4048 , piece_id=M2CBILQKSVSQ7VY4AWQQM5GHF2KYBSXH73YJGAEJZAHOR7FOBWEQ, seg_pos={0 0}, upload=274429]: Uploading piece... Aug 19 17:24:00.872920400 (263h57m55.304000531s) [upload=274429]: Next returned nil segment Aug 19 17:24:00.872945548 (263h57m55.304025679s) [upload=274429]: Waiting for error group managing segments... Aug 19 17:24:01.156214888 (263h57m55.587295019s) [noise=true , address=104.36.82.6:4874 , piece_id=QMGNWSGMEV762Y2XEDK2LIAY3JMNLD3UOVFQ6HYP7ZMLLVAVO7ZQ, seg_pos={0 0}, upload=274429]: Done uploading piece. err: <nil> Aug 19 17:24:01.181164926 (263h57m55.612245057s) [noise=true , address=104.36.82.2:4732 , piece_id=NLTK4572JAQHZQRDD2DOOIW7VBO7NK4DI4RUZRSXQ3GR72EWV3KQ, seg_pos={0 0}, upload=274429]: Done uploading piece. err: <nil> Aug 19 17:24:01.222047199 (263h57m55.653127331s) [noise=true , address=104.36.82.2:4729 , piece_id=BNX6UNYZ7VHQ4MKAH65DGSHULWRRXDSCMKOE7WFZO6T2V3ELBJ7Q, seg_pos={0 0}, upload=274429]: Done uploading piece. err: <nil> Aug 19 17:24:01.250473409 (263h57m55.68155354s) [noise=true , address=104.36.82.15:4259 , piece_id=RM7JL7JKCMFM4CNXVVZQ2QJTOEPQEQHGLW3GYXWUD6LAQ2ZJ3P5Q, seg_pos={0 0}, upload=274429]: Done uploading piece. err: <nil> Aug 19 17:24:01.259222299 (263h57m55.69030242s) [noise=true , address=104.36.82.8:4969 , piece_id=3P655Z5OUVZ4VRESACEBKYR3KUFYY56NYPWX7ZVOHL3XI2YQ5TCQ, seg_pos={0 0}, upload=274429]: Done uploading piece. err: <nil> Aug 19 17:24:01.261735818 (263h57m55.692815949s) [noise=true , address=104.36.82.10:4079 , piece_id=S3BSVJWRPP2PDGLG7FKFROUKAM3GTBDL2ALAJPAZQQQZJVRMMZ3Q, seg_pos={0 0}, upload=274429]: Done uploading piece. err: <nil> Aug 19 17:24:28.183964515 (263h58m22.615044646s) [noise=true , address=45.91.116.14:4191 , piece_id=PJ33GP36S76VCI5RVR26PBFZ6HZUUZYVWNGPJITRLGDZI6LWCJIQ, seg_pos={0 0}, upload=274429]: Done uploading piece. err: ecclient: upload failed (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, address:45.91.116.14:4191): protocol: read tcp 188.166.225.170:37646->45.91.116.14:4191: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:28.184013127 (263h58m22.615093258s) [noise=true , address=45.91.116.9:4261 , piece_id=J5IMFL3JW4YKDLI6OEY2POEB4LJR2P3BIFAD4YIF3E4JTUXIXY3A, seg_pos={0 0}, upload=274429]: Done uploading piece. err: ecclient: upload failed (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, address:45.91.116.9:4261): protocol: rpc: dial tcp 45.91.116.9:4261: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:28.184026694 (263h58m22.615106825s) [noise=true , address=45.91.116.8:4048 , piece_id=M2CBILQKSVSQ7VY4AWQQM5GHF2KYBSXH73YJGAEJZAHOR7FOBWEQ, seg_pos={0 0}, upload=274429]: Done uploading piece. err: ecclient: upload failed (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, address:45.91.116.8:4048): protocol: read tcp 188.166.225.170:45604->45.91.116.8:4048: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:28.184070568 (263h58m22.615150689s) [noise=true , address=45.91.116.5:4298 , piece_id=HMF6DXO5KFMLK67WVA7BAWLQFLC2BCVPJ7ZT255FMRCCEI3MCBNA, seg_pos={0 0}, upload=274429]: Done uploading piece. err: ecclient: upload failed (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, address:45.91.116.5:4298): protocol: read tcp 188.166.225.170:46970->45.91.116.5:4298: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:28.184103010 (263h58m22.615183141s) [noise=true , address=45.91.116.10:4083 , piece_id=V3H4T2WN2MBVALSIHZNOQHBE75WQRNYLIE3Z7DFWUUHBPYDT7IFA, seg_pos={0 0}, upload=274429]: Done uploading piece. err: ecclient: upload failed (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, address:45.91.116.10:4083): protocol: read tcp 188.166.225.170:41632->45.91.116.10:4083: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:28.184139489 (263h58m22.61521962s) [noise=true , address=45.91.116.7:4009 , piece_id=NSZYKL6FHDLDM463ZXKEQGX4ZY7JPJXENH63PPDV76XMGCBV7ZRQ, seg_pos={0 0}, upload=274429]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: read tcp 188.166.225.170:45756->45.91.116.7:4009: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:28.184171009 (263h58m22.615251141s) [noise=true , address=77.90.31.11:4555 , piece_id=GTU6PGCHI4WOFGK2B3FP2Y22VUDBWLQKKEMZBT6I4F47OKK4RPNQ, seg_pos={0 0}, upload=274429]: Done uploading piece. err: ecclient: upload failed (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, address:77.90.31.11:4555): protocol: read tcp 188.166.225.170:37438->77.90.31.11:4555: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:28.184197480 (263h58m22.615277611s) [noise=true , address=77.90.31.15:4699 , piece_id=MFF7PNMDLHYEH7PY46MGGOJSY4XPHRDPGFWJABBO4JDTJJJJ3CTQ, seg_pos={0 0}, upload=274429]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: read tcp 188.166.225.170:36788->77.90.31.15:4699: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:28.184218410 (263h58m22.615298541s) [noise=true , address=45.91.116.12:4225 , piece_id=PCJYR2VLFH23UPQEABUYESR3MHKDHIVTJWDIWVYIL6JAFGGXXXFQ, seg_pos={0 0}, upload=274429]: Done uploading piece. err: ecclient: upload failed (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, address:45.91.116.12:4225): protocol: read tcp 188.166.225.170:60636->45.91.116.12:4225: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:28.184249710 (263h58m22.615329841s) [noise=true , address=45.91.116.11:4118 , piece_id=VD2HK2QU63UVJFRASQOIRFZRWZN2HAUSAZJL2VZKYALGSV67SP2A, seg_pos={0 0}, upload=274429]: Done uploading piece. err: ecclient: upload failed (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, address:45.91.116.11:4118): protocol: read tcp 188.166.225.170:48006->45.91.116.11:4118: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:28.184274337 (263h58m22.615354468s) [seg_pos={0 0}, upload=274429]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Aug 19 17:24:28.412483516 (263h58m22.843563647s) [upload=274429]: Done uploading segments... Aug 19 17:24:28.412520628 (263h58m22.843600759s) [upload=274429]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 6.45660975s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 2.008652228s FAIL (2025-08-19 17:51:01 UTC) orbiter warning: operation: ListObjects ======ERROR====== uplink: metainfo: metaclient: context deadline exceeded ======LOGS====== Aug 19 17:35:36.607320794 (264h9m31.038400925s) [upload=274632]: Starting upload Aug 19 17:35:36.607367684 (264h9m31.038447815s) [upload=274632]: Done starting upload Aug 19 17:35:36.607435082 (264h9m31.038515213s) [upload=274632]: Uploading segments... Aug 19 17:35:36.607490538 (264h9m31.038570669s) [upload=274632]: Got next segment. Inline: true Aug 19 17:35:36.607495088 (264h9m31.038575219s) [upload=274632]: Waiting for error group managing segments... Aug 19 17:35:36.607537528 (264h9m31.038617659s) [upload=274632]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 17:35:36.607541656 (264h9m31.038621787s) [upload=274632]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 19 17:35:36.607544051 (264h9m31.038624182s) [upload=274632]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 17:35:36.889055260 (264h9m31.320135391s) [upload=274632]: Done uploading segments... Aug 19 17:35:36.889082943 (264h9m31.320163124s) [upload=274632]: Upload finished. err: <nil> Duration:* 2.008678367s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 5.767283757s FAIL (2025-08-19 17:50:59 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Got next segment. Inline: false Aug 19 17:23:57.290467622 (263h57m51.721547763s) [upload=274428]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 17:23:57.290471560 (263h57m51.721551681s) [upload=274428]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 19 17:23:57.743765742 (263h57m52.174845863s) [seg_pos={0 0}, upload=274428]: Begin upload segment... Aug 19 17:23:57.743818412 (263h57m52.174898543s) [seg_pos={0 0}, upload=274428]: Done begin upload segment. Aug 19 17:23:57.743842609 (263h57m52.17492274s) [noise=true , address=77.90.31.16:4750 , piece_id=XXJ4VHMZ4NVVT2WX322JS7MW3FHQ7UZJW4VLR4KG6FW4A4RHVN6A, seg_pos={0 0}, upload=274428]: Uploading piece... Aug 19 17:23:57.743875672 (263h57m52.174955803s) [noise=true , address=45.91.116.14:4198 , piece_id=NSLSBMDS2ZJKN62N4NGN7HRHXAUIS4WL4AGHODZZBD2CDYTXHVUA, seg_pos={0 0}, upload=274428]: Uploading piece... Aug 19 17:23:57.744218498 (263h57m52.175298629s) [noise=true , address=77.90.31.14:4667 , piece_id=BF53QUGTGVD23KCIIAA5NWUCWKWUXXSHKCAAWSDZLAORPHXQEMQQ, seg_pos={0 0}, upload=274428]: Uploading piece... Aug 19 17:23:57.744456373 (263h57m52.175536504s) [noise=true , address=77.90.31.21:4883 , piece_id=SVFJCG4T5OCDMVKBE7A33XPDIQUH6NESE4GIVTETRZRBX4N3MJDQ, seg_pos={0 0}, upload=274428]: Uploading piece... Aug 19 17:23:57.744523051 (263h57m52.175603182s) [noise=true , address=104.36.82.60:4365 , piece_id=7BXP5MWOH2B3H74CPVWOQN2NEGMXUYECIVOEHRRKINQ544WKAAHA, seg_pos={0 0}, upload=274428]: Uploading piece... Aug 19 17:23:57.744762870 (263h57m52.175843001s) [noise=true , address=77.90.31.19:4851 , piece_id=POYQI453UA5767REISB5BNE3MYV6IOYZNIK357PYGH5DM66EOBIA, seg_pos={0 0}, upload=274428]: Uploading piece... Aug 19 17:23:57.744993531 (263h57m52.176073662s) [noise=true , address=104.36.82.44:4871 , piece_id=WQGOUIWA7NVK5SUTIWX6E32ZKW62Y2KGK7E3HB5YW2CKDUR42BPQ, seg_pos={0 0}, upload=274428]: Uploading piece... Aug 19 17:23:57.745229373 (263h57m52.176309494s) [noise=true , address=79.127.205.234:4587 , piece_id=ALBAMUJVGA4RIJKGCJNJT3BT6RBD5QWU4C33RJGCEP3XIQQLW6QQ, seg_pos={0 0}, upload=274428]: Uploading piece... Aug 19 17:23:57.745473950 (263h57m52.176554081s) [noise=true , address=104.36.82.3:4771 , piece_id=2RZZG2MORQZR3ROAADIWSK46JDEZSNYBGASCDWZTKWWU72JFHRSQ, seg_pos={0 0}, upload=274428]: Uploading piece... Aug 19 17:23:57.745746030 (263h57m52.176826161s) [noise=true , address=104.36.82.34:4558 , piece_id=AJNHQTDQSMJ3R2ZKVQUCBU5XOMWNF66ID2M3RVPYDL52F5LGTPFQ, seg_pos={0 0}, upload=274428]: Uploading piece... Aug 19 17:23:57.745999426 (263h57m52.177079547s) [noise=true , address=77.90.31.18:4814 , piece_id=VUIFMMFA5PCHIKNNX7PA7RHVXIJHK3UWH4SI34R6XP5MEYTAMS6A, seg_pos={0 0}, upload=274428]: Uploading piece... Aug 19 17:23:57.746774839 (263h57m52.17785497s) [noise=true , address=104.36.82.35:4610 , piece_id=DM7HADQAK5T67ZDQR3VGUUGLHCGBLC5KJYYFF7QQNRKVE2SA5CHA, seg_pos={0 0}, upload=274428]: Uploading piece... Aug 19 17:23:57.747503714 (263h57m52.178583836s) [noise=true , address=64.49.13.137:4496 , piece_id=DW5YMZIQJZIA53K22HGK2ICRT5IVVP2ZMIVYM46BWQWAYPLNUC2Q, seg_pos={0 0}, upload=274428]: Uploading piece... Aug 19 17:23:57.747727743 (263h57m52.178807874s) [noise=true , address=77.90.31.13:4640 , piece_id=NKLBPZJDAIAVAYH3QRDCPS6VWHJBFZZWY3HCTSW2E4II77YC3BYA, seg_pos={0 0}, upload=274428]: Uploading piece... Aug 19 17:23:57.747948565 (263h57m52.179028696s) [noise=true , address=104.36.82.55:4214 , piece_id=64P34K2YEWCWZFGWSR4AQ5SRALT5CSZCB4F6FHPKDP3ZFJIOJC2Q, seg_pos={0 0}, upload=274428]: Uploading piece... Aug 19 17:23:57.748174007 (263h57m52.179254138s) [noise=true , address=104.36.82.23:4448 , piece_id=SCMNBDCYA7PUN2VP6JN37YM5LNY2GBGQHJKYSSWUWMPTZZ7LXZUQ, seg_pos={0 0}, upload=274428]: Uploading piece... Aug 19 17:23:57.748433112 (263h57m52.179513233s) [noise=true , address=104.36.82.5:4854 , piece_id=FC4DIKMBI5OPLDQCVA6P3MA7AQZV23QBIX3UAKLWJJMY724TNYHQ, seg_pos={0 0}, upload=274428]: Uploading piece... Aug 19 17:23:57.748647252 (263h57m52.179727383s) [noise=true , address=104.36.82.62:4442 , piece_id=VV5BBYB6UI4O3B4ESOBJX44I5RJBRWSLJUFR6H3GKEUZWNONTPMA, seg_pos={0 0}, upload=274428]: Uploading piece... Aug 19 17:23:58.869276532 (263h57m53.300356663s) [upload=274428]: Next returned nil segment Aug 19 17:23:58.869291511 (263h57m53.300371642s) [upload=274428]: Waiting for error group managing segments... Aug 19 17:23:59.195579562 (263h57m53.626659683s) [noise=true , address=79.127.205.234:4587 , piece_id=ALBAMUJVGA4RIJKGCJNJT3BT6RBD5QWU4C33RJGCEP3XIQQLW6QQ, seg_pos={0 0}, upload=274428]: Done uploading piece. err: <nil> Aug 19 17:23:59.239543682 (263h57m53.670623813s) [noise=true , address=64.49.13.137:4496 , piece_id=DW5YMZIQJZIA53K22HGK2ICRT5IVVP2ZMIVYM46BWQWAYPLNUC2Q, seg_pos={0 0}, upload=274428]: Done uploading piece. err: <nil> Aug 19 17:23:59.654494638 (263h57m54.085574769s) [noise=true , address=104.36.82.35:4610 , piece_id=DM7HADQAK5T67ZDQR3VGUUGLHCGBLC5KJYYFF7QQNRKVE2SA5CHA, seg_pos={0 0}, upload=274428]: Done uploading piece. err: <nil> Aug 19 17:23:59.700507838 (263h57m54.131587959s) [noise=true , address=104.36.82.34:4558 , piece_id=AJNHQTDQSMJ3R2ZKVQUCBU5XOMWNF66ID2M3RVPYDL52F5LGTPFQ, seg_pos={0 0}, upload=274428]: Done uploading piece. err: <nil> Aug 19 17:23:59.757508140 (263h57m54.188588272s) [noise=true , address=104.36.82.5:4854 , piece_id=FC4DIKMBI5OPLDQCVA6P3MA7AQZV23QBIX3UAKLWJJMY724TNYHQ, seg_pos={0 0}, upload=274428]: Done uploading piece. err: <nil> Aug 19 17:23:59.765247029 (263h57m54.19632716s) [noise=true , address=104.36.82.62:4442 , piece_id=VV5BBYB6UI4O3B4ESOBJX44I5RJBRWSLJUFR6H3GKEUZWNONTPMA, seg_pos={0 0}, upload=274428]: Done uploading piece. err: <nil> Aug 19 17:23:59.774367630 (263h57m54.205447761s) [noise=true , address=104.36.82.23:4448 , piece_id=SCMNBDCYA7PUN2VP6JN37YM5LNY2GBGQHJKYSSWUWMPTZZ7LXZUQ, seg_pos={0 0}, upload=274428]: Done uploading piece. err: <nil> Aug 19 17:23:59.827970201 (263h57m54.259050332s) [noise=true , address=104.36.82.55:4214 , piece_id=64P34K2YEWCWZFGWSR4AQ5SRALT5CSZCB4F6FHPKDP3ZFJIOJC2Q, seg_pos={0 0}, upload=274428]: Done uploading piece. err: <nil> Aug 19 17:23:59.851412395 (263h57m54.282492526s) [noise=true , address=104.36.82.44:4871 , piece_id=WQGOUIWA7NVK5SUTIWX6E32ZKW62Y2KGK7E3HB5YW2CKDUR42BPQ, seg_pos={0 0}, upload=274428]: Done uploading piece. err: <nil> Aug 19 17:23:59.946562012 (263h57m54.377642143s) [noise=true , address=104.36.82.3:4771 , piece_id=2RZZG2MORQZR3ROAADIWSK46JDEZSNYBGASCDWZTKWWU72JFHRSQ, seg_pos={0 0}, upload=274428]: Done uploading piece. err: <nil> Aug 19 17:23:59.979671513 (263h57m54.410751634s) [noise=true , address=104.36.82.60:4365 , piece_id=7BXP5MWOH2B3H74CPVWOQN2NEGMXUYECIVOEHRRKINQ544WKAAHA, seg_pos={0 0}, upload=274428]: Done uploading piece. err: <nil> Aug 19 17:24:55.494274881 (263h58m49.925355012s) [noise=true , address=77.90.31.18:4814 , piece_id=VUIFMMFA5PCHIKNNX7PA7RHVXIJHK3UWH4SI34R6XP5MEYTAMS6A, seg_pos={0 0}, upload=274428]: Done uploading piece. err: <nil> Aug 19 17:24:55.842835997 (263h58m50.273916128s) [noise=true , address=77.90.31.21:4883 , piece_id=SVFJCG4T5OCDMVKBE7A33XPDIQUH6NESE4GIVTETRZRBX4N3MJDQ, seg_pos={0 0}, upload=274428]: Done uploading piece. err: <nil> Aug 19 17:24:56.634889530 (263h58m51.065969661s) [noise=true , address=77.90.31.19:4851 , piece_id=POYQI453UA5767REISB5BNE3MYV6IOYZNIK357PYGH5DM66EOBIA, seg_pos={0 0}, upload=274428]: Done uploading piece. err: ecclient: upload failed (node:1NEV2ndHw4fohVUNsfrzwojd8S11TmMST6Sz59GVnXS8vDGhWk, address:77.90.31.19:4851): protocol: rpc: dial tcp 77.90.31.19:4851: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:56.635097748 (263h58m51.066177879s) [noise=true , address=45.91.116.14:4198 , piece_id=NSLSBMDS2ZJKN62N4NGN7HRHXAUIS4WL4AGHODZZBD2CDYTXHVUA, seg_pos={0 0}, upload=274428]: Done uploading piece. err: ecclient: upload failed (node:1xxfUrCm5PgSLpge1wYiH2EcE7kzzJ38P9rfeCYmqXTbTisioi, address:45.91.116.14:4198): protocol: rpc: dial tcp 45.91.116.14:4198: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:56.635158895 (263h58m51.066239036s) [noise=true , address=77.90.31.16:4750 , piece_id=XXJ4VHMZ4NVVT2WX322JS7MW3FHQ7UZJW4VLR4KG6FW4A4RHVN6A, seg_pos={0 0}, upload=274428]: Done uploading piece. err: ecclient: upload failed (node:12TGd8NnWdekDtHzBQHmJPgTtHjXC6g8fi6RoZYFQJVHDa5hdyE, address:77.90.31.16:4750): protocol: read tcp 188.166.225.170:54198->77.90.31.16:4750: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:56.635224120 (263h58m51.066304281s) [noise=true , address=77.90.31.14:4667 , piece_id=BF53QUGTGVD23KCIIAA5NWUCWKWUXXSHKCAAWSDZLAORPHXQEMQQ, seg_pos={0 0}, upload=274428]: Done uploading piece. err: ecclient: upload failed (node:12R5gJFhsmocYPNZf4yNnYqJwcwAKt9yiHmmqGMk3btzVjkFV4S, address:77.90.31.14:4667): protocol: read tcp 188.166.225.170:46984->77.90.31.14:4667: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:56.635280057 (263h58m51.066360198s) [noise=true , address=77.90.31.13:4640 , piece_id=NKLBPZJDAIAVAYH3QRDCPS6VWHJBFZZWY3HCTSW2E4II77YC3BYA, seg_pos={0 0}, upload=274428]: Done uploading piece. err: ecclient: upload failed (node:1XjJpLBEZm5daHcBUmUyC7QpLVhu9Vm9ex9RYcYJWEuLGk527F, address:77.90.31.13:4640): protocol: read tcp 188.166.225.170:48656->77.90.31.13:4640: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:56.635314653 (263h58m51.066394794s) [seg_pos={0 0}, upload=274428]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Aug 19 17:24:56.862515362 (263h58m51.293595493s) [upload=274428]: Done uploading segments... Aug 19 17:24:56.862536042 (263h58m51.293616163s) [upload=274428]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.767391152s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 7 out of 60 is failed (threshold=3) [...X.............X......X.X........X....XX..................] duration: 5.744296973s FAIL (2025-08-19 17:50:57 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/4M") ======LOGS====== Aug 19 17:38:09.246414094 (264h12m3.677494255s) [upload=274674]: Starting upload Aug 19 17:38:09.246486392 (264h12m3.677566533s) [upload=274674]: Done starting upload Aug 19 17:38:09.246601452 (264h12m3.677681583s) [upload=274674]: Uploading segments... Aug 19 17:38:09.246691885 (264h12m3.677772016s) [upload=274674]: Got next segment. Inline: false Aug 19 17:38:09.246724627 (264h12m3.677804758s) [upload=274674]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 17:38:09.246730909 (264h12m3.67781104s) [upload=274674]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 19 17:38:09.693242556 (264h12m4.124322687s) [seg_pos={0 0}, upload=274674]: Begin upload segment... Aug 19 17:38:09.693302230 (264h12m4.124382361s) [seg_pos={0 0}, upload=274674]: Done begin upload segment. Aug 19 17:38:09.693343529 (264h12m4.12442367s) [noise=true , address=77.90.31.18:4825 , piece_id=UQLJVRVUBBN5LFBIAQL7YQ4IRAWI3SBVBHUT4IIFBROAUHXHVZ3A, seg_pos={0 0}, upload=274674]: Uploading piece... Aug 19 17:38:09.693350092 (264h12m4.124430213s) [noise=true , address=45.91.116.14:4222 , piece_id=DXHEK43V7USJLYKPF5PKEHLJPVP4OPH2H3TQVGUBMK5MVRWQHXJQ, seg_pos={0 0}, upload=274674]: Uploading piece... Aug 19 17:38:09.693395940 (264h12m4.124476081s) [noise=true , address=45.91.116.11:4140 , piece_id=KQNAZLV4XPDYYLJQUMI2UL3GVLA7QTEZ6TU3HDYFYB7LPRBSRQFA, seg_pos={0 0}, upload=274674]: Uploading piece... Aug 19 17:38:09.693960980 (264h12m4.125041131s) [noise=true , address=79.127.201.215:4343 , piece_id=MIATMAVPWKBJPVYH3XDYXQGKW7VWKRULLP6UV2ZNKJZ7YDVJJEBQ, seg_pos={0 0}, upload=274674]: Uploading piece... Aug 19 17:38:09.694358652 (264h12m4.125438783s) [noise=true , address=104.36.82.5:4852 , piece_id=UALXJPWV34HHWPC6GNCOFNP2RDC3KF6ZY4FUHLL3MWR2UZDCXNCQ, seg_pos={0 0}, upload=274674]: Uploading piece... Aug 19 17:38:09.694689475 (264h12m4.125769606s) [noise=true , address=104.36.82.58:4269 , piece_id=CTQSY5PQFUNXUJR2JXSI7O2EHMYTOAHUSHGTMVTI42E45RYFIGJA, seg_pos={0 0}, upload=274674]: Uploading piece... Aug 19 17:38:09.695011601 (264h12m4.126091732s) [noise=true , address=104.36.82.60:4364 , piece_id=A5BGK2CC62VYRH27SR2MONG2GPSZ5WONCHAF42LFPXOTM2W7XIJA, seg_pos={0 0}, upload=274674]: Uploading piece... Aug 19 17:38:09.695441633 (264h12m4.126521764s) [noise=true , address=104.36.82.9:4028 , piece_id=4LUYCAISMQC3SGDLJZSDFDJET4R54U3N4B6ZAJ35UDXB5CT3GV2Q, seg_pos={0 0}, upload=274674]: Uploading piece... Aug 19 17:38:09.695845597 (264h12m4.126925728s) [noise=true , address=77.90.31.15:4701 , piece_id=CCTT764Z3HFZFT4PE7YOOBVVGDASLFYVFGVXGWADBLDLURBOKGGQ, seg_pos={0 0}, upload=274674]: Uploading piece... Aug 19 17:38:09.696191328 (264h12m4.127271459s) [noise=true , address=45.91.116.5:4302 , piece_id=D2ZYYIFLA7GFPEII4X75HVJGL5DOM6BZLQFXL7HYVXYLNCL4CDQQ, seg_pos={0 0}, upload=274674]: Uploading piece... Aug 19 17:38:09.696539624 (264h12m4.127619755s) [noise=true , address=77.90.31.14:4689 , piece_id=6FO4WN5BXNW6UFGW2ALS5YBYBP5FZCAZBLQZXAULFCN65NHH7N6A, seg_pos={0 0}, upload=274674]: Uploading piece... Aug 19 17:38:09.696809421 (264h12m4.127889552s) [noise=true , address=79.127.201.214:4299 , piece_id=C4JSI4OPEZPPHADT6QTVJGEXQNVSHEK42KNYQBV3S2OH7Z5Z7HHA, seg_pos={0 0}, upload=274674]: Uploading piece... Aug 19 17:38:09.697172445 (264h12m4.128252576s) [noise=true , address=104.36.82.43:4855 , piece_id=3H2PUTGFVC2PR224NBHGRB7GLWBOL2MNXXWJ7JF23HYVF53TEWFQ, seg_pos={0 0}, upload=274674]: Uploading piece... Aug 19 17:38:09.697485064 (264h12m4.128565205s) [noise=true , address=77.90.31.12:4619 , piece_id=XMWKT6JBVLOPTRUIPNEK4GXVCH5OOVO5WYWDGWMXGA2VVVD74CBA, seg_pos={0 0}, upload=274674]: Uploading piece... Aug 19 17:38:09.697774858 (264h12m4.128854989s) [noise=true , address=104.36.82.38:4714 , piece_id=SZ42LIQE4LXTBCZY4BTXWS3EL27UYIQS3AAXGWNLH65TUZWNUYSQ, seg_pos={0 0}, upload=274674]: Uploading piece... Aug 19 17:38:09.698096103 (264h12m4.129176234s) [noise=true , address=104.36.82.4:4817 , piece_id=JNKPBDLUZUWI22ZLGTCD3BZJBAUX7TZVAGXJVFZOEJPK2Z3PMADA, seg_pos={0 0}, upload=274674]: Uploading piece... Aug 19 17:38:09.698352945 (264h12m4.129433076s) [noise=true , address=45.91.116.13:4183 , piece_id=OMUE6JFSTRIF64VYM5XCHNL7U26SEXP3IGWDBHP5G2E6YJQETEUQ, seg_pos={0 0}, upload=274674]: Uploading piece... Aug 19 17:38:09.698876106 (264h12m4.129956227s) [noise=true , address=104.36.82.6:4879 , piece_id=JKK5HMPYTMR7PNWHVPBDHPH3K6H7VM7AA4K4X4WXQ5LN57VRBUFA, seg_pos={0 0}, upload=274674]: Uploading piece... Aug 19 17:38:11.192631202 (264h12m5.623711333s) [upload=274674]: Next returned nil segment Aug 19 17:38:11.192648606 (264h12m5.623728737s) [upload=274674]: Waiting for error group managing segments... Aug 19 17:38:11.591537039 (264h12m6.02261717s) [noise=true , address=45.91.116.14:4222 , piece_id=DXHEK43V7USJLYKPF5PKEHLJPVP4OPH2H3TQVGUBMK5MVRWQHXJQ, seg_pos={0 0}, upload=274674]: Done uploading piece. err: <nil> Aug 19 17:38:11.606093581 (264h12m6.037173712s) [noise=true , address=77.90.31.14:4689 , piece_id=6FO4WN5BXNW6UFGW2ALS5YBYBP5FZCAZBLQZXAULFCN65NHH7N6A, seg_pos={0 0}, upload=274674]: Done uploading piece. err: <nil> Aug 19 17:38:11.610339916 (264h12m6.041420047s) [noise=true , address=77.90.31.15:4701 , piece_id=CCTT764Z3HFZFT4PE7YOOBVVGDASLFYVFGVXGWADBLDLURBOKGGQ, seg_pos={0 0}, upload=274674]: Done uploading piece. err: <nil> Aug 19 17:38:11.630055040 (264h12m6.061135171s) [noise=true , address=79.127.201.215:4343 , piece_id=MIATMAVPWKBJPVYH3XDYXQGKW7VWKRULLP6UV2ZNKJZ7YDVJJEBQ, seg_pos={0 0}, upload=274674]: Done uploading piece. err: <nil> Aug 19 17:38:11.678703675 (264h12m6.109783797s) [noise=true , address=104.36.82.9:4028 , piece_id=4LUYCAISMQC3SGDLJZSDFDJET4R54U3N4B6ZAJ35UDXB5CT3GV2Q, seg_pos={0 0}, upload=274674]: Done uploading piece. err: <nil> Aug 19 17:38:11.693543009 (264h12m6.12462314s) [noise=true , address=104.36.82.60:4364 , piece_id=A5BGK2CC62VYRH27SR2MONG2GPSZ5WONCHAF42LFPXOTM2W7XIJA, seg_pos={0 0}, upload=274674]: Done uploading piece. err: <nil> Aug 19 17:38:11.693568448 (264h12m6.124648579s) [noise=true , address=45.91.116.13:4183 , piece_id=OMUE6JFSTRIF64VYM5XCHNL7U26SEXP3IGWDBHP5G2E6YJQETEUQ, seg_pos={0 0}, upload=274674]: Done uploading piece. err: <nil> Aug 19 17:38:11.698831207 (264h12m6.129911329s) [noise=true , address=104.36.82.5:4852 , piece_id=UALXJPWV34HHWPC6GNCOFNP2RDC3KF6ZY4FUHLL3MWR2UZDCXNCQ, seg_pos={0 0}, upload=274674]: Done uploading piece. err: <nil> Aug 19 17:38:11.698886834 (264h12m6.129966955s) [noise=true , address=77.90.31.12:4619 , piece_id=XMWKT6JBVLOPTRUIPNEK4GXVCH5OOVO5WYWDGWMXGA2VVVD74CBA, seg_pos={0 0}, upload=274674]: Done uploading piece. err: <nil> Aug 19 17:38:11.714704781 (264h12m6.145784912s) [noise=true , address=79.127.201.214:4299 , piece_id=C4JSI4OPEZPPHADT6QTVJGEXQNVSHEK42KNYQBV3S2OH7Z5Z7HHA, seg_pos={0 0}, upload=274674]: Done uploading piece. err: <nil> Aug 19 17:38:11.720319795 (264h12m6.151399956s) [noise=true , address=45.91.116.11:4140 , piece_id=KQNAZLV4XPDYYLJQUMI2UL3GVLA7QTEZ6TU3HDYFYB7LPRBSRQFA, seg_pos={0 0}, upload=274674]: Done uploading piece. err: <nil> Aug 19 17:38:11.722063401 (264h12m6.153143532s) [noise=true , address=104.36.82.6:4879 , piece_id=JKK5HMPYTMR7PNWHVPBDHPH3K6H7VM7AA4K4X4WXQ5LN57VRBUFA, seg_pos={0 0}, upload=274674]: Done uploading piece. err: <nil> Aug 19 17:38:11.726762272 (264h12m6.157842393s) [noise=true , address=104.36.82.38:4714 , piece_id=SZ42LIQE4LXTBCZY4BTXWS3EL27UYIQS3AAXGWNLH65TUZWNUYSQ, seg_pos={0 0}, upload=274674]: Done uploading piece. err: <nil> Aug 19 17:38:11.734258877 (264h12m6.165339008s) [noise=true , address=104.36.82.58:4269 , piece_id=CTQSY5PQFUNXUJR2JXSI7O2EHMYTOAHUSHGTMVTI42E45RYFIGJA, seg_pos={0 0}, upload=274674]: Done uploading piece. err: <nil> Aug 19 17:38:11.734284346 (264h12m6.165364477s) [seg_pos={0 0}, upload=274674]: Segment reached optimal threshold of 14 pieces. Aug 19 17:38:11.734416277 (264h12m6.165496408s) [noise=true , address=104.36.82.4:4817 , piece_id=JNKPBDLUZUWI22ZLGTCD3BZJBAUX7TZVAGXJVFZOEJPK2Z3PMADA, seg_pos={0 0}, upload=274674]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RSBodzJxMye65QgkyrzkWCKQ49cnfdu4haUZKW23nWpqzMcYz): protocol: expected piece hash; context canceled; EOF Aug 19 17:38:11.734486873 (264h12m6.165567004s) [noise=true , address=104.36.82.43:4855 , piece_id=3H2PUTGFVC2PR224NBHGRB7GLWBOL2MNXXWJ7JF23HYVF53TEWFQ, seg_pos={0 0}, upload=274674]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TGTRhEi2KsbrkC6ZmgGkMaJri9uhzEcmXe1J4nJjpEd1BE4ka): protocol: expected piece hash; context canceled; EOF Aug 19 17:38:11.734587926 (264h12m6.165668057s) [noise=true , address=77.90.31.18:4825 , piece_id=UQLJVRVUBBN5LFBIAQL7YQ4IRAWI3SBVBHUT4IIFBROAUHXHVZ3A, seg_pos={0 0}, upload=274674]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PqKyfFxfVKTGARbPqBcQL3Naqjdf5uR44PeLi3sJpu57dcm1x): protocol: expected piece hash; context canceled; EOF Aug 19 17:38:11.734592956 (264h12m6.165673087s) [noise=true , address=45.91.116.5:4302 , piece_id=D2ZYYIFLA7GFPEII4X75HVJGL5DOM6BZLQFXL7HYVXYLNCL4CDQQ, seg_pos={0 0}, upload=274674]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pJWhSLza8WaMuvvDwH4nae57UxQmfCdDvAkQ3v25MgPYHoZXx): protocol: expected piece hash; context canceled; EOF Aug 19 17:38:11.734645206 (264h12m6.165725337s) [seg_pos={0 0}, upload=274674]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 19 17:38:11.734756098 (264h12m6.165836239s) [upload=274674]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 19 17:38:11.734764033 (264h12m6.165844154s) [upload=274674]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 17:38:12.023665554 (264h12m6.454745685s) [upload=274674]: Done uploading segments... Aug 19 17:38:12.023686044 (264h12m6.454766195s) [upload=274674]: Upload finished. err: <nil> Duration: 5.744323142s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [................X.....X.....................................] duration: 6.080343479s FAIL (2025-08-19 17:50:39 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.080365932s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..................X...X.....................................] duration: 5.770841412s FAIL (2025-08-19 17:50:28 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 23:59.057943072 (263h57m53.489023273s) [noise=true , address=45.91.116.10:4083 , piece_id=V3H4T2WN2MBVALSIHZNOQHBE75WQRNYLIE3Z7DFWUUHBPYDT7IFA, seg_pos={0 0}, upload=274429]: Uploading piece... Aug 19 17:23:59.057955436 (263h57m53.489035557s) [noise=true , address=45.91.116.5:4298 , piece_id=HMF6DXO5KFMLK67WVA7BAWLQFLC2BCVPJ7ZT255FMRCCEI3MCBNA, seg_pos={0 0}, upload=274429]: Uploading piece... Aug 19 17:23:59.058557689 (263h57m53.48963782s) [noise=true , address=104.36.82.8:4969 , piece_id=3P655Z5OUVZ4VRESACEBKYR3KUFYY56NYPWX7ZVOHL3XI2YQ5TCQ, seg_pos={0 0}, upload=274429]: Uploading piece... Aug 19 17:23:59.059170771 (263h57m53.490250902s) [noise=true , address=45.91.116.7:4009 , piece_id=NSZYKL6FHDLDM463ZXKEQGX4ZY7JPJXENH63PPDV76XMGCBV7ZRQ, seg_pos={0 0}, upload=274429]: Uploading piece... Aug 19 17:23:59.059468281 (263h57m53.490548412s) [noise=true , address=77.90.31.15:4699 , piece_id=MFF7PNMDLHYEH7PY46MGGOJSY4XPHRDPGFWJABBO4JDTJJJJ3CTQ, seg_pos={0 0}, upload=274429]: Uploading piece... Aug 19 17:23:59.059498208 (263h57m53.490578339s) [noise=true , address=104.36.82.2:4729 , piece_id=BNX6UNYZ7VHQ4MKAH65DGSHULWRRXDSCMKOE7WFZO6T2V3ELBJ7Q, seg_pos={0 0}, upload=274429]: Uploading piece... Aug 19 17:23:59.059814182 (263h57m53.490894323s) [noise=true , address=104.36.82.6:4874 , piece_id=QMGNWSGMEV762Y2XEDK2LIAY3JMNLD3UOVFQ6HYP7ZMLLVAVO7ZQ, seg_pos={0 0}, upload=274429]: Uploading piece... Aug 19 17:23:59.059839941 (263h57m53.490920072s) [noise=true , address=45.91.116.12:4225 , piece_id=PCJYR2VLFH23UPQEABUYESR3MHKDHIVTJWDIWVYIL6JAFGGXXXFQ, seg_pos={0 0}, upload=274429]: Uploading piece... Aug 19 17:23:59.059875480 (263h57m53.490955611s) [noise=true , address=104.36.82.10:4079 , piece_id=S3BSVJWRPP2PDGLG7FKFROUKAM3GTBDL2ALAJPAZQQQZJVRMMZ3Q, seg_pos={0 0}, upload=274429]: Uploading piece... Aug 19 17:23:59.060109147 (263h57m53.491189278s) [noise=true , address=104.36.82.2:4732 , piece_id=NLTK4572JAQHZQRDD2DOOIW7VBO7NK4DI4RUZRSXQ3GR72EWV3KQ, seg_pos={0 0}, upload=274429]: Uploading piece... Aug 19 17:23:59.060142190 (263h57m53.491222321s) [noise=true , address=45.91.116.9:4261 , piece_id=J5IMFL3JW4YKDLI6OEY2POEB4LJR2P3BIFAD4YIF3E4JTUXIXY3A, seg_pos={0 0}, upload=274429]: Uploading piece... Aug 19 17:23:59.060348274 (263h57m53.491428405s) [noise=true , address=104.36.82.15:4259 , piece_id=RM7JL7JKCMFM4CNXVVZQ2QJTOEPQEQHGLW3GYXWUD6LAQ2ZJ3P5Q, seg_pos={0 0}, upload=274429]: Uploading piece... Aug 19 17:23:59.060544830 (263h57m53.491624961s) [noise=true , address=45.91.116.14:4191 , piece_id=PJ33GP36S76VCI5RVR26PBFZ6HZUUZYVWNGPJITRLGDZI6LWCJIQ, seg_pos={0 0}, upload=274429]: Uploading piece... Aug 19 17:23:59.060752227 (263h57m53.491832358s) [noise=true , address=45.91.116.11:4118 , piece_id=VD2HK2QU63UVJFRASQOIRFZRWZN2HAUSAZJL2VZKYALGSV67SP2A, seg_pos={0 0}, upload=274429]: Uploading piece... Aug 19 17:23:59.060967990 (263h57m53.492048121s) [noise=true , address=77.90.31.11:4555 , piece_id=GTU6PGCHI4WOFGK2B3FP2Y22VUDBWLQKKEMZBT6I4F47OKK4RPNQ, seg_pos={0 0}, upload=274429]: Uploading piece... Aug 19 17:23:59.061294044 (263h57m53.492374175s) [noise=true , address=45.91.116.8:4048 , piece_id=M2CBILQKSVSQ7VY4AWQQM5GHF2KYBSXH73YJGAEJZAHOR7FOBWEQ, seg_pos={0 0}, upload=274429]: Uploading piece... Aug 19 17:24:00.872920400 (263h57m55.304000531s) [upload=274429]: Next returned nil segment Aug 19 17:24:00.872945548 (263h57m55.304025679s) [upload=274429]: Waiting for error group managing segments... Aug 19 17:24:01.156214888 (263h57m55.587295019s) [noise=true , address=104.36.82.6:4874 , piece_id=QMGNWSGMEV762Y2XEDK2LIAY3JMNLD3UOVFQ6HYP7ZMLLVAVO7ZQ, seg_pos={0 0}, upload=274429]: Done uploading piece. err: <nil> Aug 19 17:24:01.181164926 (263h57m55.612245057s) [noise=true , address=104.36.82.2:4732 , piece_id=NLTK4572JAQHZQRDD2DOOIW7VBO7NK4DI4RUZRSXQ3GR72EWV3KQ, seg_pos={0 0}, upload=274429]: Done uploading piece. err: <nil> Aug 19 17:24:01.222047199 (263h57m55.653127331s) [noise=true , address=104.36.82.2:4729 , piece_id=BNX6UNYZ7VHQ4MKAH65DGSHULWRRXDSCMKOE7WFZO6T2V3ELBJ7Q, seg_pos={0 0}, upload=274429]: Done uploading piece. err: <nil> Aug 19 17:24:01.250473409 (263h57m55.68155354s) [noise=true , address=104.36.82.15:4259 , piece_id=RM7JL7JKCMFM4CNXVVZQ2QJTOEPQEQHGLW3GYXWUD6LAQ2ZJ3P5Q, seg_pos={0 0}, upload=274429]: Done uploading piece. err: <nil> Aug 19 17:24:01.259222299 (263h57m55.69030242s) [noise=true , address=104.36.82.8:4969 , piece_id=3P655Z5OUVZ4VRESACEBKYR3KUFYY56NYPWX7ZVOHL3XI2YQ5TCQ, seg_pos={0 0}, upload=274429]: Done uploading piece. err: <nil> Aug 19 17:24:01.261735818 (263h57m55.692815949s) [noise=true , address=104.36.82.10:4079 , piece_id=S3BSVJWRPP2PDGLG7FKFROUKAM3GTBDL2ALAJPAZQQQZJVRMMZ3Q, seg_pos={0 0}, upload=274429]: Done uploading piece. err: <nil> Aug 19 17:24:28.183964515 (263h58m22.615044646s) [noise=true , address=45.91.116.14:4191 , piece_id=PJ33GP36S76VCI5RVR26PBFZ6HZUUZYVWNGPJITRLGDZI6LWCJIQ, seg_pos={0 0}, upload=274429]: Done uploading piece. err: ecclient: upload failed (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, address:45.91.116.14:4191): protocol: read tcp 188.166.225.170:37646->45.91.116.14:4191: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:28.184013127 (263h58m22.615093258s) [noise=true , address=45.91.116.9:4261 , piece_id=J5IMFL3JW4YKDLI6OEY2POEB4LJR2P3BIFAD4YIF3E4JTUXIXY3A, seg_pos={0 0}, upload=274429]: Done uploading piece. err: ecclient: upload failed (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, address:45.91.116.9:4261): protocol: rpc: dial tcp 45.91.116.9:4261: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:28.184026694 (263h58m22.615106825s) [noise=true , address=45.91.116.8:4048 , piece_id=M2CBILQKSVSQ7VY4AWQQM5GHF2KYBSXH73YJGAEJZAHOR7FOBWEQ, seg_pos={0 0}, upload=274429]: Done uploading piece. err: ecclient: upload failed (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, address:45.91.116.8:4048): protocol: read tcp 188.166.225.170:45604->45.91.116.8:4048: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:28.184070568 (263h58m22.615150689s) [noise=true , address=45.91.116.5:4298 , piece_id=HMF6DXO5KFMLK67WVA7BAWLQFLC2BCVPJ7ZT255FMRCCEI3MCBNA, seg_pos={0 0}, upload=274429]: Done uploading piece. err: ecclient: upload failed (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, address:45.91.116.5:4298): protocol: read tcp 188.166.225.170:46970->45.91.116.5:4298: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:28.184103010 (263h58m22.615183141s) [noise=true , address=45.91.116.10:4083 , piece_id=V3H4T2WN2MBVALSIHZNOQHBE75WQRNYLIE3Z7DFWUUHBPYDT7IFA, seg_pos={0 0}, upload=274429]: Done uploading piece. err: ecclient: upload failed (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, address:45.91.116.10:4083): protocol: read tcp 188.166.225.170:41632->45.91.116.10:4083: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:28.184139489 (263h58m22.61521962s) [noise=true , address=45.91.116.7:4009 , piece_id=NSZYKL6FHDLDM463ZXKEQGX4ZY7JPJXENH63PPDV76XMGCBV7ZRQ, seg_pos={0 0}, upload=274429]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: read tcp 188.166.225.170:45756->45.91.116.7:4009: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:28.184171009 (263h58m22.615251141s) [noise=true , address=77.90.31.11:4555 , piece_id=GTU6PGCHI4WOFGK2B3FP2Y22VUDBWLQKKEMZBT6I4F47OKK4RPNQ, seg_pos={0 0}, upload=274429]: Done uploading piece. err: ecclient: upload failed (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, address:77.90.31.11:4555): protocol: read tcp 188.166.225.170:37438->77.90.31.11:4555: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:28.184197480 (263h58m22.615277611s) [noise=true , address=77.90.31.15:4699 , piece_id=MFF7PNMDLHYEH7PY46MGGOJSY4XPHRDPGFWJABBO4JDTJJJJ3CTQ, seg_pos={0 0}, upload=274429]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: read tcp 188.166.225.170:36788->77.90.31.15:4699: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:28.184218410 (263h58m22.615298541s) [noise=true , address=45.91.116.12:4225 , piece_id=PCJYR2VLFH23UPQEABUYESR3MHKDHIVTJWDIWVYIL6JAFGGXXXFQ, seg_pos={0 0}, upload=274429]: Done uploading piece. err: ecclient: upload failed (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, address:45.91.116.12:4225): protocol: read tcp 188.166.225.170:60636->45.91.116.12:4225: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:28.184249710 (263h58m22.615329841s) [noise=true , address=45.91.116.11:4118 , piece_id=VD2HK2QU63UVJFRASQOIRFZRWZN2HAUSAZJL2VZKYALGSV67SP2A, seg_pos={0 0}, upload=274429]: Done uploading piece. err: ecclient: upload failed (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, address:45.91.116.11:4118): protocol: read tcp 188.166.225.170:48006->45.91.116.11:4118: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:28.184274337 (263h58m22.615354468s) [seg_pos={0 0}, upload=274429]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Aug 19 17:24:28.412483516 (263h58m22.843563647s) [upload=274429]: Done uploading segments... Aug 19 17:24:28.412520628 (263h58m22.843600759s) [upload=274429]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 5.770868532s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 7 out of 60 is failed (threshold=3) [....X.............X......X.X........X....XX.................] duration: 5.750870694s FAIL (2025-08-19 17:50:16 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/4M") ======LOGS====== Aug 19 17:38:09.246414094 (264h12m3.677494255s) [upload=274674]: Starting upload Aug 19 17:38:09.246486392 (264h12m3.677566533s) [upload=274674]: Done starting upload Aug 19 17:38:09.246601452 (264h12m3.677681583s) [upload=274674]: Uploading segments... Aug 19 17:38:09.246691885 (264h12m3.677772016s) [upload=274674]: Got next segment. Inline: false Aug 19 17:38:09.246724627 (264h12m3.677804758s) [upload=274674]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 17:38:09.246730909 (264h12m3.67781104s) [upload=274674]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 19 17:38:09.693242556 (264h12m4.124322687s) [seg_pos={0 0}, upload=274674]: Begin upload segment... Aug 19 17:38:09.693302230 (264h12m4.124382361s) [seg_pos={0 0}, upload=274674]: Done begin upload segment. Aug 19 17:38:09.693343529 (264h12m4.12442367s) [noise=true , address=77.90.31.18:4825 , piece_id=UQLJVRVUBBN5LFBIAQL7YQ4IRAWI3SBVBHUT4IIFBROAUHXHVZ3A, seg_pos={0 0}, upload=274674]: Uploading piece... Aug 19 17:38:09.693350092 (264h12m4.124430213s) [noise=true , address=45.91.116.14:4222 , piece_id=DXHEK43V7USJLYKPF5PKEHLJPVP4OPH2H3TQVGUBMK5MVRWQHXJQ, seg_pos={0 0}, upload=274674]: Uploading piece... Aug 19 17:38:09.693395940 (264h12m4.124476081s) [noise=true , address=45.91.116.11:4140 , piece_id=KQNAZLV4XPDYYLJQUMI2UL3GVLA7QTEZ6TU3HDYFYB7LPRBSRQFA, seg_pos={0 0}, upload=274674]: Uploading piece... Aug 19 17:38:09.693960980 (264h12m4.125041131s) [noise=true , address=79.127.201.215:4343 , piece_id=MIATMAVPWKBJPVYH3XDYXQGKW7VWKRULLP6UV2ZNKJZ7YDVJJEBQ, seg_pos={0 0}, upload=274674]: Uploading piece... Aug 19 17:38:09.694358652 (264h12m4.125438783s) [noise=true , address=104.36.82.5:4852 , piece_id=UALXJPWV34HHWPC6GNCOFNP2RDC3KF6ZY4FUHLL3MWR2UZDCXNCQ, seg_pos={0 0}, upload=274674]: Uploading piece... Aug 19 17:38:09.694689475 (264h12m4.125769606s) [noise=true , address=104.36.82.58:4269 , piece_id=CTQSY5PQFUNXUJR2JXSI7O2EHMYTOAHUSHGTMVTI42E45RYFIGJA, seg_pos={0 0}, upload=274674]: Uploading piece... Aug 19 17:38:09.695011601 (264h12m4.126091732s) [noise=true , address=104.36.82.60:4364 , piece_id=A5BGK2CC62VYRH27SR2MONG2GPSZ5WONCHAF42LFPXOTM2W7XIJA, seg_pos={0 0}, upload=274674]: Uploading piece... Aug 19 17:38:09.695441633 (264h12m4.126521764s) [noise=true , address=104.36.82.9:4028 , piece_id=4LUYCAISMQC3SGDLJZSDFDJET4R54U3N4B6ZAJ35UDXB5CT3GV2Q, seg_pos={0 0}, upload=274674]: Uploading piece... Aug 19 17:38:09.695845597 (264h12m4.126925728s) [noise=true , address=77.90.31.15:4701 , piece_id=CCTT764Z3HFZFT4PE7YOOBVVGDASLFYVFGVXGWADBLDLURBOKGGQ, seg_pos={0 0}, upload=274674]: Uploading piece... Aug 19 17:38:09.696191328 (264h12m4.127271459s) [noise=true , address=45.91.116.5:4302 , piece_id=D2ZYYIFLA7GFPEII4X75HVJGL5DOM6BZLQFXL7HYVXYLNCL4CDQQ, seg_pos={0 0}, upload=274674]: Uploading piece... Aug 19 17:38:09.696539624 (264h12m4.127619755s) [noise=true , address=77.90.31.14:4689 , piece_id=6FO4WN5BXNW6UFGW2ALS5YBYBP5FZCAZBLQZXAULFCN65NHH7N6A, seg_pos={0 0}, upload=274674]: Uploading piece... Aug 19 17:38:09.696809421 (264h12m4.127889552s) [noise=true , address=79.127.201.214:4299 , piece_id=C4JSI4OPEZPPHADT6QTVJGEXQNVSHEK42KNYQBV3S2OH7Z5Z7HHA, seg_pos={0 0}, upload=274674]: Uploading piece... Aug 19 17:38:09.697172445 (264h12m4.128252576s) [noise=true , address=104.36.82.43:4855 , piece_id=3H2PUTGFVC2PR224NBHGRB7GLWBOL2MNXXWJ7JF23HYVF53TEWFQ, seg_pos={0 0}, upload=274674]: Uploading piece... Aug 19 17:38:09.697485064 (264h12m4.128565205s) [noise=true , address=77.90.31.12:4619 , piece_id=XMWKT6JBVLOPTRUIPNEK4GXVCH5OOVO5WYWDGWMXGA2VVVD74CBA, seg_pos={0 0}, upload=274674]: Uploading piece... Aug 19 17:38:09.697774858 (264h12m4.128854989s) [noise=true , address=104.36.82.38:4714 , piece_id=SZ42LIQE4LXTBCZY4BTXWS3EL27UYIQS3AAXGWNLH65TUZWNUYSQ, seg_pos={0 0}, upload=274674]: Uploading piece... Aug 19 17:38:09.698096103 (264h12m4.129176234s) [noise=true , address=104.36.82.4:4817 , piece_id=JNKPBDLUZUWI22ZLGTCD3BZJBAUX7TZVAGXJVFZOEJPK2Z3PMADA, seg_pos={0 0}, upload=274674]: Uploading piece... Aug 19 17:38:09.698352945 (264h12m4.129433076s) [noise=true , address=45.91.116.13:4183 , piece_id=OMUE6JFSTRIF64VYM5XCHNL7U26SEXP3IGWDBHP5G2E6YJQETEUQ, seg_pos={0 0}, upload=274674]: Uploading piece... Aug 19 17:38:09.698876106 (264h12m4.129956227s) [noise=true , address=104.36.82.6:4879 , piece_id=JKK5HMPYTMR7PNWHVPBDHPH3K6H7VM7AA4K4X4WXQ5LN57VRBUFA, seg_pos={0 0}, upload=274674]: Uploading piece... Aug 19 17:38:11.192631202 (264h12m5.623711333s) [upload=274674]: Next returned nil segment Aug 19 17:38:11.192648606 (264h12m5.623728737s) [upload=274674]: Waiting for error group managing segments... Aug 19 17:38:11.591537039 (264h12m6.02261717s) [noise=true , address=45.91.116.14:4222 , piece_id=DXHEK43V7USJLYKPF5PKEHLJPVP4OPH2H3TQVGUBMK5MVRWQHXJQ, seg_pos={0 0}, upload=274674]: Done uploading piece. err: <nil> Aug 19 17:38:11.606093581 (264h12m6.037173712s) [noise=true , address=77.90.31.14:4689 , piece_id=6FO4WN5BXNW6UFGW2ALS5YBYBP5FZCAZBLQZXAULFCN65NHH7N6A, seg_pos={0 0}, upload=274674]: Done uploading piece. err: <nil> Aug 19 17:38:11.610339916 (264h12m6.041420047s) [noise=true , address=77.90.31.15:4701 , piece_id=CCTT764Z3HFZFT4PE7YOOBVVGDASLFYVFGVXGWADBLDLURBOKGGQ, seg_pos={0 0}, upload=274674]: Done uploading piece. err: <nil> Aug 19 17:38:11.630055040 (264h12m6.061135171s) [noise=true , address=79.127.201.215:4343 , piece_id=MIATMAVPWKBJPVYH3XDYXQGKW7VWKRULLP6UV2ZNKJZ7YDVJJEBQ, seg_pos={0 0}, upload=274674]: Done uploading piece. err: <nil> Aug 19 17:38:11.678703675 (264h12m6.109783797s) [noise=true , address=104.36.82.9:4028 , piece_id=4LUYCAISMQC3SGDLJZSDFDJET4R54U3N4B6ZAJ35UDXB5CT3GV2Q, seg_pos={0 0}, upload=274674]: Done uploading piece. err: <nil> Aug 19 17:38:11.693543009 (264h12m6.12462314s) [noise=true , address=104.36.82.60:4364 , piece_id=A5BGK2CC62VYRH27SR2MONG2GPSZ5WONCHAF42LFPXOTM2W7XIJA, seg_pos={0 0}, upload=274674]: Done uploading piece. err: <nil> Aug 19 17:38:11.693568448 (264h12m6.124648579s) [noise=true , address=45.91.116.13:4183 , piece_id=OMUE6JFSTRIF64VYM5XCHNL7U26SEXP3IGWDBHP5G2E6YJQETEUQ, seg_pos={0 0}, upload=274674]: Done uploading piece. err: <nil> Aug 19 17:38:11.698831207 (264h12m6.129911329s) [noise=true , address=104.36.82.5:4852 , piece_id=UALXJPWV34HHWPC6GNCOFNP2RDC3KF6ZY4FUHLL3MWR2UZDCXNCQ, seg_pos={0 0}, upload=274674]: Done uploading piece. err: <nil> Aug 19 17:38:11.698886834 (264h12m6.129966955s) [noise=true , address=77.90.31.12:4619 , piece_id=XMWKT6JBVLOPTRUIPNEK4GXVCH5OOVO5WYWDGWMXGA2VVVD74CBA, seg_pos={0 0}, upload=274674]: Done uploading piece. err: <nil> Aug 19 17:38:11.714704781 (264h12m6.145784912s) [noise=true , address=79.127.201.214:4299 , piece_id=C4JSI4OPEZPPHADT6QTVJGEXQNVSHEK42KNYQBV3S2OH7Z5Z7HHA, seg_pos={0 0}, upload=274674]: Done uploading piece. err: <nil> Aug 19 17:38:11.720319795 (264h12m6.151399956s) [noise=true , address=45.91.116.11:4140 , piece_id=KQNAZLV4XPDYYLJQUMI2UL3GVLA7QTEZ6TU3HDYFYB7LPRBSRQFA, seg_pos={0 0}, upload=274674]: Done uploading piece. err: <nil> Aug 19 17:38:11.722063401 (264h12m6.153143532s) [noise=true , address=104.36.82.6:4879 , piece_id=JKK5HMPYTMR7PNWHVPBDHPH3K6H7VM7AA4K4X4WXQ5LN57VRBUFA, seg_pos={0 0}, upload=274674]: Done uploading piece. err: <nil> Aug 19 17:38:11.726762272 (264h12m6.157842393s) [noise=true , address=104.36.82.38:4714 , piece_id=SZ42LIQE4LXTBCZY4BTXWS3EL27UYIQS3AAXGWNLH65TUZWNUYSQ, seg_pos={0 0}, upload=274674]: Done uploading piece. err: <nil> Aug 19 17:38:11.734258877 (264h12m6.165339008s) [noise=true , address=104.36.82.58:4269 , piece_id=CTQSY5PQFUNXUJR2JXSI7O2EHMYTOAHUSHGTMVTI42E45RYFIGJA, seg_pos={0 0}, upload=274674]: Done uploading piece. err: <nil> Aug 19 17:38:11.734284346 (264h12m6.165364477s) [seg_pos={0 0}, upload=274674]: Segment reached optimal threshold of 14 pieces. Aug 19 17:38:11.734416277 (264h12m6.165496408s) [noise=true , address=104.36.82.4:4817 , piece_id=JNKPBDLUZUWI22ZLGTCD3BZJBAUX7TZVAGXJVFZOEJPK2Z3PMADA, seg_pos={0 0}, upload=274674]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RSBodzJxMye65QgkyrzkWCKQ49cnfdu4haUZKW23nWpqzMcYz): protocol: expected piece hash; context canceled; EOF Aug 19 17:38:11.734486873 (264h12m6.165567004s) [noise=true , address=104.36.82.43:4855 , piece_id=3H2PUTGFVC2PR224NBHGRB7GLWBOL2MNXXWJ7JF23HYVF53TEWFQ, seg_pos={0 0}, upload=274674]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TGTRhEi2KsbrkC6ZmgGkMaJri9uhzEcmXe1J4nJjpEd1BE4ka): protocol: expected piece hash; context canceled; EOF Aug 19 17:38:11.734587926 (264h12m6.165668057s) [noise=true , address=77.90.31.18:4825 , piece_id=UQLJVRVUBBN5LFBIAQL7YQ4IRAWI3SBVBHUT4IIFBROAUHXHVZ3A, seg_pos={0 0}, upload=274674]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PqKyfFxfVKTGARbPqBcQL3Naqjdf5uR44PeLi3sJpu57dcm1x): protocol: expected piece hash; context canceled; EOF Aug 19 17:38:11.734592956 (264h12m6.165673087s) [noise=true , address=45.91.116.5:4302 , piece_id=D2ZYYIFLA7GFPEII4X75HVJGL5DOM6BZLQFXL7HYVXYLNCL4CDQQ, seg_pos={0 0}, upload=274674]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pJWhSLza8WaMuvvDwH4nae57UxQmfCdDvAkQ3v25MgPYHoZXx): protocol: expected piece hash; context canceled; EOF Aug 19 17:38:11.734645206 (264h12m6.165725337s) [seg_pos={0 0}, upload=274674]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 19 17:38:11.734756098 (264h12m6.165836239s) [upload=274674]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 19 17:38:11.734764033 (264h12m6.165844154s) [upload=274674]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 17:38:12.023665554 (264h12m6.454745685s) [upload=274674]: Done uploading segments... Aug 19 17:38:12.023686044 (264h12m6.454766195s) [upload=274674]: Upload finished. err: <nil> Duration: 5.750908938s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.................X.....X....................................] duration: 5.847039351s FAIL (2025-08-19 17:50:07 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.847061684s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 5.850088083s FAIL (2025-08-19 17:50:05 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Got next segment. Inline: false Aug 19 17:23:57.290467622 (263h57m51.721547763s) [upload=274428]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 17:23:57.290471560 (263h57m51.721551681s) [upload=274428]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 19 17:23:57.743765742 (263h57m52.174845863s) [seg_pos={0 0}, upload=274428]: Begin upload segment... Aug 19 17:23:57.743818412 (263h57m52.174898543s) [seg_pos={0 0}, upload=274428]: Done begin upload segment. Aug 19 17:23:57.743842609 (263h57m52.17492274s) [noise=true , address=77.90.31.16:4750 , piece_id=XXJ4VHMZ4NVVT2WX322JS7MW3FHQ7UZJW4VLR4KG6FW4A4RHVN6A, seg_pos={0 0}, upload=274428]: Uploading piece... Aug 19 17:23:57.743875672 (263h57m52.174955803s) [noise=true , address=45.91.116.14:4198 , piece_id=NSLSBMDS2ZJKN62N4NGN7HRHXAUIS4WL4AGHODZZBD2CDYTXHVUA, seg_pos={0 0}, upload=274428]: Uploading piece... Aug 19 17:23:57.744218498 (263h57m52.175298629s) [noise=true , address=77.90.31.14:4667 , piece_id=BF53QUGTGVD23KCIIAA5NWUCWKWUXXSHKCAAWSDZLAORPHXQEMQQ, seg_pos={0 0}, upload=274428]: Uploading piece... Aug 19 17:23:57.744456373 (263h57m52.175536504s) [noise=true , address=77.90.31.21:4883 , piece_id=SVFJCG4T5OCDMVKBE7A33XPDIQUH6NESE4GIVTETRZRBX4N3MJDQ, seg_pos={0 0}, upload=274428]: Uploading piece... Aug 19 17:23:57.744523051 (263h57m52.175603182s) [noise=true , address=104.36.82.60:4365 , piece_id=7BXP5MWOH2B3H74CPVWOQN2NEGMXUYECIVOEHRRKINQ544WKAAHA, seg_pos={0 0}, upload=274428]: Uploading piece... Aug 19 17:23:57.744762870 (263h57m52.175843001s) [noise=true , address=77.90.31.19:4851 , piece_id=POYQI453UA5767REISB5BNE3MYV6IOYZNIK357PYGH5DM66EOBIA, seg_pos={0 0}, upload=274428]: Uploading piece... Aug 19 17:23:57.744993531 (263h57m52.176073662s) [noise=true , address=104.36.82.44:4871 , piece_id=WQGOUIWA7NVK5SUTIWX6E32ZKW62Y2KGK7E3HB5YW2CKDUR42BPQ, seg_pos={0 0}, upload=274428]: Uploading piece... Aug 19 17:23:57.745229373 (263h57m52.176309494s) [noise=true , address=79.127.205.234:4587 , piece_id=ALBAMUJVGA4RIJKGCJNJT3BT6RBD5QWU4C33RJGCEP3XIQQLW6QQ, seg_pos={0 0}, upload=274428]: Uploading piece... Aug 19 17:23:57.745473950 (263h57m52.176554081s) [noise=true , address=104.36.82.3:4771 , piece_id=2RZZG2MORQZR3ROAADIWSK46JDEZSNYBGASCDWZTKWWU72JFHRSQ, seg_pos={0 0}, upload=274428]: Uploading piece... Aug 19 17:23:57.745746030 (263h57m52.176826161s) [noise=true , address=104.36.82.34:4558 , piece_id=AJNHQTDQSMJ3R2ZKVQUCBU5XOMWNF66ID2M3RVPYDL52F5LGTPFQ, seg_pos={0 0}, upload=274428]: Uploading piece... Aug 19 17:23:57.745999426 (263h57m52.177079547s) [noise=true , address=77.90.31.18:4814 , piece_id=VUIFMMFA5PCHIKNNX7PA7RHVXIJHK3UWH4SI34R6XP5MEYTAMS6A, seg_pos={0 0}, upload=274428]: Uploading piece... Aug 19 17:23:57.746774839 (263h57m52.17785497s) [noise=true , address=104.36.82.35:4610 , piece_id=DM7HADQAK5T67ZDQR3VGUUGLHCGBLC5KJYYFF7QQNRKVE2SA5CHA, seg_pos={0 0}, upload=274428]: Uploading piece... Aug 19 17:23:57.747503714 (263h57m52.178583836s) [noise=true , address=64.49.13.137:4496 , piece_id=DW5YMZIQJZIA53K22HGK2ICRT5IVVP2ZMIVYM46BWQWAYPLNUC2Q, seg_pos={0 0}, upload=274428]: Uploading piece... Aug 19 17:23:57.747727743 (263h57m52.178807874s) [noise=true , address=77.90.31.13:4640 , piece_id=NKLBPZJDAIAVAYH3QRDCPS6VWHJBFZZWY3HCTSW2E4II77YC3BYA, seg_pos={0 0}, upload=274428]: Uploading piece... Aug 19 17:23:57.747948565 (263h57m52.179028696s) [noise=true , address=104.36.82.55:4214 , piece_id=64P34K2YEWCWZFGWSR4AQ5SRALT5CSZCB4F6FHPKDP3ZFJIOJC2Q, seg_pos={0 0}, upload=274428]: Uploading piece... Aug 19 17:23:57.748174007 (263h57m52.179254138s) [noise=true , address=104.36.82.23:4448 , piece_id=SCMNBDCYA7PUN2VP6JN37YM5LNY2GBGQHJKYSSWUWMPTZZ7LXZUQ, seg_pos={0 0}, upload=274428]: Uploading piece... Aug 19 17:23:57.748433112 (263h57m52.179513233s) [noise=true , address=104.36.82.5:4854 , piece_id=FC4DIKMBI5OPLDQCVA6P3MA7AQZV23QBIX3UAKLWJJMY724TNYHQ, seg_pos={0 0}, upload=274428]: Uploading piece... Aug 19 17:23:57.748647252 (263h57m52.179727383s) [noise=true , address=104.36.82.62:4442 , piece_id=VV5BBYB6UI4O3B4ESOBJX44I5RJBRWSLJUFR6H3GKEUZWNONTPMA, seg_pos={0 0}, upload=274428]: Uploading piece... Aug 19 17:23:58.869276532 (263h57m53.300356663s) [upload=274428]: Next returned nil segment Aug 19 17:23:58.869291511 (263h57m53.300371642s) [upload=274428]: Waiting for error group managing segments... Aug 19 17:23:59.195579562 (263h57m53.626659683s) [noise=true , address=79.127.205.234:4587 , piece_id=ALBAMUJVGA4RIJKGCJNJT3BT6RBD5QWU4C33RJGCEP3XIQQLW6QQ, seg_pos={0 0}, upload=274428]: Done uploading piece. err: <nil> Aug 19 17:23:59.239543682 (263h57m53.670623813s) [noise=true , address=64.49.13.137:4496 , piece_id=DW5YMZIQJZIA53K22HGK2ICRT5IVVP2ZMIVYM46BWQWAYPLNUC2Q, seg_pos={0 0}, upload=274428]: Done uploading piece. err: <nil> Aug 19 17:23:59.654494638 (263h57m54.085574769s) [noise=true , address=104.36.82.35:4610 , piece_id=DM7HADQAK5T67ZDQR3VGUUGLHCGBLC5KJYYFF7QQNRKVE2SA5CHA, seg_pos={0 0}, upload=274428]: Done uploading piece. err: <nil> Aug 19 17:23:59.700507838 (263h57m54.131587959s) [noise=true , address=104.36.82.34:4558 , piece_id=AJNHQTDQSMJ3R2ZKVQUCBU5XOMWNF66ID2M3RVPYDL52F5LGTPFQ, seg_pos={0 0}, upload=274428]: Done uploading piece. err: <nil> Aug 19 17:23:59.757508140 (263h57m54.188588272s) [noise=true , address=104.36.82.5:4854 , piece_id=FC4DIKMBI5OPLDQCVA6P3MA7AQZV23QBIX3UAKLWJJMY724TNYHQ, seg_pos={0 0}, upload=274428]: Done uploading piece. err: <nil> Aug 19 17:23:59.765247029 (263h57m54.19632716s) [noise=true , address=104.36.82.62:4442 , piece_id=VV5BBYB6UI4O3B4ESOBJX44I5RJBRWSLJUFR6H3GKEUZWNONTPMA, seg_pos={0 0}, upload=274428]: Done uploading piece. err: <nil> Aug 19 17:23:59.774367630 (263h57m54.205447761s) [noise=true , address=104.36.82.23:4448 , piece_id=SCMNBDCYA7PUN2VP6JN37YM5LNY2GBGQHJKYSSWUWMPTZZ7LXZUQ, seg_pos={0 0}, upload=274428]: Done uploading piece. err: <nil> Aug 19 17:23:59.827970201 (263h57m54.259050332s) [noise=true , address=104.36.82.55:4214 , piece_id=64P34K2YEWCWZFGWSR4AQ5SRALT5CSZCB4F6FHPKDP3ZFJIOJC2Q, seg_pos={0 0}, upload=274428]: Done uploading piece. err: <nil> Aug 19 17:23:59.851412395 (263h57m54.282492526s) [noise=true , address=104.36.82.44:4871 , piece_id=WQGOUIWA7NVK5SUTIWX6E32ZKW62Y2KGK7E3HB5YW2CKDUR42BPQ, seg_pos={0 0}, upload=274428]: Done uploading piece. err: <nil> Aug 19 17:23:59.946562012 (263h57m54.377642143s) [noise=true , address=104.36.82.3:4771 , piece_id=2RZZG2MORQZR3ROAADIWSK46JDEZSNYBGASCDWZTKWWU72JFHRSQ, seg_pos={0 0}, upload=274428]: Done uploading piece. err: <nil> Aug 19 17:23:59.979671513 (263h57m54.410751634s) [noise=true , address=104.36.82.60:4365 , piece_id=7BXP5MWOH2B3H74CPVWOQN2NEGMXUYECIVOEHRRKINQ544WKAAHA, seg_pos={0 0}, upload=274428]: Done uploading piece. err: <nil> Aug 19 17:24:55.494274881 (263h58m49.925355012s) [noise=true , address=77.90.31.18:4814 , piece_id=VUIFMMFA5PCHIKNNX7PA7RHVXIJHK3UWH4SI34R6XP5MEYTAMS6A, seg_pos={0 0}, upload=274428]: Done uploading piece. err: <nil> Aug 19 17:24:55.842835997 (263h58m50.273916128s) [noise=true , address=77.90.31.21:4883 , piece_id=SVFJCG4T5OCDMVKBE7A33XPDIQUH6NESE4GIVTETRZRBX4N3MJDQ, seg_pos={0 0}, upload=274428]: Done uploading piece. err: <nil> Aug 19 17:24:56.634889530 (263h58m51.065969661s) [noise=true , address=77.90.31.19:4851 , piece_id=POYQI453UA5767REISB5BNE3MYV6IOYZNIK357PYGH5DM66EOBIA, seg_pos={0 0}, upload=274428]: Done uploading piece. err: ecclient: upload failed (node:1NEV2ndHw4fohVUNsfrzwojd8S11TmMST6Sz59GVnXS8vDGhWk, address:77.90.31.19:4851): protocol: rpc: dial tcp 77.90.31.19:4851: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:56.635097748 (263h58m51.066177879s) [noise=true , address=45.91.116.14:4198 , piece_id=NSLSBMDS2ZJKN62N4NGN7HRHXAUIS4WL4AGHODZZBD2CDYTXHVUA, seg_pos={0 0}, upload=274428]: Done uploading piece. err: ecclient: upload failed (node:1xxfUrCm5PgSLpge1wYiH2EcE7kzzJ38P9rfeCYmqXTbTisioi, address:45.91.116.14:4198): protocol: rpc: dial tcp 45.91.116.14:4198: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:56.635158895 (263h58m51.066239036s) [noise=true , address=77.90.31.16:4750 , piece_id=XXJ4VHMZ4NVVT2WX322JS7MW3FHQ7UZJW4VLR4KG6FW4A4RHVN6A, seg_pos={0 0}, upload=274428]: Done uploading piece. err: ecclient: upload failed (node:12TGd8NnWdekDtHzBQHmJPgTtHjXC6g8fi6RoZYFQJVHDa5hdyE, address:77.90.31.16:4750): protocol: read tcp 188.166.225.170:54198->77.90.31.16:4750: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:56.635224120 (263h58m51.066304281s) [noise=true , address=77.90.31.14:4667 , piece_id=BF53QUGTGVD23KCIIAA5NWUCWKWUXXSHKCAAWSDZLAORPHXQEMQQ, seg_pos={0 0}, upload=274428]: Done uploading piece. err: ecclient: upload failed (node:12R5gJFhsmocYPNZf4yNnYqJwcwAKt9yiHmmqGMk3btzVjkFV4S, address:77.90.31.14:4667): protocol: read tcp 188.166.225.170:46984->77.90.31.14:4667: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:56.635280057 (263h58m51.066360198s) [noise=true , address=77.90.31.13:4640 , piece_id=NKLBPZJDAIAVAYH3QRDCPS6VWHJBFZZWY3HCTSW2E4II77YC3BYA, seg_pos={0 0}, upload=274428]: Done uploading piece. err: ecclient: upload failed (node:1XjJpLBEZm5daHcBUmUyC7QpLVhu9Vm9ex9RYcYJWEuLGk527F, address:77.90.31.13:4640): protocol: read tcp 188.166.225.170:48656->77.90.31.13:4640: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:56.635314653 (263h58m51.066394794s) [seg_pos={0 0}, upload=274428]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Aug 19 17:24:56.862515362 (263h58m51.293595493s) [upload=274428]: Done uploading segments... Aug 19 17:24:56.862536042 (263h58m51.293616163s) [upload=274428]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.850114694s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 2.196183141s FAIL (2025-08-19 17:50:00 UTC) orbiter warning: operation: ListObjects ======ERROR====== uplink: metainfo: metaclient: context deadline exceeded ======LOGS====== Aug 19 17:35:36.607320794 (264h9m31.038400925s) [upload=274632]: Starting upload Aug 19 17:35:36.607367684 (264h9m31.038447815s) [upload=274632]: Done starting upload Aug 19 17:35:36.607435082 (264h9m31.038515213s) [upload=274632]: Uploading segments... Aug 19 17:35:36.607490538 (264h9m31.038570669s) [upload=274632]: Got next segment. Inline: true Aug 19 17:35:36.607495088 (264h9m31.038575219s) [upload=274632]: Waiting for error group managing segments... Aug 19 17:35:36.607537528 (264h9m31.038617659s) [upload=274632]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 17:35:36.607541656 (264h9m31.038621787s) [upload=274632]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 19 17:35:36.607544051 (264h9m31.038624182s) [upload=274632]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 17:35:36.889055260 (264h9m31.320135391s) [upload=274632]: Done uploading segments... Aug 19 17:35:36.889082943 (264h9m31.320163124s) [upload=274632]: Upload finished. err: <nil> Duration:* 2.196212578s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...................X...X....................................] duration: 5.681092074s FAIL (2025-08-19 17:49:43 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 23:59.057943072 (263h57m53.489023273s) [noise=true , address=45.91.116.10:4083 , piece_id=V3H4T2WN2MBVALSIHZNOQHBE75WQRNYLIE3Z7DFWUUHBPYDT7IFA, seg_pos={0 0}, upload=274429]: Uploading piece... Aug 19 17:23:59.057955436 (263h57m53.489035557s) [noise=true , address=45.91.116.5:4298 , piece_id=HMF6DXO5KFMLK67WVA7BAWLQFLC2BCVPJ7ZT255FMRCCEI3MCBNA, seg_pos={0 0}, upload=274429]: Uploading piece... Aug 19 17:23:59.058557689 (263h57m53.48963782s) [noise=true , address=104.36.82.8:4969 , piece_id=3P655Z5OUVZ4VRESACEBKYR3KUFYY56NYPWX7ZVOHL3XI2YQ5TCQ, seg_pos={0 0}, upload=274429]: Uploading piece... Aug 19 17:23:59.059170771 (263h57m53.490250902s) [noise=true , address=45.91.116.7:4009 , piece_id=NSZYKL6FHDLDM463ZXKEQGX4ZY7JPJXENH63PPDV76XMGCBV7ZRQ, seg_pos={0 0}, upload=274429]: Uploading piece... Aug 19 17:23:59.059468281 (263h57m53.490548412s) [noise=true , address=77.90.31.15:4699 , piece_id=MFF7PNMDLHYEH7PY46MGGOJSY4XPHRDPGFWJABBO4JDTJJJJ3CTQ, seg_pos={0 0}, upload=274429]: Uploading piece... Aug 19 17:23:59.059498208 (263h57m53.490578339s) [noise=true , address=104.36.82.2:4729 , piece_id=BNX6UNYZ7VHQ4MKAH65DGSHULWRRXDSCMKOE7WFZO6T2V3ELBJ7Q, seg_pos={0 0}, upload=274429]: Uploading piece... Aug 19 17:23:59.059814182 (263h57m53.490894323s) [noise=true , address=104.36.82.6:4874 , piece_id=QMGNWSGMEV762Y2XEDK2LIAY3JMNLD3UOVFQ6HYP7ZMLLVAVO7ZQ, seg_pos={0 0}, upload=274429]: Uploading piece... Aug 19 17:23:59.059839941 (263h57m53.490920072s) [noise=true , address=45.91.116.12:4225 , piece_id=PCJYR2VLFH23UPQEABUYESR3MHKDHIVTJWDIWVYIL6JAFGGXXXFQ, seg_pos={0 0}, upload=274429]: Uploading piece... Aug 19 17:23:59.059875480 (263h57m53.490955611s) [noise=true , address=104.36.82.10:4079 , piece_id=S3BSVJWRPP2PDGLG7FKFROUKAM3GTBDL2ALAJPAZQQQZJVRMMZ3Q, seg_pos={0 0}, upload=274429]: Uploading piece... Aug 19 17:23:59.060109147 (263h57m53.491189278s) [noise=true , address=104.36.82.2:4732 , piece_id=NLTK4572JAQHZQRDD2DOOIW7VBO7NK4DI4RUZRSXQ3GR72EWV3KQ, seg_pos={0 0}, upload=274429]: Uploading piece... Aug 19 17:23:59.060142190 (263h57m53.491222321s) [noise=true , address=45.91.116.9:4261 , piece_id=J5IMFL3JW4YKDLI6OEY2POEB4LJR2P3BIFAD4YIF3E4JTUXIXY3A, seg_pos={0 0}, upload=274429]: Uploading piece... Aug 19 17:23:59.060348274 (263h57m53.491428405s) [noise=true , address=104.36.82.15:4259 , piece_id=RM7JL7JKCMFM4CNXVVZQ2QJTOEPQEQHGLW3GYXWUD6LAQ2ZJ3P5Q, seg_pos={0 0}, upload=274429]: Uploading piece... Aug 19 17:23:59.060544830 (263h57m53.491624961s) [noise=true , address=45.91.116.14:4191 , piece_id=PJ33GP36S76VCI5RVR26PBFZ6HZUUZYVWNGPJITRLGDZI6LWCJIQ, seg_pos={0 0}, upload=274429]: Uploading piece... Aug 19 17:23:59.060752227 (263h57m53.491832358s) [noise=true , address=45.91.116.11:4118 , piece_id=VD2HK2QU63UVJFRASQOIRFZRWZN2HAUSAZJL2VZKYALGSV67SP2A, seg_pos={0 0}, upload=274429]: Uploading piece... Aug 19 17:23:59.060967990 (263h57m53.492048121s) [noise=true , address=77.90.31.11:4555 , piece_id=GTU6PGCHI4WOFGK2B3FP2Y22VUDBWLQKKEMZBT6I4F47OKK4RPNQ, seg_pos={0 0}, upload=274429]: Uploading piece... Aug 19 17:23:59.061294044 (263h57m53.492374175s) [noise=true , address=45.91.116.8:4048 , piece_id=M2CBILQKSVSQ7VY4AWQQM5GHF2KYBSXH73YJGAEJZAHOR7FOBWEQ, seg_pos={0 0}, upload=274429]: Uploading piece... Aug 19 17:24:00.872920400 (263h57m55.304000531s) [upload=274429]: Next returned nil segment Aug 19 17:24:00.872945548 (263h57m55.304025679s) [upload=274429]: Waiting for error group managing segments... Aug 19 17:24:01.156214888 (263h57m55.587295019s) [noise=true , address=104.36.82.6:4874 , piece_id=QMGNWSGMEV762Y2XEDK2LIAY3JMNLD3UOVFQ6HYP7ZMLLVAVO7ZQ, seg_pos={0 0}, upload=274429]: Done uploading piece. err: <nil> Aug 19 17:24:01.181164926 (263h57m55.612245057s) [noise=true , address=104.36.82.2:4732 , piece_id=NLTK4572JAQHZQRDD2DOOIW7VBO7NK4DI4RUZRSXQ3GR72EWV3KQ, seg_pos={0 0}, upload=274429]: Done uploading piece. err: <nil> Aug 19 17:24:01.222047199 (263h57m55.653127331s) [noise=true , address=104.36.82.2:4729 , piece_id=BNX6UNYZ7VHQ4MKAH65DGSHULWRRXDSCMKOE7WFZO6T2V3ELBJ7Q, seg_pos={0 0}, upload=274429]: Done uploading piece. err: <nil> Aug 19 17:24:01.250473409 (263h57m55.68155354s) [noise=true , address=104.36.82.15:4259 , piece_id=RM7JL7JKCMFM4CNXVVZQ2QJTOEPQEQHGLW3GYXWUD6LAQ2ZJ3P5Q, seg_pos={0 0}, upload=274429]: Done uploading piece. err: <nil> Aug 19 17:24:01.259222299 (263h57m55.69030242s) [noise=true , address=104.36.82.8:4969 , piece_id=3P655Z5OUVZ4VRESACEBKYR3KUFYY56NYPWX7ZVOHL3XI2YQ5TCQ, seg_pos={0 0}, upload=274429]: Done uploading piece. err: <nil> Aug 19 17:24:01.261735818 (263h57m55.692815949s) [noise=true , address=104.36.82.10:4079 , piece_id=S3BSVJWRPP2PDGLG7FKFROUKAM3GTBDL2ALAJPAZQQQZJVRMMZ3Q, seg_pos={0 0}, upload=274429]: Done uploading piece. err: <nil> Aug 19 17:24:28.183964515 (263h58m22.615044646s) [noise=true , address=45.91.116.14:4191 , piece_id=PJ33GP36S76VCI5RVR26PBFZ6HZUUZYVWNGPJITRLGDZI6LWCJIQ, seg_pos={0 0}, upload=274429]: Done uploading piece. err: ecclient: upload failed (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, address:45.91.116.14:4191): protocol: read tcp 188.166.225.170:37646->45.91.116.14:4191: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:28.184013127 (263h58m22.615093258s) [noise=true , address=45.91.116.9:4261 , piece_id=J5IMFL3JW4YKDLI6OEY2POEB4LJR2P3BIFAD4YIF3E4JTUXIXY3A, seg_pos={0 0}, upload=274429]: Done uploading piece. err: ecclient: upload failed (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, address:45.91.116.9:4261): protocol: rpc: dial tcp 45.91.116.9:4261: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:28.184026694 (263h58m22.615106825s) [noise=true , address=45.91.116.8:4048 , piece_id=M2CBILQKSVSQ7VY4AWQQM5GHF2KYBSXH73YJGAEJZAHOR7FOBWEQ, seg_pos={0 0}, upload=274429]: Done uploading piece. err: ecclient: upload failed (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, address:45.91.116.8:4048): protocol: read tcp 188.166.225.170:45604->45.91.116.8:4048: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:28.184070568 (263h58m22.615150689s) [noise=true , address=45.91.116.5:4298 , piece_id=HMF6DXO5KFMLK67WVA7BAWLQFLC2BCVPJ7ZT255FMRCCEI3MCBNA, seg_pos={0 0}, upload=274429]: Done uploading piece. err: ecclient: upload failed (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, address:45.91.116.5:4298): protocol: read tcp 188.166.225.170:46970->45.91.116.5:4298: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:28.184103010 (263h58m22.615183141s) [noise=true , address=45.91.116.10:4083 , piece_id=V3H4T2WN2MBVALSIHZNOQHBE75WQRNYLIE3Z7DFWUUHBPYDT7IFA, seg_pos={0 0}, upload=274429]: Done uploading piece. err: ecclient: upload failed (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, address:45.91.116.10:4083): protocol: read tcp 188.166.225.170:41632->45.91.116.10:4083: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:28.184139489 (263h58m22.61521962s) [noise=true , address=45.91.116.7:4009 , piece_id=NSZYKL6FHDLDM463ZXKEQGX4ZY7JPJXENH63PPDV76XMGCBV7ZRQ, seg_pos={0 0}, upload=274429]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: read tcp 188.166.225.170:45756->45.91.116.7:4009: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:28.184171009 (263h58m22.615251141s) [noise=true , address=77.90.31.11:4555 , piece_id=GTU6PGCHI4WOFGK2B3FP2Y22VUDBWLQKKEMZBT6I4F47OKK4RPNQ, seg_pos={0 0}, upload=274429]: Done uploading piece. err: ecclient: upload failed (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, address:77.90.31.11:4555): protocol: read tcp 188.166.225.170:37438->77.90.31.11:4555: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:28.184197480 (263h58m22.615277611s) [noise=true , address=77.90.31.15:4699 , piece_id=MFF7PNMDLHYEH7PY46MGGOJSY4XPHRDPGFWJABBO4JDTJJJJ3CTQ, seg_pos={0 0}, upload=274429]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: read tcp 188.166.225.170:36788->77.90.31.15:4699: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:28.184218410 (263h58m22.615298541s) [noise=true , address=45.91.116.12:4225 , piece_id=PCJYR2VLFH23UPQEABUYESR3MHKDHIVTJWDIWVYIL6JAFGGXXXFQ, seg_pos={0 0}, upload=274429]: Done uploading piece. err: ecclient: upload failed (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, address:45.91.116.12:4225): protocol: read tcp 188.166.225.170:60636->45.91.116.12:4225: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:28.184249710 (263h58m22.615329841s) [noise=true , address=45.91.116.11:4118 , piece_id=VD2HK2QU63UVJFRASQOIRFZRWZN2HAUSAZJL2VZKYALGSV67SP2A, seg_pos={0 0}, upload=274429]: Done uploading piece. err: ecclient: upload failed (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, address:45.91.116.11:4118): protocol: read tcp 188.166.225.170:48006->45.91.116.11:4118: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:28.184274337 (263h58m22.615354468s) [seg_pos={0 0}, upload=274429]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Aug 19 17:24:28.412483516 (263h58m22.843563647s) [upload=274429]: Done uploading segments... Aug 19 17:24:28.412520628 (263h58m22.843600759s) [upload=274429]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 5.681116631s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 7 out of 60 is failed (threshold=3) [.....X.............X......X.X........X....XX................] duration: 5.765596158s FAIL (2025-08-19 17:49:16 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/4M") ======LOGS====== Aug 19 17:38:09.246414094 (264h12m3.677494255s) [upload=274674]: Starting upload Aug 19 17:38:09.246486392 (264h12m3.677566533s) [upload=274674]: Done starting upload Aug 19 17:38:09.246601452 (264h12m3.677681583s) [upload=274674]: Uploading segments... Aug 19 17:38:09.246691885 (264h12m3.677772016s) [upload=274674]: Got next segment. Inline: false Aug 19 17:38:09.246724627 (264h12m3.677804758s) [upload=274674]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 17:38:09.246730909 (264h12m3.67781104s) [upload=274674]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 19 17:38:09.693242556 (264h12m4.124322687s) [seg_pos={0 0}, upload=274674]: Begin upload segment... Aug 19 17:38:09.693302230 (264h12m4.124382361s) [seg_pos={0 0}, upload=274674]: Done begin upload segment. Aug 19 17:38:09.693343529 (264h12m4.12442367s) [noise=true , address=77.90.31.18:4825 , piece_id=UQLJVRVUBBN5LFBIAQL7YQ4IRAWI3SBVBHUT4IIFBROAUHXHVZ3A, seg_pos={0 0}, upload=274674]: Uploading piece... Aug 19 17:38:09.693350092 (264h12m4.124430213s) [noise=true , address=45.91.116.14:4222 , piece_id=DXHEK43V7USJLYKPF5PKEHLJPVP4OPH2H3TQVGUBMK5MVRWQHXJQ, seg_pos={0 0}, upload=274674]: Uploading piece... Aug 19 17:38:09.693395940 (264h12m4.124476081s) [noise=true , address=45.91.116.11:4140 , piece_id=KQNAZLV4XPDYYLJQUMI2UL3GVLA7QTEZ6TU3HDYFYB7LPRBSRQFA, seg_pos={0 0}, upload=274674]: Uploading piece... Aug 19 17:38:09.693960980 (264h12m4.125041131s) [noise=true , address=79.127.201.215:4343 , piece_id=MIATMAVPWKBJPVYH3XDYXQGKW7VWKRULLP6UV2ZNKJZ7YDVJJEBQ, seg_pos={0 0}, upload=274674]: Uploading piece... Aug 19 17:38:09.694358652 (264h12m4.125438783s) [noise=true , address=104.36.82.5:4852 , piece_id=UALXJPWV34HHWPC6GNCOFNP2RDC3KF6ZY4FUHLL3MWR2UZDCXNCQ, seg_pos={0 0}, upload=274674]: Uploading piece... Aug 19 17:38:09.694689475 (264h12m4.125769606s) [noise=true , address=104.36.82.58:4269 , piece_id=CTQSY5PQFUNXUJR2JXSI7O2EHMYTOAHUSHGTMVTI42E45RYFIGJA, seg_pos={0 0}, upload=274674]: Uploading piece... Aug 19 17:38:09.695011601 (264h12m4.126091732s) [noise=true , address=104.36.82.60:4364 , piece_id=A5BGK2CC62VYRH27SR2MONG2GPSZ5WONCHAF42LFPXOTM2W7XIJA, seg_pos={0 0}, upload=274674]: Uploading piece... Aug 19 17:38:09.695441633 (264h12m4.126521764s) [noise=true , address=104.36.82.9:4028 , piece_id=4LUYCAISMQC3SGDLJZSDFDJET4R54U3N4B6ZAJ35UDXB5CT3GV2Q, seg_pos={0 0}, upload=274674]: Uploading piece... Aug 19 17:38:09.695845597 (264h12m4.126925728s) [noise=true , address=77.90.31.15:4701 , piece_id=CCTT764Z3HFZFT4PE7YOOBVVGDASLFYVFGVXGWADBLDLURBOKGGQ, seg_pos={0 0}, upload=274674]: Uploading piece... Aug 19 17:38:09.696191328 (264h12m4.127271459s) [noise=true , address=45.91.116.5:4302 , piece_id=D2ZYYIFLA7GFPEII4X75HVJGL5DOM6BZLQFXL7HYVXYLNCL4CDQQ, seg_pos={0 0}, upload=274674]: Uploading piece... Aug 19 17:38:09.696539624 (264h12m4.127619755s) [noise=true , address=77.90.31.14:4689 , piece_id=6FO4WN5BXNW6UFGW2ALS5YBYBP5FZCAZBLQZXAULFCN65NHH7N6A, seg_pos={0 0}, upload=274674]: Uploading piece... Aug 19 17:38:09.696809421 (264h12m4.127889552s) [noise=true , address=79.127.201.214:4299 , piece_id=C4JSI4OPEZPPHADT6QTVJGEXQNVSHEK42KNYQBV3S2OH7Z5Z7HHA, seg_pos={0 0}, upload=274674]: Uploading piece... Aug 19 17:38:09.697172445 (264h12m4.128252576s) [noise=true , address=104.36.82.43:4855 , piece_id=3H2PUTGFVC2PR224NBHGRB7GLWBOL2MNXXWJ7JF23HYVF53TEWFQ, seg_pos={0 0}, upload=274674]: Uploading piece... Aug 19 17:38:09.697485064 (264h12m4.128565205s) [noise=true , address=77.90.31.12:4619 , piece_id=XMWKT6JBVLOPTRUIPNEK4GXVCH5OOVO5WYWDGWMXGA2VVVD74CBA, seg_pos={0 0}, upload=274674]: Uploading piece... Aug 19 17:38:09.697774858 (264h12m4.128854989s) [noise=true , address=104.36.82.38:4714 , piece_id=SZ42LIQE4LXTBCZY4BTXWS3EL27UYIQS3AAXGWNLH65TUZWNUYSQ, seg_pos={0 0}, upload=274674]: Uploading piece... Aug 19 17:38:09.698096103 (264h12m4.129176234s) [noise=true , address=104.36.82.4:4817 , piece_id=JNKPBDLUZUWI22ZLGTCD3BZJBAUX7TZVAGXJVFZOEJPK2Z3PMADA, seg_pos={0 0}, upload=274674]: Uploading piece... Aug 19 17:38:09.698352945 (264h12m4.129433076s) [noise=true , address=45.91.116.13:4183 , piece_id=OMUE6JFSTRIF64VYM5XCHNL7U26SEXP3IGWDBHP5G2E6YJQETEUQ, seg_pos={0 0}, upload=274674]: Uploading piece... Aug 19 17:38:09.698876106 (264h12m4.129956227s) [noise=true , address=104.36.82.6:4879 , piece_id=JKK5HMPYTMR7PNWHVPBDHPH3K6H7VM7AA4K4X4WXQ5LN57VRBUFA, seg_pos={0 0}, upload=274674]: Uploading piece... Aug 19 17:38:11.192631202 (264h12m5.623711333s) [upload=274674]: Next returned nil segment Aug 19 17:38:11.192648606 (264h12m5.623728737s) [upload=274674]: Waiting for error group managing segments... Aug 19 17:38:11.591537039 (264h12m6.02261717s) [noise=true , address=45.91.116.14:4222 , piece_id=DXHEK43V7USJLYKPF5PKEHLJPVP4OPH2H3TQVGUBMK5MVRWQHXJQ, seg_pos={0 0}, upload=274674]: Done uploading piece. err: <nil> Aug 19 17:38:11.606093581 (264h12m6.037173712s) [noise=true , address=77.90.31.14:4689 , piece_id=6FO4WN5BXNW6UFGW2ALS5YBYBP5FZCAZBLQZXAULFCN65NHH7N6A, seg_pos={0 0}, upload=274674]: Done uploading piece. err: <nil> Aug 19 17:38:11.610339916 (264h12m6.041420047s) [noise=true , address=77.90.31.15:4701 , piece_id=CCTT764Z3HFZFT4PE7YOOBVVGDASLFYVFGVXGWADBLDLURBOKGGQ, seg_pos={0 0}, upload=274674]: Done uploading piece. err: <nil> Aug 19 17:38:11.630055040 (264h12m6.061135171s) [noise=true , address=79.127.201.215:4343 , piece_id=MIATMAVPWKBJPVYH3XDYXQGKW7VWKRULLP6UV2ZNKJZ7YDVJJEBQ, seg_pos={0 0}, upload=274674]: Done uploading piece. err: <nil> Aug 19 17:38:11.678703675 (264h12m6.109783797s) [noise=true , address=104.36.82.9:4028 , piece_id=4LUYCAISMQC3SGDLJZSDFDJET4R54U3N4B6ZAJ35UDXB5CT3GV2Q, seg_pos={0 0}, upload=274674]: Done uploading piece. err: <nil> Aug 19 17:38:11.693543009 (264h12m6.12462314s) [noise=true , address=104.36.82.60:4364 , piece_id=A5BGK2CC62VYRH27SR2MONG2GPSZ5WONCHAF42LFPXOTM2W7XIJA, seg_pos={0 0}, upload=274674]: Done uploading piece. err: <nil> Aug 19 17:38:11.693568448 (264h12m6.124648579s) [noise=true , address=45.91.116.13:4183 , piece_id=OMUE6JFSTRIF64VYM5XCHNL7U26SEXP3IGWDBHP5G2E6YJQETEUQ, seg_pos={0 0}, upload=274674]: Done uploading piece. err: <nil> Aug 19 17:38:11.698831207 (264h12m6.129911329s) [noise=true , address=104.36.82.5:4852 , piece_id=UALXJPWV34HHWPC6GNCOFNP2RDC3KF6ZY4FUHLL3MWR2UZDCXNCQ, seg_pos={0 0}, upload=274674]: Done uploading piece. err: <nil> Aug 19 17:38:11.698886834 (264h12m6.129966955s) [noise=true , address=77.90.31.12:4619 , piece_id=XMWKT6JBVLOPTRUIPNEK4GXVCH5OOVO5WYWDGWMXGA2VVVD74CBA, seg_pos={0 0}, upload=274674]: Done uploading piece. err: <nil> Aug 19 17:38:11.714704781 (264h12m6.145784912s) [noise=true , address=79.127.201.214:4299 , piece_id=C4JSI4OPEZPPHADT6QTVJGEXQNVSHEK42KNYQBV3S2OH7Z5Z7HHA, seg_pos={0 0}, upload=274674]: Done uploading piece. err: <nil> Aug 19 17:38:11.720319795 (264h12m6.151399956s) [noise=true , address=45.91.116.11:4140 , piece_id=KQNAZLV4XPDYYLJQUMI2UL3GVLA7QTEZ6TU3HDYFYB7LPRBSRQFA, seg_pos={0 0}, upload=274674]: Done uploading piece. err: <nil> Aug 19 17:38:11.722063401 (264h12m6.153143532s) [noise=true , address=104.36.82.6:4879 , piece_id=JKK5HMPYTMR7PNWHVPBDHPH3K6H7VM7AA4K4X4WXQ5LN57VRBUFA, seg_pos={0 0}, upload=274674]: Done uploading piece. err: <nil> Aug 19 17:38:11.726762272 (264h12m6.157842393s) [noise=true , address=104.36.82.38:4714 , piece_id=SZ42LIQE4LXTBCZY4BTXWS3EL27UYIQS3AAXGWNLH65TUZWNUYSQ, seg_pos={0 0}, upload=274674]: Done uploading piece. err: <nil> Aug 19 17:38:11.734258877 (264h12m6.165339008s) [noise=true , address=104.36.82.58:4269 , piece_id=CTQSY5PQFUNXUJR2JXSI7O2EHMYTOAHUSHGTMVTI42E45RYFIGJA, seg_pos={0 0}, upload=274674]: Done uploading piece. err: <nil> Aug 19 17:38:11.734284346 (264h12m6.165364477s) [seg_pos={0 0}, upload=274674]: Segment reached optimal threshold of 14 pieces. Aug 19 17:38:11.734416277 (264h12m6.165496408s) [noise=true , address=104.36.82.4:4817 , piece_id=JNKPBDLUZUWI22ZLGTCD3BZJBAUX7TZVAGXJVFZOEJPK2Z3PMADA, seg_pos={0 0}, upload=274674]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RSBodzJxMye65QgkyrzkWCKQ49cnfdu4haUZKW23nWpqzMcYz): protocol: expected piece hash; context canceled; EOF Aug 19 17:38:11.734486873 (264h12m6.165567004s) [noise=true , address=104.36.82.43:4855 , piece_id=3H2PUTGFVC2PR224NBHGRB7GLWBOL2MNXXWJ7JF23HYVF53TEWFQ, seg_pos={0 0}, upload=274674]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TGTRhEi2KsbrkC6ZmgGkMaJri9uhzEcmXe1J4nJjpEd1BE4ka): protocol: expected piece hash; context canceled; EOF Aug 19 17:38:11.734587926 (264h12m6.165668057s) [noise=true , address=77.90.31.18:4825 , piece_id=UQLJVRVUBBN5LFBIAQL7YQ4IRAWI3SBVBHUT4IIFBROAUHXHVZ3A, seg_pos={0 0}, upload=274674]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PqKyfFxfVKTGARbPqBcQL3Naqjdf5uR44PeLi3sJpu57dcm1x): protocol: expected piece hash; context canceled; EOF Aug 19 17:38:11.734592956 (264h12m6.165673087s) [noise=true , address=45.91.116.5:4302 , piece_id=D2ZYYIFLA7GFPEII4X75HVJGL5DOM6BZLQFXL7HYVXYLNCL4CDQQ, seg_pos={0 0}, upload=274674]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pJWhSLza8WaMuvvDwH4nae57UxQmfCdDvAkQ3v25MgPYHoZXx): protocol: expected piece hash; context canceled; EOF Aug 19 17:38:11.734645206 (264h12m6.165725337s) [seg_pos={0 0}, upload=274674]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 19 17:38:11.734756098 (264h12m6.165836239s) [upload=274674]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 19 17:38:11.734764033 (264h12m6.165844154s) [upload=274674]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 17:38:12.023665554 (264h12m6.454745685s) [upload=274674]: Done uploading segments... Aug 19 17:38:12.023686044 (264h12m6.454766195s) [upload=274674]: Upload finished. err: <nil> Duration: 5.765621857s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..................X.....X...................................] duration: 5.937658724s FAIL (2025-08-19 17:49:13 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.937675547s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 1.921365092s FAIL (2025-08-19 17:49:12 UTC) orbiter warning: operation: ListObjects ======ERROR====== uplink: metainfo: metaclient: context deadline exceeded ======LOGS====== Aug 19 17:35:36.607320794 (264h9m31.038400925s) [upload=274632]: Starting upload Aug 19 17:35:36.607367684 (264h9m31.038447815s) [upload=274632]: Done starting upload Aug 19 17:35:36.607435082 (264h9m31.038515213s) [upload=274632]: Uploading segments... Aug 19 17:35:36.607490538 (264h9m31.038570669s) [upload=274632]: Got next segment. Inline: true Aug 19 17:35:36.607495088 (264h9m31.038575219s) [upload=274632]: Waiting for error group managing segments... Aug 19 17:35:36.607537528 (264h9m31.038617659s) [upload=274632]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 17:35:36.607541656 (264h9m31.038621787s) [upload=274632]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 19 17:35:36.607544051 (264h9m31.038624182s) [upload=274632]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 17:35:36.889055260 (264h9m31.320135391s) [upload=274632]: Done uploading segments... Aug 19 17:35:36.889082943 (264h9m31.320163124s) [upload=274632]: Upload finished. err: <nil> Duration:* 1.921397754s core uplink us-select-1 us1-placement-20 metainfo-projects

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)