Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 7 out of 60 is failed (threshold=3) [.................................................XXXXXXX....] duration: 6.459534368s FAIL (2025-08-01 22:00:29 UTC) orbiter warning: Operation: DownloadObject uplink: object not found ("orbiter_us_0/4M") Duration: 6.459566841s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [...................................................XX.X.X...] duration: 6.292650281s FAIL (2025-08-01 22:00:24 UTC) orbiter warning: Operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 7NK72474VWE4E3YOBCDZ5O5ZOH2MSA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB): protocol: write tcp 188.166.225.170:38368->104.36.82.12:4151: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:56:50.418604753 (51h59m40.356263252s) [seg_pos={0 0}, upload=49961]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 1 21:56:50.418680879 (51h59m40.356339378s) [upload=49961]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 1 21:56:50.418685998 (51h59m40.356344487s) [upload=49961]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 1 21:56:50.726563872 (51h59m40.664222371s) [upload=49961]: Done uploading segments... Aug 1 21:56:50.726586586 (51h59m40.664245085s) [upload=49961]: Upload finished. err: <nil> Aug 1 21:57:28.852869669 (52h0m18.790528158s) [upload=49973]: Starting upload Aug 1 21:57:28.852942389 (52h0m18.790600888s) [upload=49973]: Done starting upload Aug 1 21:57:28.853007122 (52h0m18.790665611s) [upload=49973]: Uploading segments... Aug 1 21:57:28.853174002 (52h0m18.790832491s) [upload=49973]: Got next segment. Inline: false Aug 1 21:57:28.853208168 (52h0m18.790866657s) [upload=49973]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 1 21:57:28.853212666 (52h0m18.790871165s) [upload=49973]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 1 21:57:29.095087519 (52h0m19.032746008s) [seg_pos={0 0}, upload=49973]: Begin upload segment... Aug 1 21:57:29.095129960 (52h0m19.032788459s) [seg_pos={0 0}, upload=49973]: Done begin upload segment. Aug 1 21:57:29.095164997 (52h0m19.032823496s) [noise=true , address=77.90.31.21:4878 , piece_id=BLNTYBOAIZWISSUJYK6BSBPTSNL6QWLGEN6AP6P4NPDVSURYG5AA, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.095171800 (52h0m19.032830289s) [noise=true , address=104.36.82.6:4875 , piece_id=CS5AMB7CIBQWWOU57Z3RCCSNVKIPE4B5QWU5YIQYI2UCWH7PAPVA, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.095510027 (52h0m19.033168526s) [noise=true , address=77.90.31.18:4806 , piece_id=52B6CAHGLFVTFIDQHHHAWKCBP2WJJY7OTTGRZZ7CEUH3HDA43Q6A, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.095544193 (52h0m19.033202692s) [noise=true , address=77.90.31.10:4518 , piece_id=WUOVXUSDG6YQTDPWG4YO7VLJNZZVLH6VRGMWOJABG2EJRU7XHP5A, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.095774283 (52h0m19.033432782s) [noise=true , address=104.36.82.7:4909 , piece_id=QHJCVBT6KTU34K2KWFNACWYVVE3WJ2RU76BHVSKL4HLCZEFXZYPA, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.095819880 (52h0m19.033478379s) [noise=true , address=45.91.116.12:4225 , piece_id=PBQPLA5ZYRFPSKHDKCEFOEBS5T3GYMP2FLSFDQWJLVY756KPLJNQ, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.096021656 (52h0m19.033680155s) [noise=true , address=45.91.116.13:4153 , piece_id=PMLKMZO66KXF3ZONO5ZZOQZICFK5XNGL6MN5FFBKICVFU5AWU6PQ, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.096069618 (52h0m19.033728117s) [noise=true , address=45.91.116.9:4262 , piece_id=A3F4L3RR6WPFMBFFKLATNGG3CASBIPWQP5O3G5URFNTXBE3ZW3VA, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.096348041 (52h0m19.03400654s) [noise=true , address=104.36.82.5:4840 , piece_id=UYVKVFCJJP7XZYRIVPS4ABGEMDUMNR7YW4QGG5ZFTFCGNCJO3CSQ, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.096354803 (52h0m19.034013292s) [noise=true , address=77.90.31.12:4590 , piece_id=D6FUBIFCQJSDRGWEP5GG34AQPNB3UROCYOATD6SRTJ7KAYVMFCOA, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.096608579 (52h0m19.034267078s) [noise=true , address=77.90.31.16:4733 , piece_id=DMASMGOUATLPEKCGLHKNNIMMFYVVGSULNUOMJU6RJY3ULBDOY6KQ, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.096830774 (52h0m19.034489273s) [noise=true , address=104.36.82.4:4802 , piece_id=CLM2G33M6AK5HIESRW6LZY33NKHIR2PGITWJ23NPILRCA6UYIEHA, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.097107233 (52h0m19.034765732s) [noise=true , address=77.90.31.19:4843 , piece_id=UFONZYHL7EAFYE5PRW32RKUVQPMZPXBEFLLIFJMAB2TXJ3C6KKNA, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.097324328 (52h0m19.034982827s) [noise=true , address=77.90.31.17:4771 , piece_id=Y3WSALJX23CWPPYQ3XYMCFOSBLBX2E2JQCCS7EOEYNBWY4CDRT3Q, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.098682446 (52h0m19.036340945s) [noise=true , address=77.90.31.15:4698 , piece_id=TN4OTPUD47FD6WOLZBEHNDI43WY6QVO5JK2JNJXNVM2IT6XMGERQ, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.099041603 (52h0m19.036700102s) [noise=true , address=45.91.116.11:4118 , piece_id=AKZIOAR6NRNO7JWCQZIKNSL7IUH5S4IG3KPG3IY2FAKG74DH3E3Q, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:30.718412255 (52h0m20.656070754s) [upload=49973]: Next returned nil segment Aug 1 21:57:30.718429869 (52h0m20.656088368s) [upload=49973]: Waiting for error group managing segments... Aug 1 21:57:31.107386675 (52h0m21.045045164s) [noise=true , address=45.91.116.13:4153 , piece_id=PMLKMZO66KXF3ZONO5ZZOQZICFK5XNGL6MN5FFBKICVFU5AWU6PQ, seg_pos={0 0}, upload=49973]: Done uploading piece. err: <nil> Aug 1 21:57:31.108529412 (52h0m21.046187911s) [noise=true , address=77.90.31.12:4590 , piece_id=D6FUBIFCQJSDRGWEP5GG34AQPNB3UROCYOATD6SRTJ7KAYVMFCOA, seg_pos={0 0}, upload=49973]: Done uploading piece. err: <nil> Aug 1 21:57:31.109975858 (52h0m21.047634347s) [noise=true , address=77.90.31.15:4698 , piece_id=TN4OTPUD47FD6WOLZBEHNDI43WY6QVO5JK2JNJXNVM2IT6XMGERQ, seg_pos={0 0}, upload=49973]: Done uploading piece. err: <nil> Aug 1 21:57:31.115273013 (52h0m21.052931502s) [noise=true , address=77.90.31.10:4518 , piece_id=WUOVXUSDG6YQTDPWG4YO7VLJNZZVLH6VRGMWOJABG2EJRU7XHP5A, seg_pos={0 0}, upload=49973]: Done uploading piece. err: <nil> Aug 1 21:57:31.116767281 (52h0m21.054425781s) [noise=true , address=77.90.31.18:4806 , piece_id=52B6CAHGLFVTFIDQHHHAWKCBP2WJJY7OTTGRZZ7CEUH3HDA43Q6A, seg_pos={0 0}, upload=49973]: Done uploading piece. err: <nil> Aug 1 21:57:31.201146107 (52h0m21.138804606s) [noise=true , address=77.90.31.16:4733 , piece_id=DMASMGOUATLPEKCGLHKNNIMMFYVVGSULNUOMJU6RJY3ULBDOY6KQ, seg_pos={0 0}, upload=49973]: Done uploading piece. err: <nil> Aug 1 21:57:31.201833732 (52h0m21.139492231s) [noise=true , address=77.90.31.19:4843 , piece_id=UFONZYHL7EAFYE5PRW32RKUVQPMZPXBEFLLIFJMAB2TXJ3C6KKNA, seg_pos={0 0}, upload=49973]: Done uploading piece. err: <nil> Aug 1 21:57:31.211007613 (52h0m21.148666112s) [noise=true , address=45.91.116.9:4262 , piece_id=A3F4L3RR6WPFMBFFKLATNGG3CASBIPWQP5O3G5URFNTXBE3ZW3VA, seg_pos={0 0}, upload=49973]: Done uploading piece. err: <nil> Aug 1 21:57:31.215929169 (52h0m21.153587668s) [noise=true , address=77.90.31.17:4771 , piece_id=Y3WSALJX23CWPPYQ3XYMCFOSBLBX2E2JQCCS7EOEYNBWY4CDRT3Q, seg_pos={0 0}, upload=49973]: Done uploading piece. err: <nil> Aug 1 21:57:31.222973975 (52h0m21.160632474s) [noise=true , address=77.90.31.21:4878 , piece_id=BLNTYBOAIZWISSUJYK6BSBPTSNL6QWLGEN6AP6P4NPDVSURYG5AA, seg_pos={0 0}, upload=49973]: Done uploading piece. err: <nil> Aug 1 21:57:31.234392441 (52h0m21.17205094s) [noise=true , address=45.91.116.12:4225 , piece_id=PBQPLA5ZYRFPSKHDKCEFOEBS5T3GYMP2FLSFDQWJLVY756KPLJNQ, seg_pos={0 0}, upload=49973]: Done uploading piece. err: <nil> Aug 1 21:57:31.284794305 (52h0m21.222452804s) [noise=true , address=45.91.116.11:4118 , piece_id=AKZIOAR6NRNO7JWCQZIKNSL7IUH5S4IG3KPG3IY2FAKG74DH3E3Q, seg_pos={0 0}, upload=49973]: Done uploading piece. err: <nil> Aug 1 21:57:58.202420932 (52h0m48.140079471s) [noise=true , address=104.36.82.6:4875 , piece_id=CS5AMB7CIBQWWOU57Z3RCCSNVKIPE4B5QWU5YIQYI2UCWH7PAPVA, seg_pos={0 0}, upload=49973]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:104.36.82.6:4875): protocol: write tcp 188.166.225.170:40166->104.36.82.6:4875: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:58.202441131 (52h0m48.14009963s) [noise=true , address=104.36.82.5:4840 , piece_id=UYVKVFCJJP7XZYRIVPS4ABGEMDUMNR7YW4QGG5ZFTFCGNCJO3CSQ, seg_pos={0 0}, upload=49973]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:104.36.82.5:4840): protocol: rpc: dial tcp 104.36.82.5:4840: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:58.202448445 (52h0m48.140106954s) [noise=true , address=104.36.82.7:4909 , piece_id=QHJCVBT6KTU34K2KWFNACWYVVE3WJ2RU76BHVSKL4HLCZEFXZYPA, seg_pos={0 0}, upload=49973]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:104.36.82.7:4909): protocol: rpc: dial tcp 104.36.82.7:4909: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:58.202496487 (52h0m48.140155026s) [noise=true , address=104.36.82.4:4802 , piece_id=CLM2G33M6AK5HIESRW6LZY33NKHIR2PGITWJ23NPILRCA6UYIEHA, seg_pos={0 0}, upload=49973]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: write tcp 188.166.225.170:47696->104.36.82.4:4802: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:58.202524240 (52h0m48.140182739s) [seg_pos={0 0}, upload=49973]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Aug 1 21:57:58.435983932 (52h0m48.373642431s) [upload=49973]: Done uploading segments... Aug 1 21:57:58.436007667 (52h0m48.373666166s) [upload=49973]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.292668947s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 7 out of 60 is failed (threshold=3) [..................................................XXXXXXX...] duration: 26.73143158s FAIL (2025-08-01 22:00:13 UTC) orbiter warning: Operation: DownloadObject uplink: object not found ("orbiter_us_0/4M") Duration: 26.731457389s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [....................................................XXX.....] duration: 5.795392195s FAIL (2025-08-01 22:00:01 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.795414437s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [.....................................................XXXX...] duration: 11.269606877s FAIL (2025-08-01 21:59:49 UTC) orbiter warning: Operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 7) ======LOGS====== K4BW277ZSGXJYQDA66OMEBVDC7DAWSOJAPX42HANNGRZN35KHQ, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.481101510 (51h59m34.418760009s) [noise=true , address=104.36.82.5:4842 , piece_id=LV7YAOMPE2AV7TF2WPWR7QJNLT4JZAIPSVZN3E23U5KDBNKZFD6A, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.481422003 (51h59m34.419080502s) [noise=true , address=79.127.205.235:4683 , piece_id=3QHDNAPG4GRLJFVVNJNFXNGKTGKUEMWWUBMW62HHE5N5LJDO4KWA, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.482030897 (51h59m34.419689396s) [noise=true , address=79.127.219.51:4213 , piece_id=L2WSPI5BEIFLGQKJIK4EWXFQXNQRQ7HVKPOWFYRPPUGNAL5DVYWA, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.482285215 (51h59m34.419943704s) [noise=true , address=104.36.82.14:4251 , piece_id=O7NADU347E2Q26TPD3IUCK6WE5UVRITC6ALZHBE5SWYTKWXFBMXA, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.482518370 (51h59m34.420176859s) [noise=true , address=79.127.220.98:4402 , piece_id=DVLSYRFFZ6UT3QRSVAN7AMXVYZNI6PWG2IITHLRUUJATJVK3QOGA, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.482724304 (51h59m34.420382803s) [noise=true , address=79.127.201.214:4297 , piece_id=MPTMGVEYC4SATIN4ANCR2ELVRXHYWRDLMHWQW7HM6UOIV2N6R7ZA, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.482928555 (51h59m34.420587054s) [noise=true , address=104.36.82.37:4680 , piece_id=X4PDUE4WXUUWX6JQVO4NWPQCBMHMHC5YHPIHDQVYYOI2FXV2GVJQ, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.483134960 (51h59m34.420793449s) [noise=true , address=104.36.82.50:4027 , piece_id=OW7BODU5N7ZEEGE2FHSLNYVJT2Z25LY3J42TZEOR5SP2EWIN6X7A, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.483338480 (51h59m34.420996979s) [noise=true , address=104.36.82.55:4204 , piece_id=R5CYXWAVV4ZEMCV7LF4D4XDC4G6KLVBONAH2TH5EIEVL7QHUTRWQ, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.483582847 (51h59m34.421241356s) [noise=true , address=104.36.82.11:4134 , piece_id=RMTNNNC2VD7MGWJDAMT6ZCF3GR43SHANFVAGJ6EHVH7WO5XA6VTA, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.483833737 (51h59m34.421492236s) [noise=true , address=104.36.82.43:4856 , piece_id=2SINBLHGRUOWUO4XHBNBG45YE3PW7RNF3QSNVCQXNCFEFHRBFNJQ, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.484021896 (51h59m34.421680385s) [noise=true , address=104.36.82.52:4101 , piece_id=XYPLQM7QITCCX7T4NKIXP5MZ7I6F24EJJTG37K4NPOBKTUFHULRQ, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.484301662 (51h59m34.421960182s) [noise=true , address=79.127.201.215:4352 , piece_id=I6BRYI5XIACAAAAAE7HQOBCF4OTALMOVPPNBWQCRSH2MM6WUUURQ, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.484712097 (51h59m34.422370596s) [noise=true , address=64.49.13.138:4484 , piece_id=DZ5Y4HTS6BKW2DJ5CVOB64TW4SXB6DNCJEGFUJPLAUXUEHRZRTUA, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:45.576021470 (51h59m35.513679959s) [upload=49960]: Next returned nil segment Aug 1 21:56:45.576036609 (51h59m35.513695098s) [upload=49960]: Waiting for error group managing segments... Aug 1 21:56:45.860378776 (51h59m35.798037275s) [noise=true , address=64.49.13.138:4484 , piece_id=DZ5Y4HTS6BKW2DJ5CVOB64TW4SXB6DNCJEGFUJPLAUXUEHRZRTUA, seg_pos={0 0}, upload=49960]: Done uploading piece. err: <nil> Aug 1 21:56:45.884184188 (51h59m35.821842687s) [noise=true , address=79.127.205.235:4683 , piece_id=3QHDNAPG4GRLJFVVNJNFXNGKTGKUEMWWUBMW62HHE5N5LJDO4KWA, seg_pos={0 0}, upload=49960]: Done uploading piece. err: <nil> Aug 1 21:56:46.145493612 (51h59m36.083152101s) [noise=true , address=79.127.220.98:4402 , piece_id=DVLSYRFFZ6UT3QRSVAN7AMXVYZNI6PWG2IITHLRUUJATJVK3QOGA, seg_pos={0 0}, upload=49960]: Done uploading piece. err: <nil> Aug 1 21:56:46.257505313 (51h59m36.195163802s) [noise=true , address=79.127.220.97:4371 , piece_id=5WGPRFKQETIHTSMA42AM2C3UGCFYJ3DMMPAI2L7DD6E2P37KJKNA, seg_pos={0 0}, upload=49960]: Done uploading piece. err: <nil> Aug 1 21:56:46.334426181 (51h59m36.27208468s) [noise=true , address=79.127.201.215:4352 , piece_id=I6BRYI5XIACAAAAAE7HQOBCF4OTALMOVPPNBWQCRSH2MM6WUUURQ, seg_pos={0 0}, upload=49960]: Done uploading piece. err: <nil> Aug 1 21:56:46.386119275 (51h59m36.323777774s) [noise=true , address=79.127.201.214:4297 , piece_id=MPTMGVEYC4SATIN4ANCR2ELVRXHYWRDLMHWQW7HM6UOIV2N6R7ZA, seg_pos={0 0}, upload=49960]: Done uploading piece. err: <nil> Aug 1 21:56:46.395362048 (51h59m36.333020547s) [noise=true , address=79.127.219.51:4213 , piece_id=L2WSPI5BEIFLGQKJIK4EWXFQXNQRQ7HVKPOWFYRPPUGNAL5DVYWA, seg_pos={0 0}, upload=49960]: Done uploading piece. err: <nil> Aug 1 21:57:43.392434383 (52h0m33.330092872s) [noise=true , address=104.36.82.37:4680 , piece_id=X4PDUE4WXUUWX6JQVO4NWPQCBMHMHC5YHPIHDQVYYOI2FXV2GVJQ, seg_pos={0 0}, upload=49960]: Done uploading piece. err: ecclient: upload failed (node:1bXAqpxGhUvzwYEDGBqYRhZdTeMJcV3TYfngkKTCumpt9Nupi3, address:104.36.82.37:4680): protocol: rpc: dial tcp 104.36.82.37:4680: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:43.392477967 (52h0m33.330136466s) [noise=true , address=104.36.82.11:4134 , piece_id=RMTNNNC2VD7MGWJDAMT6ZCF3GR43SHANFVAGJ6EHVH7WO5XA6VTA, seg_pos={0 0}, upload=49960]: Done uploading piece. err: ecclient: upload failed (node:12R3Z5AgkBmWXMv5XgRyfcyqkG2gmyJs9ogT24kcLy6nuB9bzUG, address:104.36.82.11:4134): protocol: rpc: dial tcp 104.36.82.11:4134: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:43.392514086 (52h0m33.330172585s) [noise=true , address=104.36.82.55:4204 , piece_id=R5CYXWAVV4ZEMCV7LF4D4XDC4G6KLVBONAH2TH5EIEVL7QHUTRWQ, seg_pos={0 0}, upload=49960]: Done uploading piece. err: ecclient: upload failed (node:1rBZH6aLHsnWh22uNzaMiMwmLJCDMn4bYbNC5jRG7KCEuRfqpH, address:104.36.82.55:4204): protocol: write tcp 188.166.225.170:48550->104.36.82.55:4204: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:43.392546537 (52h0m33.330205027s) [noise=true , address=104.36.82.59:4327 , piece_id=OZDEYIY6B4WYZULRDRL2UBOJH2JQNY3YWSITZKWDO7GTT72AN5NQ, seg_pos={0 0}, upload=49960]: Done uploading piece. err: ecclient: upload failed (node:1V4C6AWGbyYQWfinoSj7iKovHeTJ9hqypyC5U9rZ8pveXBBfCY, address:104.36.82.59:4327): protocol: rpc: dial tcp 104.36.82.59:4327: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:43.392579040 (52h0m33.33023754s) [noise=true , address=104.36.82.5:4842 , piece_id=LV7YAOMPE2AV7TF2WPWR7QJNLT4JZAIPSVZN3E23U5KDBNKZFD6A, seg_pos={0 0}, upload=49960]: Done uploading piece. err: ecclient: upload failed (node:12q4VdT5BcjUb16PBYHK6zrnW8dqENKrcGx9zdLtNefDVB1MUSn, address:104.36.82.5:4842): protocol: context deadline exceeded; context deadline exceeded Aug 1 21:57:43.392627953 (52h0m33.330286442s) [noise=true , address=104.36.82.52:4101 , piece_id=XYPLQM7QITCCX7T4NKIXP5MZ7I6F24EJJTG37K4NPOBKTUFHULRQ, seg_pos={0 0}, upload=49960]: Done uploading piece. err: ecclient: upload failed (node:12WWXeBjFZR9PzciNfs35fFcx87N9GwDpMsPHnksFDugtaBGWp4, address:104.36.82.52:4101): protocol: rpc: dial tcp 104.36.82.52:4101: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:43.392653192 (52h0m33.330311691s) [noise=true , address=104.36.82.14:4251 , piece_id=O7NADU347E2Q26TPD3IUCK6WE5UVRITC6ALZHBE5SWYTKWXFBMXA, seg_pos={0 0}, upload=49960]: Done uploading piece. err: ecclient: upload failed (node:12btf6Z38aTMx7MaBxhUyZtFqUPgeMtP2aU8Y6FbV8zwu57zArd, address:104.36.82.14:4251): protocol: rpc: dial tcp 104.36.82.14:4251: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:43.392670836 (52h0m33.330329335s) [noise=true , address=104.36.82.12:4166 , piece_id=6ZIWCOBY76GJ6DHZF5G23U4IYTOFB5SMQW5TW3YITYH227PV2FGA, seg_pos={0 0}, upload=49960]: Done uploading piece. err: ecclient: upload failed (node:12iCB7c4ZfpqafpD1PtpbE6BfU81wJW65LmrPVJXtD58yeuoR9t, address:104.36.82.12:4166): protocol: rpc: dial tcp 104.36.82.12:4166: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:43.392698018 (52h0m33.330356517s) [noise=true , address=104.36.82.50:4027 , piece_id=OW7BODU5N7ZEEGE2FHSLNYVJT2Z25LY3J42TZEOR5SP2EWIN6X7A, seg_pos={0 0}, upload=49960]: Done uploading piece. err: ecclient: upload failed (node:1zEQhyuVxSHwdDUgDfcTog2W3oLBx4z9bYUMHpaaoLU1BTU9DQ, address:104.36.82.50:4027): protocol: write tcp 188.166.225.170:37784->104.36.82.50:4027: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:43.392715381 (52h0m33.33037387s) [noise=true , address=104.36.82.43:4856 , piece_id=2SINBLHGRUOWUO4XHBNBG45YE3PW7RNF3QSNVCQXNCFEFHRBFNJQ, seg_pos={0 0}, upload=49960]: Done uploading piece. err: ecclient: upload failed (node:12bQxwBfPEGtmMp6mtnWKsoZt3eMEjLdyYFahETWGHeKJ7BhkiZ, address:104.36.82.43:4856): protocol: rpc: dial tcp 104.36.82.43:4856: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:43.392730059 (52h0m33.330388558s) [noise=true , address=104.36.82.10:4082 , piece_id=45K4BW277ZSGXJYQDA66OMEBVDC7DAWSOJAPX42HANNGRZN35KHQ, seg_pos={0 0}, upload=49960]: Done uploading piece. err: ecclient: upload failed (node:12DKsNCtqxLvpJwLszdLs74V3eb8NH2VAV9V3bngzUczawmNoLK, address:104.36.82.10:4082): protocol: write tcp 188.166.225.170:46638->104.36.82.10:4082: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:43.392753424 (52h0m33.330411923s) [seg_pos={0 0}, upload=49960]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil> Aug 1 21:57:43.624550997 (52h0m33.562209566s) [upload=49960]: Done uploading segments... Aug 1 21:57:43.624582868 (52h0m33.562241407s) [upload=49960]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7) Duration: 11.269636094s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 3 out of 60 is failed (threshold=3) [...................................X............X.....X.....] duration: 10.976263239s FAIL (2025-08-01 21:59:49 UTC) orbiter warning: Operation: ListObjects unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/1K Duration: 10.976282746s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 7 out of 60 is failed (threshold=3) [...................................................XXXXXXX..] duration: 18.569406911s FAIL (2025-08-01 21:59:42 UTC) orbiter warning: Operation: DownloadObject uplink: object not found ("orbiter_us_0/4M") Duration: 18.569431258s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 6.278559465s FAIL (2025-08-01 21:59:25 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.278578913s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [....................................................XX.X.X..] duration: 6.677733522s FAIL (2025-08-01 21:59:18 UTC) orbiter warning: Operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 7NK72474VWE4E3YOBCDZ5O5ZOH2MSA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB): protocol: write tcp 188.166.225.170:38368->104.36.82.12:4151: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:56:50.418604753 (51h59m40.356263252s) [seg_pos={0 0}, upload=49961]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 1 21:56:50.418680879 (51h59m40.356339378s) [upload=49961]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 1 21:56:50.418685998 (51h59m40.356344487s) [upload=49961]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 1 21:56:50.726563872 (51h59m40.664222371s) [upload=49961]: Done uploading segments... Aug 1 21:56:50.726586586 (51h59m40.664245085s) [upload=49961]: Upload finished. err: <nil> Aug 1 21:57:28.852869669 (52h0m18.790528158s) [upload=49973]: Starting upload Aug 1 21:57:28.852942389 (52h0m18.790600888s) [upload=49973]: Done starting upload Aug 1 21:57:28.853007122 (52h0m18.790665611s) [upload=49973]: Uploading segments... Aug 1 21:57:28.853174002 (52h0m18.790832491s) [upload=49973]: Got next segment. Inline: false Aug 1 21:57:28.853208168 (52h0m18.790866657s) [upload=49973]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 1 21:57:28.853212666 (52h0m18.790871165s) [upload=49973]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 1 21:57:29.095087519 (52h0m19.032746008s) [seg_pos={0 0}, upload=49973]: Begin upload segment... Aug 1 21:57:29.095129960 (52h0m19.032788459s) [seg_pos={0 0}, upload=49973]: Done begin upload segment. Aug 1 21:57:29.095164997 (52h0m19.032823496s) [noise=true , address=77.90.31.21:4878 , piece_id=BLNTYBOAIZWISSUJYK6BSBPTSNL6QWLGEN6AP6P4NPDVSURYG5AA, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.095171800 (52h0m19.032830289s) [noise=true , address=104.36.82.6:4875 , piece_id=CS5AMB7CIBQWWOU57Z3RCCSNVKIPE4B5QWU5YIQYI2UCWH7PAPVA, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.095510027 (52h0m19.033168526s) [noise=true , address=77.90.31.18:4806 , piece_id=52B6CAHGLFVTFIDQHHHAWKCBP2WJJY7OTTGRZZ7CEUH3HDA43Q6A, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.095544193 (52h0m19.033202692s) [noise=true , address=77.90.31.10:4518 , piece_id=WUOVXUSDG6YQTDPWG4YO7VLJNZZVLH6VRGMWOJABG2EJRU7XHP5A, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.095774283 (52h0m19.033432782s) [noise=true , address=104.36.82.7:4909 , piece_id=QHJCVBT6KTU34K2KWFNACWYVVE3WJ2RU76BHVSKL4HLCZEFXZYPA, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.095819880 (52h0m19.033478379s) [noise=true , address=45.91.116.12:4225 , piece_id=PBQPLA5ZYRFPSKHDKCEFOEBS5T3GYMP2FLSFDQWJLVY756KPLJNQ, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.096021656 (52h0m19.033680155s) [noise=true , address=45.91.116.13:4153 , piece_id=PMLKMZO66KXF3ZONO5ZZOQZICFK5XNGL6MN5FFBKICVFU5AWU6PQ, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.096069618 (52h0m19.033728117s) [noise=true , address=45.91.116.9:4262 , piece_id=A3F4L3RR6WPFMBFFKLATNGG3CASBIPWQP5O3G5URFNTXBE3ZW3VA, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.096348041 (52h0m19.03400654s) [noise=true , address=104.36.82.5:4840 , piece_id=UYVKVFCJJP7XZYRIVPS4ABGEMDUMNR7YW4QGG5ZFTFCGNCJO3CSQ, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.096354803 (52h0m19.034013292s) [noise=true , address=77.90.31.12:4590 , piece_id=D6FUBIFCQJSDRGWEP5GG34AQPNB3UROCYOATD6SRTJ7KAYVMFCOA, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.096608579 (52h0m19.034267078s) [noise=true , address=77.90.31.16:4733 , piece_id=DMASMGOUATLPEKCGLHKNNIMMFYVVGSULNUOMJU6RJY3ULBDOY6KQ, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.096830774 (52h0m19.034489273s) [noise=true , address=104.36.82.4:4802 , piece_id=CLM2G33M6AK5HIESRW6LZY33NKHIR2PGITWJ23NPILRCA6UYIEHA, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.097107233 (52h0m19.034765732s) [noise=true , address=77.90.31.19:4843 , piece_id=UFONZYHL7EAFYE5PRW32RKUVQPMZPXBEFLLIFJMAB2TXJ3C6KKNA, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.097324328 (52h0m19.034982827s) [noise=true , address=77.90.31.17:4771 , piece_id=Y3WSALJX23CWPPYQ3XYMCFOSBLBX2E2JQCCS7EOEYNBWY4CDRT3Q, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.098682446 (52h0m19.036340945s) [noise=true , address=77.90.31.15:4698 , piece_id=TN4OTPUD47FD6WOLZBEHNDI43WY6QVO5JK2JNJXNVM2IT6XMGERQ, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.099041603 (52h0m19.036700102s) [noise=true , address=45.91.116.11:4118 , piece_id=AKZIOAR6NRNO7JWCQZIKNSL7IUH5S4IG3KPG3IY2FAKG74DH3E3Q, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:30.718412255 (52h0m20.656070754s) [upload=49973]: Next returned nil segment Aug 1 21:57:30.718429869 (52h0m20.656088368s) [upload=49973]: Waiting for error group managing segments... Aug 1 21:57:31.107386675 (52h0m21.045045164s) [noise=true , address=45.91.116.13:4153 , piece_id=PMLKMZO66KXF3ZONO5ZZOQZICFK5XNGL6MN5FFBKICVFU5AWU6PQ, seg_pos={0 0}, upload=49973]: Done uploading piece. err: <nil> Aug 1 21:57:31.108529412 (52h0m21.046187911s) [noise=true , address=77.90.31.12:4590 , piece_id=D6FUBIFCQJSDRGWEP5GG34AQPNB3UROCYOATD6SRTJ7KAYVMFCOA, seg_pos={0 0}, upload=49973]: Done uploading piece. err: <nil> Aug 1 21:57:31.109975858 (52h0m21.047634347s) [noise=true , address=77.90.31.15:4698 , piece_id=TN4OTPUD47FD6WOLZBEHNDI43WY6QVO5JK2JNJXNVM2IT6XMGERQ, seg_pos={0 0}, upload=49973]: Done uploading piece. err: <nil> Aug 1 21:57:31.115273013 (52h0m21.052931502s) [noise=true , address=77.90.31.10:4518 , piece_id=WUOVXUSDG6YQTDPWG4YO7VLJNZZVLH6VRGMWOJABG2EJRU7XHP5A, seg_pos={0 0}, upload=49973]: Done uploading piece. err: <nil> Aug 1 21:57:31.116767281 (52h0m21.054425781s) [noise=true , address=77.90.31.18:4806 , piece_id=52B6CAHGLFVTFIDQHHHAWKCBP2WJJY7OTTGRZZ7CEUH3HDA43Q6A, seg_pos={0 0}, upload=49973]: Done uploading piece. err: <nil> Aug 1 21:57:31.201146107 (52h0m21.138804606s) [noise=true , address=77.90.31.16:4733 , piece_id=DMASMGOUATLPEKCGLHKNNIMMFYVVGSULNUOMJU6RJY3ULBDOY6KQ, seg_pos={0 0}, upload=49973]: Done uploading piece. err: <nil> Aug 1 21:57:31.201833732 (52h0m21.139492231s) [noise=true , address=77.90.31.19:4843 , piece_id=UFONZYHL7EAFYE5PRW32RKUVQPMZPXBEFLLIFJMAB2TXJ3C6KKNA, seg_pos={0 0}, upload=49973]: Done uploading piece. err: <nil> Aug 1 21:57:31.211007613 (52h0m21.148666112s) [noise=true , address=45.91.116.9:4262 , piece_id=A3F4L3RR6WPFMBFFKLATNGG3CASBIPWQP5O3G5URFNTXBE3ZW3VA, seg_pos={0 0}, upload=49973]: Done uploading piece. err: <nil> Aug 1 21:57:31.215929169 (52h0m21.153587668s) [noise=true , address=77.90.31.17:4771 , piece_id=Y3WSALJX23CWPPYQ3XYMCFOSBLBX2E2JQCCS7EOEYNBWY4CDRT3Q, seg_pos={0 0}, upload=49973]: Done uploading piece. err: <nil> Aug 1 21:57:31.222973975 (52h0m21.160632474s) [noise=true , address=77.90.31.21:4878 , piece_id=BLNTYBOAIZWISSUJYK6BSBPTSNL6QWLGEN6AP6P4NPDVSURYG5AA, seg_pos={0 0}, upload=49973]: Done uploading piece. err: <nil> Aug 1 21:57:31.234392441 (52h0m21.17205094s) [noise=true , address=45.91.116.12:4225 , piece_id=PBQPLA5ZYRFPSKHDKCEFOEBS5T3GYMP2FLSFDQWJLVY756KPLJNQ, seg_pos={0 0}, upload=49973]: Done uploading piece. err: <nil> Aug 1 21:57:31.284794305 (52h0m21.222452804s) [noise=true , address=45.91.116.11:4118 , piece_id=AKZIOAR6NRNO7JWCQZIKNSL7IUH5S4IG3KPG3IY2FAKG74DH3E3Q, seg_pos={0 0}, upload=49973]: Done uploading piece. err: <nil> Aug 1 21:57:58.202420932 (52h0m48.140079471s) [noise=true , address=104.36.82.6:4875 , piece_id=CS5AMB7CIBQWWOU57Z3RCCSNVKIPE4B5QWU5YIQYI2UCWH7PAPVA, seg_pos={0 0}, upload=49973]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:104.36.82.6:4875): protocol: write tcp 188.166.225.170:40166->104.36.82.6:4875: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:58.202441131 (52h0m48.14009963s) [noise=true , address=104.36.82.5:4840 , piece_id=UYVKVFCJJP7XZYRIVPS4ABGEMDUMNR7YW4QGG5ZFTFCGNCJO3CSQ, seg_pos={0 0}, upload=49973]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:104.36.82.5:4840): protocol: rpc: dial tcp 104.36.82.5:4840: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:58.202448445 (52h0m48.140106954s) [noise=true , address=104.36.82.7:4909 , piece_id=QHJCVBT6KTU34K2KWFNACWYVVE3WJ2RU76BHVSKL4HLCZEFXZYPA, seg_pos={0 0}, upload=49973]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:104.36.82.7:4909): protocol: rpc: dial tcp 104.36.82.7:4909: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:58.202496487 (52h0m48.140155026s) [noise=true , address=104.36.82.4:4802 , piece_id=CLM2G33M6AK5HIESRW6LZY33NKHIR2PGITWJ23NPILRCA6UYIEHA, seg_pos={0 0}, upload=49973]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: write tcp 188.166.225.170:47696->104.36.82.4:4802: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:58.202524240 (52h0m48.140182739s) [seg_pos={0 0}, upload=49973]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Aug 1 21:57:58.435983932 (52h0m48.373642431s) [upload=49973]: Done uploading segments... Aug 1 21:57:58.436007667 (52h0m48.373666166s) [upload=49973]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.677761357s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 3 out of 60 is failed (threshold=3) [....................................X............X.....X....] duration: 9.407407154s FAIL (2025-08-01 21:59:02 UTC) orbiter warning: Operation: ListObjects unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/1K Duration: 9.407436209s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [.....................................................XXX....] duration: 10.369511173s FAIL (2025-08-01 21:59:01 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 10.369562311s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 7 out of 60 is failed (threshold=3) [....................................................XXXXXXX.] duration: 12.841235596s FAIL (2025-08-01 21:58:51 UTC) orbiter warning: Operation: DownloadObject uplink: object not found ("orbiter_us_0/4M") Duration: 12.841270293s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 6.369025867s FAIL (2025-08-01 21:58:51 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.369050674s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [......................................................XXXX..] duration: 5.911720282s FAIL (2025-08-01 21:58:47 UTC) orbiter warning: Operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 7) ======LOGS====== K4BW277ZSGXJYQDA66OMEBVDC7DAWSOJAPX42HANNGRZN35KHQ, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.481101510 (51h59m34.418760009s) [noise=true , address=104.36.82.5:4842 , piece_id=LV7YAOMPE2AV7TF2WPWR7QJNLT4JZAIPSVZN3E23U5KDBNKZFD6A, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.481422003 (51h59m34.419080502s) [noise=true , address=79.127.205.235:4683 , piece_id=3QHDNAPG4GRLJFVVNJNFXNGKTGKUEMWWUBMW62HHE5N5LJDO4KWA, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.482030897 (51h59m34.419689396s) [noise=true , address=79.127.219.51:4213 , piece_id=L2WSPI5BEIFLGQKJIK4EWXFQXNQRQ7HVKPOWFYRPPUGNAL5DVYWA, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.482285215 (51h59m34.419943704s) [noise=true , address=104.36.82.14:4251 , piece_id=O7NADU347E2Q26TPD3IUCK6WE5UVRITC6ALZHBE5SWYTKWXFBMXA, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.482518370 (51h59m34.420176859s) [noise=true , address=79.127.220.98:4402 , piece_id=DVLSYRFFZ6UT3QRSVAN7AMXVYZNI6PWG2IITHLRUUJATJVK3QOGA, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.482724304 (51h59m34.420382803s) [noise=true , address=79.127.201.214:4297 , piece_id=MPTMGVEYC4SATIN4ANCR2ELVRXHYWRDLMHWQW7HM6UOIV2N6R7ZA, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.482928555 (51h59m34.420587054s) [noise=true , address=104.36.82.37:4680 , piece_id=X4PDUE4WXUUWX6JQVO4NWPQCBMHMHC5YHPIHDQVYYOI2FXV2GVJQ, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.483134960 (51h59m34.420793449s) [noise=true , address=104.36.82.50:4027 , piece_id=OW7BODU5N7ZEEGE2FHSLNYVJT2Z25LY3J42TZEOR5SP2EWIN6X7A, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.483338480 (51h59m34.420996979s) [noise=true , address=104.36.82.55:4204 , piece_id=R5CYXWAVV4ZEMCV7LF4D4XDC4G6KLVBONAH2TH5EIEVL7QHUTRWQ, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.483582847 (51h59m34.421241356s) [noise=true , address=104.36.82.11:4134 , piece_id=RMTNNNC2VD7MGWJDAMT6ZCF3GR43SHANFVAGJ6EHVH7WO5XA6VTA, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.483833737 (51h59m34.421492236s) [noise=true , address=104.36.82.43:4856 , piece_id=2SINBLHGRUOWUO4XHBNBG45YE3PW7RNF3QSNVCQXNCFEFHRBFNJQ, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.484021896 (51h59m34.421680385s) [noise=true , address=104.36.82.52:4101 , piece_id=XYPLQM7QITCCX7T4NKIXP5MZ7I6F24EJJTG37K4NPOBKTUFHULRQ, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.484301662 (51h59m34.421960182s) [noise=true , address=79.127.201.215:4352 , piece_id=I6BRYI5XIACAAAAAE7HQOBCF4OTALMOVPPNBWQCRSH2MM6WUUURQ, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.484712097 (51h59m34.422370596s) [noise=true , address=64.49.13.138:4484 , piece_id=DZ5Y4HTS6BKW2DJ5CVOB64TW4SXB6DNCJEGFUJPLAUXUEHRZRTUA, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:45.576021470 (51h59m35.513679959s) [upload=49960]: Next returned nil segment Aug 1 21:56:45.576036609 (51h59m35.513695098s) [upload=49960]: Waiting for error group managing segments... Aug 1 21:56:45.860378776 (51h59m35.798037275s) [noise=true , address=64.49.13.138:4484 , piece_id=DZ5Y4HTS6BKW2DJ5CVOB64TW4SXB6DNCJEGFUJPLAUXUEHRZRTUA, seg_pos={0 0}, upload=49960]: Done uploading piece. err: <nil> Aug 1 21:56:45.884184188 (51h59m35.821842687s) [noise=true , address=79.127.205.235:4683 , piece_id=3QHDNAPG4GRLJFVVNJNFXNGKTGKUEMWWUBMW62HHE5N5LJDO4KWA, seg_pos={0 0}, upload=49960]: Done uploading piece. err: <nil> Aug 1 21:56:46.145493612 (51h59m36.083152101s) [noise=true , address=79.127.220.98:4402 , piece_id=DVLSYRFFZ6UT3QRSVAN7AMXVYZNI6PWG2IITHLRUUJATJVK3QOGA, seg_pos={0 0}, upload=49960]: Done uploading piece. err: <nil> Aug 1 21:56:46.257505313 (51h59m36.195163802s) [noise=true , address=79.127.220.97:4371 , piece_id=5WGPRFKQETIHTSMA42AM2C3UGCFYJ3DMMPAI2L7DD6E2P37KJKNA, seg_pos={0 0}, upload=49960]: Done uploading piece. err: <nil> Aug 1 21:56:46.334426181 (51h59m36.27208468s) [noise=true , address=79.127.201.215:4352 , piece_id=I6BRYI5XIACAAAAAE7HQOBCF4OTALMOVPPNBWQCRSH2MM6WUUURQ, seg_pos={0 0}, upload=49960]: Done uploading piece. err: <nil> Aug 1 21:56:46.386119275 (51h59m36.323777774s) [noise=true , address=79.127.201.214:4297 , piece_id=MPTMGVEYC4SATIN4ANCR2ELVRXHYWRDLMHWQW7HM6UOIV2N6R7ZA, seg_pos={0 0}, upload=49960]: Done uploading piece. err: <nil> Aug 1 21:56:46.395362048 (51h59m36.333020547s) [noise=true , address=79.127.219.51:4213 , piece_id=L2WSPI5BEIFLGQKJIK4EWXFQXNQRQ7HVKPOWFYRPPUGNAL5DVYWA, seg_pos={0 0}, upload=49960]: Done uploading piece. err: <nil> Aug 1 21:57:43.392434383 (52h0m33.330092872s) [noise=true , address=104.36.82.37:4680 , piece_id=X4PDUE4WXUUWX6JQVO4NWPQCBMHMHC5YHPIHDQVYYOI2FXV2GVJQ, seg_pos={0 0}, upload=49960]: Done uploading piece. err: ecclient: upload failed (node:1bXAqpxGhUvzwYEDGBqYRhZdTeMJcV3TYfngkKTCumpt9Nupi3, address:104.36.82.37:4680): protocol: rpc: dial tcp 104.36.82.37:4680: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:43.392477967 (52h0m33.330136466s) [noise=true , address=104.36.82.11:4134 , piece_id=RMTNNNC2VD7MGWJDAMT6ZCF3GR43SHANFVAGJ6EHVH7WO5XA6VTA, seg_pos={0 0}, upload=49960]: Done uploading piece. err: ecclient: upload failed (node:12R3Z5AgkBmWXMv5XgRyfcyqkG2gmyJs9ogT24kcLy6nuB9bzUG, address:104.36.82.11:4134): protocol: rpc: dial tcp 104.36.82.11:4134: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:43.392514086 (52h0m33.330172585s) [noise=true , address=104.36.82.55:4204 , piece_id=R5CYXWAVV4ZEMCV7LF4D4XDC4G6KLVBONAH2TH5EIEVL7QHUTRWQ, seg_pos={0 0}, upload=49960]: Done uploading piece. err: ecclient: upload failed (node:1rBZH6aLHsnWh22uNzaMiMwmLJCDMn4bYbNC5jRG7KCEuRfqpH, address:104.36.82.55:4204): protocol: write tcp 188.166.225.170:48550->104.36.82.55:4204: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:43.392546537 (52h0m33.330205027s) [noise=true , address=104.36.82.59:4327 , piece_id=OZDEYIY6B4WYZULRDRL2UBOJH2JQNY3YWSITZKWDO7GTT72AN5NQ, seg_pos={0 0}, upload=49960]: Done uploading piece. err: ecclient: upload failed (node:1V4C6AWGbyYQWfinoSj7iKovHeTJ9hqypyC5U9rZ8pveXBBfCY, address:104.36.82.59:4327): protocol: rpc: dial tcp 104.36.82.59:4327: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:43.392579040 (52h0m33.33023754s) [noise=true , address=104.36.82.5:4842 , piece_id=LV7YAOMPE2AV7TF2WPWR7QJNLT4JZAIPSVZN3E23U5KDBNKZFD6A, seg_pos={0 0}, upload=49960]: Done uploading piece. err: ecclient: upload failed (node:12q4VdT5BcjUb16PBYHK6zrnW8dqENKrcGx9zdLtNefDVB1MUSn, address:104.36.82.5:4842): protocol: context deadline exceeded; context deadline exceeded Aug 1 21:57:43.392627953 (52h0m33.330286442s) [noise=true , address=104.36.82.52:4101 , piece_id=XYPLQM7QITCCX7T4NKIXP5MZ7I6F24EJJTG37K4NPOBKTUFHULRQ, seg_pos={0 0}, upload=49960]: Done uploading piece. err: ecclient: upload failed (node:12WWXeBjFZR9PzciNfs35fFcx87N9GwDpMsPHnksFDugtaBGWp4, address:104.36.82.52:4101): protocol: rpc: dial tcp 104.36.82.52:4101: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:43.392653192 (52h0m33.330311691s) [noise=true , address=104.36.82.14:4251 , piece_id=O7NADU347E2Q26TPD3IUCK6WE5UVRITC6ALZHBE5SWYTKWXFBMXA, seg_pos={0 0}, upload=49960]: Done uploading piece. err: ecclient: upload failed (node:12btf6Z38aTMx7MaBxhUyZtFqUPgeMtP2aU8Y6FbV8zwu57zArd, address:104.36.82.14:4251): protocol: rpc: dial tcp 104.36.82.14:4251: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:43.392670836 (52h0m33.330329335s) [noise=true , address=104.36.82.12:4166 , piece_id=6ZIWCOBY76GJ6DHZF5G23U4IYTOFB5SMQW5TW3YITYH227PV2FGA, seg_pos={0 0}, upload=49960]: Done uploading piece. err: ecclient: upload failed (node:12iCB7c4ZfpqafpD1PtpbE6BfU81wJW65LmrPVJXtD58yeuoR9t, address:104.36.82.12:4166): protocol: rpc: dial tcp 104.36.82.12:4166: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:43.392698018 (52h0m33.330356517s) [noise=true , address=104.36.82.50:4027 , piece_id=OW7BODU5N7ZEEGE2FHSLNYVJT2Z25LY3J42TZEOR5SP2EWIN6X7A, seg_pos={0 0}, upload=49960]: Done uploading piece. err: ecclient: upload failed (node:1zEQhyuVxSHwdDUgDfcTog2W3oLBx4z9bYUMHpaaoLU1BTU9DQ, address:104.36.82.50:4027): protocol: write tcp 188.166.225.170:37784->104.36.82.50:4027: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:43.392715381 (52h0m33.33037387s) [noise=true , address=104.36.82.43:4856 , piece_id=2SINBLHGRUOWUO4XHBNBG45YE3PW7RNF3QSNVCQXNCFEFHRBFNJQ, seg_pos={0 0}, upload=49960]: Done uploading piece. err: ecclient: upload failed (node:12bQxwBfPEGtmMp6mtnWKsoZt3eMEjLdyYFahETWGHeKJ7BhkiZ, address:104.36.82.43:4856): protocol: rpc: dial tcp 104.36.82.43:4856: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:43.392730059 (52h0m33.330388558s) [noise=true , address=104.36.82.10:4082 , piece_id=45K4BW277ZSGXJYQDA66OMEBVDC7DAWSOJAPX42HANNGRZN35KHQ, seg_pos={0 0}, upload=49960]: Done uploading piece. err: ecclient: upload failed (node:12DKsNCtqxLvpJwLszdLs74V3eb8NH2VAV9V3bngzUczawmNoLK, address:104.36.82.10:4082): protocol: write tcp 188.166.225.170:46638->104.36.82.10:4082: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:43.392753424 (52h0m33.330411923s) [seg_pos={0 0}, upload=49960]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil> Aug 1 21:57:43.624550997 (52h0m33.562209566s) [upload=49960]: Done uploading segments... Aug 1 21:57:43.624582868 (52h0m33.562241407s) [upload=49960]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7) Duration: 5.911747383s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 6.105959199s FAIL (2025-08-01 21:58:28 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.105988557s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [.......................................................XXXX.] duration: 42.085276604s FAIL (2025-08-01 21:58:25 UTC) orbiter warning: Operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 7) ======LOGS====== K4BW277ZSGXJYQDA66OMEBVDC7DAWSOJAPX42HANNGRZN35KHQ, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.481101510 (51h59m34.418760009s) [noise=true , address=104.36.82.5:4842 , piece_id=LV7YAOMPE2AV7TF2WPWR7QJNLT4JZAIPSVZN3E23U5KDBNKZFD6A, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.481422003 (51h59m34.419080502s) [noise=true , address=79.127.205.235:4683 , piece_id=3QHDNAPG4GRLJFVVNJNFXNGKTGKUEMWWUBMW62HHE5N5LJDO4KWA, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.482030897 (51h59m34.419689396s) [noise=true , address=79.127.219.51:4213 , piece_id=L2WSPI5BEIFLGQKJIK4EWXFQXNQRQ7HVKPOWFYRPPUGNAL5DVYWA, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.482285215 (51h59m34.419943704s) [noise=true , address=104.36.82.14:4251 , piece_id=O7NADU347E2Q26TPD3IUCK6WE5UVRITC6ALZHBE5SWYTKWXFBMXA, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.482518370 (51h59m34.420176859s) [noise=true , address=79.127.220.98:4402 , piece_id=DVLSYRFFZ6UT3QRSVAN7AMXVYZNI6PWG2IITHLRUUJATJVK3QOGA, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.482724304 (51h59m34.420382803s) [noise=true , address=79.127.201.214:4297 , piece_id=MPTMGVEYC4SATIN4ANCR2ELVRXHYWRDLMHWQW7HM6UOIV2N6R7ZA, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.482928555 (51h59m34.420587054s) [noise=true , address=104.36.82.37:4680 , piece_id=X4PDUE4WXUUWX6JQVO4NWPQCBMHMHC5YHPIHDQVYYOI2FXV2GVJQ, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.483134960 (51h59m34.420793449s) [noise=true , address=104.36.82.50:4027 , piece_id=OW7BODU5N7ZEEGE2FHSLNYVJT2Z25LY3J42TZEOR5SP2EWIN6X7A, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.483338480 (51h59m34.420996979s) [noise=true , address=104.36.82.55:4204 , piece_id=R5CYXWAVV4ZEMCV7LF4D4XDC4G6KLVBONAH2TH5EIEVL7QHUTRWQ, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.483582847 (51h59m34.421241356s) [noise=true , address=104.36.82.11:4134 , piece_id=RMTNNNC2VD7MGWJDAMT6ZCF3GR43SHANFVAGJ6EHVH7WO5XA6VTA, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.483833737 (51h59m34.421492236s) [noise=true , address=104.36.82.43:4856 , piece_id=2SINBLHGRUOWUO4XHBNBG45YE3PW7RNF3QSNVCQXNCFEFHRBFNJQ, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.484021896 (51h59m34.421680385s) [noise=true , address=104.36.82.52:4101 , piece_id=XYPLQM7QITCCX7T4NKIXP5MZ7I6F24EJJTG37K4NPOBKTUFHULRQ, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.484301662 (51h59m34.421960182s) [noise=true , address=79.127.201.215:4352 , piece_id=I6BRYI5XIACAAAAAE7HQOBCF4OTALMOVPPNBWQCRSH2MM6WUUURQ, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.484712097 (51h59m34.422370596s) [noise=true , address=64.49.13.138:4484 , piece_id=DZ5Y4HTS6BKW2DJ5CVOB64TW4SXB6DNCJEGFUJPLAUXUEHRZRTUA, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:45.576021470 (51h59m35.513679959s) [upload=49960]: Next returned nil segment Aug 1 21:56:45.576036609 (51h59m35.513695098s) [upload=49960]: Waiting for error group managing segments... Aug 1 21:56:45.860378776 (51h59m35.798037275s) [noise=true , address=64.49.13.138:4484 , piece_id=DZ5Y4HTS6BKW2DJ5CVOB64TW4SXB6DNCJEGFUJPLAUXUEHRZRTUA, seg_pos={0 0}, upload=49960]: Done uploading piece. err: <nil> Aug 1 21:56:45.884184188 (51h59m35.821842687s) [noise=true , address=79.127.205.235:4683 , piece_id=3QHDNAPG4GRLJFVVNJNFXNGKTGKUEMWWUBMW62HHE5N5LJDO4KWA, seg_pos={0 0}, upload=49960]: Done uploading piece. err: <nil> Aug 1 21:56:46.145493612 (51h59m36.083152101s) [noise=true , address=79.127.220.98:4402 , piece_id=DVLSYRFFZ6UT3QRSVAN7AMXVYZNI6PWG2IITHLRUUJATJVK3QOGA, seg_pos={0 0}, upload=49960]: Done uploading piece. err: <nil> Aug 1 21:56:46.257505313 (51h59m36.195163802s) [noise=true , address=79.127.220.97:4371 , piece_id=5WGPRFKQETIHTSMA42AM2C3UGCFYJ3DMMPAI2L7DD6E2P37KJKNA, seg_pos={0 0}, upload=49960]: Done uploading piece. err: <nil> Aug 1 21:56:46.334426181 (51h59m36.27208468s) [noise=true , address=79.127.201.215:4352 , piece_id=I6BRYI5XIACAAAAAE7HQOBCF4OTALMOVPPNBWQCRSH2MM6WUUURQ, seg_pos={0 0}, upload=49960]: Done uploading piece. err: <nil> Aug 1 21:56:46.386119275 (51h59m36.323777774s) [noise=true , address=79.127.201.214:4297 , piece_id=MPTMGVEYC4SATIN4ANCR2ELVRXHYWRDLMHWQW7HM6UOIV2N6R7ZA, seg_pos={0 0}, upload=49960]: Done uploading piece. err: <nil> Aug 1 21:56:46.395362048 (51h59m36.333020547s) [noise=true , address=79.127.219.51:4213 , piece_id=L2WSPI5BEIFLGQKJIK4EWXFQXNQRQ7HVKPOWFYRPPUGNAL5DVYWA, seg_pos={0 0}, upload=49960]: Done uploading piece. err: <nil> Aug 1 21:57:43.392434383 (52h0m33.330092872s) [noise=true , address=104.36.82.37:4680 , piece_id=X4PDUE4WXUUWX6JQVO4NWPQCBMHMHC5YHPIHDQVYYOI2FXV2GVJQ, seg_pos={0 0}, upload=49960]: Done uploading piece. err: ecclient: upload failed (node:1bXAqpxGhUvzwYEDGBqYRhZdTeMJcV3TYfngkKTCumpt9Nupi3, address:104.36.82.37:4680): protocol: rpc: dial tcp 104.36.82.37:4680: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:43.392477967 (52h0m33.330136466s) [noise=true , address=104.36.82.11:4134 , piece_id=RMTNNNC2VD7MGWJDAMT6ZCF3GR43SHANFVAGJ6EHVH7WO5XA6VTA, seg_pos={0 0}, upload=49960]: Done uploading piece. err: ecclient: upload failed (node:12R3Z5AgkBmWXMv5XgRyfcyqkG2gmyJs9ogT24kcLy6nuB9bzUG, address:104.36.82.11:4134): protocol: rpc: dial tcp 104.36.82.11:4134: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:43.392514086 (52h0m33.330172585s) [noise=true , address=104.36.82.55:4204 , piece_id=R5CYXWAVV4ZEMCV7LF4D4XDC4G6KLVBONAH2TH5EIEVL7QHUTRWQ, seg_pos={0 0}, upload=49960]: Done uploading piece. err: ecclient: upload failed (node:1rBZH6aLHsnWh22uNzaMiMwmLJCDMn4bYbNC5jRG7KCEuRfqpH, address:104.36.82.55:4204): protocol: write tcp 188.166.225.170:48550->104.36.82.55:4204: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:43.392546537 (52h0m33.330205027s) [noise=true , address=104.36.82.59:4327 , piece_id=OZDEYIY6B4WYZULRDRL2UBOJH2JQNY3YWSITZKWDO7GTT72AN5NQ, seg_pos={0 0}, upload=49960]: Done uploading piece. err: ecclient: upload failed (node:1V4C6AWGbyYQWfinoSj7iKovHeTJ9hqypyC5U9rZ8pveXBBfCY, address:104.36.82.59:4327): protocol: rpc: dial tcp 104.36.82.59:4327: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:43.392579040 (52h0m33.33023754s) [noise=true , address=104.36.82.5:4842 , piece_id=LV7YAOMPE2AV7TF2WPWR7QJNLT4JZAIPSVZN3E23U5KDBNKZFD6A, seg_pos={0 0}, upload=49960]: Done uploading piece. err: ecclient: upload failed (node:12q4VdT5BcjUb16PBYHK6zrnW8dqENKrcGx9zdLtNefDVB1MUSn, address:104.36.82.5:4842): protocol: context deadline exceeded; context deadline exceeded Aug 1 21:57:43.392627953 (52h0m33.330286442s) [noise=true , address=104.36.82.52:4101 , piece_id=XYPLQM7QITCCX7T4NKIXP5MZ7I6F24EJJTG37K4NPOBKTUFHULRQ, seg_pos={0 0}, upload=49960]: Done uploading piece. err: ecclient: upload failed (node:12WWXeBjFZR9PzciNfs35fFcx87N9GwDpMsPHnksFDugtaBGWp4, address:104.36.82.52:4101): protocol: rpc: dial tcp 104.36.82.52:4101: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:43.392653192 (52h0m33.330311691s) [noise=true , address=104.36.82.14:4251 , piece_id=O7NADU347E2Q26TPD3IUCK6WE5UVRITC6ALZHBE5SWYTKWXFBMXA, seg_pos={0 0}, upload=49960]: Done uploading piece. err: ecclient: upload failed (node:12btf6Z38aTMx7MaBxhUyZtFqUPgeMtP2aU8Y6FbV8zwu57zArd, address:104.36.82.14:4251): protocol: rpc: dial tcp 104.36.82.14:4251: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:43.392670836 (52h0m33.330329335s) [noise=true , address=104.36.82.12:4166 , piece_id=6ZIWCOBY76GJ6DHZF5G23U4IYTOFB5SMQW5TW3YITYH227PV2FGA, seg_pos={0 0}, upload=49960]: Done uploading piece. err: ecclient: upload failed (node:12iCB7c4ZfpqafpD1PtpbE6BfU81wJW65LmrPVJXtD58yeuoR9t, address:104.36.82.12:4166): protocol: rpc: dial tcp 104.36.82.12:4166: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:43.392698018 (52h0m33.330356517s) [noise=true , address=104.36.82.50:4027 , piece_id=OW7BODU5N7ZEEGE2FHSLNYVJT2Z25LY3J42TZEOR5SP2EWIN6X7A, seg_pos={0 0}, upload=49960]: Done uploading piece. err: ecclient: upload failed (node:1zEQhyuVxSHwdDUgDfcTog2W3oLBx4z9bYUMHpaaoLU1BTU9DQ, address:104.36.82.50:4027): protocol: write tcp 188.166.225.170:37784->104.36.82.50:4027: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:43.392715381 (52h0m33.33037387s) [noise=true , address=104.36.82.43:4856 , piece_id=2SINBLHGRUOWUO4XHBNBG45YE3PW7RNF3QSNVCQXNCFEFHRBFNJQ, seg_pos={0 0}, upload=49960]: Done uploading piece. err: ecclient: upload failed (node:12bQxwBfPEGtmMp6mtnWKsoZt3eMEjLdyYFahETWGHeKJ7BhkiZ, address:104.36.82.43:4856): protocol: rpc: dial tcp 104.36.82.43:4856: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:43.392730059 (52h0m33.330388558s) [noise=true , address=104.36.82.10:4082 , piece_id=45K4BW277ZSGXJYQDA66OMEBVDC7DAWSOJAPX42HANNGRZN35KHQ, seg_pos={0 0}, upload=49960]: Done uploading piece. err: ecclient: upload failed (node:12DKsNCtqxLvpJwLszdLs74V3eb8NH2VAV9V3bngzUczawmNoLK, address:104.36.82.10:4082): protocol: write tcp 188.166.225.170:46638->104.36.82.10:4082: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:43.392753424 (52h0m33.330411923s) [seg_pos={0 0}, upload=49960]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil> Aug 1 21:57:43.624550997 (52h0m33.562209566s) [upload=49960]: Done uploading segments... Aug 1 21:57:43.624582868 (52h0m33.562241407s) [upload=49960]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7) Duration: 42.085305118s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 7 out of 60 is failed (threshold=3) [.....................................................XXXXXXX] duration: 8.510916807s FAIL (2025-08-01 21:58:19 UTC) orbiter warning: Operation: DownloadObject uplink: object not found ("orbiter_us_0/4M") Duration: 8.510931405s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [......................................................XXX...] duration: 5.774395635s FAIL (2025-08-01 21:58:14 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.774417937s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 3 out of 60 is failed (threshold=3) [.....................................X............X.....X...] duration: 6.554759655s FAIL (2025-08-01 21:58:13 UTC) orbiter warning: Operation: ListObjects unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/1K Duration: 6.554781808s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 6 out of 60 is failed (threshold=3) [......................................................XXXXXX] duration: 32.771661648s FAIL (2025-08-01 21:58:10 UTC) orbiter warning: Operation: Commit uplink: failed to upload enough pieces (needed at least 14 but got 4) Duration: 32.771680583s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.....................................................XX.X.X.] duration: 6.623235846s FAIL (2025-08-01 21:58:05 UTC) orbiter warning: Operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 7NK72474VWE4E3YOBCDZ5O5ZOH2MSA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB): protocol: write tcp 188.166.225.170:38368->104.36.82.12:4151: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:56:50.418604753 (51h59m40.356263252s) [seg_pos={0 0}, upload=49961]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 1 21:56:50.418680879 (51h59m40.356339378s) [upload=49961]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 1 21:56:50.418685998 (51h59m40.356344487s) [upload=49961]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 1 21:56:50.726563872 (51h59m40.664222371s) [upload=49961]: Done uploading segments... Aug 1 21:56:50.726586586 (51h59m40.664245085s) [upload=49961]: Upload finished. err: <nil> Aug 1 21:57:28.852869669 (52h0m18.790528158s) [upload=49973]: Starting upload Aug 1 21:57:28.852942389 (52h0m18.790600888s) [upload=49973]: Done starting upload Aug 1 21:57:28.853007122 (52h0m18.790665611s) [upload=49973]: Uploading segments... Aug 1 21:57:28.853174002 (52h0m18.790832491s) [upload=49973]: Got next segment. Inline: false Aug 1 21:57:28.853208168 (52h0m18.790866657s) [upload=49973]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 1 21:57:28.853212666 (52h0m18.790871165s) [upload=49973]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 1 21:57:29.095087519 (52h0m19.032746008s) [seg_pos={0 0}, upload=49973]: Begin upload segment... Aug 1 21:57:29.095129960 (52h0m19.032788459s) [seg_pos={0 0}, upload=49973]: Done begin upload segment. Aug 1 21:57:29.095164997 (52h0m19.032823496s) [noise=true , address=77.90.31.21:4878 , piece_id=BLNTYBOAIZWISSUJYK6BSBPTSNL6QWLGEN6AP6P4NPDVSURYG5AA, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.095171800 (52h0m19.032830289s) [noise=true , address=104.36.82.6:4875 , piece_id=CS5AMB7CIBQWWOU57Z3RCCSNVKIPE4B5QWU5YIQYI2UCWH7PAPVA, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.095510027 (52h0m19.033168526s) [noise=true , address=77.90.31.18:4806 , piece_id=52B6CAHGLFVTFIDQHHHAWKCBP2WJJY7OTTGRZZ7CEUH3HDA43Q6A, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.095544193 (52h0m19.033202692s) [noise=true , address=77.90.31.10:4518 , piece_id=WUOVXUSDG6YQTDPWG4YO7VLJNZZVLH6VRGMWOJABG2EJRU7XHP5A, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.095774283 (52h0m19.033432782s) [noise=true , address=104.36.82.7:4909 , piece_id=QHJCVBT6KTU34K2KWFNACWYVVE3WJ2RU76BHVSKL4HLCZEFXZYPA, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.095819880 (52h0m19.033478379s) [noise=true , address=45.91.116.12:4225 , piece_id=PBQPLA5ZYRFPSKHDKCEFOEBS5T3GYMP2FLSFDQWJLVY756KPLJNQ, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.096021656 (52h0m19.033680155s) [noise=true , address=45.91.116.13:4153 , piece_id=PMLKMZO66KXF3ZONO5ZZOQZICFK5XNGL6MN5FFBKICVFU5AWU6PQ, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.096069618 (52h0m19.033728117s) [noise=true , address=45.91.116.9:4262 , piece_id=A3F4L3RR6WPFMBFFKLATNGG3CASBIPWQP5O3G5URFNTXBE3ZW3VA, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.096348041 (52h0m19.03400654s) [noise=true , address=104.36.82.5:4840 , piece_id=UYVKVFCJJP7XZYRIVPS4ABGEMDUMNR7YW4QGG5ZFTFCGNCJO3CSQ, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.096354803 (52h0m19.034013292s) [noise=true , address=77.90.31.12:4590 , piece_id=D6FUBIFCQJSDRGWEP5GG34AQPNB3UROCYOATD6SRTJ7KAYVMFCOA, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.096608579 (52h0m19.034267078s) [noise=true , address=77.90.31.16:4733 , piece_id=DMASMGOUATLPEKCGLHKNNIMMFYVVGSULNUOMJU6RJY3ULBDOY6KQ, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.096830774 (52h0m19.034489273s) [noise=true , address=104.36.82.4:4802 , piece_id=CLM2G33M6AK5HIESRW6LZY33NKHIR2PGITWJ23NPILRCA6UYIEHA, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.097107233 (52h0m19.034765732s) [noise=true , address=77.90.31.19:4843 , piece_id=UFONZYHL7EAFYE5PRW32RKUVQPMZPXBEFLLIFJMAB2TXJ3C6KKNA, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.097324328 (52h0m19.034982827s) [noise=true , address=77.90.31.17:4771 , piece_id=Y3WSALJX23CWPPYQ3XYMCFOSBLBX2E2JQCCS7EOEYNBWY4CDRT3Q, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.098682446 (52h0m19.036340945s) [noise=true , address=77.90.31.15:4698 , piece_id=TN4OTPUD47FD6WOLZBEHNDI43WY6QVO5JK2JNJXNVM2IT6XMGERQ, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.099041603 (52h0m19.036700102s) [noise=true , address=45.91.116.11:4118 , piece_id=AKZIOAR6NRNO7JWCQZIKNSL7IUH5S4IG3KPG3IY2FAKG74DH3E3Q, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:30.718412255 (52h0m20.656070754s) [upload=49973]: Next returned nil segment Aug 1 21:57:30.718429869 (52h0m20.656088368s) [upload=49973]: Waiting for error group managing segments... Aug 1 21:57:31.107386675 (52h0m21.045045164s) [noise=true , address=45.91.116.13:4153 , piece_id=PMLKMZO66KXF3ZONO5ZZOQZICFK5XNGL6MN5FFBKICVFU5AWU6PQ, seg_pos={0 0}, upload=49973]: Done uploading piece. err: <nil> Aug 1 21:57:31.108529412 (52h0m21.046187911s) [noise=true , address=77.90.31.12:4590 , piece_id=D6FUBIFCQJSDRGWEP5GG34AQPNB3UROCYOATD6SRTJ7KAYVMFCOA, seg_pos={0 0}, upload=49973]: Done uploading piece. err: <nil> Aug 1 21:57:31.109975858 (52h0m21.047634347s) [noise=true , address=77.90.31.15:4698 , piece_id=TN4OTPUD47FD6WOLZBEHNDI43WY6QVO5JK2JNJXNVM2IT6XMGERQ, seg_pos={0 0}, upload=49973]: Done uploading piece. err: <nil> Aug 1 21:57:31.115273013 (52h0m21.052931502s) [noise=true , address=77.90.31.10:4518 , piece_id=WUOVXUSDG6YQTDPWG4YO7VLJNZZVLH6VRGMWOJABG2EJRU7XHP5A, seg_pos={0 0}, upload=49973]: Done uploading piece. err: <nil> Aug 1 21:57:31.116767281 (52h0m21.054425781s) [noise=true , address=77.90.31.18:4806 , piece_id=52B6CAHGLFVTFIDQHHHAWKCBP2WJJY7OTTGRZZ7CEUH3HDA43Q6A, seg_pos={0 0}, upload=49973]: Done uploading piece. err: <nil> Aug 1 21:57:31.201146107 (52h0m21.138804606s) [noise=true , address=77.90.31.16:4733 , piece_id=DMASMGOUATLPEKCGLHKNNIMMFYVVGSULNUOMJU6RJY3ULBDOY6KQ, seg_pos={0 0}, upload=49973]: Done uploading piece. err: <nil> Aug 1 21:57:31.201833732 (52h0m21.139492231s) [noise=true , address=77.90.31.19:4843 , piece_id=UFONZYHL7EAFYE5PRW32RKUVQPMZPXBEFLLIFJMAB2TXJ3C6KKNA, seg_pos={0 0}, upload=49973]: Done uploading piece. err: <nil> Aug 1 21:57:31.211007613 (52h0m21.148666112s) [noise=true , address=45.91.116.9:4262 , piece_id=A3F4L3RR6WPFMBFFKLATNGG3CASBIPWQP5O3G5URFNTXBE3ZW3VA, seg_pos={0 0}, upload=49973]: Done uploading piece. err: <nil> Aug 1 21:57:31.215929169 (52h0m21.153587668s) [noise=true , address=77.90.31.17:4771 , piece_id=Y3WSALJX23CWPPYQ3XYMCFOSBLBX2E2JQCCS7EOEYNBWY4CDRT3Q, seg_pos={0 0}, upload=49973]: Done uploading piece. err: <nil> Aug 1 21:57:31.222973975 (52h0m21.160632474s) [noise=true , address=77.90.31.21:4878 , piece_id=BLNTYBOAIZWISSUJYK6BSBPTSNL6QWLGEN6AP6P4NPDVSURYG5AA, seg_pos={0 0}, upload=49973]: Done uploading piece. err: <nil> Aug 1 21:57:31.234392441 (52h0m21.17205094s) [noise=true , address=45.91.116.12:4225 , piece_id=PBQPLA5ZYRFPSKHDKCEFOEBS5T3GYMP2FLSFDQWJLVY756KPLJNQ, seg_pos={0 0}, upload=49973]: Done uploading piece. err: <nil> Aug 1 21:57:31.284794305 (52h0m21.222452804s) [noise=true , address=45.91.116.11:4118 , piece_id=AKZIOAR6NRNO7JWCQZIKNSL7IUH5S4IG3KPG3IY2FAKG74DH3E3Q, seg_pos={0 0}, upload=49973]: Done uploading piece. err: <nil> Aug 1 21:57:58.202420932 (52h0m48.140079471s) [noise=true , address=104.36.82.6:4875 , piece_id=CS5AMB7CIBQWWOU57Z3RCCSNVKIPE4B5QWU5YIQYI2UCWH7PAPVA, seg_pos={0 0}, upload=49973]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:104.36.82.6:4875): protocol: write tcp 188.166.225.170:40166->104.36.82.6:4875: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:58.202441131 (52h0m48.14009963s) [noise=true , address=104.36.82.5:4840 , piece_id=UYVKVFCJJP7XZYRIVPS4ABGEMDUMNR7YW4QGG5ZFTFCGNCJO3CSQ, seg_pos={0 0}, upload=49973]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:104.36.82.5:4840): protocol: rpc: dial tcp 104.36.82.5:4840: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:58.202448445 (52h0m48.140106954s) [noise=true , address=104.36.82.7:4909 , piece_id=QHJCVBT6KTU34K2KWFNACWYVVE3WJ2RU76BHVSKL4HLCZEFXZYPA, seg_pos={0 0}, upload=49973]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:104.36.82.7:4909): protocol: rpc: dial tcp 104.36.82.7:4909: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:58.202496487 (52h0m48.140155026s) [noise=true , address=104.36.82.4:4802 , piece_id=CLM2G33M6AK5HIESRW6LZY33NKHIR2PGITWJ23NPILRCA6UYIEHA, seg_pos={0 0}, upload=49973]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: write tcp 188.166.225.170:47696->104.36.82.4:4802: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:58.202524240 (52h0m48.140182739s) [seg_pos={0 0}, upload=49973]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Aug 1 21:57:58.435983932 (52h0m48.373642431s) [upload=49973]: Done uploading segments... Aug 1 21:57:58.436007667 (52h0m48.373666166s) [upload=49973]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.623254752s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [......................................................XX.X.X] duration: 30.234791772s FAIL (2025-08-01 21:57:58 UTC) orbiter warning: Operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 7NK72474VWE4E3YOBCDZ5O5ZOH2MSA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB): protocol: write tcp 188.166.225.170:38368->104.36.82.12:4151: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:56:50.418604753 (51h59m40.356263252s) [seg_pos={0 0}, upload=49961]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 1 21:56:50.418680879 (51h59m40.356339378s) [upload=49961]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 1 21:56:50.418685998 (51h59m40.356344487s) [upload=49961]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 1 21:56:50.726563872 (51h59m40.664222371s) [upload=49961]: Done uploading segments... Aug 1 21:56:50.726586586 (51h59m40.664245085s) [upload=49961]: Upload finished. err: <nil> Aug 1 21:57:28.852869669 (52h0m18.790528158s) [upload=49973]: Starting upload Aug 1 21:57:28.852942389 (52h0m18.790600888s) [upload=49973]: Done starting upload Aug 1 21:57:28.853007122 (52h0m18.790665611s) [upload=49973]: Uploading segments... Aug 1 21:57:28.853174002 (52h0m18.790832491s) [upload=49973]: Got next segment. Inline: false Aug 1 21:57:28.853208168 (52h0m18.790866657s) [upload=49973]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 1 21:57:28.853212666 (52h0m18.790871165s) [upload=49973]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 1 21:57:29.095087519 (52h0m19.032746008s) [seg_pos={0 0}, upload=49973]: Begin upload segment... Aug 1 21:57:29.095129960 (52h0m19.032788459s) [seg_pos={0 0}, upload=49973]: Done begin upload segment. Aug 1 21:57:29.095164997 (52h0m19.032823496s) [noise=true , address=77.90.31.21:4878 , piece_id=BLNTYBOAIZWISSUJYK6BSBPTSNL6QWLGEN6AP6P4NPDVSURYG5AA, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.095171800 (52h0m19.032830289s) [noise=true , address=104.36.82.6:4875 , piece_id=CS5AMB7CIBQWWOU57Z3RCCSNVKIPE4B5QWU5YIQYI2UCWH7PAPVA, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.095510027 (52h0m19.033168526s) [noise=true , address=77.90.31.18:4806 , piece_id=52B6CAHGLFVTFIDQHHHAWKCBP2WJJY7OTTGRZZ7CEUH3HDA43Q6A, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.095544193 (52h0m19.033202692s) [noise=true , address=77.90.31.10:4518 , piece_id=WUOVXUSDG6YQTDPWG4YO7VLJNZZVLH6VRGMWOJABG2EJRU7XHP5A, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.095774283 (52h0m19.033432782s) [noise=true , address=104.36.82.7:4909 , piece_id=QHJCVBT6KTU34K2KWFNACWYVVE3WJ2RU76BHVSKL4HLCZEFXZYPA, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.095819880 (52h0m19.033478379s) [noise=true , address=45.91.116.12:4225 , piece_id=PBQPLA5ZYRFPSKHDKCEFOEBS5T3GYMP2FLSFDQWJLVY756KPLJNQ, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.096021656 (52h0m19.033680155s) [noise=true , address=45.91.116.13:4153 , piece_id=PMLKMZO66KXF3ZONO5ZZOQZICFK5XNGL6MN5FFBKICVFU5AWU6PQ, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.096069618 (52h0m19.033728117s) [noise=true , address=45.91.116.9:4262 , piece_id=A3F4L3RR6WPFMBFFKLATNGG3CASBIPWQP5O3G5URFNTXBE3ZW3VA, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.096348041 (52h0m19.03400654s) [noise=true , address=104.36.82.5:4840 , piece_id=UYVKVFCJJP7XZYRIVPS4ABGEMDUMNR7YW4QGG5ZFTFCGNCJO3CSQ, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.096354803 (52h0m19.034013292s) [noise=true , address=77.90.31.12:4590 , piece_id=D6FUBIFCQJSDRGWEP5GG34AQPNB3UROCYOATD6SRTJ7KAYVMFCOA, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.096608579 (52h0m19.034267078s) [noise=true , address=77.90.31.16:4733 , piece_id=DMASMGOUATLPEKCGLHKNNIMMFYVVGSULNUOMJU6RJY3ULBDOY6KQ, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.096830774 (52h0m19.034489273s) [noise=true , address=104.36.82.4:4802 , piece_id=CLM2G33M6AK5HIESRW6LZY33NKHIR2PGITWJ23NPILRCA6UYIEHA, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.097107233 (52h0m19.034765732s) [noise=true , address=77.90.31.19:4843 , piece_id=UFONZYHL7EAFYE5PRW32RKUVQPMZPXBEFLLIFJMAB2TXJ3C6KKNA, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.097324328 (52h0m19.034982827s) [noise=true , address=77.90.31.17:4771 , piece_id=Y3WSALJX23CWPPYQ3XYMCFOSBLBX2E2JQCCS7EOEYNBWY4CDRT3Q, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.098682446 (52h0m19.036340945s) [noise=true , address=77.90.31.15:4698 , piece_id=TN4OTPUD47FD6WOLZBEHNDI43WY6QVO5JK2JNJXNVM2IT6XMGERQ, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.099041603 (52h0m19.036700102s) [noise=true , address=45.91.116.11:4118 , piece_id=AKZIOAR6NRNO7JWCQZIKNSL7IUH5S4IG3KPG3IY2FAKG74DH3E3Q, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:30.718412255 (52h0m20.656070754s) [upload=49973]: Next returned nil segment Aug 1 21:57:30.718429869 (52h0m20.656088368s) [upload=49973]: Waiting for error group managing segments... Aug 1 21:57:31.107386675 (52h0m21.045045164s) [noise=true , address=45.91.116.13:4153 , piece_id=PMLKMZO66KXF3ZONO5ZZOQZICFK5XNGL6MN5FFBKICVFU5AWU6PQ, seg_pos={0 0}, upload=49973]: Done uploading piece. err: <nil> Aug 1 21:57:31.108529412 (52h0m21.046187911s) [noise=true , address=77.90.31.12:4590 , piece_id=D6FUBIFCQJSDRGWEP5GG34AQPNB3UROCYOATD6SRTJ7KAYVMFCOA, seg_pos={0 0}, upload=49973]: Done uploading piece. err: <nil> Aug 1 21:57:31.109975858 (52h0m21.047634347s) [noise=true , address=77.90.31.15:4698 , piece_id=TN4OTPUD47FD6WOLZBEHNDI43WY6QVO5JK2JNJXNVM2IT6XMGERQ, seg_pos={0 0}, upload=49973]: Done uploading piece. err: <nil> Aug 1 21:57:31.115273013 (52h0m21.052931502s) [noise=true , address=77.90.31.10:4518 , piece_id=WUOVXUSDG6YQTDPWG4YO7VLJNZZVLH6VRGMWOJABG2EJRU7XHP5A, seg_pos={0 0}, upload=49973]: Done uploading piece. err: <nil> Aug 1 21:57:31.116767281 (52h0m21.054425781s) [noise=true , address=77.90.31.18:4806 , piece_id=52B6CAHGLFVTFIDQHHHAWKCBP2WJJY7OTTGRZZ7CEUH3HDA43Q6A, seg_pos={0 0}, upload=49973]: Done uploading piece. err: <nil> Aug 1 21:57:31.201146107 (52h0m21.138804606s) [noise=true , address=77.90.31.16:4733 , piece_id=DMASMGOUATLPEKCGLHKNNIMMFYVVGSULNUOMJU6RJY3ULBDOY6KQ, seg_pos={0 0}, upload=49973]: Done uploading piece. err: <nil> Aug 1 21:57:31.201833732 (52h0m21.139492231s) [noise=true , address=77.90.31.19:4843 , piece_id=UFONZYHL7EAFYE5PRW32RKUVQPMZPXBEFLLIFJMAB2TXJ3C6KKNA, seg_pos={0 0}, upload=49973]: Done uploading piece. err: <nil> Aug 1 21:57:31.211007613 (52h0m21.148666112s) [noise=true , address=45.91.116.9:4262 , piece_id=A3F4L3RR6WPFMBFFKLATNGG3CASBIPWQP5O3G5URFNTXBE3ZW3VA, seg_pos={0 0}, upload=49973]: Done uploading piece. err: <nil> Aug 1 21:57:31.215929169 (52h0m21.153587668s) [noise=true , address=77.90.31.17:4771 , piece_id=Y3WSALJX23CWPPYQ3XYMCFOSBLBX2E2JQCCS7EOEYNBWY4CDRT3Q, seg_pos={0 0}, upload=49973]: Done uploading piece. err: <nil> Aug 1 21:57:31.222973975 (52h0m21.160632474s) [noise=true , address=77.90.31.21:4878 , piece_id=BLNTYBOAIZWISSUJYK6BSBPTSNL6QWLGEN6AP6P4NPDVSURYG5AA, seg_pos={0 0}, upload=49973]: Done uploading piece. err: <nil> Aug 1 21:57:31.234392441 (52h0m21.17205094s) [noise=true , address=45.91.116.12:4225 , piece_id=PBQPLA5ZYRFPSKHDKCEFOEBS5T3GYMP2FLSFDQWJLVY756KPLJNQ, seg_pos={0 0}, upload=49973]: Done uploading piece. err: <nil> Aug 1 21:57:31.284794305 (52h0m21.222452804s) [noise=true , address=45.91.116.11:4118 , piece_id=AKZIOAR6NRNO7JWCQZIKNSL7IUH5S4IG3KPG3IY2FAKG74DH3E3Q, seg_pos={0 0}, upload=49973]: Done uploading piece. err: <nil> Aug 1 21:57:58.202420932 (52h0m48.140079471s) [noise=true , address=104.36.82.6:4875 , piece_id=CS5AMB7CIBQWWOU57Z3RCCSNVKIPE4B5QWU5YIQYI2UCWH7PAPVA, seg_pos={0 0}, upload=49973]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:104.36.82.6:4875): protocol: write tcp 188.166.225.170:40166->104.36.82.6:4875: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:58.202441131 (52h0m48.14009963s) [noise=true , address=104.36.82.5:4840 , piece_id=UYVKVFCJJP7XZYRIVPS4ABGEMDUMNR7YW4QGG5ZFTFCGNCJO3CSQ, seg_pos={0 0}, upload=49973]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:104.36.82.5:4840): protocol: rpc: dial tcp 104.36.82.5:4840: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:58.202448445 (52h0m48.140106954s) [noise=true , address=104.36.82.7:4909 , piece_id=QHJCVBT6KTU34K2KWFNACWYVVE3WJ2RU76BHVSKL4HLCZEFXZYPA, seg_pos={0 0}, upload=49973]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:104.36.82.7:4909): protocol: rpc: dial tcp 104.36.82.7:4909: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:58.202496487 (52h0m48.140155026s) [noise=true , address=104.36.82.4:4802 , piece_id=CLM2G33M6AK5HIESRW6LZY33NKHIR2PGITWJ23NPILRCA6UYIEHA, seg_pos={0 0}, upload=49973]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: write tcp 188.166.225.170:47696->104.36.82.4:4802: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:58.202524240 (52h0m48.140182739s) [seg_pos={0 0}, upload=49973]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Aug 1 21:57:58.435983932 (52h0m48.373642431s) [upload=49973]: Done uploading segments... Aug 1 21:57:58.436007667 (52h0m48.373666166s) [upload=49973]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.234815577s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [.......................................................XXX..] duration: 5.848516236s FAIL (2025-08-01 21:57:50 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.848538117s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [........................................................XXXX] duration: 1m0.234200112s FAIL (2025-08-01 21:57:43 UTC) orbiter warning: Operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 7) ======LOGS====== K4BW277ZSGXJYQDA66OMEBVDC7DAWSOJAPX42HANNGRZN35KHQ, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.481101510 (51h59m34.418760009s) [noise=true , address=104.36.82.5:4842 , piece_id=LV7YAOMPE2AV7TF2WPWR7QJNLT4JZAIPSVZN3E23U5KDBNKZFD6A, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.481422003 (51h59m34.419080502s) [noise=true , address=79.127.205.235:4683 , piece_id=3QHDNAPG4GRLJFVVNJNFXNGKTGKUEMWWUBMW62HHE5N5LJDO4KWA, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.482030897 (51h59m34.419689396s) [noise=true , address=79.127.219.51:4213 , piece_id=L2WSPI5BEIFLGQKJIK4EWXFQXNQRQ7HVKPOWFYRPPUGNAL5DVYWA, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.482285215 (51h59m34.419943704s) [noise=true , address=104.36.82.14:4251 , piece_id=O7NADU347E2Q26TPD3IUCK6WE5UVRITC6ALZHBE5SWYTKWXFBMXA, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.482518370 (51h59m34.420176859s) [noise=true , address=79.127.220.98:4402 , piece_id=DVLSYRFFZ6UT3QRSVAN7AMXVYZNI6PWG2IITHLRUUJATJVK3QOGA, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.482724304 (51h59m34.420382803s) [noise=true , address=79.127.201.214:4297 , piece_id=MPTMGVEYC4SATIN4ANCR2ELVRXHYWRDLMHWQW7HM6UOIV2N6R7ZA, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.482928555 (51h59m34.420587054s) [noise=true , address=104.36.82.37:4680 , piece_id=X4PDUE4WXUUWX6JQVO4NWPQCBMHMHC5YHPIHDQVYYOI2FXV2GVJQ, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.483134960 (51h59m34.420793449s) [noise=true , address=104.36.82.50:4027 , piece_id=OW7BODU5N7ZEEGE2FHSLNYVJT2Z25LY3J42TZEOR5SP2EWIN6X7A, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.483338480 (51h59m34.420996979s) [noise=true , address=104.36.82.55:4204 , piece_id=R5CYXWAVV4ZEMCV7LF4D4XDC4G6KLVBONAH2TH5EIEVL7QHUTRWQ, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.483582847 (51h59m34.421241356s) [noise=true , address=104.36.82.11:4134 , piece_id=RMTNNNC2VD7MGWJDAMT6ZCF3GR43SHANFVAGJ6EHVH7WO5XA6VTA, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.483833737 (51h59m34.421492236s) [noise=true , address=104.36.82.43:4856 , piece_id=2SINBLHGRUOWUO4XHBNBG45YE3PW7RNF3QSNVCQXNCFEFHRBFNJQ, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.484021896 (51h59m34.421680385s) [noise=true , address=104.36.82.52:4101 , piece_id=XYPLQM7QITCCX7T4NKIXP5MZ7I6F24EJJTG37K4NPOBKTUFHULRQ, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.484301662 (51h59m34.421960182s) [noise=true , address=79.127.201.215:4352 , piece_id=I6BRYI5XIACAAAAAE7HQOBCF4OTALMOVPPNBWQCRSH2MM6WUUURQ, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.484712097 (51h59m34.422370596s) [noise=true , address=64.49.13.138:4484 , piece_id=DZ5Y4HTS6BKW2DJ5CVOB64TW4SXB6DNCJEGFUJPLAUXUEHRZRTUA, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:45.576021470 (51h59m35.513679959s) [upload=49960]: Next returned nil segment Aug 1 21:56:45.576036609 (51h59m35.513695098s) [upload=49960]: Waiting for error group managing segments... Aug 1 21:56:45.860378776 (51h59m35.798037275s) [noise=true , address=64.49.13.138:4484 , piece_id=DZ5Y4HTS6BKW2DJ5CVOB64TW4SXB6DNCJEGFUJPLAUXUEHRZRTUA, seg_pos={0 0}, upload=49960]: Done uploading piece. err: <nil> Aug 1 21:56:45.884184188 (51h59m35.821842687s) [noise=true , address=79.127.205.235:4683 , piece_id=3QHDNAPG4GRLJFVVNJNFXNGKTGKUEMWWUBMW62HHE5N5LJDO4KWA, seg_pos={0 0}, upload=49960]: Done uploading piece. err: <nil> Aug 1 21:56:46.145493612 (51h59m36.083152101s) [noise=true , address=79.127.220.98:4402 , piece_id=DVLSYRFFZ6UT3QRSVAN7AMXVYZNI6PWG2IITHLRUUJATJVK3QOGA, seg_pos={0 0}, upload=49960]: Done uploading piece. err: <nil> Aug 1 21:56:46.257505313 (51h59m36.195163802s) [noise=true , address=79.127.220.97:4371 , piece_id=5WGPRFKQETIHTSMA42AM2C3UGCFYJ3DMMPAI2L7DD6E2P37KJKNA, seg_pos={0 0}, upload=49960]: Done uploading piece. err: <nil> Aug 1 21:56:46.334426181 (51h59m36.27208468s) [noise=true , address=79.127.201.215:4352 , piece_id=I6BRYI5XIACAAAAAE7HQOBCF4OTALMOVPPNBWQCRSH2MM6WUUURQ, seg_pos={0 0}, upload=49960]: Done uploading piece. err: <nil> Aug 1 21:56:46.386119275 (51h59m36.323777774s) [noise=true , address=79.127.201.214:4297 , piece_id=MPTMGVEYC4SATIN4ANCR2ELVRXHYWRDLMHWQW7HM6UOIV2N6R7ZA, seg_pos={0 0}, upload=49960]: Done uploading piece. err: <nil> Aug 1 21:56:46.395362048 (51h59m36.333020547s) [noise=true , address=79.127.219.51:4213 , piece_id=L2WSPI5BEIFLGQKJIK4EWXFQXNQRQ7HVKPOWFYRPPUGNAL5DVYWA, seg_pos={0 0}, upload=49960]: Done uploading piece. err: <nil> Aug 1 21:57:43.392434383 (52h0m33.330092872s) [noise=true , address=104.36.82.37:4680 , piece_id=X4PDUE4WXUUWX6JQVO4NWPQCBMHMHC5YHPIHDQVYYOI2FXV2GVJQ, seg_pos={0 0}, upload=49960]: Done uploading piece. err: ecclient: upload failed (node:1bXAqpxGhUvzwYEDGBqYRhZdTeMJcV3TYfngkKTCumpt9Nupi3, address:104.36.82.37:4680): protocol: rpc: dial tcp 104.36.82.37:4680: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:43.392477967 (52h0m33.330136466s) [noise=true , address=104.36.82.11:4134 , piece_id=RMTNNNC2VD7MGWJDAMT6ZCF3GR43SHANFVAGJ6EHVH7WO5XA6VTA, seg_pos={0 0}, upload=49960]: Done uploading piece. err: ecclient: upload failed (node:12R3Z5AgkBmWXMv5XgRyfcyqkG2gmyJs9ogT24kcLy6nuB9bzUG, address:104.36.82.11:4134): protocol: rpc: dial tcp 104.36.82.11:4134: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:43.392514086 (52h0m33.330172585s) [noise=true , address=104.36.82.55:4204 , piece_id=R5CYXWAVV4ZEMCV7LF4D4XDC4G6KLVBONAH2TH5EIEVL7QHUTRWQ, seg_pos={0 0}, upload=49960]: Done uploading piece. err: ecclient: upload failed (node:1rBZH6aLHsnWh22uNzaMiMwmLJCDMn4bYbNC5jRG7KCEuRfqpH, address:104.36.82.55:4204): protocol: write tcp 188.166.225.170:48550->104.36.82.55:4204: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:43.392546537 (52h0m33.330205027s) [noise=true , address=104.36.82.59:4327 , piece_id=OZDEYIY6B4WYZULRDRL2UBOJH2JQNY3YWSITZKWDO7GTT72AN5NQ, seg_pos={0 0}, upload=49960]: Done uploading piece. err: ecclient: upload failed (node:1V4C6AWGbyYQWfinoSj7iKovHeTJ9hqypyC5U9rZ8pveXBBfCY, address:104.36.82.59:4327): protocol: rpc: dial tcp 104.36.82.59:4327: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:43.392579040 (52h0m33.33023754s) [noise=true , address=104.36.82.5:4842 , piece_id=LV7YAOMPE2AV7TF2WPWR7QJNLT4JZAIPSVZN3E23U5KDBNKZFD6A, seg_pos={0 0}, upload=49960]: Done uploading piece. err: ecclient: upload failed (node:12q4VdT5BcjUb16PBYHK6zrnW8dqENKrcGx9zdLtNefDVB1MUSn, address:104.36.82.5:4842): protocol: context deadline exceeded; context deadline exceeded Aug 1 21:57:43.392627953 (52h0m33.330286442s) [noise=true , address=104.36.82.52:4101 , piece_id=XYPLQM7QITCCX7T4NKIXP5MZ7I6F24EJJTG37K4NPOBKTUFHULRQ, seg_pos={0 0}, upload=49960]: Done uploading piece. err: ecclient: upload failed (node:12WWXeBjFZR9PzciNfs35fFcx87N9GwDpMsPHnksFDugtaBGWp4, address:104.36.82.52:4101): protocol: rpc: dial tcp 104.36.82.52:4101: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:43.392653192 (52h0m33.330311691s) [noise=true , address=104.36.82.14:4251 , piece_id=O7NADU347E2Q26TPD3IUCK6WE5UVRITC6ALZHBE5SWYTKWXFBMXA, seg_pos={0 0}, upload=49960]: Done uploading piece. err: ecclient: upload failed (node:12btf6Z38aTMx7MaBxhUyZtFqUPgeMtP2aU8Y6FbV8zwu57zArd, address:104.36.82.14:4251): protocol: rpc: dial tcp 104.36.82.14:4251: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:43.392670836 (52h0m33.330329335s) [noise=true , address=104.36.82.12:4166 , piece_id=6ZIWCOBY76GJ6DHZF5G23U4IYTOFB5SMQW5TW3YITYH227PV2FGA, seg_pos={0 0}, upload=49960]: Done uploading piece. err: ecclient: upload failed (node:12iCB7c4ZfpqafpD1PtpbE6BfU81wJW65LmrPVJXtD58yeuoR9t, address:104.36.82.12:4166): protocol: rpc: dial tcp 104.36.82.12:4166: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:43.392698018 (52h0m33.330356517s) [noise=true , address=104.36.82.50:4027 , piece_id=OW7BODU5N7ZEEGE2FHSLNYVJT2Z25LY3J42TZEOR5SP2EWIN6X7A, seg_pos={0 0}, upload=49960]: Done uploading piece. err: ecclient: upload failed (node:1zEQhyuVxSHwdDUgDfcTog2W3oLBx4z9bYUMHpaaoLU1BTU9DQ, address:104.36.82.50:4027): protocol: write tcp 188.166.225.170:37784->104.36.82.50:4027: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:43.392715381 (52h0m33.33037387s) [noise=true , address=104.36.82.43:4856 , piece_id=2SINBLHGRUOWUO4XHBNBG45YE3PW7RNF3QSNVCQXNCFEFHRBFNJQ, seg_pos={0 0}, upload=49960]: Done uploading piece. err: ecclient: upload failed (node:12bQxwBfPEGtmMp6mtnWKsoZt3eMEjLdyYFahETWGHeKJ7BhkiZ, address:104.36.82.43:4856): protocol: rpc: dial tcp 104.36.82.43:4856: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:43.392730059 (52h0m33.330388558s) [noise=true , address=104.36.82.10:4082 , piece_id=45K4BW277ZSGXJYQDA66OMEBVDC7DAWSOJAPX42HANNGRZN35KHQ, seg_pos={0 0}, upload=49960]: Done uploading piece. err: ecclient: upload failed (node:12DKsNCtqxLvpJwLszdLs74V3eb8NH2VAV9V3bngzUczawmNoLK, address:104.36.82.10:4082): protocol: write tcp 188.166.225.170:46638->104.36.82.10:4082: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:43.392753424 (52h0m33.330411923s) [seg_pos={0 0}, upload=49960]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil> Aug 1 21:57:43.624550997 (52h0m33.562209566s) [upload=49960]: Done uploading segments... Aug 1 21:57:43.624582868 (52h0m33.562241407s) [upload=49960]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7) Duration: 1m0.234223968s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [........................................................XXX.] duration: 55.830889871s FAIL (2025-08-01 21:57:29 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 55.83090026s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 3 out of 60 is failed (threshold=3) [......................................X............X.....X..] duration: 2.031964888s FAIL (2025-08-01 21:57:22 UTC) orbiter warning: Operation: ListObjects unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/1K Duration: 2.032003312s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 6.187124477s FAIL (2025-08-01 21:57:18 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.18714632s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 3 out of 60 is failed (threshold=3) [.......................................X............X.....X.] duration: 2.060039429s FAIL (2025-08-01 21:57:06 UTC) orbiter warning: Operation: ListObjects unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/1K Duration: 2.060086429s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 5 out of 60 is failed (threshold=3) [.......................................................XXXXX] duration: 30.234900448s FAIL (2025-08-01 21:57:03 UTC) orbiter warning: Operation: Commit uplink: failed to upload enough pieces (needed at least 14 but got 3) Duration: 30.23492136s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.......................................................XX.X.] duration: 5.873428582s FAIL (2025-08-01 21:56:53 UTC) orbiter warning: Operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== PHZDYICM22IPO5JISYWPVMDHEKLQ, seg_pos={0 0}, upload=49934]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5): protocol: write tcp 188.166.225.170:51948->104.36.82.3:4767: use of closed network connection; conn closed: op; piecestore: piecestore close: context canceled Aug 1 21:55:19.088488708 (51h58m9.026147237s) [seg_pos={0 0}, upload=49934]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 1 21:55:19.088596624 (51h58m9.026255143s) [upload=49934]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 1 21:55:19.088603167 (51h58m9.026261706s) [upload=49934]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 1 21:55:19.368865049 (51h58m9.306523548s) [upload=49934]: Done uploading segments... Aug 1 21:55:19.368916437 (51h58m9.306574936s) [upload=49934]: Upload finished. err: <nil> Aug 1 21:56:17.505345190 (51h59m7.443003689s) [upload=49953]: Starting upload Aug 1 21:56:17.505417057 (51h59m7.443075556s) [upload=49953]: Done starting upload Aug 1 21:56:17.505616580 (51h59m7.443275079s) [upload=49953]: Uploading segments... Aug 1 21:56:17.505648681 (51h59m7.44330717s) [upload=49953]: Got next segment. Inline: false Aug 1 21:56:17.505670042 (51h59m7.443328531s) [upload=49953]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 1 21:56:17.505674160 (51h59m7.443332659s) [upload=49953]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 1 21:56:17.740419662 (51h59m7.678078161s) [seg_pos={0 0}, upload=49953]: Begin upload segment... Aug 1 21:56:17.740470409 (51h59m7.678128908s) [seg_pos={0 0}, upload=49953]: Done begin upload segment. Aug 1 21:56:17.740494956 (51h59m7.678153455s) [noise=true , address=45.91.116.5:4299 , piece_id=3SB6BSR5AK3KTJIK2AWKVU47EZEPRS42LGP7TX4F46HN5NQ767FA, seg_pos={0 0}, upload=49953]: Uploading piece... Aug 1 21:56:17.740530123 (51h59m7.678188622s) [noise=true , address=104.36.82.8:4971 , piece_id=WK63NO3PBWOXPYRQWKNVHHWIZDF2LZQTYVPNMUUJOG3J6I42PS7A, seg_pos={0 0}, upload=49953]: Uploading piece... Aug 1 21:56:17.740846219 (51h59m7.678504718s) [noise=true , address=77.90.31.19:4842 , piece_id=GVTBKMNYSMPY5IF2O2MOUYHIUPK4P5V4KIYSIWBXBXBVA34QOM3A, seg_pos={0 0}, upload=49953]: Uploading piece... Aug 1 21:56:17.741111195 (51h59m7.678769694s) [noise=true , address=77.90.31.21:4877 , piece_id=MEPTGIEH6XWK2STUCZFU6CO524P4VGX4GMCTYFTWVBP3MPSHNY3A, seg_pos={0 0}, upload=49953]: Uploading piece... Aug 1 21:56:17.741486403 (51h59m7.679144922s) [noise=true , address=45.91.116.13:4154 , piece_id=4F6HX6YABGLIDHO7H6MWCYQ57533PJ2YGQFQJMBXJYOS4Z4XY5SA, seg_pos={0 0}, upload=49953]: Uploading piece... Aug 1 21:56:17.741534395 (51h59m7.679192884s) [noise=true , address=45.91.116.10:4083 , piece_id=572YKDEMWDBADDOY3WGN6CECJGER5JECQWZXWHOQKKNFNQTIVQDQ, seg_pos={0 0}, upload=49953]: Uploading piece... Aug 1 21:56:17.741862303 (51h59m7.679520832s) [noise=true , address=104.36.82.2:4731 , piece_id=SLLCYPXHYJU2UWVEYA7ATJLBAIFBDDBHCN2EJEO2HOO67C4DOXJA, seg_pos={0 0}, upload=49953]: Uploading piece... Aug 1 21:56:17.742153339 (51h59m7.679811838s) [noise=true , address=77.90.31.15:4699 , piece_id=EQLOXQDBN57AR5BENQ6FAVCFES2PTRGUADAFZNHQKFCNSRF27GJQ, seg_pos={0 0}, upload=49953]: Uploading piece... Aug 1 21:56:17.742300440 (51h59m7.679958939s) [noise=true , address=45.91.116.11:4119 , piece_id=Y7EG3JIEFXBT5I4VOB7YJBCBOCBDK4P2MCOVSGU7LO6XXNYD42AQ, seg_pos={0 0}, upload=49953]: Uploading piece... Aug 1 21:56:17.742350346 (51h59m7.680008845s) [noise=true , address=45.91.116.14:4191 , piece_id=AWA66MJC4C6UK2323UFYG3P7LRENFWTE3I6EGFWS254NLTBT256A, seg_pos={0 0}, upload=49953]: Uploading piece... Aug 1 21:56:17.742547453 (51h59m7.680205952s) [noise=true , address=45.91.116.4:4940 , piece_id=7TAVU47XZ5QZPKQUSBDU2VAU3NR4LMLI6NELCNPOQY7BKX4QDKEQ, seg_pos={0 0}, upload=49953]: Uploading piece... Aug 1 21:56:17.742734120 (51h59m7.680392619s) [noise=true , address=77.90.31.12:4590 , piece_id=FJD6CZBMQ3DU4ZAGIYOYUWRR6KMGNFNGXDNO55MGHA55MGYGGUCA, seg_pos={0 0}, upload=49953]: Uploading piece... Aug 1 21:56:17.742927009 (51h59m7.680585508s) [noise=true , address=77.90.31.11:4554 , piece_id=Q2W634RB76ZPR3D2PJZUG45FXMS7FCFN4IF745NEFQW44RU5RUFA, seg_pos={0 0}, upload=49953]: Uploading piece... Aug 1 21:56:17.743122072 (51h59m7.680780571s) [noise=true , address=104.36.82.4:4802 , piece_id=NESKRXISSKAEDLXBMTHYDZ2OVUJUGCYZ6JVG2ZHBTKIO4SZSZDMA, seg_pos={0 0}, upload=49953]: Uploading piece... Aug 1 21:56:17.743324500 (51h59m7.680982999s) [noise=true , address=104.36.82.7:4910 , piece_id=SH2RMPWPAW24CX6MHIBJC5Z7S2DMPTC6DKJVXKGCIZNGZJQNV5NQ, seg_pos={0 0}, upload=49953]: Uploading piece... Aug 1 21:56:17.743537677 (51h59m7.681196166s) [noise=true , address=77.90.31.14:4661 , piece_id=L3JNMTKXW7A4JLPIUNPKS7V76UDQUQO2IHAWCHUF2AYLPPBTIXEQ, seg_pos={0 0}, upload=49953]: Uploading piece... Aug 1 21:56:19.449135562 (51h59m9.386794061s) [upload=49953]: Next returned nil segment Aug 1 21:56:19.449151993 (51h59m9.386810512s) [upload=49953]: Waiting for error group managing segments... Aug 1 21:56:19.759811709 (51h59m9.697470208s) [noise=true , address=77.90.31.12:4590 , piece_id=FJD6CZBMQ3DU4ZAGIYOYUWRR6KMGNFNGXDNO55MGHA55MGYGGUCA, seg_pos={0 0}, upload=49953]: Done uploading piece. err: <nil> Aug 1 21:56:19.764469559 (51h59m9.702128058s) [noise=true , address=77.90.31.15:4699 , piece_id=EQLOXQDBN57AR5BENQ6FAVCFES2PTRGUADAFZNHQKFCNSRF27GJQ, seg_pos={0 0}, upload=49953]: Done uploading piece. err: <nil> Aug 1 21:56:19.769315670 (51h59m9.706974169s) [noise=true , address=45.91.116.10:4083 , piece_id=572YKDEMWDBADDOY3WGN6CECJGER5JECQWZXWHOQKKNFNQTIVQDQ, seg_pos={0 0}, upload=49953]: Done uploading piece. err: <nil> Aug 1 21:56:19.770933436 (51h59m9.708591935s) [noise=true , address=45.91.116.13:4154 , piece_id=4F6HX6YABGLIDHO7H6MWCYQ57533PJ2YGQFQJMBXJYOS4Z4XY5SA, seg_pos={0 0}, upload=49953]: Done uploading piece. err: <nil> Aug 1 21:56:19.780547048 (51h59m9.718205547s) [noise=true , address=77.90.31.11:4554 , piece_id=Q2W634RB76ZPR3D2PJZUG45FXMS7FCFN4IF745NEFQW44RU5RUFA, seg_pos={0 0}, upload=49953]: Done uploading piece. err: <nil> Aug 1 21:56:19.847756306 (51h59m9.785414805s) [noise=true , address=45.91.116.11:4119 , piece_id=Y7EG3JIEFXBT5I4VOB7YJBCBOCBDK4P2MCOVSGU7LO6XXNYD42AQ, seg_pos={0 0}, upload=49953]: Done uploading piece. err: <nil> Aug 1 21:56:19.852604211 (51h59m9.79026271s) [noise=true , address=45.91.116.14:4191 , piece_id=AWA66MJC4C6UK2323UFYG3P7LRENFWTE3I6EGFWS254NLTBT256A, seg_pos={0 0}, upload=49953]: Done uploading piece. err: <nil> Aug 1 21:56:19.858421450 (51h59m9.796079949s) [noise=true , address=45.91.116.4:4940 , piece_id=7TAVU47XZ5QZPKQUSBDU2VAU3NR4LMLI6NELCNPOQY7BKX4QDKEQ, seg_pos={0 0}, upload=49953]: Done uploading piece. err: <nil> Aug 1 21:56:19.858590934 (51h59m9.796249433s) [noise=true , address=77.90.31.21:4877 , piece_id=MEPTGIEH6XWK2STUCZFU6CO524P4VGX4GMCTYFTWVBP3MPSHNY3A, seg_pos={0 0}, upload=49953]: Done uploading piece. err: <nil> Aug 1 21:56:19.910467720 (51h59m9.848126219s) [noise=true , address=77.90.31.19:4842 , piece_id=GVTBKMNYSMPY5IF2O2MOUYHIUPK4P5V4KIYSIWBXBXBVA34QOM3A, seg_pos={0 0}, upload=49953]: Done uploading piece. err: <nil> Aug 1 21:56:19.926072253 (51h59m9.863730742s) [noise=true , address=77.90.31.14:4661 , piece_id=L3JNMTKXW7A4JLPIUNPKS7V76UDQUQO2IHAWCHUF2AYLPPBTIXEQ, seg_pos={0 0}, upload=49953]: Done uploading piece. err: <nil> Aug 1 21:56:19.942164790 (51h59m9.879823279s) [noise=true , address=45.91.116.5:4299 , piece_id=3SB6BSR5AK3KTJIK2AWKVU47EZEPRS42LGP7TX4F46HN5NQ767FA, seg_pos={0 0}, upload=49953]: Done uploading piece. err: <nil> Aug 1 21:56:46.858074017 (51h59m36.795732516s) [noise=true , address=104.36.82.8:4971 , piece_id=WK63NO3PBWOXPYRQWKNVHHWIZDF2LZQTYVPNMUUJOG3J6I42PS7A, seg_pos={0 0}, upload=49953]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:104.36.82.8:4971): protocol: rpc: dial tcp 104.36.82.8:4971: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:56:46.858123221 (51h59m36.79578172s) [noise=true , address=104.36.82.4:4802 , piece_id=NESKRXISSKAEDLXBMTHYDZ2OVUJUGCYZ6JVG2ZHBTKIO4SZSZDMA, seg_pos={0 0}, upload=49953]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: write tcp 188.166.225.170:54466->104.36.82.4:4802: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:56:46.858152237 (51h59m36.795810726s) [noise=true , address=104.36.82.2:4731 , piece_id=SLLCYPXHYJU2UWVEYA7ATJLBAIFBDDBHCN2EJEO2HOO67C4DOXJA, seg_pos={0 0}, upload=49953]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:104.36.82.2:4731): protocol: write tcp 188.166.225.170:53948->104.36.82.2:4731: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:56:46.858245054 (51h59m36.795903553s) [noise=true , address=104.36.82.7:4910 , piece_id=SH2RMPWPAW24CX6MHIBJC5Z7S2DMPTC6DKJVXKGCIZNGZJQNV5NQ, seg_pos={0 0}, upload=49953]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: rpc: dial tcp 104.36.82.7:4910: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:56:46.858280922 (51h59m36.795939421s) [seg_pos={0 0}, upload=49953]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Aug 1 21:56:47.084606176 (51h59m37.022264675s) [upload=49953]: Done uploading segments... Aug 1 21:56:47.084631495 (51h59m37.022289994s) [upload=49953]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.873444813s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 5.990379122s FAIL (2025-08-01 21:56:48 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.990391537s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [........................................................XX.X] duration: 30.228264641s FAIL (2025-08-01 21:56:47 UTC) orbiter warning: Operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== PHZDYICM22IPO5JISYWPVMDHEKLQ, seg_pos={0 0}, upload=49934]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5): protocol: write tcp 188.166.225.170:51948->104.36.82.3:4767: use of closed network connection; conn closed: op; piecestore: piecestore close: context canceled Aug 1 21:55:19.088488708 (51h58m9.026147237s) [seg_pos={0 0}, upload=49934]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 1 21:55:19.088596624 (51h58m9.026255143s) [upload=49934]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 1 21:55:19.088603167 (51h58m9.026261706s) [upload=49934]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 1 21:55:19.368865049 (51h58m9.306523548s) [upload=49934]: Done uploading segments... Aug 1 21:55:19.368916437 (51h58m9.306574936s) [upload=49934]: Upload finished. err: <nil> Aug 1 21:56:17.505345190 (51h59m7.443003689s) [upload=49953]: Starting upload Aug 1 21:56:17.505417057 (51h59m7.443075556s) [upload=49953]: Done starting upload Aug 1 21:56:17.505616580 (51h59m7.443275079s) [upload=49953]: Uploading segments... Aug 1 21:56:17.505648681 (51h59m7.44330717s) [upload=49953]: Got next segment. Inline: false Aug 1 21:56:17.505670042 (51h59m7.443328531s) [upload=49953]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 1 21:56:17.505674160 (51h59m7.443332659s) [upload=49953]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 1 21:56:17.740419662 (51h59m7.678078161s) [seg_pos={0 0}, upload=49953]: Begin upload segment... Aug 1 21:56:17.740470409 (51h59m7.678128908s) [seg_pos={0 0}, upload=49953]: Done begin upload segment. Aug 1 21:56:17.740494956 (51h59m7.678153455s) [noise=true , address=45.91.116.5:4299 , piece_id=3SB6BSR5AK3KTJIK2AWKVU47EZEPRS42LGP7TX4F46HN5NQ767FA, seg_pos={0 0}, upload=49953]: Uploading piece... Aug 1 21:56:17.740530123 (51h59m7.678188622s) [noise=true , address=104.36.82.8:4971 , piece_id=WK63NO3PBWOXPYRQWKNVHHWIZDF2LZQTYVPNMUUJOG3J6I42PS7A, seg_pos={0 0}, upload=49953]: Uploading piece... Aug 1 21:56:17.740846219 (51h59m7.678504718s) [noise=true , address=77.90.31.19:4842 , piece_id=GVTBKMNYSMPY5IF2O2MOUYHIUPK4P5V4KIYSIWBXBXBVA34QOM3A, seg_pos={0 0}, upload=49953]: Uploading piece... Aug 1 21:56:17.741111195 (51h59m7.678769694s) [noise=true , address=77.90.31.21:4877 , piece_id=MEPTGIEH6XWK2STUCZFU6CO524P4VGX4GMCTYFTWVBP3MPSHNY3A, seg_pos={0 0}, upload=49953]: Uploading piece... Aug 1 21:56:17.741486403 (51h59m7.679144922s) [noise=true , address=45.91.116.13:4154 , piece_id=4F6HX6YABGLIDHO7H6MWCYQ57533PJ2YGQFQJMBXJYOS4Z4XY5SA, seg_pos={0 0}, upload=49953]: Uploading piece... Aug 1 21:56:17.741534395 (51h59m7.679192884s) [noise=true , address=45.91.116.10:4083 , piece_id=572YKDEMWDBADDOY3WGN6CECJGER5JECQWZXWHOQKKNFNQTIVQDQ, seg_pos={0 0}, upload=49953]: Uploading piece... Aug 1 21:56:17.741862303 (51h59m7.679520832s) [noise=true , address=104.36.82.2:4731 , piece_id=SLLCYPXHYJU2UWVEYA7ATJLBAIFBDDBHCN2EJEO2HOO67C4DOXJA, seg_pos={0 0}, upload=49953]: Uploading piece... Aug 1 21:56:17.742153339 (51h59m7.679811838s) [noise=true , address=77.90.31.15:4699 , piece_id=EQLOXQDBN57AR5BENQ6FAVCFES2PTRGUADAFZNHQKFCNSRF27GJQ, seg_pos={0 0}, upload=49953]: Uploading piece... Aug 1 21:56:17.742300440 (51h59m7.679958939s) [noise=true , address=45.91.116.11:4119 , piece_id=Y7EG3JIEFXBT5I4VOB7YJBCBOCBDK4P2MCOVSGU7LO6XXNYD42AQ, seg_pos={0 0}, upload=49953]: Uploading piece... Aug 1 21:56:17.742350346 (51h59m7.680008845s) [noise=true , address=45.91.116.14:4191 , piece_id=AWA66MJC4C6UK2323UFYG3P7LRENFWTE3I6EGFWS254NLTBT256A, seg_pos={0 0}, upload=49953]: Uploading piece... Aug 1 21:56:17.742547453 (51h59m7.680205952s) [noise=true , address=45.91.116.4:4940 , piece_id=7TAVU47XZ5QZPKQUSBDU2VAU3NR4LMLI6NELCNPOQY7BKX4QDKEQ, seg_pos={0 0}, upload=49953]: Uploading piece... Aug 1 21:56:17.742734120 (51h59m7.680392619s) [noise=true , address=77.90.31.12:4590 , piece_id=FJD6CZBMQ3DU4ZAGIYOYUWRR6KMGNFNGXDNO55MGHA55MGYGGUCA, seg_pos={0 0}, upload=49953]: Uploading piece... Aug 1 21:56:17.742927009 (51h59m7.680585508s) [noise=true , address=77.90.31.11:4554 , piece_id=Q2W634RB76ZPR3D2PJZUG45FXMS7FCFN4IF745NEFQW44RU5RUFA, seg_pos={0 0}, upload=49953]: Uploading piece... Aug 1 21:56:17.743122072 (51h59m7.680780571s) [noise=true , address=104.36.82.4:4802 , piece_id=NESKRXISSKAEDLXBMTHYDZ2OVUJUGCYZ6JVG2ZHBTKIO4SZSZDMA, seg_pos={0 0}, upload=49953]: Uploading piece... Aug 1 21:56:17.743324500 (51h59m7.680982999s) [noise=true , address=104.36.82.7:4910 , piece_id=SH2RMPWPAW24CX6MHIBJC5Z7S2DMPTC6DKJVXKGCIZNGZJQNV5NQ, seg_pos={0 0}, upload=49953]: Uploading piece... Aug 1 21:56:17.743537677 (51h59m7.681196166s) [noise=true , address=77.90.31.14:4661 , piece_id=L3JNMTKXW7A4JLPIUNPKS7V76UDQUQO2IHAWCHUF2AYLPPBTIXEQ, seg_pos={0 0}, upload=49953]: Uploading piece... Aug 1 21:56:19.449135562 (51h59m9.386794061s) [upload=49953]: Next returned nil segment Aug 1 21:56:19.449151993 (51h59m9.386810512s) [upload=49953]: Waiting for error group managing segments... Aug 1 21:56:19.759811709 (51h59m9.697470208s) [noise=true , address=77.90.31.12:4590 , piece_id=FJD6CZBMQ3DU4ZAGIYOYUWRR6KMGNFNGXDNO55MGHA55MGYGGUCA, seg_pos={0 0}, upload=49953]: Done uploading piece. err: <nil> Aug 1 21:56:19.764469559 (51h59m9.702128058s) [noise=true , address=77.90.31.15:4699 , piece_id=EQLOXQDBN57AR5BENQ6FAVCFES2PTRGUADAFZNHQKFCNSRF27GJQ, seg_pos={0 0}, upload=49953]: Done uploading piece. err: <nil> Aug 1 21:56:19.769315670 (51h59m9.706974169s) [noise=true , address=45.91.116.10:4083 , piece_id=572YKDEMWDBADDOY3WGN6CECJGER5JECQWZXWHOQKKNFNQTIVQDQ, seg_pos={0 0}, upload=49953]: Done uploading piece. err: <nil> Aug 1 21:56:19.770933436 (51h59m9.708591935s) [noise=true , address=45.91.116.13:4154 , piece_id=4F6HX6YABGLIDHO7H6MWCYQ57533PJ2YGQFQJMBXJYOS4Z4XY5SA, seg_pos={0 0}, upload=49953]: Done uploading piece. err: <nil> Aug 1 21:56:19.780547048 (51h59m9.718205547s) [noise=true , address=77.90.31.11:4554 , piece_id=Q2W634RB76ZPR3D2PJZUG45FXMS7FCFN4IF745NEFQW44RU5RUFA, seg_pos={0 0}, upload=49953]: Done uploading piece. err: <nil> Aug 1 21:56:19.847756306 (51h59m9.785414805s) [noise=true , address=45.91.116.11:4119 , piece_id=Y7EG3JIEFXBT5I4VOB7YJBCBOCBDK4P2MCOVSGU7LO6XXNYD42AQ, seg_pos={0 0}, upload=49953]: Done uploading piece. err: <nil> Aug 1 21:56:19.852604211 (51h59m9.79026271s) [noise=true , address=45.91.116.14:4191 , piece_id=AWA66MJC4C6UK2323UFYG3P7LRENFWTE3I6EGFWS254NLTBT256A, seg_pos={0 0}, upload=49953]: Done uploading piece. err: <nil> Aug 1 21:56:19.858421450 (51h59m9.796079949s) [noise=true , address=45.91.116.4:4940 , piece_id=7TAVU47XZ5QZPKQUSBDU2VAU3NR4LMLI6NELCNPOQY7BKX4QDKEQ, seg_pos={0 0}, upload=49953]: Done uploading piece. err: <nil> Aug 1 21:56:19.858590934 (51h59m9.796249433s) [noise=true , address=77.90.31.21:4877 , piece_id=MEPTGIEH6XWK2STUCZFU6CO524P4VGX4GMCTYFTWVBP3MPSHNY3A, seg_pos={0 0}, upload=49953]: Done uploading piece. err: <nil> Aug 1 21:56:19.910467720 (51h59m9.848126219s) [noise=true , address=77.90.31.19:4842 , piece_id=GVTBKMNYSMPY5IF2O2MOUYHIUPK4P5V4KIYSIWBXBXBVA34QOM3A, seg_pos={0 0}, upload=49953]: Done uploading piece. err: <nil> Aug 1 21:56:19.926072253 (51h59m9.863730742s) [noise=true , address=77.90.31.14:4661 , piece_id=L3JNMTKXW7A4JLPIUNPKS7V76UDQUQO2IHAWCHUF2AYLPPBTIXEQ, seg_pos={0 0}, upload=49953]: Done uploading piece. err: <nil> Aug 1 21:56:19.942164790 (51h59m9.879823279s) [noise=true , address=45.91.116.5:4299 , piece_id=3SB6BSR5AK3KTJIK2AWKVU47EZEPRS42LGP7TX4F46HN5NQ767FA, seg_pos={0 0}, upload=49953]: Done uploading piece. err: <nil> Aug 1 21:56:46.858074017 (51h59m36.795732516s) [noise=true , address=104.36.82.8:4971 , piece_id=WK63NO3PBWOXPYRQWKNVHHWIZDF2LZQTYVPNMUUJOG3J6I42PS7A, seg_pos={0 0}, upload=49953]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:104.36.82.8:4971): protocol: rpc: dial tcp 104.36.82.8:4971: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:56:46.858123221 (51h59m36.79578172s) [noise=true , address=104.36.82.4:4802 , piece_id=NESKRXISSKAEDLXBMTHYDZ2OVUJUGCYZ6JVG2ZHBTKIO4SZSZDMA, seg_pos={0 0}, upload=49953]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: write tcp 188.166.225.170:54466->104.36.82.4:4802: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:56:46.858152237 (51h59m36.795810726s) [noise=true , address=104.36.82.2:4731 , piece_id=SLLCYPXHYJU2UWVEYA7ATJLBAIFBDDBHCN2EJEO2HOO67C4DOXJA, seg_pos={0 0}, upload=49953]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:104.36.82.2:4731): protocol: write tcp 188.166.225.170:53948->104.36.82.2:4731: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:56:46.858245054 (51h59m36.795903553s) [noise=true , address=104.36.82.7:4910 , piece_id=SH2RMPWPAW24CX6MHIBJC5Z7S2DMPTC6DKJVXKGCIZNGZJQNV5NQ, seg_pos={0 0}, upload=49953]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: rpc: dial tcp 104.36.82.7:4910: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:56:46.858280922 (51h59m36.795939421s) [seg_pos={0 0}, upload=49953]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Aug 1 21:56:47.084606176 (51h59m37.022264675s) [upload=49953]: Done uploading segments... Aug 1 21:56:47.084631495 (51h59m37.022289994s) [upload=49953]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.228283558s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 30.001406267s FAIL (2025-08-01 21:56:42 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.00143388s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [.........................................................XXX] duration: 1m0.232193164s FAIL (2025-08-01 21:56:36 UTC) orbiter warning: Operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 4) ======LOGS====== 24443 (51h58m27.791782942s) [noise=true , address=104.36.82.5:4857 , piece_id=TO43PPRSKS7WE4VNWMOKHB4VJ2OLGWX3QS6ZUFEWLNQMW564355A, seg_pos={0 0}, upload=49939]: Uploading piece... Aug 1 21:55:37.854408256 (51h58m27.792066755s) [noise=true , address=104.36.82.53:4135 , piece_id=E37VS4ROR6Y7SXXOJAHVJQKDO6ZRJLUAT4W22L2LLPFNJ2T7NGYA, seg_pos={0 0}, upload=49939]: Uploading piece... Aug 1 21:55:37.854720404 (51h58m27.792378893s) [noise=true , address=104.36.82.55:4223 , piece_id=QYWA2LTTYXB3BQ3YS4QCN2V3GNPU3M25CGO66CKWUZVIELWQ3IMQ, seg_pos={0 0}, upload=49939]: Uploading piece... Aug 1 21:55:37.855083067 (51h58m27.792741566s) [noise=true , address=104.36.82.4:4820 , piece_id=WZWXDLSRL3A5Q3NQ7RL46A3MJCJT67UFEJVEKUPIH3S4YT44DCUQ, seg_pos={0 0}, upload=49939]: Uploading piece... Aug 1 21:55:37.855150826 (51h58m27.792809325s) [noise=true , address=104.36.82.54:4184 , piece_id=WJXCJO677FI3R7DXJTM7ZSR7B3BV5HFFJHMXNZ6OVAKMKIGP5UMQ, seg_pos={0 0}, upload=49939]: Uploading piece... Aug 1 21:55:37.855406306 (51h58m27.793064805s) [noise=true , address=104.36.82.58:4278 , piece_id=7PPKOT74X7P7DLAD6KCTETYRFQJF6DUPN3QQRF53MBEV2XPEIROQ, seg_pos={0 0}, upload=49939]: Uploading piece... Aug 1 21:55:37.855711129 (51h58m27.793369618s) [noise=true , address=104.36.82.60:4365 , piece_id=B4WFJGL6QGZXCB6JHONUZY3YKQRVMGS5N3HBTW35C576MT5CEN7Q, seg_pos={0 0}, upload=49939]: Uploading piece... Aug 1 21:55:37.856014169 (51h58m27.793672668s) [noise=true , address=104.36.82.9:4021 , piece_id=6BCXAVMTNII4T3GVGPGFYEG3SVHNX4MHEO56XKZB7PZSHUG6S4GA, seg_pos={0 0}, upload=49939]: Uploading piece... Aug 1 21:55:37.856081678 (51h58m27.793740177s) [noise=true , address=104.36.82.12:4179 , piece_id=6EOT4UVJI2IKRVE4YF24W5LLJF2TF7WUUAOQFHRLQIN2KA7N4P6A, seg_pos={0 0}, upload=49939]: Uploading piece... Aug 1 21:55:37.856644083 (51h58m27.794302582s) [noise=true , address=104.36.82.44:4902 , piece_id=7YESGBHQ7O3IO4BENMQAE36SMYHBVSL6YOHT46XDUKBLIHRDZN2Q, seg_pos={0 0}, upload=49939]: Uploading piece... Aug 1 21:55:37.856856941 (51h58m27.79451544s) [noise=true , address=104.36.82.16:4317 , piece_id=IUNNVLIGKEXBAOF3AORBGHGNJAYQEPUCN7DQFFE5PU4EOZBUCSHA, seg_pos={0 0}, upload=49939]: Uploading piece... Aug 1 21:55:38.938850363 (51h58m28.876508862s) [upload=49939]: Next returned nil segment Aug 1 21:55:38.938862617 (51h58m28.876521106s) [upload=49939]: Waiting for error group managing segments... Aug 1 21:55:39.255192139 (51h58m29.192850628s) [noise=true , address=79.127.205.245:4243 , piece_id=LAPNLSGYCJA3B6VRVMMYM5H46GTUFDMUSP55RYGSBHOF3XHRM6YQ, seg_pos={0 0}, upload=49939]: Done uploading piece. err: <nil> Aug 1 21:55:39.273260756 (51h58m29.210919245s) [noise=true , address=79.127.205.234:4596 , piece_id=WQ6G2NC4HPUK5TGQ73E3YW2ADEIYMVEXZIWVY6L34PHO7VCFPBVQ, seg_pos={0 0}, upload=49939]: Done uploading piece. err: <nil> Aug 1 21:55:39.556009305 (51h58m29.493667804s) [noise=true , address=79.127.220.97:4377 , piece_id=U3S3UY7LSKMCYSSSSTKCF2EKBCP5DJLDMN5LGYKPRE5KL4X6EZHA, seg_pos={0 0}, upload=49939]: Done uploading piece. err: <nil> Aug 1 21:55:39.784785758 (51h58m29.722444257s) [noise=true , address=79.127.201.215:4351 , piece_id=G7JZ2QEEILPBEZKSIKIEVDGTUOZP4KGGT7C3EZTLI5ALM52DVEBQ, seg_pos={0 0}, upload=49939]: Done uploading piece. err: <nil> Aug 1 21:56:36.753504550 (51h59m26.691163049s) [noise=true , address=104.36.82.58:4278 , piece_id=7PPKOT74X7P7DLAD6KCTETYRFQJF6DUPN3QQRF53MBEV2XPEIROQ, seg_pos={0 0}, upload=49939]: Done uploading piece. err: ecclient: upload failed (node:12A56U9nehknX9qwMC9j4h2AjfAvEGbjwFFt648E7haU3jusW76, address:104.36.82.58:4278): protocol: rpc: dial tcp 104.36.82.58:4278: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:56:36.753547313 (51h59m26.691205812s) [noise=true , address=104.36.82.44:4902 , piece_id=7YESGBHQ7O3IO4BENMQAE36SMYHBVSL6YOHT46XDUKBLIHRDZN2Q, seg_pos={0 0}, upload=49939]: Done uploading piece. err: ecclient: upload failed (node:1Be9T5kXUwLU3C6wBeKFV3Q7aZFbZDTNG1oaEft8V6rmySUcEE, address:104.36.82.44:4902): protocol: rpc: dial tcp 104.36.82.44:4902: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:56:36.753579324 (51h59m26.691237823s) [noise=true , address=104.36.82.51:4078 , piece_id=B5OKSGZMLH3PNJPTFN7DIBGMMUJZ5U6FYCMVZT7HJZZ6A55IONPQ, seg_pos={0 0}, upload=49939]: Done uploading piece. err: ecclient: upload failed (node:1tzgtenV6us5ssD7xWGpZmVUoWEfbFhBNewbK9irHtZPkyDtRd, address:104.36.82.51:4078): protocol: write tcp 188.166.225.170:60338->104.36.82.51:4078: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:56:36.753607798 (51h59m26.691266297s) [noise=true , address=104.36.82.60:4365 , piece_id=B4WFJGL6QGZXCB6JHONUZY3YKQRVMGS5N3HBTW35C576MT5CEN7Q, seg_pos={0 0}, upload=49939]: Done uploading piece. err: ecclient: upload failed (node:122EX4PNuiRTCG2gbgbaL5dAtD8s5SN7ZesShk7NqUTGCZ3gA2a, address:104.36.82.60:4365): protocol: rpc: dial tcp 104.36.82.60:4365: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:56:36.753636744 (51h59m26.691295243s) [noise=true , address=104.36.82.10:4081 , piece_id=6UEEQLAJ46NZ5S6QRUFF4JOLL6XFP2PX2OBEJMPYWQFPY5VQ7LPA, seg_pos={0 0}, upload=49939]: Done uploading piece. err: ecclient: upload failed (node:12t9rGfCQ7bLkNnZzaPVgkE2Y9rXsiFp1XGdeECzv9M1jk8Qqmp, address:104.36.82.10:4081): protocol: rpc: dial tcp 104.36.82.10:4081: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:56:36.753664877 (51h59m26.691323376s) [noise=true , address=104.36.82.5:4857 , piece_id=TO43PPRSKS7WE4VNWMOKHB4VJ2OLGWX3QS6ZUFEWLNQMW564355A, seg_pos={0 0}, upload=49939]: Done uploading piece. err: ecclient: upload failed (node:1qjxNyfMA6ZjUurQZty2L8U5SUGo6wcLfCb3KVbmWn3ZaDXzQz, address:104.36.82.5:4857): protocol: rpc: dial tcp 104.36.82.5:4857: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:56:36.753690637 (51h59m26.691349136s) [noise=true , address=104.36.82.53:4135 , piece_id=E37VS4ROR6Y7SXXOJAHVJQKDO6ZRJLUAT4W22L2LLPFNJ2T7NGYA, seg_pos={0 0}, upload=49939]: Done uploading piece. err: ecclient: upload failed (node:128j2QoFq4UDWCXjV8uChmN9KB1sQMnhdF6hGgNQygdqHvw9tPj, address:104.36.82.53:4135): protocol: rpc: dial tcp 104.36.82.53:4135: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:56:36.753736906 (51h59m26.691395405s) [noise=true , address=104.36.82.60:4370 , piece_id=TXX223AJIZI7JDOLPKTY2XUTJQOQ3GE3J2WIJYYNNQOMD4PSNZFA, seg_pos={0 0}, upload=49939]: Done uploading piece. err: ecclient: upload failed (node:14LBgixGGgNeYs63ZSpkYJmrEGr3W9Ma1s2Sj8cCxU6mUxm1aC, address:104.36.82.60:4370): protocol: write tcp 188.166.225.170:57844->104.36.82.60:4370: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:56:36.753757364 (51h59m26.691415863s) [noise=true , address=104.36.82.55:4223 , piece_id=QYWA2LTTYXB3BQ3YS4QCN2V3GNPU3M25CGO66CKWUZVIELWQ3IMQ, seg_pos={0 0}, upload=49939]: Done uploading piece. err: ecclient: upload failed (node:1FijvRUiKwjk3A6yFZxNeLhPQ4bj5cxeEo6mjEs2DtRnK5WJr6, address:104.36.82.55:4223): protocol: rpc: dial tcp 104.36.82.55:4223: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:56:36.753770830 (51h59m26.691429329s) [noise=true , address=104.36.82.54:4184 , piece_id=WJXCJO677FI3R7DXJTM7ZSR7B3BV5HFFJHMXNZ6OVAKMKIGP5UMQ, seg_pos={0 0}, upload=49939]: Done uploading piece. err: ecclient: upload failed (node:1VYcRgiFADDQ1rLsiMufThN1K51cuoqKRXNjbp2w9Exu1d3gQG, address:104.36.82.54:4184): protocol: rpc: dial tcp 104.36.82.54:4184: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:56:36.753826897 (51h59m26.691485397s) [noise=true , address=104.36.82.9:4021 , piece_id=6BCXAVMTNII4T3GVGPGFYEG3SVHNX4MHEO56XKZB7PZSHUG6S4GA, seg_pos={0 0}, upload=49939]: Done uploading piece. err: ecclient: upload failed (node:12ts2cPQYUxdscD7RBpVxiHk36KAtJnXnAP9FwY7gCpBXHeS1Cg, address:104.36.82.9:4021): protocol: rpc: dial tcp 104.36.82.9:4021: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:56:36.753835714 (51h59m26.691494213s) [noise=true , address=104.36.82.12:4179 , piece_id=6EOT4UVJI2IKRVE4YF24W5LLJF2TF7WUUAOQFHRLQIN2KA7N4P6A, seg_pos={0 0}, upload=49939]: Done uploading piece. err: ecclient: upload failed (node:129pP9f3rFfTvP6MhG894N7WHuJRZP8aWhzJf1zBwZ76wbzVbZa, address:104.36.82.12:4179): protocol: rpc: dial tcp 104.36.82.12:4179: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:56:36.753845133 (51h59m26.691503632s) [noise=true , address=104.36.82.4:4820 , piece_id=WZWXDLSRL3A5Q3NQ7RL46A3MJCJT67UFEJVEKUPIH3S4YT44DCUQ, seg_pos={0 0}, upload=49939]: Done uploading piece. err: ecclient: upload failed (node:12TnUNGbwHSf335XW7vNuq1ffwhFkgxomckBQmF189779tRU59k, address:104.36.82.4:4820): protocol: rpc: dial tcp 104.36.82.4:4820: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:56:36.753858538 (51h59m26.691517027s) [noise=true , address=104.36.82.16:4317 , piece_id=IUNNVLIGKEXBAOF3AORBGHGNJAYQEPUCN7DQFFE5PU4EOZBUCSHA, seg_pos={0 0}, upload=49939]: Done uploading piece. err: ecclient: upload failed (node:1FdJoeW4vejiQx1QkQ1nCkApgQaW6fB66d2usAYVfFuK1PPmRP, address:104.36.82.16:4317): protocol: write tcp 188.166.225.170:32856->104.36.82.16:4317: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:56:36.753877915 (51h59m26.691536414s) [seg_pos={0 0}, upload=49939]: Done waiting for segment. successful: 4 optimal: 14 errs: <nil> Aug 1 21:56:36.984224293 (51h59m26.921882792s) [upload=49939]: Done uploading segments... Aug 1 21:56:36.984244802 (51h59m26.921903301s) [upload=49939]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 4) Duration: 1m0.2322073s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [.........................................................XXX] duration: 1m0.001711796s FAIL (2025-08-01 21:56:33 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.001732585s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 3 out of 60 is failed (threshold=3) [........................................X............X.....X] duration: 2.188364279s FAIL (2025-08-01 21:56:10 UTC) orbiter warning: Operation: ListObjects unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/1K Duration: 2.188392091s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 4 out of 60 is failed (threshold=3) [........................................................XXXX] duration: 30.230746717s FAIL (2025-08-01 21:56:07 UTC) orbiter warning: Operation: Commit uplink: failed to upload enough pieces (needed at least 14 but got 7) Duration: 30.230776525s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [.........................................X............X.....] duration: 1.823575178s FAIL (2025-08-01 21:55:47 UTC) orbiter warning: Operation: sync2.Copy digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/1K):: after 7b07c632440be552d064c57547a519a1464d9714c90d6f11ea4290210b73bd46 != before b944efe6550e3eab95f8314489d530afd89693d29ea288891d748fdb1ed0ffd1 Duration: 1.823608681s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 3 out of 60 is failed (threshold=3) [.........................................................XXX] duration: 30.229794039s FAIL (2025-08-01 21:55:37 UTC) orbiter warning: Operation: Commit uplink: failed to upload enough pieces (needed at least 14 but got 1) Duration: 30.229818256s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [..........................................................XX] duration: 1m0.230837417s FAIL (2025-08-01 21:55:36 UTC) orbiter warning: Operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 7) ======LOGS====== , piece_id=NQISD2P3JDLMISNZNKKMHWDE5J2XK6YROI76HP363EC5AX43O5WQ, seg_pos={0 0}, upload=49924]: Uploading piece... Aug 1 21:54:37.411454506 (51h57m27.349113005s) [noise=true , address=79.127.205.246:4257 , piece_id=AGVNBPLRAWXTMK3NB3UK446KYJVL5KJUDYVPQUG2EMZIX7HED24Q, seg_pos={0 0}, upload=49924]: Uploading piece... Aug 1 21:54:37.411738670 (51h57m27.349397169s) [noise=true , address=64.49.13.138:4487 , piece_id=OQV2GGHLMZNMUZLNXED2QFTAQ6UTX4RF2HLYLMDC6GMGFQIMU2DA, seg_pos={0 0}, upload=49924]: Uploading piece... Aug 1 21:54:37.412218778 (51h57m27.349877277s) [noise=true , address=104.36.82.44:4902 , piece_id=VNYKJ6BDHHVRE2IPLMPKNZH3NS5RI63LFYVL2BVPGQNIHPJRMO3Q, seg_pos={0 0}, upload=49924]: Uploading piece... Aug 1 21:54:37.412522029 (51h57m27.350180528s) [noise=true , address=104.36.82.11:4140 , piece_id=KVGT5NNH43L2DTKO3DD63SUSOWSDX2LMOUUA2UNODKNWYDT3DL6Q, seg_pos={0 0}, upload=49924]: Uploading piece... Aug 1 21:54:37.412799950 (51h57m27.350458449s) [noise=true , address=104.36.82.7:4920 , piece_id=S7TENDI6NAEYV7DBF5RSDIXP7443OILNUVD7NUTRMD5KQVNG42CQ, seg_pos={0 0}, upload=49924]: Uploading piece... Aug 1 21:54:37.413084023 (51h57m27.350742522s) [noise=true , address=79.127.220.98:4402 , piece_id=7HB6X2LMJD3D2LKLJNEI7ZL6AT4YRT7GDQEKS4YBIL4EBJHGVCSA, seg_pos={0 0}, upload=49924]: Uploading piece... Aug 1 21:54:37.413346847 (51h57m27.351005346s) [noise=true , address=104.36.82.46:4487 , piece_id=R4YFTZQVNQ4NVG2EK26UCUTLQFCXQXGO7YEDLYWZHWZZ36GHAEOA, seg_pos={0 0}, upload=49924]: Uploading piece... Aug 1 21:54:37.413623596 (51h57m27.351282095s) [noise=true , address=104.36.82.59:4329 , piece_id=XMWSQ4XIAHT7ZMCNGZCB2GEUPBS6SDCBTKGKMRP2ECGCPJRIAUPA, seg_pos={0 0}, upload=49924]: Uploading piece... Aug 1 21:54:37.413714159 (51h57m27.351372658s) [noise=true , address=79.127.219.51:4183 , piece_id=VVWXVLI63I3KI7QQMNC5PVAJ7BSTDTNY4BUT2ZTVDZT2HFYEE3LQ, seg_pos={0 0}, upload=49924]: Uploading piece... Aug 1 21:54:37.414021988 (51h57m27.351680487s) [noise=true , address=104.36.82.9:4027 , piece_id=DKLKAOHXKQWENZHTXO6IVYRKXJJJ4KUVNE5V7FEUA4IN7MUBVDKQ, seg_pos={0 0}, upload=49924]: Uploading piece... Aug 1 21:54:37.414258871 (51h57m27.351917371s) [noise=true , address=104.36.82.12:4159 , piece_id=J3MUQU4LXK2IQYFSXK64WPDTRZV6VIMWYJCAB2MU2TPTVSRR4G7Q, seg_pos={0 0}, upload=49924]: Uploading piece... Aug 1 21:54:37.414545229 (51h57m27.352203728s) [noise=true , address=104.36.82.16:4321 , piece_id=CGQVSDXDAFTSREI4YVERIBYNOFFRC2OLXK2JTPSX3S5VKJW3RD4Q, seg_pos={0 0}, upload=49924]: Uploading piece... Aug 1 21:54:37.414754269 (51h57m27.352412768s) [noise=true , address=104.36.82.60:4368 , piece_id=HX7NYMGYVUJ6RNEV4JHMLAZGCFH47DGI7WCUZYDQY33JYCERVR4A, seg_pos={0 0}, upload=49924]: Uploading piece... Aug 1 21:54:37.415011191 (51h57m27.35266969s) [noise=true , address=104.36.82.23:4480 , piece_id=CLQA4AABFCS557L7RPNYA4ICW4ESLHIMZELGIPPJ3H3AUEK2NO7A, seg_pos={0 0}, upload=49924]: Uploading piece... Aug 1 21:54:38.530398829 (51h57m28.468057328s) [upload=49924]: Next returned nil segment Aug 1 21:54:38.530416693 (51h57m28.468075182s) [upload=49924]: Waiting for error group managing segments... Aug 1 21:54:38.858479353 (51h57m28.796137852s) [noise=true , address=79.127.205.246:4257 , piece_id=AGVNBPLRAWXTMK3NB3UK446KYJVL5KJUDYVPQUG2EMZIX7HED24Q, seg_pos={0 0}, upload=49924]: Done uploading piece. err: <nil> Aug 1 21:54:38.914441033 (51h57m28.852099532s) [noise=true , address=64.49.13.138:4487 , piece_id=OQV2GGHLMZNMUZLNXED2QFTAQ6UTX4RF2HLYLMDC6GMGFQIMU2DA, seg_pos={0 0}, upload=49924]: Done uploading piece. err: <nil> Aug 1 21:54:39.180916249 (51h57m29.118574748s) [noise=true , address=79.127.220.98:4402 , piece_id=7HB6X2LMJD3D2LKLJNEI7ZL6AT4YRT7GDQEKS4YBIL4EBJHGVCSA, seg_pos={0 0}, upload=49924]: Done uploading piece. err: <nil> Aug 1 21:54:39.314570134 (51h57m29.252228623s) [noise=true , address=79.127.201.215:4351 , piece_id=TOYCB2XATZO47OQKULDA2ZHRLOKRCGUHCLQRNU54T7GOBREWGIHA, seg_pos={0 0}, upload=49924]: Done uploading piece. err: <nil> Aug 1 21:54:39.354898255 (51h57m29.292556754s) [noise=true , address=79.127.219.51:4183 , piece_id=VVWXVLI63I3KI7QQMNC5PVAJ7BSTDTNY4BUT2ZTVDZT2HFYEE3LQ, seg_pos={0 0}, upload=49924]: Done uploading piece. err: <nil> Aug 1 21:54:39.407393494 (51h57m29.345051993s) [noise=true , address=77.90.31.14:4685 , piece_id=MN22AIVFB4TWMNSJZF7ANY6VG577WWH34FZOOIMIKEERO3NK6O3A, seg_pos={0 0}, upload=49924]: Done uploading piece. err: <nil> Aug 1 21:54:39.441986854 (51h57m29.379645353s) [noise=true , address=45.91.116.7:4026 , piece_id=NQISD2P3JDLMISNZNKKMHWDE5J2XK6YROI76HP363EC5AX43O5WQ, seg_pos={0 0}, upload=49924]: Done uploading piece. err: <nil> Aug 1 21:55:36.314105310 (51h58m26.251763809s) [noise=true , address=104.36.82.60:4368 , piece_id=HX7NYMGYVUJ6RNEV4JHMLAZGCFH47DGI7WCUZYDQY33JYCERVR4A, seg_pos={0 0}, upload=49924]: Done uploading piece. err: ecclient: upload failed (node:12Q6suyRuFmW1rG5qq8faCFbCkfAPaNfYQaBv6qhL1DNavUffkB, address:104.36.82.60:4368): protocol: rpc: dial tcp 104.36.82.60:4368: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:55:36.314168280 (51h58m26.251826779s) [noise=true , address=104.36.82.23:4480 , piece_id=CLQA4AABFCS557L7RPNYA4ICW4ESLHIMZELGIPPJ3H3AUEK2NO7A, seg_pos={0 0}, upload=49924]: Done uploading piece. err: ecclient: upload failed (node:1qQhWAdN83i1jMTYByNQ8SWGXQaXRgBGezkLTf1BTwLyKskjjU, address:104.36.82.23:4480): protocol: rpc: dial tcp 104.36.82.23:4480: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:55:36.314204910 (51h58m26.251863399s) [noise=true , address=104.36.82.54:4180 , piece_id=J3GP4MJZ5GMSV7JMYIJBW5QQHKRWUP7BX4YDR7KPCL6GN3UXLBBQ, seg_pos={0 0}, upload=49924]: Done uploading piece. err: ecclient: upload failed (node:1rhkcipPBZNFScBjBhWdigzSMwENSmzhCucnC2iuyc9udpweE3, address:104.36.82.54:4180): protocol: rpc: dial tcp 104.36.82.54:4180: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:55:36.314250748 (51h58m26.251909247s) [noise=true , address=104.36.82.11:4140 , piece_id=KVGT5NNH43L2DTKO3DD63SUSOWSDX2LMOUUA2UNODKNWYDT3DL6Q, seg_pos={0 0}, upload=49924]: Done uploading piece. err: ecclient: upload failed (node:12AYQRkKcRprBgX1BdUwnd4MCQ1swYucsG9aYvLLEaskgrngkCB, address:104.36.82.11:4140): protocol: rpc: dial tcp 104.36.82.11:4140: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:55:36.314284062 (51h58m26.251942561s) [noise=true , address=104.36.82.46:4487 , piece_id=R4YFTZQVNQ4NVG2EK26UCUTLQFCXQXGO7YEDLYWZHWZZ36GHAEOA, seg_pos={0 0}, upload=49924]: Done uploading piece. err: ecclient: upload failed (node:1AdZLDV7UBbV2DimCHzmoNxycLigvLMx6Fy9U9QB35WTtiV1KX, address:104.36.82.46:4487): protocol: rpc: dial tcp 104.36.82.46:4487: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:55:36.314342964 (51h58m26.252001463s) [noise=true , address=104.36.82.7:4920 , piece_id=S7TENDI6NAEYV7DBF5RSDIXP7443OILNUVD7NUTRMD5KQVNG42CQ, seg_pos={0 0}, upload=49924]: Done uploading piece. err: ecclient: upload failed (node:1vBccAgciQwbLaMMe9QKfYM3zxYTHqG3pfYQ7nh8eETr8JUskb, address:104.36.82.7:4920): protocol: rpc: dial tcp 104.36.82.7:4920: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:55:36.314375476 (51h58m26.252033965s) [noise=true , address=104.36.82.59:4329 , piece_id=XMWSQ4XIAHT7ZMCNGZCB2GEUPBS6SDCBTKGKMRP2ECGCPJRIAUPA, seg_pos={0 0}, upload=49924]: Done uploading piece. err: ecclient: upload failed (node:1dGY6BG4FULjuRwnEKd9wV4cZPVJsgo32oFHKZn5BgjxVaojbk, address:104.36.82.59:4329): protocol: rpc: dial tcp 104.36.82.59:4329: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:55:36.314427386 (51h58m26.252085945s) [noise=true , address=104.36.82.9:4027 , piece_id=DKLKAOHXKQWENZHTXO6IVYRKXJJJ4KUVNE5V7FEUA4IN7MUBVDKQ, seg_pos={0 0}, upload=49924]: Done uploading piece. err: ecclient: upload failed (node:1SWk13XvkrCYwKcdZ46SK2THQvhsD6pf4hWJKgARztB9qwfMZU, address:104.36.82.9:4027): protocol: rpc: dial tcp 104.36.82.9:4027: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:55:36.314456592 (51h58m26.252115091s) [noise=true , address=104.36.82.44:4902 , piece_id=VNYKJ6BDHHVRE2IPLMPKNZH3NS5RI63LFYVL2BVPGQNIHPJRMO3Q, seg_pos={0 0}, upload=49924]: Done uploading piece. err: ecclient: upload failed (node:1Be9T5kXUwLU3C6wBeKFV3Q7aZFbZDTNG1oaEft8V6rmySUcEE, address:104.36.82.44:4902): protocol: rpc: dial tcp 104.36.82.44:4902: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:55:36.314534330 (51h58m26.25219283s) [noise=true , address=104.36.82.12:4159 , piece_id=J3MUQU4LXK2IQYFSXK64WPDTRZV6VIMWYJCAB2MU2TPTVSRR4G7Q, seg_pos={0 0}, upload=49924]: Done uploading piece. err: ecclient: upload failed (node:1taQkf86SDCwKJaXVcfu1YcSNz8SfpAtChruvcZVFfy55qZdtX, address:104.36.82.12:4159): protocol: rpc: dial tcp 104.36.82.12:4159: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:55:36.314550171 (51h58m26.25220868s) [noise=true , address=104.36.82.16:4321 , piece_id=CGQVSDXDAFTSREI4YVERIBYNOFFRC2OLXK2JTPSX3S5VKJW3RD4Q, seg_pos={0 0}, upload=49924]: Done uploading piece. err: ecclient: upload failed (node:12edsHqD9iRBDgMU8vYhSM1EWG8Sn57GZ8TPiTytri3rp7W9Eae, address:104.36.82.16:4321): protocol: write tcp 188.166.225.170:37028->104.36.82.16:4321: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:55:36.314584406 (51h58m26.252242906s) [seg_pos={0 0}, upload=49924]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil> Aug 1 21:55:36.543350319 (51h58m26.481008819s) [upload=49924]: Done uploading segments... Aug 1 21:55:36.543374376 (51h58m26.481032875s) [upload=49924]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7) Duration: 1m0.230853358s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [..........................................X............X....] duration: 1.836637345s FAIL (2025-08-01 21:55:29 UTC) orbiter warning: Operation: sync2.Copy digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/1K):: after 7b07c632440be552d064c57547a519a1464d9714c90d6f11ea4290210b73bd46 != before b944efe6550e3eab95f8314489d530afd89693d29ea288891d748fdb1ed0ffd1 Duration: 1.836694203s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........................................................XX.] duration: 6.110564941s FAIL (2025-08-01 21:55:22 UTC) orbiter warning: Operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 007): protocol: rpc: dial tcp 104.36.82.9:4007: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:53:47.873667171 (51h56m37.811325671s) [noise=true , address=104.36.82.2:4730 , piece_id=LWUE3RLN3V7OB6MCA6DQKN4JIXSY4HXY7REESWOGPY2U4PGUIYJA, seg_pos={0 0}, upload=49906]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:104.36.82.2:4730): protocol: write tcp 188.166.225.170:41370->104.36.82.2:4730: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:53:47.873679725 (51h56m37.811338224s) [seg_pos={0 0}, upload=49906]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Aug 1 21:53:48.106630115 (51h56m38.044288604s) [upload=49906]: Done uploading segments... Aug 1 21:53:48.106654551 (51h56m38.04431305s) [upload=49906]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Aug 1 21:54:21.582824425 (51h57m11.520482924s) [upload=49920]: Starting upload Aug 1 21:54:21.582881163 (51h57m11.520539652s) [upload=49920]: Done starting upload Aug 1 21:54:21.583036930 (51h57m11.52069543s) [upload=49920]: Uploading segments... Aug 1 21:54:21.583068862 (51h57m11.520727361s) [upload=49920]: Got next segment. Inline: false Aug 1 21:54:21.583094451 (51h57m11.520752951s) [upload=49920]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 1 21:54:21.583098469 (51h57m11.520756968s) [upload=49920]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 1 21:54:21.819679043 (51h57m11.757337542s) [seg_pos={0 0}, upload=49920]: Begin upload segment... Aug 1 21:54:21.819722976 (51h57m11.757381475s) [seg_pos={0 0}, upload=49920]: Done begin upload segment. Aug 1 21:54:21.819745309 (51h57m11.757403808s) [noise=true , address=104.36.82.3:4765 , piece_id=5JWL3TF4SXMGRDI4YWLVKRYIPVNKMG7OATF7GJGR2ILCSUJ3373Q, seg_pos={0 0}, upload=49920]: Uploading piece... Aug 1 21:54:21.819805614 (51h57m11.757464113s) [noise=true , address=77.90.31.13:4628 , piece_id=EJXU6EFBSFHKTLKPYHZDY4UNNJNKDQXQBDHGNARDSPZA23YBWHGQ, seg_pos={0 0}, upload=49920]: Uploading piece... Aug 1 21:54:21.820108003 (51h57m11.757766502s) [noise=true , address=77.90.31.19:4841 , piece_id=IPVDNLM5DJ4DAP2MYOD2FGFILZCW7TVT7MTXHRLOVOKCKPJCV7EA, seg_pos={0 0}, upload=49920]: Uploading piece... Aug 1 21:54:21.820431432 (51h57m11.758089931s) [noise=true , address=45.91.116.10:4081 , piece_id=XQ7ZZC6HKVZDEWKBUJWDBYKO2LHVG2MWXE5HRHDQNCDNY36GYLTA, seg_pos={0 0}, upload=49920]: Uploading piece... Aug 1 21:54:21.820441350 (51h57m11.758099849s) [noise=true , address=45.91.116.6:4973 , piece_id=NF2Y5SV7Q3S3MTFSI3DZA4JNRPOM7BYKKCE33D2YYNWI34IC7L3A, seg_pos={0 0}, upload=49920]: Uploading piece... Aug 1 21:54:21.820791661 (51h57m11.75845015s) [noise=true , address=104.36.82.4:4802 , piece_id=N4XRJIP6SE4J5H2QFLESLGAXRU7F7KIFAEVPKNXU4CRYFMODFRYQ, seg_pos={0 0}, upload=49920]: Uploading piece... Aug 1 21:54:21.820800017 (51h57m11.758458506s) [noise=true , address=104.36.82.9:4006 , piece_id=ECQRTBZP4T5NFT2GER5Y4RT2YND7HEAWX5NUI6XG6K5WGVCL3IWQ, seg_pos={0 0}, upload=49920]: Uploading piece... Aug 1 21:54:21.821095071 (51h57m11.75875357s) [noise=true , address=77.90.31.10:4519 , piece_id=ZAWNIOP3QVGFIN4EZ6BHC2FGO527NYSWHBTMKZWDVSRSBDUGUKUA, seg_pos={0 0}, upload=49920]: Uploading piece... Aug 1 21:54:21.821103337 (51h57m11.758761836s) [noise=true , address=104.36.82.5:4837 , piece_id=OVKX6V5TNG5B7BL6DU7H3ALXVIW4UHRTXI4AQSHEO6WIGPJNTEVQ, seg_pos={0 0}, upload=49920]: Uploading piece... Aug 1 21:54:21.821337484 (51h57m11.758995983s) [noise=true , address=45.91.116.12:4226 , piece_id=VI2Y5I64SLS6SWYBC32HF3QLC5JMACOFLUQW7B7C3QFJT7Y3Z6PQ, seg_pos={0 0}, upload=49920]: Uploading piece... Aug 1 21:54:21.821400215 (51h57m11.759058714s) [noise=true , address=77.90.31.19:4842 , piece_id=5LEMN3QJTN5N5HEIJYIFCIAERN63DSDJN7GCYGQJQJ7ACM7SDQLA, seg_pos={0 0}, upload=49920]: Uploading piece... Aug 1 21:54:21.821614334 (51h57m11.759272823s) [noise=true , address=45.91.116.9:4262 , piece_id=PHBAXJ65HH223XRTVRPGS3SL4XB4XV3JCRQGS5XLCIGBZYKTEVHQ, seg_pos={0 0}, upload=49920]: Uploading piece... Aug 1 21:54:21.821679149 (51h57m11.759337648s) [noise=true , address=45.91.116.4:4938 , piece_id=YLILN4A5BP32B7DRNGCHJIG3CZPMCFVSGAQFVLWNKEV3FTPXMT6Q, seg_pos={0 0}, upload=49920]: Uploading piece... Aug 1 21:54:21.821864212 (51h57m11.759522711s) [noise=true , address=45.91.116.10:4083 , piece_id=FN7XQNW6RF45GFJCWGYMG6TU37TN3UOFI2CR2FYL3XJDSOIACBKQ, seg_pos={0 0}, upload=49920]: Uploading piece... Aug 1 21:54:21.822884644 (51h57m11.760543143s) [noise=true , address=45.91.116.11:4119 , piece_id=S5C7ZF4PDNNAOYWC3XL3SNXIV4YSRKOCAC6L3N3PHDGA2TVFKMEA, seg_pos={0 0}, upload=49920]: Uploading piece... Aug 1 21:54:21.823100417 (51h57m11.760758906s) [noise=true , address=77.90.31.12:4589 , piece_id=Z6NQQBOBSJRHY5GC7BWHDM35GMPMLBYKRE6DN7VSR7WNCL3KRVNQ, seg_pos={0 0}, upload=49920]: Uploading piece... Aug 1 21:54:23.564683254 (51h57m13.502341753s) [upload=49920]: Next returned nil segment Aug 1 21:54:23.564700097 (51h57m13.502358586s) [upload=49920]: Waiting for error group managing segments... Aug 1 21:54:23.843501825 (51h57m13.781160334s) [noise=true , address=77.90.31.13:4628 , piece_id=EJXU6EFBSFHKTLKPYHZDY4UNNJNKDQXQBDHGNARDSPZA23YBWHGQ, seg_pos={0 0}, upload=49920]: Done uploading piece. err: <nil> Aug 1 21:54:23.845236794 (51h57m13.782895303s) [noise=true , address=77.90.31.10:4519 , piece_id=ZAWNIOP3QVGFIN4EZ6BHC2FGO527NYSWHBTMKZWDVSRSBDUGUKUA, seg_pos={0 0}, upload=49920]: Done uploading piece. err: <nil> Aug 1 21:54:23.851919939 (51h57m13.789578428s) [noise=true , address=45.91.116.10:4081 , piece_id=XQ7ZZC6HKVZDEWKBUJWDBYKO2LHVG2MWXE5HRHDQNCDNY36GYLTA, seg_pos={0 0}, upload=49920]: Done uploading piece. err: <nil> Aug 1 21:54:23.863342793 (51h57m13.801001292s) [noise=true , address=77.90.31.19:4841 , piece_id=IPVDNLM5DJ4DAP2MYOD2FGFILZCW7TVT7MTXHRLOVOKCKPJCV7EA, seg_pos={0 0}, upload=49920]: Done uploading piece. err: <nil> Aug 1 21:54:23.921412726 (51h57m13.859071225s) [noise=true , address=45.91.116.12:4226 , piece_id=VI2Y5I64SLS6SWYBC32HF3QLC5JMACOFLUQW7B7C3QFJT7Y3Z6PQ, seg_pos={0 0}, upload=49920]: Done uploading piece. err: <nil> Aug 1 21:54:23.922233376 (51h57m13.859891875s) [noise=true , address=45.91.116.9:4262 , piece_id=PHBAXJ65HH223XRTVRPGS3SL4XB4XV3JCRQGS5XLCIGBZYKTEVHQ, seg_pos={0 0}, upload=49920]: Done uploading piece. err: <nil> Aug 1 21:54:23.926507283 (51h57m13.864165782s) [noise=true , address=77.90.31.19:4842 , piece_id=5LEMN3QJTN5N5HEIJYIFCIAERN63DSDJN7GCYGQJQJ7ACM7SDQLA, seg_pos={0 0}, upload=49920]: Done uploading piece. err: <nil> Aug 1 21:54:23.929820481 (51h57m13.86747898s) [noise=true , address=45.91.116.10:4083 , piece_id=FN7XQNW6RF45GFJCWGYMG6TU37TN3UOFI2CR2FYL3XJDSOIACBKQ, seg_pos={0 0}, upload=49920]: Done uploading piece. err: <nil> Aug 1 21:54:23.935253295 (51h57m13.872911794s) [noise=true , address=45.91.116.6:4973 , piece_id=NF2Y5SV7Q3S3MTFSI3DZA4JNRPOM7BYKKCE33D2YYNWI34IC7L3A, seg_pos={0 0}, upload=49920]: Done uploading piece. err: <nil> Aug 1 21:54:23.936465073 (51h57m13.874123572s) [noise=true , address=45.91.116.11:4119 , piece_id=S5C7ZF4PDNNAOYWC3XL3SNXIV4YSRKOCAC6L3N3PHDGA2TVFKMEA, seg_pos={0 0}, upload=49920]: Done uploading piece. err: <nil> Aug 1 21:54:24.008511355 (51h57m13.946169854s) [noise=true , address=45.91.116.4:4938 , piece_id=YLILN4A5BP32B7DRNGCHJIG3CZPMCFVSGAQFVLWNKEV3FTPXMT6Q, seg_pos={0 0}, upload=49920]: Done uploading piece. err: <nil> Aug 1 21:54:24.163933144 (51h57m14.101591643s) [noise=true , address=77.90.31.12:4589 , piece_id=Z6NQQBOBSJRHY5GC7BWHDM35GMPMLBYKRE6DN7VSR7WNCL3KRVNQ, seg_pos={0 0}, upload=49920]: Done uploading piece. err: <nil> Aug 1 21:54:50.931068973 (51h57m40.868727462s) [noise=true , address=104.36.82.4:4802 , piece_id=N4XRJIP6SE4J5H2QFLESLGAXRU7F7KIFAEVPKNXU4CRYFMODFRYQ, seg_pos={0 0}, upload=49920]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: write tcp 188.166.225.170:47794->104.36.82.4:4802: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:54:50.931082940 (51h57m40.868741449s) [noise=true , address=104.36.82.3:4765 , piece_id=5JWL3TF4SXMGRDI4YWLVKRYIPVNKMG7OATF7GJGR2ILCSUJ3373Q, seg_pos={0 0}, upload=49920]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:104.36.82.3:4765): protocol: context deadline exceeded; context deadline exceeded Aug 1 21:54:50.931120421 (51h57m40.86877892s) [noise=true , address=104.36.82.9:4006 , piece_id=ECQRTBZP4T5NFT2GER5Y4RT2YND7HEAWX5NUI6XG6K5WGVCL3IWQ, seg_pos={0 0}, upload=49920]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: write tcp 188.166.225.170:48514->104.36.82.9:4006: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:54:50.931183922 (51h57m40.868842421s) [noise=true , address=104.36.82.5:4837 , piece_id=OVKX6V5TNG5B7BL6DU7H3ALXVIW4UHRTXI4AQSHEO6WIGPJNTEVQ, seg_pos={0 0}, upload=49920]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:104.36.82.5:4837): protocol: rpc: dial tcp 104.36.82.5:4837: i/o timeout; conn closed: op Aug 1 21:54:50.931200685 (51h57m40.868859184s) [seg_pos={0 0}, upload=49920]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Aug 1 21:54:51.166648722 (51h57m41.104307221s) [upload=49920]: Done uploading segments... Aug 1 21:54:51.166672908 (51h57m41.104331407s) [upload=49920]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.110590491s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [..........................................................XX] duration: 1m0.00157596s FAIL (2025-08-01 21:55:20 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.001586991s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........................................................XX] duration: 30.237224352s FAIL (2025-08-01 21:54:51 UTC) orbiter warning: Operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 007): protocol: rpc: dial tcp 104.36.82.9:4007: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:53:47.873667171 (51h56m37.811325671s) [noise=true , address=104.36.82.2:4730 , piece_id=LWUE3RLN3V7OB6MCA6DQKN4JIXSY4HXY7REESWOGPY2U4PGUIYJA, seg_pos={0 0}, upload=49906]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:104.36.82.2:4730): protocol: write tcp 188.166.225.170:41370->104.36.82.2:4730: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:53:47.873679725 (51h56m37.811338224s) [seg_pos={0 0}, upload=49906]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Aug 1 21:53:48.106630115 (51h56m38.044288604s) [upload=49906]: Done uploading segments... Aug 1 21:53:48.106654551 (51h56m38.04431305s) [upload=49906]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Aug 1 21:54:21.582824425 (51h57m11.520482924s) [upload=49920]: Starting upload Aug 1 21:54:21.582881163 (51h57m11.520539652s) [upload=49920]: Done starting upload Aug 1 21:54:21.583036930 (51h57m11.52069543s) [upload=49920]: Uploading segments... Aug 1 21:54:21.583068862 (51h57m11.520727361s) [upload=49920]: Got next segment. Inline: false Aug 1 21:54:21.583094451 (51h57m11.520752951s) [upload=49920]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 1 21:54:21.583098469 (51h57m11.520756968s) [upload=49920]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 1 21:54:21.819679043 (51h57m11.757337542s) [seg_pos={0 0}, upload=49920]: Begin upload segment... Aug 1 21:54:21.819722976 (51h57m11.757381475s) [seg_pos={0 0}, upload=49920]: Done begin upload segment. Aug 1 21:54:21.819745309 (51h57m11.757403808s) [noise=true , address=104.36.82.3:4765 , piece_id=5JWL3TF4SXMGRDI4YWLVKRYIPVNKMG7OATF7GJGR2ILCSUJ3373Q, seg_pos={0 0}, upload=49920]: Uploading piece... Aug 1 21:54:21.819805614 (51h57m11.757464113s) [noise=true , address=77.90.31.13:4628 , piece_id=EJXU6EFBSFHKTLKPYHZDY4UNNJNKDQXQBDHGNARDSPZA23YBWHGQ, seg_pos={0 0}, upload=49920]: Uploading piece... Aug 1 21:54:21.820108003 (51h57m11.757766502s) [noise=true , address=77.90.31.19:4841 , piece_id=IPVDNLM5DJ4DAP2MYOD2FGFILZCW7TVT7MTXHRLOVOKCKPJCV7EA, seg_pos={0 0}, upload=49920]: Uploading piece... Aug 1 21:54:21.820431432 (51h57m11.758089931s) [noise=true , address=45.91.116.10:4081 , piece_id=XQ7ZZC6HKVZDEWKBUJWDBYKO2LHVG2MWXE5HRHDQNCDNY36GYLTA, seg_pos={0 0}, upload=49920]: Uploading piece... Aug 1 21:54:21.820441350 (51h57m11.758099849s) [noise=true , address=45.91.116.6:4973 , piece_id=NF2Y5SV7Q3S3MTFSI3DZA4JNRPOM7BYKKCE33D2YYNWI34IC7L3A, seg_pos={0 0}, upload=49920]: Uploading piece... Aug 1 21:54:21.820791661 (51h57m11.75845015s) [noise=true , address=104.36.82.4:4802 , piece_id=N4XRJIP6SE4J5H2QFLESLGAXRU7F7KIFAEVPKNXU4CRYFMODFRYQ, seg_pos={0 0}, upload=49920]: Uploading piece... Aug 1 21:54:21.820800017 (51h57m11.758458506s) [noise=true , address=104.36.82.9:4006 , piece_id=ECQRTBZP4T5NFT2GER5Y4RT2YND7HEAWX5NUI6XG6K5WGVCL3IWQ, seg_pos={0 0}, upload=49920]: Uploading piece... Aug 1 21:54:21.821095071 (51h57m11.75875357s) [noise=true , address=77.90.31.10:4519 , piece_id=ZAWNIOP3QVGFIN4EZ6BHC2FGO527NYSWHBTMKZWDVSRSBDUGUKUA, seg_pos={0 0}, upload=49920]: Uploading piece... Aug 1 21:54:21.821103337 (51h57m11.758761836s) [noise=true , address=104.36.82.5:4837 , piece_id=OVKX6V5TNG5B7BL6DU7H3ALXVIW4UHRTXI4AQSHEO6WIGPJNTEVQ, seg_pos={0 0}, upload=49920]: Uploading piece... Aug 1 21:54:21.821337484 (51h57m11.758995983s) [noise=true , address=45.91.116.12:4226 , piece_id=VI2Y5I64SLS6SWYBC32HF3QLC5JMACOFLUQW7B7C3QFJT7Y3Z6PQ, seg_pos={0 0}, upload=49920]: Uploading piece... Aug 1 21:54:21.821400215 (51h57m11.759058714s) [noise=true , address=77.90.31.19:4842 , piece_id=5LEMN3QJTN5N5HEIJYIFCIAERN63DSDJN7GCYGQJQJ7ACM7SDQLA, seg_pos={0 0}, upload=49920]: Uploading piece... Aug 1 21:54:21.821614334 (51h57m11.759272823s) [noise=true , address=45.91.116.9:4262 , piece_id=PHBAXJ65HH223XRTVRPGS3SL4XB4XV3JCRQGS5XLCIGBZYKTEVHQ, seg_pos={0 0}, upload=49920]: Uploading piece... Aug 1 21:54:21.821679149 (51h57m11.759337648s) [noise=true , address=45.91.116.4:4938 , piece_id=YLILN4A5BP32B7DRNGCHJIG3CZPMCFVSGAQFVLWNKEV3FTPXMT6Q, seg_pos={0 0}, upload=49920]: Uploading piece... Aug 1 21:54:21.821864212 (51h57m11.759522711s) [noise=true , address=45.91.116.10:4083 , piece_id=FN7XQNW6RF45GFJCWGYMG6TU37TN3UOFI2CR2FYL3XJDSOIACBKQ, seg_pos={0 0}, upload=49920]: Uploading piece... Aug 1 21:54:21.822884644 (51h57m11.760543143s) [noise=true , address=45.91.116.11:4119 , piece_id=S5C7ZF4PDNNAOYWC3XL3SNXIV4YSRKOCAC6L3N3PHDGA2TVFKMEA, seg_pos={0 0}, upload=49920]: Uploading piece... Aug 1 21:54:21.823100417 (51h57m11.760758906s) [noise=true , address=77.90.31.12:4589 , piece_id=Z6NQQBOBSJRHY5GC7BWHDM35GMPMLBYKRE6DN7VSR7WNCL3KRVNQ, seg_pos={0 0}, upload=49920]: Uploading piece... Aug 1 21:54:23.564683254 (51h57m13.502341753s) [upload=49920]: Next returned nil segment Aug 1 21:54:23.564700097 (51h57m13.502358586s) [upload=49920]: Waiting for error group managing segments... Aug 1 21:54:23.843501825 (51h57m13.781160334s) [noise=true , address=77.90.31.13:4628 , piece_id=EJXU6EFBSFHKTLKPYHZDY4UNNJNKDQXQBDHGNARDSPZA23YBWHGQ, seg_pos={0 0}, upload=49920]: Done uploading piece. err: <nil> Aug 1 21:54:23.845236794 (51h57m13.782895303s) [noise=true , address=77.90.31.10:4519 , piece_id=ZAWNIOP3QVGFIN4EZ6BHC2FGO527NYSWHBTMKZWDVSRSBDUGUKUA, seg_pos={0 0}, upload=49920]: Done uploading piece. err: <nil> Aug 1 21:54:23.851919939 (51h57m13.789578428s) [noise=true , address=45.91.116.10:4081 , piece_id=XQ7ZZC6HKVZDEWKBUJWDBYKO2LHVG2MWXE5HRHDQNCDNY36GYLTA, seg_pos={0 0}, upload=49920]: Done uploading piece. err: <nil> Aug 1 21:54:23.863342793 (51h57m13.801001292s) [noise=true , address=77.90.31.19:4841 , piece_id=IPVDNLM5DJ4DAP2MYOD2FGFILZCW7TVT7MTXHRLOVOKCKPJCV7EA, seg_pos={0 0}, upload=49920]: Done uploading piece. err: <nil> Aug 1 21:54:23.921412726 (51h57m13.859071225s) [noise=true , address=45.91.116.12:4226 , piece_id=VI2Y5I64SLS6SWYBC32HF3QLC5JMACOFLUQW7B7C3QFJT7Y3Z6PQ, seg_pos={0 0}, upload=49920]: Done uploading piece. err: <nil> Aug 1 21:54:23.922233376 (51h57m13.859891875s) [noise=true , address=45.91.116.9:4262 , piece_id=PHBAXJ65HH223XRTVRPGS3SL4XB4XV3JCRQGS5XLCIGBZYKTEVHQ, seg_pos={0 0}, upload=49920]: Done uploading piece. err: <nil> Aug 1 21:54:23.926507283 (51h57m13.864165782s) [noise=true , address=77.90.31.19:4842 , piece_id=5LEMN3QJTN5N5HEIJYIFCIAERN63DSDJN7GCYGQJQJ7ACM7SDQLA, seg_pos={0 0}, upload=49920]: Done uploading piece. err: <nil> Aug 1 21:54:23.929820481 (51h57m13.86747898s) [noise=true , address=45.91.116.10:4083 , piece_id=FN7XQNW6RF45GFJCWGYMG6TU37TN3UOFI2CR2FYL3XJDSOIACBKQ, seg_pos={0 0}, upload=49920]: Done uploading piece. err: <nil> Aug 1 21:54:23.935253295 (51h57m13.872911794s) [noise=true , address=45.91.116.6:4973 , piece_id=NF2Y5SV7Q3S3MTFSI3DZA4JNRPOM7BYKKCE33D2YYNWI34IC7L3A, seg_pos={0 0}, upload=49920]: Done uploading piece. err: <nil> Aug 1 21:54:23.936465073 (51h57m13.874123572s) [noise=true , address=45.91.116.11:4119 , piece_id=S5C7ZF4PDNNAOYWC3XL3SNXIV4YSRKOCAC6L3N3PHDGA2TVFKMEA, seg_pos={0 0}, upload=49920]: Done uploading piece. err: <nil> Aug 1 21:54:24.008511355 (51h57m13.946169854s) [noise=true , address=45.91.116.4:4938 , piece_id=YLILN4A5BP32B7DRNGCHJIG3CZPMCFVSGAQFVLWNKEV3FTPXMT6Q, seg_pos={0 0}, upload=49920]: Done uploading piece. err: <nil> Aug 1 21:54:24.163933144 (51h57m14.101591643s) [noise=true , address=77.90.31.12:4589 , piece_id=Z6NQQBOBSJRHY5GC7BWHDM35GMPMLBYKRE6DN7VSR7WNCL3KRVNQ, seg_pos={0 0}, upload=49920]: Done uploading piece. err: <nil> Aug 1 21:54:50.931068973 (51h57m40.868727462s) [noise=true , address=104.36.82.4:4802 , piece_id=N4XRJIP6SE4J5H2QFLESLGAXRU7F7KIFAEVPKNXU4CRYFMODFRYQ, seg_pos={0 0}, upload=49920]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: write tcp 188.166.225.170:47794->104.36.82.4:4802: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:54:50.931082940 (51h57m40.868741449s) [noise=true , address=104.36.82.3:4765 , piece_id=5JWL3TF4SXMGRDI4YWLVKRYIPVNKMG7OATF7GJGR2ILCSUJ3373Q, seg_pos={0 0}, upload=49920]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:104.36.82.3:4765): protocol: context deadline exceeded; context deadline exceeded Aug 1 21:54:50.931120421 (51h57m40.86877892s) [noise=true , address=104.36.82.9:4006 , piece_id=ECQRTBZP4T5NFT2GER5Y4RT2YND7HEAWX5NUI6XG6K5WGVCL3IWQ, seg_pos={0 0}, upload=49920]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: write tcp 188.166.225.170:48514->104.36.82.9:4006: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:54:50.931183922 (51h57m40.868842421s) [noise=true , address=104.36.82.5:4837 , piece_id=OVKX6V5TNG5B7BL6DU7H3ALXVIW4UHRTXI4AQSHEO6WIGPJNTEVQ, seg_pos={0 0}, upload=49920]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:104.36.82.5:4837): protocol: rpc: dial tcp 104.36.82.5:4837: i/o timeout; conn closed: op Aug 1 21:54:50.931200685 (51h57m40.868859184s) [seg_pos={0 0}, upload=49920]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Aug 1 21:54:51.166648722 (51h57m41.104307221s) [upload=49920]: Done uploading segments... Aug 1 21:54:51.166672908 (51h57m41.104331407s) [upload=49920]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.23724965s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 1m0.241331431s FAIL (2025-08-01 21:54:36 UTC) orbiter warning: Operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 8) ======LOGS====== 04.36.82.11:4136 , piece_id=P5OG5DMNQCN547W4KHGJSUKL2J4SW5Y73Y3UIWDT5YDONHIJG6WA, seg_pos={0 0}, upload=49913]: Uploading piece... Aug 1 21:53:36.932275359 (51h56m26.869933848s) [noise=true , address=45.91.116.4:4965 , piece_id=5T7ZPLUAXCYZHVHVNVEXBKWK7HFT75TMFTNZ7CORZ5S5DWOWYHRQ, seg_pos={0 0}, upload=49913]: Uploading piece... Aug 1 21:53:36.932368747 (51h56m26.870027246s) [noise=true , address=104.36.82.7:4926 , piece_id=SUNQISGZUFOGJENNNNDBEZ2GZ7ACI7J5OP4AKHUWZVHB7OG4IIFA, seg_pos={0 0}, upload=49913]: Uploading piece... Aug 1 21:53:36.932575363 (51h56m26.870233862s) [noise=true , address=104.36.82.62:4435 , piece_id=44JKKSRJQPAM3KPISY25UWCAZG6LHBWIUABIZUI4LSD2D7FEFJXQ, seg_pos={0 0}, upload=49913]: Uploading piece... Aug 1 21:53:36.932645576 (51h56m26.870304075s) [noise=true , address=79.127.201.215:4331 , piece_id=KAOLFPN2O3MET4KR7RI7ZPC5DBEJ3XXIB2M5GQELYJSYPXCGGY7Q, seg_pos={0 0}, upload=49913]: Uploading piece... Aug 1 21:53:36.932773161 (51h56m26.87043166s) [noise=true , address=104.36.82.5:4856 , piece_id=ASSNJ4X2LWCI3NKWWCUDV2C6GQAPGRS7JS5YSQ3IB2VIFKBPE5NA, seg_pos={0 0}, upload=49913]: Uploading piece... Aug 1 21:53:36.933018170 (51h56m26.870676669s) [noise=true , address=104.36.82.44:4903 , piece_id=OJ65H5TY35TTNJZDLI5PH4RNZBLOEG4CY7JAOM2YZNB44S6BVIQQ, seg_pos={0 0}, upload=49913]: Uploading piece... Aug 1 21:53:36.933097892 (51h56m26.870756391s) [noise=true , address=104.36.82.56:4257 , piece_id=QOCYYNULSHDKFQD34DIGU5GFTEWJLS7J5F6BGS7MLKOGU6I3KROA, seg_pos={0 0}, upload=49913]: Uploading piece... Aug 1 21:53:36.933251596 (51h56m26.870910095s) [noise=true , address=79.127.219.51:4181 , piece_id=WC2JG4BH3I6D7HGHBZA6CAZYS65J7BHHGCRHRFQ2WEBU5JTJBQ4A, seg_pos={0 0}, upload=49913]: Uploading piece... Aug 1 21:53:36.933474853 (51h56m26.871133352s) [noise=true , address=79.127.219.38:4009 , piece_id=JMGASGNIKJKBG2BOFYV24CWH2UFPGMMWX6CNI4KHBXIU3ADQBEFA, seg_pos={0 0}, upload=49913]: Uploading piece... Aug 1 21:53:36.933518096 (51h56m26.871176595s) [noise=true , address=104.36.82.16:4325 , piece_id=RCNED4PYNDWKO6VBU264QOWA2IH2ZHDJTEK26L2CSHX3CNTVAKWQ, seg_pos={0 0}, upload=49913]: Uploading piece... Aug 1 21:53:36.933761853 (51h56m26.871420342s) [noise=true , address=79.127.205.235:4663 , piece_id=OVDHMDT2G2KAR27EESEPXSX62MDZXYK2SYLDZ3H4M7XAHHZ4FYFA, seg_pos={0 0}, upload=49913]: Uploading piece... Aug 1 21:53:36.933853417 (51h56m26.871511917s) [noise=true , address=104.36.82.42:4823 , piece_id=7G4BMOAJE2XFTZEWNZORLY3RUWZVR6D42HTVCRA27CRTY4IR3XDA, seg_pos={0 0}, upload=49913]: Uploading piece... Aug 1 21:53:36.934027099 (51h56m26.871685588s) [noise=true , address=104.36.82.46:4485 , piece_id=7RDAF7FFURF3URCPE6KBZ35GXQQXJDRMBPUOHWA5LNVP7ZMPHFXQ, seg_pos={0 0}, upload=49913]: Uploading piece... Aug 1 21:53:36.934158271 (51h56m26.87181677s) [noise=true , address=79.127.205.245:4221 , piece_id=6N2CC4DFHPJT7Z66UI7K6AFD6B62AANF67H6HA3GYBS6C75SW6IQ, seg_pos={0 0}, upload=49913]: Uploading piece... Aug 1 21:53:36.934234767 (51h56m26.871893256s) [noise=true , address=104.36.82.9:4008 , piece_id=Z4SZDEFU2EUUIO4BSSNLVHESPTHDHIIJ4UICJWJAAOLEEUA5QJVQ, seg_pos={0 0}, upload=49913]: Uploading piece... Aug 1 21:53:38.030411540 (51h56m27.968070039s) [upload=49913]: Next returned nil segment Aug 1 21:53:38.030446827 (51h56m27.968105326s) [upload=49913]: Waiting for error group managing segments... Aug 1 21:53:38.335616368 (51h56m28.273274867s) [noise=true , address=79.127.205.245:4221 , piece_id=6N2CC4DFHPJT7Z66UI7K6AFD6B62AANF67H6HA3GYBS6C75SW6IQ, seg_pos={0 0}, upload=49913]: Done uploading piece. err: <nil> Aug 1 21:53:38.357265865 (51h56m28.294924364s) [noise=true , address=79.127.205.235:4663 , piece_id=OVDHMDT2G2KAR27EESEPXSX62MDZXYK2SYLDZ3H4M7XAHHZ4FYFA, seg_pos={0 0}, upload=49913]: Done uploading piece. err: <nil> Aug 1 21:53:38.422838513 (51h56m28.360497012s) [noise=true , address=64.49.13.169:4715 , piece_id=FG4TMHKYQ25XIPND3YM27JJDBT5TBXUYFBPD6BETNDFWPNVMBYNA, seg_pos={0 0}, upload=49913]: Done uploading piece. err: <nil> Aug 1 21:53:38.809077231 (51h56m28.74673573s) [noise=true , address=79.127.201.215:4331 , piece_id=KAOLFPN2O3MET4KR7RI7ZPC5DBEJ3XXIB2M5GQELYJSYPXCGGY7Q, seg_pos={0 0}, upload=49913]: Done uploading piece. err: <nil> Aug 1 21:53:38.841598979 (51h56m28.779257468s) [noise=true , address=45.91.116.4:4965 , piece_id=5T7ZPLUAXCYZHVHVNVEXBKWK7HFT75TMFTNZ7CORZ5S5DWOWYHRQ, seg_pos={0 0}, upload=49913]: Done uploading piece. err: <nil> Aug 1 21:53:38.887017590 (51h56m28.824676079s) [noise=true , address=79.127.219.38:4009 , piece_id=JMGASGNIKJKBG2BOFYV24CWH2UFPGMMWX6CNI4KHBXIU3ADQBEFA, seg_pos={0 0}, upload=49913]: Done uploading piece. err: <nil> Aug 1 21:53:38.901108378 (51h56m28.838766877s) [noise=true , address=79.127.219.51:4181 , piece_id=WC2JG4BH3I6D7HGHBZA6CAZYS65J7BHHGCRHRFQ2WEBU5JTJBQ4A, seg_pos={0 0}, upload=49913]: Done uploading piece. err: <nil> Aug 1 21:53:39.003328845 (51h56m28.940987355s) [noise=true , address=77.90.31.14:4674 , piece_id=FSNLVXPLB3N7DM6TNG7KYKGW4ASGEIXT45622YK7XHTKOKW43BXA, seg_pos={0 0}, upload=49913]: Done uploading piece. err: <nil> Aug 1 21:54:35.824604480 (51h57m25.762262979s) [noise=true , address=104.36.82.11:4136 , piece_id=P5OG5DMNQCN547W4KHGJSUKL2J4SW5Y73Y3UIWDT5YDONHIJG6WA, seg_pos={0 0}, upload=49913]: Done uploading piece. err: ecclient: upload failed (node:127n8CMbZQ7haiv6tGHrVg9V8Yf12sYkcbL9mEbSpQ4WkT9isnp, address:104.36.82.11:4136): protocol: rpc: dial tcp 104.36.82.11:4136: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:54:35.824666348 (51h57m25.762324847s) [noise=true , address=104.36.82.5:4856 , piece_id=ASSNJ4X2LWCI3NKWWCUDV2C6GQAPGRS7JS5YSQ3IB2VIFKBPE5NA, seg_pos={0 0}, upload=49913]: Done uploading piece. err: ecclient: upload failed (node:12mXTGa9rxx8jU8oKLmR9obX3Utn7p5rQH2GeK6LeJ6jiNvfYZ6, address:104.36.82.5:4856): protocol: rpc: dial tcp 104.36.82.5:4856: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:54:35.824707157 (51h57m25.762365656s) [noise=true , address=104.36.82.16:4325 , piece_id=RCNED4PYNDWKO6VBU264QOWA2IH2ZHDJTEK26L2CSHX3CNTVAKWQ, seg_pos={0 0}, upload=49913]: Done uploading piece. err: ecclient: upload failed (node:1aTYNxPbXJGM29Xqovc5VK4cun64wr2DoduaVCvKDj38xCLGKJ, address:104.36.82.16:4325): protocol: rpc: dial tcp 104.36.82.16:4325: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:54:35.824750559 (51h57m25.762409058s) [noise=true , address=104.36.82.46:4485 , piece_id=7RDAF7FFURF3URCPE6KBZ35GXQQXJDRMBPUOHWA5LNVP7ZMPHFXQ, seg_pos={0 0}, upload=49913]: Done uploading piece. err: ecclient: upload failed (node:12w74nT4D4F3tttRavcoKiPzyct4CxvGQfbn6EcVypHzaexCxNJ, address:104.36.82.46:4485): protocol: rpc: dial tcp 104.36.82.46:4485: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:54:35.824796327 (51h57m25.762454826s) [noise=true , address=104.36.82.7:4926 , piece_id=SUNQISGZUFOGJENNNNDBEZ2GZ7ACI7J5OP4AKHUWZVHB7OG4IIFA, seg_pos={0 0}, upload=49913]: Done uploading piece. err: ecclient: upload failed (node:12EXjvaPQL9GhezbMgqhQBZvEKk9UTEMp8n8G2Pet5F3r4XLpDr, address:104.36.82.7:4926): protocol: rpc: dial tcp 104.36.82.7:4926: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:54:35.824837986 (51h57m25.762496485s) [noise=true , address=104.36.82.56:4257 , piece_id=QOCYYNULSHDKFQD34DIGU5GFTEWJLS7J5F6BGS7MLKOGU6I3KROA, seg_pos={0 0}, upload=49913]: Done uploading piece. err: ecclient: upload failed (node:12iw1KEPBAYhvSachvkigv7bPXfGN5XEugvxGC5YTBV2Muymir3, address:104.36.82.56:4257): protocol: rpc: dial tcp 104.36.82.56:4257: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:54:35.824883063 (51h57m25.762541552s) [noise=true , address=104.36.82.42:4823 , piece_id=7G4BMOAJE2XFTZEWNZORLY3RUWZVR6D42HTVCRA27CRTY4IR3XDA, seg_pos={0 0}, upload=49913]: Done uploading piece. err: ecclient: upload failed (node:1fkA5DEj9RqsR6N85z7azvujy8tPkUhB5GCqBpUmFb4vvwbd8y, address:104.36.82.42:4823): protocol: rpc: dial tcp 104.36.82.42:4823: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:54:35.824926395 (51h57m25.762584894s) [noise=true , address=104.36.82.9:4008 , piece_id=Z4SZDEFU2EUUIO4BSSNLVHESPTHDHIIJ4UICJWJAAOLEEUA5QJVQ, seg_pos={0 0}, upload=49913]: Done uploading piece. err: ecclient: upload failed (node:148CqQmrsJSfoGByHYRPoZmvP1VgQQpaMVLb7RgRQvNBuo7air, address:104.36.82.9:4008): protocol: rpc: dial tcp 104.36.82.9:4008: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:54:35.824969528 (51h57m25.762628027s) [noise=true , address=104.36.82.62:4435 , piece_id=44JKKSRJQPAM3KPISY25UWCAZG6LHBWIUABIZUI4LSD2D7FEFJXQ, seg_pos={0 0}, upload=49913]: Done uploading piece. err: ecclient: upload failed (node:12o9DvdWBWefvKcvWd62RS9pX7utf7KwbyDaYxfwdXtrMTvcUTh, address:104.36.82.62:4435): protocol: rpc: dial tcp 104.36.82.62:4435: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:54:35.825014264 (51h57m25.762672763s) [noise=true , address=104.36.82.44:4903 , piece_id=OJ65H5TY35TTNJZDLI5PH4RNZBLOEG4CY7JAOM2YZNB44S6BVIQQ, seg_pos={0 0}, upload=49913]: Done uploading piece. err: ecclient: upload failed (node:12mY4YcLLVbnVuXtKSHwyv4sgTycqzxUZHGVQha5WfXXTPjNGLq, address:104.36.82.44:4903): protocol: write tcp 188.166.225.170:60788->104.36.82.44:4903: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:54:35.825029733 (51h57m25.762688232s) [seg_pos={0 0}, upload=49913]: Done waiting for segment. successful: 8 optimal: 14 errs: <nil> Aug 1 21:54:36.063442984 (51h57m26.001101483s) [upload=49913]: Done uploading segments... Aug 1 21:54:36.063469305 (51h57m26.001127794s) [upload=49913]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 8) Duration: 1m0.241380775s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [...........................................X............X...] duration: 1.877679443s FAIL (2025-08-01 21:54:27 UTC) orbiter warning: Operation: sync2.Copy digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/1K):: after 7b07c632440be552d064c57547a519a1464d9714c90d6f11ea4290210b73bd46 != before b944efe6550e3eab95f8314489d530afd89693d29ea288891d748fdb1ed0ffd1 Duration: 1.877702878s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 3 out of 60 is failed (threshold=3) [X.........................................................XX] duration: 30.237253773s FAIL (2025-08-01 21:54:23 UTC) orbiter warning: Operation: Commit uplink: failed to upload enough pieces (needed at least 14 but got 3) Duration: 30.237265825s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 1m0.000961493s FAIL (2025-08-01 21:54:20 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.001006138s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [.X.........................................................X] duration: 30.229115632s FAIL (2025-08-01 21:53:53 UTC) orbiter warning: Operation: Commit uplink: failed to upload enough pieces (needed at least 14 but got 4) Duration: 30.229135891s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 30.235086345s FAIL (2025-08-01 21:53:48 UTC) orbiter warning: Operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: expected piece hash; context canceled; EOF Aug 1 21:52:50.659296438 (51h55m40.596954927s) [seg_pos={0 0}, upload=49901]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 1 21:52:50.659354139 (51h55m40.597012638s) [upload=49901]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 1 21:52:50.659357996 (51h55m40.597016485s) [upload=49901]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 1 21:52:50.944433262 (51h55m40.882091771s) [upload=49901]: Done uploading segments... Aug 1 21:52:50.944451486 (51h55m40.882109985s) [upload=49901]: Upload finished. err: <nil> Aug 1 21:53:18.518537294 (51h56m8.456195793s) [upload=49906]: Starting upload Aug 1 21:53:18.518590285 (51h56m8.456248785s) [upload=49906]: Done starting upload Aug 1 21:53:18.518789918 (51h56m8.456448417s) [upload=49906]: Uploading segments... Aug 1 21:53:18.519391288 (51h56m8.457049787s) [upload=49906]: Got next segment. Inline: false Aug 1 21:53:18.519430303 (51h56m8.457088812s) [upload=49906]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 1 21:53:18.519435022 (51h56m8.457093511s) [upload=49906]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 1 21:53:18.755803378 (51h56m8.693461867s) [seg_pos={0 0}, upload=49906]: Begin upload segment... Aug 1 21:53:18.755845529 (51h56m8.693504028s) [seg_pos={0 0}, upload=49906]: Done begin upload segment. Aug 1 21:53:18.755867481 (51h56m8.693525971s) [noise=true , address=77.90.31.16:4735 , piece_id=SQL2QSOG2CNSNNTPMGFDKIEOTRU4AF4I5CD2PG656G2QOQDL5S7A, seg_pos={0 0}, upload=49906]: Uploading piece... Aug 1 21:53:18.755889854 (51h56m8.693548354s) [noise=true , address=45.91.116.8:4045 , piece_id=BIV75MAMSUJ5SX576NAEOTOHO2JUXROKNEY43FZAGMQROBBSDIFA, seg_pos={0 0}, upload=49906]: Uploading piece... Aug 1 21:53:18.756249282 (51h56m8.693907781s) [noise=true , address=45.91.116.5:4299 , piece_id=EUBHDLYXRGDY7OBIKRRXQPOVQBNY7UMWFMBQQ5MH6DKGLWCI6RWA, seg_pos={0 0}, upload=49906]: Uploading piece... Aug 1 21:53:18.756357479 (51h56m8.694015978s) [noise=true , address=77.90.31.11:4555 , piece_id=BKJ2EGTOJWYQRJZMUDK7UD37PS5RTYALOFNAPOYMQTZZBWM5UQZA, seg_pos={0 0}, upload=49906]: Uploading piece... Aug 1 21:53:18.756506283 (51h56m8.694164783s) [noise=true , address=77.90.31.18:4807 , piece_id=ZEV6S7ZJOKWNRD3T5XNESFKNVOPUXRIKYYIGYONEP2LOAHAK25HA, seg_pos={0 0}, upload=49906]: Uploading piece... Aug 1 21:53:18.756660810 (51h56m8.694319299s) [noise=true , address=77.90.31.16:4734 , piece_id=MY3X2QHUXO6Q3MOP75BA2OZ2ULS2VORDC245BLT7RWKVQ4JMFMYQ, seg_pos={0 0}, upload=49906]: Uploading piece... Aug 1 21:53:18.756951665 (51h56m8.694610164s) [noise=true , address=77.90.31.17:4770 , piece_id=XYNTVZYVJLNCB6SENLKJQSTTFLDI5N4EYAXGSWIRLIWAWQJPMOZA, seg_pos={0 0}, upload=49906]: Uploading piece... Aug 1 21:53:18.756961305 (51h56m8.694619804s) [noise=true , address=104.36.82.14:4223 , piece_id=3H4L4HYKYCAJWUMZXXXRNW7CXKNM4BQTNIDRPJSOYZGPGNKUESGA, seg_pos={0 0}, upload=49906]: Uploading piece... Aug 1 21:53:18.757420192 (51h56m8.695078691s) [noise=true , address=104.36.82.9:4007 , piece_id=D2VDLLQIDOK3NJEPRTICTFFUA4KEPN7RYXIE5JRB6NIKT7D3F7FQ, seg_pos={0 0}, upload=49906]: Uploading piece... Aug 1 21:53:18.757705118 (51h56m8.695363617s) [noise=true , address=104.36.82.6:4875 , piece_id=6A2VGCQ5RIHWHGAPUGYFD4ZXFLQPAU3KDBHHWLTJ4GPYFBUENGPQ, seg_pos={0 0}, upload=49906]: Uploading piece... Aug 1 21:53:18.758335524 (51h56m8.695994023s) [noise=true , address=77.90.31.14:4663 , piece_id=FYQ4R4WRB2HAFABBY3U3FMN6G3TK3UY2H3WYUALI3AGIICHTDQ5A, seg_pos={0 0}, upload=49906]: Uploading piece... Aug 1 21:53:18.758815091 (51h56m8.6964736s) [noise=true , address=104.36.82.2:4730 , piece_id=LWUE3RLN3V7OB6MCA6DQKN4JIXSY4HXY7REESWOGPY2U4PGUIYJA, seg_pos={0 0}, upload=49906]: Uploading piece... Aug 1 21:53:18.759175290 (51h56m8.696833789s) [noise=true , address=45.91.116.13:4155 , piece_id=JZX6SUHKWKCD2KNER3P4WWPGHS6HFHPGAZ5OTPUQCO3OAVXWOKVQ, seg_pos={0 0}, upload=49906]: Uploading piece... Aug 1 21:53:18.759456798 (51h56m8.697115297s) [noise=true , address=77.90.31.21:4877 , piece_id=UOVC5YL3EICG3DQO7PJLVSKQWEJA75ZVBVW7FT4GHSW5O3ZFH6VA, seg_pos={0 0}, upload=49906]: Uploading piece... Aug 1 21:53:18.759686118 (51h56m8.697344617s) [noise=true , address=104.36.82.3:4767 , piece_id=F6FMRRUDCGKY2LZNPSWERL6QXHPDZZBF3CDZIWKHENCHI4PWF7JQ, seg_pos={0 0}, upload=49906]: Uploading piece... Aug 1 21:53:18.759897782 (51h56m8.697556281s) [noise=true , address=45.91.116.7:4011 , piece_id=XCRLWKI2VFMQHANJU5J6ZV5JYD7GIHK6VAVVZQHVM67HZLYCFURA, seg_pos={0 0}, upload=49906]: Uploading piece... Aug 1 21:53:20.433266739 (51h56m10.370925238s) [upload=49906]: Next returned nil segment Aug 1 21:53:20.433280476 (51h56m10.370938975s) [upload=49906]: Waiting for error group managing segments... Aug 1 21:53:20.772434388 (51h56m10.710092887s) [noise=true , address=45.91.116.5:4299 , piece_id=EUBHDLYXRGDY7OBIKRRXQPOVQBNY7UMWFMBQQ5MH6DKGLWCI6RWA, seg_pos={0 0}, upload=49906]: Done uploading piece. err: <nil> Aug 1 21:53:20.774343902 (51h56m10.712002401s) [noise=true , address=77.90.31.16:4734 , piece_id=MY3X2QHUXO6Q3MOP75BA2OZ2ULS2VORDC245BLT7RWKVQ4JMFMYQ, seg_pos={0 0}, upload=49906]: Done uploading piece. err: <nil> Aug 1 21:53:20.781733871 (51h56m10.71939237s) [noise=true , address=77.90.31.17:4770 , piece_id=XYNTVZYVJLNCB6SENLKJQSTTFLDI5N4EYAXGSWIRLIWAWQJPMOZA, seg_pos={0 0}, upload=49906]: Done uploading piece. err: <nil> Aug 1 21:53:20.791277791 (51h56m10.72893629s) [noise=true , address=77.90.31.14:4663 , piece_id=FYQ4R4WRB2HAFABBY3U3FMN6G3TK3UY2H3WYUALI3AGIICHTDQ5A, seg_pos={0 0}, upload=49906]: Done uploading piece. err: <nil> Aug 1 21:53:20.802267898 (51h56m10.739926397s) [noise=true , address=77.90.31.21:4877 , piece_id=UOVC5YL3EICG3DQO7PJLVSKQWEJA75ZVBVW7FT4GHSW5O3ZFH6VA, seg_pos={0 0}, upload=49906]: Done uploading piece. err: <nil> Aug 1 21:53:20.856636360 (51h56m10.794294859s) [noise=true , address=77.90.31.11:4555 , piece_id=BKJ2EGTOJWYQRJZMUDK7UD37PS5RTYALOFNAPOYMQTZZBWM5UQZA, seg_pos={0 0}, upload=49906]: Done uploading piece. err: <nil> Aug 1 21:53:20.858019667 (51h56m10.795678166s) [noise=true , address=45.91.116.8:4045 , piece_id=BIV75MAMSUJ5SX576NAEOTOHO2JUXROKNEY43FZAGMQROBBSDIFA, seg_pos={0 0}, upload=49906]: Done uploading piece. err: <nil> Aug 1 21:53:20.867979653 (51h56m10.805638172s) [noise=true , address=45.91.116.13:4155 , piece_id=JZX6SUHKWKCD2KNER3P4WWPGHS6HFHPGAZ5OTPUQCO3OAVXWOKVQ, seg_pos={0 0}, upload=49906]: Done uploading piece. err: <nil> Aug 1 21:53:20.879459447 (51h56m10.817117946s) [noise=true , address=77.90.31.18:4807 , piece_id=ZEV6S7ZJOKWNRD3T5XNESFKNVOPUXRIKYYIGYONEP2LOAHAK25HA, seg_pos={0 0}, upload=49906]: Done uploading piece. err: <nil> Aug 1 21:53:20.885831678 (51h56m10.823490197s) [noise=true , address=45.91.116.7:4011 , piece_id=XCRLWKI2VFMQHANJU5J6ZV5JYD7GIHK6VAVVZQHVM67HZLYCFURA, seg_pos={0 0}, upload=49906]: Done uploading piece. err: <nil> Aug 1 21:53:21.081904849 (51h56m11.019563348s) [noise=true , address=77.90.31.16:4735 , piece_id=SQL2QSOG2CNSNNTPMGFDKIEOTRU4AF4I5CD2PG656G2QOQDL5S7A, seg_pos={0 0}, upload=49906]: Done uploading piece. err: <nil> Aug 1 21:53:47.873480294 (51h56m37.811138793s) [noise=true , address=104.36.82.14:4223 , piece_id=3H4L4HYKYCAJWUMZXXXRNW7CXKNM4BQTNIDRPJSOYZGPGNKUESGA, seg_pos={0 0}, upload=49906]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:104.36.82.14:4223): protocol: rpc: dial tcp 104.36.82.14:4223: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:53:47.873557562 (51h56m37.811216061s) [noise=true , address=104.36.82.3:4767 , piece_id=F6FMRRUDCGKY2LZNPSWERL6QXHPDZZBF3CDZIWKHENCHI4PWF7JQ, seg_pos={0 0}, upload=49906]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:104.36.82.3:4767): protocol: rpc: dial tcp 104.36.82.3:4767: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:53:47.873599993 (51h56m37.811258492s) [noise=true , address=104.36.82.6:4875 , piece_id=6A2VGCQ5RIHWHGAPUGYFD4ZXFLQPAU3KDBHHWLTJ4GPYFBUENGPQ, seg_pos={0 0}, upload=49906]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:104.36.82.6:4875): protocol: rpc: dial tcp 104.36.82.6:4875: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:53:47.873635010 (51h56m37.811293509s) [noise=true , address=104.36.82.9:4007 , piece_id=D2VDLLQIDOK3NJEPRTICTFFUA4KEPN7RYXIE5JRB6NIKT7D3F7FQ, seg_pos={0 0}, upload=49906]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:104.36.82.9:4007): protocol: rpc: dial tcp 104.36.82.9:4007: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:53:47.873667171 (51h56m37.811325671s) [noise=true , address=104.36.82.2:4730 , piece_id=LWUE3RLN3V7OB6MCA6DQKN4JIXSY4HXY7REESWOGPY2U4PGUIYJA, seg_pos={0 0}, upload=49906]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:104.36.82.2:4730): protocol: write tcp 188.166.225.170:41370->104.36.82.2:4730: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:53:47.873679725 (51h56m37.811338224s) [seg_pos={0 0}, upload=49906]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Aug 1 21:53:48.106630115 (51h56m38.044288604s) [upload=49906]: Done uploading segments... Aug 1 21:53:48.106654551 (51h56m38.04431305s) [upload=49906]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 30.23520974s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)