Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [..XX.X.X....................................................] duration: 5.844150475s FAIL (2025-08-01 22:38:40 UTC) orbiter warning: Operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 7NK72474VWE4E3YOBCDZ5O5ZOH2MSA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB): protocol: write tcp 188.166.225.170:38368->104.36.82.12:4151: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:56:50.418604753 (51h59m40.356263252s) [seg_pos={0 0}, upload=49961]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 1 21:56:50.418680879 (51h59m40.356339378s) [upload=49961]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 1 21:56:50.418685998 (51h59m40.356344487s) [upload=49961]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 1 21:56:50.726563872 (51h59m40.664222371s) [upload=49961]: Done uploading segments... Aug 1 21:56:50.726586586 (51h59m40.664245085s) [upload=49961]: Upload finished. err: <nil> Aug 1 21:57:28.852869669 (52h0m18.790528158s) [upload=49973]: Starting upload Aug 1 21:57:28.852942389 (52h0m18.790600888s) [upload=49973]: Done starting upload Aug 1 21:57:28.853007122 (52h0m18.790665611s) [upload=49973]: Uploading segments... Aug 1 21:57:28.853174002 (52h0m18.790832491s) [upload=49973]: Got next segment. Inline: false Aug 1 21:57:28.853208168 (52h0m18.790866657s) [upload=49973]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 1 21:57:28.853212666 (52h0m18.790871165s) [upload=49973]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 1 21:57:29.095087519 (52h0m19.032746008s) [seg_pos={0 0}, upload=49973]: Begin upload segment... Aug 1 21:57:29.095129960 (52h0m19.032788459s) [seg_pos={0 0}, upload=49973]: Done begin upload segment. Aug 1 21:57:29.095164997 (52h0m19.032823496s) [noise=true , address=77.90.31.21:4878 , piece_id=BLNTYBOAIZWISSUJYK6BSBPTSNL6QWLGEN6AP6P4NPDVSURYG5AA, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.095171800 (52h0m19.032830289s) [noise=true , address=104.36.82.6:4875 , piece_id=CS5AMB7CIBQWWOU57Z3RCCSNVKIPE4B5QWU5YIQYI2UCWH7PAPVA, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.095510027 (52h0m19.033168526s) [noise=true , address=77.90.31.18:4806 , piece_id=52B6CAHGLFVTFIDQHHHAWKCBP2WJJY7OTTGRZZ7CEUH3HDA43Q6A, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.095544193 (52h0m19.033202692s) [noise=true , address=77.90.31.10:4518 , piece_id=WUOVXUSDG6YQTDPWG4YO7VLJNZZVLH6VRGMWOJABG2EJRU7XHP5A, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.095774283 (52h0m19.033432782s) [noise=true , address=104.36.82.7:4909 , piece_id=QHJCVBT6KTU34K2KWFNACWYVVE3WJ2RU76BHVSKL4HLCZEFXZYPA, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.095819880 (52h0m19.033478379s) [noise=true , address=45.91.116.12:4225 , piece_id=PBQPLA5ZYRFPSKHDKCEFOEBS5T3GYMP2FLSFDQWJLVY756KPLJNQ, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.096021656 (52h0m19.033680155s) [noise=true , address=45.91.116.13:4153 , piece_id=PMLKMZO66KXF3ZONO5ZZOQZICFK5XNGL6MN5FFBKICVFU5AWU6PQ, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.096069618 (52h0m19.033728117s) [noise=true , address=45.91.116.9:4262 , piece_id=A3F4L3RR6WPFMBFFKLATNGG3CASBIPWQP5O3G5URFNTXBE3ZW3VA, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.096348041 (52h0m19.03400654s) [noise=true , address=104.36.82.5:4840 , piece_id=UYVKVFCJJP7XZYRIVPS4ABGEMDUMNR7YW4QGG5ZFTFCGNCJO3CSQ, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.096354803 (52h0m19.034013292s) [noise=true , address=77.90.31.12:4590 , piece_id=D6FUBIFCQJSDRGWEP5GG34AQPNB3UROCYOATD6SRTJ7KAYVMFCOA, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.096608579 (52h0m19.034267078s) [noise=true , address=77.90.31.16:4733 , piece_id=DMASMGOUATLPEKCGLHKNNIMMFYVVGSULNUOMJU6RJY3ULBDOY6KQ, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.096830774 (52h0m19.034489273s) [noise=true , address=104.36.82.4:4802 , piece_id=CLM2G33M6AK5HIESRW6LZY33NKHIR2PGITWJ23NPILRCA6UYIEHA, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.097107233 (52h0m19.034765732s) [noise=true , address=77.90.31.19:4843 , piece_id=UFONZYHL7EAFYE5PRW32RKUVQPMZPXBEFLLIFJMAB2TXJ3C6KKNA, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.097324328 (52h0m19.034982827s) [noise=true , address=77.90.31.17:4771 , piece_id=Y3WSALJX23CWPPYQ3XYMCFOSBLBX2E2JQCCS7EOEYNBWY4CDRT3Q, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.098682446 (52h0m19.036340945s) [noise=true , address=77.90.31.15:4698 , piece_id=TN4OTPUD47FD6WOLZBEHNDI43WY6QVO5JK2JNJXNVM2IT6XMGERQ, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.099041603 (52h0m19.036700102s) [noise=true , address=45.91.116.11:4118 , piece_id=AKZIOAR6NRNO7JWCQZIKNSL7IUH5S4IG3KPG3IY2FAKG74DH3E3Q, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:30.718412255 (52h0m20.656070754s) [upload=49973]: Next returned nil segment Aug 1 21:57:30.718429869 (52h0m20.656088368s) [upload=49973]: Waiting for error group managing segments... Aug 1 21:57:31.107386675 (52h0m21.045045164s) [noise=true , address=45.91.116.13:4153 , piece_id=PMLKMZO66KXF3ZONO5ZZOQZICFK5XNGL6MN5FFBKICVFU5AWU6PQ, seg_pos={0 0}, upload=49973]: Done uploading piece. err: <nil> Aug 1 21:57:31.108529412 (52h0m21.046187911s) [noise=true , address=77.90.31.12:4590 , piece_id=D6FUBIFCQJSDRGWEP5GG34AQPNB3UROCYOATD6SRTJ7KAYVMFCOA, seg_pos={0 0}, upload=49973]: Done uploading piece. err: <nil> Aug 1 21:57:31.109975858 (52h0m21.047634347s) [noise=true , address=77.90.31.15:4698 , piece_id=TN4OTPUD47FD6WOLZBEHNDI43WY6QVO5JK2JNJXNVM2IT6XMGERQ, seg_pos={0 0}, upload=49973]: Done uploading piece. err: <nil> Aug 1 21:57:31.115273013 (52h0m21.052931502s) [noise=true , address=77.90.31.10:4518 , piece_id=WUOVXUSDG6YQTDPWG4YO7VLJNZZVLH6VRGMWOJABG2EJRU7XHP5A, seg_pos={0 0}, upload=49973]: Done uploading piece. err: <nil> Aug 1 21:57:31.116767281 (52h0m21.054425781s) [noise=true , address=77.90.31.18:4806 , piece_id=52B6CAHGLFVTFIDQHHHAWKCBP2WJJY7OTTGRZZ7CEUH3HDA43Q6A, seg_pos={0 0}, upload=49973]: Done uploading piece. err: <nil> Aug 1 21:57:31.201146107 (52h0m21.138804606s) [noise=true , address=77.90.31.16:4733 , piece_id=DMASMGOUATLPEKCGLHKNNIMMFYVVGSULNUOMJU6RJY3ULBDOY6KQ, seg_pos={0 0}, upload=49973]: Done uploading piece. err: <nil> Aug 1 21:57:31.201833732 (52h0m21.139492231s) [noise=true , address=77.90.31.19:4843 , piece_id=UFONZYHL7EAFYE5PRW32RKUVQPMZPXBEFLLIFJMAB2TXJ3C6KKNA, seg_pos={0 0}, upload=49973]: Done uploading piece. err: <nil> Aug 1 21:57:31.211007613 (52h0m21.148666112s) [noise=true , address=45.91.116.9:4262 , piece_id=A3F4L3RR6WPFMBFFKLATNGG3CASBIPWQP5O3G5URFNTXBE3ZW3VA, seg_pos={0 0}, upload=49973]: Done uploading piece. err: <nil> Aug 1 21:57:31.215929169 (52h0m21.153587668s) [noise=true , address=77.90.31.17:4771 , piece_id=Y3WSALJX23CWPPYQ3XYMCFOSBLBX2E2JQCCS7EOEYNBWY4CDRT3Q, seg_pos={0 0}, upload=49973]: Done uploading piece. err: <nil> Aug 1 21:57:31.222973975 (52h0m21.160632474s) [noise=true , address=77.90.31.21:4878 , piece_id=BLNTYBOAIZWISSUJYK6BSBPTSNL6QWLGEN6AP6P4NPDVSURYG5AA, seg_pos={0 0}, upload=49973]: Done uploading piece. err: <nil> Aug 1 21:57:31.234392441 (52h0m21.17205094s) [noise=true , address=45.91.116.12:4225 , piece_id=PBQPLA5ZYRFPSKHDKCEFOEBS5T3GYMP2FLSFDQWJLVY756KPLJNQ, seg_pos={0 0}, upload=49973]: Done uploading piece. err: <nil> Aug 1 21:57:31.284794305 (52h0m21.222452804s) [noise=true , address=45.91.116.11:4118 , piece_id=AKZIOAR6NRNO7JWCQZIKNSL7IUH5S4IG3KPG3IY2FAKG74DH3E3Q, seg_pos={0 0}, upload=49973]: Done uploading piece. err: <nil> Aug 1 21:57:58.202420932 (52h0m48.140079471s) [noise=true , address=104.36.82.6:4875 , piece_id=CS5AMB7CIBQWWOU57Z3RCCSNVKIPE4B5QWU5YIQYI2UCWH7PAPVA, seg_pos={0 0}, upload=49973]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:104.36.82.6:4875): protocol: write tcp 188.166.225.170:40166->104.36.82.6:4875: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:58.202441131 (52h0m48.14009963s) [noise=true , address=104.36.82.5:4840 , piece_id=UYVKVFCJJP7XZYRIVPS4ABGEMDUMNR7YW4QGG5ZFTFCGNCJO3CSQ, seg_pos={0 0}, upload=49973]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:104.36.82.5:4840): protocol: rpc: dial tcp 104.36.82.5:4840: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:58.202448445 (52h0m48.140106954s) [noise=true , address=104.36.82.7:4909 , piece_id=QHJCVBT6KTU34K2KWFNACWYVVE3WJ2RU76BHVSKL4HLCZEFXZYPA, seg_pos={0 0}, upload=49973]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:104.36.82.7:4909): protocol: rpc: dial tcp 104.36.82.7:4909: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:58.202496487 (52h0m48.140155026s) [noise=true , address=104.36.82.4:4802 , piece_id=CLM2G33M6AK5HIESRW6LZY33NKHIR2PGITWJ23NPILRCA6UYIEHA, seg_pos={0 0}, upload=49973]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: write tcp 188.166.225.170:47696->104.36.82.4:4802: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:58.202524240 (52h0m48.140182739s) [seg_pos={0 0}, upload=49973]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Aug 1 21:57:58.435983932 (52h0m48.373642431s) [upload=49973]: Done uploading segments... Aug 1 21:57:58.436007667 (52h0m48.373666166s) [upload=49973]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.844177688s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 3 out of 60 is failed (threshold=3) [....X.......X.....X.........................................] duration: 1.902243903s FAIL (2025-08-01 22:38:32 UTC) orbiter warning: Operation: ListObjects unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/1K Duration: 1.902271607s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 3 out of 60 is failed (threshold=3) [.....X.......X.....X........................................] duration: 2.134268946s FAIL (2025-08-01 22:38:15 UTC) orbiter warning: Operation: ListObjects unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/1K Duration: 2.134293184s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [XXX.........................................................] duration: 5.477636546s FAIL (2025-08-01 22:38:14 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.477678486s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 5 out of 60 is failed (threshold=3) [XXXX........................X...............................] duration: 5.855117931s FAIL (2025-08-01 22:38:00 UTC) orbiter warning: Operation: sync2.Copy digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/4M):: after 92e23610b5d37ceb174fa119b5e47214ac2cd632235616ccf3b7f0b13a4a6031 != before 92313f7b7503eb9340206bbda9efbdb260cf41b185fb20ecbdf772c5fb959aac Duration: 5.855141186s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [..XXXX......................................................] duration: 5.944629441s FAIL (2025-08-01 22:37:50 UTC) orbiter warning: Operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 7) ======LOGS====== K4BW277ZSGXJYQDA66OMEBVDC7DAWSOJAPX42HANNGRZN35KHQ, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.481101510 (51h59m34.418760009s) [noise=true , address=104.36.82.5:4842 , piece_id=LV7YAOMPE2AV7TF2WPWR7QJNLT4JZAIPSVZN3E23U5KDBNKZFD6A, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.481422003 (51h59m34.419080502s) [noise=true , address=79.127.205.235:4683 , piece_id=3QHDNAPG4GRLJFVVNJNFXNGKTGKUEMWWUBMW62HHE5N5LJDO4KWA, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.482030897 (51h59m34.419689396s) [noise=true , address=79.127.219.51:4213 , piece_id=L2WSPI5BEIFLGQKJIK4EWXFQXNQRQ7HVKPOWFYRPPUGNAL5DVYWA, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.482285215 (51h59m34.419943704s) [noise=true , address=104.36.82.14:4251 , piece_id=O7NADU347E2Q26TPD3IUCK6WE5UVRITC6ALZHBE5SWYTKWXFBMXA, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.482518370 (51h59m34.420176859s) [noise=true , address=79.127.220.98:4402 , piece_id=DVLSYRFFZ6UT3QRSVAN7AMXVYZNI6PWG2IITHLRUUJATJVK3QOGA, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.482724304 (51h59m34.420382803s) [noise=true , address=79.127.201.214:4297 , piece_id=MPTMGVEYC4SATIN4ANCR2ELVRXHYWRDLMHWQW7HM6UOIV2N6R7ZA, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.482928555 (51h59m34.420587054s) [noise=true , address=104.36.82.37:4680 , piece_id=X4PDUE4WXUUWX6JQVO4NWPQCBMHMHC5YHPIHDQVYYOI2FXV2GVJQ, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.483134960 (51h59m34.420793449s) [noise=true , address=104.36.82.50:4027 , piece_id=OW7BODU5N7ZEEGE2FHSLNYVJT2Z25LY3J42TZEOR5SP2EWIN6X7A, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.483338480 (51h59m34.420996979s) [noise=true , address=104.36.82.55:4204 , piece_id=R5CYXWAVV4ZEMCV7LF4D4XDC4G6KLVBONAH2TH5EIEVL7QHUTRWQ, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.483582847 (51h59m34.421241356s) [noise=true , address=104.36.82.11:4134 , piece_id=RMTNNNC2VD7MGWJDAMT6ZCF3GR43SHANFVAGJ6EHVH7WO5XA6VTA, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.483833737 (51h59m34.421492236s) [noise=true , address=104.36.82.43:4856 , piece_id=2SINBLHGRUOWUO4XHBNBG45YE3PW7RNF3QSNVCQXNCFEFHRBFNJQ, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.484021896 (51h59m34.421680385s) [noise=true , address=104.36.82.52:4101 , piece_id=XYPLQM7QITCCX7T4NKIXP5MZ7I6F24EJJTG37K4NPOBKTUFHULRQ, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.484301662 (51h59m34.421960182s) [noise=true , address=79.127.201.215:4352 , piece_id=I6BRYI5XIACAAAAAE7HQOBCF4OTALMOVPPNBWQCRSH2MM6WUUURQ, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.484712097 (51h59m34.422370596s) [noise=true , address=64.49.13.138:4484 , piece_id=DZ5Y4HTS6BKW2DJ5CVOB64TW4SXB6DNCJEGFUJPLAUXUEHRZRTUA, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:45.576021470 (51h59m35.513679959s) [upload=49960]: Next returned nil segment Aug 1 21:56:45.576036609 (51h59m35.513695098s) [upload=49960]: Waiting for error group managing segments... Aug 1 21:56:45.860378776 (51h59m35.798037275s) [noise=true , address=64.49.13.138:4484 , piece_id=DZ5Y4HTS6BKW2DJ5CVOB64TW4SXB6DNCJEGFUJPLAUXUEHRZRTUA, seg_pos={0 0}, upload=49960]: Done uploading piece. err: <nil> Aug 1 21:56:45.884184188 (51h59m35.821842687s) [noise=true , address=79.127.205.235:4683 , piece_id=3QHDNAPG4GRLJFVVNJNFXNGKTGKUEMWWUBMW62HHE5N5LJDO4KWA, seg_pos={0 0}, upload=49960]: Done uploading piece. err: <nil> Aug 1 21:56:46.145493612 (51h59m36.083152101s) [noise=true , address=79.127.220.98:4402 , piece_id=DVLSYRFFZ6UT3QRSVAN7AMXVYZNI6PWG2IITHLRUUJATJVK3QOGA, seg_pos={0 0}, upload=49960]: Done uploading piece. err: <nil> Aug 1 21:56:46.257505313 (51h59m36.195163802s) [noise=true , address=79.127.220.97:4371 , piece_id=5WGPRFKQETIHTSMA42AM2C3UGCFYJ3DMMPAI2L7DD6E2P37KJKNA, seg_pos={0 0}, upload=49960]: Done uploading piece. err: <nil> Aug 1 21:56:46.334426181 (51h59m36.27208468s) [noise=true , address=79.127.201.215:4352 , piece_id=I6BRYI5XIACAAAAAE7HQOBCF4OTALMOVPPNBWQCRSH2MM6WUUURQ, seg_pos={0 0}, upload=49960]: Done uploading piece. err: <nil> Aug 1 21:56:46.386119275 (51h59m36.323777774s) [noise=true , address=79.127.201.214:4297 , piece_id=MPTMGVEYC4SATIN4ANCR2ELVRXHYWRDLMHWQW7HM6UOIV2N6R7ZA, seg_pos={0 0}, upload=49960]: Done uploading piece. err: <nil> Aug 1 21:56:46.395362048 (51h59m36.333020547s) [noise=true , address=79.127.219.51:4213 , piece_id=L2WSPI5BEIFLGQKJIK4EWXFQXNQRQ7HVKPOWFYRPPUGNAL5DVYWA, seg_pos={0 0}, upload=49960]: Done uploading piece. err: <nil> Aug 1 21:57:43.392434383 (52h0m33.330092872s) [noise=true , address=104.36.82.37:4680 , piece_id=X4PDUE4WXUUWX6JQVO4NWPQCBMHMHC5YHPIHDQVYYOI2FXV2GVJQ, seg_pos={0 0}, upload=49960]: Done uploading piece. err: ecclient: upload failed (node:1bXAqpxGhUvzwYEDGBqYRhZdTeMJcV3TYfngkKTCumpt9Nupi3, address:104.36.82.37:4680): protocol: rpc: dial tcp 104.36.82.37:4680: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:43.392477967 (52h0m33.330136466s) [noise=true , address=104.36.82.11:4134 , piece_id=RMTNNNC2VD7MGWJDAMT6ZCF3GR43SHANFVAGJ6EHVH7WO5XA6VTA, seg_pos={0 0}, upload=49960]: Done uploading piece. err: ecclient: upload failed (node:12R3Z5AgkBmWXMv5XgRyfcyqkG2gmyJs9ogT24kcLy6nuB9bzUG, address:104.36.82.11:4134): protocol: rpc: dial tcp 104.36.82.11:4134: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:43.392514086 (52h0m33.330172585s) [noise=true , address=104.36.82.55:4204 , piece_id=R5CYXWAVV4ZEMCV7LF4D4XDC4G6KLVBONAH2TH5EIEVL7QHUTRWQ, seg_pos={0 0}, upload=49960]: Done uploading piece. err: ecclient: upload failed (node:1rBZH6aLHsnWh22uNzaMiMwmLJCDMn4bYbNC5jRG7KCEuRfqpH, address:104.36.82.55:4204): protocol: write tcp 188.166.225.170:48550->104.36.82.55:4204: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:43.392546537 (52h0m33.330205027s) [noise=true , address=104.36.82.59:4327 , piece_id=OZDEYIY6B4WYZULRDRL2UBOJH2JQNY3YWSITZKWDO7GTT72AN5NQ, seg_pos={0 0}, upload=49960]: Done uploading piece. err: ecclient: upload failed (node:1V4C6AWGbyYQWfinoSj7iKovHeTJ9hqypyC5U9rZ8pveXBBfCY, address:104.36.82.59:4327): protocol: rpc: dial tcp 104.36.82.59:4327: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:43.392579040 (52h0m33.33023754s) [noise=true , address=104.36.82.5:4842 , piece_id=LV7YAOMPE2AV7TF2WPWR7QJNLT4JZAIPSVZN3E23U5KDBNKZFD6A, seg_pos={0 0}, upload=49960]: Done uploading piece. err: ecclient: upload failed (node:12q4VdT5BcjUb16PBYHK6zrnW8dqENKrcGx9zdLtNefDVB1MUSn, address:104.36.82.5:4842): protocol: context deadline exceeded; context deadline exceeded Aug 1 21:57:43.392627953 (52h0m33.330286442s) [noise=true , address=104.36.82.52:4101 , piece_id=XYPLQM7QITCCX7T4NKIXP5MZ7I6F24EJJTG37K4NPOBKTUFHULRQ, seg_pos={0 0}, upload=49960]: Done uploading piece. err: ecclient: upload failed (node:12WWXeBjFZR9PzciNfs35fFcx87N9GwDpMsPHnksFDugtaBGWp4, address:104.36.82.52:4101): protocol: rpc: dial tcp 104.36.82.52:4101: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:43.392653192 (52h0m33.330311691s) [noise=true , address=104.36.82.14:4251 , piece_id=O7NADU347E2Q26TPD3IUCK6WE5UVRITC6ALZHBE5SWYTKWXFBMXA, seg_pos={0 0}, upload=49960]: Done uploading piece. err: ecclient: upload failed (node:12btf6Z38aTMx7MaBxhUyZtFqUPgeMtP2aU8Y6FbV8zwu57zArd, address:104.36.82.14:4251): protocol: rpc: dial tcp 104.36.82.14:4251: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:43.392670836 (52h0m33.330329335s) [noise=true , address=104.36.82.12:4166 , piece_id=6ZIWCOBY76GJ6DHZF5G23U4IYTOFB5SMQW5TW3YITYH227PV2FGA, seg_pos={0 0}, upload=49960]: Done uploading piece. err: ecclient: upload failed (node:12iCB7c4ZfpqafpD1PtpbE6BfU81wJW65LmrPVJXtD58yeuoR9t, address:104.36.82.12:4166): protocol: rpc: dial tcp 104.36.82.12:4166: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:43.392698018 (52h0m33.330356517s) [noise=true , address=104.36.82.50:4027 , piece_id=OW7BODU5N7ZEEGE2FHSLNYVJT2Z25LY3J42TZEOR5SP2EWIN6X7A, seg_pos={0 0}, upload=49960]: Done uploading piece. err: ecclient: upload failed (node:1zEQhyuVxSHwdDUgDfcTog2W3oLBx4z9bYUMHpaaoLU1BTU9DQ, address:104.36.82.50:4027): protocol: write tcp 188.166.225.170:37784->104.36.82.50:4027: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:43.392715381 (52h0m33.33037387s) [noise=true , address=104.36.82.43:4856 , piece_id=2SINBLHGRUOWUO4XHBNBG45YE3PW7RNF3QSNVCQXNCFEFHRBFNJQ, seg_pos={0 0}, upload=49960]: Done uploading piece. err: ecclient: upload failed (node:12bQxwBfPEGtmMp6mtnWKsoZt3eMEjLdyYFahETWGHeKJ7BhkiZ, address:104.36.82.43:4856): protocol: rpc: dial tcp 104.36.82.43:4856: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:43.392730059 (52h0m33.330388558s) [noise=true , address=104.36.82.10:4082 , piece_id=45K4BW277ZSGXJYQDA66OMEBVDC7DAWSOJAPX42HANNGRZN35KHQ, seg_pos={0 0}, upload=49960]: Done uploading piece. err: ecclient: upload failed (node:12DKsNCtqxLvpJwLszdLs74V3eb8NH2VAV9V3bngzUczawmNoLK, address:104.36.82.10:4082): protocol: write tcp 188.166.225.170:46638->104.36.82.10:4082: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:43.392753424 (52h0m33.330411923s) [seg_pos={0 0}, upload=49960]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil> Aug 1 21:57:43.624550997 (52h0m33.562209566s) [upload=49960]: Done uploading segments... Aug 1 21:57:43.624582868 (52h0m33.562241407s) [upload=49960]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7) Duration: 5.944685889s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 4 out of 60 is failed (threshold=3) [X.....X.......X.....X.......................................] duration: 1.848605587s FAIL (2025-08-01 22:37:48 UTC) orbiter warning: Operation: ListObjects unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/1K Duration: 1.848634284s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [.XXX........................................................] duration: 5.646926078s FAIL (2025-08-01 22:37:41 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.646953892s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 6 out of 60 is failed (threshold=3) [XXXXX........................X..............................] duration: 6.019792551s FAIL (2025-08-01 22:37:37 UTC) orbiter warning: Operation: sync2.Copy digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/4M):: after 92e23610b5d37ceb174fa119b5e47214ac2cd632235616ccf3b7f0b13a4a6031 != before 92313f7b7503eb9340206bbda9efbdb260cf41b185fb20ecbdf772c5fb959aac Duration: 6.019822007s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [...XX.X.X...................................................] duration: 5.691275841s FAIL (2025-08-01 22:37:31 UTC) orbiter warning: Operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 7NK72474VWE4E3YOBCDZ5O5ZOH2MSA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB): protocol: write tcp 188.166.225.170:38368->104.36.82.12:4151: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:56:50.418604753 (51h59m40.356263252s) [seg_pos={0 0}, upload=49961]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 1 21:56:50.418680879 (51h59m40.356339378s) [upload=49961]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 1 21:56:50.418685998 (51h59m40.356344487s) [upload=49961]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 1 21:56:50.726563872 (51h59m40.664222371s) [upload=49961]: Done uploading segments... Aug 1 21:56:50.726586586 (51h59m40.664245085s) [upload=49961]: Upload finished. err: <nil> Aug 1 21:57:28.852869669 (52h0m18.790528158s) [upload=49973]: Starting upload Aug 1 21:57:28.852942389 (52h0m18.790600888s) [upload=49973]: Done starting upload Aug 1 21:57:28.853007122 (52h0m18.790665611s) [upload=49973]: Uploading segments... Aug 1 21:57:28.853174002 (52h0m18.790832491s) [upload=49973]: Got next segment. Inline: false Aug 1 21:57:28.853208168 (52h0m18.790866657s) [upload=49973]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 1 21:57:28.853212666 (52h0m18.790871165s) [upload=49973]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 1 21:57:29.095087519 (52h0m19.032746008s) [seg_pos={0 0}, upload=49973]: Begin upload segment... Aug 1 21:57:29.095129960 (52h0m19.032788459s) [seg_pos={0 0}, upload=49973]: Done begin upload segment. Aug 1 21:57:29.095164997 (52h0m19.032823496s) [noise=true , address=77.90.31.21:4878 , piece_id=BLNTYBOAIZWISSUJYK6BSBPTSNL6QWLGEN6AP6P4NPDVSURYG5AA, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.095171800 (52h0m19.032830289s) [noise=true , address=104.36.82.6:4875 , piece_id=CS5AMB7CIBQWWOU57Z3RCCSNVKIPE4B5QWU5YIQYI2UCWH7PAPVA, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.095510027 (52h0m19.033168526s) [noise=true , address=77.90.31.18:4806 , piece_id=52B6CAHGLFVTFIDQHHHAWKCBP2WJJY7OTTGRZZ7CEUH3HDA43Q6A, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.095544193 (52h0m19.033202692s) [noise=true , address=77.90.31.10:4518 , piece_id=WUOVXUSDG6YQTDPWG4YO7VLJNZZVLH6VRGMWOJABG2EJRU7XHP5A, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.095774283 (52h0m19.033432782s) [noise=true , address=104.36.82.7:4909 , piece_id=QHJCVBT6KTU34K2KWFNACWYVVE3WJ2RU76BHVSKL4HLCZEFXZYPA, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.095819880 (52h0m19.033478379s) [noise=true , address=45.91.116.12:4225 , piece_id=PBQPLA5ZYRFPSKHDKCEFOEBS5T3GYMP2FLSFDQWJLVY756KPLJNQ, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.096021656 (52h0m19.033680155s) [noise=true , address=45.91.116.13:4153 , piece_id=PMLKMZO66KXF3ZONO5ZZOQZICFK5XNGL6MN5FFBKICVFU5AWU6PQ, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.096069618 (52h0m19.033728117s) [noise=true , address=45.91.116.9:4262 , piece_id=A3F4L3RR6WPFMBFFKLATNGG3CASBIPWQP5O3G5URFNTXBE3ZW3VA, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.096348041 (52h0m19.03400654s) [noise=true , address=104.36.82.5:4840 , piece_id=UYVKVFCJJP7XZYRIVPS4ABGEMDUMNR7YW4QGG5ZFTFCGNCJO3CSQ, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.096354803 (52h0m19.034013292s) [noise=true , address=77.90.31.12:4590 , piece_id=D6FUBIFCQJSDRGWEP5GG34AQPNB3UROCYOATD6SRTJ7KAYVMFCOA, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.096608579 (52h0m19.034267078s) [noise=true , address=77.90.31.16:4733 , piece_id=DMASMGOUATLPEKCGLHKNNIMMFYVVGSULNUOMJU6RJY3ULBDOY6KQ, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.096830774 (52h0m19.034489273s) [noise=true , address=104.36.82.4:4802 , piece_id=CLM2G33M6AK5HIESRW6LZY33NKHIR2PGITWJ23NPILRCA6UYIEHA, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.097107233 (52h0m19.034765732s) [noise=true , address=77.90.31.19:4843 , piece_id=UFONZYHL7EAFYE5PRW32RKUVQPMZPXBEFLLIFJMAB2TXJ3C6KKNA, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.097324328 (52h0m19.034982827s) [noise=true , address=77.90.31.17:4771 , piece_id=Y3WSALJX23CWPPYQ3XYMCFOSBLBX2E2JQCCS7EOEYNBWY4CDRT3Q, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.098682446 (52h0m19.036340945s) [noise=true , address=77.90.31.15:4698 , piece_id=TN4OTPUD47FD6WOLZBEHNDI43WY6QVO5JK2JNJXNVM2IT6XMGERQ, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.099041603 (52h0m19.036700102s) [noise=true , address=45.91.116.11:4118 , piece_id=AKZIOAR6NRNO7JWCQZIKNSL7IUH5S4IG3KPG3IY2FAKG74DH3E3Q, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:30.718412255 (52h0m20.656070754s) [upload=49973]: Next returned nil segment Aug 1 21:57:30.718429869 (52h0m20.656088368s) [upload=49973]: Waiting for error group managing segments... Aug 1 21:57:31.107386675 (52h0m21.045045164s) [noise=true , address=45.91.116.13:4153 , piece_id=PMLKMZO66KXF3ZONO5ZZOQZICFK5XNGL6MN5FFBKICVFU5AWU6PQ, seg_pos={0 0}, upload=49973]: Done uploading piece. err: <nil> Aug 1 21:57:31.108529412 (52h0m21.046187911s) [noise=true , address=77.90.31.12:4590 , piece_id=D6FUBIFCQJSDRGWEP5GG34AQPNB3UROCYOATD6SRTJ7KAYVMFCOA, seg_pos={0 0}, upload=49973]: Done uploading piece. err: <nil> Aug 1 21:57:31.109975858 (52h0m21.047634347s) [noise=true , address=77.90.31.15:4698 , piece_id=TN4OTPUD47FD6WOLZBEHNDI43WY6QVO5JK2JNJXNVM2IT6XMGERQ, seg_pos={0 0}, upload=49973]: Done uploading piece. err: <nil> Aug 1 21:57:31.115273013 (52h0m21.052931502s) [noise=true , address=77.90.31.10:4518 , piece_id=WUOVXUSDG6YQTDPWG4YO7VLJNZZVLH6VRGMWOJABG2EJRU7XHP5A, seg_pos={0 0}, upload=49973]: Done uploading piece. err: <nil> Aug 1 21:57:31.116767281 (52h0m21.054425781s) [noise=true , address=77.90.31.18:4806 , piece_id=52B6CAHGLFVTFIDQHHHAWKCBP2WJJY7OTTGRZZ7CEUH3HDA43Q6A, seg_pos={0 0}, upload=49973]: Done uploading piece. err: <nil> Aug 1 21:57:31.201146107 (52h0m21.138804606s) [noise=true , address=77.90.31.16:4733 , piece_id=DMASMGOUATLPEKCGLHKNNIMMFYVVGSULNUOMJU6RJY3ULBDOY6KQ, seg_pos={0 0}, upload=49973]: Done uploading piece. err: <nil> Aug 1 21:57:31.201833732 (52h0m21.139492231s) [noise=true , address=77.90.31.19:4843 , piece_id=UFONZYHL7EAFYE5PRW32RKUVQPMZPXBEFLLIFJMAB2TXJ3C6KKNA, seg_pos={0 0}, upload=49973]: Done uploading piece. err: <nil> Aug 1 21:57:31.211007613 (52h0m21.148666112s) [noise=true , address=45.91.116.9:4262 , piece_id=A3F4L3RR6WPFMBFFKLATNGG3CASBIPWQP5O3G5URFNTXBE3ZW3VA, seg_pos={0 0}, upload=49973]: Done uploading piece. err: <nil> Aug 1 21:57:31.215929169 (52h0m21.153587668s) [noise=true , address=77.90.31.17:4771 , piece_id=Y3WSALJX23CWPPYQ3XYMCFOSBLBX2E2JQCCS7EOEYNBWY4CDRT3Q, seg_pos={0 0}, upload=49973]: Done uploading piece. err: <nil> Aug 1 21:57:31.222973975 (52h0m21.160632474s) [noise=true , address=77.90.31.21:4878 , piece_id=BLNTYBOAIZWISSUJYK6BSBPTSNL6QWLGEN6AP6P4NPDVSURYG5AA, seg_pos={0 0}, upload=49973]: Done uploading piece. err: <nil> Aug 1 21:57:31.234392441 (52h0m21.17205094s) [noise=true , address=45.91.116.12:4225 , piece_id=PBQPLA5ZYRFPSKHDKCEFOEBS5T3GYMP2FLSFDQWJLVY756KPLJNQ, seg_pos={0 0}, upload=49973]: Done uploading piece. err: <nil> Aug 1 21:57:31.284794305 (52h0m21.222452804s) [noise=true , address=45.91.116.11:4118 , piece_id=AKZIOAR6NRNO7JWCQZIKNSL7IUH5S4IG3KPG3IY2FAKG74DH3E3Q, seg_pos={0 0}, upload=49973]: Done uploading piece. err: <nil> Aug 1 21:57:58.202420932 (52h0m48.140079471s) [noise=true , address=104.36.82.6:4875 , piece_id=CS5AMB7CIBQWWOU57Z3RCCSNVKIPE4B5QWU5YIQYI2UCWH7PAPVA, seg_pos={0 0}, upload=49973]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:104.36.82.6:4875): protocol: write tcp 188.166.225.170:40166->104.36.82.6:4875: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:58.202441131 (52h0m48.14009963s) [noise=true , address=104.36.82.5:4840 , piece_id=UYVKVFCJJP7XZYRIVPS4ABGEMDUMNR7YW4QGG5ZFTFCGNCJO3CSQ, seg_pos={0 0}, upload=49973]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:104.36.82.5:4840): protocol: rpc: dial tcp 104.36.82.5:4840: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:58.202448445 (52h0m48.140106954s) [noise=true , address=104.36.82.7:4909 , piece_id=QHJCVBT6KTU34K2KWFNACWYVVE3WJ2RU76BHVSKL4HLCZEFXZYPA, seg_pos={0 0}, upload=49973]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:104.36.82.7:4909): protocol: rpc: dial tcp 104.36.82.7:4909: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:58.202496487 (52h0m48.140155026s) [noise=true , address=104.36.82.4:4802 , piece_id=CLM2G33M6AK5HIESRW6LZY33NKHIR2PGITWJ23NPILRCA6UYIEHA, seg_pos={0 0}, upload=49973]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: write tcp 188.166.225.170:47696->104.36.82.4:4802: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:58.202524240 (52h0m48.140182739s) [seg_pos={0 0}, upload=49973]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Aug 1 21:57:58.435983932 (52h0m48.373642431s) [upload=49973]: Done uploading segments... Aug 1 21:57:58.436007667 (52h0m48.373666166s) [upload=49973]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.69130147s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 6.154713498s FAIL (2025-08-01 22:37:30 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.154734017s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [....XX.X.X..................................................] duration: 5.954412519s FAIL (2025-08-01 22:37:03 UTC) orbiter warning: Operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 7NK72474VWE4E3YOBCDZ5O5ZOH2MSA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB): protocol: write tcp 188.166.225.170:38368->104.36.82.12:4151: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:56:50.418604753 (51h59m40.356263252s) [seg_pos={0 0}, upload=49961]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 1 21:56:50.418680879 (51h59m40.356339378s) [upload=49961]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 1 21:56:50.418685998 (51h59m40.356344487s) [upload=49961]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 1 21:56:50.726563872 (51h59m40.664222371s) [upload=49961]: Done uploading segments... Aug 1 21:56:50.726586586 (51h59m40.664245085s) [upload=49961]: Upload finished. err: <nil> Aug 1 21:57:28.852869669 (52h0m18.790528158s) [upload=49973]: Starting upload Aug 1 21:57:28.852942389 (52h0m18.790600888s) [upload=49973]: Done starting upload Aug 1 21:57:28.853007122 (52h0m18.790665611s) [upload=49973]: Uploading segments... Aug 1 21:57:28.853174002 (52h0m18.790832491s) [upload=49973]: Got next segment. Inline: false Aug 1 21:57:28.853208168 (52h0m18.790866657s) [upload=49973]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 1 21:57:28.853212666 (52h0m18.790871165s) [upload=49973]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 1 21:57:29.095087519 (52h0m19.032746008s) [seg_pos={0 0}, upload=49973]: Begin upload segment... Aug 1 21:57:29.095129960 (52h0m19.032788459s) [seg_pos={0 0}, upload=49973]: Done begin upload segment. Aug 1 21:57:29.095164997 (52h0m19.032823496s) [noise=true , address=77.90.31.21:4878 , piece_id=BLNTYBOAIZWISSUJYK6BSBPTSNL6QWLGEN6AP6P4NPDVSURYG5AA, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.095171800 (52h0m19.032830289s) [noise=true , address=104.36.82.6:4875 , piece_id=CS5AMB7CIBQWWOU57Z3RCCSNVKIPE4B5QWU5YIQYI2UCWH7PAPVA, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.095510027 (52h0m19.033168526s) [noise=true , address=77.90.31.18:4806 , piece_id=52B6CAHGLFVTFIDQHHHAWKCBP2WJJY7OTTGRZZ7CEUH3HDA43Q6A, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.095544193 (52h0m19.033202692s) [noise=true , address=77.90.31.10:4518 , piece_id=WUOVXUSDG6YQTDPWG4YO7VLJNZZVLH6VRGMWOJABG2EJRU7XHP5A, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.095774283 (52h0m19.033432782s) [noise=true , address=104.36.82.7:4909 , piece_id=QHJCVBT6KTU34K2KWFNACWYVVE3WJ2RU76BHVSKL4HLCZEFXZYPA, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.095819880 (52h0m19.033478379s) [noise=true , address=45.91.116.12:4225 , piece_id=PBQPLA5ZYRFPSKHDKCEFOEBS5T3GYMP2FLSFDQWJLVY756KPLJNQ, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.096021656 (52h0m19.033680155s) [noise=true , address=45.91.116.13:4153 , piece_id=PMLKMZO66KXF3ZONO5ZZOQZICFK5XNGL6MN5FFBKICVFU5AWU6PQ, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.096069618 (52h0m19.033728117s) [noise=true , address=45.91.116.9:4262 , piece_id=A3F4L3RR6WPFMBFFKLATNGG3CASBIPWQP5O3G5URFNTXBE3ZW3VA, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.096348041 (52h0m19.03400654s) [noise=true , address=104.36.82.5:4840 , piece_id=UYVKVFCJJP7XZYRIVPS4ABGEMDUMNR7YW4QGG5ZFTFCGNCJO3CSQ, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.096354803 (52h0m19.034013292s) [noise=true , address=77.90.31.12:4590 , piece_id=D6FUBIFCQJSDRGWEP5GG34AQPNB3UROCYOATD6SRTJ7KAYVMFCOA, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.096608579 (52h0m19.034267078s) [noise=true , address=77.90.31.16:4733 , piece_id=DMASMGOUATLPEKCGLHKNNIMMFYVVGSULNUOMJU6RJY3ULBDOY6KQ, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.096830774 (52h0m19.034489273s) [noise=true , address=104.36.82.4:4802 , piece_id=CLM2G33M6AK5HIESRW6LZY33NKHIR2PGITWJ23NPILRCA6UYIEHA, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.097107233 (52h0m19.034765732s) [noise=true , address=77.90.31.19:4843 , piece_id=UFONZYHL7EAFYE5PRW32RKUVQPMZPXBEFLLIFJMAB2TXJ3C6KKNA, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.097324328 (52h0m19.034982827s) [noise=true , address=77.90.31.17:4771 , piece_id=Y3WSALJX23CWPPYQ3XYMCFOSBLBX2E2JQCCS7EOEYNBWY4CDRT3Q, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.098682446 (52h0m19.036340945s) [noise=true , address=77.90.31.15:4698 , piece_id=TN4OTPUD47FD6WOLZBEHNDI43WY6QVO5JK2JNJXNVM2IT6XMGERQ, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.099041603 (52h0m19.036700102s) [noise=true , address=45.91.116.11:4118 , piece_id=AKZIOAR6NRNO7JWCQZIKNSL7IUH5S4IG3KPG3IY2FAKG74DH3E3Q, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:30.718412255 (52h0m20.656070754s) [upload=49973]: Next returned nil segment Aug 1 21:57:30.718429869 (52h0m20.656088368s) [upload=49973]: Waiting for error group managing segments... Aug 1 21:57:31.107386675 (52h0m21.045045164s) [noise=true , address=45.91.116.13:4153 , piece_id=PMLKMZO66KXF3ZONO5ZZOQZICFK5XNGL6MN5FFBKICVFU5AWU6PQ, seg_pos={0 0}, upload=49973]: Done uploading piece. err: <nil> Aug 1 21:57:31.108529412 (52h0m21.046187911s) [noise=true , address=77.90.31.12:4590 , piece_id=D6FUBIFCQJSDRGWEP5GG34AQPNB3UROCYOATD6SRTJ7KAYVMFCOA, seg_pos={0 0}, upload=49973]: Done uploading piece. err: <nil> Aug 1 21:57:31.109975858 (52h0m21.047634347s) [noise=true , address=77.90.31.15:4698 , piece_id=TN4OTPUD47FD6WOLZBEHNDI43WY6QVO5JK2JNJXNVM2IT6XMGERQ, seg_pos={0 0}, upload=49973]: Done uploading piece. err: <nil> Aug 1 21:57:31.115273013 (52h0m21.052931502s) [noise=true , address=77.90.31.10:4518 , piece_id=WUOVXUSDG6YQTDPWG4YO7VLJNZZVLH6VRGMWOJABG2EJRU7XHP5A, seg_pos={0 0}, upload=49973]: Done uploading piece. err: <nil> Aug 1 21:57:31.116767281 (52h0m21.054425781s) [noise=true , address=77.90.31.18:4806 , piece_id=52B6CAHGLFVTFIDQHHHAWKCBP2WJJY7OTTGRZZ7CEUH3HDA43Q6A, seg_pos={0 0}, upload=49973]: Done uploading piece. err: <nil> Aug 1 21:57:31.201146107 (52h0m21.138804606s) [noise=true , address=77.90.31.16:4733 , piece_id=DMASMGOUATLPEKCGLHKNNIMMFYVVGSULNUOMJU6RJY3ULBDOY6KQ, seg_pos={0 0}, upload=49973]: Done uploading piece. err: <nil> Aug 1 21:57:31.201833732 (52h0m21.139492231s) [noise=true , address=77.90.31.19:4843 , piece_id=UFONZYHL7EAFYE5PRW32RKUVQPMZPXBEFLLIFJMAB2TXJ3C6KKNA, seg_pos={0 0}, upload=49973]: Done uploading piece. err: <nil> Aug 1 21:57:31.211007613 (52h0m21.148666112s) [noise=true , address=45.91.116.9:4262 , piece_id=A3F4L3RR6WPFMBFFKLATNGG3CASBIPWQP5O3G5URFNTXBE3ZW3VA, seg_pos={0 0}, upload=49973]: Done uploading piece. err: <nil> Aug 1 21:57:31.215929169 (52h0m21.153587668s) [noise=true , address=77.90.31.17:4771 , piece_id=Y3WSALJX23CWPPYQ3XYMCFOSBLBX2E2JQCCS7EOEYNBWY4CDRT3Q, seg_pos={0 0}, upload=49973]: Done uploading piece. err: <nil> Aug 1 21:57:31.222973975 (52h0m21.160632474s) [noise=true , address=77.90.31.21:4878 , piece_id=BLNTYBOAIZWISSUJYK6BSBPTSNL6QWLGEN6AP6P4NPDVSURYG5AA, seg_pos={0 0}, upload=49973]: Done uploading piece. err: <nil> Aug 1 21:57:31.234392441 (52h0m21.17205094s) [noise=true , address=45.91.116.12:4225 , piece_id=PBQPLA5ZYRFPSKHDKCEFOEBS5T3GYMP2FLSFDQWJLVY756KPLJNQ, seg_pos={0 0}, upload=49973]: Done uploading piece. err: <nil> Aug 1 21:57:31.284794305 (52h0m21.222452804s) [noise=true , address=45.91.116.11:4118 , piece_id=AKZIOAR6NRNO7JWCQZIKNSL7IUH5S4IG3KPG3IY2FAKG74DH3E3Q, seg_pos={0 0}, upload=49973]: Done uploading piece. err: <nil> Aug 1 21:57:58.202420932 (52h0m48.140079471s) [noise=true , address=104.36.82.6:4875 , piece_id=CS5AMB7CIBQWWOU57Z3RCCSNVKIPE4B5QWU5YIQYI2UCWH7PAPVA, seg_pos={0 0}, upload=49973]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:104.36.82.6:4875): protocol: write tcp 188.166.225.170:40166->104.36.82.6:4875: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:58.202441131 (52h0m48.14009963s) [noise=true , address=104.36.82.5:4840 , piece_id=UYVKVFCJJP7XZYRIVPS4ABGEMDUMNR7YW4QGG5ZFTFCGNCJO3CSQ, seg_pos={0 0}, upload=49973]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:104.36.82.5:4840): protocol: rpc: dial tcp 104.36.82.5:4840: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:58.202448445 (52h0m48.140106954s) [noise=true , address=104.36.82.7:4909 , piece_id=QHJCVBT6KTU34K2KWFNACWYVVE3WJ2RU76BHVSKL4HLCZEFXZYPA, seg_pos={0 0}, upload=49973]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:104.36.82.7:4909): protocol: rpc: dial tcp 104.36.82.7:4909: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:58.202496487 (52h0m48.140155026s) [noise=true , address=104.36.82.4:4802 , piece_id=CLM2G33M6AK5HIESRW6LZY33NKHIR2PGITWJ23NPILRCA6UYIEHA, seg_pos={0 0}, upload=49973]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: write tcp 188.166.225.170:47696->104.36.82.4:4802: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:58.202524240 (52h0m48.140182739s) [seg_pos={0 0}, upload=49973]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Aug 1 21:57:58.435983932 (52h0m48.373642431s) [upload=49973]: Done uploading segments... Aug 1 21:57:58.436007667 (52h0m48.373666166s) [upload=49973]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.954447296s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [...XXXX.....................................................] duration: 6.235275352s FAIL (2025-08-01 22:37:02 UTC) orbiter warning: Operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 7) ======LOGS====== K4BW277ZSGXJYQDA66OMEBVDC7DAWSOJAPX42HANNGRZN35KHQ, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.481101510 (51h59m34.418760009s) [noise=true , address=104.36.82.5:4842 , piece_id=LV7YAOMPE2AV7TF2WPWR7QJNLT4JZAIPSVZN3E23U5KDBNKZFD6A, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.481422003 (51h59m34.419080502s) [noise=true , address=79.127.205.235:4683 , piece_id=3QHDNAPG4GRLJFVVNJNFXNGKTGKUEMWWUBMW62HHE5N5LJDO4KWA, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.482030897 (51h59m34.419689396s) [noise=true , address=79.127.219.51:4213 , piece_id=L2WSPI5BEIFLGQKJIK4EWXFQXNQRQ7HVKPOWFYRPPUGNAL5DVYWA, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.482285215 (51h59m34.419943704s) [noise=true , address=104.36.82.14:4251 , piece_id=O7NADU347E2Q26TPD3IUCK6WE5UVRITC6ALZHBE5SWYTKWXFBMXA, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.482518370 (51h59m34.420176859s) [noise=true , address=79.127.220.98:4402 , piece_id=DVLSYRFFZ6UT3QRSVAN7AMXVYZNI6PWG2IITHLRUUJATJVK3QOGA, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.482724304 (51h59m34.420382803s) [noise=true , address=79.127.201.214:4297 , piece_id=MPTMGVEYC4SATIN4ANCR2ELVRXHYWRDLMHWQW7HM6UOIV2N6R7ZA, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.482928555 (51h59m34.420587054s) [noise=true , address=104.36.82.37:4680 , piece_id=X4PDUE4WXUUWX6JQVO4NWPQCBMHMHC5YHPIHDQVYYOI2FXV2GVJQ, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.483134960 (51h59m34.420793449s) [noise=true , address=104.36.82.50:4027 , piece_id=OW7BODU5N7ZEEGE2FHSLNYVJT2Z25LY3J42TZEOR5SP2EWIN6X7A, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.483338480 (51h59m34.420996979s) [noise=true , address=104.36.82.55:4204 , piece_id=R5CYXWAVV4ZEMCV7LF4D4XDC4G6KLVBONAH2TH5EIEVL7QHUTRWQ, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.483582847 (51h59m34.421241356s) [noise=true , address=104.36.82.11:4134 , piece_id=RMTNNNC2VD7MGWJDAMT6ZCF3GR43SHANFVAGJ6EHVH7WO5XA6VTA, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.483833737 (51h59m34.421492236s) [noise=true , address=104.36.82.43:4856 , piece_id=2SINBLHGRUOWUO4XHBNBG45YE3PW7RNF3QSNVCQXNCFEFHRBFNJQ, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.484021896 (51h59m34.421680385s) [noise=true , address=104.36.82.52:4101 , piece_id=XYPLQM7QITCCX7T4NKIXP5MZ7I6F24EJJTG37K4NPOBKTUFHULRQ, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.484301662 (51h59m34.421960182s) [noise=true , address=79.127.201.215:4352 , piece_id=I6BRYI5XIACAAAAAE7HQOBCF4OTALMOVPPNBWQCRSH2MM6WUUURQ, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.484712097 (51h59m34.422370596s) [noise=true , address=64.49.13.138:4484 , piece_id=DZ5Y4HTS6BKW2DJ5CVOB64TW4SXB6DNCJEGFUJPLAUXUEHRZRTUA, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:45.576021470 (51h59m35.513679959s) [upload=49960]: Next returned nil segment Aug 1 21:56:45.576036609 (51h59m35.513695098s) [upload=49960]: Waiting for error group managing segments... Aug 1 21:56:45.860378776 (51h59m35.798037275s) [noise=true , address=64.49.13.138:4484 , piece_id=DZ5Y4HTS6BKW2DJ5CVOB64TW4SXB6DNCJEGFUJPLAUXUEHRZRTUA, seg_pos={0 0}, upload=49960]: Done uploading piece. err: <nil> Aug 1 21:56:45.884184188 (51h59m35.821842687s) [noise=true , address=79.127.205.235:4683 , piece_id=3QHDNAPG4GRLJFVVNJNFXNGKTGKUEMWWUBMW62HHE5N5LJDO4KWA, seg_pos={0 0}, upload=49960]: Done uploading piece. err: <nil> Aug 1 21:56:46.145493612 (51h59m36.083152101s) [noise=true , address=79.127.220.98:4402 , piece_id=DVLSYRFFZ6UT3QRSVAN7AMXVYZNI6PWG2IITHLRUUJATJVK3QOGA, seg_pos={0 0}, upload=49960]: Done uploading piece. err: <nil> Aug 1 21:56:46.257505313 (51h59m36.195163802s) [noise=true , address=79.127.220.97:4371 , piece_id=5WGPRFKQETIHTSMA42AM2C3UGCFYJ3DMMPAI2L7DD6E2P37KJKNA, seg_pos={0 0}, upload=49960]: Done uploading piece. err: <nil> Aug 1 21:56:46.334426181 (51h59m36.27208468s) [noise=true , address=79.127.201.215:4352 , piece_id=I6BRYI5XIACAAAAAE7HQOBCF4OTALMOVPPNBWQCRSH2MM6WUUURQ, seg_pos={0 0}, upload=49960]: Done uploading piece. err: <nil> Aug 1 21:56:46.386119275 (51h59m36.323777774s) [noise=true , address=79.127.201.214:4297 , piece_id=MPTMGVEYC4SATIN4ANCR2ELVRXHYWRDLMHWQW7HM6UOIV2N6R7ZA, seg_pos={0 0}, upload=49960]: Done uploading piece. err: <nil> Aug 1 21:56:46.395362048 (51h59m36.333020547s) [noise=true , address=79.127.219.51:4213 , piece_id=L2WSPI5BEIFLGQKJIK4EWXFQXNQRQ7HVKPOWFYRPPUGNAL5DVYWA, seg_pos={0 0}, upload=49960]: Done uploading piece. err: <nil> Aug 1 21:57:43.392434383 (52h0m33.330092872s) [noise=true , address=104.36.82.37:4680 , piece_id=X4PDUE4WXUUWX6JQVO4NWPQCBMHMHC5YHPIHDQVYYOI2FXV2GVJQ, seg_pos={0 0}, upload=49960]: Done uploading piece. err: ecclient: upload failed (node:1bXAqpxGhUvzwYEDGBqYRhZdTeMJcV3TYfngkKTCumpt9Nupi3, address:104.36.82.37:4680): protocol: rpc: dial tcp 104.36.82.37:4680: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:43.392477967 (52h0m33.330136466s) [noise=true , address=104.36.82.11:4134 , piece_id=RMTNNNC2VD7MGWJDAMT6ZCF3GR43SHANFVAGJ6EHVH7WO5XA6VTA, seg_pos={0 0}, upload=49960]: Done uploading piece. err: ecclient: upload failed (node:12R3Z5AgkBmWXMv5XgRyfcyqkG2gmyJs9ogT24kcLy6nuB9bzUG, address:104.36.82.11:4134): protocol: rpc: dial tcp 104.36.82.11:4134: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:43.392514086 (52h0m33.330172585s) [noise=true , address=104.36.82.55:4204 , piece_id=R5CYXWAVV4ZEMCV7LF4D4XDC4G6KLVBONAH2TH5EIEVL7QHUTRWQ, seg_pos={0 0}, upload=49960]: Done uploading piece. err: ecclient: upload failed (node:1rBZH6aLHsnWh22uNzaMiMwmLJCDMn4bYbNC5jRG7KCEuRfqpH, address:104.36.82.55:4204): protocol: write tcp 188.166.225.170:48550->104.36.82.55:4204: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:43.392546537 (52h0m33.330205027s) [noise=true , address=104.36.82.59:4327 , piece_id=OZDEYIY6B4WYZULRDRL2UBOJH2JQNY3YWSITZKWDO7GTT72AN5NQ, seg_pos={0 0}, upload=49960]: Done uploading piece. err: ecclient: upload failed (node:1V4C6AWGbyYQWfinoSj7iKovHeTJ9hqypyC5U9rZ8pveXBBfCY, address:104.36.82.59:4327): protocol: rpc: dial tcp 104.36.82.59:4327: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:43.392579040 (52h0m33.33023754s) [noise=true , address=104.36.82.5:4842 , piece_id=LV7YAOMPE2AV7TF2WPWR7QJNLT4JZAIPSVZN3E23U5KDBNKZFD6A, seg_pos={0 0}, upload=49960]: Done uploading piece. err: ecclient: upload failed (node:12q4VdT5BcjUb16PBYHK6zrnW8dqENKrcGx9zdLtNefDVB1MUSn, address:104.36.82.5:4842): protocol: context deadline exceeded; context deadline exceeded Aug 1 21:57:43.392627953 (52h0m33.330286442s) [noise=true , address=104.36.82.52:4101 , piece_id=XYPLQM7QITCCX7T4NKIXP5MZ7I6F24EJJTG37K4NPOBKTUFHULRQ, seg_pos={0 0}, upload=49960]: Done uploading piece. err: ecclient: upload failed (node:12WWXeBjFZR9PzciNfs35fFcx87N9GwDpMsPHnksFDugtaBGWp4, address:104.36.82.52:4101): protocol: rpc: dial tcp 104.36.82.52:4101: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:43.392653192 (52h0m33.330311691s) [noise=true , address=104.36.82.14:4251 , piece_id=O7NADU347E2Q26TPD3IUCK6WE5UVRITC6ALZHBE5SWYTKWXFBMXA, seg_pos={0 0}, upload=49960]: Done uploading piece. err: ecclient: upload failed (node:12btf6Z38aTMx7MaBxhUyZtFqUPgeMtP2aU8Y6FbV8zwu57zArd, address:104.36.82.14:4251): protocol: rpc: dial tcp 104.36.82.14:4251: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:43.392670836 (52h0m33.330329335s) [noise=true , address=104.36.82.12:4166 , piece_id=6ZIWCOBY76GJ6DHZF5G23U4IYTOFB5SMQW5TW3YITYH227PV2FGA, seg_pos={0 0}, upload=49960]: Done uploading piece. err: ecclient: upload failed (node:12iCB7c4ZfpqafpD1PtpbE6BfU81wJW65LmrPVJXtD58yeuoR9t, address:104.36.82.12:4166): protocol: rpc: dial tcp 104.36.82.12:4166: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:43.392698018 (52h0m33.330356517s) [noise=true , address=104.36.82.50:4027 , piece_id=OW7BODU5N7ZEEGE2FHSLNYVJT2Z25LY3J42TZEOR5SP2EWIN6X7A, seg_pos={0 0}, upload=49960]: Done uploading piece. err: ecclient: upload failed (node:1zEQhyuVxSHwdDUgDfcTog2W3oLBx4z9bYUMHpaaoLU1BTU9DQ, address:104.36.82.50:4027): protocol: write tcp 188.166.225.170:37784->104.36.82.50:4027: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:43.392715381 (52h0m33.33037387s) [noise=true , address=104.36.82.43:4856 , piece_id=2SINBLHGRUOWUO4XHBNBG45YE3PW7RNF3QSNVCQXNCFEFHRBFNJQ, seg_pos={0 0}, upload=49960]: Done uploading piece. err: ecclient: upload failed (node:12bQxwBfPEGtmMp6mtnWKsoZt3eMEjLdyYFahETWGHeKJ7BhkiZ, address:104.36.82.43:4856): protocol: rpc: dial tcp 104.36.82.43:4856: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:43.392730059 (52h0m33.330388558s) [noise=true , address=104.36.82.10:4082 , piece_id=45K4BW277ZSGXJYQDA66OMEBVDC7DAWSOJAPX42HANNGRZN35KHQ, seg_pos={0 0}, upload=49960]: Done uploading piece. err: ecclient: upload failed (node:12DKsNCtqxLvpJwLszdLs74V3eb8NH2VAV9V3bngzUczawmNoLK, address:104.36.82.10:4082): protocol: write tcp 188.166.225.170:46638->104.36.82.10:4082: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:43.392753424 (52h0m33.330411923s) [seg_pos={0 0}, upload=49960]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil> Aug 1 21:57:43.624550997 (52h0m33.562209566s) [upload=49960]: Done uploading segments... Aug 1 21:57:43.624582868 (52h0m33.562241407s) [upload=49960]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7) Duration: 6.235299669s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [..XXX.......................................................] duration: 5.460234882s FAIL (2025-08-01 22:36:54 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.460258707s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 7 out of 60 is failed (threshold=3) [XXXXXX........................X.............................] duration: 6.025911096s FAIL (2025-08-01 22:36:46 UTC) orbiter warning: Operation: sync2.Copy digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/4M):: after 92e23610b5d37ceb174fa119b5e47214ac2cd632235616ccf3b7f0b13a4a6031 != before 92313f7b7503eb9340206bbda9efbdb260cf41b185fb20ecbdf772c5fb959aac Duration: 6.025943199s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 4 out of 60 is failed (threshold=3) [.X.....X.......X.....X......................................] duration: 2.013766951s FAIL (2025-08-01 22:36:46 UTC) orbiter warning: Operation: ListObjects unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/1K Duration: 2.013859209s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [....XXXX....................................................] duration: 5.820279876s FAIL (2025-08-01 22:36:30 UTC) orbiter warning: Operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 7) ======LOGS====== K4BW277ZSGXJYQDA66OMEBVDC7DAWSOJAPX42HANNGRZN35KHQ, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.481101510 (51h59m34.418760009s) [noise=true , address=104.36.82.5:4842 , piece_id=LV7YAOMPE2AV7TF2WPWR7QJNLT4JZAIPSVZN3E23U5KDBNKZFD6A, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.481422003 (51h59m34.419080502s) [noise=true , address=79.127.205.235:4683 , piece_id=3QHDNAPG4GRLJFVVNJNFXNGKTGKUEMWWUBMW62HHE5N5LJDO4KWA, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.482030897 (51h59m34.419689396s) [noise=true , address=79.127.219.51:4213 , piece_id=L2WSPI5BEIFLGQKJIK4EWXFQXNQRQ7HVKPOWFYRPPUGNAL5DVYWA, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.482285215 (51h59m34.419943704s) [noise=true , address=104.36.82.14:4251 , piece_id=O7NADU347E2Q26TPD3IUCK6WE5UVRITC6ALZHBE5SWYTKWXFBMXA, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.482518370 (51h59m34.420176859s) [noise=true , address=79.127.220.98:4402 , piece_id=DVLSYRFFZ6UT3QRSVAN7AMXVYZNI6PWG2IITHLRUUJATJVK3QOGA, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.482724304 (51h59m34.420382803s) [noise=true , address=79.127.201.214:4297 , piece_id=MPTMGVEYC4SATIN4ANCR2ELVRXHYWRDLMHWQW7HM6UOIV2N6R7ZA, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.482928555 (51h59m34.420587054s) [noise=true , address=104.36.82.37:4680 , piece_id=X4PDUE4WXUUWX6JQVO4NWPQCBMHMHC5YHPIHDQVYYOI2FXV2GVJQ, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.483134960 (51h59m34.420793449s) [noise=true , address=104.36.82.50:4027 , piece_id=OW7BODU5N7ZEEGE2FHSLNYVJT2Z25LY3J42TZEOR5SP2EWIN6X7A, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.483338480 (51h59m34.420996979s) [noise=true , address=104.36.82.55:4204 , piece_id=R5CYXWAVV4ZEMCV7LF4D4XDC4G6KLVBONAH2TH5EIEVL7QHUTRWQ, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.483582847 (51h59m34.421241356s) [noise=true , address=104.36.82.11:4134 , piece_id=RMTNNNC2VD7MGWJDAMT6ZCF3GR43SHANFVAGJ6EHVH7WO5XA6VTA, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.483833737 (51h59m34.421492236s) [noise=true , address=104.36.82.43:4856 , piece_id=2SINBLHGRUOWUO4XHBNBG45YE3PW7RNF3QSNVCQXNCFEFHRBFNJQ, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.484021896 (51h59m34.421680385s) [noise=true , address=104.36.82.52:4101 , piece_id=XYPLQM7QITCCX7T4NKIXP5MZ7I6F24EJJTG37K4NPOBKTUFHULRQ, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.484301662 (51h59m34.421960182s) [noise=true , address=79.127.201.215:4352 , piece_id=I6BRYI5XIACAAAAAE7HQOBCF4OTALMOVPPNBWQCRSH2MM6WUUURQ, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.484712097 (51h59m34.422370596s) [noise=true , address=64.49.13.138:4484 , piece_id=DZ5Y4HTS6BKW2DJ5CVOB64TW4SXB6DNCJEGFUJPLAUXUEHRZRTUA, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:45.576021470 (51h59m35.513679959s) [upload=49960]: Next returned nil segment Aug 1 21:56:45.576036609 (51h59m35.513695098s) [upload=49960]: Waiting for error group managing segments... Aug 1 21:56:45.860378776 (51h59m35.798037275s) [noise=true , address=64.49.13.138:4484 , piece_id=DZ5Y4HTS6BKW2DJ5CVOB64TW4SXB6DNCJEGFUJPLAUXUEHRZRTUA, seg_pos={0 0}, upload=49960]: Done uploading piece. err: <nil> Aug 1 21:56:45.884184188 (51h59m35.821842687s) [noise=true , address=79.127.205.235:4683 , piece_id=3QHDNAPG4GRLJFVVNJNFXNGKTGKUEMWWUBMW62HHE5N5LJDO4KWA, seg_pos={0 0}, upload=49960]: Done uploading piece. err: <nil> Aug 1 21:56:46.145493612 (51h59m36.083152101s) [noise=true , address=79.127.220.98:4402 , piece_id=DVLSYRFFZ6UT3QRSVAN7AMXVYZNI6PWG2IITHLRUUJATJVK3QOGA, seg_pos={0 0}, upload=49960]: Done uploading piece. err: <nil> Aug 1 21:56:46.257505313 (51h59m36.195163802s) [noise=true , address=79.127.220.97:4371 , piece_id=5WGPRFKQETIHTSMA42AM2C3UGCFYJ3DMMPAI2L7DD6E2P37KJKNA, seg_pos={0 0}, upload=49960]: Done uploading piece. err: <nil> Aug 1 21:56:46.334426181 (51h59m36.27208468s) [noise=true , address=79.127.201.215:4352 , piece_id=I6BRYI5XIACAAAAAE7HQOBCF4OTALMOVPPNBWQCRSH2MM6WUUURQ, seg_pos={0 0}, upload=49960]: Done uploading piece. err: <nil> Aug 1 21:56:46.386119275 (51h59m36.323777774s) [noise=true , address=79.127.201.214:4297 , piece_id=MPTMGVEYC4SATIN4ANCR2ELVRXHYWRDLMHWQW7HM6UOIV2N6R7ZA, seg_pos={0 0}, upload=49960]: Done uploading piece. err: <nil> Aug 1 21:56:46.395362048 (51h59m36.333020547s) [noise=true , address=79.127.219.51:4213 , piece_id=L2WSPI5BEIFLGQKJIK4EWXFQXNQRQ7HVKPOWFYRPPUGNAL5DVYWA, seg_pos={0 0}, upload=49960]: Done uploading piece. err: <nil> Aug 1 21:57:43.392434383 (52h0m33.330092872s) [noise=true , address=104.36.82.37:4680 , piece_id=X4PDUE4WXUUWX6JQVO4NWPQCBMHMHC5YHPIHDQVYYOI2FXV2GVJQ, seg_pos={0 0}, upload=49960]: Done uploading piece. err: ecclient: upload failed (node:1bXAqpxGhUvzwYEDGBqYRhZdTeMJcV3TYfngkKTCumpt9Nupi3, address:104.36.82.37:4680): protocol: rpc: dial tcp 104.36.82.37:4680: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:43.392477967 (52h0m33.330136466s) [noise=true , address=104.36.82.11:4134 , piece_id=RMTNNNC2VD7MGWJDAMT6ZCF3GR43SHANFVAGJ6EHVH7WO5XA6VTA, seg_pos={0 0}, upload=49960]: Done uploading piece. err: ecclient: upload failed (node:12R3Z5AgkBmWXMv5XgRyfcyqkG2gmyJs9ogT24kcLy6nuB9bzUG, address:104.36.82.11:4134): protocol: rpc: dial tcp 104.36.82.11:4134: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:43.392514086 (52h0m33.330172585s) [noise=true , address=104.36.82.55:4204 , piece_id=R5CYXWAVV4ZEMCV7LF4D4XDC4G6KLVBONAH2TH5EIEVL7QHUTRWQ, seg_pos={0 0}, upload=49960]: Done uploading piece. err: ecclient: upload failed (node:1rBZH6aLHsnWh22uNzaMiMwmLJCDMn4bYbNC5jRG7KCEuRfqpH, address:104.36.82.55:4204): protocol: write tcp 188.166.225.170:48550->104.36.82.55:4204: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:43.392546537 (52h0m33.330205027s) [noise=true , address=104.36.82.59:4327 , piece_id=OZDEYIY6B4WYZULRDRL2UBOJH2JQNY3YWSITZKWDO7GTT72AN5NQ, seg_pos={0 0}, upload=49960]: Done uploading piece. err: ecclient: upload failed (node:1V4C6AWGbyYQWfinoSj7iKovHeTJ9hqypyC5U9rZ8pveXBBfCY, address:104.36.82.59:4327): protocol: rpc: dial tcp 104.36.82.59:4327: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:43.392579040 (52h0m33.33023754s) [noise=true , address=104.36.82.5:4842 , piece_id=LV7YAOMPE2AV7TF2WPWR7QJNLT4JZAIPSVZN3E23U5KDBNKZFD6A, seg_pos={0 0}, upload=49960]: Done uploading piece. err: ecclient: upload failed (node:12q4VdT5BcjUb16PBYHK6zrnW8dqENKrcGx9zdLtNefDVB1MUSn, address:104.36.82.5:4842): protocol: context deadline exceeded; context deadline exceeded Aug 1 21:57:43.392627953 (52h0m33.330286442s) [noise=true , address=104.36.82.52:4101 , piece_id=XYPLQM7QITCCX7T4NKIXP5MZ7I6F24EJJTG37K4NPOBKTUFHULRQ, seg_pos={0 0}, upload=49960]: Done uploading piece. err: ecclient: upload failed (node:12WWXeBjFZR9PzciNfs35fFcx87N9GwDpMsPHnksFDugtaBGWp4, address:104.36.82.52:4101): protocol: rpc: dial tcp 104.36.82.52:4101: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:43.392653192 (52h0m33.330311691s) [noise=true , address=104.36.82.14:4251 , piece_id=O7NADU347E2Q26TPD3IUCK6WE5UVRITC6ALZHBE5SWYTKWXFBMXA, seg_pos={0 0}, upload=49960]: Done uploading piece. err: ecclient: upload failed (node:12btf6Z38aTMx7MaBxhUyZtFqUPgeMtP2aU8Y6FbV8zwu57zArd, address:104.36.82.14:4251): protocol: rpc: dial tcp 104.36.82.14:4251: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:43.392670836 (52h0m33.330329335s) [noise=true , address=104.36.82.12:4166 , piece_id=6ZIWCOBY76GJ6DHZF5G23U4IYTOFB5SMQW5TW3YITYH227PV2FGA, seg_pos={0 0}, upload=49960]: Done uploading piece. err: ecclient: upload failed (node:12iCB7c4ZfpqafpD1PtpbE6BfU81wJW65LmrPVJXtD58yeuoR9t, address:104.36.82.12:4166): protocol: rpc: dial tcp 104.36.82.12:4166: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:43.392698018 (52h0m33.330356517s) [noise=true , address=104.36.82.50:4027 , piece_id=OW7BODU5N7ZEEGE2FHSLNYVJT2Z25LY3J42TZEOR5SP2EWIN6X7A, seg_pos={0 0}, upload=49960]: Done uploading piece. err: ecclient: upload failed (node:1zEQhyuVxSHwdDUgDfcTog2W3oLBx4z9bYUMHpaaoLU1BTU9DQ, address:104.36.82.50:4027): protocol: write tcp 188.166.225.170:37784->104.36.82.50:4027: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:43.392715381 (52h0m33.33037387s) [noise=true , address=104.36.82.43:4856 , piece_id=2SINBLHGRUOWUO4XHBNBG45YE3PW7RNF3QSNVCQXNCFEFHRBFNJQ, seg_pos={0 0}, upload=49960]: Done uploading piece. err: ecclient: upload failed (node:12bQxwBfPEGtmMp6mtnWKsoZt3eMEjLdyYFahETWGHeKJ7BhkiZ, address:104.36.82.43:4856): protocol: rpc: dial tcp 104.36.82.43:4856: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:43.392730059 (52h0m33.330388558s) [noise=true , address=104.36.82.10:4082 , piece_id=45K4BW277ZSGXJYQDA66OMEBVDC7DAWSOJAPX42HANNGRZN35KHQ, seg_pos={0 0}, upload=49960]: Done uploading piece. err: ecclient: upload failed (node:12DKsNCtqxLvpJwLszdLs74V3eb8NH2VAV9V3bngzUczawmNoLK, address:104.36.82.10:4082): protocol: write tcp 188.166.225.170:46638->104.36.82.10:4082: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:43.392753424 (52h0m33.330411923s) [seg_pos={0 0}, upload=49960]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil> Aug 1 21:57:43.624550997 (52h0m33.562209566s) [upload=49960]: Done uploading segments... Aug 1 21:57:43.624582868 (52h0m33.562241407s) [upload=49960]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7) Duration: 5.820304713s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 6.093456283s FAIL (2025-08-01 22:36:24 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.093475621s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 8 out of 60 is failed (threshold=3) [XXXXXXX........................X............................] duration: 5.895561573s FAIL (2025-08-01 22:36:13 UTC) orbiter warning: Operation: sync2.Copy digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/4M):: after 92e23610b5d37ceb174fa119b5e47214ac2cd632235616ccf3b7f0b13a4a6031 != before 92313f7b7503eb9340206bbda9efbdb260cf41b185fb20ecbdf772c5fb959aac Duration: 5.895579357s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [...XXX......................................................] duration: 5.754847255s FAIL (2025-08-01 22:36:07 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.754875208s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 5.865951451s FAIL (2025-08-01 22:36:05 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.865967331s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.....XX.X.X.................................................] duration: 5.858338163s FAIL (2025-08-01 22:36:03 UTC) orbiter warning: Operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 7NK72474VWE4E3YOBCDZ5O5ZOH2MSA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB): protocol: write tcp 188.166.225.170:38368->104.36.82.12:4151: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:56:50.418604753 (51h59m40.356263252s) [seg_pos={0 0}, upload=49961]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 1 21:56:50.418680879 (51h59m40.356339378s) [upload=49961]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 1 21:56:50.418685998 (51h59m40.356344487s) [upload=49961]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 1 21:56:50.726563872 (51h59m40.664222371s) [upload=49961]: Done uploading segments... Aug 1 21:56:50.726586586 (51h59m40.664245085s) [upload=49961]: Upload finished. err: <nil> Aug 1 21:57:28.852869669 (52h0m18.790528158s) [upload=49973]: Starting upload Aug 1 21:57:28.852942389 (52h0m18.790600888s) [upload=49973]: Done starting upload Aug 1 21:57:28.853007122 (52h0m18.790665611s) [upload=49973]: Uploading segments... Aug 1 21:57:28.853174002 (52h0m18.790832491s) [upload=49973]: Got next segment. Inline: false Aug 1 21:57:28.853208168 (52h0m18.790866657s) [upload=49973]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 1 21:57:28.853212666 (52h0m18.790871165s) [upload=49973]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 1 21:57:29.095087519 (52h0m19.032746008s) [seg_pos={0 0}, upload=49973]: Begin upload segment... Aug 1 21:57:29.095129960 (52h0m19.032788459s) [seg_pos={0 0}, upload=49973]: Done begin upload segment. Aug 1 21:57:29.095164997 (52h0m19.032823496s) [noise=true , address=77.90.31.21:4878 , piece_id=BLNTYBOAIZWISSUJYK6BSBPTSNL6QWLGEN6AP6P4NPDVSURYG5AA, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.095171800 (52h0m19.032830289s) [noise=true , address=104.36.82.6:4875 , piece_id=CS5AMB7CIBQWWOU57Z3RCCSNVKIPE4B5QWU5YIQYI2UCWH7PAPVA, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.095510027 (52h0m19.033168526s) [noise=true , address=77.90.31.18:4806 , piece_id=52B6CAHGLFVTFIDQHHHAWKCBP2WJJY7OTTGRZZ7CEUH3HDA43Q6A, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.095544193 (52h0m19.033202692s) [noise=true , address=77.90.31.10:4518 , piece_id=WUOVXUSDG6YQTDPWG4YO7VLJNZZVLH6VRGMWOJABG2EJRU7XHP5A, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.095774283 (52h0m19.033432782s) [noise=true , address=104.36.82.7:4909 , piece_id=QHJCVBT6KTU34K2KWFNACWYVVE3WJ2RU76BHVSKL4HLCZEFXZYPA, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.095819880 (52h0m19.033478379s) [noise=true , address=45.91.116.12:4225 , piece_id=PBQPLA5ZYRFPSKHDKCEFOEBS5T3GYMP2FLSFDQWJLVY756KPLJNQ, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.096021656 (52h0m19.033680155s) [noise=true , address=45.91.116.13:4153 , piece_id=PMLKMZO66KXF3ZONO5ZZOQZICFK5XNGL6MN5FFBKICVFU5AWU6PQ, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.096069618 (52h0m19.033728117s) [noise=true , address=45.91.116.9:4262 , piece_id=A3F4L3RR6WPFMBFFKLATNGG3CASBIPWQP5O3G5URFNTXBE3ZW3VA, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.096348041 (52h0m19.03400654s) [noise=true , address=104.36.82.5:4840 , piece_id=UYVKVFCJJP7XZYRIVPS4ABGEMDUMNR7YW4QGG5ZFTFCGNCJO3CSQ, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.096354803 (52h0m19.034013292s) [noise=true , address=77.90.31.12:4590 , piece_id=D6FUBIFCQJSDRGWEP5GG34AQPNB3UROCYOATD6SRTJ7KAYVMFCOA, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.096608579 (52h0m19.034267078s) [noise=true , address=77.90.31.16:4733 , piece_id=DMASMGOUATLPEKCGLHKNNIMMFYVVGSULNUOMJU6RJY3ULBDOY6KQ, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.096830774 (52h0m19.034489273s) [noise=true , address=104.36.82.4:4802 , piece_id=CLM2G33M6AK5HIESRW6LZY33NKHIR2PGITWJ23NPILRCA6UYIEHA, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.097107233 (52h0m19.034765732s) [noise=true , address=77.90.31.19:4843 , piece_id=UFONZYHL7EAFYE5PRW32RKUVQPMZPXBEFLLIFJMAB2TXJ3C6KKNA, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.097324328 (52h0m19.034982827s) [noise=true , address=77.90.31.17:4771 , piece_id=Y3WSALJX23CWPPYQ3XYMCFOSBLBX2E2JQCCS7EOEYNBWY4CDRT3Q, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.098682446 (52h0m19.036340945s) [noise=true , address=77.90.31.15:4698 , piece_id=TN4OTPUD47FD6WOLZBEHNDI43WY6QVO5JK2JNJXNVM2IT6XMGERQ, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.099041603 (52h0m19.036700102s) [noise=true , address=45.91.116.11:4118 , piece_id=AKZIOAR6NRNO7JWCQZIKNSL7IUH5S4IG3KPG3IY2FAKG74DH3E3Q, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:30.718412255 (52h0m20.656070754s) [upload=49973]: Next returned nil segment Aug 1 21:57:30.718429869 (52h0m20.656088368s) [upload=49973]: Waiting for error group managing segments... Aug 1 21:57:31.107386675 (52h0m21.045045164s) [noise=true , address=45.91.116.13:4153 , piece_id=PMLKMZO66KXF3ZONO5ZZOQZICFK5XNGL6MN5FFBKICVFU5AWU6PQ, seg_pos={0 0}, upload=49973]: Done uploading piece. err: <nil> Aug 1 21:57:31.108529412 (52h0m21.046187911s) [noise=true , address=77.90.31.12:4590 , piece_id=D6FUBIFCQJSDRGWEP5GG34AQPNB3UROCYOATD6SRTJ7KAYVMFCOA, seg_pos={0 0}, upload=49973]: Done uploading piece. err: <nil> Aug 1 21:57:31.109975858 (52h0m21.047634347s) [noise=true , address=77.90.31.15:4698 , piece_id=TN4OTPUD47FD6WOLZBEHNDI43WY6QVO5JK2JNJXNVM2IT6XMGERQ, seg_pos={0 0}, upload=49973]: Done uploading piece. err: <nil> Aug 1 21:57:31.115273013 (52h0m21.052931502s) [noise=true , address=77.90.31.10:4518 , piece_id=WUOVXUSDG6YQTDPWG4YO7VLJNZZVLH6VRGMWOJABG2EJRU7XHP5A, seg_pos={0 0}, upload=49973]: Done uploading piece. err: <nil> Aug 1 21:57:31.116767281 (52h0m21.054425781s) [noise=true , address=77.90.31.18:4806 , piece_id=52B6CAHGLFVTFIDQHHHAWKCBP2WJJY7OTTGRZZ7CEUH3HDA43Q6A, seg_pos={0 0}, upload=49973]: Done uploading piece. err: <nil> Aug 1 21:57:31.201146107 (52h0m21.138804606s) [noise=true , address=77.90.31.16:4733 , piece_id=DMASMGOUATLPEKCGLHKNNIMMFYVVGSULNUOMJU6RJY3ULBDOY6KQ, seg_pos={0 0}, upload=49973]: Done uploading piece. err: <nil> Aug 1 21:57:31.201833732 (52h0m21.139492231s) [noise=true , address=77.90.31.19:4843 , piece_id=UFONZYHL7EAFYE5PRW32RKUVQPMZPXBEFLLIFJMAB2TXJ3C6KKNA, seg_pos={0 0}, upload=49973]: Done uploading piece. err: <nil> Aug 1 21:57:31.211007613 (52h0m21.148666112s) [noise=true , address=45.91.116.9:4262 , piece_id=A3F4L3RR6WPFMBFFKLATNGG3CASBIPWQP5O3G5URFNTXBE3ZW3VA, seg_pos={0 0}, upload=49973]: Done uploading piece. err: <nil> Aug 1 21:57:31.215929169 (52h0m21.153587668s) [noise=true , address=77.90.31.17:4771 , piece_id=Y3WSALJX23CWPPYQ3XYMCFOSBLBX2E2JQCCS7EOEYNBWY4CDRT3Q, seg_pos={0 0}, upload=49973]: Done uploading piece. err: <nil> Aug 1 21:57:31.222973975 (52h0m21.160632474s) [noise=true , address=77.90.31.21:4878 , piece_id=BLNTYBOAIZWISSUJYK6BSBPTSNL6QWLGEN6AP6P4NPDVSURYG5AA, seg_pos={0 0}, upload=49973]: Done uploading piece. err: <nil> Aug 1 21:57:31.234392441 (52h0m21.17205094s) [noise=true , address=45.91.116.12:4225 , piece_id=PBQPLA5ZYRFPSKHDKCEFOEBS5T3GYMP2FLSFDQWJLVY756KPLJNQ, seg_pos={0 0}, upload=49973]: Done uploading piece. err: <nil> Aug 1 21:57:31.284794305 (52h0m21.222452804s) [noise=true , address=45.91.116.11:4118 , piece_id=AKZIOAR6NRNO7JWCQZIKNSL7IUH5S4IG3KPG3IY2FAKG74DH3E3Q, seg_pos={0 0}, upload=49973]: Done uploading piece. err: <nil> Aug 1 21:57:58.202420932 (52h0m48.140079471s) [noise=true , address=104.36.82.6:4875 , piece_id=CS5AMB7CIBQWWOU57Z3RCCSNVKIPE4B5QWU5YIQYI2UCWH7PAPVA, seg_pos={0 0}, upload=49973]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:104.36.82.6:4875): protocol: write tcp 188.166.225.170:40166->104.36.82.6:4875: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:58.202441131 (52h0m48.14009963s) [noise=true , address=104.36.82.5:4840 , piece_id=UYVKVFCJJP7XZYRIVPS4ABGEMDUMNR7YW4QGG5ZFTFCGNCJO3CSQ, seg_pos={0 0}, upload=49973]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:104.36.82.5:4840): protocol: rpc: dial tcp 104.36.82.5:4840: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:58.202448445 (52h0m48.140106954s) [noise=true , address=104.36.82.7:4909 , piece_id=QHJCVBT6KTU34K2KWFNACWYVVE3WJ2RU76BHVSKL4HLCZEFXZYPA, seg_pos={0 0}, upload=49973]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:104.36.82.7:4909): protocol: rpc: dial tcp 104.36.82.7:4909: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:58.202496487 (52h0m48.140155026s) [noise=true , address=104.36.82.4:4802 , piece_id=CLM2G33M6AK5HIESRW6LZY33NKHIR2PGITWJ23NPILRCA6UYIEHA, seg_pos={0 0}, upload=49973]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: write tcp 188.166.225.170:47696->104.36.82.4:4802: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:58.202524240 (52h0m48.140182739s) [seg_pos={0 0}, upload=49973]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Aug 1 21:57:58.435983932 (52h0m48.373642431s) [upload=49973]: Done uploading segments... Aug 1 21:57:58.436007667 (52h0m48.373666166s) [upload=49973]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.8583679s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [......XX.X.X................................................] duration: 5.781478225s FAIL (2025-08-01 22:35:38 UTC) orbiter warning: Operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 7NK72474VWE4E3YOBCDZ5O5ZOH2MSA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB): protocol: write tcp 188.166.225.170:38368->104.36.82.12:4151: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:56:50.418604753 (51h59m40.356263252s) [seg_pos={0 0}, upload=49961]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 1 21:56:50.418680879 (51h59m40.356339378s) [upload=49961]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 1 21:56:50.418685998 (51h59m40.356344487s) [upload=49961]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 1 21:56:50.726563872 (51h59m40.664222371s) [upload=49961]: Done uploading segments... Aug 1 21:56:50.726586586 (51h59m40.664245085s) [upload=49961]: Upload finished. err: <nil> Aug 1 21:57:28.852869669 (52h0m18.790528158s) [upload=49973]: Starting upload Aug 1 21:57:28.852942389 (52h0m18.790600888s) [upload=49973]: Done starting upload Aug 1 21:57:28.853007122 (52h0m18.790665611s) [upload=49973]: Uploading segments... Aug 1 21:57:28.853174002 (52h0m18.790832491s) [upload=49973]: Got next segment. Inline: false Aug 1 21:57:28.853208168 (52h0m18.790866657s) [upload=49973]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 1 21:57:28.853212666 (52h0m18.790871165s) [upload=49973]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 1 21:57:29.095087519 (52h0m19.032746008s) [seg_pos={0 0}, upload=49973]: Begin upload segment... Aug 1 21:57:29.095129960 (52h0m19.032788459s) [seg_pos={0 0}, upload=49973]: Done begin upload segment. Aug 1 21:57:29.095164997 (52h0m19.032823496s) [noise=true , address=77.90.31.21:4878 , piece_id=BLNTYBOAIZWISSUJYK6BSBPTSNL6QWLGEN6AP6P4NPDVSURYG5AA, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.095171800 (52h0m19.032830289s) [noise=true , address=104.36.82.6:4875 , piece_id=CS5AMB7CIBQWWOU57Z3RCCSNVKIPE4B5QWU5YIQYI2UCWH7PAPVA, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.095510027 (52h0m19.033168526s) [noise=true , address=77.90.31.18:4806 , piece_id=52B6CAHGLFVTFIDQHHHAWKCBP2WJJY7OTTGRZZ7CEUH3HDA43Q6A, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.095544193 (52h0m19.033202692s) [noise=true , address=77.90.31.10:4518 , piece_id=WUOVXUSDG6YQTDPWG4YO7VLJNZZVLH6VRGMWOJABG2EJRU7XHP5A, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.095774283 (52h0m19.033432782s) [noise=true , address=104.36.82.7:4909 , piece_id=QHJCVBT6KTU34K2KWFNACWYVVE3WJ2RU76BHVSKL4HLCZEFXZYPA, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.095819880 (52h0m19.033478379s) [noise=true , address=45.91.116.12:4225 , piece_id=PBQPLA5ZYRFPSKHDKCEFOEBS5T3GYMP2FLSFDQWJLVY756KPLJNQ, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.096021656 (52h0m19.033680155s) [noise=true , address=45.91.116.13:4153 , piece_id=PMLKMZO66KXF3ZONO5ZZOQZICFK5XNGL6MN5FFBKICVFU5AWU6PQ, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.096069618 (52h0m19.033728117s) [noise=true , address=45.91.116.9:4262 , piece_id=A3F4L3RR6WPFMBFFKLATNGG3CASBIPWQP5O3G5URFNTXBE3ZW3VA, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.096348041 (52h0m19.03400654s) [noise=true , address=104.36.82.5:4840 , piece_id=UYVKVFCJJP7XZYRIVPS4ABGEMDUMNR7YW4QGG5ZFTFCGNCJO3CSQ, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.096354803 (52h0m19.034013292s) [noise=true , address=77.90.31.12:4590 , piece_id=D6FUBIFCQJSDRGWEP5GG34AQPNB3UROCYOATD6SRTJ7KAYVMFCOA, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.096608579 (52h0m19.034267078s) [noise=true , address=77.90.31.16:4733 , piece_id=DMASMGOUATLPEKCGLHKNNIMMFYVVGSULNUOMJU6RJY3ULBDOY6KQ, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.096830774 (52h0m19.034489273s) [noise=true , address=104.36.82.4:4802 , piece_id=CLM2G33M6AK5HIESRW6LZY33NKHIR2PGITWJ23NPILRCA6UYIEHA, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.097107233 (52h0m19.034765732s) [noise=true , address=77.90.31.19:4843 , piece_id=UFONZYHL7EAFYE5PRW32RKUVQPMZPXBEFLLIFJMAB2TXJ3C6KKNA, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.097324328 (52h0m19.034982827s) [noise=true , address=77.90.31.17:4771 , piece_id=Y3WSALJX23CWPPYQ3XYMCFOSBLBX2E2JQCCS7EOEYNBWY4CDRT3Q, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.098682446 (52h0m19.036340945s) [noise=true , address=77.90.31.15:4698 , piece_id=TN4OTPUD47FD6WOLZBEHNDI43WY6QVO5JK2JNJXNVM2IT6XMGERQ, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.099041603 (52h0m19.036700102s) [noise=true , address=45.91.116.11:4118 , piece_id=AKZIOAR6NRNO7JWCQZIKNSL7IUH5S4IG3KPG3IY2FAKG74DH3E3Q, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:30.718412255 (52h0m20.656070754s) [upload=49973]: Next returned nil segment Aug 1 21:57:30.718429869 (52h0m20.656088368s) [upload=49973]: Waiting for error group managing segments... Aug 1 21:57:31.107386675 (52h0m21.045045164s) [noise=true , address=45.91.116.13:4153 , piece_id=PMLKMZO66KXF3ZONO5ZZOQZICFK5XNGL6MN5FFBKICVFU5AWU6PQ, seg_pos={0 0}, upload=49973]: Done uploading piece. err: <nil> Aug 1 21:57:31.108529412 (52h0m21.046187911s) [noise=true , address=77.90.31.12:4590 , piece_id=D6FUBIFCQJSDRGWEP5GG34AQPNB3UROCYOATD6SRTJ7KAYVMFCOA, seg_pos={0 0}, upload=49973]: Done uploading piece. err: <nil> Aug 1 21:57:31.109975858 (52h0m21.047634347s) [noise=true , address=77.90.31.15:4698 , piece_id=TN4OTPUD47FD6WOLZBEHNDI43WY6QVO5JK2JNJXNVM2IT6XMGERQ, seg_pos={0 0}, upload=49973]: Done uploading piece. err: <nil> Aug 1 21:57:31.115273013 (52h0m21.052931502s) [noise=true , address=77.90.31.10:4518 , piece_id=WUOVXUSDG6YQTDPWG4YO7VLJNZZVLH6VRGMWOJABG2EJRU7XHP5A, seg_pos={0 0}, upload=49973]: Done uploading piece. err: <nil> Aug 1 21:57:31.116767281 (52h0m21.054425781s) [noise=true , address=77.90.31.18:4806 , piece_id=52B6CAHGLFVTFIDQHHHAWKCBP2WJJY7OTTGRZZ7CEUH3HDA43Q6A, seg_pos={0 0}, upload=49973]: Done uploading piece. err: <nil> Aug 1 21:57:31.201146107 (52h0m21.138804606s) [noise=true , address=77.90.31.16:4733 , piece_id=DMASMGOUATLPEKCGLHKNNIMMFYVVGSULNUOMJU6RJY3ULBDOY6KQ, seg_pos={0 0}, upload=49973]: Done uploading piece. err: <nil> Aug 1 21:57:31.201833732 (52h0m21.139492231s) [noise=true , address=77.90.31.19:4843 , piece_id=UFONZYHL7EAFYE5PRW32RKUVQPMZPXBEFLLIFJMAB2TXJ3C6KKNA, seg_pos={0 0}, upload=49973]: Done uploading piece. err: <nil> Aug 1 21:57:31.211007613 (52h0m21.148666112s) [noise=true , address=45.91.116.9:4262 , piece_id=A3F4L3RR6WPFMBFFKLATNGG3CASBIPWQP5O3G5URFNTXBE3ZW3VA, seg_pos={0 0}, upload=49973]: Done uploading piece. err: <nil> Aug 1 21:57:31.215929169 (52h0m21.153587668s) [noise=true , address=77.90.31.17:4771 , piece_id=Y3WSALJX23CWPPYQ3XYMCFOSBLBX2E2JQCCS7EOEYNBWY4CDRT3Q, seg_pos={0 0}, upload=49973]: Done uploading piece. err: <nil> Aug 1 21:57:31.222973975 (52h0m21.160632474s) [noise=true , address=77.90.31.21:4878 , piece_id=BLNTYBOAIZWISSUJYK6BSBPTSNL6QWLGEN6AP6P4NPDVSURYG5AA, seg_pos={0 0}, upload=49973]: Done uploading piece. err: <nil> Aug 1 21:57:31.234392441 (52h0m21.17205094s) [noise=true , address=45.91.116.12:4225 , piece_id=PBQPLA5ZYRFPSKHDKCEFOEBS5T3GYMP2FLSFDQWJLVY756KPLJNQ, seg_pos={0 0}, upload=49973]: Done uploading piece. err: <nil> Aug 1 21:57:31.284794305 (52h0m21.222452804s) [noise=true , address=45.91.116.11:4118 , piece_id=AKZIOAR6NRNO7JWCQZIKNSL7IUH5S4IG3KPG3IY2FAKG74DH3E3Q, seg_pos={0 0}, upload=49973]: Done uploading piece. err: <nil> Aug 1 21:57:58.202420932 (52h0m48.140079471s) [noise=true , address=104.36.82.6:4875 , piece_id=CS5AMB7CIBQWWOU57Z3RCCSNVKIPE4B5QWU5YIQYI2UCWH7PAPVA, seg_pos={0 0}, upload=49973]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:104.36.82.6:4875): protocol: write tcp 188.166.225.170:40166->104.36.82.6:4875: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:58.202441131 (52h0m48.14009963s) [noise=true , address=104.36.82.5:4840 , piece_id=UYVKVFCJJP7XZYRIVPS4ABGEMDUMNR7YW4QGG5ZFTFCGNCJO3CSQ, seg_pos={0 0}, upload=49973]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:104.36.82.5:4840): protocol: rpc: dial tcp 104.36.82.5:4840: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:58.202448445 (52h0m48.140106954s) [noise=true , address=104.36.82.7:4909 , piece_id=QHJCVBT6KTU34K2KWFNACWYVVE3WJ2RU76BHVSKL4HLCZEFXZYPA, seg_pos={0 0}, upload=49973]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:104.36.82.7:4909): protocol: rpc: dial tcp 104.36.82.7:4909: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:58.202496487 (52h0m48.140155026s) [noise=true , address=104.36.82.4:4802 , piece_id=CLM2G33M6AK5HIESRW6LZY33NKHIR2PGITWJ23NPILRCA6UYIEHA, seg_pos={0 0}, upload=49973]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: write tcp 188.166.225.170:47696->104.36.82.4:4802: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:58.202524240 (52h0m48.140182739s) [seg_pos={0 0}, upload=49973]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Aug 1 21:57:58.435983932 (52h0m48.373642431s) [upload=49973]: Done uploading segments... Aug 1 21:57:58.436007667 (52h0m48.373666166s) [upload=49973]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.781522099s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 4 out of 60 is failed (threshold=3) [..X.....X.......X.....X.....................................] duration: 1.861868843s FAIL (2025-08-01 22:35:38 UTC) orbiter warning: Operation: ListObjects unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/1K Duration: 1.861895416s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [....XXX.....................................................] duration: 5.568439055s FAIL (2025-08-01 22:35:30 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.568458192s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [.....XXXX...................................................] duration: 5.845960678s FAIL (2025-08-01 22:35:23 UTC) orbiter warning: Operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 7) ======LOGS====== K4BW277ZSGXJYQDA66OMEBVDC7DAWSOJAPX42HANNGRZN35KHQ, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.481101510 (51h59m34.418760009s) [noise=true , address=104.36.82.5:4842 , piece_id=LV7YAOMPE2AV7TF2WPWR7QJNLT4JZAIPSVZN3E23U5KDBNKZFD6A, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.481422003 (51h59m34.419080502s) [noise=true , address=79.127.205.235:4683 , piece_id=3QHDNAPG4GRLJFVVNJNFXNGKTGKUEMWWUBMW62HHE5N5LJDO4KWA, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.482030897 (51h59m34.419689396s) [noise=true , address=79.127.219.51:4213 , piece_id=L2WSPI5BEIFLGQKJIK4EWXFQXNQRQ7HVKPOWFYRPPUGNAL5DVYWA, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.482285215 (51h59m34.419943704s) [noise=true , address=104.36.82.14:4251 , piece_id=O7NADU347E2Q26TPD3IUCK6WE5UVRITC6ALZHBE5SWYTKWXFBMXA, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.482518370 (51h59m34.420176859s) [noise=true , address=79.127.220.98:4402 , piece_id=DVLSYRFFZ6UT3QRSVAN7AMXVYZNI6PWG2IITHLRUUJATJVK3QOGA, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.482724304 (51h59m34.420382803s) [noise=true , address=79.127.201.214:4297 , piece_id=MPTMGVEYC4SATIN4ANCR2ELVRXHYWRDLMHWQW7HM6UOIV2N6R7ZA, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.482928555 (51h59m34.420587054s) [noise=true , address=104.36.82.37:4680 , piece_id=X4PDUE4WXUUWX6JQVO4NWPQCBMHMHC5YHPIHDQVYYOI2FXV2GVJQ, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.483134960 (51h59m34.420793449s) [noise=true , address=104.36.82.50:4027 , piece_id=OW7BODU5N7ZEEGE2FHSLNYVJT2Z25LY3J42TZEOR5SP2EWIN6X7A, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.483338480 (51h59m34.420996979s) [noise=true , address=104.36.82.55:4204 , piece_id=R5CYXWAVV4ZEMCV7LF4D4XDC4G6KLVBONAH2TH5EIEVL7QHUTRWQ, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.483582847 (51h59m34.421241356s) [noise=true , address=104.36.82.11:4134 , piece_id=RMTNNNC2VD7MGWJDAMT6ZCF3GR43SHANFVAGJ6EHVH7WO5XA6VTA, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.483833737 (51h59m34.421492236s) [noise=true , address=104.36.82.43:4856 , piece_id=2SINBLHGRUOWUO4XHBNBG45YE3PW7RNF3QSNVCQXNCFEFHRBFNJQ, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.484021896 (51h59m34.421680385s) [noise=true , address=104.36.82.52:4101 , piece_id=XYPLQM7QITCCX7T4NKIXP5MZ7I6F24EJJTG37K4NPOBKTUFHULRQ, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.484301662 (51h59m34.421960182s) [noise=true , address=79.127.201.215:4352 , piece_id=I6BRYI5XIACAAAAAE7HQOBCF4OTALMOVPPNBWQCRSH2MM6WUUURQ, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.484712097 (51h59m34.422370596s) [noise=true , address=64.49.13.138:4484 , piece_id=DZ5Y4HTS6BKW2DJ5CVOB64TW4SXB6DNCJEGFUJPLAUXUEHRZRTUA, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:45.576021470 (51h59m35.513679959s) [upload=49960]: Next returned nil segment Aug 1 21:56:45.576036609 (51h59m35.513695098s) [upload=49960]: Waiting for error group managing segments... Aug 1 21:56:45.860378776 (51h59m35.798037275s) [noise=true , address=64.49.13.138:4484 , piece_id=DZ5Y4HTS6BKW2DJ5CVOB64TW4SXB6DNCJEGFUJPLAUXUEHRZRTUA, seg_pos={0 0}, upload=49960]: Done uploading piece. err: <nil> Aug 1 21:56:45.884184188 (51h59m35.821842687s) [noise=true , address=79.127.205.235:4683 , piece_id=3QHDNAPG4GRLJFVVNJNFXNGKTGKUEMWWUBMW62HHE5N5LJDO4KWA, seg_pos={0 0}, upload=49960]: Done uploading piece. err: <nil> Aug 1 21:56:46.145493612 (51h59m36.083152101s) [noise=true , address=79.127.220.98:4402 , piece_id=DVLSYRFFZ6UT3QRSVAN7AMXVYZNI6PWG2IITHLRUUJATJVK3QOGA, seg_pos={0 0}, upload=49960]: Done uploading piece. err: <nil> Aug 1 21:56:46.257505313 (51h59m36.195163802s) [noise=true , address=79.127.220.97:4371 , piece_id=5WGPRFKQETIHTSMA42AM2C3UGCFYJ3DMMPAI2L7DD6E2P37KJKNA, seg_pos={0 0}, upload=49960]: Done uploading piece. err: <nil> Aug 1 21:56:46.334426181 (51h59m36.27208468s) [noise=true , address=79.127.201.215:4352 , piece_id=I6BRYI5XIACAAAAAE7HQOBCF4OTALMOVPPNBWQCRSH2MM6WUUURQ, seg_pos={0 0}, upload=49960]: Done uploading piece. err: <nil> Aug 1 21:56:46.386119275 (51h59m36.323777774s) [noise=true , address=79.127.201.214:4297 , piece_id=MPTMGVEYC4SATIN4ANCR2ELVRXHYWRDLMHWQW7HM6UOIV2N6R7ZA, seg_pos={0 0}, upload=49960]: Done uploading piece. err: <nil> Aug 1 21:56:46.395362048 (51h59m36.333020547s) [noise=true , address=79.127.219.51:4213 , piece_id=L2WSPI5BEIFLGQKJIK4EWXFQXNQRQ7HVKPOWFYRPPUGNAL5DVYWA, seg_pos={0 0}, upload=49960]: Done uploading piece. err: <nil> Aug 1 21:57:43.392434383 (52h0m33.330092872s) [noise=true , address=104.36.82.37:4680 , piece_id=X4PDUE4WXUUWX6JQVO4NWPQCBMHMHC5YHPIHDQVYYOI2FXV2GVJQ, seg_pos={0 0}, upload=49960]: Done uploading piece. err: ecclient: upload failed (node:1bXAqpxGhUvzwYEDGBqYRhZdTeMJcV3TYfngkKTCumpt9Nupi3, address:104.36.82.37:4680): protocol: rpc: dial tcp 104.36.82.37:4680: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:43.392477967 (52h0m33.330136466s) [noise=true , address=104.36.82.11:4134 , piece_id=RMTNNNC2VD7MGWJDAMT6ZCF3GR43SHANFVAGJ6EHVH7WO5XA6VTA, seg_pos={0 0}, upload=49960]: Done uploading piece. err: ecclient: upload failed (node:12R3Z5AgkBmWXMv5XgRyfcyqkG2gmyJs9ogT24kcLy6nuB9bzUG, address:104.36.82.11:4134): protocol: rpc: dial tcp 104.36.82.11:4134: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:43.392514086 (52h0m33.330172585s) [noise=true , address=104.36.82.55:4204 , piece_id=R5CYXWAVV4ZEMCV7LF4D4XDC4G6KLVBONAH2TH5EIEVL7QHUTRWQ, seg_pos={0 0}, upload=49960]: Done uploading piece. err: ecclient: upload failed (node:1rBZH6aLHsnWh22uNzaMiMwmLJCDMn4bYbNC5jRG7KCEuRfqpH, address:104.36.82.55:4204): protocol: write tcp 188.166.225.170:48550->104.36.82.55:4204: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:43.392546537 (52h0m33.330205027s) [noise=true , address=104.36.82.59:4327 , piece_id=OZDEYIY6B4WYZULRDRL2UBOJH2JQNY3YWSITZKWDO7GTT72AN5NQ, seg_pos={0 0}, upload=49960]: Done uploading piece. err: ecclient: upload failed (node:1V4C6AWGbyYQWfinoSj7iKovHeTJ9hqypyC5U9rZ8pveXBBfCY, address:104.36.82.59:4327): protocol: rpc: dial tcp 104.36.82.59:4327: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:43.392579040 (52h0m33.33023754s) [noise=true , address=104.36.82.5:4842 , piece_id=LV7YAOMPE2AV7TF2WPWR7QJNLT4JZAIPSVZN3E23U5KDBNKZFD6A, seg_pos={0 0}, upload=49960]: Done uploading piece. err: ecclient: upload failed (node:12q4VdT5BcjUb16PBYHK6zrnW8dqENKrcGx9zdLtNefDVB1MUSn, address:104.36.82.5:4842): protocol: context deadline exceeded; context deadline exceeded Aug 1 21:57:43.392627953 (52h0m33.330286442s) [noise=true , address=104.36.82.52:4101 , piece_id=XYPLQM7QITCCX7T4NKIXP5MZ7I6F24EJJTG37K4NPOBKTUFHULRQ, seg_pos={0 0}, upload=49960]: Done uploading piece. err: ecclient: upload failed (node:12WWXeBjFZR9PzciNfs35fFcx87N9GwDpMsPHnksFDugtaBGWp4, address:104.36.82.52:4101): protocol: rpc: dial tcp 104.36.82.52:4101: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:43.392653192 (52h0m33.330311691s) [noise=true , address=104.36.82.14:4251 , piece_id=O7NADU347E2Q26TPD3IUCK6WE5UVRITC6ALZHBE5SWYTKWXFBMXA, seg_pos={0 0}, upload=49960]: Done uploading piece. err: ecclient: upload failed (node:12btf6Z38aTMx7MaBxhUyZtFqUPgeMtP2aU8Y6FbV8zwu57zArd, address:104.36.82.14:4251): protocol: rpc: dial tcp 104.36.82.14:4251: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:43.392670836 (52h0m33.330329335s) [noise=true , address=104.36.82.12:4166 , piece_id=6ZIWCOBY76GJ6DHZF5G23U4IYTOFB5SMQW5TW3YITYH227PV2FGA, seg_pos={0 0}, upload=49960]: Done uploading piece. err: ecclient: upload failed (node:12iCB7c4ZfpqafpD1PtpbE6BfU81wJW65LmrPVJXtD58yeuoR9t, address:104.36.82.12:4166): protocol: rpc: dial tcp 104.36.82.12:4166: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:43.392698018 (52h0m33.330356517s) [noise=true , address=104.36.82.50:4027 , piece_id=OW7BODU5N7ZEEGE2FHSLNYVJT2Z25LY3J42TZEOR5SP2EWIN6X7A, seg_pos={0 0}, upload=49960]: Done uploading piece. err: ecclient: upload failed (node:1zEQhyuVxSHwdDUgDfcTog2W3oLBx4z9bYUMHpaaoLU1BTU9DQ, address:104.36.82.50:4027): protocol: write tcp 188.166.225.170:37784->104.36.82.50:4027: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:43.392715381 (52h0m33.33037387s) [noise=true , address=104.36.82.43:4856 , piece_id=2SINBLHGRUOWUO4XHBNBG45YE3PW7RNF3QSNVCQXNCFEFHRBFNJQ, seg_pos={0 0}, upload=49960]: Done uploading piece. err: ecclient: upload failed (node:12bQxwBfPEGtmMp6mtnWKsoZt3eMEjLdyYFahETWGHeKJ7BhkiZ, address:104.36.82.43:4856): protocol: rpc: dial tcp 104.36.82.43:4856: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:43.392730059 (52h0m33.330388558s) [noise=true , address=104.36.82.10:4082 , piece_id=45K4BW277ZSGXJYQDA66OMEBVDC7DAWSOJAPX42HANNGRZN35KHQ, seg_pos={0 0}, upload=49960]: Done uploading piece. err: ecclient: upload failed (node:12DKsNCtqxLvpJwLszdLs74V3eb8NH2VAV9V3bngzUczawmNoLK, address:104.36.82.10:4082): protocol: write tcp 188.166.225.170:46638->104.36.82.10:4082: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:43.392753424 (52h0m33.330411923s) [seg_pos={0 0}, upload=49960]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil> Aug 1 21:57:43.624550997 (52h0m33.562209566s) [upload=49960]: Done uploading segments... Aug 1 21:57:43.624582868 (52h0m33.562241407s) [upload=49960]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7) Duration: 5.845981887s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 8 out of 60 is failed (threshold=3) [.XXXXXXX........................X...........................] duration: 6.285198731s FAIL (2025-08-01 22:35:17 UTC) orbiter warning: Operation: sync2.Copy digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/4M):: after 92e23610b5d37ceb174fa119b5e47214ac2cd632235616ccf3b7f0b13a4a6031 != before 92313f7b7503eb9340206bbda9efbdb260cf41b185fb20ecbdf772c5fb959aac Duration: 6.28522848s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 4 out of 60 is failed (threshold=3) [...X.....X.......X.....X....................................] duration: 1.93894798s FAIL (2025-08-01 22:35:17 UTC) orbiter warning: Operation: ListObjects unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/1K Duration: 1.939012665s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [......XXXX..................................................] duration: 6.110950043s FAIL (2025-08-01 22:35:02 UTC) orbiter warning: Operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 7) ======LOGS====== K4BW277ZSGXJYQDA66OMEBVDC7DAWSOJAPX42HANNGRZN35KHQ, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.481101510 (51h59m34.418760009s) [noise=true , address=104.36.82.5:4842 , piece_id=LV7YAOMPE2AV7TF2WPWR7QJNLT4JZAIPSVZN3E23U5KDBNKZFD6A, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.481422003 (51h59m34.419080502s) [noise=true , address=79.127.205.235:4683 , piece_id=3QHDNAPG4GRLJFVVNJNFXNGKTGKUEMWWUBMW62HHE5N5LJDO4KWA, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.482030897 (51h59m34.419689396s) [noise=true , address=79.127.219.51:4213 , piece_id=L2WSPI5BEIFLGQKJIK4EWXFQXNQRQ7HVKPOWFYRPPUGNAL5DVYWA, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.482285215 (51h59m34.419943704s) [noise=true , address=104.36.82.14:4251 , piece_id=O7NADU347E2Q26TPD3IUCK6WE5UVRITC6ALZHBE5SWYTKWXFBMXA, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.482518370 (51h59m34.420176859s) [noise=true , address=79.127.220.98:4402 , piece_id=DVLSYRFFZ6UT3QRSVAN7AMXVYZNI6PWG2IITHLRUUJATJVK3QOGA, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.482724304 (51h59m34.420382803s) [noise=true , address=79.127.201.214:4297 , piece_id=MPTMGVEYC4SATIN4ANCR2ELVRXHYWRDLMHWQW7HM6UOIV2N6R7ZA, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.482928555 (51h59m34.420587054s) [noise=true , address=104.36.82.37:4680 , piece_id=X4PDUE4WXUUWX6JQVO4NWPQCBMHMHC5YHPIHDQVYYOI2FXV2GVJQ, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.483134960 (51h59m34.420793449s) [noise=true , address=104.36.82.50:4027 , piece_id=OW7BODU5N7ZEEGE2FHSLNYVJT2Z25LY3J42TZEOR5SP2EWIN6X7A, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.483338480 (51h59m34.420996979s) [noise=true , address=104.36.82.55:4204 , piece_id=R5CYXWAVV4ZEMCV7LF4D4XDC4G6KLVBONAH2TH5EIEVL7QHUTRWQ, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.483582847 (51h59m34.421241356s) [noise=true , address=104.36.82.11:4134 , piece_id=RMTNNNC2VD7MGWJDAMT6ZCF3GR43SHANFVAGJ6EHVH7WO5XA6VTA, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.483833737 (51h59m34.421492236s) [noise=true , address=104.36.82.43:4856 , piece_id=2SINBLHGRUOWUO4XHBNBG45YE3PW7RNF3QSNVCQXNCFEFHRBFNJQ, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.484021896 (51h59m34.421680385s) [noise=true , address=104.36.82.52:4101 , piece_id=XYPLQM7QITCCX7T4NKIXP5MZ7I6F24EJJTG37K4NPOBKTUFHULRQ, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.484301662 (51h59m34.421960182s) [noise=true , address=79.127.201.215:4352 , piece_id=I6BRYI5XIACAAAAAE7HQOBCF4OTALMOVPPNBWQCRSH2MM6WUUURQ, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.484712097 (51h59m34.422370596s) [noise=true , address=64.49.13.138:4484 , piece_id=DZ5Y4HTS6BKW2DJ5CVOB64TW4SXB6DNCJEGFUJPLAUXUEHRZRTUA, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:45.576021470 (51h59m35.513679959s) [upload=49960]: Next returned nil segment Aug 1 21:56:45.576036609 (51h59m35.513695098s) [upload=49960]: Waiting for error group managing segments... Aug 1 21:56:45.860378776 (51h59m35.798037275s) [noise=true , address=64.49.13.138:4484 , piece_id=DZ5Y4HTS6BKW2DJ5CVOB64TW4SXB6DNCJEGFUJPLAUXUEHRZRTUA, seg_pos={0 0}, upload=49960]: Done uploading piece. err: <nil> Aug 1 21:56:45.884184188 (51h59m35.821842687s) [noise=true , address=79.127.205.235:4683 , piece_id=3QHDNAPG4GRLJFVVNJNFXNGKTGKUEMWWUBMW62HHE5N5LJDO4KWA, seg_pos={0 0}, upload=49960]: Done uploading piece. err: <nil> Aug 1 21:56:46.145493612 (51h59m36.083152101s) [noise=true , address=79.127.220.98:4402 , piece_id=DVLSYRFFZ6UT3QRSVAN7AMXVYZNI6PWG2IITHLRUUJATJVK3QOGA, seg_pos={0 0}, upload=49960]: Done uploading piece. err: <nil> Aug 1 21:56:46.257505313 (51h59m36.195163802s) [noise=true , address=79.127.220.97:4371 , piece_id=5WGPRFKQETIHTSMA42AM2C3UGCFYJ3DMMPAI2L7DD6E2P37KJKNA, seg_pos={0 0}, upload=49960]: Done uploading piece. err: <nil> Aug 1 21:56:46.334426181 (51h59m36.27208468s) [noise=true , address=79.127.201.215:4352 , piece_id=I6BRYI5XIACAAAAAE7HQOBCF4OTALMOVPPNBWQCRSH2MM6WUUURQ, seg_pos={0 0}, upload=49960]: Done uploading piece. err: <nil> Aug 1 21:56:46.386119275 (51h59m36.323777774s) [noise=true , address=79.127.201.214:4297 , piece_id=MPTMGVEYC4SATIN4ANCR2ELVRXHYWRDLMHWQW7HM6UOIV2N6R7ZA, seg_pos={0 0}, upload=49960]: Done uploading piece. err: <nil> Aug 1 21:56:46.395362048 (51h59m36.333020547s) [noise=true , address=79.127.219.51:4213 , piece_id=L2WSPI5BEIFLGQKJIK4EWXFQXNQRQ7HVKPOWFYRPPUGNAL5DVYWA, seg_pos={0 0}, upload=49960]: Done uploading piece. err: <nil> Aug 1 21:57:43.392434383 (52h0m33.330092872s) [noise=true , address=104.36.82.37:4680 , piece_id=X4PDUE4WXUUWX6JQVO4NWPQCBMHMHC5YHPIHDQVYYOI2FXV2GVJQ, seg_pos={0 0}, upload=49960]: Done uploading piece. err: ecclient: upload failed (node:1bXAqpxGhUvzwYEDGBqYRhZdTeMJcV3TYfngkKTCumpt9Nupi3, address:104.36.82.37:4680): protocol: rpc: dial tcp 104.36.82.37:4680: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:43.392477967 (52h0m33.330136466s) [noise=true , address=104.36.82.11:4134 , piece_id=RMTNNNC2VD7MGWJDAMT6ZCF3GR43SHANFVAGJ6EHVH7WO5XA6VTA, seg_pos={0 0}, upload=49960]: Done uploading piece. err: ecclient: upload failed (node:12R3Z5AgkBmWXMv5XgRyfcyqkG2gmyJs9ogT24kcLy6nuB9bzUG, address:104.36.82.11:4134): protocol: rpc: dial tcp 104.36.82.11:4134: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:43.392514086 (52h0m33.330172585s) [noise=true , address=104.36.82.55:4204 , piece_id=R5CYXWAVV4ZEMCV7LF4D4XDC4G6KLVBONAH2TH5EIEVL7QHUTRWQ, seg_pos={0 0}, upload=49960]: Done uploading piece. err: ecclient: upload failed (node:1rBZH6aLHsnWh22uNzaMiMwmLJCDMn4bYbNC5jRG7KCEuRfqpH, address:104.36.82.55:4204): protocol: write tcp 188.166.225.170:48550->104.36.82.55:4204: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:43.392546537 (52h0m33.330205027s) [noise=true , address=104.36.82.59:4327 , piece_id=OZDEYIY6B4WYZULRDRL2UBOJH2JQNY3YWSITZKWDO7GTT72AN5NQ, seg_pos={0 0}, upload=49960]: Done uploading piece. err: ecclient: upload failed (node:1V4C6AWGbyYQWfinoSj7iKovHeTJ9hqypyC5U9rZ8pveXBBfCY, address:104.36.82.59:4327): protocol: rpc: dial tcp 104.36.82.59:4327: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:43.392579040 (52h0m33.33023754s) [noise=true , address=104.36.82.5:4842 , piece_id=LV7YAOMPE2AV7TF2WPWR7QJNLT4JZAIPSVZN3E23U5KDBNKZFD6A, seg_pos={0 0}, upload=49960]: Done uploading piece. err: ecclient: upload failed (node:12q4VdT5BcjUb16PBYHK6zrnW8dqENKrcGx9zdLtNefDVB1MUSn, address:104.36.82.5:4842): protocol: context deadline exceeded; context deadline exceeded Aug 1 21:57:43.392627953 (52h0m33.330286442s) [noise=true , address=104.36.82.52:4101 , piece_id=XYPLQM7QITCCX7T4NKIXP5MZ7I6F24EJJTG37K4NPOBKTUFHULRQ, seg_pos={0 0}, upload=49960]: Done uploading piece. err: ecclient: upload failed (node:12WWXeBjFZR9PzciNfs35fFcx87N9GwDpMsPHnksFDugtaBGWp4, address:104.36.82.52:4101): protocol: rpc: dial tcp 104.36.82.52:4101: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:43.392653192 (52h0m33.330311691s) [noise=true , address=104.36.82.14:4251 , piece_id=O7NADU347E2Q26TPD3IUCK6WE5UVRITC6ALZHBE5SWYTKWXFBMXA, seg_pos={0 0}, upload=49960]: Done uploading piece. err: ecclient: upload failed (node:12btf6Z38aTMx7MaBxhUyZtFqUPgeMtP2aU8Y6FbV8zwu57zArd, address:104.36.82.14:4251): protocol: rpc: dial tcp 104.36.82.14:4251: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:43.392670836 (52h0m33.330329335s) [noise=true , address=104.36.82.12:4166 , piece_id=6ZIWCOBY76GJ6DHZF5G23U4IYTOFB5SMQW5TW3YITYH227PV2FGA, seg_pos={0 0}, upload=49960]: Done uploading piece. err: ecclient: upload failed (node:12iCB7c4ZfpqafpD1PtpbE6BfU81wJW65LmrPVJXtD58yeuoR9t, address:104.36.82.12:4166): protocol: rpc: dial tcp 104.36.82.12:4166: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:43.392698018 (52h0m33.330356517s) [noise=true , address=104.36.82.50:4027 , piece_id=OW7BODU5N7ZEEGE2FHSLNYVJT2Z25LY3J42TZEOR5SP2EWIN6X7A, seg_pos={0 0}, upload=49960]: Done uploading piece. err: ecclient: upload failed (node:1zEQhyuVxSHwdDUgDfcTog2W3oLBx4z9bYUMHpaaoLU1BTU9DQ, address:104.36.82.50:4027): protocol: write tcp 188.166.225.170:37784->104.36.82.50:4027: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:43.392715381 (52h0m33.33037387s) [noise=true , address=104.36.82.43:4856 , piece_id=2SINBLHGRUOWUO4XHBNBG45YE3PW7RNF3QSNVCQXNCFEFHRBFNJQ, seg_pos={0 0}, upload=49960]: Done uploading piece. err: ecclient: upload failed (node:12bQxwBfPEGtmMp6mtnWKsoZt3eMEjLdyYFahETWGHeKJ7BhkiZ, address:104.36.82.43:4856): protocol: rpc: dial tcp 104.36.82.43:4856: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:43.392730059 (52h0m33.330388558s) [noise=true , address=104.36.82.10:4082 , piece_id=45K4BW277ZSGXJYQDA66OMEBVDC7DAWSOJAPX42HANNGRZN35KHQ, seg_pos={0 0}, upload=49960]: Done uploading piece. err: ecclient: upload failed (node:12DKsNCtqxLvpJwLszdLs74V3eb8NH2VAV9V3bngzUczawmNoLK, address:104.36.82.10:4082): protocol: write tcp 188.166.225.170:46638->104.36.82.10:4082: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:43.392753424 (52h0m33.330411923s) [seg_pos={0 0}, upload=49960]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil> Aug 1 21:57:43.624550997 (52h0m33.562209566s) [upload=49960]: Done uploading segments... Aug 1 21:57:43.624582868 (52h0m33.562241407s) [upload=49960]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7) Duration: 6.110981633s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 6.098120505s FAIL (2025-08-01 22:34:57 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.098151074s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 8 out of 60 is failed (threshold=3) [..XXXXXXX........................X..........................] duration: 6.052984875s FAIL (2025-08-01 22:34:56 UTC) orbiter warning: Operation: sync2.Copy digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/4M):: after 92e23610b5d37ceb174fa119b5e47214ac2cd632235616ccf3b7f0b13a4a6031 != before 92313f7b7503eb9340206bbda9efbdb260cf41b185fb20ecbdf772c5fb959aac Duration: 6.053012557s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [.....XXX....................................................] duration: 5.795996295s FAIL (2025-08-01 22:34:37 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.796016092s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 4 out of 60 is failed (threshold=3) [....X.....X.......X.....X...................................] duration: 1.923238234s FAIL (2025-08-01 22:34:27 UTC) orbiter warning: Operation: ListObjects unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/1K Duration: 1.923267179s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.......XX.X.X...............................................] duration: 5.713896823s FAIL (2025-08-01 22:34:25 UTC) orbiter warning: Operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 7NK72474VWE4E3YOBCDZ5O5ZOH2MSA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB): protocol: write tcp 188.166.225.170:38368->104.36.82.12:4151: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:56:50.418604753 (51h59m40.356263252s) [seg_pos={0 0}, upload=49961]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 1 21:56:50.418680879 (51h59m40.356339378s) [upload=49961]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 1 21:56:50.418685998 (51h59m40.356344487s) [upload=49961]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 1 21:56:50.726563872 (51h59m40.664222371s) [upload=49961]: Done uploading segments... Aug 1 21:56:50.726586586 (51h59m40.664245085s) [upload=49961]: Upload finished. err: <nil> Aug 1 21:57:28.852869669 (52h0m18.790528158s) [upload=49973]: Starting upload Aug 1 21:57:28.852942389 (52h0m18.790600888s) [upload=49973]: Done starting upload Aug 1 21:57:28.853007122 (52h0m18.790665611s) [upload=49973]: Uploading segments... Aug 1 21:57:28.853174002 (52h0m18.790832491s) [upload=49973]: Got next segment. Inline: false Aug 1 21:57:28.853208168 (52h0m18.790866657s) [upload=49973]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 1 21:57:28.853212666 (52h0m18.790871165s) [upload=49973]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 1 21:57:29.095087519 (52h0m19.032746008s) [seg_pos={0 0}, upload=49973]: Begin upload segment... Aug 1 21:57:29.095129960 (52h0m19.032788459s) [seg_pos={0 0}, upload=49973]: Done begin upload segment. Aug 1 21:57:29.095164997 (52h0m19.032823496s) [noise=true , address=77.90.31.21:4878 , piece_id=BLNTYBOAIZWISSUJYK6BSBPTSNL6QWLGEN6AP6P4NPDVSURYG5AA, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.095171800 (52h0m19.032830289s) [noise=true , address=104.36.82.6:4875 , piece_id=CS5AMB7CIBQWWOU57Z3RCCSNVKIPE4B5QWU5YIQYI2UCWH7PAPVA, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.095510027 (52h0m19.033168526s) [noise=true , address=77.90.31.18:4806 , piece_id=52B6CAHGLFVTFIDQHHHAWKCBP2WJJY7OTTGRZZ7CEUH3HDA43Q6A, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.095544193 (52h0m19.033202692s) [noise=true , address=77.90.31.10:4518 , piece_id=WUOVXUSDG6YQTDPWG4YO7VLJNZZVLH6VRGMWOJABG2EJRU7XHP5A, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.095774283 (52h0m19.033432782s) [noise=true , address=104.36.82.7:4909 , piece_id=QHJCVBT6KTU34K2KWFNACWYVVE3WJ2RU76BHVSKL4HLCZEFXZYPA, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.095819880 (52h0m19.033478379s) [noise=true , address=45.91.116.12:4225 , piece_id=PBQPLA5ZYRFPSKHDKCEFOEBS5T3GYMP2FLSFDQWJLVY756KPLJNQ, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.096021656 (52h0m19.033680155s) [noise=true , address=45.91.116.13:4153 , piece_id=PMLKMZO66KXF3ZONO5ZZOQZICFK5XNGL6MN5FFBKICVFU5AWU6PQ, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.096069618 (52h0m19.033728117s) [noise=true , address=45.91.116.9:4262 , piece_id=A3F4L3RR6WPFMBFFKLATNGG3CASBIPWQP5O3G5URFNTXBE3ZW3VA, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.096348041 (52h0m19.03400654s) [noise=true , address=104.36.82.5:4840 , piece_id=UYVKVFCJJP7XZYRIVPS4ABGEMDUMNR7YW4QGG5ZFTFCGNCJO3CSQ, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.096354803 (52h0m19.034013292s) [noise=true , address=77.90.31.12:4590 , piece_id=D6FUBIFCQJSDRGWEP5GG34AQPNB3UROCYOATD6SRTJ7KAYVMFCOA, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.096608579 (52h0m19.034267078s) [noise=true , address=77.90.31.16:4733 , piece_id=DMASMGOUATLPEKCGLHKNNIMMFYVVGSULNUOMJU6RJY3ULBDOY6KQ, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.096830774 (52h0m19.034489273s) [noise=true , address=104.36.82.4:4802 , piece_id=CLM2G33M6AK5HIESRW6LZY33NKHIR2PGITWJ23NPILRCA6UYIEHA, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.097107233 (52h0m19.034765732s) [noise=true , address=77.90.31.19:4843 , piece_id=UFONZYHL7EAFYE5PRW32RKUVQPMZPXBEFLLIFJMAB2TXJ3C6KKNA, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.097324328 (52h0m19.034982827s) [noise=true , address=77.90.31.17:4771 , piece_id=Y3WSALJX23CWPPYQ3XYMCFOSBLBX2E2JQCCS7EOEYNBWY4CDRT3Q, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.098682446 (52h0m19.036340945s) [noise=true , address=77.90.31.15:4698 , piece_id=TN4OTPUD47FD6WOLZBEHNDI43WY6QVO5JK2JNJXNVM2IT6XMGERQ, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.099041603 (52h0m19.036700102s) [noise=true , address=45.91.116.11:4118 , piece_id=AKZIOAR6NRNO7JWCQZIKNSL7IUH5S4IG3KPG3IY2FAKG74DH3E3Q, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:30.718412255 (52h0m20.656070754s) [upload=49973]: Next returned nil segment Aug 1 21:57:30.718429869 (52h0m20.656088368s) [upload=49973]: Waiting for error group managing segments... Aug 1 21:57:31.107386675 (52h0m21.045045164s) [noise=true , address=45.91.116.13:4153 , piece_id=PMLKMZO66KXF3ZONO5ZZOQZICFK5XNGL6MN5FFBKICVFU5AWU6PQ, seg_pos={0 0}, upload=49973]: Done uploading piece. err: <nil> Aug 1 21:57:31.108529412 (52h0m21.046187911s) [noise=true , address=77.90.31.12:4590 , piece_id=D6FUBIFCQJSDRGWEP5GG34AQPNB3UROCYOATD6SRTJ7KAYVMFCOA, seg_pos={0 0}, upload=49973]: Done uploading piece. err: <nil> Aug 1 21:57:31.109975858 (52h0m21.047634347s) [noise=true , address=77.90.31.15:4698 , piece_id=TN4OTPUD47FD6WOLZBEHNDI43WY6QVO5JK2JNJXNVM2IT6XMGERQ, seg_pos={0 0}, upload=49973]: Done uploading piece. err: <nil> Aug 1 21:57:31.115273013 (52h0m21.052931502s) [noise=true , address=77.90.31.10:4518 , piece_id=WUOVXUSDG6YQTDPWG4YO7VLJNZZVLH6VRGMWOJABG2EJRU7XHP5A, seg_pos={0 0}, upload=49973]: Done uploading piece. err: <nil> Aug 1 21:57:31.116767281 (52h0m21.054425781s) [noise=true , address=77.90.31.18:4806 , piece_id=52B6CAHGLFVTFIDQHHHAWKCBP2WJJY7OTTGRZZ7CEUH3HDA43Q6A, seg_pos={0 0}, upload=49973]: Done uploading piece. err: <nil> Aug 1 21:57:31.201146107 (52h0m21.138804606s) [noise=true , address=77.90.31.16:4733 , piece_id=DMASMGOUATLPEKCGLHKNNIMMFYVVGSULNUOMJU6RJY3ULBDOY6KQ, seg_pos={0 0}, upload=49973]: Done uploading piece. err: <nil> Aug 1 21:57:31.201833732 (52h0m21.139492231s) [noise=true , address=77.90.31.19:4843 , piece_id=UFONZYHL7EAFYE5PRW32RKUVQPMZPXBEFLLIFJMAB2TXJ3C6KKNA, seg_pos={0 0}, upload=49973]: Done uploading piece. err: <nil> Aug 1 21:57:31.211007613 (52h0m21.148666112s) [noise=true , address=45.91.116.9:4262 , piece_id=A3F4L3RR6WPFMBFFKLATNGG3CASBIPWQP5O3G5URFNTXBE3ZW3VA, seg_pos={0 0}, upload=49973]: Done uploading piece. err: <nil> Aug 1 21:57:31.215929169 (52h0m21.153587668s) [noise=true , address=77.90.31.17:4771 , piece_id=Y3WSALJX23CWPPYQ3XYMCFOSBLBX2E2JQCCS7EOEYNBWY4CDRT3Q, seg_pos={0 0}, upload=49973]: Done uploading piece. err: <nil> Aug 1 21:57:31.222973975 (52h0m21.160632474s) [noise=true , address=77.90.31.21:4878 , piece_id=BLNTYBOAIZWISSUJYK6BSBPTSNL6QWLGEN6AP6P4NPDVSURYG5AA, seg_pos={0 0}, upload=49973]: Done uploading piece. err: <nil> Aug 1 21:57:31.234392441 (52h0m21.17205094s) [noise=true , address=45.91.116.12:4225 , piece_id=PBQPLA5ZYRFPSKHDKCEFOEBS5T3GYMP2FLSFDQWJLVY756KPLJNQ, seg_pos={0 0}, upload=49973]: Done uploading piece. err: <nil> Aug 1 21:57:31.284794305 (52h0m21.222452804s) [noise=true , address=45.91.116.11:4118 , piece_id=AKZIOAR6NRNO7JWCQZIKNSL7IUH5S4IG3KPG3IY2FAKG74DH3E3Q, seg_pos={0 0}, upload=49973]: Done uploading piece. err: <nil> Aug 1 21:57:58.202420932 (52h0m48.140079471s) [noise=true , address=104.36.82.6:4875 , piece_id=CS5AMB7CIBQWWOU57Z3RCCSNVKIPE4B5QWU5YIQYI2UCWH7PAPVA, seg_pos={0 0}, upload=49973]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:104.36.82.6:4875): protocol: write tcp 188.166.225.170:40166->104.36.82.6:4875: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:58.202441131 (52h0m48.14009963s) [noise=true , address=104.36.82.5:4840 , piece_id=UYVKVFCJJP7XZYRIVPS4ABGEMDUMNR7YW4QGG5ZFTFCGNCJO3CSQ, seg_pos={0 0}, upload=49973]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:104.36.82.5:4840): protocol: rpc: dial tcp 104.36.82.5:4840: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:58.202448445 (52h0m48.140106954s) [noise=true , address=104.36.82.7:4909 , piece_id=QHJCVBT6KTU34K2KWFNACWYVVE3WJ2RU76BHVSKL4HLCZEFXZYPA, seg_pos={0 0}, upload=49973]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:104.36.82.7:4909): protocol: rpc: dial tcp 104.36.82.7:4909: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:58.202496487 (52h0m48.140155026s) [noise=true , address=104.36.82.4:4802 , piece_id=CLM2G33M6AK5HIESRW6LZY33NKHIR2PGITWJ23NPILRCA6UYIEHA, seg_pos={0 0}, upload=49973]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: write tcp 188.166.225.170:47696->104.36.82.4:4802: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:58.202524240 (52h0m48.140182739s) [seg_pos={0 0}, upload=49973]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Aug 1 21:57:58.435983932 (52h0m48.373642431s) [upload=49973]: Done uploading segments... Aug 1 21:57:58.436007667 (52h0m48.373666166s) [upload=49973]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.713935768s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [......XXX...................................................] duration: 5.777346611s FAIL (2025-08-01 22:34:22 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.777367833s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 8 out of 60 is failed (threshold=3) [...XXXXXXX........................X.........................] duration: 5.900881633s FAIL (2025-08-01 22:34:18 UTC) orbiter warning: Operation: sync2.Copy digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/4M):: after 92e23610b5d37ceb174fa119b5e47214ac2cd632235616ccf3b7f0b13a4a6031 != before 92313f7b7503eb9340206bbda9efbdb260cf41b185fb20ecbdf772c5fb959aac Duration: 5.900905658s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 6.146731766s FAIL (2025-08-01 22:34:09 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.14677528s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 4 out of 60 is failed (threshold=3) [.....X.....X.......X.....X..................................] duration: 1.894599821s FAIL (2025-08-01 22:34:08 UTC) orbiter warning: Operation: ListObjects unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/1K Duration: 1.894623526s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [.......XXX..................................................] duration: 5.784378183s FAIL (2025-08-01 22:34:02 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.784400106s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [.......XXXX.................................................] duration: 5.724137018s FAIL (2025-08-01 22:33:55 UTC) orbiter warning: Operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 7) ======LOGS====== K4BW277ZSGXJYQDA66OMEBVDC7DAWSOJAPX42HANNGRZN35KHQ, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.481101510 (51h59m34.418760009s) [noise=true , address=104.36.82.5:4842 , piece_id=LV7YAOMPE2AV7TF2WPWR7QJNLT4JZAIPSVZN3E23U5KDBNKZFD6A, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.481422003 (51h59m34.419080502s) [noise=true , address=79.127.205.235:4683 , piece_id=3QHDNAPG4GRLJFVVNJNFXNGKTGKUEMWWUBMW62HHE5N5LJDO4KWA, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.482030897 (51h59m34.419689396s) [noise=true , address=79.127.219.51:4213 , piece_id=L2WSPI5BEIFLGQKJIK4EWXFQXNQRQ7HVKPOWFYRPPUGNAL5DVYWA, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.482285215 (51h59m34.419943704s) [noise=true , address=104.36.82.14:4251 , piece_id=O7NADU347E2Q26TPD3IUCK6WE5UVRITC6ALZHBE5SWYTKWXFBMXA, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.482518370 (51h59m34.420176859s) [noise=true , address=79.127.220.98:4402 , piece_id=DVLSYRFFZ6UT3QRSVAN7AMXVYZNI6PWG2IITHLRUUJATJVK3QOGA, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.482724304 (51h59m34.420382803s) [noise=true , address=79.127.201.214:4297 , piece_id=MPTMGVEYC4SATIN4ANCR2ELVRXHYWRDLMHWQW7HM6UOIV2N6R7ZA, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.482928555 (51h59m34.420587054s) [noise=true , address=104.36.82.37:4680 , piece_id=X4PDUE4WXUUWX6JQVO4NWPQCBMHMHC5YHPIHDQVYYOI2FXV2GVJQ, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.483134960 (51h59m34.420793449s) [noise=true , address=104.36.82.50:4027 , piece_id=OW7BODU5N7ZEEGE2FHSLNYVJT2Z25LY3J42TZEOR5SP2EWIN6X7A, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.483338480 (51h59m34.420996979s) [noise=true , address=104.36.82.55:4204 , piece_id=R5CYXWAVV4ZEMCV7LF4D4XDC4G6KLVBONAH2TH5EIEVL7QHUTRWQ, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.483582847 (51h59m34.421241356s) [noise=true , address=104.36.82.11:4134 , piece_id=RMTNNNC2VD7MGWJDAMT6ZCF3GR43SHANFVAGJ6EHVH7WO5XA6VTA, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.483833737 (51h59m34.421492236s) [noise=true , address=104.36.82.43:4856 , piece_id=2SINBLHGRUOWUO4XHBNBG45YE3PW7RNF3QSNVCQXNCFEFHRBFNJQ, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.484021896 (51h59m34.421680385s) [noise=true , address=104.36.82.52:4101 , piece_id=XYPLQM7QITCCX7T4NKIXP5MZ7I6F24EJJTG37K4NPOBKTUFHULRQ, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.484301662 (51h59m34.421960182s) [noise=true , address=79.127.201.215:4352 , piece_id=I6BRYI5XIACAAAAAE7HQOBCF4OTALMOVPPNBWQCRSH2MM6WUUURQ, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.484712097 (51h59m34.422370596s) [noise=true , address=64.49.13.138:4484 , piece_id=DZ5Y4HTS6BKW2DJ5CVOB64TW4SXB6DNCJEGFUJPLAUXUEHRZRTUA, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:45.576021470 (51h59m35.513679959s) [upload=49960]: Next returned nil segment Aug 1 21:56:45.576036609 (51h59m35.513695098s) [upload=49960]: Waiting for error group managing segments... Aug 1 21:56:45.860378776 (51h59m35.798037275s) [noise=true , address=64.49.13.138:4484 , piece_id=DZ5Y4HTS6BKW2DJ5CVOB64TW4SXB6DNCJEGFUJPLAUXUEHRZRTUA, seg_pos={0 0}, upload=49960]: Done uploading piece. err: <nil> Aug 1 21:56:45.884184188 (51h59m35.821842687s) [noise=true , address=79.127.205.235:4683 , piece_id=3QHDNAPG4GRLJFVVNJNFXNGKTGKUEMWWUBMW62HHE5N5LJDO4KWA, seg_pos={0 0}, upload=49960]: Done uploading piece. err: <nil> Aug 1 21:56:46.145493612 (51h59m36.083152101s) [noise=true , address=79.127.220.98:4402 , piece_id=DVLSYRFFZ6UT3QRSVAN7AMXVYZNI6PWG2IITHLRUUJATJVK3QOGA, seg_pos={0 0}, upload=49960]: Done uploading piece. err: <nil> Aug 1 21:56:46.257505313 (51h59m36.195163802s) [noise=true , address=79.127.220.97:4371 , piece_id=5WGPRFKQETIHTSMA42AM2C3UGCFYJ3DMMPAI2L7DD6E2P37KJKNA, seg_pos={0 0}, upload=49960]: Done uploading piece. err: <nil> Aug 1 21:56:46.334426181 (51h59m36.27208468s) [noise=true , address=79.127.201.215:4352 , piece_id=I6BRYI5XIACAAAAAE7HQOBCF4OTALMOVPPNBWQCRSH2MM6WUUURQ, seg_pos={0 0}, upload=49960]: Done uploading piece. err: <nil> Aug 1 21:56:46.386119275 (51h59m36.323777774s) [noise=true , address=79.127.201.214:4297 , piece_id=MPTMGVEYC4SATIN4ANCR2ELVRXHYWRDLMHWQW7HM6UOIV2N6R7ZA, seg_pos={0 0}, upload=49960]: Done uploading piece. err: <nil> Aug 1 21:56:46.395362048 (51h59m36.333020547s) [noise=true , address=79.127.219.51:4213 , piece_id=L2WSPI5BEIFLGQKJIK4EWXFQXNQRQ7HVKPOWFYRPPUGNAL5DVYWA, seg_pos={0 0}, upload=49960]: Done uploading piece. err: <nil> Aug 1 21:57:43.392434383 (52h0m33.330092872s) [noise=true , address=104.36.82.37:4680 , piece_id=X4PDUE4WXUUWX6JQVO4NWPQCBMHMHC5YHPIHDQVYYOI2FXV2GVJQ, seg_pos={0 0}, upload=49960]: Done uploading piece. err: ecclient: upload failed (node:1bXAqpxGhUvzwYEDGBqYRhZdTeMJcV3TYfngkKTCumpt9Nupi3, address:104.36.82.37:4680): protocol: rpc: dial tcp 104.36.82.37:4680: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:43.392477967 (52h0m33.330136466s) [noise=true , address=104.36.82.11:4134 , piece_id=RMTNNNC2VD7MGWJDAMT6ZCF3GR43SHANFVAGJ6EHVH7WO5XA6VTA, seg_pos={0 0}, upload=49960]: Done uploading piece. err: ecclient: upload failed (node:12R3Z5AgkBmWXMv5XgRyfcyqkG2gmyJs9ogT24kcLy6nuB9bzUG, address:104.36.82.11:4134): protocol: rpc: dial tcp 104.36.82.11:4134: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:43.392514086 (52h0m33.330172585s) [noise=true , address=104.36.82.55:4204 , piece_id=R5CYXWAVV4ZEMCV7LF4D4XDC4G6KLVBONAH2TH5EIEVL7QHUTRWQ, seg_pos={0 0}, upload=49960]: Done uploading piece. err: ecclient: upload failed (node:1rBZH6aLHsnWh22uNzaMiMwmLJCDMn4bYbNC5jRG7KCEuRfqpH, address:104.36.82.55:4204): protocol: write tcp 188.166.225.170:48550->104.36.82.55:4204: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:43.392546537 (52h0m33.330205027s) [noise=true , address=104.36.82.59:4327 , piece_id=OZDEYIY6B4WYZULRDRL2UBOJH2JQNY3YWSITZKWDO7GTT72AN5NQ, seg_pos={0 0}, upload=49960]: Done uploading piece. err: ecclient: upload failed (node:1V4C6AWGbyYQWfinoSj7iKovHeTJ9hqypyC5U9rZ8pveXBBfCY, address:104.36.82.59:4327): protocol: rpc: dial tcp 104.36.82.59:4327: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:43.392579040 (52h0m33.33023754s) [noise=true , address=104.36.82.5:4842 , piece_id=LV7YAOMPE2AV7TF2WPWR7QJNLT4JZAIPSVZN3E23U5KDBNKZFD6A, seg_pos={0 0}, upload=49960]: Done uploading piece. err: ecclient: upload failed (node:12q4VdT5BcjUb16PBYHK6zrnW8dqENKrcGx9zdLtNefDVB1MUSn, address:104.36.82.5:4842): protocol: context deadline exceeded; context deadline exceeded Aug 1 21:57:43.392627953 (52h0m33.330286442s) [noise=true , address=104.36.82.52:4101 , piece_id=XYPLQM7QITCCX7T4NKIXP5MZ7I6F24EJJTG37K4NPOBKTUFHULRQ, seg_pos={0 0}, upload=49960]: Done uploading piece. err: ecclient: upload failed (node:12WWXeBjFZR9PzciNfs35fFcx87N9GwDpMsPHnksFDugtaBGWp4, address:104.36.82.52:4101): protocol: rpc: dial tcp 104.36.82.52:4101: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:43.392653192 (52h0m33.330311691s) [noise=true , address=104.36.82.14:4251 , piece_id=O7NADU347E2Q26TPD3IUCK6WE5UVRITC6ALZHBE5SWYTKWXFBMXA, seg_pos={0 0}, upload=49960]: Done uploading piece. err: ecclient: upload failed (node:12btf6Z38aTMx7MaBxhUyZtFqUPgeMtP2aU8Y6FbV8zwu57zArd, address:104.36.82.14:4251): protocol: rpc: dial tcp 104.36.82.14:4251: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:43.392670836 (52h0m33.330329335s) [noise=true , address=104.36.82.12:4166 , piece_id=6ZIWCOBY76GJ6DHZF5G23U4IYTOFB5SMQW5TW3YITYH227PV2FGA, seg_pos={0 0}, upload=49960]: Done uploading piece. err: ecclient: upload failed (node:12iCB7c4ZfpqafpD1PtpbE6BfU81wJW65LmrPVJXtD58yeuoR9t, address:104.36.82.12:4166): protocol: rpc: dial tcp 104.36.82.12:4166: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:43.392698018 (52h0m33.330356517s) [noise=true , address=104.36.82.50:4027 , piece_id=OW7BODU5N7ZEEGE2FHSLNYVJT2Z25LY3J42TZEOR5SP2EWIN6X7A, seg_pos={0 0}, upload=49960]: Done uploading piece. err: ecclient: upload failed (node:1zEQhyuVxSHwdDUgDfcTog2W3oLBx4z9bYUMHpaaoLU1BTU9DQ, address:104.36.82.50:4027): protocol: write tcp 188.166.225.170:37784->104.36.82.50:4027: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:43.392715381 (52h0m33.33037387s) [noise=true , address=104.36.82.43:4856 , piece_id=2SINBLHGRUOWUO4XHBNBG45YE3PW7RNF3QSNVCQXNCFEFHRBFNJQ, seg_pos={0 0}, upload=49960]: Done uploading piece. err: ecclient: upload failed (node:12bQxwBfPEGtmMp6mtnWKsoZt3eMEjLdyYFahETWGHeKJ7BhkiZ, address:104.36.82.43:4856): protocol: rpc: dial tcp 104.36.82.43:4856: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:43.392730059 (52h0m33.330388558s) [noise=true , address=104.36.82.10:4082 , piece_id=45K4BW277ZSGXJYQDA66OMEBVDC7DAWSOJAPX42HANNGRZN35KHQ, seg_pos={0 0}, upload=49960]: Done uploading piece. err: ecclient: upload failed (node:12DKsNCtqxLvpJwLszdLs74V3eb8NH2VAV9V3bngzUczawmNoLK, address:104.36.82.10:4082): protocol: write tcp 188.166.225.170:46638->104.36.82.10:4082: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:43.392753424 (52h0m33.330411923s) [seg_pos={0 0}, upload=49960]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil> Aug 1 21:57:43.624550997 (52h0m33.562209566s) [upload=49960]: Done uploading segments... Aug 1 21:57:43.624582868 (52h0m33.562241407s) [upload=49960]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7) Duration: 5.724165803s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [........XXX.................................................] duration: 5.464466428s FAIL (2025-08-01 22:33:46 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.464481488s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 8 out of 60 is failed (threshold=3) [....XXXXXXX........................X........................] duration: 6.021274896s FAIL (2025-08-01 22:33:42 UTC) orbiter warning: Operation: sync2.Copy digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/4M):: after 92e23610b5d37ceb174fa119b5e47214ac2cd632235616ccf3b7f0b13a4a6031 != before 92313f7b7503eb9340206bbda9efbdb260cf41b185fb20ecbdf772c5fb959aac Duration: 6.021311067s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 6.157629384s FAIL (2025-08-01 22:33:38 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.157647439s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [........XX.X.X..............................................] duration: 5.878632361s FAIL (2025-08-01 22:33:32 UTC) orbiter warning: Operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 7NK72474VWE4E3YOBCDZ5O5ZOH2MSA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB): protocol: write tcp 188.166.225.170:38368->104.36.82.12:4151: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:56:50.418604753 (51h59m40.356263252s) [seg_pos={0 0}, upload=49961]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 1 21:56:50.418680879 (51h59m40.356339378s) [upload=49961]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 1 21:56:50.418685998 (51h59m40.356344487s) [upload=49961]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 1 21:56:50.726563872 (51h59m40.664222371s) [upload=49961]: Done uploading segments... Aug 1 21:56:50.726586586 (51h59m40.664245085s) [upload=49961]: Upload finished. err: <nil> Aug 1 21:57:28.852869669 (52h0m18.790528158s) [upload=49973]: Starting upload Aug 1 21:57:28.852942389 (52h0m18.790600888s) [upload=49973]: Done starting upload Aug 1 21:57:28.853007122 (52h0m18.790665611s) [upload=49973]: Uploading segments... Aug 1 21:57:28.853174002 (52h0m18.790832491s) [upload=49973]: Got next segment. Inline: false Aug 1 21:57:28.853208168 (52h0m18.790866657s) [upload=49973]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 1 21:57:28.853212666 (52h0m18.790871165s) [upload=49973]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 1 21:57:29.095087519 (52h0m19.032746008s) [seg_pos={0 0}, upload=49973]: Begin upload segment... Aug 1 21:57:29.095129960 (52h0m19.032788459s) [seg_pos={0 0}, upload=49973]: Done begin upload segment. Aug 1 21:57:29.095164997 (52h0m19.032823496s) [noise=true , address=77.90.31.21:4878 , piece_id=BLNTYBOAIZWISSUJYK6BSBPTSNL6QWLGEN6AP6P4NPDVSURYG5AA, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.095171800 (52h0m19.032830289s) [noise=true , address=104.36.82.6:4875 , piece_id=CS5AMB7CIBQWWOU57Z3RCCSNVKIPE4B5QWU5YIQYI2UCWH7PAPVA, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.095510027 (52h0m19.033168526s) [noise=true , address=77.90.31.18:4806 , piece_id=52B6CAHGLFVTFIDQHHHAWKCBP2WJJY7OTTGRZZ7CEUH3HDA43Q6A, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.095544193 (52h0m19.033202692s) [noise=true , address=77.90.31.10:4518 , piece_id=WUOVXUSDG6YQTDPWG4YO7VLJNZZVLH6VRGMWOJABG2EJRU7XHP5A, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.095774283 (52h0m19.033432782s) [noise=true , address=104.36.82.7:4909 , piece_id=QHJCVBT6KTU34K2KWFNACWYVVE3WJ2RU76BHVSKL4HLCZEFXZYPA, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.095819880 (52h0m19.033478379s) [noise=true , address=45.91.116.12:4225 , piece_id=PBQPLA5ZYRFPSKHDKCEFOEBS5T3GYMP2FLSFDQWJLVY756KPLJNQ, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.096021656 (52h0m19.033680155s) [noise=true , address=45.91.116.13:4153 , piece_id=PMLKMZO66KXF3ZONO5ZZOQZICFK5XNGL6MN5FFBKICVFU5AWU6PQ, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.096069618 (52h0m19.033728117s) [noise=true , address=45.91.116.9:4262 , piece_id=A3F4L3RR6WPFMBFFKLATNGG3CASBIPWQP5O3G5URFNTXBE3ZW3VA, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.096348041 (52h0m19.03400654s) [noise=true , address=104.36.82.5:4840 , piece_id=UYVKVFCJJP7XZYRIVPS4ABGEMDUMNR7YW4QGG5ZFTFCGNCJO3CSQ, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.096354803 (52h0m19.034013292s) [noise=true , address=77.90.31.12:4590 , piece_id=D6FUBIFCQJSDRGWEP5GG34AQPNB3UROCYOATD6SRTJ7KAYVMFCOA, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.096608579 (52h0m19.034267078s) [noise=true , address=77.90.31.16:4733 , piece_id=DMASMGOUATLPEKCGLHKNNIMMFYVVGSULNUOMJU6RJY3ULBDOY6KQ, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.096830774 (52h0m19.034489273s) [noise=true , address=104.36.82.4:4802 , piece_id=CLM2G33M6AK5HIESRW6LZY33NKHIR2PGITWJ23NPILRCA6UYIEHA, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.097107233 (52h0m19.034765732s) [noise=true , address=77.90.31.19:4843 , piece_id=UFONZYHL7EAFYE5PRW32RKUVQPMZPXBEFLLIFJMAB2TXJ3C6KKNA, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.097324328 (52h0m19.034982827s) [noise=true , address=77.90.31.17:4771 , piece_id=Y3WSALJX23CWPPYQ3XYMCFOSBLBX2E2JQCCS7EOEYNBWY4CDRT3Q, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.098682446 (52h0m19.036340945s) [noise=true , address=77.90.31.15:4698 , piece_id=TN4OTPUD47FD6WOLZBEHNDI43WY6QVO5JK2JNJXNVM2IT6XMGERQ, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.099041603 (52h0m19.036700102s) [noise=true , address=45.91.116.11:4118 , piece_id=AKZIOAR6NRNO7JWCQZIKNSL7IUH5S4IG3KPG3IY2FAKG74DH3E3Q, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:30.718412255 (52h0m20.656070754s) [upload=49973]: Next returned nil segment Aug 1 21:57:30.718429869 (52h0m20.656088368s) [upload=49973]: Waiting for error group managing segments... Aug 1 21:57:31.107386675 (52h0m21.045045164s) [noise=true , address=45.91.116.13:4153 , piece_id=PMLKMZO66KXF3ZONO5ZZOQZICFK5XNGL6MN5FFBKICVFU5AWU6PQ, seg_pos={0 0}, upload=49973]: Done uploading piece. err: <nil> Aug 1 21:57:31.108529412 (52h0m21.046187911s) [noise=true , address=77.90.31.12:4590 , piece_id=D6FUBIFCQJSDRGWEP5GG34AQPNB3UROCYOATD6SRTJ7KAYVMFCOA, seg_pos={0 0}, upload=49973]: Done uploading piece. err: <nil> Aug 1 21:57:31.109975858 (52h0m21.047634347s) [noise=true , address=77.90.31.15:4698 , piece_id=TN4OTPUD47FD6WOLZBEHNDI43WY6QVO5JK2JNJXNVM2IT6XMGERQ, seg_pos={0 0}, upload=49973]: Done uploading piece. err: <nil> Aug 1 21:57:31.115273013 (52h0m21.052931502s) [noise=true , address=77.90.31.10:4518 , piece_id=WUOVXUSDG6YQTDPWG4YO7VLJNZZVLH6VRGMWOJABG2EJRU7XHP5A, seg_pos={0 0}, upload=49973]: Done uploading piece. err: <nil> Aug 1 21:57:31.116767281 (52h0m21.054425781s) [noise=true , address=77.90.31.18:4806 , piece_id=52B6CAHGLFVTFIDQHHHAWKCBP2WJJY7OTTGRZZ7CEUH3HDA43Q6A, seg_pos={0 0}, upload=49973]: Done uploading piece. err: <nil> Aug 1 21:57:31.201146107 (52h0m21.138804606s) [noise=true , address=77.90.31.16:4733 , piece_id=DMASMGOUATLPEKCGLHKNNIMMFYVVGSULNUOMJU6RJY3ULBDOY6KQ, seg_pos={0 0}, upload=49973]: Done uploading piece. err: <nil> Aug 1 21:57:31.201833732 (52h0m21.139492231s) [noise=true , address=77.90.31.19:4843 , piece_id=UFONZYHL7EAFYE5PRW32RKUVQPMZPXBEFLLIFJMAB2TXJ3C6KKNA, seg_pos={0 0}, upload=49973]: Done uploading piece. err: <nil> Aug 1 21:57:31.211007613 (52h0m21.148666112s) [noise=true , address=45.91.116.9:4262 , piece_id=A3F4L3RR6WPFMBFFKLATNGG3CASBIPWQP5O3G5URFNTXBE3ZW3VA, seg_pos={0 0}, upload=49973]: Done uploading piece. err: <nil> Aug 1 21:57:31.215929169 (52h0m21.153587668s) [noise=true , address=77.90.31.17:4771 , piece_id=Y3WSALJX23CWPPYQ3XYMCFOSBLBX2E2JQCCS7EOEYNBWY4CDRT3Q, seg_pos={0 0}, upload=49973]: Done uploading piece. err: <nil> Aug 1 21:57:31.222973975 (52h0m21.160632474s) [noise=true , address=77.90.31.21:4878 , piece_id=BLNTYBOAIZWISSUJYK6BSBPTSNL6QWLGEN6AP6P4NPDVSURYG5AA, seg_pos={0 0}, upload=49973]: Done uploading piece. err: <nil> Aug 1 21:57:31.234392441 (52h0m21.17205094s) [noise=true , address=45.91.116.12:4225 , piece_id=PBQPLA5ZYRFPSKHDKCEFOEBS5T3GYMP2FLSFDQWJLVY756KPLJNQ, seg_pos={0 0}, upload=49973]: Done uploading piece. err: <nil> Aug 1 21:57:31.284794305 (52h0m21.222452804s) [noise=true , address=45.91.116.11:4118 , piece_id=AKZIOAR6NRNO7JWCQZIKNSL7IUH5S4IG3KPG3IY2FAKG74DH3E3Q, seg_pos={0 0}, upload=49973]: Done uploading piece. err: <nil> Aug 1 21:57:58.202420932 (52h0m48.140079471s) [noise=true , address=104.36.82.6:4875 , piece_id=CS5AMB7CIBQWWOU57Z3RCCSNVKIPE4B5QWU5YIQYI2UCWH7PAPVA, seg_pos={0 0}, upload=49973]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:104.36.82.6:4875): protocol: write tcp 188.166.225.170:40166->104.36.82.6:4875: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:58.202441131 (52h0m48.14009963s) [noise=true , address=104.36.82.5:4840 , piece_id=UYVKVFCJJP7XZYRIVPS4ABGEMDUMNR7YW4QGG5ZFTFCGNCJO3CSQ, seg_pos={0 0}, upload=49973]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:104.36.82.5:4840): protocol: rpc: dial tcp 104.36.82.5:4840: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:58.202448445 (52h0m48.140106954s) [noise=true , address=104.36.82.7:4909 , piece_id=QHJCVBT6KTU34K2KWFNACWYVVE3WJ2RU76BHVSKL4HLCZEFXZYPA, seg_pos={0 0}, upload=49973]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:104.36.82.7:4909): protocol: rpc: dial tcp 104.36.82.7:4909: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:58.202496487 (52h0m48.140155026s) [noise=true , address=104.36.82.4:4802 , piece_id=CLM2G33M6AK5HIESRW6LZY33NKHIR2PGITWJ23NPILRCA6UYIEHA, seg_pos={0 0}, upload=49973]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: write tcp 188.166.225.170:47696->104.36.82.4:4802: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:58.202524240 (52h0m48.140182739s) [seg_pos={0 0}, upload=49973]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Aug 1 21:57:58.435983932 (52h0m48.373642431s) [upload=49973]: Done uploading segments... Aug 1 21:57:58.436007667 (52h0m48.373666166s) [upload=49973]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.878671204s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [........XXXX................................................] duration: 5.881715729s FAIL (2025-08-01 22:33:30 UTC) orbiter warning: Operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 7) ======LOGS====== K4BW277ZSGXJYQDA66OMEBVDC7DAWSOJAPX42HANNGRZN35KHQ, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.481101510 (51h59m34.418760009s) [noise=true , address=104.36.82.5:4842 , piece_id=LV7YAOMPE2AV7TF2WPWR7QJNLT4JZAIPSVZN3E23U5KDBNKZFD6A, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.481422003 (51h59m34.419080502s) [noise=true , address=79.127.205.235:4683 , piece_id=3QHDNAPG4GRLJFVVNJNFXNGKTGKUEMWWUBMW62HHE5N5LJDO4KWA, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.482030897 (51h59m34.419689396s) [noise=true , address=79.127.219.51:4213 , piece_id=L2WSPI5BEIFLGQKJIK4EWXFQXNQRQ7HVKPOWFYRPPUGNAL5DVYWA, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.482285215 (51h59m34.419943704s) [noise=true , address=104.36.82.14:4251 , piece_id=O7NADU347E2Q26TPD3IUCK6WE5UVRITC6ALZHBE5SWYTKWXFBMXA, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.482518370 (51h59m34.420176859s) [noise=true , address=79.127.220.98:4402 , piece_id=DVLSYRFFZ6UT3QRSVAN7AMXVYZNI6PWG2IITHLRUUJATJVK3QOGA, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.482724304 (51h59m34.420382803s) [noise=true , address=79.127.201.214:4297 , piece_id=MPTMGVEYC4SATIN4ANCR2ELVRXHYWRDLMHWQW7HM6UOIV2N6R7ZA, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.482928555 (51h59m34.420587054s) [noise=true , address=104.36.82.37:4680 , piece_id=X4PDUE4WXUUWX6JQVO4NWPQCBMHMHC5YHPIHDQVYYOI2FXV2GVJQ, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.483134960 (51h59m34.420793449s) [noise=true , address=104.36.82.50:4027 , piece_id=OW7BODU5N7ZEEGE2FHSLNYVJT2Z25LY3J42TZEOR5SP2EWIN6X7A, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.483338480 (51h59m34.420996979s) [noise=true , address=104.36.82.55:4204 , piece_id=R5CYXWAVV4ZEMCV7LF4D4XDC4G6KLVBONAH2TH5EIEVL7QHUTRWQ, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.483582847 (51h59m34.421241356s) [noise=true , address=104.36.82.11:4134 , piece_id=RMTNNNC2VD7MGWJDAMT6ZCF3GR43SHANFVAGJ6EHVH7WO5XA6VTA, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.483833737 (51h59m34.421492236s) [noise=true , address=104.36.82.43:4856 , piece_id=2SINBLHGRUOWUO4XHBNBG45YE3PW7RNF3QSNVCQXNCFEFHRBFNJQ, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.484021896 (51h59m34.421680385s) [noise=true , address=104.36.82.52:4101 , piece_id=XYPLQM7QITCCX7T4NKIXP5MZ7I6F24EJJTG37K4NPOBKTUFHULRQ, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.484301662 (51h59m34.421960182s) [noise=true , address=79.127.201.215:4352 , piece_id=I6BRYI5XIACAAAAAE7HQOBCF4OTALMOVPPNBWQCRSH2MM6WUUURQ, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.484712097 (51h59m34.422370596s) [noise=true , address=64.49.13.138:4484 , piece_id=DZ5Y4HTS6BKW2DJ5CVOB64TW4SXB6DNCJEGFUJPLAUXUEHRZRTUA, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:45.576021470 (51h59m35.513679959s) [upload=49960]: Next returned nil segment Aug 1 21:56:45.576036609 (51h59m35.513695098s) [upload=49960]: Waiting for error group managing segments... Aug 1 21:56:45.860378776 (51h59m35.798037275s) [noise=true , address=64.49.13.138:4484 , piece_id=DZ5Y4HTS6BKW2DJ5CVOB64TW4SXB6DNCJEGFUJPLAUXUEHRZRTUA, seg_pos={0 0}, upload=49960]: Done uploading piece. err: <nil> Aug 1 21:56:45.884184188 (51h59m35.821842687s) [noise=true , address=79.127.205.235:4683 , piece_id=3QHDNAPG4GRLJFVVNJNFXNGKTGKUEMWWUBMW62HHE5N5LJDO4KWA, seg_pos={0 0}, upload=49960]: Done uploading piece. err: <nil> Aug 1 21:56:46.145493612 (51h59m36.083152101s) [noise=true , address=79.127.220.98:4402 , piece_id=DVLSYRFFZ6UT3QRSVAN7AMXVYZNI6PWG2IITHLRUUJATJVK3QOGA, seg_pos={0 0}, upload=49960]: Done uploading piece. err: <nil> Aug 1 21:56:46.257505313 (51h59m36.195163802s) [noise=true , address=79.127.220.97:4371 , piece_id=5WGPRFKQETIHTSMA42AM2C3UGCFYJ3DMMPAI2L7DD6E2P37KJKNA, seg_pos={0 0}, upload=49960]: Done uploading piece. err: <nil> Aug 1 21:56:46.334426181 (51h59m36.27208468s) [noise=true , address=79.127.201.215:4352 , piece_id=I6BRYI5XIACAAAAAE7HQOBCF4OTALMOVPPNBWQCRSH2MM6WUUURQ, seg_pos={0 0}, upload=49960]: Done uploading piece. err: <nil> Aug 1 21:56:46.386119275 (51h59m36.323777774s) [noise=true , address=79.127.201.214:4297 , piece_id=MPTMGVEYC4SATIN4ANCR2ELVRXHYWRDLMHWQW7HM6UOIV2N6R7ZA, seg_pos={0 0}, upload=49960]: Done uploading piece. err: <nil> Aug 1 21:56:46.395362048 (51h59m36.333020547s) [noise=true , address=79.127.219.51:4213 , piece_id=L2WSPI5BEIFLGQKJIK4EWXFQXNQRQ7HVKPOWFYRPPUGNAL5DVYWA, seg_pos={0 0}, upload=49960]: Done uploading piece. err: <nil> Aug 1 21:57:43.392434383 (52h0m33.330092872s) [noise=true , address=104.36.82.37:4680 , piece_id=X4PDUE4WXUUWX6JQVO4NWPQCBMHMHC5YHPIHDQVYYOI2FXV2GVJQ, seg_pos={0 0}, upload=49960]: Done uploading piece. err: ecclient: upload failed (node:1bXAqpxGhUvzwYEDGBqYRhZdTeMJcV3TYfngkKTCumpt9Nupi3, address:104.36.82.37:4680): protocol: rpc: dial tcp 104.36.82.37:4680: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:43.392477967 (52h0m33.330136466s) [noise=true , address=104.36.82.11:4134 , piece_id=RMTNNNC2VD7MGWJDAMT6ZCF3GR43SHANFVAGJ6EHVH7WO5XA6VTA, seg_pos={0 0}, upload=49960]: Done uploading piece. err: ecclient: upload failed (node:12R3Z5AgkBmWXMv5XgRyfcyqkG2gmyJs9ogT24kcLy6nuB9bzUG, address:104.36.82.11:4134): protocol: rpc: dial tcp 104.36.82.11:4134: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:43.392514086 (52h0m33.330172585s) [noise=true , address=104.36.82.55:4204 , piece_id=R5CYXWAVV4ZEMCV7LF4D4XDC4G6KLVBONAH2TH5EIEVL7QHUTRWQ, seg_pos={0 0}, upload=49960]: Done uploading piece. err: ecclient: upload failed (node:1rBZH6aLHsnWh22uNzaMiMwmLJCDMn4bYbNC5jRG7KCEuRfqpH, address:104.36.82.55:4204): protocol: write tcp 188.166.225.170:48550->104.36.82.55:4204: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:43.392546537 (52h0m33.330205027s) [noise=true , address=104.36.82.59:4327 , piece_id=OZDEYIY6B4WYZULRDRL2UBOJH2JQNY3YWSITZKWDO7GTT72AN5NQ, seg_pos={0 0}, upload=49960]: Done uploading piece. err: ecclient: upload failed (node:1V4C6AWGbyYQWfinoSj7iKovHeTJ9hqypyC5U9rZ8pveXBBfCY, address:104.36.82.59:4327): protocol: rpc: dial tcp 104.36.82.59:4327: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:43.392579040 (52h0m33.33023754s) [noise=true , address=104.36.82.5:4842 , piece_id=LV7YAOMPE2AV7TF2WPWR7QJNLT4JZAIPSVZN3E23U5KDBNKZFD6A, seg_pos={0 0}, upload=49960]: Done uploading piece. err: ecclient: upload failed (node:12q4VdT5BcjUb16PBYHK6zrnW8dqENKrcGx9zdLtNefDVB1MUSn, address:104.36.82.5:4842): protocol: context deadline exceeded; context deadline exceeded Aug 1 21:57:43.392627953 (52h0m33.330286442s) [noise=true , address=104.36.82.52:4101 , piece_id=XYPLQM7QITCCX7T4NKIXP5MZ7I6F24EJJTG37K4NPOBKTUFHULRQ, seg_pos={0 0}, upload=49960]: Done uploading piece. err: ecclient: upload failed (node:12WWXeBjFZR9PzciNfs35fFcx87N9GwDpMsPHnksFDugtaBGWp4, address:104.36.82.52:4101): protocol: rpc: dial tcp 104.36.82.52:4101: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:43.392653192 (52h0m33.330311691s) [noise=true , address=104.36.82.14:4251 , piece_id=O7NADU347E2Q26TPD3IUCK6WE5UVRITC6ALZHBE5SWYTKWXFBMXA, seg_pos={0 0}, upload=49960]: Done uploading piece. err: ecclient: upload failed (node:12btf6Z38aTMx7MaBxhUyZtFqUPgeMtP2aU8Y6FbV8zwu57zArd, address:104.36.82.14:4251): protocol: rpc: dial tcp 104.36.82.14:4251: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:43.392670836 (52h0m33.330329335s) [noise=true , address=104.36.82.12:4166 , piece_id=6ZIWCOBY76GJ6DHZF5G23U4IYTOFB5SMQW5TW3YITYH227PV2FGA, seg_pos={0 0}, upload=49960]: Done uploading piece. err: ecclient: upload failed (node:12iCB7c4ZfpqafpD1PtpbE6BfU81wJW65LmrPVJXtD58yeuoR9t, address:104.36.82.12:4166): protocol: rpc: dial tcp 104.36.82.12:4166: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:43.392698018 (52h0m33.330356517s) [noise=true , address=104.36.82.50:4027 , piece_id=OW7BODU5N7ZEEGE2FHSLNYVJT2Z25LY3J42TZEOR5SP2EWIN6X7A, seg_pos={0 0}, upload=49960]: Done uploading piece. err: ecclient: upload failed (node:1zEQhyuVxSHwdDUgDfcTog2W3oLBx4z9bYUMHpaaoLU1BTU9DQ, address:104.36.82.50:4027): protocol: write tcp 188.166.225.170:37784->104.36.82.50:4027: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:43.392715381 (52h0m33.33037387s) [noise=true , address=104.36.82.43:4856 , piece_id=2SINBLHGRUOWUO4XHBNBG45YE3PW7RNF3QSNVCQXNCFEFHRBFNJQ, seg_pos={0 0}, upload=49960]: Done uploading piece. err: ecclient: upload failed (node:12bQxwBfPEGtmMp6mtnWKsoZt3eMEjLdyYFahETWGHeKJ7BhkiZ, address:104.36.82.43:4856): protocol: rpc: dial tcp 104.36.82.43:4856: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:43.392730059 (52h0m33.330388558s) [noise=true , address=104.36.82.10:4082 , piece_id=45K4BW277ZSGXJYQDA66OMEBVDC7DAWSOJAPX42HANNGRZN35KHQ, seg_pos={0 0}, upload=49960]: Done uploading piece. err: ecclient: upload failed (node:12DKsNCtqxLvpJwLszdLs74V3eb8NH2VAV9V3bngzUczawmNoLK, address:104.36.82.10:4082): protocol: write tcp 188.166.225.170:46638->104.36.82.10:4082: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:43.392753424 (52h0m33.330411923s) [seg_pos={0 0}, upload=49960]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil> Aug 1 21:57:43.624550997 (52h0m33.562209566s) [upload=49960]: Done uploading segments... Aug 1 21:57:43.624582868 (52h0m33.562241407s) [upload=49960]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7) Duration: 5.881739245s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 4 out of 60 is failed (threshold=3) [......X.....X.......X.....X.................................] duration: 1.882016884s FAIL (2025-08-01 22:33:29 UTC) orbiter warning: Operation: ListObjects unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/1K Duration: 1.88204052s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [.........XXX................................................] duration: 5.558229543s FAIL (2025-08-01 22:33:18 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.558254571s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.........XX.X.X.............................................] duration: 5.741038684s FAIL (2025-08-01 22:33:13 UTC) orbiter warning: Operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 7NK72474VWE4E3YOBCDZ5O5ZOH2MSA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB): protocol: write tcp 188.166.225.170:38368->104.36.82.12:4151: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:56:50.418604753 (51h59m40.356263252s) [seg_pos={0 0}, upload=49961]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 1 21:56:50.418680879 (51h59m40.356339378s) [upload=49961]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 1 21:56:50.418685998 (51h59m40.356344487s) [upload=49961]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 1 21:56:50.726563872 (51h59m40.664222371s) [upload=49961]: Done uploading segments... Aug 1 21:56:50.726586586 (51h59m40.664245085s) [upload=49961]: Upload finished. err: <nil> Aug 1 21:57:28.852869669 (52h0m18.790528158s) [upload=49973]: Starting upload Aug 1 21:57:28.852942389 (52h0m18.790600888s) [upload=49973]: Done starting upload Aug 1 21:57:28.853007122 (52h0m18.790665611s) [upload=49973]: Uploading segments... Aug 1 21:57:28.853174002 (52h0m18.790832491s) [upload=49973]: Got next segment. Inline: false Aug 1 21:57:28.853208168 (52h0m18.790866657s) [upload=49973]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 1 21:57:28.853212666 (52h0m18.790871165s) [upload=49973]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 1 21:57:29.095087519 (52h0m19.032746008s) [seg_pos={0 0}, upload=49973]: Begin upload segment... Aug 1 21:57:29.095129960 (52h0m19.032788459s) [seg_pos={0 0}, upload=49973]: Done begin upload segment. Aug 1 21:57:29.095164997 (52h0m19.032823496s) [noise=true , address=77.90.31.21:4878 , piece_id=BLNTYBOAIZWISSUJYK6BSBPTSNL6QWLGEN6AP6P4NPDVSURYG5AA, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.095171800 (52h0m19.032830289s) [noise=true , address=104.36.82.6:4875 , piece_id=CS5AMB7CIBQWWOU57Z3RCCSNVKIPE4B5QWU5YIQYI2UCWH7PAPVA, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.095510027 (52h0m19.033168526s) [noise=true , address=77.90.31.18:4806 , piece_id=52B6CAHGLFVTFIDQHHHAWKCBP2WJJY7OTTGRZZ7CEUH3HDA43Q6A, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.095544193 (52h0m19.033202692s) [noise=true , address=77.90.31.10:4518 , piece_id=WUOVXUSDG6YQTDPWG4YO7VLJNZZVLH6VRGMWOJABG2EJRU7XHP5A, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.095774283 (52h0m19.033432782s) [noise=true , address=104.36.82.7:4909 , piece_id=QHJCVBT6KTU34K2KWFNACWYVVE3WJ2RU76BHVSKL4HLCZEFXZYPA, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.095819880 (52h0m19.033478379s) [noise=true , address=45.91.116.12:4225 , piece_id=PBQPLA5ZYRFPSKHDKCEFOEBS5T3GYMP2FLSFDQWJLVY756KPLJNQ, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.096021656 (52h0m19.033680155s) [noise=true , address=45.91.116.13:4153 , piece_id=PMLKMZO66KXF3ZONO5ZZOQZICFK5XNGL6MN5FFBKICVFU5AWU6PQ, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.096069618 (52h0m19.033728117s) [noise=true , address=45.91.116.9:4262 , piece_id=A3F4L3RR6WPFMBFFKLATNGG3CASBIPWQP5O3G5URFNTXBE3ZW3VA, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.096348041 (52h0m19.03400654s) [noise=true , address=104.36.82.5:4840 , piece_id=UYVKVFCJJP7XZYRIVPS4ABGEMDUMNR7YW4QGG5ZFTFCGNCJO3CSQ, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.096354803 (52h0m19.034013292s) [noise=true , address=77.90.31.12:4590 , piece_id=D6FUBIFCQJSDRGWEP5GG34AQPNB3UROCYOATD6SRTJ7KAYVMFCOA, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.096608579 (52h0m19.034267078s) [noise=true , address=77.90.31.16:4733 , piece_id=DMASMGOUATLPEKCGLHKNNIMMFYVVGSULNUOMJU6RJY3ULBDOY6KQ, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.096830774 (52h0m19.034489273s) [noise=true , address=104.36.82.4:4802 , piece_id=CLM2G33M6AK5HIESRW6LZY33NKHIR2PGITWJ23NPILRCA6UYIEHA, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.097107233 (52h0m19.034765732s) [noise=true , address=77.90.31.19:4843 , piece_id=UFONZYHL7EAFYE5PRW32RKUVQPMZPXBEFLLIFJMAB2TXJ3C6KKNA, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.097324328 (52h0m19.034982827s) [noise=true , address=77.90.31.17:4771 , piece_id=Y3WSALJX23CWPPYQ3XYMCFOSBLBX2E2JQCCS7EOEYNBWY4CDRT3Q, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.098682446 (52h0m19.036340945s) [noise=true , address=77.90.31.15:4698 , piece_id=TN4OTPUD47FD6WOLZBEHNDI43WY6QVO5JK2JNJXNVM2IT6XMGERQ, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.099041603 (52h0m19.036700102s) [noise=true , address=45.91.116.11:4118 , piece_id=AKZIOAR6NRNO7JWCQZIKNSL7IUH5S4IG3KPG3IY2FAKG74DH3E3Q, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:30.718412255 (52h0m20.656070754s) [upload=49973]: Next returned nil segment Aug 1 21:57:30.718429869 (52h0m20.656088368s) [upload=49973]: Waiting for error group managing segments... Aug 1 21:57:31.107386675 (52h0m21.045045164s) [noise=true , address=45.91.116.13:4153 , piece_id=PMLKMZO66KXF3ZONO5ZZOQZICFK5XNGL6MN5FFBKICVFU5AWU6PQ, seg_pos={0 0}, upload=49973]: Done uploading piece. err: <nil> Aug 1 21:57:31.108529412 (52h0m21.046187911s) [noise=true , address=77.90.31.12:4590 , piece_id=D6FUBIFCQJSDRGWEP5GG34AQPNB3UROCYOATD6SRTJ7KAYVMFCOA, seg_pos={0 0}, upload=49973]: Done uploading piece. err: <nil> Aug 1 21:57:31.109975858 (52h0m21.047634347s) [noise=true , address=77.90.31.15:4698 , piece_id=TN4OTPUD47FD6WOLZBEHNDI43WY6QVO5JK2JNJXNVM2IT6XMGERQ, seg_pos={0 0}, upload=49973]: Done uploading piece. err: <nil> Aug 1 21:57:31.115273013 (52h0m21.052931502s) [noise=true , address=77.90.31.10:4518 , piece_id=WUOVXUSDG6YQTDPWG4YO7VLJNZZVLH6VRGMWOJABG2EJRU7XHP5A, seg_pos={0 0}, upload=49973]: Done uploading piece. err: <nil> Aug 1 21:57:31.116767281 (52h0m21.054425781s) [noise=true , address=77.90.31.18:4806 , piece_id=52B6CAHGLFVTFIDQHHHAWKCBP2WJJY7OTTGRZZ7CEUH3HDA43Q6A, seg_pos={0 0}, upload=49973]: Done uploading piece. err: <nil> Aug 1 21:57:31.201146107 (52h0m21.138804606s) [noise=true , address=77.90.31.16:4733 , piece_id=DMASMGOUATLPEKCGLHKNNIMMFYVVGSULNUOMJU6RJY3ULBDOY6KQ, seg_pos={0 0}, upload=49973]: Done uploading piece. err: <nil> Aug 1 21:57:31.201833732 (52h0m21.139492231s) [noise=true , address=77.90.31.19:4843 , piece_id=UFONZYHL7EAFYE5PRW32RKUVQPMZPXBEFLLIFJMAB2TXJ3C6KKNA, seg_pos={0 0}, upload=49973]: Done uploading piece. err: <nil> Aug 1 21:57:31.211007613 (52h0m21.148666112s) [noise=true , address=45.91.116.9:4262 , piece_id=A3F4L3RR6WPFMBFFKLATNGG3CASBIPWQP5O3G5URFNTXBE3ZW3VA, seg_pos={0 0}, upload=49973]: Done uploading piece. err: <nil> Aug 1 21:57:31.215929169 (52h0m21.153587668s) [noise=true , address=77.90.31.17:4771 , piece_id=Y3WSALJX23CWPPYQ3XYMCFOSBLBX2E2JQCCS7EOEYNBWY4CDRT3Q, seg_pos={0 0}, upload=49973]: Done uploading piece. err: <nil> Aug 1 21:57:31.222973975 (52h0m21.160632474s) [noise=true , address=77.90.31.21:4878 , piece_id=BLNTYBOAIZWISSUJYK6BSBPTSNL6QWLGEN6AP6P4NPDVSURYG5AA, seg_pos={0 0}, upload=49973]: Done uploading piece. err: <nil> Aug 1 21:57:31.234392441 (52h0m21.17205094s) [noise=true , address=45.91.116.12:4225 , piece_id=PBQPLA5ZYRFPSKHDKCEFOEBS5T3GYMP2FLSFDQWJLVY756KPLJNQ, seg_pos={0 0}, upload=49973]: Done uploading piece. err: <nil> Aug 1 21:57:31.284794305 (52h0m21.222452804s) [noise=true , address=45.91.116.11:4118 , piece_id=AKZIOAR6NRNO7JWCQZIKNSL7IUH5S4IG3KPG3IY2FAKG74DH3E3Q, seg_pos={0 0}, upload=49973]: Done uploading piece. err: <nil> Aug 1 21:57:58.202420932 (52h0m48.140079471s) [noise=true , address=104.36.82.6:4875 , piece_id=CS5AMB7CIBQWWOU57Z3RCCSNVKIPE4B5QWU5YIQYI2UCWH7PAPVA, seg_pos={0 0}, upload=49973]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:104.36.82.6:4875): protocol: write tcp 188.166.225.170:40166->104.36.82.6:4875: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:58.202441131 (52h0m48.14009963s) [noise=true , address=104.36.82.5:4840 , piece_id=UYVKVFCJJP7XZYRIVPS4ABGEMDUMNR7YW4QGG5ZFTFCGNCJO3CSQ, seg_pos={0 0}, upload=49973]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:104.36.82.5:4840): protocol: rpc: dial tcp 104.36.82.5:4840: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:58.202448445 (52h0m48.140106954s) [noise=true , address=104.36.82.7:4909 , piece_id=QHJCVBT6KTU34K2KWFNACWYVVE3WJ2RU76BHVSKL4HLCZEFXZYPA, seg_pos={0 0}, upload=49973]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:104.36.82.7:4909): protocol: rpc: dial tcp 104.36.82.7:4909: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:58.202496487 (52h0m48.140155026s) [noise=true , address=104.36.82.4:4802 , piece_id=CLM2G33M6AK5HIESRW6LZY33NKHIR2PGITWJ23NPILRCA6UYIEHA, seg_pos={0 0}, upload=49973]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: write tcp 188.166.225.170:47696->104.36.82.4:4802: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:58.202524240 (52h0m48.140182739s) [seg_pos={0 0}, upload=49973]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Aug 1 21:57:58.435983932 (52h0m48.373642431s) [upload=49973]: Done uploading segments... Aug 1 21:57:58.436007667 (52h0m48.373666166s) [upload=49973]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.741060997s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 8 out of 60 is failed (threshold=3) [.....XXXXXXX........................X.......................] duration: 6.042309312s FAIL (2025-08-01 22:33:03 UTC) orbiter warning: Operation: sync2.Copy digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/4M):: after 92e23610b5d37ceb174fa119b5e47214ac2cd632235616ccf3b7f0b13a4a6031 != before 92313f7b7503eb9340206bbda9efbdb260cf41b185fb20ecbdf772c5fb959aac Duration: 6.04233955s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [.........XXXX...............................................] duration: 5.66009214s FAIL (2025-08-01 22:32:58 UTC) orbiter warning: Operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 7) ======LOGS====== K4BW277ZSGXJYQDA66OMEBVDC7DAWSOJAPX42HANNGRZN35KHQ, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.481101510 (51h59m34.418760009s) [noise=true , address=104.36.82.5:4842 , piece_id=LV7YAOMPE2AV7TF2WPWR7QJNLT4JZAIPSVZN3E23U5KDBNKZFD6A, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.481422003 (51h59m34.419080502s) [noise=true , address=79.127.205.235:4683 , piece_id=3QHDNAPG4GRLJFVVNJNFXNGKTGKUEMWWUBMW62HHE5N5LJDO4KWA, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.482030897 (51h59m34.419689396s) [noise=true , address=79.127.219.51:4213 , piece_id=L2WSPI5BEIFLGQKJIK4EWXFQXNQRQ7HVKPOWFYRPPUGNAL5DVYWA, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.482285215 (51h59m34.419943704s) [noise=true , address=104.36.82.14:4251 , piece_id=O7NADU347E2Q26TPD3IUCK6WE5UVRITC6ALZHBE5SWYTKWXFBMXA, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.482518370 (51h59m34.420176859s) [noise=true , address=79.127.220.98:4402 , piece_id=DVLSYRFFZ6UT3QRSVAN7AMXVYZNI6PWG2IITHLRUUJATJVK3QOGA, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.482724304 (51h59m34.420382803s) [noise=true , address=79.127.201.214:4297 , piece_id=MPTMGVEYC4SATIN4ANCR2ELVRXHYWRDLMHWQW7HM6UOIV2N6R7ZA, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.482928555 (51h59m34.420587054s) [noise=true , address=104.36.82.37:4680 , piece_id=X4PDUE4WXUUWX6JQVO4NWPQCBMHMHC5YHPIHDQVYYOI2FXV2GVJQ, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.483134960 (51h59m34.420793449s) [noise=true , address=104.36.82.50:4027 , piece_id=OW7BODU5N7ZEEGE2FHSLNYVJT2Z25LY3J42TZEOR5SP2EWIN6X7A, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.483338480 (51h59m34.420996979s) [noise=true , address=104.36.82.55:4204 , piece_id=R5CYXWAVV4ZEMCV7LF4D4XDC4G6KLVBONAH2TH5EIEVL7QHUTRWQ, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.483582847 (51h59m34.421241356s) [noise=true , address=104.36.82.11:4134 , piece_id=RMTNNNC2VD7MGWJDAMT6ZCF3GR43SHANFVAGJ6EHVH7WO5XA6VTA, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.483833737 (51h59m34.421492236s) [noise=true , address=104.36.82.43:4856 , piece_id=2SINBLHGRUOWUO4XHBNBG45YE3PW7RNF3QSNVCQXNCFEFHRBFNJQ, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.484021896 (51h59m34.421680385s) [noise=true , address=104.36.82.52:4101 , piece_id=XYPLQM7QITCCX7T4NKIXP5MZ7I6F24EJJTG37K4NPOBKTUFHULRQ, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.484301662 (51h59m34.421960182s) [noise=true , address=79.127.201.215:4352 , piece_id=I6BRYI5XIACAAAAAE7HQOBCF4OTALMOVPPNBWQCRSH2MM6WUUURQ, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.484712097 (51h59m34.422370596s) [noise=true , address=64.49.13.138:4484 , piece_id=DZ5Y4HTS6BKW2DJ5CVOB64TW4SXB6DNCJEGFUJPLAUXUEHRZRTUA, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:45.576021470 (51h59m35.513679959s) [upload=49960]: Next returned nil segment Aug 1 21:56:45.576036609 (51h59m35.513695098s) [upload=49960]: Waiting for error group managing segments... Aug 1 21:56:45.860378776 (51h59m35.798037275s) [noise=true , address=64.49.13.138:4484 , piece_id=DZ5Y4HTS6BKW2DJ5CVOB64TW4SXB6DNCJEGFUJPLAUXUEHRZRTUA, seg_pos={0 0}, upload=49960]: Done uploading piece. err: <nil> Aug 1 21:56:45.884184188 (51h59m35.821842687s) [noise=true , address=79.127.205.235:4683 , piece_id=3QHDNAPG4GRLJFVVNJNFXNGKTGKUEMWWUBMW62HHE5N5LJDO4KWA, seg_pos={0 0}, upload=49960]: Done uploading piece. err: <nil> Aug 1 21:56:46.145493612 (51h59m36.083152101s) [noise=true , address=79.127.220.98:4402 , piece_id=DVLSYRFFZ6UT3QRSVAN7AMXVYZNI6PWG2IITHLRUUJATJVK3QOGA, seg_pos={0 0}, upload=49960]: Done uploading piece. err: <nil> Aug 1 21:56:46.257505313 (51h59m36.195163802s) [noise=true , address=79.127.220.97:4371 , piece_id=5WGPRFKQETIHTSMA42AM2C3UGCFYJ3DMMPAI2L7DD6E2P37KJKNA, seg_pos={0 0}, upload=49960]: Done uploading piece. err: <nil> Aug 1 21:56:46.334426181 (51h59m36.27208468s) [noise=true , address=79.127.201.215:4352 , piece_id=I6BRYI5XIACAAAAAE7HQOBCF4OTALMOVPPNBWQCRSH2MM6WUUURQ, seg_pos={0 0}, upload=49960]: Done uploading piece. err: <nil> Aug 1 21:56:46.386119275 (51h59m36.323777774s) [noise=true , address=79.127.201.214:4297 , piece_id=MPTMGVEYC4SATIN4ANCR2ELVRXHYWRDLMHWQW7HM6UOIV2N6R7ZA, seg_pos={0 0}, upload=49960]: Done uploading piece. err: <nil> Aug 1 21:56:46.395362048 (51h59m36.333020547s) [noise=true , address=79.127.219.51:4213 , piece_id=L2WSPI5BEIFLGQKJIK4EWXFQXNQRQ7HVKPOWFYRPPUGNAL5DVYWA, seg_pos={0 0}, upload=49960]: Done uploading piece. err: <nil> Aug 1 21:57:43.392434383 (52h0m33.330092872s) [noise=true , address=104.36.82.37:4680 , piece_id=X4PDUE4WXUUWX6JQVO4NWPQCBMHMHC5YHPIHDQVYYOI2FXV2GVJQ, seg_pos={0 0}, upload=49960]: Done uploading piece. err: ecclient: upload failed (node:1bXAqpxGhUvzwYEDGBqYRhZdTeMJcV3TYfngkKTCumpt9Nupi3, address:104.36.82.37:4680): protocol: rpc: dial tcp 104.36.82.37:4680: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:43.392477967 (52h0m33.330136466s) [noise=true , address=104.36.82.11:4134 , piece_id=RMTNNNC2VD7MGWJDAMT6ZCF3GR43SHANFVAGJ6EHVH7WO5XA6VTA, seg_pos={0 0}, upload=49960]: Done uploading piece. err: ecclient: upload failed (node:12R3Z5AgkBmWXMv5XgRyfcyqkG2gmyJs9ogT24kcLy6nuB9bzUG, address:104.36.82.11:4134): protocol: rpc: dial tcp 104.36.82.11:4134: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:43.392514086 (52h0m33.330172585s) [noise=true , address=104.36.82.55:4204 , piece_id=R5CYXWAVV4ZEMCV7LF4D4XDC4G6KLVBONAH2TH5EIEVL7QHUTRWQ, seg_pos={0 0}, upload=49960]: Done uploading piece. err: ecclient: upload failed (node:1rBZH6aLHsnWh22uNzaMiMwmLJCDMn4bYbNC5jRG7KCEuRfqpH, address:104.36.82.55:4204): protocol: write tcp 188.166.225.170:48550->104.36.82.55:4204: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:43.392546537 (52h0m33.330205027s) [noise=true , address=104.36.82.59:4327 , piece_id=OZDEYIY6B4WYZULRDRL2UBOJH2JQNY3YWSITZKWDO7GTT72AN5NQ, seg_pos={0 0}, upload=49960]: Done uploading piece. err: ecclient: upload failed (node:1V4C6AWGbyYQWfinoSj7iKovHeTJ9hqypyC5U9rZ8pveXBBfCY, address:104.36.82.59:4327): protocol: rpc: dial tcp 104.36.82.59:4327: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:43.392579040 (52h0m33.33023754s) [noise=true , address=104.36.82.5:4842 , piece_id=LV7YAOMPE2AV7TF2WPWR7QJNLT4JZAIPSVZN3E23U5KDBNKZFD6A, seg_pos={0 0}, upload=49960]: Done uploading piece. err: ecclient: upload failed (node:12q4VdT5BcjUb16PBYHK6zrnW8dqENKrcGx9zdLtNefDVB1MUSn, address:104.36.82.5:4842): protocol: context deadline exceeded; context deadline exceeded Aug 1 21:57:43.392627953 (52h0m33.330286442s) [noise=true , address=104.36.82.52:4101 , piece_id=XYPLQM7QITCCX7T4NKIXP5MZ7I6F24EJJTG37K4NPOBKTUFHULRQ, seg_pos={0 0}, upload=49960]: Done uploading piece. err: ecclient: upload failed (node:12WWXeBjFZR9PzciNfs35fFcx87N9GwDpMsPHnksFDugtaBGWp4, address:104.36.82.52:4101): protocol: rpc: dial tcp 104.36.82.52:4101: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:43.392653192 (52h0m33.330311691s) [noise=true , address=104.36.82.14:4251 , piece_id=O7NADU347E2Q26TPD3IUCK6WE5UVRITC6ALZHBE5SWYTKWXFBMXA, seg_pos={0 0}, upload=49960]: Done uploading piece. err: ecclient: upload failed (node:12btf6Z38aTMx7MaBxhUyZtFqUPgeMtP2aU8Y6FbV8zwu57zArd, address:104.36.82.14:4251): protocol: rpc: dial tcp 104.36.82.14:4251: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:43.392670836 (52h0m33.330329335s) [noise=true , address=104.36.82.12:4166 , piece_id=6ZIWCOBY76GJ6DHZF5G23U4IYTOFB5SMQW5TW3YITYH227PV2FGA, seg_pos={0 0}, upload=49960]: Done uploading piece. err: ecclient: upload failed (node:12iCB7c4ZfpqafpD1PtpbE6BfU81wJW65LmrPVJXtD58yeuoR9t, address:104.36.82.12:4166): protocol: rpc: dial tcp 104.36.82.12:4166: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:43.392698018 (52h0m33.330356517s) [noise=true , address=104.36.82.50:4027 , piece_id=OW7BODU5N7ZEEGE2FHSLNYVJT2Z25LY3J42TZEOR5SP2EWIN6X7A, seg_pos={0 0}, upload=49960]: Done uploading piece. err: ecclient: upload failed (node:1zEQhyuVxSHwdDUgDfcTog2W3oLBx4z9bYUMHpaaoLU1BTU9DQ, address:104.36.82.50:4027): protocol: write tcp 188.166.225.170:37784->104.36.82.50:4027: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:43.392715381 (52h0m33.33037387s) [noise=true , address=104.36.82.43:4856 , piece_id=2SINBLHGRUOWUO4XHBNBG45YE3PW7RNF3QSNVCQXNCFEFHRBFNJQ, seg_pos={0 0}, upload=49960]: Done uploading piece. err: ecclient: upload failed (node:12bQxwBfPEGtmMp6mtnWKsoZt3eMEjLdyYFahETWGHeKJ7BhkiZ, address:104.36.82.43:4856): protocol: rpc: dial tcp 104.36.82.43:4856: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:43.392730059 (52h0m33.330388558s) [noise=true , address=104.36.82.10:4082 , piece_id=45K4BW277ZSGXJYQDA66OMEBVDC7DAWSOJAPX42HANNGRZN35KHQ, seg_pos={0 0}, upload=49960]: Done uploading piece. err: ecclient: upload failed (node:12DKsNCtqxLvpJwLszdLs74V3eb8NH2VAV9V3bngzUczawmNoLK, address:104.36.82.10:4082): protocol: write tcp 188.166.225.170:46638->104.36.82.10:4082: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:43.392753424 (52h0m33.330411923s) [seg_pos={0 0}, upload=49960]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil> Aug 1 21:57:43.624550997 (52h0m33.562209566s) [upload=49960]: Done uploading segments... Aug 1 21:57:43.624582868 (52h0m33.562241407s) [upload=49960]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7) Duration: 5.66012363s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)