Orbiter History

checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 6.444427383s FAIL (2025-08-01 22:16:25 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.44445167s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.............................XX.X.X.........................] duration: 5.991062891s FAIL (2025-08-01 22:16:18 UTC) orbiter warning: Operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 7NK72474VWE4E3YOBCDZ5O5ZOH2MSA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB): protocol: write tcp 188.166.225.170:38368->104.36.82.12:4151: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:56:50.418604753 (51h59m40.356263252s) [seg_pos={0 0}, upload=49961]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 1 21:56:50.418680879 (51h59m40.356339378s) [upload=49961]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 1 21:56:50.418685998 (51h59m40.356344487s) [upload=49961]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 1 21:56:50.726563872 (51h59m40.664222371s) [upload=49961]: Done uploading segments... Aug 1 21:56:50.726586586 (51h59m40.664245085s) [upload=49961]: Upload finished. err: <nil> Aug 1 21:57:28.852869669 (52h0m18.790528158s) [upload=49973]: Starting upload Aug 1 21:57:28.852942389 (52h0m18.790600888s) [upload=49973]: Done starting upload Aug 1 21:57:28.853007122 (52h0m18.790665611s) [upload=49973]: Uploading segments... Aug 1 21:57:28.853174002 (52h0m18.790832491s) [upload=49973]: Got next segment. Inline: false Aug 1 21:57:28.853208168 (52h0m18.790866657s) [upload=49973]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 1 21:57:28.853212666 (52h0m18.790871165s) [upload=49973]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 1 21:57:29.095087519 (52h0m19.032746008s) [seg_pos={0 0}, upload=49973]: Begin upload segment... Aug 1 21:57:29.095129960 (52h0m19.032788459s) [seg_pos={0 0}, upload=49973]: Done begin upload segment. Aug 1 21:57:29.095164997 (52h0m19.032823496s) [noise=true , address=77.90.31.21:4878 , piece_id=BLNTYBOAIZWISSUJYK6BSBPTSNL6QWLGEN6AP6P4NPDVSURYG5AA, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.095171800 (52h0m19.032830289s) [noise=true , address=104.36.82.6:4875 , piece_id=CS5AMB7CIBQWWOU57Z3RCCSNVKIPE4B5QWU5YIQYI2UCWH7PAPVA, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.095510027 (52h0m19.033168526s) [noise=true , address=77.90.31.18:4806 , piece_id=52B6CAHGLFVTFIDQHHHAWKCBP2WJJY7OTTGRZZ7CEUH3HDA43Q6A, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.095544193 (52h0m19.033202692s) [noise=true , address=77.90.31.10:4518 , piece_id=WUOVXUSDG6YQTDPWG4YO7VLJNZZVLH6VRGMWOJABG2EJRU7XHP5A, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.095774283 (52h0m19.033432782s) [noise=true , address=104.36.82.7:4909 , piece_id=QHJCVBT6KTU34K2KWFNACWYVVE3WJ2RU76BHVSKL4HLCZEFXZYPA, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.095819880 (52h0m19.033478379s) [noise=true , address=45.91.116.12:4225 , piece_id=PBQPLA5ZYRFPSKHDKCEFOEBS5T3GYMP2FLSFDQWJLVY756KPLJNQ, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.096021656 (52h0m19.033680155s) [noise=true , address=45.91.116.13:4153 , piece_id=PMLKMZO66KXF3ZONO5ZZOQZICFK5XNGL6MN5FFBKICVFU5AWU6PQ, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.096069618 (52h0m19.033728117s) [noise=true , address=45.91.116.9:4262 , piece_id=A3F4L3RR6WPFMBFFKLATNGG3CASBIPWQP5O3G5URFNTXBE3ZW3VA, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.096348041 (52h0m19.03400654s) [noise=true , address=104.36.82.5:4840 , piece_id=UYVKVFCJJP7XZYRIVPS4ABGEMDUMNR7YW4QGG5ZFTFCGNCJO3CSQ, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.096354803 (52h0m19.034013292s) [noise=true , address=77.90.31.12:4590 , piece_id=D6FUBIFCQJSDRGWEP5GG34AQPNB3UROCYOATD6SRTJ7KAYVMFCOA, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.096608579 (52h0m19.034267078s) [noise=true , address=77.90.31.16:4733 , piece_id=DMASMGOUATLPEKCGLHKNNIMMFYVVGSULNUOMJU6RJY3ULBDOY6KQ, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.096830774 (52h0m19.034489273s) [noise=true , address=104.36.82.4:4802 , piece_id=CLM2G33M6AK5HIESRW6LZY33NKHIR2PGITWJ23NPILRCA6UYIEHA, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.097107233 (52h0m19.034765732s) [noise=true , address=77.90.31.19:4843 , piece_id=UFONZYHL7EAFYE5PRW32RKUVQPMZPXBEFLLIFJMAB2TXJ3C6KKNA, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.097324328 (52h0m19.034982827s) [noise=true , address=77.90.31.17:4771 , piece_id=Y3WSALJX23CWPPYQ3XYMCFOSBLBX2E2JQCCS7EOEYNBWY4CDRT3Q, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.098682446 (52h0m19.036340945s) [noise=true , address=77.90.31.15:4698 , piece_id=TN4OTPUD47FD6WOLZBEHNDI43WY6QVO5JK2JNJXNVM2IT6XMGERQ, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.099041603 (52h0m19.036700102s) [noise=true , address=45.91.116.11:4118 , piece_id=AKZIOAR6NRNO7JWCQZIKNSL7IUH5S4IG3KPG3IY2FAKG74DH3E3Q, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:30.718412255 (52h0m20.656070754s) [upload=49973]: Next returned nil segment Aug 1 21:57:30.718429869 (52h0m20.656088368s) [upload=49973]: Waiting for error group managing segments... Aug 1 21:57:31.107386675 (52h0m21.045045164s) [noise=true , address=45.91.116.13:4153 , piece_id=PMLKMZO66KXF3ZONO5ZZOQZICFK5XNGL6MN5FFBKICVFU5AWU6PQ, seg_pos={0 0}, upload=49973]: Done uploading piece. err: <nil> Aug 1 21:57:31.108529412 (52h0m21.046187911s) [noise=true , address=77.90.31.12:4590 , piece_id=D6FUBIFCQJSDRGWEP5GG34AQPNB3UROCYOATD6SRTJ7KAYVMFCOA, seg_pos={0 0}, upload=49973]: Done uploading piece. err: <nil> Aug 1 21:57:31.109975858 (52h0m21.047634347s) [noise=true , address=77.90.31.15:4698 , piece_id=TN4OTPUD47FD6WOLZBEHNDI43WY6QVO5JK2JNJXNVM2IT6XMGERQ, seg_pos={0 0}, upload=49973]: Done uploading piece. err: <nil> Aug 1 21:57:31.115273013 (52h0m21.052931502s) [noise=true , address=77.90.31.10:4518 , piece_id=WUOVXUSDG6YQTDPWG4YO7VLJNZZVLH6VRGMWOJABG2EJRU7XHP5A, seg_pos={0 0}, upload=49973]: Done uploading piece. err: <nil> Aug 1 21:57:31.116767281 (52h0m21.054425781s) [noise=true , address=77.90.31.18:4806 , piece_id=52B6CAHGLFVTFIDQHHHAWKCBP2WJJY7OTTGRZZ7CEUH3HDA43Q6A, seg_pos={0 0}, upload=49973]: Done uploading piece. err: <nil> Aug 1 21:57:31.201146107 (52h0m21.138804606s) [noise=true , address=77.90.31.16:4733 , piece_id=DMASMGOUATLPEKCGLHKNNIMMFYVVGSULNUOMJU6RJY3ULBDOY6KQ, seg_pos={0 0}, upload=49973]: Done uploading piece. err: <nil> Aug 1 21:57:31.201833732 (52h0m21.139492231s) [noise=true , address=77.90.31.19:4843 , piece_id=UFONZYHL7EAFYE5PRW32RKUVQPMZPXBEFLLIFJMAB2TXJ3C6KKNA, seg_pos={0 0}, upload=49973]: Done uploading piece. err: <nil> Aug 1 21:57:31.211007613 (52h0m21.148666112s) [noise=true , address=45.91.116.9:4262 , piece_id=A3F4L3RR6WPFMBFFKLATNGG3CASBIPWQP5O3G5URFNTXBE3ZW3VA, seg_pos={0 0}, upload=49973]: Done uploading piece. err: <nil> Aug 1 21:57:31.215929169 (52h0m21.153587668s) [noise=true , address=77.90.31.17:4771 , piece_id=Y3WSALJX23CWPPYQ3XYMCFOSBLBX2E2JQCCS7EOEYNBWY4CDRT3Q, seg_pos={0 0}, upload=49973]: Done uploading piece. err: <nil> Aug 1 21:57:31.222973975 (52h0m21.160632474s) [noise=true , address=77.90.31.21:4878 , piece_id=BLNTYBOAIZWISSUJYK6BSBPTSNL6QWLGEN6AP6P4NPDVSURYG5AA, seg_pos={0 0}, upload=49973]: Done uploading piece. err: <nil> Aug 1 21:57:31.234392441 (52h0m21.17205094s) [noise=true , address=45.91.116.12:4225 , piece_id=PBQPLA5ZYRFPSKHDKCEFOEBS5T3GYMP2FLSFDQWJLVY756KPLJNQ, seg_pos={0 0}, upload=49973]: Done uploading piece. err: <nil> Aug 1 21:57:31.284794305 (52h0m21.222452804s) [noise=true , address=45.91.116.11:4118 , piece_id=AKZIOAR6NRNO7JWCQZIKNSL7IUH5S4IG3KPG3IY2FAKG74DH3E3Q, seg_pos={0 0}, upload=49973]: Done uploading piece. err: <nil> Aug 1 21:57:58.202420932 (52h0m48.140079471s) [noise=true , address=104.36.82.6:4875 , piece_id=CS5AMB7CIBQWWOU57Z3RCCSNVKIPE4B5QWU5YIQYI2UCWH7PAPVA, seg_pos={0 0}, upload=49973]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:104.36.82.6:4875): protocol: write tcp 188.166.225.170:40166->104.36.82.6:4875: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:58.202441131 (52h0m48.14009963s) [noise=true , address=104.36.82.5:4840 , piece_id=UYVKVFCJJP7XZYRIVPS4ABGEMDUMNR7YW4QGG5ZFTFCGNCJO3CSQ, seg_pos={0 0}, upload=49973]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:104.36.82.5:4840): protocol: rpc: dial tcp 104.36.82.5:4840: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:58.202448445 (52h0m48.140106954s) [noise=true , address=104.36.82.7:4909 , piece_id=QHJCVBT6KTU34K2KWFNACWYVVE3WJ2RU76BHVSKL4HLCZEFXZYPA, seg_pos={0 0}, upload=49973]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:104.36.82.7:4909): protocol: rpc: dial tcp 104.36.82.7:4909: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:58.202496487 (52h0m48.140155026s) [noise=true , address=104.36.82.4:4802 , piece_id=CLM2G33M6AK5HIESRW6LZY33NKHIR2PGITWJ23NPILRCA6UYIEHA, seg_pos={0 0}, upload=49973]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: write tcp 188.166.225.170:47696->104.36.82.4:4802: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:58.202524240 (52h0m48.140182739s) [seg_pos={0 0}, upload=49973]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Aug 1 21:57:58.435983932 (52h0m48.373642431s) [upload=49973]: Done uploading segments... Aug 1 21:57:58.436007667 (52h0m48.373666166s) [upload=49973]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.991086846s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [.............................XXXX...........................] duration: 6.047199144s FAIL (2025-08-01 22:16:12 UTC) orbiter warning: Operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 7) ======LOGS====== K4BW277ZSGXJYQDA66OMEBVDC7DAWSOJAPX42HANNGRZN35KHQ, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.481101510 (51h59m34.418760009s) [noise=true , address=104.36.82.5:4842 , piece_id=LV7YAOMPE2AV7TF2WPWR7QJNLT4JZAIPSVZN3E23U5KDBNKZFD6A, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.481422003 (51h59m34.419080502s) [noise=true , address=79.127.205.235:4683 , piece_id=3QHDNAPG4GRLJFVVNJNFXNGKTGKUEMWWUBMW62HHE5N5LJDO4KWA, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.482030897 (51h59m34.419689396s) [noise=true , address=79.127.219.51:4213 , piece_id=L2WSPI5BEIFLGQKJIK4EWXFQXNQRQ7HVKPOWFYRPPUGNAL5DVYWA, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.482285215 (51h59m34.419943704s) [noise=true , address=104.36.82.14:4251 , piece_id=O7NADU347E2Q26TPD3IUCK6WE5UVRITC6ALZHBE5SWYTKWXFBMXA, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.482518370 (51h59m34.420176859s) [noise=true , address=79.127.220.98:4402 , piece_id=DVLSYRFFZ6UT3QRSVAN7AMXVYZNI6PWG2IITHLRUUJATJVK3QOGA, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.482724304 (51h59m34.420382803s) [noise=true , address=79.127.201.214:4297 , piece_id=MPTMGVEYC4SATIN4ANCR2ELVRXHYWRDLMHWQW7HM6UOIV2N6R7ZA, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.482928555 (51h59m34.420587054s) [noise=true , address=104.36.82.37:4680 , piece_id=X4PDUE4WXUUWX6JQVO4NWPQCBMHMHC5YHPIHDQVYYOI2FXV2GVJQ, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.483134960 (51h59m34.420793449s) [noise=true , address=104.36.82.50:4027 , piece_id=OW7BODU5N7ZEEGE2FHSLNYVJT2Z25LY3J42TZEOR5SP2EWIN6X7A, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.483338480 (51h59m34.420996979s) [noise=true , address=104.36.82.55:4204 , piece_id=R5CYXWAVV4ZEMCV7LF4D4XDC4G6KLVBONAH2TH5EIEVL7QHUTRWQ, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.483582847 (51h59m34.421241356s) [noise=true , address=104.36.82.11:4134 , piece_id=RMTNNNC2VD7MGWJDAMT6ZCF3GR43SHANFVAGJ6EHVH7WO5XA6VTA, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.483833737 (51h59m34.421492236s) [noise=true , address=104.36.82.43:4856 , piece_id=2SINBLHGRUOWUO4XHBNBG45YE3PW7RNF3QSNVCQXNCFEFHRBFNJQ, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.484021896 (51h59m34.421680385s) [noise=true , address=104.36.82.52:4101 , piece_id=XYPLQM7QITCCX7T4NKIXP5MZ7I6F24EJJTG37K4NPOBKTUFHULRQ, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.484301662 (51h59m34.421960182s) [noise=true , address=79.127.201.215:4352 , piece_id=I6BRYI5XIACAAAAAE7HQOBCF4OTALMOVPPNBWQCRSH2MM6WUUURQ, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.484712097 (51h59m34.422370596s) [noise=true , address=64.49.13.138:4484 , piece_id=DZ5Y4HTS6BKW2DJ5CVOB64TW4SXB6DNCJEGFUJPLAUXUEHRZRTUA, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:45.576021470 (51h59m35.513679959s) [upload=49960]: Next returned nil segment Aug 1 21:56:45.576036609 (51h59m35.513695098s) [upload=49960]: Waiting for error group managing segments... Aug 1 21:56:45.860378776 (51h59m35.798037275s) [noise=true , address=64.49.13.138:4484 , piece_id=DZ5Y4HTS6BKW2DJ5CVOB64TW4SXB6DNCJEGFUJPLAUXUEHRZRTUA, seg_pos={0 0}, upload=49960]: Done uploading piece. err: <nil> Aug 1 21:56:45.884184188 (51h59m35.821842687s) [noise=true , address=79.127.205.235:4683 , piece_id=3QHDNAPG4GRLJFVVNJNFXNGKTGKUEMWWUBMW62HHE5N5LJDO4KWA, seg_pos={0 0}, upload=49960]: Done uploading piece. err: <nil> Aug 1 21:56:46.145493612 (51h59m36.083152101s) [noise=true , address=79.127.220.98:4402 , piece_id=DVLSYRFFZ6UT3QRSVAN7AMXVYZNI6PWG2IITHLRUUJATJVK3QOGA, seg_pos={0 0}, upload=49960]: Done uploading piece. err: <nil> Aug 1 21:56:46.257505313 (51h59m36.195163802s) [noise=true , address=79.127.220.97:4371 , piece_id=5WGPRFKQETIHTSMA42AM2C3UGCFYJ3DMMPAI2L7DD6E2P37KJKNA, seg_pos={0 0}, upload=49960]: Done uploading piece. err: <nil> Aug 1 21:56:46.334426181 (51h59m36.27208468s) [noise=true , address=79.127.201.215:4352 , piece_id=I6BRYI5XIACAAAAAE7HQOBCF4OTALMOVPPNBWQCRSH2MM6WUUURQ, seg_pos={0 0}, upload=49960]: Done uploading piece. err: <nil> Aug 1 21:56:46.386119275 (51h59m36.323777774s) [noise=true , address=79.127.201.214:4297 , piece_id=MPTMGVEYC4SATIN4ANCR2ELVRXHYWRDLMHWQW7HM6UOIV2N6R7ZA, seg_pos={0 0}, upload=49960]: Done uploading piece. err: <nil> Aug 1 21:56:46.395362048 (51h59m36.333020547s) [noise=true , address=79.127.219.51:4213 , piece_id=L2WSPI5BEIFLGQKJIK4EWXFQXNQRQ7HVKPOWFYRPPUGNAL5DVYWA, seg_pos={0 0}, upload=49960]: Done uploading piece. err: <nil> Aug 1 21:57:43.392434383 (52h0m33.330092872s) [noise=true , address=104.36.82.37:4680 , piece_id=X4PDUE4WXUUWX6JQVO4NWPQCBMHMHC5YHPIHDQVYYOI2FXV2GVJQ, seg_pos={0 0}, upload=49960]: Done uploading piece. err: ecclient: upload failed (node:1bXAqpxGhUvzwYEDGBqYRhZdTeMJcV3TYfngkKTCumpt9Nupi3, address:104.36.82.37:4680): protocol: rpc: dial tcp 104.36.82.37:4680: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:43.392477967 (52h0m33.330136466s) [noise=true , address=104.36.82.11:4134 , piece_id=RMTNNNC2VD7MGWJDAMT6ZCF3GR43SHANFVAGJ6EHVH7WO5XA6VTA, seg_pos={0 0}, upload=49960]: Done uploading piece. err: ecclient: upload failed (node:12R3Z5AgkBmWXMv5XgRyfcyqkG2gmyJs9ogT24kcLy6nuB9bzUG, address:104.36.82.11:4134): protocol: rpc: dial tcp 104.36.82.11:4134: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:43.392514086 (52h0m33.330172585s) [noise=true , address=104.36.82.55:4204 , piece_id=R5CYXWAVV4ZEMCV7LF4D4XDC4G6KLVBONAH2TH5EIEVL7QHUTRWQ, seg_pos={0 0}, upload=49960]: Done uploading piece. err: ecclient: upload failed (node:1rBZH6aLHsnWh22uNzaMiMwmLJCDMn4bYbNC5jRG7KCEuRfqpH, address:104.36.82.55:4204): protocol: write tcp 188.166.225.170:48550->104.36.82.55:4204: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:43.392546537 (52h0m33.330205027s) [noise=true , address=104.36.82.59:4327 , piece_id=OZDEYIY6B4WYZULRDRL2UBOJH2JQNY3YWSITZKWDO7GTT72AN5NQ, seg_pos={0 0}, upload=49960]: Done uploading piece. err: ecclient: upload failed (node:1V4C6AWGbyYQWfinoSj7iKovHeTJ9hqypyC5U9rZ8pveXBBfCY, address:104.36.82.59:4327): protocol: rpc: dial tcp 104.36.82.59:4327: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:43.392579040 (52h0m33.33023754s) [noise=true , address=104.36.82.5:4842 , piece_id=LV7YAOMPE2AV7TF2WPWR7QJNLT4JZAIPSVZN3E23U5KDBNKZFD6A, seg_pos={0 0}, upload=49960]: Done uploading piece. err: ecclient: upload failed (node:12q4VdT5BcjUb16PBYHK6zrnW8dqENKrcGx9zdLtNefDVB1MUSn, address:104.36.82.5:4842): protocol: context deadline exceeded; context deadline exceeded Aug 1 21:57:43.392627953 (52h0m33.330286442s) [noise=true , address=104.36.82.52:4101 , piece_id=XYPLQM7QITCCX7T4NKIXP5MZ7I6F24EJJTG37K4NPOBKTUFHULRQ, seg_pos={0 0}, upload=49960]: Done uploading piece. err: ecclient: upload failed (node:12WWXeBjFZR9PzciNfs35fFcx87N9GwDpMsPHnksFDugtaBGWp4, address:104.36.82.52:4101): protocol: rpc: dial tcp 104.36.82.52:4101: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:43.392653192 (52h0m33.330311691s) [noise=true , address=104.36.82.14:4251 , piece_id=O7NADU347E2Q26TPD3IUCK6WE5UVRITC6ALZHBE5SWYTKWXFBMXA, seg_pos={0 0}, upload=49960]: Done uploading piece. err: ecclient: upload failed (node:12btf6Z38aTMx7MaBxhUyZtFqUPgeMtP2aU8Y6FbV8zwu57zArd, address:104.36.82.14:4251): protocol: rpc: dial tcp 104.36.82.14:4251: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:43.392670836 (52h0m33.330329335s) [noise=true , address=104.36.82.12:4166 , piece_id=6ZIWCOBY76GJ6DHZF5G23U4IYTOFB5SMQW5TW3YITYH227PV2FGA, seg_pos={0 0}, upload=49960]: Done uploading piece. err: ecclient: upload failed (node:12iCB7c4ZfpqafpD1PtpbE6BfU81wJW65LmrPVJXtD58yeuoR9t, address:104.36.82.12:4166): protocol: rpc: dial tcp 104.36.82.12:4166: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:43.392698018 (52h0m33.330356517s) [noise=true , address=104.36.82.50:4027 , piece_id=OW7BODU5N7ZEEGE2FHSLNYVJT2Z25LY3J42TZEOR5SP2EWIN6X7A, seg_pos={0 0}, upload=49960]: Done uploading piece. err: ecclient: upload failed (node:1zEQhyuVxSHwdDUgDfcTog2W3oLBx4z9bYUMHpaaoLU1BTU9DQ, address:104.36.82.50:4027): protocol: write tcp 188.166.225.170:37784->104.36.82.50:4027: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:43.392715381 (52h0m33.33037387s) [noise=true , address=104.36.82.43:4856 , piece_id=2SINBLHGRUOWUO4XHBNBG45YE3PW7RNF3QSNVCQXNCFEFHRBFNJQ, seg_pos={0 0}, upload=49960]: Done uploading piece. err: ecclient: upload failed (node:12bQxwBfPEGtmMp6mtnWKsoZt3eMEjLdyYFahETWGHeKJ7BhkiZ, address:104.36.82.43:4856): protocol: rpc: dial tcp 104.36.82.43:4856: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:43.392730059 (52h0m33.330388558s) [noise=true , address=104.36.82.10:4082 , piece_id=45K4BW277ZSGXJYQDA66OMEBVDC7DAWSOJAPX42HANNGRZN35KHQ, seg_pos={0 0}, upload=49960]: Done uploading piece. err: ecclient: upload failed (node:12DKsNCtqxLvpJwLszdLs74V3eb8NH2VAV9V3bngzUczawmNoLK, address:104.36.82.10:4082): protocol: write tcp 188.166.225.170:46638->104.36.82.10:4082: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:43.392753424 (52h0m33.330411923s) [seg_pos={0 0}, upload=49960]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil> Aug 1 21:57:43.624550997 (52h0m33.562209566s) [upload=49960]: Done uploading segments... Aug 1 21:57:43.624582868 (52h0m33.562241407s) [upload=49960]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7) Duration: 6.047232328s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 5 out of 60 is failed (threshold=3) [...............X............X.....X.......X.....X...........] duration: 1.952749916s FAIL (2025-08-01 22:15:57 UTC) orbiter warning: Operation: ListObjects unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/1K Duration: 1.952784432s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [..............................XXX...........................] duration: 5.49269891s FAIL (2025-08-01 22:15:50 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.492754706s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 8 out of 60 is failed (threshold=3) [............................XXXXXXX........................X] duration: 6.073767858s FAIL (2025-08-01 22:15:42 UTC) orbiter warning: Operation: sync2.Copy digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/4M):: after 92e23610b5d37ceb174fa119b5e47214ac2cd632235616ccf3b7f0b13a4a6031 != before 92313f7b7503eb9340206bbda9efbdb260cf41b185fb20ecbdf772c5fb959aac Duration: 6.073787706s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [..............................XX.X.X........................] duration: 6.215218958s FAIL (2025-08-01 22:15:37 UTC) orbiter warning: Operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 7NK72474VWE4E3YOBCDZ5O5ZOH2MSA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB): protocol: write tcp 188.166.225.170:38368->104.36.82.12:4151: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:56:50.418604753 (51h59m40.356263252s) [seg_pos={0 0}, upload=49961]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 1 21:56:50.418680879 (51h59m40.356339378s) [upload=49961]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 1 21:56:50.418685998 (51h59m40.356344487s) [upload=49961]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 1 21:56:50.726563872 (51h59m40.664222371s) [upload=49961]: Done uploading segments... Aug 1 21:56:50.726586586 (51h59m40.664245085s) [upload=49961]: Upload finished. err: <nil> Aug 1 21:57:28.852869669 (52h0m18.790528158s) [upload=49973]: Starting upload Aug 1 21:57:28.852942389 (52h0m18.790600888s) [upload=49973]: Done starting upload Aug 1 21:57:28.853007122 (52h0m18.790665611s) [upload=49973]: Uploading segments... Aug 1 21:57:28.853174002 (52h0m18.790832491s) [upload=49973]: Got next segment. Inline: false Aug 1 21:57:28.853208168 (52h0m18.790866657s) [upload=49973]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 1 21:57:28.853212666 (52h0m18.790871165s) [upload=49973]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 1 21:57:29.095087519 (52h0m19.032746008s) [seg_pos={0 0}, upload=49973]: Begin upload segment... Aug 1 21:57:29.095129960 (52h0m19.032788459s) [seg_pos={0 0}, upload=49973]: Done begin upload segment. Aug 1 21:57:29.095164997 (52h0m19.032823496s) [noise=true , address=77.90.31.21:4878 , piece_id=BLNTYBOAIZWISSUJYK6BSBPTSNL6QWLGEN6AP6P4NPDVSURYG5AA, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.095171800 (52h0m19.032830289s) [noise=true , address=104.36.82.6:4875 , piece_id=CS5AMB7CIBQWWOU57Z3RCCSNVKIPE4B5QWU5YIQYI2UCWH7PAPVA, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.095510027 (52h0m19.033168526s) [noise=true , address=77.90.31.18:4806 , piece_id=52B6CAHGLFVTFIDQHHHAWKCBP2WJJY7OTTGRZZ7CEUH3HDA43Q6A, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.095544193 (52h0m19.033202692s) [noise=true , address=77.90.31.10:4518 , piece_id=WUOVXUSDG6YQTDPWG4YO7VLJNZZVLH6VRGMWOJABG2EJRU7XHP5A, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.095774283 (52h0m19.033432782s) [noise=true , address=104.36.82.7:4909 , piece_id=QHJCVBT6KTU34K2KWFNACWYVVE3WJ2RU76BHVSKL4HLCZEFXZYPA, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.095819880 (52h0m19.033478379s) [noise=true , address=45.91.116.12:4225 , piece_id=PBQPLA5ZYRFPSKHDKCEFOEBS5T3GYMP2FLSFDQWJLVY756KPLJNQ, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.096021656 (52h0m19.033680155s) [noise=true , address=45.91.116.13:4153 , piece_id=PMLKMZO66KXF3ZONO5ZZOQZICFK5XNGL6MN5FFBKICVFU5AWU6PQ, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.096069618 (52h0m19.033728117s) [noise=true , address=45.91.116.9:4262 , piece_id=A3F4L3RR6WPFMBFFKLATNGG3CASBIPWQP5O3G5URFNTXBE3ZW3VA, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.096348041 (52h0m19.03400654s) [noise=true , address=104.36.82.5:4840 , piece_id=UYVKVFCJJP7XZYRIVPS4ABGEMDUMNR7YW4QGG5ZFTFCGNCJO3CSQ, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.096354803 (52h0m19.034013292s) [noise=true , address=77.90.31.12:4590 , piece_id=D6FUBIFCQJSDRGWEP5GG34AQPNB3UROCYOATD6SRTJ7KAYVMFCOA, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.096608579 (52h0m19.034267078s) [noise=true , address=77.90.31.16:4733 , piece_id=DMASMGOUATLPEKCGLHKNNIMMFYVVGSULNUOMJU6RJY3ULBDOY6KQ, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.096830774 (52h0m19.034489273s) [noise=true , address=104.36.82.4:4802 , piece_id=CLM2G33M6AK5HIESRW6LZY33NKHIR2PGITWJ23NPILRCA6UYIEHA, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.097107233 (52h0m19.034765732s) [noise=true , address=77.90.31.19:4843 , piece_id=UFONZYHL7EAFYE5PRW32RKUVQPMZPXBEFLLIFJMAB2TXJ3C6KKNA, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.097324328 (52h0m19.034982827s) [noise=true , address=77.90.31.17:4771 , piece_id=Y3WSALJX23CWPPYQ3XYMCFOSBLBX2E2JQCCS7EOEYNBWY4CDRT3Q, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.098682446 (52h0m19.036340945s) [noise=true , address=77.90.31.15:4698 , piece_id=TN4OTPUD47FD6WOLZBEHNDI43WY6QVO5JK2JNJXNVM2IT6XMGERQ, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.099041603 (52h0m19.036700102s) [noise=true , address=45.91.116.11:4118 , piece_id=AKZIOAR6NRNO7JWCQZIKNSL7IUH5S4IG3KPG3IY2FAKG74DH3E3Q, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:30.718412255 (52h0m20.656070754s) [upload=49973]: Next returned nil segment Aug 1 21:57:30.718429869 (52h0m20.656088368s) [upload=49973]: Waiting for error group managing segments... Aug 1 21:57:31.107386675 (52h0m21.045045164s) [noise=true , address=45.91.116.13:4153 , piece_id=PMLKMZO66KXF3ZONO5ZZOQZICFK5XNGL6MN5FFBKICVFU5AWU6PQ, seg_pos={0 0}, upload=49973]: Done uploading piece. err: <nil> Aug 1 21:57:31.108529412 (52h0m21.046187911s) [noise=true , address=77.90.31.12:4590 , piece_id=D6FUBIFCQJSDRGWEP5GG34AQPNB3UROCYOATD6SRTJ7KAYVMFCOA, seg_pos={0 0}, upload=49973]: Done uploading piece. err: <nil> Aug 1 21:57:31.109975858 (52h0m21.047634347s) [noise=true , address=77.90.31.15:4698 , piece_id=TN4OTPUD47FD6WOLZBEHNDI43WY6QVO5JK2JNJXNVM2IT6XMGERQ, seg_pos={0 0}, upload=49973]: Done uploading piece. err: <nil> Aug 1 21:57:31.115273013 (52h0m21.052931502s) [noise=true , address=77.90.31.10:4518 , piece_id=WUOVXUSDG6YQTDPWG4YO7VLJNZZVLH6VRGMWOJABG2EJRU7XHP5A, seg_pos={0 0}, upload=49973]: Done uploading piece. err: <nil> Aug 1 21:57:31.116767281 (52h0m21.054425781s) [noise=true , address=77.90.31.18:4806 , piece_id=52B6CAHGLFVTFIDQHHHAWKCBP2WJJY7OTTGRZZ7CEUH3HDA43Q6A, seg_pos={0 0}, upload=49973]: Done uploading piece. err: <nil> Aug 1 21:57:31.201146107 (52h0m21.138804606s) [noise=true , address=77.90.31.16:4733 , piece_id=DMASMGOUATLPEKCGLHKNNIMMFYVVGSULNUOMJU6RJY3ULBDOY6KQ, seg_pos={0 0}, upload=49973]: Done uploading piece. err: <nil> Aug 1 21:57:31.201833732 (52h0m21.139492231s) [noise=true , address=77.90.31.19:4843 , piece_id=UFONZYHL7EAFYE5PRW32RKUVQPMZPXBEFLLIFJMAB2TXJ3C6KKNA, seg_pos={0 0}, upload=49973]: Done uploading piece. err: <nil> Aug 1 21:57:31.211007613 (52h0m21.148666112s) [noise=true , address=45.91.116.9:4262 , piece_id=A3F4L3RR6WPFMBFFKLATNGG3CASBIPWQP5O3G5URFNTXBE3ZW3VA, seg_pos={0 0}, upload=49973]: Done uploading piece. err: <nil> Aug 1 21:57:31.215929169 (52h0m21.153587668s) [noise=true , address=77.90.31.17:4771 , piece_id=Y3WSALJX23CWPPYQ3XYMCFOSBLBX2E2JQCCS7EOEYNBWY4CDRT3Q, seg_pos={0 0}, upload=49973]: Done uploading piece. err: <nil> Aug 1 21:57:31.222973975 (52h0m21.160632474s) [noise=true , address=77.90.31.21:4878 , piece_id=BLNTYBOAIZWISSUJYK6BSBPTSNL6QWLGEN6AP6P4NPDVSURYG5AA, seg_pos={0 0}, upload=49973]: Done uploading piece. err: <nil> Aug 1 21:57:31.234392441 (52h0m21.17205094s) [noise=true , address=45.91.116.12:4225 , piece_id=PBQPLA5ZYRFPSKHDKCEFOEBS5T3GYMP2FLSFDQWJLVY756KPLJNQ, seg_pos={0 0}, upload=49973]: Done uploading piece. err: <nil> Aug 1 21:57:31.284794305 (52h0m21.222452804s) [noise=true , address=45.91.116.11:4118 , piece_id=AKZIOAR6NRNO7JWCQZIKNSL7IUH5S4IG3KPG3IY2FAKG74DH3E3Q, seg_pos={0 0}, upload=49973]: Done uploading piece. err: <nil> Aug 1 21:57:58.202420932 (52h0m48.140079471s) [noise=true , address=104.36.82.6:4875 , piece_id=CS5AMB7CIBQWWOU57Z3RCCSNVKIPE4B5QWU5YIQYI2UCWH7PAPVA, seg_pos={0 0}, upload=49973]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:104.36.82.6:4875): protocol: write tcp 188.166.225.170:40166->104.36.82.6:4875: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:58.202441131 (52h0m48.14009963s) [noise=true , address=104.36.82.5:4840 , piece_id=UYVKVFCJJP7XZYRIVPS4ABGEMDUMNR7YW4QGG5ZFTFCGNCJO3CSQ, seg_pos={0 0}, upload=49973]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:104.36.82.5:4840): protocol: rpc: dial tcp 104.36.82.5:4840: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:58.202448445 (52h0m48.140106954s) [noise=true , address=104.36.82.7:4909 , piece_id=QHJCVBT6KTU34K2KWFNACWYVVE3WJ2RU76BHVSKL4HLCZEFXZYPA, seg_pos={0 0}, upload=49973]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:104.36.82.7:4909): protocol: rpc: dial tcp 104.36.82.7:4909: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:58.202496487 (52h0m48.140155026s) [noise=true , address=104.36.82.4:4802 , piece_id=CLM2G33M6AK5HIESRW6LZY33NKHIR2PGITWJ23NPILRCA6UYIEHA, seg_pos={0 0}, upload=49973]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: write tcp 188.166.225.170:47696->104.36.82.4:4802: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:58.202524240 (52h0m48.140182739s) [seg_pos={0 0}, upload=49973]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Aug 1 21:57:58.435983932 (52h0m48.373642431s) [upload=49973]: Done uploading segments... Aug 1 21:57:58.436007667 (52h0m48.373666166s) [upload=49973]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.215247181s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 6.090607512s FAIL (2025-08-01 22:15:37 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.090631657s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [..............................XXXX..........................] duration: 5.929679136s FAIL (2025-08-01 22:15:24 UTC) orbiter warning: Operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 7) ======LOGS====== K4BW277ZSGXJYQDA66OMEBVDC7DAWSOJAPX42HANNGRZN35KHQ, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.481101510 (51h59m34.418760009s) [noise=true , address=104.36.82.5:4842 , piece_id=LV7YAOMPE2AV7TF2WPWR7QJNLT4JZAIPSVZN3E23U5KDBNKZFD6A, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.481422003 (51h59m34.419080502s) [noise=true , address=79.127.205.235:4683 , piece_id=3QHDNAPG4GRLJFVVNJNFXNGKTGKUEMWWUBMW62HHE5N5LJDO4KWA, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.482030897 (51h59m34.419689396s) [noise=true , address=79.127.219.51:4213 , piece_id=L2WSPI5BEIFLGQKJIK4EWXFQXNQRQ7HVKPOWFYRPPUGNAL5DVYWA, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.482285215 (51h59m34.419943704s) [noise=true , address=104.36.82.14:4251 , piece_id=O7NADU347E2Q26TPD3IUCK6WE5UVRITC6ALZHBE5SWYTKWXFBMXA, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.482518370 (51h59m34.420176859s) [noise=true , address=79.127.220.98:4402 , piece_id=DVLSYRFFZ6UT3QRSVAN7AMXVYZNI6PWG2IITHLRUUJATJVK3QOGA, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.482724304 (51h59m34.420382803s) [noise=true , address=79.127.201.214:4297 , piece_id=MPTMGVEYC4SATIN4ANCR2ELVRXHYWRDLMHWQW7HM6UOIV2N6R7ZA, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.482928555 (51h59m34.420587054s) [noise=true , address=104.36.82.37:4680 , piece_id=X4PDUE4WXUUWX6JQVO4NWPQCBMHMHC5YHPIHDQVYYOI2FXV2GVJQ, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.483134960 (51h59m34.420793449s) [noise=true , address=104.36.82.50:4027 , piece_id=OW7BODU5N7ZEEGE2FHSLNYVJT2Z25LY3J42TZEOR5SP2EWIN6X7A, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.483338480 (51h59m34.420996979s) [noise=true , address=104.36.82.55:4204 , piece_id=R5CYXWAVV4ZEMCV7LF4D4XDC4G6KLVBONAH2TH5EIEVL7QHUTRWQ, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.483582847 (51h59m34.421241356s) [noise=true , address=104.36.82.11:4134 , piece_id=RMTNNNC2VD7MGWJDAMT6ZCF3GR43SHANFVAGJ6EHVH7WO5XA6VTA, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.483833737 (51h59m34.421492236s) [noise=true , address=104.36.82.43:4856 , piece_id=2SINBLHGRUOWUO4XHBNBG45YE3PW7RNF3QSNVCQXNCFEFHRBFNJQ, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.484021896 (51h59m34.421680385s) [noise=true , address=104.36.82.52:4101 , piece_id=XYPLQM7QITCCX7T4NKIXP5MZ7I6F24EJJTG37K4NPOBKTUFHULRQ, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.484301662 (51h59m34.421960182s) [noise=true , address=79.127.201.215:4352 , piece_id=I6BRYI5XIACAAAAAE7HQOBCF4OTALMOVPPNBWQCRSH2MM6WUUURQ, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.484712097 (51h59m34.422370596s) [noise=true , address=64.49.13.138:4484 , piece_id=DZ5Y4HTS6BKW2DJ5CVOB64TW4SXB6DNCJEGFUJPLAUXUEHRZRTUA, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:45.576021470 (51h59m35.513679959s) [upload=49960]: Next returned nil segment Aug 1 21:56:45.576036609 (51h59m35.513695098s) [upload=49960]: Waiting for error group managing segments... Aug 1 21:56:45.860378776 (51h59m35.798037275s) [noise=true , address=64.49.13.138:4484 , piece_id=DZ5Y4HTS6BKW2DJ5CVOB64TW4SXB6DNCJEGFUJPLAUXUEHRZRTUA, seg_pos={0 0}, upload=49960]: Done uploading piece. err: <nil> Aug 1 21:56:45.884184188 (51h59m35.821842687s) [noise=true , address=79.127.205.235:4683 , piece_id=3QHDNAPG4GRLJFVVNJNFXNGKTGKUEMWWUBMW62HHE5N5LJDO4KWA, seg_pos={0 0}, upload=49960]: Done uploading piece. err: <nil> Aug 1 21:56:46.145493612 (51h59m36.083152101s) [noise=true , address=79.127.220.98:4402 , piece_id=DVLSYRFFZ6UT3QRSVAN7AMXVYZNI6PWG2IITHLRUUJATJVK3QOGA, seg_pos={0 0}, upload=49960]: Done uploading piece. err: <nil> Aug 1 21:56:46.257505313 (51h59m36.195163802s) [noise=true , address=79.127.220.97:4371 , piece_id=5WGPRFKQETIHTSMA42AM2C3UGCFYJ3DMMPAI2L7DD6E2P37KJKNA, seg_pos={0 0}, upload=49960]: Done uploading piece. err: <nil> Aug 1 21:56:46.334426181 (51h59m36.27208468s) [noise=true , address=79.127.201.215:4352 , piece_id=I6BRYI5XIACAAAAAE7HQOBCF4OTALMOVPPNBWQCRSH2MM6WUUURQ, seg_pos={0 0}, upload=49960]: Done uploading piece. err: <nil> Aug 1 21:56:46.386119275 (51h59m36.323777774s) [noise=true , address=79.127.201.214:4297 , piece_id=MPTMGVEYC4SATIN4ANCR2ELVRXHYWRDLMHWQW7HM6UOIV2N6R7ZA, seg_pos={0 0}, upload=49960]: Done uploading piece. err: <nil> Aug 1 21:56:46.395362048 (51h59m36.333020547s) [noise=true , address=79.127.219.51:4213 , piece_id=L2WSPI5BEIFLGQKJIK4EWXFQXNQRQ7HVKPOWFYRPPUGNAL5DVYWA, seg_pos={0 0}, upload=49960]: Done uploading piece. err: <nil> Aug 1 21:57:43.392434383 (52h0m33.330092872s) [noise=true , address=104.36.82.37:4680 , piece_id=X4PDUE4WXUUWX6JQVO4NWPQCBMHMHC5YHPIHDQVYYOI2FXV2GVJQ, seg_pos={0 0}, upload=49960]: Done uploading piece. err: ecclient: upload failed (node:1bXAqpxGhUvzwYEDGBqYRhZdTeMJcV3TYfngkKTCumpt9Nupi3, address:104.36.82.37:4680): protocol: rpc: dial tcp 104.36.82.37:4680: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:43.392477967 (52h0m33.330136466s) [noise=true , address=104.36.82.11:4134 , piece_id=RMTNNNC2VD7MGWJDAMT6ZCF3GR43SHANFVAGJ6EHVH7WO5XA6VTA, seg_pos={0 0}, upload=49960]: Done uploading piece. err: ecclient: upload failed (node:12R3Z5AgkBmWXMv5XgRyfcyqkG2gmyJs9ogT24kcLy6nuB9bzUG, address:104.36.82.11:4134): protocol: rpc: dial tcp 104.36.82.11:4134: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:43.392514086 (52h0m33.330172585s) [noise=true , address=104.36.82.55:4204 , piece_id=R5CYXWAVV4ZEMCV7LF4D4XDC4G6KLVBONAH2TH5EIEVL7QHUTRWQ, seg_pos={0 0}, upload=49960]: Done uploading piece. err: ecclient: upload failed (node:1rBZH6aLHsnWh22uNzaMiMwmLJCDMn4bYbNC5jRG7KCEuRfqpH, address:104.36.82.55:4204): protocol: write tcp 188.166.225.170:48550->104.36.82.55:4204: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:43.392546537 (52h0m33.330205027s) [noise=true , address=104.36.82.59:4327 , piece_id=OZDEYIY6B4WYZULRDRL2UBOJH2JQNY3YWSITZKWDO7GTT72AN5NQ, seg_pos={0 0}, upload=49960]: Done uploading piece. err: ecclient: upload failed (node:1V4C6AWGbyYQWfinoSj7iKovHeTJ9hqypyC5U9rZ8pveXBBfCY, address:104.36.82.59:4327): protocol: rpc: dial tcp 104.36.82.59:4327: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:43.392579040 (52h0m33.33023754s) [noise=true , address=104.36.82.5:4842 , piece_id=LV7YAOMPE2AV7TF2WPWR7QJNLT4JZAIPSVZN3E23U5KDBNKZFD6A, seg_pos={0 0}, upload=49960]: Done uploading piece. err: ecclient: upload failed (node:12q4VdT5BcjUb16PBYHK6zrnW8dqENKrcGx9zdLtNefDVB1MUSn, address:104.36.82.5:4842): protocol: context deadline exceeded; context deadline exceeded Aug 1 21:57:43.392627953 (52h0m33.330286442s) [noise=true , address=104.36.82.52:4101 , piece_id=XYPLQM7QITCCX7T4NKIXP5MZ7I6F24EJJTG37K4NPOBKTUFHULRQ, seg_pos={0 0}, upload=49960]: Done uploading piece. err: ecclient: upload failed (node:12WWXeBjFZR9PzciNfs35fFcx87N9GwDpMsPHnksFDugtaBGWp4, address:104.36.82.52:4101): protocol: rpc: dial tcp 104.36.82.52:4101: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:43.392653192 (52h0m33.330311691s) [noise=true , address=104.36.82.14:4251 , piece_id=O7NADU347E2Q26TPD3IUCK6WE5UVRITC6ALZHBE5SWYTKWXFBMXA, seg_pos={0 0}, upload=49960]: Done uploading piece. err: ecclient: upload failed (node:12btf6Z38aTMx7MaBxhUyZtFqUPgeMtP2aU8Y6FbV8zwu57zArd, address:104.36.82.14:4251): protocol: rpc: dial tcp 104.36.82.14:4251: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:43.392670836 (52h0m33.330329335s) [noise=true , address=104.36.82.12:4166 , piece_id=6ZIWCOBY76GJ6DHZF5G23U4IYTOFB5SMQW5TW3YITYH227PV2FGA, seg_pos={0 0}, upload=49960]: Done uploading piece. err: ecclient: upload failed (node:12iCB7c4ZfpqafpD1PtpbE6BfU81wJW65LmrPVJXtD58yeuoR9t, address:104.36.82.12:4166): protocol: rpc: dial tcp 104.36.82.12:4166: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:43.392698018 (52h0m33.330356517s) [noise=true , address=104.36.82.50:4027 , piece_id=OW7BODU5N7ZEEGE2FHSLNYVJT2Z25LY3J42TZEOR5SP2EWIN6X7A, seg_pos={0 0}, upload=49960]: Done uploading piece. err: ecclient: upload failed (node:1zEQhyuVxSHwdDUgDfcTog2W3oLBx4z9bYUMHpaaoLU1BTU9DQ, address:104.36.82.50:4027): protocol: write tcp 188.166.225.170:37784->104.36.82.50:4027: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:43.392715381 (52h0m33.33037387s) [noise=true , address=104.36.82.43:4856 , piece_id=2SINBLHGRUOWUO4XHBNBG45YE3PW7RNF3QSNVCQXNCFEFHRBFNJQ, seg_pos={0 0}, upload=49960]: Done uploading piece. err: ecclient: upload failed (node:12bQxwBfPEGtmMp6mtnWKsoZt3eMEjLdyYFahETWGHeKJ7BhkiZ, address:104.36.82.43:4856): protocol: rpc: dial tcp 104.36.82.43:4856: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:43.392730059 (52h0m33.330388558s) [noise=true , address=104.36.82.10:4082 , piece_id=45K4BW277ZSGXJYQDA66OMEBVDC7DAWSOJAPX42HANNGRZN35KHQ, seg_pos={0 0}, upload=49960]: Done uploading piece. err: ecclient: upload failed (node:12DKsNCtqxLvpJwLszdLs74V3eb8NH2VAV9V3bngzUczawmNoLK, address:104.36.82.10:4082): protocol: write tcp 188.166.225.170:46638->104.36.82.10:4082: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:43.392753424 (52h0m33.330411923s) [seg_pos={0 0}, upload=49960]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil> Aug 1 21:57:43.624550997 (52h0m33.562209566s) [upload=49960]: Done uploading segments... Aug 1 21:57:43.624582868 (52h0m33.562241407s) [upload=49960]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7) Duration: 5.929707501s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [...............................XXX..........................] duration: 5.855348682s FAIL (2025-08-01 22:15:17 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.855398888s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 6.128621509s FAIL (2025-08-01 22:15:13 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.128643952s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 5 out of 60 is failed (threshold=3) [................X............X.....X.......X.....X..........] duration: 2.475539471s FAIL (2025-08-01 22:15:05 UTC) orbiter warning: Operation: ListObjects unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/1K Duration: 2.47556046s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 7 out of 60 is failed (threshold=3) [.............................XXXXXXX........................] duration: 6.119195805s FAIL (2025-08-01 22:15:04 UTC) orbiter warning: Operation: DownloadObject uplink: object not found ("orbiter_us_0/4M") Duration: 6.119224088s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [...............................XXXX.........................] duration: 5.907750371s FAIL (2025-08-01 22:15:00 UTC) orbiter warning: Operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 7) ======LOGS====== K4BW277ZSGXJYQDA66OMEBVDC7DAWSOJAPX42HANNGRZN35KHQ, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.481101510 (51h59m34.418760009s) [noise=true , address=104.36.82.5:4842 , piece_id=LV7YAOMPE2AV7TF2WPWR7QJNLT4JZAIPSVZN3E23U5KDBNKZFD6A, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.481422003 (51h59m34.419080502s) [noise=true , address=79.127.205.235:4683 , piece_id=3QHDNAPG4GRLJFVVNJNFXNGKTGKUEMWWUBMW62HHE5N5LJDO4KWA, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.482030897 (51h59m34.419689396s) [noise=true , address=79.127.219.51:4213 , piece_id=L2WSPI5BEIFLGQKJIK4EWXFQXNQRQ7HVKPOWFYRPPUGNAL5DVYWA, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.482285215 (51h59m34.419943704s) [noise=true , address=104.36.82.14:4251 , piece_id=O7NADU347E2Q26TPD3IUCK6WE5UVRITC6ALZHBE5SWYTKWXFBMXA, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.482518370 (51h59m34.420176859s) [noise=true , address=79.127.220.98:4402 , piece_id=DVLSYRFFZ6UT3QRSVAN7AMXVYZNI6PWG2IITHLRUUJATJVK3QOGA, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.482724304 (51h59m34.420382803s) [noise=true , address=79.127.201.214:4297 , piece_id=MPTMGVEYC4SATIN4ANCR2ELVRXHYWRDLMHWQW7HM6UOIV2N6R7ZA, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.482928555 (51h59m34.420587054s) [noise=true , address=104.36.82.37:4680 , piece_id=X4PDUE4WXUUWX6JQVO4NWPQCBMHMHC5YHPIHDQVYYOI2FXV2GVJQ, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.483134960 (51h59m34.420793449s) [noise=true , address=104.36.82.50:4027 , piece_id=OW7BODU5N7ZEEGE2FHSLNYVJT2Z25LY3J42TZEOR5SP2EWIN6X7A, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.483338480 (51h59m34.420996979s) [noise=true , address=104.36.82.55:4204 , piece_id=R5CYXWAVV4ZEMCV7LF4D4XDC4G6KLVBONAH2TH5EIEVL7QHUTRWQ, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.483582847 (51h59m34.421241356s) [noise=true , address=104.36.82.11:4134 , piece_id=RMTNNNC2VD7MGWJDAMT6ZCF3GR43SHANFVAGJ6EHVH7WO5XA6VTA, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.483833737 (51h59m34.421492236s) [noise=true , address=104.36.82.43:4856 , piece_id=2SINBLHGRUOWUO4XHBNBG45YE3PW7RNF3QSNVCQXNCFEFHRBFNJQ, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.484021896 (51h59m34.421680385s) [noise=true , address=104.36.82.52:4101 , piece_id=XYPLQM7QITCCX7T4NKIXP5MZ7I6F24EJJTG37K4NPOBKTUFHULRQ, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.484301662 (51h59m34.421960182s) [noise=true , address=79.127.201.215:4352 , piece_id=I6BRYI5XIACAAAAAE7HQOBCF4OTALMOVPPNBWQCRSH2MM6WUUURQ, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.484712097 (51h59m34.422370596s) [noise=true , address=64.49.13.138:4484 , piece_id=DZ5Y4HTS6BKW2DJ5CVOB64TW4SXB6DNCJEGFUJPLAUXUEHRZRTUA, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:45.576021470 (51h59m35.513679959s) [upload=49960]: Next returned nil segment Aug 1 21:56:45.576036609 (51h59m35.513695098s) [upload=49960]: Waiting for error group managing segments... Aug 1 21:56:45.860378776 (51h59m35.798037275s) [noise=true , address=64.49.13.138:4484 , piece_id=DZ5Y4HTS6BKW2DJ5CVOB64TW4SXB6DNCJEGFUJPLAUXUEHRZRTUA, seg_pos={0 0}, upload=49960]: Done uploading piece. err: <nil> Aug 1 21:56:45.884184188 (51h59m35.821842687s) [noise=true , address=79.127.205.235:4683 , piece_id=3QHDNAPG4GRLJFVVNJNFXNGKTGKUEMWWUBMW62HHE5N5LJDO4KWA, seg_pos={0 0}, upload=49960]: Done uploading piece. err: <nil> Aug 1 21:56:46.145493612 (51h59m36.083152101s) [noise=true , address=79.127.220.98:4402 , piece_id=DVLSYRFFZ6UT3QRSVAN7AMXVYZNI6PWG2IITHLRUUJATJVK3QOGA, seg_pos={0 0}, upload=49960]: Done uploading piece. err: <nil> Aug 1 21:56:46.257505313 (51h59m36.195163802s) [noise=true , address=79.127.220.97:4371 , piece_id=5WGPRFKQETIHTSMA42AM2C3UGCFYJ3DMMPAI2L7DD6E2P37KJKNA, seg_pos={0 0}, upload=49960]: Done uploading piece. err: <nil> Aug 1 21:56:46.334426181 (51h59m36.27208468s) [noise=true , address=79.127.201.215:4352 , piece_id=I6BRYI5XIACAAAAAE7HQOBCF4OTALMOVPPNBWQCRSH2MM6WUUURQ, seg_pos={0 0}, upload=49960]: Done uploading piece. err: <nil> Aug 1 21:56:46.386119275 (51h59m36.323777774s) [noise=true , address=79.127.201.214:4297 , piece_id=MPTMGVEYC4SATIN4ANCR2ELVRXHYWRDLMHWQW7HM6UOIV2N6R7ZA, seg_pos={0 0}, upload=49960]: Done uploading piece. err: <nil> Aug 1 21:56:46.395362048 (51h59m36.333020547s) [noise=true , address=79.127.219.51:4213 , piece_id=L2WSPI5BEIFLGQKJIK4EWXFQXNQRQ7HVKPOWFYRPPUGNAL5DVYWA, seg_pos={0 0}, upload=49960]: Done uploading piece. err: <nil> Aug 1 21:57:43.392434383 (52h0m33.330092872s) [noise=true , address=104.36.82.37:4680 , piece_id=X4PDUE4WXUUWX6JQVO4NWPQCBMHMHC5YHPIHDQVYYOI2FXV2GVJQ, seg_pos={0 0}, upload=49960]: Done uploading piece. err: ecclient: upload failed (node:1bXAqpxGhUvzwYEDGBqYRhZdTeMJcV3TYfngkKTCumpt9Nupi3, address:104.36.82.37:4680): protocol: rpc: dial tcp 104.36.82.37:4680: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:43.392477967 (52h0m33.330136466s) [noise=true , address=104.36.82.11:4134 , piece_id=RMTNNNC2VD7MGWJDAMT6ZCF3GR43SHANFVAGJ6EHVH7WO5XA6VTA, seg_pos={0 0}, upload=49960]: Done uploading piece. err: ecclient: upload failed (node:12R3Z5AgkBmWXMv5XgRyfcyqkG2gmyJs9ogT24kcLy6nuB9bzUG, address:104.36.82.11:4134): protocol: rpc: dial tcp 104.36.82.11:4134: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:43.392514086 (52h0m33.330172585s) [noise=true , address=104.36.82.55:4204 , piece_id=R5CYXWAVV4ZEMCV7LF4D4XDC4G6KLVBONAH2TH5EIEVL7QHUTRWQ, seg_pos={0 0}, upload=49960]: Done uploading piece. err: ecclient: upload failed (node:1rBZH6aLHsnWh22uNzaMiMwmLJCDMn4bYbNC5jRG7KCEuRfqpH, address:104.36.82.55:4204): protocol: write tcp 188.166.225.170:48550->104.36.82.55:4204: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:43.392546537 (52h0m33.330205027s) [noise=true , address=104.36.82.59:4327 , piece_id=OZDEYIY6B4WYZULRDRL2UBOJH2JQNY3YWSITZKWDO7GTT72AN5NQ, seg_pos={0 0}, upload=49960]: Done uploading piece. err: ecclient: upload failed (node:1V4C6AWGbyYQWfinoSj7iKovHeTJ9hqypyC5U9rZ8pveXBBfCY, address:104.36.82.59:4327): protocol: rpc: dial tcp 104.36.82.59:4327: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:43.392579040 (52h0m33.33023754s) [noise=true , address=104.36.82.5:4842 , piece_id=LV7YAOMPE2AV7TF2WPWR7QJNLT4JZAIPSVZN3E23U5KDBNKZFD6A, seg_pos={0 0}, upload=49960]: Done uploading piece. err: ecclient: upload failed (node:12q4VdT5BcjUb16PBYHK6zrnW8dqENKrcGx9zdLtNefDVB1MUSn, address:104.36.82.5:4842): protocol: context deadline exceeded; context deadline exceeded Aug 1 21:57:43.392627953 (52h0m33.330286442s) [noise=true , address=104.36.82.52:4101 , piece_id=XYPLQM7QITCCX7T4NKIXP5MZ7I6F24EJJTG37K4NPOBKTUFHULRQ, seg_pos={0 0}, upload=49960]: Done uploading piece. err: ecclient: upload failed (node:12WWXeBjFZR9PzciNfs35fFcx87N9GwDpMsPHnksFDugtaBGWp4, address:104.36.82.52:4101): protocol: rpc: dial tcp 104.36.82.52:4101: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:43.392653192 (52h0m33.330311691s) [noise=true , address=104.36.82.14:4251 , piece_id=O7NADU347E2Q26TPD3IUCK6WE5UVRITC6ALZHBE5SWYTKWXFBMXA, seg_pos={0 0}, upload=49960]: Done uploading piece. err: ecclient: upload failed (node:12btf6Z38aTMx7MaBxhUyZtFqUPgeMtP2aU8Y6FbV8zwu57zArd, address:104.36.82.14:4251): protocol: rpc: dial tcp 104.36.82.14:4251: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:43.392670836 (52h0m33.330329335s) [noise=true , address=104.36.82.12:4166 , piece_id=6ZIWCOBY76GJ6DHZF5G23U4IYTOFB5SMQW5TW3YITYH227PV2FGA, seg_pos={0 0}, upload=49960]: Done uploading piece. err: ecclient: upload failed (node:12iCB7c4ZfpqafpD1PtpbE6BfU81wJW65LmrPVJXtD58yeuoR9t, address:104.36.82.12:4166): protocol: rpc: dial tcp 104.36.82.12:4166: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:43.392698018 (52h0m33.330356517s) [noise=true , address=104.36.82.50:4027 , piece_id=OW7BODU5N7ZEEGE2FHSLNYVJT2Z25LY3J42TZEOR5SP2EWIN6X7A, seg_pos={0 0}, upload=49960]: Done uploading piece. err: ecclient: upload failed (node:1zEQhyuVxSHwdDUgDfcTog2W3oLBx4z9bYUMHpaaoLU1BTU9DQ, address:104.36.82.50:4027): protocol: write tcp 188.166.225.170:37784->104.36.82.50:4027: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:43.392715381 (52h0m33.33037387s) [noise=true , address=104.36.82.43:4856 , piece_id=2SINBLHGRUOWUO4XHBNBG45YE3PW7RNF3QSNVCQXNCFEFHRBFNJQ, seg_pos={0 0}, upload=49960]: Done uploading piece. err: ecclient: upload failed (node:12bQxwBfPEGtmMp6mtnWKsoZt3eMEjLdyYFahETWGHeKJ7BhkiZ, address:104.36.82.43:4856): protocol: rpc: dial tcp 104.36.82.43:4856: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:43.392730059 (52h0m33.330388558s) [noise=true , address=104.36.82.10:4082 , piece_id=45K4BW277ZSGXJYQDA66OMEBVDC7DAWSOJAPX42HANNGRZN35KHQ, seg_pos={0 0}, upload=49960]: Done uploading piece. err: ecclient: upload failed (node:12DKsNCtqxLvpJwLszdLs74V3eb8NH2VAV9V3bngzUczawmNoLK, address:104.36.82.10:4082): protocol: write tcp 188.166.225.170:46638->104.36.82.10:4082: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:43.392753424 (52h0m33.330411923s) [seg_pos={0 0}, upload=49960]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil> Aug 1 21:57:43.624550997 (52h0m33.562209566s) [upload=49960]: Done uploading segments... Aug 1 21:57:43.624582868 (52h0m33.562241407s) [upload=49960]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7) Duration: 5.907777413s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 11.585423479s FAIL (2025-08-01 22:14:58 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 11.585441874s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [................................XXX.........................] duration: 5.675273067s FAIL (2025-08-01 22:14:52 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.675297303s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [...............................XX.X.X.......................] duration: 5.668050508s FAIL (2025-08-01 22:14:45 UTC) orbiter warning: Operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 7NK72474VWE4E3YOBCDZ5O5ZOH2MSA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB): protocol: write tcp 188.166.225.170:38368->104.36.82.12:4151: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:56:50.418604753 (51h59m40.356263252s) [seg_pos={0 0}, upload=49961]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 1 21:56:50.418680879 (51h59m40.356339378s) [upload=49961]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 1 21:56:50.418685998 (51h59m40.356344487s) [upload=49961]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 1 21:56:50.726563872 (51h59m40.664222371s) [upload=49961]: Done uploading segments... Aug 1 21:56:50.726586586 (51h59m40.664245085s) [upload=49961]: Upload finished. err: <nil> Aug 1 21:57:28.852869669 (52h0m18.790528158s) [upload=49973]: Starting upload Aug 1 21:57:28.852942389 (52h0m18.790600888s) [upload=49973]: Done starting upload Aug 1 21:57:28.853007122 (52h0m18.790665611s) [upload=49973]: Uploading segments... Aug 1 21:57:28.853174002 (52h0m18.790832491s) [upload=49973]: Got next segment. Inline: false Aug 1 21:57:28.853208168 (52h0m18.790866657s) [upload=49973]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 1 21:57:28.853212666 (52h0m18.790871165s) [upload=49973]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 1 21:57:29.095087519 (52h0m19.032746008s) [seg_pos={0 0}, upload=49973]: Begin upload segment... Aug 1 21:57:29.095129960 (52h0m19.032788459s) [seg_pos={0 0}, upload=49973]: Done begin upload segment. Aug 1 21:57:29.095164997 (52h0m19.032823496s) [noise=true , address=77.90.31.21:4878 , piece_id=BLNTYBOAIZWISSUJYK6BSBPTSNL6QWLGEN6AP6P4NPDVSURYG5AA, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.095171800 (52h0m19.032830289s) [noise=true , address=104.36.82.6:4875 , piece_id=CS5AMB7CIBQWWOU57Z3RCCSNVKIPE4B5QWU5YIQYI2UCWH7PAPVA, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.095510027 (52h0m19.033168526s) [noise=true , address=77.90.31.18:4806 , piece_id=52B6CAHGLFVTFIDQHHHAWKCBP2WJJY7OTTGRZZ7CEUH3HDA43Q6A, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.095544193 (52h0m19.033202692s) [noise=true , address=77.90.31.10:4518 , piece_id=WUOVXUSDG6YQTDPWG4YO7VLJNZZVLH6VRGMWOJABG2EJRU7XHP5A, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.095774283 (52h0m19.033432782s) [noise=true , address=104.36.82.7:4909 , piece_id=QHJCVBT6KTU34K2KWFNACWYVVE3WJ2RU76BHVSKL4HLCZEFXZYPA, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.095819880 (52h0m19.033478379s) [noise=true , address=45.91.116.12:4225 , piece_id=PBQPLA5ZYRFPSKHDKCEFOEBS5T3GYMP2FLSFDQWJLVY756KPLJNQ, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.096021656 (52h0m19.033680155s) [noise=true , address=45.91.116.13:4153 , piece_id=PMLKMZO66KXF3ZONO5ZZOQZICFK5XNGL6MN5FFBKICVFU5AWU6PQ, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.096069618 (52h0m19.033728117s) [noise=true , address=45.91.116.9:4262 , piece_id=A3F4L3RR6WPFMBFFKLATNGG3CASBIPWQP5O3G5URFNTXBE3ZW3VA, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.096348041 (52h0m19.03400654s) [noise=true , address=104.36.82.5:4840 , piece_id=UYVKVFCJJP7XZYRIVPS4ABGEMDUMNR7YW4QGG5ZFTFCGNCJO3CSQ, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.096354803 (52h0m19.034013292s) [noise=true , address=77.90.31.12:4590 , piece_id=D6FUBIFCQJSDRGWEP5GG34AQPNB3UROCYOATD6SRTJ7KAYVMFCOA, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.096608579 (52h0m19.034267078s) [noise=true , address=77.90.31.16:4733 , piece_id=DMASMGOUATLPEKCGLHKNNIMMFYVVGSULNUOMJU6RJY3ULBDOY6KQ, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.096830774 (52h0m19.034489273s) [noise=true , address=104.36.82.4:4802 , piece_id=CLM2G33M6AK5HIESRW6LZY33NKHIR2PGITWJ23NPILRCA6UYIEHA, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.097107233 (52h0m19.034765732s) [noise=true , address=77.90.31.19:4843 , piece_id=UFONZYHL7EAFYE5PRW32RKUVQPMZPXBEFLLIFJMAB2TXJ3C6KKNA, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.097324328 (52h0m19.034982827s) [noise=true , address=77.90.31.17:4771 , piece_id=Y3WSALJX23CWPPYQ3XYMCFOSBLBX2E2JQCCS7EOEYNBWY4CDRT3Q, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.098682446 (52h0m19.036340945s) [noise=true , address=77.90.31.15:4698 , piece_id=TN4OTPUD47FD6WOLZBEHNDI43WY6QVO5JK2JNJXNVM2IT6XMGERQ, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.099041603 (52h0m19.036700102s) [noise=true , address=45.91.116.11:4118 , piece_id=AKZIOAR6NRNO7JWCQZIKNSL7IUH5S4IG3KPG3IY2FAKG74DH3E3Q, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:30.718412255 (52h0m20.656070754s) [upload=49973]: Next returned nil segment Aug 1 21:57:30.718429869 (52h0m20.656088368s) [upload=49973]: Waiting for error group managing segments... Aug 1 21:57:31.107386675 (52h0m21.045045164s) [noise=true , address=45.91.116.13:4153 , piece_id=PMLKMZO66KXF3ZONO5ZZOQZICFK5XNGL6MN5FFBKICVFU5AWU6PQ, seg_pos={0 0}, upload=49973]: Done uploading piece. err: <nil> Aug 1 21:57:31.108529412 (52h0m21.046187911s) [noise=true , address=77.90.31.12:4590 , piece_id=D6FUBIFCQJSDRGWEP5GG34AQPNB3UROCYOATD6SRTJ7KAYVMFCOA, seg_pos={0 0}, upload=49973]: Done uploading piece. err: <nil> Aug 1 21:57:31.109975858 (52h0m21.047634347s) [noise=true , address=77.90.31.15:4698 , piece_id=TN4OTPUD47FD6WOLZBEHNDI43WY6QVO5JK2JNJXNVM2IT6XMGERQ, seg_pos={0 0}, upload=49973]: Done uploading piece. err: <nil> Aug 1 21:57:31.115273013 (52h0m21.052931502s) [noise=true , address=77.90.31.10:4518 , piece_id=WUOVXUSDG6YQTDPWG4YO7VLJNZZVLH6VRGMWOJABG2EJRU7XHP5A, seg_pos={0 0}, upload=49973]: Done uploading piece. err: <nil> Aug 1 21:57:31.116767281 (52h0m21.054425781s) [noise=true , address=77.90.31.18:4806 , piece_id=52B6CAHGLFVTFIDQHHHAWKCBP2WJJY7OTTGRZZ7CEUH3HDA43Q6A, seg_pos={0 0}, upload=49973]: Done uploading piece. err: <nil> Aug 1 21:57:31.201146107 (52h0m21.138804606s) [noise=true , address=77.90.31.16:4733 , piece_id=DMASMGOUATLPEKCGLHKNNIMMFYVVGSULNUOMJU6RJY3ULBDOY6KQ, seg_pos={0 0}, upload=49973]: Done uploading piece. err: <nil> Aug 1 21:57:31.201833732 (52h0m21.139492231s) [noise=true , address=77.90.31.19:4843 , piece_id=UFONZYHL7EAFYE5PRW32RKUVQPMZPXBEFLLIFJMAB2TXJ3C6KKNA, seg_pos={0 0}, upload=49973]: Done uploading piece. err: <nil> Aug 1 21:57:31.211007613 (52h0m21.148666112s) [noise=true , address=45.91.116.9:4262 , piece_id=A3F4L3RR6WPFMBFFKLATNGG3CASBIPWQP5O3G5URFNTXBE3ZW3VA, seg_pos={0 0}, upload=49973]: Done uploading piece. err: <nil> Aug 1 21:57:31.215929169 (52h0m21.153587668s) [noise=true , address=77.90.31.17:4771 , piece_id=Y3WSALJX23CWPPYQ3XYMCFOSBLBX2E2JQCCS7EOEYNBWY4CDRT3Q, seg_pos={0 0}, upload=49973]: Done uploading piece. err: <nil> Aug 1 21:57:31.222973975 (52h0m21.160632474s) [noise=true , address=77.90.31.21:4878 , piece_id=BLNTYBOAIZWISSUJYK6BSBPTSNL6QWLGEN6AP6P4NPDVSURYG5AA, seg_pos={0 0}, upload=49973]: Done uploading piece. err: <nil> Aug 1 21:57:31.234392441 (52h0m21.17205094s) [noise=true , address=45.91.116.12:4225 , piece_id=PBQPLA5ZYRFPSKHDKCEFOEBS5T3GYMP2FLSFDQWJLVY756KPLJNQ, seg_pos={0 0}, upload=49973]: Done uploading piece. err: <nil> Aug 1 21:57:31.284794305 (52h0m21.222452804s) [noise=true , address=45.91.116.11:4118 , piece_id=AKZIOAR6NRNO7JWCQZIKNSL7IUH5S4IG3KPG3IY2FAKG74DH3E3Q, seg_pos={0 0}, upload=49973]: Done uploading piece. err: <nil> Aug 1 21:57:58.202420932 (52h0m48.140079471s) [noise=true , address=104.36.82.6:4875 , piece_id=CS5AMB7CIBQWWOU57Z3RCCSNVKIPE4B5QWU5YIQYI2UCWH7PAPVA, seg_pos={0 0}, upload=49973]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:104.36.82.6:4875): protocol: write tcp 188.166.225.170:40166->104.36.82.6:4875: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:58.202441131 (52h0m48.14009963s) [noise=true , address=104.36.82.5:4840 , piece_id=UYVKVFCJJP7XZYRIVPS4ABGEMDUMNR7YW4QGG5ZFTFCGNCJO3CSQ, seg_pos={0 0}, upload=49973]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:104.36.82.5:4840): protocol: rpc: dial tcp 104.36.82.5:4840: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:58.202448445 (52h0m48.140106954s) [noise=true , address=104.36.82.7:4909 , piece_id=QHJCVBT6KTU34K2KWFNACWYVVE3WJ2RU76BHVSKL4HLCZEFXZYPA, seg_pos={0 0}, upload=49973]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:104.36.82.7:4909): protocol: rpc: dial tcp 104.36.82.7:4909: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:58.202496487 (52h0m48.140155026s) [noise=true , address=104.36.82.4:4802 , piece_id=CLM2G33M6AK5HIESRW6LZY33NKHIR2PGITWJ23NPILRCA6UYIEHA, seg_pos={0 0}, upload=49973]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: write tcp 188.166.225.170:47696->104.36.82.4:4802: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:58.202524240 (52h0m48.140182739s) [seg_pos={0 0}, upload=49973]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Aug 1 21:57:58.435983932 (52h0m48.373642431s) [upload=49973]: Done uploading segments... Aug 1 21:57:58.436007667 (52h0m48.373666166s) [upload=49973]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.668073193s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 7 out of 60 is failed (threshold=3) [..............................XXXXXXX.......................] duration: 5.86792317s FAIL (2025-08-01 22:14:40 UTC) orbiter warning: Operation: DownloadObject uplink: object not found ("orbiter_us_0/4M") Duration: 5.867949269s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [................................XX.X.X......................] duration: 5.985285798s FAIL (2025-08-01 22:14:28 UTC) orbiter warning: Operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 7NK72474VWE4E3YOBCDZ5O5ZOH2MSA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB): protocol: write tcp 188.166.225.170:38368->104.36.82.12:4151: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:56:50.418604753 (51h59m40.356263252s) [seg_pos={0 0}, upload=49961]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 1 21:56:50.418680879 (51h59m40.356339378s) [upload=49961]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 1 21:56:50.418685998 (51h59m40.356344487s) [upload=49961]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 1 21:56:50.726563872 (51h59m40.664222371s) [upload=49961]: Done uploading segments... Aug 1 21:56:50.726586586 (51h59m40.664245085s) [upload=49961]: Upload finished. err: <nil> Aug 1 21:57:28.852869669 (52h0m18.790528158s) [upload=49973]: Starting upload Aug 1 21:57:28.852942389 (52h0m18.790600888s) [upload=49973]: Done starting upload Aug 1 21:57:28.853007122 (52h0m18.790665611s) [upload=49973]: Uploading segments... Aug 1 21:57:28.853174002 (52h0m18.790832491s) [upload=49973]: Got next segment. Inline: false Aug 1 21:57:28.853208168 (52h0m18.790866657s) [upload=49973]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 1 21:57:28.853212666 (52h0m18.790871165s) [upload=49973]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 1 21:57:29.095087519 (52h0m19.032746008s) [seg_pos={0 0}, upload=49973]: Begin upload segment... Aug 1 21:57:29.095129960 (52h0m19.032788459s) [seg_pos={0 0}, upload=49973]: Done begin upload segment. Aug 1 21:57:29.095164997 (52h0m19.032823496s) [noise=true , address=77.90.31.21:4878 , piece_id=BLNTYBOAIZWISSUJYK6BSBPTSNL6QWLGEN6AP6P4NPDVSURYG5AA, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.095171800 (52h0m19.032830289s) [noise=true , address=104.36.82.6:4875 , piece_id=CS5AMB7CIBQWWOU57Z3RCCSNVKIPE4B5QWU5YIQYI2UCWH7PAPVA, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.095510027 (52h0m19.033168526s) [noise=true , address=77.90.31.18:4806 , piece_id=52B6CAHGLFVTFIDQHHHAWKCBP2WJJY7OTTGRZZ7CEUH3HDA43Q6A, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.095544193 (52h0m19.033202692s) [noise=true , address=77.90.31.10:4518 , piece_id=WUOVXUSDG6YQTDPWG4YO7VLJNZZVLH6VRGMWOJABG2EJRU7XHP5A, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.095774283 (52h0m19.033432782s) [noise=true , address=104.36.82.7:4909 , piece_id=QHJCVBT6KTU34K2KWFNACWYVVE3WJ2RU76BHVSKL4HLCZEFXZYPA, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.095819880 (52h0m19.033478379s) [noise=true , address=45.91.116.12:4225 , piece_id=PBQPLA5ZYRFPSKHDKCEFOEBS5T3GYMP2FLSFDQWJLVY756KPLJNQ, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.096021656 (52h0m19.033680155s) [noise=true , address=45.91.116.13:4153 , piece_id=PMLKMZO66KXF3ZONO5ZZOQZICFK5XNGL6MN5FFBKICVFU5AWU6PQ, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.096069618 (52h0m19.033728117s) [noise=true , address=45.91.116.9:4262 , piece_id=A3F4L3RR6WPFMBFFKLATNGG3CASBIPWQP5O3G5URFNTXBE3ZW3VA, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.096348041 (52h0m19.03400654s) [noise=true , address=104.36.82.5:4840 , piece_id=UYVKVFCJJP7XZYRIVPS4ABGEMDUMNR7YW4QGG5ZFTFCGNCJO3CSQ, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.096354803 (52h0m19.034013292s) [noise=true , address=77.90.31.12:4590 , piece_id=D6FUBIFCQJSDRGWEP5GG34AQPNB3UROCYOATD6SRTJ7KAYVMFCOA, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.096608579 (52h0m19.034267078s) [noise=true , address=77.90.31.16:4733 , piece_id=DMASMGOUATLPEKCGLHKNNIMMFYVVGSULNUOMJU6RJY3ULBDOY6KQ, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.096830774 (52h0m19.034489273s) [noise=true , address=104.36.82.4:4802 , piece_id=CLM2G33M6AK5HIESRW6LZY33NKHIR2PGITWJ23NPILRCA6UYIEHA, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.097107233 (52h0m19.034765732s) [noise=true , address=77.90.31.19:4843 , piece_id=UFONZYHL7EAFYE5PRW32RKUVQPMZPXBEFLLIFJMAB2TXJ3C6KKNA, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.097324328 (52h0m19.034982827s) [noise=true , address=77.90.31.17:4771 , piece_id=Y3WSALJX23CWPPYQ3XYMCFOSBLBX2E2JQCCS7EOEYNBWY4CDRT3Q, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.098682446 (52h0m19.036340945s) [noise=true , address=77.90.31.15:4698 , piece_id=TN4OTPUD47FD6WOLZBEHNDI43WY6QVO5JK2JNJXNVM2IT6XMGERQ, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.099041603 (52h0m19.036700102s) [noise=true , address=45.91.116.11:4118 , piece_id=AKZIOAR6NRNO7JWCQZIKNSL7IUH5S4IG3KPG3IY2FAKG74DH3E3Q, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:30.718412255 (52h0m20.656070754s) [upload=49973]: Next returned nil segment Aug 1 21:57:30.718429869 (52h0m20.656088368s) [upload=49973]: Waiting for error group managing segments... Aug 1 21:57:31.107386675 (52h0m21.045045164s) [noise=true , address=45.91.116.13:4153 , piece_id=PMLKMZO66KXF3ZONO5ZZOQZICFK5XNGL6MN5FFBKICVFU5AWU6PQ, seg_pos={0 0}, upload=49973]: Done uploading piece. err: <nil> Aug 1 21:57:31.108529412 (52h0m21.046187911s) [noise=true , address=77.90.31.12:4590 , piece_id=D6FUBIFCQJSDRGWEP5GG34AQPNB3UROCYOATD6SRTJ7KAYVMFCOA, seg_pos={0 0}, upload=49973]: Done uploading piece. err: <nil> Aug 1 21:57:31.109975858 (52h0m21.047634347s) [noise=true , address=77.90.31.15:4698 , piece_id=TN4OTPUD47FD6WOLZBEHNDI43WY6QVO5JK2JNJXNVM2IT6XMGERQ, seg_pos={0 0}, upload=49973]: Done uploading piece. err: <nil> Aug 1 21:57:31.115273013 (52h0m21.052931502s) [noise=true , address=77.90.31.10:4518 , piece_id=WUOVXUSDG6YQTDPWG4YO7VLJNZZVLH6VRGMWOJABG2EJRU7XHP5A, seg_pos={0 0}, upload=49973]: Done uploading piece. err: <nil> Aug 1 21:57:31.116767281 (52h0m21.054425781s) [noise=true , address=77.90.31.18:4806 , piece_id=52B6CAHGLFVTFIDQHHHAWKCBP2WJJY7OTTGRZZ7CEUH3HDA43Q6A, seg_pos={0 0}, upload=49973]: Done uploading piece. err: <nil> Aug 1 21:57:31.201146107 (52h0m21.138804606s) [noise=true , address=77.90.31.16:4733 , piece_id=DMASMGOUATLPEKCGLHKNNIMMFYVVGSULNUOMJU6RJY3ULBDOY6KQ, seg_pos={0 0}, upload=49973]: Done uploading piece. err: <nil> Aug 1 21:57:31.201833732 (52h0m21.139492231s) [noise=true , address=77.90.31.19:4843 , piece_id=UFONZYHL7EAFYE5PRW32RKUVQPMZPXBEFLLIFJMAB2TXJ3C6KKNA, seg_pos={0 0}, upload=49973]: Done uploading piece. err: <nil> Aug 1 21:57:31.211007613 (52h0m21.148666112s) [noise=true , address=45.91.116.9:4262 , piece_id=A3F4L3RR6WPFMBFFKLATNGG3CASBIPWQP5O3G5URFNTXBE3ZW3VA, seg_pos={0 0}, upload=49973]: Done uploading piece. err: <nil> Aug 1 21:57:31.215929169 (52h0m21.153587668s) [noise=true , address=77.90.31.17:4771 , piece_id=Y3WSALJX23CWPPYQ3XYMCFOSBLBX2E2JQCCS7EOEYNBWY4CDRT3Q, seg_pos={0 0}, upload=49973]: Done uploading piece. err: <nil> Aug 1 21:57:31.222973975 (52h0m21.160632474s) [noise=true , address=77.90.31.21:4878 , piece_id=BLNTYBOAIZWISSUJYK6BSBPTSNL6QWLGEN6AP6P4NPDVSURYG5AA, seg_pos={0 0}, upload=49973]: Done uploading piece. err: <nil> Aug 1 21:57:31.234392441 (52h0m21.17205094s) [noise=true , address=45.91.116.12:4225 , piece_id=PBQPLA5ZYRFPSKHDKCEFOEBS5T3GYMP2FLSFDQWJLVY756KPLJNQ, seg_pos={0 0}, upload=49973]: Done uploading piece. err: <nil> Aug 1 21:57:31.284794305 (52h0m21.222452804s) [noise=true , address=45.91.116.11:4118 , piece_id=AKZIOAR6NRNO7JWCQZIKNSL7IUH5S4IG3KPG3IY2FAKG74DH3E3Q, seg_pos={0 0}, upload=49973]: Done uploading piece. err: <nil> Aug 1 21:57:58.202420932 (52h0m48.140079471s) [noise=true , address=104.36.82.6:4875 , piece_id=CS5AMB7CIBQWWOU57Z3RCCSNVKIPE4B5QWU5YIQYI2UCWH7PAPVA, seg_pos={0 0}, upload=49973]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:104.36.82.6:4875): protocol: write tcp 188.166.225.170:40166->104.36.82.6:4875: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:58.202441131 (52h0m48.14009963s) [noise=true , address=104.36.82.5:4840 , piece_id=UYVKVFCJJP7XZYRIVPS4ABGEMDUMNR7YW4QGG5ZFTFCGNCJO3CSQ, seg_pos={0 0}, upload=49973]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:104.36.82.5:4840): protocol: rpc: dial tcp 104.36.82.5:4840: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:58.202448445 (52h0m48.140106954s) [noise=true , address=104.36.82.7:4909 , piece_id=QHJCVBT6KTU34K2KWFNACWYVVE3WJ2RU76BHVSKL4HLCZEFXZYPA, seg_pos={0 0}, upload=49973]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:104.36.82.7:4909): protocol: rpc: dial tcp 104.36.82.7:4909: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:58.202496487 (52h0m48.140155026s) [noise=true , address=104.36.82.4:4802 , piece_id=CLM2G33M6AK5HIESRW6LZY33NKHIR2PGITWJ23NPILRCA6UYIEHA, seg_pos={0 0}, upload=49973]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: write tcp 188.166.225.170:47696->104.36.82.4:4802: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:58.202524240 (52h0m48.140182739s) [seg_pos={0 0}, upload=49973]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Aug 1 21:57:58.435983932 (52h0m48.373642431s) [upload=49973]: Done uploading segments... Aug 1 21:57:58.436007667 (52h0m48.373666166s) [upload=49973]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.985314293s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 5 out of 60 is failed (threshold=3) [.................X............X.....X.......X.....X.........] duration: 2.239731875s FAIL (2025-08-01 22:14:11 UTC) orbiter warning: Operation: ListObjects unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/1K Duration: 2.239759086s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [................................XXXX........................] duration: 5.89461603s FAIL (2025-08-01 22:14:09 UTC) orbiter warning: Operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 7) ======LOGS====== K4BW277ZSGXJYQDA66OMEBVDC7DAWSOJAPX42HANNGRZN35KHQ, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.481101510 (51h59m34.418760009s) [noise=true , address=104.36.82.5:4842 , piece_id=LV7YAOMPE2AV7TF2WPWR7QJNLT4JZAIPSVZN3E23U5KDBNKZFD6A, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.481422003 (51h59m34.419080502s) [noise=true , address=79.127.205.235:4683 , piece_id=3QHDNAPG4GRLJFVVNJNFXNGKTGKUEMWWUBMW62HHE5N5LJDO4KWA, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.482030897 (51h59m34.419689396s) [noise=true , address=79.127.219.51:4213 , piece_id=L2WSPI5BEIFLGQKJIK4EWXFQXNQRQ7HVKPOWFYRPPUGNAL5DVYWA, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.482285215 (51h59m34.419943704s) [noise=true , address=104.36.82.14:4251 , piece_id=O7NADU347E2Q26TPD3IUCK6WE5UVRITC6ALZHBE5SWYTKWXFBMXA, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.482518370 (51h59m34.420176859s) [noise=true , address=79.127.220.98:4402 , piece_id=DVLSYRFFZ6UT3QRSVAN7AMXVYZNI6PWG2IITHLRUUJATJVK3QOGA, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.482724304 (51h59m34.420382803s) [noise=true , address=79.127.201.214:4297 , piece_id=MPTMGVEYC4SATIN4ANCR2ELVRXHYWRDLMHWQW7HM6UOIV2N6R7ZA, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.482928555 (51h59m34.420587054s) [noise=true , address=104.36.82.37:4680 , piece_id=X4PDUE4WXUUWX6JQVO4NWPQCBMHMHC5YHPIHDQVYYOI2FXV2GVJQ, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.483134960 (51h59m34.420793449s) [noise=true , address=104.36.82.50:4027 , piece_id=OW7BODU5N7ZEEGE2FHSLNYVJT2Z25LY3J42TZEOR5SP2EWIN6X7A, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.483338480 (51h59m34.420996979s) [noise=true , address=104.36.82.55:4204 , piece_id=R5CYXWAVV4ZEMCV7LF4D4XDC4G6KLVBONAH2TH5EIEVL7QHUTRWQ, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.483582847 (51h59m34.421241356s) [noise=true , address=104.36.82.11:4134 , piece_id=RMTNNNC2VD7MGWJDAMT6ZCF3GR43SHANFVAGJ6EHVH7WO5XA6VTA, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.483833737 (51h59m34.421492236s) [noise=true , address=104.36.82.43:4856 , piece_id=2SINBLHGRUOWUO4XHBNBG45YE3PW7RNF3QSNVCQXNCFEFHRBFNJQ, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.484021896 (51h59m34.421680385s) [noise=true , address=104.36.82.52:4101 , piece_id=XYPLQM7QITCCX7T4NKIXP5MZ7I6F24EJJTG37K4NPOBKTUFHULRQ, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.484301662 (51h59m34.421960182s) [noise=true , address=79.127.201.215:4352 , piece_id=I6BRYI5XIACAAAAAE7HQOBCF4OTALMOVPPNBWQCRSH2MM6WUUURQ, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.484712097 (51h59m34.422370596s) [noise=true , address=64.49.13.138:4484 , piece_id=DZ5Y4HTS6BKW2DJ5CVOB64TW4SXB6DNCJEGFUJPLAUXUEHRZRTUA, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:45.576021470 (51h59m35.513679959s) [upload=49960]: Next returned nil segment Aug 1 21:56:45.576036609 (51h59m35.513695098s) [upload=49960]: Waiting for error group managing segments... Aug 1 21:56:45.860378776 (51h59m35.798037275s) [noise=true , address=64.49.13.138:4484 , piece_id=DZ5Y4HTS6BKW2DJ5CVOB64TW4SXB6DNCJEGFUJPLAUXUEHRZRTUA, seg_pos={0 0}, upload=49960]: Done uploading piece. err: <nil> Aug 1 21:56:45.884184188 (51h59m35.821842687s) [noise=true , address=79.127.205.235:4683 , piece_id=3QHDNAPG4GRLJFVVNJNFXNGKTGKUEMWWUBMW62HHE5N5LJDO4KWA, seg_pos={0 0}, upload=49960]: Done uploading piece. err: <nil> Aug 1 21:56:46.145493612 (51h59m36.083152101s) [noise=true , address=79.127.220.98:4402 , piece_id=DVLSYRFFZ6UT3QRSVAN7AMXVYZNI6PWG2IITHLRUUJATJVK3QOGA, seg_pos={0 0}, upload=49960]: Done uploading piece. err: <nil> Aug 1 21:56:46.257505313 (51h59m36.195163802s) [noise=true , address=79.127.220.97:4371 , piece_id=5WGPRFKQETIHTSMA42AM2C3UGCFYJ3DMMPAI2L7DD6E2P37KJKNA, seg_pos={0 0}, upload=49960]: Done uploading piece. err: <nil> Aug 1 21:56:46.334426181 (51h59m36.27208468s) [noise=true , address=79.127.201.215:4352 , piece_id=I6BRYI5XIACAAAAAE7HQOBCF4OTALMOVPPNBWQCRSH2MM6WUUURQ, seg_pos={0 0}, upload=49960]: Done uploading piece. err: <nil> Aug 1 21:56:46.386119275 (51h59m36.323777774s) [noise=true , address=79.127.201.214:4297 , piece_id=MPTMGVEYC4SATIN4ANCR2ELVRXHYWRDLMHWQW7HM6UOIV2N6R7ZA, seg_pos={0 0}, upload=49960]: Done uploading piece. err: <nil> Aug 1 21:56:46.395362048 (51h59m36.333020547s) [noise=true , address=79.127.219.51:4213 , piece_id=L2WSPI5BEIFLGQKJIK4EWXFQXNQRQ7HVKPOWFYRPPUGNAL5DVYWA, seg_pos={0 0}, upload=49960]: Done uploading piece. err: <nil> Aug 1 21:57:43.392434383 (52h0m33.330092872s) [noise=true , address=104.36.82.37:4680 , piece_id=X4PDUE4WXUUWX6JQVO4NWPQCBMHMHC5YHPIHDQVYYOI2FXV2GVJQ, seg_pos={0 0}, upload=49960]: Done uploading piece. err: ecclient: upload failed (node:1bXAqpxGhUvzwYEDGBqYRhZdTeMJcV3TYfngkKTCumpt9Nupi3, address:104.36.82.37:4680): protocol: rpc: dial tcp 104.36.82.37:4680: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:43.392477967 (52h0m33.330136466s) [noise=true , address=104.36.82.11:4134 , piece_id=RMTNNNC2VD7MGWJDAMT6ZCF3GR43SHANFVAGJ6EHVH7WO5XA6VTA, seg_pos={0 0}, upload=49960]: Done uploading piece. err: ecclient: upload failed (node:12R3Z5AgkBmWXMv5XgRyfcyqkG2gmyJs9ogT24kcLy6nuB9bzUG, address:104.36.82.11:4134): protocol: rpc: dial tcp 104.36.82.11:4134: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:43.392514086 (52h0m33.330172585s) [noise=true , address=104.36.82.55:4204 , piece_id=R5CYXWAVV4ZEMCV7LF4D4XDC4G6KLVBONAH2TH5EIEVL7QHUTRWQ, seg_pos={0 0}, upload=49960]: Done uploading piece. err: ecclient: upload failed (node:1rBZH6aLHsnWh22uNzaMiMwmLJCDMn4bYbNC5jRG7KCEuRfqpH, address:104.36.82.55:4204): protocol: write tcp 188.166.225.170:48550->104.36.82.55:4204: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:43.392546537 (52h0m33.330205027s) [noise=true , address=104.36.82.59:4327 , piece_id=OZDEYIY6B4WYZULRDRL2UBOJH2JQNY3YWSITZKWDO7GTT72AN5NQ, seg_pos={0 0}, upload=49960]: Done uploading piece. err: ecclient: upload failed (node:1V4C6AWGbyYQWfinoSj7iKovHeTJ9hqypyC5U9rZ8pveXBBfCY, address:104.36.82.59:4327): protocol: rpc: dial tcp 104.36.82.59:4327: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:43.392579040 (52h0m33.33023754s) [noise=true , address=104.36.82.5:4842 , piece_id=LV7YAOMPE2AV7TF2WPWR7QJNLT4JZAIPSVZN3E23U5KDBNKZFD6A, seg_pos={0 0}, upload=49960]: Done uploading piece. err: ecclient: upload failed (node:12q4VdT5BcjUb16PBYHK6zrnW8dqENKrcGx9zdLtNefDVB1MUSn, address:104.36.82.5:4842): protocol: context deadline exceeded; context deadline exceeded Aug 1 21:57:43.392627953 (52h0m33.330286442s) [noise=true , address=104.36.82.52:4101 , piece_id=XYPLQM7QITCCX7T4NKIXP5MZ7I6F24EJJTG37K4NPOBKTUFHULRQ, seg_pos={0 0}, upload=49960]: Done uploading piece. err: ecclient: upload failed (node:12WWXeBjFZR9PzciNfs35fFcx87N9GwDpMsPHnksFDugtaBGWp4, address:104.36.82.52:4101): protocol: rpc: dial tcp 104.36.82.52:4101: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:43.392653192 (52h0m33.330311691s) [noise=true , address=104.36.82.14:4251 , piece_id=O7NADU347E2Q26TPD3IUCK6WE5UVRITC6ALZHBE5SWYTKWXFBMXA, seg_pos={0 0}, upload=49960]: Done uploading piece. err: ecclient: upload failed (node:12btf6Z38aTMx7MaBxhUyZtFqUPgeMtP2aU8Y6FbV8zwu57zArd, address:104.36.82.14:4251): protocol: rpc: dial tcp 104.36.82.14:4251: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:43.392670836 (52h0m33.330329335s) [noise=true , address=104.36.82.12:4166 , piece_id=6ZIWCOBY76GJ6DHZF5G23U4IYTOFB5SMQW5TW3YITYH227PV2FGA, seg_pos={0 0}, upload=49960]: Done uploading piece. err: ecclient: upload failed (node:12iCB7c4ZfpqafpD1PtpbE6BfU81wJW65LmrPVJXtD58yeuoR9t, address:104.36.82.12:4166): protocol: rpc: dial tcp 104.36.82.12:4166: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:43.392698018 (52h0m33.330356517s) [noise=true , address=104.36.82.50:4027 , piece_id=OW7BODU5N7ZEEGE2FHSLNYVJT2Z25LY3J42TZEOR5SP2EWIN6X7A, seg_pos={0 0}, upload=49960]: Done uploading piece. err: ecclient: upload failed (node:1zEQhyuVxSHwdDUgDfcTog2W3oLBx4z9bYUMHpaaoLU1BTU9DQ, address:104.36.82.50:4027): protocol: write tcp 188.166.225.170:37784->104.36.82.50:4027: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:43.392715381 (52h0m33.33037387s) [noise=true , address=104.36.82.43:4856 , piece_id=2SINBLHGRUOWUO4XHBNBG45YE3PW7RNF3QSNVCQXNCFEFHRBFNJQ, seg_pos={0 0}, upload=49960]: Done uploading piece. err: ecclient: upload failed (node:12bQxwBfPEGtmMp6mtnWKsoZt3eMEjLdyYFahETWGHeKJ7BhkiZ, address:104.36.82.43:4856): protocol: rpc: dial tcp 104.36.82.43:4856: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:43.392730059 (52h0m33.330388558s) [noise=true , address=104.36.82.10:4082 , piece_id=45K4BW277ZSGXJYQDA66OMEBVDC7DAWSOJAPX42HANNGRZN35KHQ, seg_pos={0 0}, upload=49960]: Done uploading piece. err: ecclient: upload failed (node:12DKsNCtqxLvpJwLszdLs74V3eb8NH2VAV9V3bngzUczawmNoLK, address:104.36.82.10:4082): protocol: write tcp 188.166.225.170:46638->104.36.82.10:4082: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:43.392753424 (52h0m33.330411923s) [seg_pos={0 0}, upload=49960]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil> Aug 1 21:57:43.624550997 (52h0m33.562209566s) [upload=49960]: Done uploading segments... Aug 1 21:57:43.624582868 (52h0m33.562241407s) [upload=49960]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7) Duration: 5.894643233s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [.................................XXX........................] duration: 5.598842283s FAIL (2025-08-01 22:14:08 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.598864575s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 5.998405166s FAIL (2025-08-01 22:13:55 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.998427298s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 7 out of 60 is failed (threshold=3) [...............................XXXXXXX......................] duration: 5.927298741s FAIL (2025-08-01 22:13:53 UTC) orbiter warning: Operation: DownloadObject uplink: object not found ("orbiter_us_0/4M") Duration: 5.927323679s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [.................................XXXX.......................] duration: 6.025760357s FAIL (2025-08-01 22:13:49 UTC) orbiter warning: Operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 7) ======LOGS====== K4BW277ZSGXJYQDA66OMEBVDC7DAWSOJAPX42HANNGRZN35KHQ, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.481101510 (51h59m34.418760009s) [noise=true , address=104.36.82.5:4842 , piece_id=LV7YAOMPE2AV7TF2WPWR7QJNLT4JZAIPSVZN3E23U5KDBNKZFD6A, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.481422003 (51h59m34.419080502s) [noise=true , address=79.127.205.235:4683 , piece_id=3QHDNAPG4GRLJFVVNJNFXNGKTGKUEMWWUBMW62HHE5N5LJDO4KWA, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.482030897 (51h59m34.419689396s) [noise=true , address=79.127.219.51:4213 , piece_id=L2WSPI5BEIFLGQKJIK4EWXFQXNQRQ7HVKPOWFYRPPUGNAL5DVYWA, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.482285215 (51h59m34.419943704s) [noise=true , address=104.36.82.14:4251 , piece_id=O7NADU347E2Q26TPD3IUCK6WE5UVRITC6ALZHBE5SWYTKWXFBMXA, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.482518370 (51h59m34.420176859s) [noise=true , address=79.127.220.98:4402 , piece_id=DVLSYRFFZ6UT3QRSVAN7AMXVYZNI6PWG2IITHLRUUJATJVK3QOGA, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.482724304 (51h59m34.420382803s) [noise=true , address=79.127.201.214:4297 , piece_id=MPTMGVEYC4SATIN4ANCR2ELVRXHYWRDLMHWQW7HM6UOIV2N6R7ZA, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.482928555 (51h59m34.420587054s) [noise=true , address=104.36.82.37:4680 , piece_id=X4PDUE4WXUUWX6JQVO4NWPQCBMHMHC5YHPIHDQVYYOI2FXV2GVJQ, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.483134960 (51h59m34.420793449s) [noise=true , address=104.36.82.50:4027 , piece_id=OW7BODU5N7ZEEGE2FHSLNYVJT2Z25LY3J42TZEOR5SP2EWIN6X7A, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.483338480 (51h59m34.420996979s) [noise=true , address=104.36.82.55:4204 , piece_id=R5CYXWAVV4ZEMCV7LF4D4XDC4G6KLVBONAH2TH5EIEVL7QHUTRWQ, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.483582847 (51h59m34.421241356s) [noise=true , address=104.36.82.11:4134 , piece_id=RMTNNNC2VD7MGWJDAMT6ZCF3GR43SHANFVAGJ6EHVH7WO5XA6VTA, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.483833737 (51h59m34.421492236s) [noise=true , address=104.36.82.43:4856 , piece_id=2SINBLHGRUOWUO4XHBNBG45YE3PW7RNF3QSNVCQXNCFEFHRBFNJQ, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.484021896 (51h59m34.421680385s) [noise=true , address=104.36.82.52:4101 , piece_id=XYPLQM7QITCCX7T4NKIXP5MZ7I6F24EJJTG37K4NPOBKTUFHULRQ, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.484301662 (51h59m34.421960182s) [noise=true , address=79.127.201.215:4352 , piece_id=I6BRYI5XIACAAAAAE7HQOBCF4OTALMOVPPNBWQCRSH2MM6WUUURQ, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.484712097 (51h59m34.422370596s) [noise=true , address=64.49.13.138:4484 , piece_id=DZ5Y4HTS6BKW2DJ5CVOB64TW4SXB6DNCJEGFUJPLAUXUEHRZRTUA, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:45.576021470 (51h59m35.513679959s) [upload=49960]: Next returned nil segment Aug 1 21:56:45.576036609 (51h59m35.513695098s) [upload=49960]: Waiting for error group managing segments... Aug 1 21:56:45.860378776 (51h59m35.798037275s) [noise=true , address=64.49.13.138:4484 , piece_id=DZ5Y4HTS6BKW2DJ5CVOB64TW4SXB6DNCJEGFUJPLAUXUEHRZRTUA, seg_pos={0 0}, upload=49960]: Done uploading piece. err: <nil> Aug 1 21:56:45.884184188 (51h59m35.821842687s) [noise=true , address=79.127.205.235:4683 , piece_id=3QHDNAPG4GRLJFVVNJNFXNGKTGKUEMWWUBMW62HHE5N5LJDO4KWA, seg_pos={0 0}, upload=49960]: Done uploading piece. err: <nil> Aug 1 21:56:46.145493612 (51h59m36.083152101s) [noise=true , address=79.127.220.98:4402 , piece_id=DVLSYRFFZ6UT3QRSVAN7AMXVYZNI6PWG2IITHLRUUJATJVK3QOGA, seg_pos={0 0}, upload=49960]: Done uploading piece. err: <nil> Aug 1 21:56:46.257505313 (51h59m36.195163802s) [noise=true , address=79.127.220.97:4371 , piece_id=5WGPRFKQETIHTSMA42AM2C3UGCFYJ3DMMPAI2L7DD6E2P37KJKNA, seg_pos={0 0}, upload=49960]: Done uploading piece. err: <nil> Aug 1 21:56:46.334426181 (51h59m36.27208468s) [noise=true , address=79.127.201.215:4352 , piece_id=I6BRYI5XIACAAAAAE7HQOBCF4OTALMOVPPNBWQCRSH2MM6WUUURQ, seg_pos={0 0}, upload=49960]: Done uploading piece. err: <nil> Aug 1 21:56:46.386119275 (51h59m36.323777774s) [noise=true , address=79.127.201.214:4297 , piece_id=MPTMGVEYC4SATIN4ANCR2ELVRXHYWRDLMHWQW7HM6UOIV2N6R7ZA, seg_pos={0 0}, upload=49960]: Done uploading piece. err: <nil> Aug 1 21:56:46.395362048 (51h59m36.333020547s) [noise=true , address=79.127.219.51:4213 , piece_id=L2WSPI5BEIFLGQKJIK4EWXFQXNQRQ7HVKPOWFYRPPUGNAL5DVYWA, seg_pos={0 0}, upload=49960]: Done uploading piece. err: <nil> Aug 1 21:57:43.392434383 (52h0m33.330092872s) [noise=true , address=104.36.82.37:4680 , piece_id=X4PDUE4WXUUWX6JQVO4NWPQCBMHMHC5YHPIHDQVYYOI2FXV2GVJQ, seg_pos={0 0}, upload=49960]: Done uploading piece. err: ecclient: upload failed (node:1bXAqpxGhUvzwYEDGBqYRhZdTeMJcV3TYfngkKTCumpt9Nupi3, address:104.36.82.37:4680): protocol: rpc: dial tcp 104.36.82.37:4680: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:43.392477967 (52h0m33.330136466s) [noise=true , address=104.36.82.11:4134 , piece_id=RMTNNNC2VD7MGWJDAMT6ZCF3GR43SHANFVAGJ6EHVH7WO5XA6VTA, seg_pos={0 0}, upload=49960]: Done uploading piece. err: ecclient: upload failed (node:12R3Z5AgkBmWXMv5XgRyfcyqkG2gmyJs9ogT24kcLy6nuB9bzUG, address:104.36.82.11:4134): protocol: rpc: dial tcp 104.36.82.11:4134: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:43.392514086 (52h0m33.330172585s) [noise=true , address=104.36.82.55:4204 , piece_id=R5CYXWAVV4ZEMCV7LF4D4XDC4G6KLVBONAH2TH5EIEVL7QHUTRWQ, seg_pos={0 0}, upload=49960]: Done uploading piece. err: ecclient: upload failed (node:1rBZH6aLHsnWh22uNzaMiMwmLJCDMn4bYbNC5jRG7KCEuRfqpH, address:104.36.82.55:4204): protocol: write tcp 188.166.225.170:48550->104.36.82.55:4204: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:43.392546537 (52h0m33.330205027s) [noise=true , address=104.36.82.59:4327 , piece_id=OZDEYIY6B4WYZULRDRL2UBOJH2JQNY3YWSITZKWDO7GTT72AN5NQ, seg_pos={0 0}, upload=49960]: Done uploading piece. err: ecclient: upload failed (node:1V4C6AWGbyYQWfinoSj7iKovHeTJ9hqypyC5U9rZ8pveXBBfCY, address:104.36.82.59:4327): protocol: rpc: dial tcp 104.36.82.59:4327: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:43.392579040 (52h0m33.33023754s) [noise=true , address=104.36.82.5:4842 , piece_id=LV7YAOMPE2AV7TF2WPWR7QJNLT4JZAIPSVZN3E23U5KDBNKZFD6A, seg_pos={0 0}, upload=49960]: Done uploading piece. err: ecclient: upload failed (node:12q4VdT5BcjUb16PBYHK6zrnW8dqENKrcGx9zdLtNefDVB1MUSn, address:104.36.82.5:4842): protocol: context deadline exceeded; context deadline exceeded Aug 1 21:57:43.392627953 (52h0m33.330286442s) [noise=true , address=104.36.82.52:4101 , piece_id=XYPLQM7QITCCX7T4NKIXP5MZ7I6F24EJJTG37K4NPOBKTUFHULRQ, seg_pos={0 0}, upload=49960]: Done uploading piece. err: ecclient: upload failed (node:12WWXeBjFZR9PzciNfs35fFcx87N9GwDpMsPHnksFDugtaBGWp4, address:104.36.82.52:4101): protocol: rpc: dial tcp 104.36.82.52:4101: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:43.392653192 (52h0m33.330311691s) [noise=true , address=104.36.82.14:4251 , piece_id=O7NADU347E2Q26TPD3IUCK6WE5UVRITC6ALZHBE5SWYTKWXFBMXA, seg_pos={0 0}, upload=49960]: Done uploading piece. err: ecclient: upload failed (node:12btf6Z38aTMx7MaBxhUyZtFqUPgeMtP2aU8Y6FbV8zwu57zArd, address:104.36.82.14:4251): protocol: rpc: dial tcp 104.36.82.14:4251: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:43.392670836 (52h0m33.330329335s) [noise=true , address=104.36.82.12:4166 , piece_id=6ZIWCOBY76GJ6DHZF5G23U4IYTOFB5SMQW5TW3YITYH227PV2FGA, seg_pos={0 0}, upload=49960]: Done uploading piece. err: ecclient: upload failed (node:12iCB7c4ZfpqafpD1PtpbE6BfU81wJW65LmrPVJXtD58yeuoR9t, address:104.36.82.12:4166): protocol: rpc: dial tcp 104.36.82.12:4166: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:43.392698018 (52h0m33.330356517s) [noise=true , address=104.36.82.50:4027 , piece_id=OW7BODU5N7ZEEGE2FHSLNYVJT2Z25LY3J42TZEOR5SP2EWIN6X7A, seg_pos={0 0}, upload=49960]: Done uploading piece. err: ecclient: upload failed (node:1zEQhyuVxSHwdDUgDfcTog2W3oLBx4z9bYUMHpaaoLU1BTU9DQ, address:104.36.82.50:4027): protocol: write tcp 188.166.225.170:37784->104.36.82.50:4027: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:43.392715381 (52h0m33.33037387s) [noise=true , address=104.36.82.43:4856 , piece_id=2SINBLHGRUOWUO4XHBNBG45YE3PW7RNF3QSNVCQXNCFEFHRBFNJQ, seg_pos={0 0}, upload=49960]: Done uploading piece. err: ecclient: upload failed (node:12bQxwBfPEGtmMp6mtnWKsoZt3eMEjLdyYFahETWGHeKJ7BhkiZ, address:104.36.82.43:4856): protocol: rpc: dial tcp 104.36.82.43:4856: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:43.392730059 (52h0m33.330388558s) [noise=true , address=104.36.82.10:4082 , piece_id=45K4BW277ZSGXJYQDA66OMEBVDC7DAWSOJAPX42HANNGRZN35KHQ, seg_pos={0 0}, upload=49960]: Done uploading piece. err: ecclient: upload failed (node:12DKsNCtqxLvpJwLszdLs74V3eb8NH2VAV9V3bngzUczawmNoLK, address:104.36.82.10:4082): protocol: write tcp 188.166.225.170:46638->104.36.82.10:4082: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:43.392753424 (52h0m33.330411923s) [seg_pos={0 0}, upload=49960]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil> Aug 1 21:57:43.624550997 (52h0m33.562209566s) [upload=49960]: Done uploading segments... Aug 1 21:57:43.624582868 (52h0m33.562241407s) [upload=49960]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7) Duration: 6.025795755s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [..................................XXX.......................] duration: 5.6011506s FAIL (2025-08-01 22:13:41 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.601168715s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 5 out of 60 is failed (threshold=3) [..................X............X.....X.......X.....X........] duration: 2.043234728s FAIL (2025-08-01 22:13:34 UTC) orbiter warning: Operation: ListObjects unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/1K Duration: 2.04326185s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.................................XX.X.X.....................] duration: 5.946896129s FAIL (2025-08-01 22:13:25 UTC) orbiter warning: Operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 7NK72474VWE4E3YOBCDZ5O5ZOH2MSA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB): protocol: write tcp 188.166.225.170:38368->104.36.82.12:4151: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:56:50.418604753 (51h59m40.356263252s) [seg_pos={0 0}, upload=49961]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 1 21:56:50.418680879 (51h59m40.356339378s) [upload=49961]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 1 21:56:50.418685998 (51h59m40.356344487s) [upload=49961]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 1 21:56:50.726563872 (51h59m40.664222371s) [upload=49961]: Done uploading segments... Aug 1 21:56:50.726586586 (51h59m40.664245085s) [upload=49961]: Upload finished. err: <nil> Aug 1 21:57:28.852869669 (52h0m18.790528158s) [upload=49973]: Starting upload Aug 1 21:57:28.852942389 (52h0m18.790600888s) [upload=49973]: Done starting upload Aug 1 21:57:28.853007122 (52h0m18.790665611s) [upload=49973]: Uploading segments... Aug 1 21:57:28.853174002 (52h0m18.790832491s) [upload=49973]: Got next segment. Inline: false Aug 1 21:57:28.853208168 (52h0m18.790866657s) [upload=49973]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 1 21:57:28.853212666 (52h0m18.790871165s) [upload=49973]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 1 21:57:29.095087519 (52h0m19.032746008s) [seg_pos={0 0}, upload=49973]: Begin upload segment... Aug 1 21:57:29.095129960 (52h0m19.032788459s) [seg_pos={0 0}, upload=49973]: Done begin upload segment. Aug 1 21:57:29.095164997 (52h0m19.032823496s) [noise=true , address=77.90.31.21:4878 , piece_id=BLNTYBOAIZWISSUJYK6BSBPTSNL6QWLGEN6AP6P4NPDVSURYG5AA, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.095171800 (52h0m19.032830289s) [noise=true , address=104.36.82.6:4875 , piece_id=CS5AMB7CIBQWWOU57Z3RCCSNVKIPE4B5QWU5YIQYI2UCWH7PAPVA, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.095510027 (52h0m19.033168526s) [noise=true , address=77.90.31.18:4806 , piece_id=52B6CAHGLFVTFIDQHHHAWKCBP2WJJY7OTTGRZZ7CEUH3HDA43Q6A, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.095544193 (52h0m19.033202692s) [noise=true , address=77.90.31.10:4518 , piece_id=WUOVXUSDG6YQTDPWG4YO7VLJNZZVLH6VRGMWOJABG2EJRU7XHP5A, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.095774283 (52h0m19.033432782s) [noise=true , address=104.36.82.7:4909 , piece_id=QHJCVBT6KTU34K2KWFNACWYVVE3WJ2RU76BHVSKL4HLCZEFXZYPA, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.095819880 (52h0m19.033478379s) [noise=true , address=45.91.116.12:4225 , piece_id=PBQPLA5ZYRFPSKHDKCEFOEBS5T3GYMP2FLSFDQWJLVY756KPLJNQ, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.096021656 (52h0m19.033680155s) [noise=true , address=45.91.116.13:4153 , piece_id=PMLKMZO66KXF3ZONO5ZZOQZICFK5XNGL6MN5FFBKICVFU5AWU6PQ, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.096069618 (52h0m19.033728117s) [noise=true , address=45.91.116.9:4262 , piece_id=A3F4L3RR6WPFMBFFKLATNGG3CASBIPWQP5O3G5URFNTXBE3ZW3VA, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.096348041 (52h0m19.03400654s) [noise=true , address=104.36.82.5:4840 , piece_id=UYVKVFCJJP7XZYRIVPS4ABGEMDUMNR7YW4QGG5ZFTFCGNCJO3CSQ, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.096354803 (52h0m19.034013292s) [noise=true , address=77.90.31.12:4590 , piece_id=D6FUBIFCQJSDRGWEP5GG34AQPNB3UROCYOATD6SRTJ7KAYVMFCOA, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.096608579 (52h0m19.034267078s) [noise=true , address=77.90.31.16:4733 , piece_id=DMASMGOUATLPEKCGLHKNNIMMFYVVGSULNUOMJU6RJY3ULBDOY6KQ, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.096830774 (52h0m19.034489273s) [noise=true , address=104.36.82.4:4802 , piece_id=CLM2G33M6AK5HIESRW6LZY33NKHIR2PGITWJ23NPILRCA6UYIEHA, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.097107233 (52h0m19.034765732s) [noise=true , address=77.90.31.19:4843 , piece_id=UFONZYHL7EAFYE5PRW32RKUVQPMZPXBEFLLIFJMAB2TXJ3C6KKNA, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.097324328 (52h0m19.034982827s) [noise=true , address=77.90.31.17:4771 , piece_id=Y3WSALJX23CWPPYQ3XYMCFOSBLBX2E2JQCCS7EOEYNBWY4CDRT3Q, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.098682446 (52h0m19.036340945s) [noise=true , address=77.90.31.15:4698 , piece_id=TN4OTPUD47FD6WOLZBEHNDI43WY6QVO5JK2JNJXNVM2IT6XMGERQ, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.099041603 (52h0m19.036700102s) [noise=true , address=45.91.116.11:4118 , piece_id=AKZIOAR6NRNO7JWCQZIKNSL7IUH5S4IG3KPG3IY2FAKG74DH3E3Q, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:30.718412255 (52h0m20.656070754s) [upload=49973]: Next returned nil segment Aug 1 21:57:30.718429869 (52h0m20.656088368s) [upload=49973]: Waiting for error group managing segments... Aug 1 21:57:31.107386675 (52h0m21.045045164s) [noise=true , address=45.91.116.13:4153 , piece_id=PMLKMZO66KXF3ZONO5ZZOQZICFK5XNGL6MN5FFBKICVFU5AWU6PQ, seg_pos={0 0}, upload=49973]: Done uploading piece. err: <nil> Aug 1 21:57:31.108529412 (52h0m21.046187911s) [noise=true , address=77.90.31.12:4590 , piece_id=D6FUBIFCQJSDRGWEP5GG34AQPNB3UROCYOATD6SRTJ7KAYVMFCOA, seg_pos={0 0}, upload=49973]: Done uploading piece. err: <nil> Aug 1 21:57:31.109975858 (52h0m21.047634347s) [noise=true , address=77.90.31.15:4698 , piece_id=TN4OTPUD47FD6WOLZBEHNDI43WY6QVO5JK2JNJXNVM2IT6XMGERQ, seg_pos={0 0}, upload=49973]: Done uploading piece. err: <nil> Aug 1 21:57:31.115273013 (52h0m21.052931502s) [noise=true , address=77.90.31.10:4518 , piece_id=WUOVXUSDG6YQTDPWG4YO7VLJNZZVLH6VRGMWOJABG2EJRU7XHP5A, seg_pos={0 0}, upload=49973]: Done uploading piece. err: <nil> Aug 1 21:57:31.116767281 (52h0m21.054425781s) [noise=true , address=77.90.31.18:4806 , piece_id=52B6CAHGLFVTFIDQHHHAWKCBP2WJJY7OTTGRZZ7CEUH3HDA43Q6A, seg_pos={0 0}, upload=49973]: Done uploading piece. err: <nil> Aug 1 21:57:31.201146107 (52h0m21.138804606s) [noise=true , address=77.90.31.16:4733 , piece_id=DMASMGOUATLPEKCGLHKNNIMMFYVVGSULNUOMJU6RJY3ULBDOY6KQ, seg_pos={0 0}, upload=49973]: Done uploading piece. err: <nil> Aug 1 21:57:31.201833732 (52h0m21.139492231s) [noise=true , address=77.90.31.19:4843 , piece_id=UFONZYHL7EAFYE5PRW32RKUVQPMZPXBEFLLIFJMAB2TXJ3C6KKNA, seg_pos={0 0}, upload=49973]: Done uploading piece. err: <nil> Aug 1 21:57:31.211007613 (52h0m21.148666112s) [noise=true , address=45.91.116.9:4262 , piece_id=A3F4L3RR6WPFMBFFKLATNGG3CASBIPWQP5O3G5URFNTXBE3ZW3VA, seg_pos={0 0}, upload=49973]: Done uploading piece. err: <nil> Aug 1 21:57:31.215929169 (52h0m21.153587668s) [noise=true , address=77.90.31.17:4771 , piece_id=Y3WSALJX23CWPPYQ3XYMCFOSBLBX2E2JQCCS7EOEYNBWY4CDRT3Q, seg_pos={0 0}, upload=49973]: Done uploading piece. err: <nil> Aug 1 21:57:31.222973975 (52h0m21.160632474s) [noise=true , address=77.90.31.21:4878 , piece_id=BLNTYBOAIZWISSUJYK6BSBPTSNL6QWLGEN6AP6P4NPDVSURYG5AA, seg_pos={0 0}, upload=49973]: Done uploading piece. err: <nil> Aug 1 21:57:31.234392441 (52h0m21.17205094s) [noise=true , address=45.91.116.12:4225 , piece_id=PBQPLA5ZYRFPSKHDKCEFOEBS5T3GYMP2FLSFDQWJLVY756KPLJNQ, seg_pos={0 0}, upload=49973]: Done uploading piece. err: <nil> Aug 1 21:57:31.284794305 (52h0m21.222452804s) [noise=true , address=45.91.116.11:4118 , piece_id=AKZIOAR6NRNO7JWCQZIKNSL7IUH5S4IG3KPG3IY2FAKG74DH3E3Q, seg_pos={0 0}, upload=49973]: Done uploading piece. err: <nil> Aug 1 21:57:58.202420932 (52h0m48.140079471s) [noise=true , address=104.36.82.6:4875 , piece_id=CS5AMB7CIBQWWOU57Z3RCCSNVKIPE4B5QWU5YIQYI2UCWH7PAPVA, seg_pos={0 0}, upload=49973]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:104.36.82.6:4875): protocol: write tcp 188.166.225.170:40166->104.36.82.6:4875: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:58.202441131 (52h0m48.14009963s) [noise=true , address=104.36.82.5:4840 , piece_id=UYVKVFCJJP7XZYRIVPS4ABGEMDUMNR7YW4QGG5ZFTFCGNCJO3CSQ, seg_pos={0 0}, upload=49973]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:104.36.82.5:4840): protocol: rpc: dial tcp 104.36.82.5:4840: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:58.202448445 (52h0m48.140106954s) [noise=true , address=104.36.82.7:4909 , piece_id=QHJCVBT6KTU34K2KWFNACWYVVE3WJ2RU76BHVSKL4HLCZEFXZYPA, seg_pos={0 0}, upload=49973]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:104.36.82.7:4909): protocol: rpc: dial tcp 104.36.82.7:4909: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:58.202496487 (52h0m48.140155026s) [noise=true , address=104.36.82.4:4802 , piece_id=CLM2G33M6AK5HIESRW6LZY33NKHIR2PGITWJ23NPILRCA6UYIEHA, seg_pos={0 0}, upload=49973]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: write tcp 188.166.225.170:47696->104.36.82.4:4802: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:58.202524240 (52h0m48.140182739s) [seg_pos={0 0}, upload=49973]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Aug 1 21:57:58.435983932 (52h0m48.373642431s) [upload=49973]: Done uploading segments... Aug 1 21:57:58.436007667 (52h0m48.373666166s) [upload=49973]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.946921237s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [..................................XXXX......................] duration: 5.74634695s FAIL (2025-08-01 22:13:18 UTC) orbiter warning: Operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 7) ======LOGS====== K4BW277ZSGXJYQDA66OMEBVDC7DAWSOJAPX42HANNGRZN35KHQ, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.481101510 (51h59m34.418760009s) [noise=true , address=104.36.82.5:4842 , piece_id=LV7YAOMPE2AV7TF2WPWR7QJNLT4JZAIPSVZN3E23U5KDBNKZFD6A, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.481422003 (51h59m34.419080502s) [noise=true , address=79.127.205.235:4683 , piece_id=3QHDNAPG4GRLJFVVNJNFXNGKTGKUEMWWUBMW62HHE5N5LJDO4KWA, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.482030897 (51h59m34.419689396s) [noise=true , address=79.127.219.51:4213 , piece_id=L2WSPI5BEIFLGQKJIK4EWXFQXNQRQ7HVKPOWFYRPPUGNAL5DVYWA, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.482285215 (51h59m34.419943704s) [noise=true , address=104.36.82.14:4251 , piece_id=O7NADU347E2Q26TPD3IUCK6WE5UVRITC6ALZHBE5SWYTKWXFBMXA, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.482518370 (51h59m34.420176859s) [noise=true , address=79.127.220.98:4402 , piece_id=DVLSYRFFZ6UT3QRSVAN7AMXVYZNI6PWG2IITHLRUUJATJVK3QOGA, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.482724304 (51h59m34.420382803s) [noise=true , address=79.127.201.214:4297 , piece_id=MPTMGVEYC4SATIN4ANCR2ELVRXHYWRDLMHWQW7HM6UOIV2N6R7ZA, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.482928555 (51h59m34.420587054s) [noise=true , address=104.36.82.37:4680 , piece_id=X4PDUE4WXUUWX6JQVO4NWPQCBMHMHC5YHPIHDQVYYOI2FXV2GVJQ, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.483134960 (51h59m34.420793449s) [noise=true , address=104.36.82.50:4027 , piece_id=OW7BODU5N7ZEEGE2FHSLNYVJT2Z25LY3J42TZEOR5SP2EWIN6X7A, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.483338480 (51h59m34.420996979s) [noise=true , address=104.36.82.55:4204 , piece_id=R5CYXWAVV4ZEMCV7LF4D4XDC4G6KLVBONAH2TH5EIEVL7QHUTRWQ, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.483582847 (51h59m34.421241356s) [noise=true , address=104.36.82.11:4134 , piece_id=RMTNNNC2VD7MGWJDAMT6ZCF3GR43SHANFVAGJ6EHVH7WO5XA6VTA, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.483833737 (51h59m34.421492236s) [noise=true , address=104.36.82.43:4856 , piece_id=2SINBLHGRUOWUO4XHBNBG45YE3PW7RNF3QSNVCQXNCFEFHRBFNJQ, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.484021896 (51h59m34.421680385s) [noise=true , address=104.36.82.52:4101 , piece_id=XYPLQM7QITCCX7T4NKIXP5MZ7I6F24EJJTG37K4NPOBKTUFHULRQ, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.484301662 (51h59m34.421960182s) [noise=true , address=79.127.201.215:4352 , piece_id=I6BRYI5XIACAAAAAE7HQOBCF4OTALMOVPPNBWQCRSH2MM6WUUURQ, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.484712097 (51h59m34.422370596s) [noise=true , address=64.49.13.138:4484 , piece_id=DZ5Y4HTS6BKW2DJ5CVOB64TW4SXB6DNCJEGFUJPLAUXUEHRZRTUA, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:45.576021470 (51h59m35.513679959s) [upload=49960]: Next returned nil segment Aug 1 21:56:45.576036609 (51h59m35.513695098s) [upload=49960]: Waiting for error group managing segments... Aug 1 21:56:45.860378776 (51h59m35.798037275s) [noise=true , address=64.49.13.138:4484 , piece_id=DZ5Y4HTS6BKW2DJ5CVOB64TW4SXB6DNCJEGFUJPLAUXUEHRZRTUA, seg_pos={0 0}, upload=49960]: Done uploading piece. err: <nil> Aug 1 21:56:45.884184188 (51h59m35.821842687s) [noise=true , address=79.127.205.235:4683 , piece_id=3QHDNAPG4GRLJFVVNJNFXNGKTGKUEMWWUBMW62HHE5N5LJDO4KWA, seg_pos={0 0}, upload=49960]: Done uploading piece. err: <nil> Aug 1 21:56:46.145493612 (51h59m36.083152101s) [noise=true , address=79.127.220.98:4402 , piece_id=DVLSYRFFZ6UT3QRSVAN7AMXVYZNI6PWG2IITHLRUUJATJVK3QOGA, seg_pos={0 0}, upload=49960]: Done uploading piece. err: <nil> Aug 1 21:56:46.257505313 (51h59m36.195163802s) [noise=true , address=79.127.220.97:4371 , piece_id=5WGPRFKQETIHTSMA42AM2C3UGCFYJ3DMMPAI2L7DD6E2P37KJKNA, seg_pos={0 0}, upload=49960]: Done uploading piece. err: <nil> Aug 1 21:56:46.334426181 (51h59m36.27208468s) [noise=true , address=79.127.201.215:4352 , piece_id=I6BRYI5XIACAAAAAE7HQOBCF4OTALMOVPPNBWQCRSH2MM6WUUURQ, seg_pos={0 0}, upload=49960]: Done uploading piece. err: <nil> Aug 1 21:56:46.386119275 (51h59m36.323777774s) [noise=true , address=79.127.201.214:4297 , piece_id=MPTMGVEYC4SATIN4ANCR2ELVRXHYWRDLMHWQW7HM6UOIV2N6R7ZA, seg_pos={0 0}, upload=49960]: Done uploading piece. err: <nil> Aug 1 21:56:46.395362048 (51h59m36.333020547s) [noise=true , address=79.127.219.51:4213 , piece_id=L2WSPI5BEIFLGQKJIK4EWXFQXNQRQ7HVKPOWFYRPPUGNAL5DVYWA, seg_pos={0 0}, upload=49960]: Done uploading piece. err: <nil> Aug 1 21:57:43.392434383 (52h0m33.330092872s) [noise=true , address=104.36.82.37:4680 , piece_id=X4PDUE4WXUUWX6JQVO4NWPQCBMHMHC5YHPIHDQVYYOI2FXV2GVJQ, seg_pos={0 0}, upload=49960]: Done uploading piece. err: ecclient: upload failed (node:1bXAqpxGhUvzwYEDGBqYRhZdTeMJcV3TYfngkKTCumpt9Nupi3, address:104.36.82.37:4680): protocol: rpc: dial tcp 104.36.82.37:4680: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:43.392477967 (52h0m33.330136466s) [noise=true , address=104.36.82.11:4134 , piece_id=RMTNNNC2VD7MGWJDAMT6ZCF3GR43SHANFVAGJ6EHVH7WO5XA6VTA, seg_pos={0 0}, upload=49960]: Done uploading piece. err: ecclient: upload failed (node:12R3Z5AgkBmWXMv5XgRyfcyqkG2gmyJs9ogT24kcLy6nuB9bzUG, address:104.36.82.11:4134): protocol: rpc: dial tcp 104.36.82.11:4134: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:43.392514086 (52h0m33.330172585s) [noise=true , address=104.36.82.55:4204 , piece_id=R5CYXWAVV4ZEMCV7LF4D4XDC4G6KLVBONAH2TH5EIEVL7QHUTRWQ, seg_pos={0 0}, upload=49960]: Done uploading piece. err: ecclient: upload failed (node:1rBZH6aLHsnWh22uNzaMiMwmLJCDMn4bYbNC5jRG7KCEuRfqpH, address:104.36.82.55:4204): protocol: write tcp 188.166.225.170:48550->104.36.82.55:4204: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:43.392546537 (52h0m33.330205027s) [noise=true , address=104.36.82.59:4327 , piece_id=OZDEYIY6B4WYZULRDRL2UBOJH2JQNY3YWSITZKWDO7GTT72AN5NQ, seg_pos={0 0}, upload=49960]: Done uploading piece. err: ecclient: upload failed (node:1V4C6AWGbyYQWfinoSj7iKovHeTJ9hqypyC5U9rZ8pveXBBfCY, address:104.36.82.59:4327): protocol: rpc: dial tcp 104.36.82.59:4327: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:43.392579040 (52h0m33.33023754s) [noise=true , address=104.36.82.5:4842 , piece_id=LV7YAOMPE2AV7TF2WPWR7QJNLT4JZAIPSVZN3E23U5KDBNKZFD6A, seg_pos={0 0}, upload=49960]: Done uploading piece. err: ecclient: upload failed (node:12q4VdT5BcjUb16PBYHK6zrnW8dqENKrcGx9zdLtNefDVB1MUSn, address:104.36.82.5:4842): protocol: context deadline exceeded; context deadline exceeded Aug 1 21:57:43.392627953 (52h0m33.330286442s) [noise=true , address=104.36.82.52:4101 , piece_id=XYPLQM7QITCCX7T4NKIXP5MZ7I6F24EJJTG37K4NPOBKTUFHULRQ, seg_pos={0 0}, upload=49960]: Done uploading piece. err: ecclient: upload failed (node:12WWXeBjFZR9PzciNfs35fFcx87N9GwDpMsPHnksFDugtaBGWp4, address:104.36.82.52:4101): protocol: rpc: dial tcp 104.36.82.52:4101: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:43.392653192 (52h0m33.330311691s) [noise=true , address=104.36.82.14:4251 , piece_id=O7NADU347E2Q26TPD3IUCK6WE5UVRITC6ALZHBE5SWYTKWXFBMXA, seg_pos={0 0}, upload=49960]: Done uploading piece. err: ecclient: upload failed (node:12btf6Z38aTMx7MaBxhUyZtFqUPgeMtP2aU8Y6FbV8zwu57zArd, address:104.36.82.14:4251): protocol: rpc: dial tcp 104.36.82.14:4251: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:43.392670836 (52h0m33.330329335s) [noise=true , address=104.36.82.12:4166 , piece_id=6ZIWCOBY76GJ6DHZF5G23U4IYTOFB5SMQW5TW3YITYH227PV2FGA, seg_pos={0 0}, upload=49960]: Done uploading piece. err: ecclient: upload failed (node:12iCB7c4ZfpqafpD1PtpbE6BfU81wJW65LmrPVJXtD58yeuoR9t, address:104.36.82.12:4166): protocol: rpc: dial tcp 104.36.82.12:4166: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:43.392698018 (52h0m33.330356517s) [noise=true , address=104.36.82.50:4027 , piece_id=OW7BODU5N7ZEEGE2FHSLNYVJT2Z25LY3J42TZEOR5SP2EWIN6X7A, seg_pos={0 0}, upload=49960]: Done uploading piece. err: ecclient: upload failed (node:1zEQhyuVxSHwdDUgDfcTog2W3oLBx4z9bYUMHpaaoLU1BTU9DQ, address:104.36.82.50:4027): protocol: write tcp 188.166.225.170:37784->104.36.82.50:4027: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:43.392715381 (52h0m33.33037387s) [noise=true , address=104.36.82.43:4856 , piece_id=2SINBLHGRUOWUO4XHBNBG45YE3PW7RNF3QSNVCQXNCFEFHRBFNJQ, seg_pos={0 0}, upload=49960]: Done uploading piece. err: ecclient: upload failed (node:12bQxwBfPEGtmMp6mtnWKsoZt3eMEjLdyYFahETWGHeKJ7BhkiZ, address:104.36.82.43:4856): protocol: rpc: dial tcp 104.36.82.43:4856: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:43.392730059 (52h0m33.330388558s) [noise=true , address=104.36.82.10:4082 , piece_id=45K4BW277ZSGXJYQDA66OMEBVDC7DAWSOJAPX42HANNGRZN35KHQ, seg_pos={0 0}, upload=49960]: Done uploading piece. err: ecclient: upload failed (node:12DKsNCtqxLvpJwLszdLs74V3eb8NH2VAV9V3bngzUczawmNoLK, address:104.36.82.10:4082): protocol: write tcp 188.166.225.170:46638->104.36.82.10:4082: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:43.392753424 (52h0m33.330411923s) [seg_pos={0 0}, upload=49960]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil> Aug 1 21:57:43.624550997 (52h0m33.562209566s) [upload=49960]: Done uploading segments... Aug 1 21:57:43.624582868 (52h0m33.562241407s) [upload=49960]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7) Duration: 5.746369354s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 6.227564625s FAIL (2025-08-01 22:13:16 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.227585917s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 7 out of 60 is failed (threshold=3) [................................XXXXXXX.....................] duration: 5.943696808s FAIL (2025-08-01 22:13:13 UTC) orbiter warning: Operation: DownloadObject uplink: object not found ("orbiter_us_0/4M") Duration: 5.943722928s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 5 out of 60 is failed (threshold=3) [...................X............X.....X.......X.....X.......] duration: 2.029088764s FAIL (2025-08-01 22:13:08 UTC) orbiter warning: Operation: ListObjects unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/1K Duration: 2.029116678s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 6.152593826s FAIL (2025-08-01 22:13:01 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.152638913s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [...................................XXX......................] duration: 5.843387688s FAIL (2025-08-01 22:13:01 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.843413127s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [...................................XXXX.....................] duration: 6.013406841s FAIL (2025-08-01 22:12:53 UTC) orbiter warning: Operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 7) ======LOGS====== K4BW277ZSGXJYQDA66OMEBVDC7DAWSOJAPX42HANNGRZN35KHQ, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.481101510 (51h59m34.418760009s) [noise=true , address=104.36.82.5:4842 , piece_id=LV7YAOMPE2AV7TF2WPWR7QJNLT4JZAIPSVZN3E23U5KDBNKZFD6A, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.481422003 (51h59m34.419080502s) [noise=true , address=79.127.205.235:4683 , piece_id=3QHDNAPG4GRLJFVVNJNFXNGKTGKUEMWWUBMW62HHE5N5LJDO4KWA, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.482030897 (51h59m34.419689396s) [noise=true , address=79.127.219.51:4213 , piece_id=L2WSPI5BEIFLGQKJIK4EWXFQXNQRQ7HVKPOWFYRPPUGNAL5DVYWA, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.482285215 (51h59m34.419943704s) [noise=true , address=104.36.82.14:4251 , piece_id=O7NADU347E2Q26TPD3IUCK6WE5UVRITC6ALZHBE5SWYTKWXFBMXA, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.482518370 (51h59m34.420176859s) [noise=true , address=79.127.220.98:4402 , piece_id=DVLSYRFFZ6UT3QRSVAN7AMXVYZNI6PWG2IITHLRUUJATJVK3QOGA, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.482724304 (51h59m34.420382803s) [noise=true , address=79.127.201.214:4297 , piece_id=MPTMGVEYC4SATIN4ANCR2ELVRXHYWRDLMHWQW7HM6UOIV2N6R7ZA, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.482928555 (51h59m34.420587054s) [noise=true , address=104.36.82.37:4680 , piece_id=X4PDUE4WXUUWX6JQVO4NWPQCBMHMHC5YHPIHDQVYYOI2FXV2GVJQ, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.483134960 (51h59m34.420793449s) [noise=true , address=104.36.82.50:4027 , piece_id=OW7BODU5N7ZEEGE2FHSLNYVJT2Z25LY3J42TZEOR5SP2EWIN6X7A, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.483338480 (51h59m34.420996979s) [noise=true , address=104.36.82.55:4204 , piece_id=R5CYXWAVV4ZEMCV7LF4D4XDC4G6KLVBONAH2TH5EIEVL7QHUTRWQ, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.483582847 (51h59m34.421241356s) [noise=true , address=104.36.82.11:4134 , piece_id=RMTNNNC2VD7MGWJDAMT6ZCF3GR43SHANFVAGJ6EHVH7WO5XA6VTA, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.483833737 (51h59m34.421492236s) [noise=true , address=104.36.82.43:4856 , piece_id=2SINBLHGRUOWUO4XHBNBG45YE3PW7RNF3QSNVCQXNCFEFHRBFNJQ, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.484021896 (51h59m34.421680385s) [noise=true , address=104.36.82.52:4101 , piece_id=XYPLQM7QITCCX7T4NKIXP5MZ7I6F24EJJTG37K4NPOBKTUFHULRQ, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.484301662 (51h59m34.421960182s) [noise=true , address=79.127.201.215:4352 , piece_id=I6BRYI5XIACAAAAAE7HQOBCF4OTALMOVPPNBWQCRSH2MM6WUUURQ, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.484712097 (51h59m34.422370596s) [noise=true , address=64.49.13.138:4484 , piece_id=DZ5Y4HTS6BKW2DJ5CVOB64TW4SXB6DNCJEGFUJPLAUXUEHRZRTUA, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:45.576021470 (51h59m35.513679959s) [upload=49960]: Next returned nil segment Aug 1 21:56:45.576036609 (51h59m35.513695098s) [upload=49960]: Waiting for error group managing segments... Aug 1 21:56:45.860378776 (51h59m35.798037275s) [noise=true , address=64.49.13.138:4484 , piece_id=DZ5Y4HTS6BKW2DJ5CVOB64TW4SXB6DNCJEGFUJPLAUXUEHRZRTUA, seg_pos={0 0}, upload=49960]: Done uploading piece. err: <nil> Aug 1 21:56:45.884184188 (51h59m35.821842687s) [noise=true , address=79.127.205.235:4683 , piece_id=3QHDNAPG4GRLJFVVNJNFXNGKTGKUEMWWUBMW62HHE5N5LJDO4KWA, seg_pos={0 0}, upload=49960]: Done uploading piece. err: <nil> Aug 1 21:56:46.145493612 (51h59m36.083152101s) [noise=true , address=79.127.220.98:4402 , piece_id=DVLSYRFFZ6UT3QRSVAN7AMXVYZNI6PWG2IITHLRUUJATJVK3QOGA, seg_pos={0 0}, upload=49960]: Done uploading piece. err: <nil> Aug 1 21:56:46.257505313 (51h59m36.195163802s) [noise=true , address=79.127.220.97:4371 , piece_id=5WGPRFKQETIHTSMA42AM2C3UGCFYJ3DMMPAI2L7DD6E2P37KJKNA, seg_pos={0 0}, upload=49960]: Done uploading piece. err: <nil> Aug 1 21:56:46.334426181 (51h59m36.27208468s) [noise=true , address=79.127.201.215:4352 , piece_id=I6BRYI5XIACAAAAAE7HQOBCF4OTALMOVPPNBWQCRSH2MM6WUUURQ, seg_pos={0 0}, upload=49960]: Done uploading piece. err: <nil> Aug 1 21:56:46.386119275 (51h59m36.323777774s) [noise=true , address=79.127.201.214:4297 , piece_id=MPTMGVEYC4SATIN4ANCR2ELVRXHYWRDLMHWQW7HM6UOIV2N6R7ZA, seg_pos={0 0}, upload=49960]: Done uploading piece. err: <nil> Aug 1 21:56:46.395362048 (51h59m36.333020547s) [noise=true , address=79.127.219.51:4213 , piece_id=L2WSPI5BEIFLGQKJIK4EWXFQXNQRQ7HVKPOWFYRPPUGNAL5DVYWA, seg_pos={0 0}, upload=49960]: Done uploading piece. err: <nil> Aug 1 21:57:43.392434383 (52h0m33.330092872s) [noise=true , address=104.36.82.37:4680 , piece_id=X4PDUE4WXUUWX6JQVO4NWPQCBMHMHC5YHPIHDQVYYOI2FXV2GVJQ, seg_pos={0 0}, upload=49960]: Done uploading piece. err: ecclient: upload failed (node:1bXAqpxGhUvzwYEDGBqYRhZdTeMJcV3TYfngkKTCumpt9Nupi3, address:104.36.82.37:4680): protocol: rpc: dial tcp 104.36.82.37:4680: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:43.392477967 (52h0m33.330136466s) [noise=true , address=104.36.82.11:4134 , piece_id=RMTNNNC2VD7MGWJDAMT6ZCF3GR43SHANFVAGJ6EHVH7WO5XA6VTA, seg_pos={0 0}, upload=49960]: Done uploading piece. err: ecclient: upload failed (node:12R3Z5AgkBmWXMv5XgRyfcyqkG2gmyJs9ogT24kcLy6nuB9bzUG, address:104.36.82.11:4134): protocol: rpc: dial tcp 104.36.82.11:4134: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:43.392514086 (52h0m33.330172585s) [noise=true , address=104.36.82.55:4204 , piece_id=R5CYXWAVV4ZEMCV7LF4D4XDC4G6KLVBONAH2TH5EIEVL7QHUTRWQ, seg_pos={0 0}, upload=49960]: Done uploading piece. err: ecclient: upload failed (node:1rBZH6aLHsnWh22uNzaMiMwmLJCDMn4bYbNC5jRG7KCEuRfqpH, address:104.36.82.55:4204): protocol: write tcp 188.166.225.170:48550->104.36.82.55:4204: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:43.392546537 (52h0m33.330205027s) [noise=true , address=104.36.82.59:4327 , piece_id=OZDEYIY6B4WYZULRDRL2UBOJH2JQNY3YWSITZKWDO7GTT72AN5NQ, seg_pos={0 0}, upload=49960]: Done uploading piece. err: ecclient: upload failed (node:1V4C6AWGbyYQWfinoSj7iKovHeTJ9hqypyC5U9rZ8pveXBBfCY, address:104.36.82.59:4327): protocol: rpc: dial tcp 104.36.82.59:4327: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:43.392579040 (52h0m33.33023754s) [noise=true , address=104.36.82.5:4842 , piece_id=LV7YAOMPE2AV7TF2WPWR7QJNLT4JZAIPSVZN3E23U5KDBNKZFD6A, seg_pos={0 0}, upload=49960]: Done uploading piece. err: ecclient: upload failed (node:12q4VdT5BcjUb16PBYHK6zrnW8dqENKrcGx9zdLtNefDVB1MUSn, address:104.36.82.5:4842): protocol: context deadline exceeded; context deadline exceeded Aug 1 21:57:43.392627953 (52h0m33.330286442s) [noise=true , address=104.36.82.52:4101 , piece_id=XYPLQM7QITCCX7T4NKIXP5MZ7I6F24EJJTG37K4NPOBKTUFHULRQ, seg_pos={0 0}, upload=49960]: Done uploading piece. err: ecclient: upload failed (node:12WWXeBjFZR9PzciNfs35fFcx87N9GwDpMsPHnksFDugtaBGWp4, address:104.36.82.52:4101): protocol: rpc: dial tcp 104.36.82.52:4101: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:43.392653192 (52h0m33.330311691s) [noise=true , address=104.36.82.14:4251 , piece_id=O7NADU347E2Q26TPD3IUCK6WE5UVRITC6ALZHBE5SWYTKWXFBMXA, seg_pos={0 0}, upload=49960]: Done uploading piece. err: ecclient: upload failed (node:12btf6Z38aTMx7MaBxhUyZtFqUPgeMtP2aU8Y6FbV8zwu57zArd, address:104.36.82.14:4251): protocol: rpc: dial tcp 104.36.82.14:4251: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:43.392670836 (52h0m33.330329335s) [noise=true , address=104.36.82.12:4166 , piece_id=6ZIWCOBY76GJ6DHZF5G23U4IYTOFB5SMQW5TW3YITYH227PV2FGA, seg_pos={0 0}, upload=49960]: Done uploading piece. err: ecclient: upload failed (node:12iCB7c4ZfpqafpD1PtpbE6BfU81wJW65LmrPVJXtD58yeuoR9t, address:104.36.82.12:4166): protocol: rpc: dial tcp 104.36.82.12:4166: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:43.392698018 (52h0m33.330356517s) [noise=true , address=104.36.82.50:4027 , piece_id=OW7BODU5N7ZEEGE2FHSLNYVJT2Z25LY3J42TZEOR5SP2EWIN6X7A, seg_pos={0 0}, upload=49960]: Done uploading piece. err: ecclient: upload failed (node:1zEQhyuVxSHwdDUgDfcTog2W3oLBx4z9bYUMHpaaoLU1BTU9DQ, address:104.36.82.50:4027): protocol: write tcp 188.166.225.170:37784->104.36.82.50:4027: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:43.392715381 (52h0m33.33037387s) [noise=true , address=104.36.82.43:4856 , piece_id=2SINBLHGRUOWUO4XHBNBG45YE3PW7RNF3QSNVCQXNCFEFHRBFNJQ, seg_pos={0 0}, upload=49960]: Done uploading piece. err: ecclient: upload failed (node:12bQxwBfPEGtmMp6mtnWKsoZt3eMEjLdyYFahETWGHeKJ7BhkiZ, address:104.36.82.43:4856): protocol: rpc: dial tcp 104.36.82.43:4856: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:43.392730059 (52h0m33.330388558s) [noise=true , address=104.36.82.10:4082 , piece_id=45K4BW277ZSGXJYQDA66OMEBVDC7DAWSOJAPX42HANNGRZN35KHQ, seg_pos={0 0}, upload=49960]: Done uploading piece. err: ecclient: upload failed (node:12DKsNCtqxLvpJwLszdLs74V3eb8NH2VAV9V3bngzUczawmNoLK, address:104.36.82.10:4082): protocol: write tcp 188.166.225.170:46638->104.36.82.10:4082: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:43.392753424 (52h0m33.330411923s) [seg_pos={0 0}, upload=49960]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil> Aug 1 21:57:43.624550997 (52h0m33.562209566s) [upload=49960]: Done uploading segments... Aug 1 21:57:43.624582868 (52h0m33.562241407s) [upload=49960]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7) Duration: 6.013457939s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [....................................XXX.....................] duration: 5.571210987s FAIL (2025-08-01 22:12:24 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.571235754s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 7 out of 60 is failed (threshold=3) [.................................XXXXXXX....................] duration: 5.83281983s FAIL (2025-08-01 22:12:14 UTC) orbiter warning: Operation: DownloadObject uplink: object not found ("orbiter_us_0/4M") Duration: 5.832855168s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [..................................XX.X.X....................] duration: 6.152962759s FAIL (2025-08-01 22:12:13 UTC) orbiter warning: Operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 7NK72474VWE4E3YOBCDZ5O5ZOH2MSA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB): protocol: write tcp 188.166.225.170:38368->104.36.82.12:4151: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:56:50.418604753 (51h59m40.356263252s) [seg_pos={0 0}, upload=49961]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 1 21:56:50.418680879 (51h59m40.356339378s) [upload=49961]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 1 21:56:50.418685998 (51h59m40.356344487s) [upload=49961]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 1 21:56:50.726563872 (51h59m40.664222371s) [upload=49961]: Done uploading segments... Aug 1 21:56:50.726586586 (51h59m40.664245085s) [upload=49961]: Upload finished. err: <nil> Aug 1 21:57:28.852869669 (52h0m18.790528158s) [upload=49973]: Starting upload Aug 1 21:57:28.852942389 (52h0m18.790600888s) [upload=49973]: Done starting upload Aug 1 21:57:28.853007122 (52h0m18.790665611s) [upload=49973]: Uploading segments... Aug 1 21:57:28.853174002 (52h0m18.790832491s) [upload=49973]: Got next segment. Inline: false Aug 1 21:57:28.853208168 (52h0m18.790866657s) [upload=49973]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 1 21:57:28.853212666 (52h0m18.790871165s) [upload=49973]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 1 21:57:29.095087519 (52h0m19.032746008s) [seg_pos={0 0}, upload=49973]: Begin upload segment... Aug 1 21:57:29.095129960 (52h0m19.032788459s) [seg_pos={0 0}, upload=49973]: Done begin upload segment. Aug 1 21:57:29.095164997 (52h0m19.032823496s) [noise=true , address=77.90.31.21:4878 , piece_id=BLNTYBOAIZWISSUJYK6BSBPTSNL6QWLGEN6AP6P4NPDVSURYG5AA, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.095171800 (52h0m19.032830289s) [noise=true , address=104.36.82.6:4875 , piece_id=CS5AMB7CIBQWWOU57Z3RCCSNVKIPE4B5QWU5YIQYI2UCWH7PAPVA, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.095510027 (52h0m19.033168526s) [noise=true , address=77.90.31.18:4806 , piece_id=52B6CAHGLFVTFIDQHHHAWKCBP2WJJY7OTTGRZZ7CEUH3HDA43Q6A, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.095544193 (52h0m19.033202692s) [noise=true , address=77.90.31.10:4518 , piece_id=WUOVXUSDG6YQTDPWG4YO7VLJNZZVLH6VRGMWOJABG2EJRU7XHP5A, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.095774283 (52h0m19.033432782s) [noise=true , address=104.36.82.7:4909 , piece_id=QHJCVBT6KTU34K2KWFNACWYVVE3WJ2RU76BHVSKL4HLCZEFXZYPA, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.095819880 (52h0m19.033478379s) [noise=true , address=45.91.116.12:4225 , piece_id=PBQPLA5ZYRFPSKHDKCEFOEBS5T3GYMP2FLSFDQWJLVY756KPLJNQ, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.096021656 (52h0m19.033680155s) [noise=true , address=45.91.116.13:4153 , piece_id=PMLKMZO66KXF3ZONO5ZZOQZICFK5XNGL6MN5FFBKICVFU5AWU6PQ, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.096069618 (52h0m19.033728117s) [noise=true , address=45.91.116.9:4262 , piece_id=A3F4L3RR6WPFMBFFKLATNGG3CASBIPWQP5O3G5URFNTXBE3ZW3VA, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.096348041 (52h0m19.03400654s) [noise=true , address=104.36.82.5:4840 , piece_id=UYVKVFCJJP7XZYRIVPS4ABGEMDUMNR7YW4QGG5ZFTFCGNCJO3CSQ, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.096354803 (52h0m19.034013292s) [noise=true , address=77.90.31.12:4590 , piece_id=D6FUBIFCQJSDRGWEP5GG34AQPNB3UROCYOATD6SRTJ7KAYVMFCOA, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.096608579 (52h0m19.034267078s) [noise=true , address=77.90.31.16:4733 , piece_id=DMASMGOUATLPEKCGLHKNNIMMFYVVGSULNUOMJU6RJY3ULBDOY6KQ, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.096830774 (52h0m19.034489273s) [noise=true , address=104.36.82.4:4802 , piece_id=CLM2G33M6AK5HIESRW6LZY33NKHIR2PGITWJ23NPILRCA6UYIEHA, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.097107233 (52h0m19.034765732s) [noise=true , address=77.90.31.19:4843 , piece_id=UFONZYHL7EAFYE5PRW32RKUVQPMZPXBEFLLIFJMAB2TXJ3C6KKNA, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.097324328 (52h0m19.034982827s) [noise=true , address=77.90.31.17:4771 , piece_id=Y3WSALJX23CWPPYQ3XYMCFOSBLBX2E2JQCCS7EOEYNBWY4CDRT3Q, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.098682446 (52h0m19.036340945s) [noise=true , address=77.90.31.15:4698 , piece_id=TN4OTPUD47FD6WOLZBEHNDI43WY6QVO5JK2JNJXNVM2IT6XMGERQ, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.099041603 (52h0m19.036700102s) [noise=true , address=45.91.116.11:4118 , piece_id=AKZIOAR6NRNO7JWCQZIKNSL7IUH5S4IG3KPG3IY2FAKG74DH3E3Q, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:30.718412255 (52h0m20.656070754s) [upload=49973]: Next returned nil segment Aug 1 21:57:30.718429869 (52h0m20.656088368s) [upload=49973]: Waiting for error group managing segments... Aug 1 21:57:31.107386675 (52h0m21.045045164s) [noise=true , address=45.91.116.13:4153 , piece_id=PMLKMZO66KXF3ZONO5ZZOQZICFK5XNGL6MN5FFBKICVFU5AWU6PQ, seg_pos={0 0}, upload=49973]: Done uploading piece. err: <nil> Aug 1 21:57:31.108529412 (52h0m21.046187911s) [noise=true , address=77.90.31.12:4590 , piece_id=D6FUBIFCQJSDRGWEP5GG34AQPNB3UROCYOATD6SRTJ7KAYVMFCOA, seg_pos={0 0}, upload=49973]: Done uploading piece. err: <nil> Aug 1 21:57:31.109975858 (52h0m21.047634347s) [noise=true , address=77.90.31.15:4698 , piece_id=TN4OTPUD47FD6WOLZBEHNDI43WY6QVO5JK2JNJXNVM2IT6XMGERQ, seg_pos={0 0}, upload=49973]: Done uploading piece. err: <nil> Aug 1 21:57:31.115273013 (52h0m21.052931502s) [noise=true , address=77.90.31.10:4518 , piece_id=WUOVXUSDG6YQTDPWG4YO7VLJNZZVLH6VRGMWOJABG2EJRU7XHP5A, seg_pos={0 0}, upload=49973]: Done uploading piece. err: <nil> Aug 1 21:57:31.116767281 (52h0m21.054425781s) [noise=true , address=77.90.31.18:4806 , piece_id=52B6CAHGLFVTFIDQHHHAWKCBP2WJJY7OTTGRZZ7CEUH3HDA43Q6A, seg_pos={0 0}, upload=49973]: Done uploading piece. err: <nil> Aug 1 21:57:31.201146107 (52h0m21.138804606s) [noise=true , address=77.90.31.16:4733 , piece_id=DMASMGOUATLPEKCGLHKNNIMMFYVVGSULNUOMJU6RJY3ULBDOY6KQ, seg_pos={0 0}, upload=49973]: Done uploading piece. err: <nil> Aug 1 21:57:31.201833732 (52h0m21.139492231s) [noise=true , address=77.90.31.19:4843 , piece_id=UFONZYHL7EAFYE5PRW32RKUVQPMZPXBEFLLIFJMAB2TXJ3C6KKNA, seg_pos={0 0}, upload=49973]: Done uploading piece. err: <nil> Aug 1 21:57:31.211007613 (52h0m21.148666112s) [noise=true , address=45.91.116.9:4262 , piece_id=A3F4L3RR6WPFMBFFKLATNGG3CASBIPWQP5O3G5URFNTXBE3ZW3VA, seg_pos={0 0}, upload=49973]: Done uploading piece. err: <nil> Aug 1 21:57:31.215929169 (52h0m21.153587668s) [noise=true , address=77.90.31.17:4771 , piece_id=Y3WSALJX23CWPPYQ3XYMCFOSBLBX2E2JQCCS7EOEYNBWY4CDRT3Q, seg_pos={0 0}, upload=49973]: Done uploading piece. err: <nil> Aug 1 21:57:31.222973975 (52h0m21.160632474s) [noise=true , address=77.90.31.21:4878 , piece_id=BLNTYBOAIZWISSUJYK6BSBPTSNL6QWLGEN6AP6P4NPDVSURYG5AA, seg_pos={0 0}, upload=49973]: Done uploading piece. err: <nil> Aug 1 21:57:31.234392441 (52h0m21.17205094s) [noise=true , address=45.91.116.12:4225 , piece_id=PBQPLA5ZYRFPSKHDKCEFOEBS5T3GYMP2FLSFDQWJLVY756KPLJNQ, seg_pos={0 0}, upload=49973]: Done uploading piece. err: <nil> Aug 1 21:57:31.284794305 (52h0m21.222452804s) [noise=true , address=45.91.116.11:4118 , piece_id=AKZIOAR6NRNO7JWCQZIKNSL7IUH5S4IG3KPG3IY2FAKG74DH3E3Q, seg_pos={0 0}, upload=49973]: Done uploading piece. err: <nil> Aug 1 21:57:58.202420932 (52h0m48.140079471s) [noise=true , address=104.36.82.6:4875 , piece_id=CS5AMB7CIBQWWOU57Z3RCCSNVKIPE4B5QWU5YIQYI2UCWH7PAPVA, seg_pos={0 0}, upload=49973]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:104.36.82.6:4875): protocol: write tcp 188.166.225.170:40166->104.36.82.6:4875: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:58.202441131 (52h0m48.14009963s) [noise=true , address=104.36.82.5:4840 , piece_id=UYVKVFCJJP7XZYRIVPS4ABGEMDUMNR7YW4QGG5ZFTFCGNCJO3CSQ, seg_pos={0 0}, upload=49973]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:104.36.82.5:4840): protocol: rpc: dial tcp 104.36.82.5:4840: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:58.202448445 (52h0m48.140106954s) [noise=true , address=104.36.82.7:4909 , piece_id=QHJCVBT6KTU34K2KWFNACWYVVE3WJ2RU76BHVSKL4HLCZEFXZYPA, seg_pos={0 0}, upload=49973]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:104.36.82.7:4909): protocol: rpc: dial tcp 104.36.82.7:4909: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:58.202496487 (52h0m48.140155026s) [noise=true , address=104.36.82.4:4802 , piece_id=CLM2G33M6AK5HIESRW6LZY33NKHIR2PGITWJ23NPILRCA6UYIEHA, seg_pos={0 0}, upload=49973]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: write tcp 188.166.225.170:47696->104.36.82.4:4802: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:58.202524240 (52h0m48.140182739s) [seg_pos={0 0}, upload=49973]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Aug 1 21:57:58.435983932 (52h0m48.373642431s) [upload=49973]: Done uploading segments... Aug 1 21:57:58.436007667 (52h0m48.373666166s) [upload=49973]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.152987547s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 5 out of 60 is failed (threshold=3) [....................X............X.....X.......X.....X......] duration: 2.299972579s FAIL (2025-08-01 22:11:55 UTC) orbiter warning: Operation: ListObjects unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/1K Duration: 2.300000143s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [...................................XX.X.X...................] duration: 5.93382571s FAIL (2025-08-01 22:11:54 UTC) orbiter warning: Operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 7NK72474VWE4E3YOBCDZ5O5ZOH2MSA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB): protocol: write tcp 188.166.225.170:38368->104.36.82.12:4151: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:56:50.418604753 (51h59m40.356263252s) [seg_pos={0 0}, upload=49961]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 1 21:56:50.418680879 (51h59m40.356339378s) [upload=49961]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 1 21:56:50.418685998 (51h59m40.356344487s) [upload=49961]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 1 21:56:50.726563872 (51h59m40.664222371s) [upload=49961]: Done uploading segments... Aug 1 21:56:50.726586586 (51h59m40.664245085s) [upload=49961]: Upload finished. err: <nil> Aug 1 21:57:28.852869669 (52h0m18.790528158s) [upload=49973]: Starting upload Aug 1 21:57:28.852942389 (52h0m18.790600888s) [upload=49973]: Done starting upload Aug 1 21:57:28.853007122 (52h0m18.790665611s) [upload=49973]: Uploading segments... Aug 1 21:57:28.853174002 (52h0m18.790832491s) [upload=49973]: Got next segment. Inline: false Aug 1 21:57:28.853208168 (52h0m18.790866657s) [upload=49973]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 1 21:57:28.853212666 (52h0m18.790871165s) [upload=49973]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 1 21:57:29.095087519 (52h0m19.032746008s) [seg_pos={0 0}, upload=49973]: Begin upload segment... Aug 1 21:57:29.095129960 (52h0m19.032788459s) [seg_pos={0 0}, upload=49973]: Done begin upload segment. Aug 1 21:57:29.095164997 (52h0m19.032823496s) [noise=true , address=77.90.31.21:4878 , piece_id=BLNTYBOAIZWISSUJYK6BSBPTSNL6QWLGEN6AP6P4NPDVSURYG5AA, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.095171800 (52h0m19.032830289s) [noise=true , address=104.36.82.6:4875 , piece_id=CS5AMB7CIBQWWOU57Z3RCCSNVKIPE4B5QWU5YIQYI2UCWH7PAPVA, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.095510027 (52h0m19.033168526s) [noise=true , address=77.90.31.18:4806 , piece_id=52B6CAHGLFVTFIDQHHHAWKCBP2WJJY7OTTGRZZ7CEUH3HDA43Q6A, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.095544193 (52h0m19.033202692s) [noise=true , address=77.90.31.10:4518 , piece_id=WUOVXUSDG6YQTDPWG4YO7VLJNZZVLH6VRGMWOJABG2EJRU7XHP5A, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.095774283 (52h0m19.033432782s) [noise=true , address=104.36.82.7:4909 , piece_id=QHJCVBT6KTU34K2KWFNACWYVVE3WJ2RU76BHVSKL4HLCZEFXZYPA, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.095819880 (52h0m19.033478379s) [noise=true , address=45.91.116.12:4225 , piece_id=PBQPLA5ZYRFPSKHDKCEFOEBS5T3GYMP2FLSFDQWJLVY756KPLJNQ, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.096021656 (52h0m19.033680155s) [noise=true , address=45.91.116.13:4153 , piece_id=PMLKMZO66KXF3ZONO5ZZOQZICFK5XNGL6MN5FFBKICVFU5AWU6PQ, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.096069618 (52h0m19.033728117s) [noise=true , address=45.91.116.9:4262 , piece_id=A3F4L3RR6WPFMBFFKLATNGG3CASBIPWQP5O3G5URFNTXBE3ZW3VA, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.096348041 (52h0m19.03400654s) [noise=true , address=104.36.82.5:4840 , piece_id=UYVKVFCJJP7XZYRIVPS4ABGEMDUMNR7YW4QGG5ZFTFCGNCJO3CSQ, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.096354803 (52h0m19.034013292s) [noise=true , address=77.90.31.12:4590 , piece_id=D6FUBIFCQJSDRGWEP5GG34AQPNB3UROCYOATD6SRTJ7KAYVMFCOA, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.096608579 (52h0m19.034267078s) [noise=true , address=77.90.31.16:4733 , piece_id=DMASMGOUATLPEKCGLHKNNIMMFYVVGSULNUOMJU6RJY3ULBDOY6KQ, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.096830774 (52h0m19.034489273s) [noise=true , address=104.36.82.4:4802 , piece_id=CLM2G33M6AK5HIESRW6LZY33NKHIR2PGITWJ23NPILRCA6UYIEHA, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.097107233 (52h0m19.034765732s) [noise=true , address=77.90.31.19:4843 , piece_id=UFONZYHL7EAFYE5PRW32RKUVQPMZPXBEFLLIFJMAB2TXJ3C6KKNA, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.097324328 (52h0m19.034982827s) [noise=true , address=77.90.31.17:4771 , piece_id=Y3WSALJX23CWPPYQ3XYMCFOSBLBX2E2JQCCS7EOEYNBWY4CDRT3Q, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.098682446 (52h0m19.036340945s) [noise=true , address=77.90.31.15:4698 , piece_id=TN4OTPUD47FD6WOLZBEHNDI43WY6QVO5JK2JNJXNVM2IT6XMGERQ, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.099041603 (52h0m19.036700102s) [noise=true , address=45.91.116.11:4118 , piece_id=AKZIOAR6NRNO7JWCQZIKNSL7IUH5S4IG3KPG3IY2FAKG74DH3E3Q, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:30.718412255 (52h0m20.656070754s) [upload=49973]: Next returned nil segment Aug 1 21:57:30.718429869 (52h0m20.656088368s) [upload=49973]: Waiting for error group managing segments... Aug 1 21:57:31.107386675 (52h0m21.045045164s) [noise=true , address=45.91.116.13:4153 , piece_id=PMLKMZO66KXF3ZONO5ZZOQZICFK5XNGL6MN5FFBKICVFU5AWU6PQ, seg_pos={0 0}, upload=49973]: Done uploading piece. err: <nil> Aug 1 21:57:31.108529412 (52h0m21.046187911s) [noise=true , address=77.90.31.12:4590 , piece_id=D6FUBIFCQJSDRGWEP5GG34AQPNB3UROCYOATD6SRTJ7KAYVMFCOA, seg_pos={0 0}, upload=49973]: Done uploading piece. err: <nil> Aug 1 21:57:31.109975858 (52h0m21.047634347s) [noise=true , address=77.90.31.15:4698 , piece_id=TN4OTPUD47FD6WOLZBEHNDI43WY6QVO5JK2JNJXNVM2IT6XMGERQ, seg_pos={0 0}, upload=49973]: Done uploading piece. err: <nil> Aug 1 21:57:31.115273013 (52h0m21.052931502s) [noise=true , address=77.90.31.10:4518 , piece_id=WUOVXUSDG6YQTDPWG4YO7VLJNZZVLH6VRGMWOJABG2EJRU7XHP5A, seg_pos={0 0}, upload=49973]: Done uploading piece. err: <nil> Aug 1 21:57:31.116767281 (52h0m21.054425781s) [noise=true , address=77.90.31.18:4806 , piece_id=52B6CAHGLFVTFIDQHHHAWKCBP2WJJY7OTTGRZZ7CEUH3HDA43Q6A, seg_pos={0 0}, upload=49973]: Done uploading piece. err: <nil> Aug 1 21:57:31.201146107 (52h0m21.138804606s) [noise=true , address=77.90.31.16:4733 , piece_id=DMASMGOUATLPEKCGLHKNNIMMFYVVGSULNUOMJU6RJY3ULBDOY6KQ, seg_pos={0 0}, upload=49973]: Done uploading piece. err: <nil> Aug 1 21:57:31.201833732 (52h0m21.139492231s) [noise=true , address=77.90.31.19:4843 , piece_id=UFONZYHL7EAFYE5PRW32RKUVQPMZPXBEFLLIFJMAB2TXJ3C6KKNA, seg_pos={0 0}, upload=49973]: Done uploading piece. err: <nil> Aug 1 21:57:31.211007613 (52h0m21.148666112s) [noise=true , address=45.91.116.9:4262 , piece_id=A3F4L3RR6WPFMBFFKLATNGG3CASBIPWQP5O3G5URFNTXBE3ZW3VA, seg_pos={0 0}, upload=49973]: Done uploading piece. err: <nil> Aug 1 21:57:31.215929169 (52h0m21.153587668s) [noise=true , address=77.90.31.17:4771 , piece_id=Y3WSALJX23CWPPYQ3XYMCFOSBLBX2E2JQCCS7EOEYNBWY4CDRT3Q, seg_pos={0 0}, upload=49973]: Done uploading piece. err: <nil> Aug 1 21:57:31.222973975 (52h0m21.160632474s) [noise=true , address=77.90.31.21:4878 , piece_id=BLNTYBOAIZWISSUJYK6BSBPTSNL6QWLGEN6AP6P4NPDVSURYG5AA, seg_pos={0 0}, upload=49973]: Done uploading piece. err: <nil> Aug 1 21:57:31.234392441 (52h0m21.17205094s) [noise=true , address=45.91.116.12:4225 , piece_id=PBQPLA5ZYRFPSKHDKCEFOEBS5T3GYMP2FLSFDQWJLVY756KPLJNQ, seg_pos={0 0}, upload=49973]: Done uploading piece. err: <nil> Aug 1 21:57:31.284794305 (52h0m21.222452804s) [noise=true , address=45.91.116.11:4118 , piece_id=AKZIOAR6NRNO7JWCQZIKNSL7IUH5S4IG3KPG3IY2FAKG74DH3E3Q, seg_pos={0 0}, upload=49973]: Done uploading piece. err: <nil> Aug 1 21:57:58.202420932 (52h0m48.140079471s) [noise=true , address=104.36.82.6:4875 , piece_id=CS5AMB7CIBQWWOU57Z3RCCSNVKIPE4B5QWU5YIQYI2UCWH7PAPVA, seg_pos={0 0}, upload=49973]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:104.36.82.6:4875): protocol: write tcp 188.166.225.170:40166->104.36.82.6:4875: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:58.202441131 (52h0m48.14009963s) [noise=true , address=104.36.82.5:4840 , piece_id=UYVKVFCJJP7XZYRIVPS4ABGEMDUMNR7YW4QGG5ZFTFCGNCJO3CSQ, seg_pos={0 0}, upload=49973]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:104.36.82.5:4840): protocol: rpc: dial tcp 104.36.82.5:4840: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:58.202448445 (52h0m48.140106954s) [noise=true , address=104.36.82.7:4909 , piece_id=QHJCVBT6KTU34K2KWFNACWYVVE3WJ2RU76BHVSKL4HLCZEFXZYPA, seg_pos={0 0}, upload=49973]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:104.36.82.7:4909): protocol: rpc: dial tcp 104.36.82.7:4909: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:58.202496487 (52h0m48.140155026s) [noise=true , address=104.36.82.4:4802 , piece_id=CLM2G33M6AK5HIESRW6LZY33NKHIR2PGITWJ23NPILRCA6UYIEHA, seg_pos={0 0}, upload=49973]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: write tcp 188.166.225.170:47696->104.36.82.4:4802: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:58.202524240 (52h0m48.140182739s) [seg_pos={0 0}, upload=49973]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Aug 1 21:57:58.435983932 (52h0m48.373642431s) [upload=49973]: Done uploading segments... Aug 1 21:57:58.436007667 (52h0m48.373666166s) [upload=49973]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.933861878s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [....................................XXXX....................] duration: 5.988781457s FAIL (2025-08-01 22:11:52 UTC) orbiter warning: Operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 7) ======LOGS====== K4BW277ZSGXJYQDA66OMEBVDC7DAWSOJAPX42HANNGRZN35KHQ, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.481101510 (51h59m34.418760009s) [noise=true , address=104.36.82.5:4842 , piece_id=LV7YAOMPE2AV7TF2WPWR7QJNLT4JZAIPSVZN3E23U5KDBNKZFD6A, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.481422003 (51h59m34.419080502s) [noise=true , address=79.127.205.235:4683 , piece_id=3QHDNAPG4GRLJFVVNJNFXNGKTGKUEMWWUBMW62HHE5N5LJDO4KWA, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.482030897 (51h59m34.419689396s) [noise=true , address=79.127.219.51:4213 , piece_id=L2WSPI5BEIFLGQKJIK4EWXFQXNQRQ7HVKPOWFYRPPUGNAL5DVYWA, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.482285215 (51h59m34.419943704s) [noise=true , address=104.36.82.14:4251 , piece_id=O7NADU347E2Q26TPD3IUCK6WE5UVRITC6ALZHBE5SWYTKWXFBMXA, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.482518370 (51h59m34.420176859s) [noise=true , address=79.127.220.98:4402 , piece_id=DVLSYRFFZ6UT3QRSVAN7AMXVYZNI6PWG2IITHLRUUJATJVK3QOGA, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.482724304 (51h59m34.420382803s) [noise=true , address=79.127.201.214:4297 , piece_id=MPTMGVEYC4SATIN4ANCR2ELVRXHYWRDLMHWQW7HM6UOIV2N6R7ZA, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.482928555 (51h59m34.420587054s) [noise=true , address=104.36.82.37:4680 , piece_id=X4PDUE4WXUUWX6JQVO4NWPQCBMHMHC5YHPIHDQVYYOI2FXV2GVJQ, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.483134960 (51h59m34.420793449s) [noise=true , address=104.36.82.50:4027 , piece_id=OW7BODU5N7ZEEGE2FHSLNYVJT2Z25LY3J42TZEOR5SP2EWIN6X7A, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.483338480 (51h59m34.420996979s) [noise=true , address=104.36.82.55:4204 , piece_id=R5CYXWAVV4ZEMCV7LF4D4XDC4G6KLVBONAH2TH5EIEVL7QHUTRWQ, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.483582847 (51h59m34.421241356s) [noise=true , address=104.36.82.11:4134 , piece_id=RMTNNNC2VD7MGWJDAMT6ZCF3GR43SHANFVAGJ6EHVH7WO5XA6VTA, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.483833737 (51h59m34.421492236s) [noise=true , address=104.36.82.43:4856 , piece_id=2SINBLHGRUOWUO4XHBNBG45YE3PW7RNF3QSNVCQXNCFEFHRBFNJQ, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.484021896 (51h59m34.421680385s) [noise=true , address=104.36.82.52:4101 , piece_id=XYPLQM7QITCCX7T4NKIXP5MZ7I6F24EJJTG37K4NPOBKTUFHULRQ, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.484301662 (51h59m34.421960182s) [noise=true , address=79.127.201.215:4352 , piece_id=I6BRYI5XIACAAAAAE7HQOBCF4OTALMOVPPNBWQCRSH2MM6WUUURQ, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.484712097 (51h59m34.422370596s) [noise=true , address=64.49.13.138:4484 , piece_id=DZ5Y4HTS6BKW2DJ5CVOB64TW4SXB6DNCJEGFUJPLAUXUEHRZRTUA, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:45.576021470 (51h59m35.513679959s) [upload=49960]: Next returned nil segment Aug 1 21:56:45.576036609 (51h59m35.513695098s) [upload=49960]: Waiting for error group managing segments... Aug 1 21:56:45.860378776 (51h59m35.798037275s) [noise=true , address=64.49.13.138:4484 , piece_id=DZ5Y4HTS6BKW2DJ5CVOB64TW4SXB6DNCJEGFUJPLAUXUEHRZRTUA, seg_pos={0 0}, upload=49960]: Done uploading piece. err: <nil> Aug 1 21:56:45.884184188 (51h59m35.821842687s) [noise=true , address=79.127.205.235:4683 , piece_id=3QHDNAPG4GRLJFVVNJNFXNGKTGKUEMWWUBMW62HHE5N5LJDO4KWA, seg_pos={0 0}, upload=49960]: Done uploading piece. err: <nil> Aug 1 21:56:46.145493612 (51h59m36.083152101s) [noise=true , address=79.127.220.98:4402 , piece_id=DVLSYRFFZ6UT3QRSVAN7AMXVYZNI6PWG2IITHLRUUJATJVK3QOGA, seg_pos={0 0}, upload=49960]: Done uploading piece. err: <nil> Aug 1 21:56:46.257505313 (51h59m36.195163802s) [noise=true , address=79.127.220.97:4371 , piece_id=5WGPRFKQETIHTSMA42AM2C3UGCFYJ3DMMPAI2L7DD6E2P37KJKNA, seg_pos={0 0}, upload=49960]: Done uploading piece. err: <nil> Aug 1 21:56:46.334426181 (51h59m36.27208468s) [noise=true , address=79.127.201.215:4352 , piece_id=I6BRYI5XIACAAAAAE7HQOBCF4OTALMOVPPNBWQCRSH2MM6WUUURQ, seg_pos={0 0}, upload=49960]: Done uploading piece. err: <nil> Aug 1 21:56:46.386119275 (51h59m36.323777774s) [noise=true , address=79.127.201.214:4297 , piece_id=MPTMGVEYC4SATIN4ANCR2ELVRXHYWRDLMHWQW7HM6UOIV2N6R7ZA, seg_pos={0 0}, upload=49960]: Done uploading piece. err: <nil> Aug 1 21:56:46.395362048 (51h59m36.333020547s) [noise=true , address=79.127.219.51:4213 , piece_id=L2WSPI5BEIFLGQKJIK4EWXFQXNQRQ7HVKPOWFYRPPUGNAL5DVYWA, seg_pos={0 0}, upload=49960]: Done uploading piece. err: <nil> Aug 1 21:57:43.392434383 (52h0m33.330092872s) [noise=true , address=104.36.82.37:4680 , piece_id=X4PDUE4WXUUWX6JQVO4NWPQCBMHMHC5YHPIHDQVYYOI2FXV2GVJQ, seg_pos={0 0}, upload=49960]: Done uploading piece. err: ecclient: upload failed (node:1bXAqpxGhUvzwYEDGBqYRhZdTeMJcV3TYfngkKTCumpt9Nupi3, address:104.36.82.37:4680): protocol: rpc: dial tcp 104.36.82.37:4680: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:43.392477967 (52h0m33.330136466s) [noise=true , address=104.36.82.11:4134 , piece_id=RMTNNNC2VD7MGWJDAMT6ZCF3GR43SHANFVAGJ6EHVH7WO5XA6VTA, seg_pos={0 0}, upload=49960]: Done uploading piece. err: ecclient: upload failed (node:12R3Z5AgkBmWXMv5XgRyfcyqkG2gmyJs9ogT24kcLy6nuB9bzUG, address:104.36.82.11:4134): protocol: rpc: dial tcp 104.36.82.11:4134: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:43.392514086 (52h0m33.330172585s) [noise=true , address=104.36.82.55:4204 , piece_id=R5CYXWAVV4ZEMCV7LF4D4XDC4G6KLVBONAH2TH5EIEVL7QHUTRWQ, seg_pos={0 0}, upload=49960]: Done uploading piece. err: ecclient: upload failed (node:1rBZH6aLHsnWh22uNzaMiMwmLJCDMn4bYbNC5jRG7KCEuRfqpH, address:104.36.82.55:4204): protocol: write tcp 188.166.225.170:48550->104.36.82.55:4204: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:43.392546537 (52h0m33.330205027s) [noise=true , address=104.36.82.59:4327 , piece_id=OZDEYIY6B4WYZULRDRL2UBOJH2JQNY3YWSITZKWDO7GTT72AN5NQ, seg_pos={0 0}, upload=49960]: Done uploading piece. err: ecclient: upload failed (node:1V4C6AWGbyYQWfinoSj7iKovHeTJ9hqypyC5U9rZ8pveXBBfCY, address:104.36.82.59:4327): protocol: rpc: dial tcp 104.36.82.59:4327: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:43.392579040 (52h0m33.33023754s) [noise=true , address=104.36.82.5:4842 , piece_id=LV7YAOMPE2AV7TF2WPWR7QJNLT4JZAIPSVZN3E23U5KDBNKZFD6A, seg_pos={0 0}, upload=49960]: Done uploading piece. err: ecclient: upload failed (node:12q4VdT5BcjUb16PBYHK6zrnW8dqENKrcGx9zdLtNefDVB1MUSn, address:104.36.82.5:4842): protocol: context deadline exceeded; context deadline exceeded Aug 1 21:57:43.392627953 (52h0m33.330286442s) [noise=true , address=104.36.82.52:4101 , piece_id=XYPLQM7QITCCX7T4NKIXP5MZ7I6F24EJJTG37K4NPOBKTUFHULRQ, seg_pos={0 0}, upload=49960]: Done uploading piece. err: ecclient: upload failed (node:12WWXeBjFZR9PzciNfs35fFcx87N9GwDpMsPHnksFDugtaBGWp4, address:104.36.82.52:4101): protocol: rpc: dial tcp 104.36.82.52:4101: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:43.392653192 (52h0m33.330311691s) [noise=true , address=104.36.82.14:4251 , piece_id=O7NADU347E2Q26TPD3IUCK6WE5UVRITC6ALZHBE5SWYTKWXFBMXA, seg_pos={0 0}, upload=49960]: Done uploading piece. err: ecclient: upload failed (node:12btf6Z38aTMx7MaBxhUyZtFqUPgeMtP2aU8Y6FbV8zwu57zArd, address:104.36.82.14:4251): protocol: rpc: dial tcp 104.36.82.14:4251: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:43.392670836 (52h0m33.330329335s) [noise=true , address=104.36.82.12:4166 , piece_id=6ZIWCOBY76GJ6DHZF5G23U4IYTOFB5SMQW5TW3YITYH227PV2FGA, seg_pos={0 0}, upload=49960]: Done uploading piece. err: ecclient: upload failed (node:12iCB7c4ZfpqafpD1PtpbE6BfU81wJW65LmrPVJXtD58yeuoR9t, address:104.36.82.12:4166): protocol: rpc: dial tcp 104.36.82.12:4166: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:43.392698018 (52h0m33.330356517s) [noise=true , address=104.36.82.50:4027 , piece_id=OW7BODU5N7ZEEGE2FHSLNYVJT2Z25LY3J42TZEOR5SP2EWIN6X7A, seg_pos={0 0}, upload=49960]: Done uploading piece. err: ecclient: upload failed (node:1zEQhyuVxSHwdDUgDfcTog2W3oLBx4z9bYUMHpaaoLU1BTU9DQ, address:104.36.82.50:4027): protocol: write tcp 188.166.225.170:37784->104.36.82.50:4027: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:43.392715381 (52h0m33.33037387s) [noise=true , address=104.36.82.43:4856 , piece_id=2SINBLHGRUOWUO4XHBNBG45YE3PW7RNF3QSNVCQXNCFEFHRBFNJQ, seg_pos={0 0}, upload=49960]: Done uploading piece. err: ecclient: upload failed (node:12bQxwBfPEGtmMp6mtnWKsoZt3eMEjLdyYFahETWGHeKJ7BhkiZ, address:104.36.82.43:4856): protocol: rpc: dial tcp 104.36.82.43:4856: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:43.392730059 (52h0m33.330388558s) [noise=true , address=104.36.82.10:4082 , piece_id=45K4BW277ZSGXJYQDA66OMEBVDC7DAWSOJAPX42HANNGRZN35KHQ, seg_pos={0 0}, upload=49960]: Done uploading piece. err: ecclient: upload failed (node:12DKsNCtqxLvpJwLszdLs74V3eb8NH2VAV9V3bngzUczawmNoLK, address:104.36.82.10:4082): protocol: write tcp 188.166.225.170:46638->104.36.82.10:4082: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:43.392753424 (52h0m33.330411923s) [seg_pos={0 0}, upload=49960]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil> Aug 1 21:57:43.624550997 (52h0m33.562209566s) [upload=49960]: Done uploading segments... Aug 1 21:57:43.624582868 (52h0m33.562241407s) [upload=49960]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7) Duration: 5.988798371s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 6.215768726s FAIL (2025-08-01 22:11:52 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.215788996s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 7 out of 60 is failed (threshold=3) [..................................XXXXXXX...................] duration: 6.207637039s FAIL (2025-08-01 22:11:24 UTC) orbiter warning: Operation: DownloadObject uplink: object not found ("orbiter_us_0/4M") Duration: 6.2076685s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [.....................................XXX....................] duration: 5.563279801s FAIL (2025-08-01 22:11:21 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.563293368s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [....................................XX.X.X..................] duration: 5.870732866s FAIL (2025-08-01 22:11:11 UTC) orbiter warning: Operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 7NK72474VWE4E3YOBCDZ5O5ZOH2MSA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB): protocol: write tcp 188.166.225.170:38368->104.36.82.12:4151: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:56:50.418604753 (51h59m40.356263252s) [seg_pos={0 0}, upload=49961]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 1 21:56:50.418680879 (51h59m40.356339378s) [upload=49961]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 1 21:56:50.418685998 (51h59m40.356344487s) [upload=49961]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 1 21:56:50.726563872 (51h59m40.664222371s) [upload=49961]: Done uploading segments... Aug 1 21:56:50.726586586 (51h59m40.664245085s) [upload=49961]: Upload finished. err: <nil> Aug 1 21:57:28.852869669 (52h0m18.790528158s) [upload=49973]: Starting upload Aug 1 21:57:28.852942389 (52h0m18.790600888s) [upload=49973]: Done starting upload Aug 1 21:57:28.853007122 (52h0m18.790665611s) [upload=49973]: Uploading segments... Aug 1 21:57:28.853174002 (52h0m18.790832491s) [upload=49973]: Got next segment. Inline: false Aug 1 21:57:28.853208168 (52h0m18.790866657s) [upload=49973]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 1 21:57:28.853212666 (52h0m18.790871165s) [upload=49973]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 1 21:57:29.095087519 (52h0m19.032746008s) [seg_pos={0 0}, upload=49973]: Begin upload segment... Aug 1 21:57:29.095129960 (52h0m19.032788459s) [seg_pos={0 0}, upload=49973]: Done begin upload segment. Aug 1 21:57:29.095164997 (52h0m19.032823496s) [noise=true , address=77.90.31.21:4878 , piece_id=BLNTYBOAIZWISSUJYK6BSBPTSNL6QWLGEN6AP6P4NPDVSURYG5AA, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.095171800 (52h0m19.032830289s) [noise=true , address=104.36.82.6:4875 , piece_id=CS5AMB7CIBQWWOU57Z3RCCSNVKIPE4B5QWU5YIQYI2UCWH7PAPVA, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.095510027 (52h0m19.033168526s) [noise=true , address=77.90.31.18:4806 , piece_id=52B6CAHGLFVTFIDQHHHAWKCBP2WJJY7OTTGRZZ7CEUH3HDA43Q6A, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.095544193 (52h0m19.033202692s) [noise=true , address=77.90.31.10:4518 , piece_id=WUOVXUSDG6YQTDPWG4YO7VLJNZZVLH6VRGMWOJABG2EJRU7XHP5A, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.095774283 (52h0m19.033432782s) [noise=true , address=104.36.82.7:4909 , piece_id=QHJCVBT6KTU34K2KWFNACWYVVE3WJ2RU76BHVSKL4HLCZEFXZYPA, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.095819880 (52h0m19.033478379s) [noise=true , address=45.91.116.12:4225 , piece_id=PBQPLA5ZYRFPSKHDKCEFOEBS5T3GYMP2FLSFDQWJLVY756KPLJNQ, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.096021656 (52h0m19.033680155s) [noise=true , address=45.91.116.13:4153 , piece_id=PMLKMZO66KXF3ZONO5ZZOQZICFK5XNGL6MN5FFBKICVFU5AWU6PQ, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.096069618 (52h0m19.033728117s) [noise=true , address=45.91.116.9:4262 , piece_id=A3F4L3RR6WPFMBFFKLATNGG3CASBIPWQP5O3G5URFNTXBE3ZW3VA, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.096348041 (52h0m19.03400654s) [noise=true , address=104.36.82.5:4840 , piece_id=UYVKVFCJJP7XZYRIVPS4ABGEMDUMNR7YW4QGG5ZFTFCGNCJO3CSQ, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.096354803 (52h0m19.034013292s) [noise=true , address=77.90.31.12:4590 , piece_id=D6FUBIFCQJSDRGWEP5GG34AQPNB3UROCYOATD6SRTJ7KAYVMFCOA, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.096608579 (52h0m19.034267078s) [noise=true , address=77.90.31.16:4733 , piece_id=DMASMGOUATLPEKCGLHKNNIMMFYVVGSULNUOMJU6RJY3ULBDOY6KQ, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.096830774 (52h0m19.034489273s) [noise=true , address=104.36.82.4:4802 , piece_id=CLM2G33M6AK5HIESRW6LZY33NKHIR2PGITWJ23NPILRCA6UYIEHA, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.097107233 (52h0m19.034765732s) [noise=true , address=77.90.31.19:4843 , piece_id=UFONZYHL7EAFYE5PRW32RKUVQPMZPXBEFLLIFJMAB2TXJ3C6KKNA, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.097324328 (52h0m19.034982827s) [noise=true , address=77.90.31.17:4771 , piece_id=Y3WSALJX23CWPPYQ3XYMCFOSBLBX2E2JQCCS7EOEYNBWY4CDRT3Q, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.098682446 (52h0m19.036340945s) [noise=true , address=77.90.31.15:4698 , piece_id=TN4OTPUD47FD6WOLZBEHNDI43WY6QVO5JK2JNJXNVM2IT6XMGERQ, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.099041603 (52h0m19.036700102s) [noise=true , address=45.91.116.11:4118 , piece_id=AKZIOAR6NRNO7JWCQZIKNSL7IUH5S4IG3KPG3IY2FAKG74DH3E3Q, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:30.718412255 (52h0m20.656070754s) [upload=49973]: Next returned nil segment Aug 1 21:57:30.718429869 (52h0m20.656088368s) [upload=49973]: Waiting for error group managing segments... Aug 1 21:57:31.107386675 (52h0m21.045045164s) [noise=true , address=45.91.116.13:4153 , piece_id=PMLKMZO66KXF3ZONO5ZZOQZICFK5XNGL6MN5FFBKICVFU5AWU6PQ, seg_pos={0 0}, upload=49973]: Done uploading piece. err: <nil> Aug 1 21:57:31.108529412 (52h0m21.046187911s) [noise=true , address=77.90.31.12:4590 , piece_id=D6FUBIFCQJSDRGWEP5GG34AQPNB3UROCYOATD6SRTJ7KAYVMFCOA, seg_pos={0 0}, upload=49973]: Done uploading piece. err: <nil> Aug 1 21:57:31.109975858 (52h0m21.047634347s) [noise=true , address=77.90.31.15:4698 , piece_id=TN4OTPUD47FD6WOLZBEHNDI43WY6QVO5JK2JNJXNVM2IT6XMGERQ, seg_pos={0 0}, upload=49973]: Done uploading piece. err: <nil> Aug 1 21:57:31.115273013 (52h0m21.052931502s) [noise=true , address=77.90.31.10:4518 , piece_id=WUOVXUSDG6YQTDPWG4YO7VLJNZZVLH6VRGMWOJABG2EJRU7XHP5A, seg_pos={0 0}, upload=49973]: Done uploading piece. err: <nil> Aug 1 21:57:31.116767281 (52h0m21.054425781s) [noise=true , address=77.90.31.18:4806 , piece_id=52B6CAHGLFVTFIDQHHHAWKCBP2WJJY7OTTGRZZ7CEUH3HDA43Q6A, seg_pos={0 0}, upload=49973]: Done uploading piece. err: <nil> Aug 1 21:57:31.201146107 (52h0m21.138804606s) [noise=true , address=77.90.31.16:4733 , piece_id=DMASMGOUATLPEKCGLHKNNIMMFYVVGSULNUOMJU6RJY3ULBDOY6KQ, seg_pos={0 0}, upload=49973]: Done uploading piece. err: <nil> Aug 1 21:57:31.201833732 (52h0m21.139492231s) [noise=true , address=77.90.31.19:4843 , piece_id=UFONZYHL7EAFYE5PRW32RKUVQPMZPXBEFLLIFJMAB2TXJ3C6KKNA, seg_pos={0 0}, upload=49973]: Done uploading piece. err: <nil> Aug 1 21:57:31.211007613 (52h0m21.148666112s) [noise=true , address=45.91.116.9:4262 , piece_id=A3F4L3RR6WPFMBFFKLATNGG3CASBIPWQP5O3G5URFNTXBE3ZW3VA, seg_pos={0 0}, upload=49973]: Done uploading piece. err: <nil> Aug 1 21:57:31.215929169 (52h0m21.153587668s) [noise=true , address=77.90.31.17:4771 , piece_id=Y3WSALJX23CWPPYQ3XYMCFOSBLBX2E2JQCCS7EOEYNBWY4CDRT3Q, seg_pos={0 0}, upload=49973]: Done uploading piece. err: <nil> Aug 1 21:57:31.222973975 (52h0m21.160632474s) [noise=true , address=77.90.31.21:4878 , piece_id=BLNTYBOAIZWISSUJYK6BSBPTSNL6QWLGEN6AP6P4NPDVSURYG5AA, seg_pos={0 0}, upload=49973]: Done uploading piece. err: <nil> Aug 1 21:57:31.234392441 (52h0m21.17205094s) [noise=true , address=45.91.116.12:4225 , piece_id=PBQPLA5ZYRFPSKHDKCEFOEBS5T3GYMP2FLSFDQWJLVY756KPLJNQ, seg_pos={0 0}, upload=49973]: Done uploading piece. err: <nil> Aug 1 21:57:31.284794305 (52h0m21.222452804s) [noise=true , address=45.91.116.11:4118 , piece_id=AKZIOAR6NRNO7JWCQZIKNSL7IUH5S4IG3KPG3IY2FAKG74DH3E3Q, seg_pos={0 0}, upload=49973]: Done uploading piece. err: <nil> Aug 1 21:57:58.202420932 (52h0m48.140079471s) [noise=true , address=104.36.82.6:4875 , piece_id=CS5AMB7CIBQWWOU57Z3RCCSNVKIPE4B5QWU5YIQYI2UCWH7PAPVA, seg_pos={0 0}, upload=49973]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:104.36.82.6:4875): protocol: write tcp 188.166.225.170:40166->104.36.82.6:4875: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:58.202441131 (52h0m48.14009963s) [noise=true , address=104.36.82.5:4840 , piece_id=UYVKVFCJJP7XZYRIVPS4ABGEMDUMNR7YW4QGG5ZFTFCGNCJO3CSQ, seg_pos={0 0}, upload=49973]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:104.36.82.5:4840): protocol: rpc: dial tcp 104.36.82.5:4840: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:58.202448445 (52h0m48.140106954s) [noise=true , address=104.36.82.7:4909 , piece_id=QHJCVBT6KTU34K2KWFNACWYVVE3WJ2RU76BHVSKL4HLCZEFXZYPA, seg_pos={0 0}, upload=49973]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:104.36.82.7:4909): protocol: rpc: dial tcp 104.36.82.7:4909: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:58.202496487 (52h0m48.140155026s) [noise=true , address=104.36.82.4:4802 , piece_id=CLM2G33M6AK5HIESRW6LZY33NKHIR2PGITWJ23NPILRCA6UYIEHA, seg_pos={0 0}, upload=49973]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: write tcp 188.166.225.170:47696->104.36.82.4:4802: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:58.202524240 (52h0m48.140182739s) [seg_pos={0 0}, upload=49973]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Aug 1 21:57:58.435983932 (52h0m48.373642431s) [upload=49973]: Done uploading segments... Aug 1 21:57:58.436007667 (52h0m48.373666166s) [upload=49973]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.870753247s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [.....................................XXXX...................] duration: 6.14052848s FAIL (2025-08-01 22:11:02 UTC) orbiter warning: Operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 7) ======LOGS====== K4BW277ZSGXJYQDA66OMEBVDC7DAWSOJAPX42HANNGRZN35KHQ, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.481101510 (51h59m34.418760009s) [noise=true , address=104.36.82.5:4842 , piece_id=LV7YAOMPE2AV7TF2WPWR7QJNLT4JZAIPSVZN3E23U5KDBNKZFD6A, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.481422003 (51h59m34.419080502s) [noise=true , address=79.127.205.235:4683 , piece_id=3QHDNAPG4GRLJFVVNJNFXNGKTGKUEMWWUBMW62HHE5N5LJDO4KWA, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.482030897 (51h59m34.419689396s) [noise=true , address=79.127.219.51:4213 , piece_id=L2WSPI5BEIFLGQKJIK4EWXFQXNQRQ7HVKPOWFYRPPUGNAL5DVYWA, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.482285215 (51h59m34.419943704s) [noise=true , address=104.36.82.14:4251 , piece_id=O7NADU347E2Q26TPD3IUCK6WE5UVRITC6ALZHBE5SWYTKWXFBMXA, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.482518370 (51h59m34.420176859s) [noise=true , address=79.127.220.98:4402 , piece_id=DVLSYRFFZ6UT3QRSVAN7AMXVYZNI6PWG2IITHLRUUJATJVK3QOGA, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.482724304 (51h59m34.420382803s) [noise=true , address=79.127.201.214:4297 , piece_id=MPTMGVEYC4SATIN4ANCR2ELVRXHYWRDLMHWQW7HM6UOIV2N6R7ZA, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.482928555 (51h59m34.420587054s) [noise=true , address=104.36.82.37:4680 , piece_id=X4PDUE4WXUUWX6JQVO4NWPQCBMHMHC5YHPIHDQVYYOI2FXV2GVJQ, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.483134960 (51h59m34.420793449s) [noise=true , address=104.36.82.50:4027 , piece_id=OW7BODU5N7ZEEGE2FHSLNYVJT2Z25LY3J42TZEOR5SP2EWIN6X7A, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.483338480 (51h59m34.420996979s) [noise=true , address=104.36.82.55:4204 , piece_id=R5CYXWAVV4ZEMCV7LF4D4XDC4G6KLVBONAH2TH5EIEVL7QHUTRWQ, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.483582847 (51h59m34.421241356s) [noise=true , address=104.36.82.11:4134 , piece_id=RMTNNNC2VD7MGWJDAMT6ZCF3GR43SHANFVAGJ6EHVH7WO5XA6VTA, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.483833737 (51h59m34.421492236s) [noise=true , address=104.36.82.43:4856 , piece_id=2SINBLHGRUOWUO4XHBNBG45YE3PW7RNF3QSNVCQXNCFEFHRBFNJQ, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.484021896 (51h59m34.421680385s) [noise=true , address=104.36.82.52:4101 , piece_id=XYPLQM7QITCCX7T4NKIXP5MZ7I6F24EJJTG37K4NPOBKTUFHULRQ, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.484301662 (51h59m34.421960182s) [noise=true , address=79.127.201.215:4352 , piece_id=I6BRYI5XIACAAAAAE7HQOBCF4OTALMOVPPNBWQCRSH2MM6WUUURQ, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:44.484712097 (51h59m34.422370596s) [noise=true , address=64.49.13.138:4484 , piece_id=DZ5Y4HTS6BKW2DJ5CVOB64TW4SXB6DNCJEGFUJPLAUXUEHRZRTUA, seg_pos={0 0}, upload=49960]: Uploading piece... Aug 1 21:56:45.576021470 (51h59m35.513679959s) [upload=49960]: Next returned nil segment Aug 1 21:56:45.576036609 (51h59m35.513695098s) [upload=49960]: Waiting for error group managing segments... Aug 1 21:56:45.860378776 (51h59m35.798037275s) [noise=true , address=64.49.13.138:4484 , piece_id=DZ5Y4HTS6BKW2DJ5CVOB64TW4SXB6DNCJEGFUJPLAUXUEHRZRTUA, seg_pos={0 0}, upload=49960]: Done uploading piece. err: <nil> Aug 1 21:56:45.884184188 (51h59m35.821842687s) [noise=true , address=79.127.205.235:4683 , piece_id=3QHDNAPG4GRLJFVVNJNFXNGKTGKUEMWWUBMW62HHE5N5LJDO4KWA, seg_pos={0 0}, upload=49960]: Done uploading piece. err: <nil> Aug 1 21:56:46.145493612 (51h59m36.083152101s) [noise=true , address=79.127.220.98:4402 , piece_id=DVLSYRFFZ6UT3QRSVAN7AMXVYZNI6PWG2IITHLRUUJATJVK3QOGA, seg_pos={0 0}, upload=49960]: Done uploading piece. err: <nil> Aug 1 21:56:46.257505313 (51h59m36.195163802s) [noise=true , address=79.127.220.97:4371 , piece_id=5WGPRFKQETIHTSMA42AM2C3UGCFYJ3DMMPAI2L7DD6E2P37KJKNA, seg_pos={0 0}, upload=49960]: Done uploading piece. err: <nil> Aug 1 21:56:46.334426181 (51h59m36.27208468s) [noise=true , address=79.127.201.215:4352 , piece_id=I6BRYI5XIACAAAAAE7HQOBCF4OTALMOVPPNBWQCRSH2MM6WUUURQ, seg_pos={0 0}, upload=49960]: Done uploading piece. err: <nil> Aug 1 21:56:46.386119275 (51h59m36.323777774s) [noise=true , address=79.127.201.214:4297 , piece_id=MPTMGVEYC4SATIN4ANCR2ELVRXHYWRDLMHWQW7HM6UOIV2N6R7ZA, seg_pos={0 0}, upload=49960]: Done uploading piece. err: <nil> Aug 1 21:56:46.395362048 (51h59m36.333020547s) [noise=true , address=79.127.219.51:4213 , piece_id=L2WSPI5BEIFLGQKJIK4EWXFQXNQRQ7HVKPOWFYRPPUGNAL5DVYWA, seg_pos={0 0}, upload=49960]: Done uploading piece. err: <nil> Aug 1 21:57:43.392434383 (52h0m33.330092872s) [noise=true , address=104.36.82.37:4680 , piece_id=X4PDUE4WXUUWX6JQVO4NWPQCBMHMHC5YHPIHDQVYYOI2FXV2GVJQ, seg_pos={0 0}, upload=49960]: Done uploading piece. err: ecclient: upload failed (node:1bXAqpxGhUvzwYEDGBqYRhZdTeMJcV3TYfngkKTCumpt9Nupi3, address:104.36.82.37:4680): protocol: rpc: dial tcp 104.36.82.37:4680: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:43.392477967 (52h0m33.330136466s) [noise=true , address=104.36.82.11:4134 , piece_id=RMTNNNC2VD7MGWJDAMT6ZCF3GR43SHANFVAGJ6EHVH7WO5XA6VTA, seg_pos={0 0}, upload=49960]: Done uploading piece. err: ecclient: upload failed (node:12R3Z5AgkBmWXMv5XgRyfcyqkG2gmyJs9ogT24kcLy6nuB9bzUG, address:104.36.82.11:4134): protocol: rpc: dial tcp 104.36.82.11:4134: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:43.392514086 (52h0m33.330172585s) [noise=true , address=104.36.82.55:4204 , piece_id=R5CYXWAVV4ZEMCV7LF4D4XDC4G6KLVBONAH2TH5EIEVL7QHUTRWQ, seg_pos={0 0}, upload=49960]: Done uploading piece. err: ecclient: upload failed (node:1rBZH6aLHsnWh22uNzaMiMwmLJCDMn4bYbNC5jRG7KCEuRfqpH, address:104.36.82.55:4204): protocol: write tcp 188.166.225.170:48550->104.36.82.55:4204: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:43.392546537 (52h0m33.330205027s) [noise=true , address=104.36.82.59:4327 , piece_id=OZDEYIY6B4WYZULRDRL2UBOJH2JQNY3YWSITZKWDO7GTT72AN5NQ, seg_pos={0 0}, upload=49960]: Done uploading piece. err: ecclient: upload failed (node:1V4C6AWGbyYQWfinoSj7iKovHeTJ9hqypyC5U9rZ8pveXBBfCY, address:104.36.82.59:4327): protocol: rpc: dial tcp 104.36.82.59:4327: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:43.392579040 (52h0m33.33023754s) [noise=true , address=104.36.82.5:4842 , piece_id=LV7YAOMPE2AV7TF2WPWR7QJNLT4JZAIPSVZN3E23U5KDBNKZFD6A, seg_pos={0 0}, upload=49960]: Done uploading piece. err: ecclient: upload failed (node:12q4VdT5BcjUb16PBYHK6zrnW8dqENKrcGx9zdLtNefDVB1MUSn, address:104.36.82.5:4842): protocol: context deadline exceeded; context deadline exceeded Aug 1 21:57:43.392627953 (52h0m33.330286442s) [noise=true , address=104.36.82.52:4101 , piece_id=XYPLQM7QITCCX7T4NKIXP5MZ7I6F24EJJTG37K4NPOBKTUFHULRQ, seg_pos={0 0}, upload=49960]: Done uploading piece. err: ecclient: upload failed (node:12WWXeBjFZR9PzciNfs35fFcx87N9GwDpMsPHnksFDugtaBGWp4, address:104.36.82.52:4101): protocol: rpc: dial tcp 104.36.82.52:4101: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:43.392653192 (52h0m33.330311691s) [noise=true , address=104.36.82.14:4251 , piece_id=O7NADU347E2Q26TPD3IUCK6WE5UVRITC6ALZHBE5SWYTKWXFBMXA, seg_pos={0 0}, upload=49960]: Done uploading piece. err: ecclient: upload failed (node:12btf6Z38aTMx7MaBxhUyZtFqUPgeMtP2aU8Y6FbV8zwu57zArd, address:104.36.82.14:4251): protocol: rpc: dial tcp 104.36.82.14:4251: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:43.392670836 (52h0m33.330329335s) [noise=true , address=104.36.82.12:4166 , piece_id=6ZIWCOBY76GJ6DHZF5G23U4IYTOFB5SMQW5TW3YITYH227PV2FGA, seg_pos={0 0}, upload=49960]: Done uploading piece. err: ecclient: upload failed (node:12iCB7c4ZfpqafpD1PtpbE6BfU81wJW65LmrPVJXtD58yeuoR9t, address:104.36.82.12:4166): protocol: rpc: dial tcp 104.36.82.12:4166: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:43.392698018 (52h0m33.330356517s) [noise=true , address=104.36.82.50:4027 , piece_id=OW7BODU5N7ZEEGE2FHSLNYVJT2Z25LY3J42TZEOR5SP2EWIN6X7A, seg_pos={0 0}, upload=49960]: Done uploading piece. err: ecclient: upload failed (node:1zEQhyuVxSHwdDUgDfcTog2W3oLBx4z9bYUMHpaaoLU1BTU9DQ, address:104.36.82.50:4027): protocol: write tcp 188.166.225.170:37784->104.36.82.50:4027: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:43.392715381 (52h0m33.33037387s) [noise=true , address=104.36.82.43:4856 , piece_id=2SINBLHGRUOWUO4XHBNBG45YE3PW7RNF3QSNVCQXNCFEFHRBFNJQ, seg_pos={0 0}, upload=49960]: Done uploading piece. err: ecclient: upload failed (node:12bQxwBfPEGtmMp6mtnWKsoZt3eMEjLdyYFahETWGHeKJ7BhkiZ, address:104.36.82.43:4856): protocol: rpc: dial tcp 104.36.82.43:4856: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:43.392730059 (52h0m33.330388558s) [noise=true , address=104.36.82.10:4082 , piece_id=45K4BW277ZSGXJYQDA66OMEBVDC7DAWSOJAPX42HANNGRZN35KHQ, seg_pos={0 0}, upload=49960]: Done uploading piece. err: ecclient: upload failed (node:12DKsNCtqxLvpJwLszdLs74V3eb8NH2VAV9V3bngzUczawmNoLK, address:104.36.82.10:4082): protocol: write tcp 188.166.225.170:46638->104.36.82.10:4082: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:43.392753424 (52h0m33.330411923s) [seg_pos={0 0}, upload=49960]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil> Aug 1 21:57:43.624550997 (52h0m33.562209566s) [upload=49960]: Done uploading segments... Aug 1 21:57:43.624582868 (52h0m33.562241407s) [upload=49960]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7) Duration: 6.140557506s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 7 out of 60 is failed (threshold=3) [...................................XXXXXXX..................] duration: 5.63569333s FAIL (2025-08-01 22:10:51 UTC) orbiter warning: Operation: DownloadObject uplink: object not found ("orbiter_us_0/4M") Duration: 5.635752854s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.....................................XX.X.X.................] duration: 5.902993244s FAIL (2025-08-01 22:10:43 UTC) orbiter warning: Operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 7NK72474VWE4E3YOBCDZ5O5ZOH2MSA, seg_pos={0 0}, upload=49961]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB): protocol: write tcp 188.166.225.170:38368->104.36.82.12:4151: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:56:50.418604753 (51h59m40.356263252s) [seg_pos={0 0}, upload=49961]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 1 21:56:50.418680879 (51h59m40.356339378s) [upload=49961]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 1 21:56:50.418685998 (51h59m40.356344487s) [upload=49961]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 1 21:56:50.726563872 (51h59m40.664222371s) [upload=49961]: Done uploading segments... Aug 1 21:56:50.726586586 (51h59m40.664245085s) [upload=49961]: Upload finished. err: <nil> Aug 1 21:57:28.852869669 (52h0m18.790528158s) [upload=49973]: Starting upload Aug 1 21:57:28.852942389 (52h0m18.790600888s) [upload=49973]: Done starting upload Aug 1 21:57:28.853007122 (52h0m18.790665611s) [upload=49973]: Uploading segments... Aug 1 21:57:28.853174002 (52h0m18.790832491s) [upload=49973]: Got next segment. Inline: false Aug 1 21:57:28.853208168 (52h0m18.790866657s) [upload=49973]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 1 21:57:28.853212666 (52h0m18.790871165s) [upload=49973]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 1 21:57:29.095087519 (52h0m19.032746008s) [seg_pos={0 0}, upload=49973]: Begin upload segment... Aug 1 21:57:29.095129960 (52h0m19.032788459s) [seg_pos={0 0}, upload=49973]: Done begin upload segment. Aug 1 21:57:29.095164997 (52h0m19.032823496s) [noise=true , address=77.90.31.21:4878 , piece_id=BLNTYBOAIZWISSUJYK6BSBPTSNL6QWLGEN6AP6P4NPDVSURYG5AA, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.095171800 (52h0m19.032830289s) [noise=true , address=104.36.82.6:4875 , piece_id=CS5AMB7CIBQWWOU57Z3RCCSNVKIPE4B5QWU5YIQYI2UCWH7PAPVA, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.095510027 (52h0m19.033168526s) [noise=true , address=77.90.31.18:4806 , piece_id=52B6CAHGLFVTFIDQHHHAWKCBP2WJJY7OTTGRZZ7CEUH3HDA43Q6A, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.095544193 (52h0m19.033202692s) [noise=true , address=77.90.31.10:4518 , piece_id=WUOVXUSDG6YQTDPWG4YO7VLJNZZVLH6VRGMWOJABG2EJRU7XHP5A, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.095774283 (52h0m19.033432782s) [noise=true , address=104.36.82.7:4909 , piece_id=QHJCVBT6KTU34K2KWFNACWYVVE3WJ2RU76BHVSKL4HLCZEFXZYPA, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.095819880 (52h0m19.033478379s) [noise=true , address=45.91.116.12:4225 , piece_id=PBQPLA5ZYRFPSKHDKCEFOEBS5T3GYMP2FLSFDQWJLVY756KPLJNQ, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.096021656 (52h0m19.033680155s) [noise=true , address=45.91.116.13:4153 , piece_id=PMLKMZO66KXF3ZONO5ZZOQZICFK5XNGL6MN5FFBKICVFU5AWU6PQ, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.096069618 (52h0m19.033728117s) [noise=true , address=45.91.116.9:4262 , piece_id=A3F4L3RR6WPFMBFFKLATNGG3CASBIPWQP5O3G5URFNTXBE3ZW3VA, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.096348041 (52h0m19.03400654s) [noise=true , address=104.36.82.5:4840 , piece_id=UYVKVFCJJP7XZYRIVPS4ABGEMDUMNR7YW4QGG5ZFTFCGNCJO3CSQ, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.096354803 (52h0m19.034013292s) [noise=true , address=77.90.31.12:4590 , piece_id=D6FUBIFCQJSDRGWEP5GG34AQPNB3UROCYOATD6SRTJ7KAYVMFCOA, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.096608579 (52h0m19.034267078s) [noise=true , address=77.90.31.16:4733 , piece_id=DMASMGOUATLPEKCGLHKNNIMMFYVVGSULNUOMJU6RJY3ULBDOY6KQ, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.096830774 (52h0m19.034489273s) [noise=true , address=104.36.82.4:4802 , piece_id=CLM2G33M6AK5HIESRW6LZY33NKHIR2PGITWJ23NPILRCA6UYIEHA, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.097107233 (52h0m19.034765732s) [noise=true , address=77.90.31.19:4843 , piece_id=UFONZYHL7EAFYE5PRW32RKUVQPMZPXBEFLLIFJMAB2TXJ3C6KKNA, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.097324328 (52h0m19.034982827s) [noise=true , address=77.90.31.17:4771 , piece_id=Y3WSALJX23CWPPYQ3XYMCFOSBLBX2E2JQCCS7EOEYNBWY4CDRT3Q, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.098682446 (52h0m19.036340945s) [noise=true , address=77.90.31.15:4698 , piece_id=TN4OTPUD47FD6WOLZBEHNDI43WY6QVO5JK2JNJXNVM2IT6XMGERQ, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:29.099041603 (52h0m19.036700102s) [noise=true , address=45.91.116.11:4118 , piece_id=AKZIOAR6NRNO7JWCQZIKNSL7IUH5S4IG3KPG3IY2FAKG74DH3E3Q, seg_pos={0 0}, upload=49973]: Uploading piece... Aug 1 21:57:30.718412255 (52h0m20.656070754s) [upload=49973]: Next returned nil segment Aug 1 21:57:30.718429869 (52h0m20.656088368s) [upload=49973]: Waiting for error group managing segments... Aug 1 21:57:31.107386675 (52h0m21.045045164s) [noise=true , address=45.91.116.13:4153 , piece_id=PMLKMZO66KXF3ZONO5ZZOQZICFK5XNGL6MN5FFBKICVFU5AWU6PQ, seg_pos={0 0}, upload=49973]: Done uploading piece. err: <nil> Aug 1 21:57:31.108529412 (52h0m21.046187911s) [noise=true , address=77.90.31.12:4590 , piece_id=D6FUBIFCQJSDRGWEP5GG34AQPNB3UROCYOATD6SRTJ7KAYVMFCOA, seg_pos={0 0}, upload=49973]: Done uploading piece. err: <nil> Aug 1 21:57:31.109975858 (52h0m21.047634347s) [noise=true , address=77.90.31.15:4698 , piece_id=TN4OTPUD47FD6WOLZBEHNDI43WY6QVO5JK2JNJXNVM2IT6XMGERQ, seg_pos={0 0}, upload=49973]: Done uploading piece. err: <nil> Aug 1 21:57:31.115273013 (52h0m21.052931502s) [noise=true , address=77.90.31.10:4518 , piece_id=WUOVXUSDG6YQTDPWG4YO7VLJNZZVLH6VRGMWOJABG2EJRU7XHP5A, seg_pos={0 0}, upload=49973]: Done uploading piece. err: <nil> Aug 1 21:57:31.116767281 (52h0m21.054425781s) [noise=true , address=77.90.31.18:4806 , piece_id=52B6CAHGLFVTFIDQHHHAWKCBP2WJJY7OTTGRZZ7CEUH3HDA43Q6A, seg_pos={0 0}, upload=49973]: Done uploading piece. err: <nil> Aug 1 21:57:31.201146107 (52h0m21.138804606s) [noise=true , address=77.90.31.16:4733 , piece_id=DMASMGOUATLPEKCGLHKNNIMMFYVVGSULNUOMJU6RJY3ULBDOY6KQ, seg_pos={0 0}, upload=49973]: Done uploading piece. err: <nil> Aug 1 21:57:31.201833732 (52h0m21.139492231s) [noise=true , address=77.90.31.19:4843 , piece_id=UFONZYHL7EAFYE5PRW32RKUVQPMZPXBEFLLIFJMAB2TXJ3C6KKNA, seg_pos={0 0}, upload=49973]: Done uploading piece. err: <nil> Aug 1 21:57:31.211007613 (52h0m21.148666112s) [noise=true , address=45.91.116.9:4262 , piece_id=A3F4L3RR6WPFMBFFKLATNGG3CASBIPWQP5O3G5URFNTXBE3ZW3VA, seg_pos={0 0}, upload=49973]: Done uploading piece. err: <nil> Aug 1 21:57:31.215929169 (52h0m21.153587668s) [noise=true , address=77.90.31.17:4771 , piece_id=Y3WSALJX23CWPPYQ3XYMCFOSBLBX2E2JQCCS7EOEYNBWY4CDRT3Q, seg_pos={0 0}, upload=49973]: Done uploading piece. err: <nil> Aug 1 21:57:31.222973975 (52h0m21.160632474s) [noise=true , address=77.90.31.21:4878 , piece_id=BLNTYBOAIZWISSUJYK6BSBPTSNL6QWLGEN6AP6P4NPDVSURYG5AA, seg_pos={0 0}, upload=49973]: Done uploading piece. err: <nil> Aug 1 21:57:31.234392441 (52h0m21.17205094s) [noise=true , address=45.91.116.12:4225 , piece_id=PBQPLA5ZYRFPSKHDKCEFOEBS5T3GYMP2FLSFDQWJLVY756KPLJNQ, seg_pos={0 0}, upload=49973]: Done uploading piece. err: <nil> Aug 1 21:57:31.284794305 (52h0m21.222452804s) [noise=true , address=45.91.116.11:4118 , piece_id=AKZIOAR6NRNO7JWCQZIKNSL7IUH5S4IG3KPG3IY2FAKG74DH3E3Q, seg_pos={0 0}, upload=49973]: Done uploading piece. err: <nil> Aug 1 21:57:58.202420932 (52h0m48.140079471s) [noise=true , address=104.36.82.6:4875 , piece_id=CS5AMB7CIBQWWOU57Z3RCCSNVKIPE4B5QWU5YIQYI2UCWH7PAPVA, seg_pos={0 0}, upload=49973]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:104.36.82.6:4875): protocol: write tcp 188.166.225.170:40166->104.36.82.6:4875: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:58.202441131 (52h0m48.14009963s) [noise=true , address=104.36.82.5:4840 , piece_id=UYVKVFCJJP7XZYRIVPS4ABGEMDUMNR7YW4QGG5ZFTFCGNCJO3CSQ, seg_pos={0 0}, upload=49973]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:104.36.82.5:4840): protocol: rpc: dial tcp 104.36.82.5:4840: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:58.202448445 (52h0m48.140106954s) [noise=true , address=104.36.82.7:4909 , piece_id=QHJCVBT6KTU34K2KWFNACWYVVE3WJ2RU76BHVSKL4HLCZEFXZYPA, seg_pos={0 0}, upload=49973]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:104.36.82.7:4909): protocol: rpc: dial tcp 104.36.82.7:4909: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:58.202496487 (52h0m48.140155026s) [noise=true , address=104.36.82.4:4802 , piece_id=CLM2G33M6AK5HIESRW6LZY33NKHIR2PGITWJ23NPILRCA6UYIEHA, seg_pos={0 0}, upload=49973]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: write tcp 188.166.225.170:47696->104.36.82.4:4802: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 1 21:57:58.202524240 (52h0m48.140182739s) [seg_pos={0 0}, upload=49973]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Aug 1 21:57:58.435983932 (52h0m48.373642431s) [upload=49973]: Done uploading segments... Aug 1 21:57:58.436007667 (52h0m48.373666166s) [upload=49973]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.903022028s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 5 out of 60 is failed (threshold=3) [.....................X............X.....X.......X.....X.....] duration: 2.289202502s FAIL (2025-08-01 22:10:41 UTC) orbiter warning: Operation: ListObjects unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/1K Duration: 2.28923214s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 6.294609792s FAIL (2025-08-01 22:10:41 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.294638587s edge gateway us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)