Orbiter History

checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway) antiblip: 4 out of 60 is failed (threshold=3) [...................................................X.XXX....] duration: 810.436072ms target: https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410 FAIL (2025-08-16 14:42:42 UTC) orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": dial tcp 45.38.24.2:443: i/o timeout Duration: 810.458487ms edge gateway vhost us-select-1
checkers.HTTP{"GET" "http://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt", ExpectedStatus: 308, ExpectedRedirect: "https://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt"} failure (antiblip window 3/60): (http.us_select_1.linksharing) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 478.591993ms target: http://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt FAIL (2025-08-16 14:42:40 UTC) orbiter warning: Get "http://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt": dial tcp 45.38.24.2:80: i/o timeout Duration: 478.614405ms edge linksharing us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [.......................................X........XXX..XXXXXX.] duration: 26.001175558s FAIL (2025-08-16 14:42:39 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 26.00118678s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [.................................................XXX..XXXXX.] duration: 7.102224187s FAIL (2025-08-16 14:42:38 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Aug 16 14:41:18.059283749 (189h15m12.49036388s) [upload=196900]: Starting upload Aug 16 14:41:18.059332062 (189h15m12.490412183s) [upload=196900]: Done starting upload Aug 16 14:41:18.059504060 (189h15m12.490584191s) [upload=196900]: Uploading segments... Aug 16 14:41:18.059537265 (189h15m12.490617396s) [upload=196900]: Got next segment. Inline: false Aug 16 14:41:18.059560799 (189h15m12.49064093s) [upload=196900]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 14:41:18.059565188 (189h15m12.490645319s) [upload=196900]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 14:41:18.298282637 (189h15m12.729362778s) [seg_pos={0 0}, upload=196900]: Begin upload segment... Aug 16 14:41:18.298344817 (189h15m12.729424948s) [seg_pos={0 0}, upload=196900]: Done begin upload segment. Aug 16 14:41:18.298376758 (189h15m12.729456889s) [noise=true , address=104.36.82.6:4875 , piece_id=3Y2WW3FPBDP26AKZOGMXMRKGO5SUZ4KCAGQK7VE53AG7BLY7ENYA, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.298401766 (189h15m12.729481897s) [noise=true , address=104.36.82.15:4258 , piece_id=TK44FRS6QGGDFDXTEOEZ6H6HH2MQMAZ7VPOTZQGMU6I2UMABBECQ, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.298802282 (189h15m12.729882413s) [noise=true , address=45.91.116.7:4009 , piece_id=T35MI4CZIKKH55YQMWGZYNAOHQIRD634NKWBP5ESTCUSNIZ53ZYQ, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.299803298 (189h15m12.730883429s) [noise=true , address=104.36.82.10:4079 , piece_id=QYDL2MKDZQDA2Q3KS2X5YRCOXH4VEINXVUNY66M5P7X3TAWMYI5Q, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.301061816 (189h15m12.732141947s) [noise=true , address=45.91.116.7:4010 , piece_id=H27FVU5GTAZI5TSTOPNG4HTL6NXJ2XFATT2NP5DG7PZHM5IAXFOA, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.301518819 (189h15m12.73259895s) [noise=true , address=104.36.82.5:4840 , piece_id=CWKO2G6GPYCRQDY4RHQFHHZOVTJTIA5V2EWFVHLIAPEGZOWF2IWA, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.302477774 (189h15m12.733557905s) [noise=true , address=45.91.116.11:4117 , piece_id=LDGIVV667A6BG3YMHWLQTK7ZBRA3LXO3GLLSCTT7HNIIZ4YAQBRQ, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.302776165 (189h15m12.733856296s) [noise=true , address=104.36.82.7:4910 , piece_id=W7T6BWBZQ3HOWSNC3ZF2EXIW2WFPKFT6YXORO4GY7PIZDC5R3B7Q, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.303009963 (189h15m12.734090124s) [noise=true , address=104.36.82.12:4150 , piece_id=5SDVCZCTMXX4GSU5H4ZAKQGKQDHLKCZ3TNWI35AP4TRIA7BLW66A, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.303201981 (189h15m12.734282112s) [noise=true , address=45.91.116.10:4082 , piece_id=XI3YGRGLJIRA2LN7U2MMQCLN5PYYBFMW3DY7HMWOTXZY2WTSD42A, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.303441258 (189h15m12.734521389s) [noise=true , address=104.36.82.9:4006 , piece_id=OJ6O4A5DVJ2NQTS6PFW5VS4V6WDTIJFYD3UVYNZYOUAE7RFNBUTQ, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.303652101 (189h15m12.734732233s) [noise=true , address=104.36.82.11:4114 , piece_id=2NXRA6JZ32CFPKKRO5RLYXMDUJB5236Q6I6CN2NJI4X2RNXDNKAA, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.303878915 (189h15m12.734959036s) [noise=true , address=104.36.82.2:4732 , piece_id=AUJI3CUB57YJNGX2QLUYM2MGBHNNJBUBJIPZQ5QKNIUWAECOTS4A, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.304097604 (189h15m12.735177735s) [noise=true , address=104.36.82.15:4259 , piece_id=VIKWOS7EK7JKKKZRSYZ5PZIYPZVMSOV3C3H7WXF6RCOR33NSW6EA, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.304419109 (189h15m12.73549924s) [noise=true , address=45.91.116.12:4226 , piece_id=QBOLY2YJ7HB4EO4IKC7WTPYMOXA4SDIIMVMVEQM673HTM4AWXJ7A, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.304654881 (189h15m12.735735002s) [noise=true , address=45.91.116.5:4299 , piece_id=IHMNTP3BZI64MFONUUNGNKW7WTB2IVCKQM63QUMEQJGIDAZ6QGHA, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:20.156503810 (189h15m14.587583941s) [upload=196900]: Next returned nil segment Aug 16 14:41:20.156524930 (189h15m14.587605061s) [upload=196900]: Waiting for error group managing segments... Aug 16 14:41:20.462807729 (189h15m14.89388786s) [noise=true , address=104.36.82.6:4875 , piece_id=3Y2WW3FPBDP26AKZOGMXMRKGO5SUZ4KCAGQK7VE53AG7BLY7ENYA, seg_pos={0 0}, upload=196900]: Done uploading piece. err: <nil> Aug 16 14:41:20.490967610 (189h15m14.922047741s) [noise=true , address=104.36.82.15:4259 , piece_id=VIKWOS7EK7JKKKZRSYZ5PZIYPZVMSOV3C3H7WXF6RCOR33NSW6EA, seg_pos={0 0}, upload=196900]: Done uploading piece. err: <nil> Aug 16 14:41:20.501478531 (189h15m14.932558662s) [noise=true , address=104.36.82.11:4114 , piece_id=2NXRA6JZ32CFPKKRO5RLYXMDUJB5236Q6I6CN2NJI4X2RNXDNKAA, seg_pos={0 0}, upload=196900]: Done uploading piece. err: <nil> Aug 16 14:41:20.507709423 (189h15m14.938789554s) [noise=true , address=104.36.82.9:4006 , piece_id=OJ6O4A5DVJ2NQTS6PFW5VS4V6WDTIJFYD3UVYNZYOUAE7RFNBUTQ, seg_pos={0 0}, upload=196900]: Done uploading piece. err: <nil> Aug 16 14:41:20.522813604 (189h15m14.953893725s) [noise=true , address=104.36.82.12:4150 , piece_id=5SDVCZCTMXX4GSU5H4ZAKQGKQDHLKCZ3TNWI35AP4TRIA7BLW66A, seg_pos={0 0}, upload=196900]: Done uploading piece. err: <nil> Aug 16 14:41:20.578876905 (189h15m15.009957036s) [noise=true , address=104.36.82.15:4258 , piece_id=TK44FRS6QGGDFDXTEOEZ6H6HH2MQMAZ7VPOTZQGMU6I2UMABBECQ, seg_pos={0 0}, upload=196900]: Done uploading piece. err: <nil> Aug 16 14:41:20.658903978 (189h15m15.089984149s) [noise=true , address=104.36.82.10:4079 , piece_id=QYDL2MKDZQDA2Q3KS2X5YRCOXH4VEINXVUNY66M5P7X3TAWMYI5Q, seg_pos={0 0}, upload=196900]: Done uploading piece. err: <nil> Aug 16 14:41:21.998036313 (189h15m16.429116444s) [noise=true , address=45.91.116.12:4226 , piece_id=QBOLY2YJ7HB4EO4IKC7WTPYMOXA4SDIIMVMVEQM673HTM4AWXJ7A, seg_pos={0 0}, upload=196900]: Done uploading piece. err: <nil> Aug 16 14:41:22.398912736 (189h15m16.829992867s) [noise=true , address=45.91.116.10:4082 , piece_id=XI3YGRGLJIRA2LN7U2MMQCLN5PYYBFMW3DY7HMWOTXZY2WTSD42A, seg_pos={0 0}, upload=196900]: Done uploading piece. err: <nil> Aug 16 14:41:23.643969778 (189h15m18.075049909s) [noise=true , address=45.91.116.7:4010 , piece_id=H27FVU5GTAZI5TSTOPNG4HTL6NXJ2XFATT2NP5DG7PZHM5IAXFOA, seg_pos={0 0}, upload=196900]: Done uploading piece. err: <nil> Aug 16 14:41:35.220987509 (189h15m29.65206764s) [noise=true , address=104.36.82.7:4910 , piece_id=W7T6BWBZQ3HOWSNC3ZF2EXIW2WFPKFT6YXORO4GY7PIZDC5R3B7Q, seg_pos={0 0}, upload=196900]: Done uploading piece. err: <nil> Aug 16 14:41:47.419790979 (189h15m41.8508711s) [noise=true , address=104.36.82.2:4732 , piece_id=AUJI3CUB57YJNGX2QLUYM2MGBHNNJBUBJIPZQ5QKNIUWAECOTS4A, seg_pos={0 0}, upload=196900]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:104.36.82.2:4732): protocol: write tcp 188.166.225.170:36840->104.36.82.2:4732: use of closed network connection; write tcp 188.166.225.170:36840->104.36.82.2:4732: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36840->104.36.82.2:4732: use of closed network connection Aug 16 14:41:47.419887454 (189h15m41.850967585s) [noise=true , address=45.91.116.7:4009 , piece_id=T35MI4CZIKKH55YQMWGZYNAOHQIRD634NKWBP5ESTCUSNIZ53ZYQ, seg_pos={0 0}, upload=196900]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: write tcp 188.166.225.170:49112->45.91.116.7:4009: use of closed network connection; write tcp 188.166.225.170:49112->45.91.116.7:4009: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49112->45.91.116.7:4009: use of closed network connection Aug 16 14:41:47.419939924 (189h15m41.851020055s) [noise=true , address=45.91.116.11:4117 , piece_id=LDGIVV667A6BG3YMHWLQTK7ZBRA3LXO3GLLSCTT7HNIIZ4YAQBRQ, seg_pos={0 0}, upload=196900]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: write tcp 188.166.225.170:49474->45.91.116.11:4117: use of closed network connection; write tcp 188.166.225.170:49474->45.91.116.11:4117: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49474->45.91.116.11:4117: use of closed network connection Aug 16 14:41:47.420020879 (189h15m41.851101s) [noise=true , address=45.91.116.5:4299 , piece_id=IHMNTP3BZI64MFONUUNGNKW7WTB2IVCKQM63QUMEQJGIDAZ6QGHA, seg_pos={0 0}, upload=196900]: Done uploading piece. err: ecclient: upload failed (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, address:45.91.116.5:4299): protocol: read tcp 188.166.225.170:47592->45.91.116.5:4299: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:41:47.420030317 (189h15m41.851110448s) [noise=true , address=104.36.82.5:4840 , piece_id=CWKO2G6GPYCRQDY4RHQFHHZOVTJTIA5V2EWFVHLIAPEGZOWF2IWA, seg_pos={0 0}, upload=196900]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:104.36.82.5:4840): protocol: read tcp 188.166.225.170:54090->104.36.82.5:4840: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:41:47.420076175 (189h15m41.851156306s) [seg_pos={0 0}, upload=196900]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Aug 16 14:41:47.645353128 (189h15m42.076433249s) [upload=196900]: Done uploading segments... Aug 16 14:41:47.645385871 (189h15m42.076466002s) [upload=196900]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 7.10225215s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 5 out of 60 is failed (threshold=3) [...................................................XX..XXX..] duration: 26.705380485s FAIL (2025-08-16 14:42:36 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 26.705482389s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 4 out of 60 is failed (threshold=3) [........X...............X............................X....X.] duration: 4.027192666s FAIL (2025-08-16 14:42:35 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/1K):: after 5b4e9c59e1b603b8baec6a7847bc559340eec1a18700944fbb5f0cb717754552 != before 08bb3ceaa2907a9c695559f1a049d1db95ca5e65c937769756841140f9b12378 ======LOGS====== Aug 16 14:41:26.568660143 (189h15m20.999740275s) [upload=196903]: Starting upload Aug 16 14:41:26.568720509 (189h15m20.99980064s) [upload=196903]: Done starting upload Aug 16 14:41:26.568786014 (189h15m20.999866155s) [upload=196903]: Uploading segments... Aug 16 14:41:26.568826642 (189h15m20.999906773s) [upload=196903]: Got next segment. Inline: true Aug 16 14:41:26.568831321 (189h15m20.999911452s) [upload=196903]: Waiting for error group managing segments... Aug 16 14:41:26.568870796 (189h15m20.999950957s) [upload=196903]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 14:41:26.568875666 (189h15m20.999955807s) [upload=196903]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 16 14:41:26.568879223 (189h15m20.999959364s) [upload=196903]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 14:41:27.069024505 (189h15m21.500104636s) [upload=196903]: Done uploading segments... Aug 16 14:41:27.069041257 (189h15m21.500121388s) [upload=196903]: Upload finished. err: <nil> Duration:* 4.027216021s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect) antiblip: 1 out of 15 is failed (threshold=6) [............X..] duration: 9.763701775s remote: "tcp:3.109.243.18:443" response headers: Date=Sat, 16 Aug 2025 14:40:59 GMT Content-Type=text/html Content-Length=164 X-Cluster-Name=ap-south-1-prod-hosting-red target: https://www.storj.io/dcs-satellites FAIL (2025-08-16 14:42:35 UTC) orbiter warning: unexpected status code "504 Gateway Time-out" "<html>\r\n<head><title>504 Gateway Time-out</title></head>\r\n<body>\r\n<center><h1>504 Gateway Time-out</h1></center>\r\n<hr><center>openresty</center>\r\n</body>\r\n</html>\r\n" Duration: 9.763724669s core website
checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 4.017244723s FAIL (2025-08-16 14:42:33 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": dial tcp 45.38.24.2:443: i/o timeout Duration: 4.017276634s edge authservice http us-select-1
checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 2.962141867s FAIL (2025-08-16 14:42:16 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": dial tcp 45.38.24.2:443: i/o timeout Duration: 2.962167667s edge authservice http us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [........................................X........XXX..XXXXXX] duration: 30.001309559s FAIL (2025-08-16 14:42:13 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.001324407s edge gateway us-select-1
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway) antiblip: 4 out of 60 is failed (threshold=3) [....................................................X.XXX...] duration: 1.646766819s target: https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410 FAIL (2025-08-16 14:42:06 UTC) orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": dial tcp 45.38.24.2:443: i/o timeout Duration: 1.646797537s edge gateway vhost us-select-1
checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 3.363270191s FAIL (2025-08-16 14:41:58 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": dial tcp 45.38.24.2:443: i/o timeout Duration: 3.363313153s edge authservice http us-select-1
checkers.HTTP{"GET" "http://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt", ExpectedStatus: 308, ExpectedRedirect: "https://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt"} failure (antiblip window 3/60): (http.us_select_1.linksharing) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 1.8131s target: http://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt FAIL (2025-08-16 14:41:50 UTC) orbiter warning: Get "http://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt": dial tcp 45.38.24.2:80: i/o timeout Duration: 1.81312587s edge linksharing us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [..................................................XXX..XXXXX] duration: 30.227946502s FAIL (2025-08-16 14:41:47 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Aug 16 14:41:18.059283749 (189h15m12.49036388s) [upload=196900]: Starting upload Aug 16 14:41:18.059332062 (189h15m12.490412183s) [upload=196900]: Done starting upload Aug 16 14:41:18.059504060 (189h15m12.490584191s) [upload=196900]: Uploading segments... Aug 16 14:41:18.059537265 (189h15m12.490617396s) [upload=196900]: Got next segment. Inline: false Aug 16 14:41:18.059560799 (189h15m12.49064093s) [upload=196900]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 14:41:18.059565188 (189h15m12.490645319s) [upload=196900]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 14:41:18.298282637 (189h15m12.729362778s) [seg_pos={0 0}, upload=196900]: Begin upload segment... Aug 16 14:41:18.298344817 (189h15m12.729424948s) [seg_pos={0 0}, upload=196900]: Done begin upload segment. Aug 16 14:41:18.298376758 (189h15m12.729456889s) [noise=true , address=104.36.82.6:4875 , piece_id=3Y2WW3FPBDP26AKZOGMXMRKGO5SUZ4KCAGQK7VE53AG7BLY7ENYA, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.298401766 (189h15m12.729481897s) [noise=true , address=104.36.82.15:4258 , piece_id=TK44FRS6QGGDFDXTEOEZ6H6HH2MQMAZ7VPOTZQGMU6I2UMABBECQ, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.298802282 (189h15m12.729882413s) [noise=true , address=45.91.116.7:4009 , piece_id=T35MI4CZIKKH55YQMWGZYNAOHQIRD634NKWBP5ESTCUSNIZ53ZYQ, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.299803298 (189h15m12.730883429s) [noise=true , address=104.36.82.10:4079 , piece_id=QYDL2MKDZQDA2Q3KS2X5YRCOXH4VEINXVUNY66M5P7X3TAWMYI5Q, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.301061816 (189h15m12.732141947s) [noise=true , address=45.91.116.7:4010 , piece_id=H27FVU5GTAZI5TSTOPNG4HTL6NXJ2XFATT2NP5DG7PZHM5IAXFOA, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.301518819 (189h15m12.73259895s) [noise=true , address=104.36.82.5:4840 , piece_id=CWKO2G6GPYCRQDY4RHQFHHZOVTJTIA5V2EWFVHLIAPEGZOWF2IWA, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.302477774 (189h15m12.733557905s) [noise=true , address=45.91.116.11:4117 , piece_id=LDGIVV667A6BG3YMHWLQTK7ZBRA3LXO3GLLSCTT7HNIIZ4YAQBRQ, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.302776165 (189h15m12.733856296s) [noise=true , address=104.36.82.7:4910 , piece_id=W7T6BWBZQ3HOWSNC3ZF2EXIW2WFPKFT6YXORO4GY7PIZDC5R3B7Q, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.303009963 (189h15m12.734090124s) [noise=true , address=104.36.82.12:4150 , piece_id=5SDVCZCTMXX4GSU5H4ZAKQGKQDHLKCZ3TNWI35AP4TRIA7BLW66A, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.303201981 (189h15m12.734282112s) [noise=true , address=45.91.116.10:4082 , piece_id=XI3YGRGLJIRA2LN7U2MMQCLN5PYYBFMW3DY7HMWOTXZY2WTSD42A, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.303441258 (189h15m12.734521389s) [noise=true , address=104.36.82.9:4006 , piece_id=OJ6O4A5DVJ2NQTS6PFW5VS4V6WDTIJFYD3UVYNZYOUAE7RFNBUTQ, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.303652101 (189h15m12.734732233s) [noise=true , address=104.36.82.11:4114 , piece_id=2NXRA6JZ32CFPKKRO5RLYXMDUJB5236Q6I6CN2NJI4X2RNXDNKAA, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.303878915 (189h15m12.734959036s) [noise=true , address=104.36.82.2:4732 , piece_id=AUJI3CUB57YJNGX2QLUYM2MGBHNNJBUBJIPZQ5QKNIUWAECOTS4A, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.304097604 (189h15m12.735177735s) [noise=true , address=104.36.82.15:4259 , piece_id=VIKWOS7EK7JKKKZRSYZ5PZIYPZVMSOV3C3H7WXF6RCOR33NSW6EA, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.304419109 (189h15m12.73549924s) [noise=true , address=45.91.116.12:4226 , piece_id=QBOLY2YJ7HB4EO4IKC7WTPYMOXA4SDIIMVMVEQM673HTM4AWXJ7A, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:18.304654881 (189h15m12.735735002s) [noise=true , address=45.91.116.5:4299 , piece_id=IHMNTP3BZI64MFONUUNGNKW7WTB2IVCKQM63QUMEQJGIDAZ6QGHA, seg_pos={0 0}, upload=196900]: Uploading piece... Aug 16 14:41:20.156503810 (189h15m14.587583941s) [upload=196900]: Next returned nil segment Aug 16 14:41:20.156524930 (189h15m14.587605061s) [upload=196900]: Waiting for error group managing segments... Aug 16 14:41:20.462807729 (189h15m14.89388786s) [noise=true , address=104.36.82.6:4875 , piece_id=3Y2WW3FPBDP26AKZOGMXMRKGO5SUZ4KCAGQK7VE53AG7BLY7ENYA, seg_pos={0 0}, upload=196900]: Done uploading piece. err: <nil> Aug 16 14:41:20.490967610 (189h15m14.922047741s) [noise=true , address=104.36.82.15:4259 , piece_id=VIKWOS7EK7JKKKZRSYZ5PZIYPZVMSOV3C3H7WXF6RCOR33NSW6EA, seg_pos={0 0}, upload=196900]: Done uploading piece. err: <nil> Aug 16 14:41:20.501478531 (189h15m14.932558662s) [noise=true , address=104.36.82.11:4114 , piece_id=2NXRA6JZ32CFPKKRO5RLYXMDUJB5236Q6I6CN2NJI4X2RNXDNKAA, seg_pos={0 0}, upload=196900]: Done uploading piece. err: <nil> Aug 16 14:41:20.507709423 (189h15m14.938789554s) [noise=true , address=104.36.82.9:4006 , piece_id=OJ6O4A5DVJ2NQTS6PFW5VS4V6WDTIJFYD3UVYNZYOUAE7RFNBUTQ, seg_pos={0 0}, upload=196900]: Done uploading piece. err: <nil> Aug 16 14:41:20.522813604 (189h15m14.953893725s) [noise=true , address=104.36.82.12:4150 , piece_id=5SDVCZCTMXX4GSU5H4ZAKQGKQDHLKCZ3TNWI35AP4TRIA7BLW66A, seg_pos={0 0}, upload=196900]: Done uploading piece. err: <nil> Aug 16 14:41:20.578876905 (189h15m15.009957036s) [noise=true , address=104.36.82.15:4258 , piece_id=TK44FRS6QGGDFDXTEOEZ6H6HH2MQMAZ7VPOTZQGMU6I2UMABBECQ, seg_pos={0 0}, upload=196900]: Done uploading piece. err: <nil> Aug 16 14:41:20.658903978 (189h15m15.089984149s) [noise=true , address=104.36.82.10:4079 , piece_id=QYDL2MKDZQDA2Q3KS2X5YRCOXH4VEINXVUNY66M5P7X3TAWMYI5Q, seg_pos={0 0}, upload=196900]: Done uploading piece. err: <nil> Aug 16 14:41:21.998036313 (189h15m16.429116444s) [noise=true , address=45.91.116.12:4226 , piece_id=QBOLY2YJ7HB4EO4IKC7WTPYMOXA4SDIIMVMVEQM673HTM4AWXJ7A, seg_pos={0 0}, upload=196900]: Done uploading piece. err: <nil> Aug 16 14:41:22.398912736 (189h15m16.829992867s) [noise=true , address=45.91.116.10:4082 , piece_id=XI3YGRGLJIRA2LN7U2MMQCLN5PYYBFMW3DY7HMWOTXZY2WTSD42A, seg_pos={0 0}, upload=196900]: Done uploading piece. err: <nil> Aug 16 14:41:23.643969778 (189h15m18.075049909s) [noise=true , address=45.91.116.7:4010 , piece_id=H27FVU5GTAZI5TSTOPNG4HTL6NXJ2XFATT2NP5DG7PZHM5IAXFOA, seg_pos={0 0}, upload=196900]: Done uploading piece. err: <nil> Aug 16 14:41:35.220987509 (189h15m29.65206764s) [noise=true , address=104.36.82.7:4910 , piece_id=W7T6BWBZQ3HOWSNC3ZF2EXIW2WFPKFT6YXORO4GY7PIZDC5R3B7Q, seg_pos={0 0}, upload=196900]: Done uploading piece. err: <nil> Aug 16 14:41:47.419790979 (189h15m41.8508711s) [noise=true , address=104.36.82.2:4732 , piece_id=AUJI3CUB57YJNGX2QLUYM2MGBHNNJBUBJIPZQ5QKNIUWAECOTS4A, seg_pos={0 0}, upload=196900]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:104.36.82.2:4732): protocol: write tcp 188.166.225.170:36840->104.36.82.2:4732: use of closed network connection; write tcp 188.166.225.170:36840->104.36.82.2:4732: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36840->104.36.82.2:4732: use of closed network connection Aug 16 14:41:47.419887454 (189h15m41.850967585s) [noise=true , address=45.91.116.7:4009 , piece_id=T35MI4CZIKKH55YQMWGZYNAOHQIRD634NKWBP5ESTCUSNIZ53ZYQ, seg_pos={0 0}, upload=196900]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: write tcp 188.166.225.170:49112->45.91.116.7:4009: use of closed network connection; write tcp 188.166.225.170:49112->45.91.116.7:4009: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49112->45.91.116.7:4009: use of closed network connection Aug 16 14:41:47.419939924 (189h15m41.851020055s) [noise=true , address=45.91.116.11:4117 , piece_id=LDGIVV667A6BG3YMHWLQTK7ZBRA3LXO3GLLSCTT7HNIIZ4YAQBRQ, seg_pos={0 0}, upload=196900]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: write tcp 188.166.225.170:49474->45.91.116.11:4117: use of closed network connection; write tcp 188.166.225.170:49474->45.91.116.11:4117: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49474->45.91.116.11:4117: use of closed network connection Aug 16 14:41:47.420020879 (189h15m41.851101s) [noise=true , address=45.91.116.5:4299 , piece_id=IHMNTP3BZI64MFONUUNGNKW7WTB2IVCKQM63QUMEQJGIDAZ6QGHA, seg_pos={0 0}, upload=196900]: Done uploading piece. err: ecclient: upload failed (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, address:45.91.116.5:4299): protocol: read tcp 188.166.225.170:47592->45.91.116.5:4299: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:41:47.420030317 (189h15m41.851110448s) [noise=true , address=104.36.82.5:4840 , piece_id=CWKO2G6GPYCRQDY4RHQFHHZOVTJTIA5V2EWFVHLIAPEGZOWF2IWA, seg_pos={0 0}, upload=196900]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:104.36.82.5:4840): protocol: read tcp 188.166.225.170:54090->104.36.82.5:4840: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:41:47.420076175 (189h15m41.851156306s) [seg_pos={0 0}, upload=196900]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Aug 16 14:41:47.645353128 (189h15m42.076433249s) [upload=196900]: Done uploading segments... Aug 16 14:41:47.645385871 (189h15m42.076466002s) [upload=196900]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 30.227974275s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [....................................................X.X.XXX.] duration: 40.447853556s FAIL (2025-08-16 14:41:45 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 9) ======LOGS====== ess=104.36.82.44:4884 , piece_id=2VREOXCLUGM2FUX7VRQRHZID7CSMAJZ6JRWLOJ4GKREJ4OHUMZOA, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.166352764 (189h13m48.597432895s) [noise=true , address=77.90.31.10:4541 , piece_id=HAOVML65R5HZ5HGAFUDBUGB7S3QKJ2DHF3X7AQTMGEXVVFZC3RYA, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.166371479 (189h13m48.59745161s) [noise=true , address=77.90.31.21:4902 , piece_id=WUTMD3HBQ3OUMVOIRSDYOWEQAVV7MIEOONMWDDCKXRXVEZ7UH6WQ, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.166641586 (189h13m48.597721707s) [noise=true , address=104.36.82.23:4463 , piece_id=B2BOEACKYAULFJIJZFCOJQ7C36SOJCKKVCQIP3QYNSBE6YZS66WQ, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.166717972 (189h13m48.597798093s) [noise=true , address=77.90.31.18:4835 , piece_id=XURSIUMVEHTCRCXBWEIZ2ZJCSKTQXCVOQIKXWISDAYMRW7M4NOTQ, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.166907264 (189h13m48.597987395s) [noise=true , address=45.91.116.12:4240 , piece_id=5O3C47NABFGQ2E45ZL67PGJC5F67XNFDBQ7GYKGACMN6ZPAL62XA, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.167135330 (189h13m48.598215461s) [noise=true , address=104.36.82.58:4299 , piece_id=KSQRSQVB6EGT37PKX3YV7QUQTJZSJKEUD3CKCFPCAD6S4NJAL7QA, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.167357465 (189h13m48.598437586s) [noise=true , address=79.127.201.215:4350 , piece_id=6MWZ4MNZXE6KVJ4RLOTRWM24DER64OZW7CMCUPP7RYJQR7LWCTJQ, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.167565724 (189h13m48.598645855s) [noise=true , address=45.91.116.6:4991 , piece_id=6MTNZT65LSI6PXSVMUJHT64GQVMU6CWOVUH2KGOINQIAUSKSDQRQ, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.167779904 (189h13m48.598860035s) [noise=true , address=79.127.220.101:4061 , piece_id=UURYVARKKAUJAAVMCDWQWRB6NKVOHVWT4GEG7B72RZDJ2CI3MLLQ, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.167991478 (189h13m48.599071609s) [noise=true , address=79.127.220.98:4401 , piece_id=6NZAXKDQ6HSBIWHHARLGLUL3QI2NBH6EMGIOYMNUFKJPOBKWDH6A, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.168183566 (189h13m48.599263697s) [noise=true , address=104.36.82.14:4247 , piece_id=G56NHF7M7FCPXDXRXEEADHQ2EZKINRBRED7Q4XI3662EOYODPWAQ, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.168419999 (189h13m48.59950012s) [noise=true , address=45.91.116.7:4017 , piece_id=IO6CYVMTFTQG2SAFKJIWEVJH72MYZA34ZGCT6WZR6PWXPICIQGNQ, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.168687240 (189h13m48.599767371s) [noise=true , address=45.91.116.13:4159 , piece_id=33U4IEPB3Z5GEDQOHFWTAR5ATFJXN236X5Y66R6QNZ2FMK7KTO4A, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.168925556 (189h13m48.600005677s) [noise=true , address=64.49.13.169:4706 , piece_id=5VF6GL7HRPMPQT4FKJ2KQOWPUM6B5LEQZ2SSG62B6LPCAVGBTLBQ, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.169130949 (189h13m48.60021107s) [noise=true , address=104.36.82.60:4348 , piece_id=ZJXZKILKMDTWN7ZFW67F3CGZIQHA7OYXQIONNW3CG72WX57ZIQQQ, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:55.311744797 (189h13m49.742824938s) [upload=196876]: Next returned nil segment Aug 16 14:39:55.311772059 (189h13m49.74285219s) [upload=196876]: Waiting for error group managing segments... Aug 16 14:39:55.601879518 (189h13m50.032959639s) [noise=true , address=64.49.13.169:4706 , piece_id=5VF6GL7HRPMPQT4FKJ2KQOWPUM6B5LEQZ2SSG62B6LPCAVGBTLBQ, seg_pos={0 0}, upload=196876]: Done uploading piece. err: <nil> Aug 16 14:39:56.004918829 (189h13m50.435998971s) [noise=true , address=79.127.220.101:4061 , piece_id=UURYVARKKAUJAAVMCDWQWRB6NKVOHVWT4GEG7B72RZDJ2CI3MLLQ, seg_pos={0 0}, upload=196876]: Done uploading piece. err: <nil> Aug 16 14:39:56.027693798 (189h13m50.458773929s) [noise=true , address=79.127.220.98:4401 , piece_id=6NZAXKDQ6HSBIWHHARLGLUL3QI2NBH6EMGIOYMNUFKJPOBKWDH6A, seg_pos={0 0}, upload=196876]: Done uploading piece. err: <nil> Aug 16 14:39:57.572686072 (189h13m52.003766203s) [noise=true , address=45.91.116.12:4240 , piece_id=5O3C47NABFGQ2E45ZL67PGJC5F67XNFDBQ7GYKGACMN6ZPAL62XA, seg_pos={0 0}, upload=196876]: Done uploading piece. err: <nil> Aug 16 14:39:58.673547510 (189h13m53.104627681s) [noise=true , address=77.90.31.10:4541 , piece_id=HAOVML65R5HZ5HGAFUDBUGB7S3QKJ2DHF3X7AQTMGEXVVFZC3RYA, seg_pos={0 0}, upload=196876]: Done uploading piece. err: <nil> Aug 16 14:40:26.090374323 (189h14m20.521454444s) [noise=true , address=79.127.201.215:4350 , piece_id=6MWZ4MNZXE6KVJ4RLOTRWM24DER64OZW7CMCUPP7RYJQR7LWCTJQ, seg_pos={0 0}, upload=196876]: Done uploading piece. err: <nil> Aug 16 14:40:32.402083903 (189h14m26.833164034s) [noise=true , address=104.36.82.14:4247 , piece_id=G56NHF7M7FCPXDXRXEEADHQ2EZKINRBRED7Q4XI3662EOYODPWAQ, seg_pos={0 0}, upload=196876]: Done uploading piece. err: <nil> Aug 16 14:40:38.857957123 (189h14m33.289037244s) [noise=true , address=104.36.82.58:4299 , piece_id=KSQRSQVB6EGT37PKX3YV7QUQTJZSJKEUD3CKCFPCAD6S4NJAL7QA, seg_pos={0 0}, upload=196876]: Done uploading piece. err: <nil> Aug 16 14:40:49.164615516 (189h14m43.595695647s) [noise=true , address=77.90.31.18:4835 , piece_id=XURSIUMVEHTCRCXBWEIZ2ZJCSKTQXCVOQIKXWISDAYMRW7M4NOTQ, seg_pos={0 0}, upload=196876]: Done uploading piece. err: <nil> Aug 16 14:40:53.061942988 (189h14m47.493023119s) [noise=true , address=104.36.82.23:4463 , piece_id=B2BOEACKYAULFJIJZFCOJQ7C36SOJCKKVCQIP3QYNSBE6YZS66WQ, seg_pos={0 0}, upload=196876]: Done uploading piece. err: ecclient: upload failed (node:12d3ACC8cWirsfXo2XNPTtr6sULutTpdEpvFDweeiQuPQ3NiZyL, address:104.36.82.23:4463): protocol: expected piece hash; context deadline exceeded; EOF Aug 16 14:40:53.062241600 (189h14m47.493321731s) [noise=true , address=104.36.82.59:4310 , piece_id=64KHK5PWAB57RCI7FU4U26XDZKH6WXOEJ3NUFLTRRTHKS2UCPF2A, seg_pos={0 0}, upload=196876]: Done uploading piece. err: ecclient: upload failed (node:12uYJ3ag6xFbLiu39t7jqwvtbo1QLwHw7k9PmfrMWBZBappp8MX, address:104.36.82.59:4310): protocol: rpc: dial tcp 104.36.82.59:4310: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:40:53.062279993 (189h14m47.493360124s) [noise=true , address=45.91.116.5:4307 , piece_id=Z2JSIUQPBMW2JSYVMSZVBMCQXW3L2C2NEWPSRF6FUIJQRMEFHSKA, seg_pos={0 0}, upload=196876]: Done uploading piece. err: ecclient: upload failed (node:1ATcyutSAE1A1uN2QcwYFdY1ZvUCZTKiybzXt66NgnqrFYoWWC, address:45.91.116.5:4307): protocol: read tcp 188.166.225.170:57434->45.91.116.5:4307: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:40:53.062286144 (189h14m47.493366275s) [noise=true , address=45.91.116.6:4991 , piece_id=6MTNZT65LSI6PXSVMUJHT64GQVMU6CWOVUH2KGOINQIAUSKSDQRQ, seg_pos={0 0}, upload=196876]: Done uploading piece. err: ecclient: upload failed (node:12muT2wQRGoRoDtpsPGS3EDjQ9sLq2qTS3jRSHWfuV1h9itzMbb, address:45.91.116.6:4991): protocol: expected piece hash; write tcp 188.166.225.170:42596->45.91.116.6:4991: use of closed network connection; write tcp 188.166.225.170:42596->45.91.116.6:4991: use of closed network connection; EOF Aug 16 14:40:53.062367069 (189h14m47.49344719s) [noise=true , address=45.91.116.13:4159 , piece_id=33U4IEPB3Z5GEDQOHFWTAR5ATFJXN236X5Y66R6QNZ2FMK7KTO4A, seg_pos={0 0}, upload=196876]: Done uploading piece. err: ecclient: upload failed (node:128V9AYe3XZKQ9o2cbdeSNYoK55MHFJ4q1KbhjGXWzKrv17Hv3t, address:45.91.116.13:4159): protocol: rpc: dial tcp 45.91.116.13:4159: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:40:53.062469385 (189h14m47.493549516s) [noise=true , address=45.91.116.7:4017 , piece_id=IO6CYVMTFTQG2SAFKJIWEVJH72MYZA34ZGCT6WZR6PWXPICIQGNQ, seg_pos={0 0}, upload=196876]: Done uploading piece. err: ecclient: upload failed (node:1XLSYete7Dj3VNt24BbP5szLp35H6F6urWr8tRAdYy1Ms1Gpj7, address:45.91.116.7:4017): protocol: rpc: dial tcp 45.91.116.7:4017: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:40:53.062564447 (189h14m47.493644578s) [noise=true , address=104.36.82.60:4348 , piece_id=ZJXZKILKMDTWN7ZFW67F3CGZIQHA7OYXQIONNW3CG72WX57ZIQQQ, seg_pos={0 0}, upload=196876]: Done uploading piece. err: ecclient: upload failed (node:12ezboNAWPbaK9UDENYoBb3iHjNU2ahn7mikiteN7y3Aj6uw4Nx, address:104.36.82.60:4348): protocol: read tcp 188.166.225.170:45622->104.36.82.60:4348: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:40:53.062589825 (189h14m47.493669956s) [noise=true , address=77.90.31.21:4902 , piece_id=WUTMD3HBQ3OUMVOIRSDYOWEQAVV7MIEOONMWDDCKXRXVEZ7UH6WQ, seg_pos={0 0}, upload=196876]: Done uploading piece. err: ecclient: upload failed (node:12sD3HFEaYLm7KV6uRvjbojtU9eQGMSy3Aewr4pX5Bxjy4jMCHx, address:77.90.31.21:4902): protocol: read tcp 188.166.225.170:59354->77.90.31.21:4902: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:40:53.062605967 (189h14m47.493686098s) [noise=true , address=104.36.82.44:4884 , piece_id=2VREOXCLUGM2FUX7VRQRHZID7CSMAJZ6JRWLOJ4GKREJ4OHUMZOA, seg_pos={0 0}, upload=196876]: Done uploading piece. err: ecclient: upload failed (node:1qGXdnm83P4eMnCtZdRt2x5yAWP7pQafWwoPG72daZJVR9pyKG, address:104.36.82.44:4884): protocol: read tcp 188.166.225.170:54416->104.36.82.44:4884: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:40:53.062642737 (189h14m47.493722868s) [seg_pos={0 0}, upload=196876]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil> Aug 16 14:40:53.290772077 (189h14m47.721852218s) [upload=196876]: Done uploading segments... Aug 16 14:40:53.290798057 (189h14m47.721878198s) [upload=196876]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9) Duration: 40.44788095s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [.........................................X........XXX..XXXXX] duration: 30.001317859s FAIL (2025-08-16 14:41:40 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.001335604s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 5 out of 60 is failed (threshold=3) [....................................................XX..XXX.] duration: 5.604038037s FAIL (2025-08-16 14:41:32 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.604057945s edge gateway us-select-1
checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 24.061949167s FAIL (2025-08-16 14:41:31 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": dial tcp 45.38.24.2:443: i/o timeout Duration: 24.061968333s edge authservice http us-select-1
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway) antiblip: 4 out of 60 is failed (threshold=3) [.....................................................X.XXX..] duration: 13.112695325s target: https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410 FAIL (2025-08-16 14:41:29 UTC) orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": dial tcp 45.38.24.2:443: i/o timeout Duration: 13.112722116s edge gateway vhost us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 8 out of 60 is failed (threshold=3) [...X...X...........X................................X.X.XXX.] duration: 17.083193167s FAIL (2025-08-16 14:41:27 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 9) ======LOGS====== Aug 16 14:40:10.271545867 (189h14m4.702625998s) [upload=196880]: Starting upload Aug 16 14:40:10.271612004 (189h14m4.702692135s) [upload=196880]: Done starting upload Aug 16 14:40:10.271706525 (189h14m4.702786646s) [upload=196880]: Uploading segments... Aug 16 14:40:10.271816746 (189h14m4.702896877s) [upload=196880]: Got next segment. Inline: false Aug 16 14:40:10.271855299 (189h14m4.70293543s) [upload=196880]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 14:40:10.271860169 (189h14m4.7029403s) [upload=196880]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 14:40:11.050260955 (189h14m5.481341086s) [seg_pos={0 0}, upload=196880]: Begin upload segment... Aug 16 14:40:11.050307854 (189h14m5.481387985s) [seg_pos={0 0}, upload=196880]: Done begin upload segment. Aug 16 14:40:11.050332451 (189h14m5.481412572s) [noise=true , address=104.36.82.18:4396 , piece_id=WKO7CZ7UW7TQ63KZIEQKQDGIX4PO2ACHQ55S5PMTSHMINFJWFJXA, seg_pos={0 0}, upload=196880]: Uploading piece... Aug 16 14:40:11.050366005 (189h14m5.481446136s) [noise=true , address=45.91.116.7:4014 , piece_id=FYDSCQXNMKJ4OLB3CBIDP5JETT2Y3TAQALPDCOATXMXLHHP25TYQ, seg_pos={0 0}, upload=196880]: Uploading piece... Aug 16 14:40:11.050812209 (189h14m5.48189234s) [noise=true , address=104.36.82.5:4847 , piece_id=XPSL4LVA4HYAKH3AAVDFKBVKLNUOXU3ETNGYP76YRMPCI66IY7TQ, seg_pos={0 0}, upload=196880]: Uploading piece... Aug 16 14:40:11.050899216 (189h14m5.481979347s) [noise=true , address=104.36.82.2:4753 , piece_id=AHUD4Q62NH5RYTZ4TMBOIQVPSOGQXNEMEQ64MQDWNEBN7DFB5N4Q, seg_pos={0 0}, upload=196880]: Uploading piece... Aug 16 14:40:11.051100741 (189h14m5.482180872s) [noise=true , address=66.23.193.175:4561 , piece_id=V6UVFCDFYHZZFZ3JSAHIHX3IHV2ACWLCGXIYEAN3S6XCFPOAQRIQ, seg_pos={0 0}, upload=196880]: Uploading piece... Aug 16 14:40:11.051246319 (189h14m5.48232645s) [noise=true , address=45.91.116.10:4110 , piece_id=MJBFOL7WU6U53FNEDCP6RSHWZ7R7QMV5CPSXB5BHLLLHGPP7X22A, seg_pos={0 0}, upload=196880]: Uploading piece... Aug 16 14:40:11.051375637 (189h14m5.482455768s) [noise=true , address=104.36.82.23:4464 , piece_id=EF2Q3MPPE6LN4YHTJ7D54XJIO6NFP3756PPSOWA2OFXPVAL2CSOQ, seg_pos={0 0}, upload=196880]: Uploading piece... Aug 16 14:40:11.051564248 (189h14m5.482644379s) [noise=true , address=104.36.82.58:4271 , piece_id=SQHWRTQHUJSBDDOVKSNNPCZZJTBLF77C54ORF333HRRP27C3FYRQ, seg_pos={0 0}, upload=196880]: Uploading piece... Aug 16 14:40:11.051656875 (189h14m5.482737006s) [noise=true , address=104.36.82.60:4339 , piece_id=W2YLIEIA4NCOADOTSURQXATJLYJX3A56LMKHEE4THPSZDQ2S5KMQ, seg_pos={0 0}, upload=196880]: Uploading piece... Aug 16 14:40:11.051875734 (189h14m5.482955865s) [noise=true , address=77.90.31.12:4593 , piece_id=BARZUGVNKOZINZDADT63F23HKAET5JTCXXP4JNWKHZFSRS3UJ4MQ, seg_pos={0 0}, upload=196880]: Uploading piece... Aug 16 14:40:11.051977859 (189h14m5.48305799s) [noise=true , address=156.146.43.225:4457 , piece_id=DSGCCX7H2WMVO5OBL42RHOODN2C3Y5V5EQ7HMJMMY6SFZSOZNK7Q, seg_pos={0 0}, upload=196880]: Uploading piece... Aug 16 14:40:11.052194885 (189h14m5.483275016s) [noise=true , address=79.127.201.215:4359 , piece_id=ZAASOGPH4FH2JSF37Y3ENYUFYDEZW3YL6LYUOBJIYYIQGMM4HCCQ, seg_pos={0 0}, upload=196880]: Uploading piece... Aug 16 14:40:11.052241494 (189h14m5.483321625s) [noise=true , address=77.90.31.21:4888 , piece_id=23OKO3PQIDSXNKN4RFSHJJQZ3T3EV4HPYVCUPON344C2SKYA6HGQ, seg_pos={0 0}, upload=196880]: Uploading piece... Aug 16 14:40:11.052607734 (189h14m5.483687866s) [noise=true , address=104.36.82.39:4744 , piece_id=KWNX53ZFT3CQ3P55KXCIKNKSEFY2C74HQLJTR5YTJPVBPJRA7DNQ, seg_pos={0 0}, upload=196880]: Uploading piece... Aug 16 14:40:11.053028068 (189h14m5.484108199s) [noise=true , address=104.36.82.44:4881 , piece_id=XQIHTBFY6EQTOP7W4ZLCZ2IULVLUJ22JF3BWKDUDC4WYO3I3V3WA, seg_pos={0 0}, upload=196880]: Uploading piece... Aug 16 14:40:11.053419859 (189h14m5.48449999s) [noise=true , address=45.91.116.8:4052 , piece_id=WTP26Y3RREPFXX5L7J3LRHANZR3WNHDU53FLV76ON7XE7C5R7VLA, seg_pos={0 0}, upload=196880]: Uploading piece... Aug 16 14:40:11.053755751 (189h14m5.484835882s) [noise=true , address=45.91.116.11:4128 , piece_id=R76MH6VVKHYBC3RIJ5NFHTFICWSKDSTYSGTOIIWDVXINI7RZ3Z3Q, seg_pos={0 0}, upload=196880]: Uploading piece... Aug 16 14:40:11.054120639 (189h14m5.48520077s) [noise=true , address=104.36.82.3:4796 , piece_id=24XWWCPAXXD7NTKH5YZKV6ALH6BSW6VFQOOWAO4F2TXEBJS5JHTQ, seg_pos={0 0}, upload=196880]: Uploading piece... Aug 16 14:40:12.229792430 (189h14m6.660872561s) [upload=196880]: Next returned nil segment Aug 16 14:40:12.229828298 (189h14m6.660908419s) [upload=196880]: Waiting for error group managing segments... Aug 16 14:40:12.565875869 (189h14m6.996956s) [noise=true , address=66.23.193.175:4561 , piece_id=V6UVFCDFYHZZFZ3JSAHIHX3IHV2ACWLCGXIYEAN3S6XCFPOAQRIQ, seg_pos={0 0}, upload=196880]: Done uploading piece. err: <nil> Aug 16 14:40:13.041908045 (189h14m7.472988166s) [noise=true , address=156.146.43.225:4457 , piece_id=DSGCCX7H2WMVO5OBL42RHOODN2C3Y5V5EQ7HMJMMY6SFZSOZNK7Q, seg_pos={0 0}, upload=196880]: Done uploading piece. err: <nil> Aug 16 14:40:13.059924989 (189h14m7.49100511s) [noise=true , address=104.36.82.44:4881 , piece_id=XQIHTBFY6EQTOP7W4ZLCZ2IULVLUJ22JF3BWKDUDC4WYO3I3V3WA, seg_pos={0 0}, upload=196880]: Done uploading piece. err: <nil> Aug 16 14:40:13.169755056 (189h14m7.600835187s) [noise=true , address=104.36.82.58:4271 , piece_id=SQHWRTQHUJSBDDOVKSNNPCZZJTBLF77C54ORF333HRRP27C3FYRQ, seg_pos={0 0}, upload=196880]: Done uploading piece. err: <nil> Aug 16 14:40:25.072613187 (189h14m19.503693318s) [noise=true , address=79.127.201.215:4359 , piece_id=ZAASOGPH4FH2JSF37Y3ENYUFYDEZW3YL6LYUOBJIYYIQGMM4HCCQ, seg_pos={0 0}, upload=196880]: Done uploading piece. err: <nil> Aug 16 14:40:26.679539187 (189h14m21.110619308s) [noise=true , address=104.36.82.39:4744 , piece_id=KWNX53ZFT3CQ3P55KXCIKNKSEFY2C74HQLJTR5YTJPVBPJRA7DNQ, seg_pos={0 0}, upload=196880]: Done uploading piece. err: <nil> Aug 16 14:40:27.717184137 (189h14m22.148264268s) [noise=true , address=104.36.82.23:4464 , piece_id=EF2Q3MPPE6LN4YHTJ7D54XJIO6NFP3756PPSOWA2OFXPVAL2CSOQ, seg_pos={0 0}, upload=196880]: Done uploading piece. err: <nil> Aug 16 14:40:36.160947329 (189h14m30.59202745s) [noise=true , address=45.91.116.10:4110 , piece_id=MJBFOL7WU6U53FNEDCP6RSHWZ7R7QMV5CPSXB5BHLLLHGPP7X22A, seg_pos={0 0}, upload=196880]: Done uploading piece. err: <nil> Aug 16 14:41:01.633322263 (189h14m56.064402414s) [noise=true , address=45.91.116.11:4128 , piece_id=R76MH6VVKHYBC3RIJ5NFHTFICWSKDSTYSGTOIIWDVXINI7RZ3Z3Q, seg_pos={0 0}, upload=196880]: Done uploading piece. err: <nil> Aug 16 14:41:09.619455088 (189h15m4.050535219s) [noise=true , address=45.91.116.8:4052 , piece_id=WTP26Y3RREPFXX5L7J3LRHANZR3WNHDU53FLV76ON7XE7C5R7VLA, seg_pos={0 0}, upload=196880]: Done uploading piece. err: ecclient: upload failed (node:1HHEiq4G4TCaHFJAN3DhL8SwXAFqcwRgYteSHmr1gNe2ThcmnS, address:45.91.116.8:4052): protocol: expected piece hash; context deadline exceeded; EOF Aug 16 14:41:09.619762696 (189h15m4.050842827s) [noise=true , address=104.36.82.18:4396 , piece_id=WKO7CZ7UW7TQ63KZIEQKQDGIX4PO2ACHQ55S5PMTSHMINFJWFJXA, seg_pos={0 0}, upload=196880]: Done uploading piece. err: ecclient: upload failed (node:1k5gzJBGWhpARcfiynburgfTqJ2t7sVNs1mAkUfToMG5saxPqx, address:104.36.82.18:4396): protocol: read tcp 188.166.225.170:58918->104.36.82.18:4396: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:41:09.619779719 (189h15m4.05085985s) [noise=true , address=104.36.82.5:4847 , piece_id=XPSL4LVA4HYAKH3AAVDFKBVKLNUOXU3ETNGYP76YRMPCI66IY7TQ, seg_pos={0 0}, upload=196880]: Done uploading piece. err: ecclient: upload failed (node:12aAq1Axcts5JgFR2ve4AmPACyCprGsyzM1PERjTYeNM4xeBH1i, address:104.36.82.5:4847): protocol: write tcp 188.166.225.170:45564->104.36.82.5:4847: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:41:09.619822731 (189h15m4.050902862s) [noise=true , address=104.36.82.2:4753 , piece_id=AHUD4Q62NH5RYTZ4TMBOIQVPSOGQXNEMEQ64MQDWNEBN7DFB5N4Q, seg_pos={0 0}, upload=196880]: Done uploading piece. err: ecclient: upload failed (node:12RvrzrVY3N4FD2iNFBkHYBgitmrfcrLreLN6mJARGobdpbgtGu, address:104.36.82.2:4753): protocol: read tcp 188.166.225.170:43476->104.36.82.2:4753: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:41:09.619861365 (189h15m4.050941496s) [noise=true , address=45.91.116.7:4014 , piece_id=FYDSCQXNMKJ4OLB3CBIDP5JETT2Y3TAQALPDCOATXMXLHHP25TYQ, seg_pos={0 0}, upload=196880]: Done uploading piece. err: ecclient: upload failed (node:1yVYauHmJiuAd6ZE6CfQvqZY2MzJTZnNA69CWLm9caDiX2fjDH, address:45.91.116.7:4014): protocol: write tcp 188.166.225.170:48072->45.91.116.7:4014: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:41:09.619868599 (189h15m4.05094873s) [noise=true , address=104.36.82.60:4339 , piece_id=W2YLIEIA4NCOADOTSURQXATJLYJX3A56LMKHEE4THPSZDQ2S5KMQ, seg_pos={0 0}, upload=196880]: Done uploading piece. err: ecclient: upload failed (node:124ACRNWgJYWgCVsDkXchmT47DiUcgvtZcdGo2MzyVqZkPjyz1R, address:104.36.82.60:4339): protocol: read tcp 188.166.225.170:44952->104.36.82.60:4339: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:41:09.619894288 (189h15m4.050974419s) [noise=true , address=77.90.31.21:4888 , piece_id=23OKO3PQIDSXNKN4RFSHJJQZ3T3EV4HPYVCUPON344C2SKYA6HGQ, seg_pos={0 0}, upload=196880]: Done uploading piece. err: ecclient: upload failed (node:12BiqwcDuvobZabx3jxDhXf27Ei8oLFS6wLp3qbpZCtJ4ZhQEZA, address:77.90.31.21:4888): protocol: read tcp 188.166.225.170:43064->77.90.31.21:4888: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:41:09.619921740 (189h15m4.051001861s) [noise=true , address=104.36.82.3:4796 , piece_id=24XWWCPAXXD7NTKH5YZKV6ALH6BSW6VFQOOWAO4F2TXEBJS5JHTQ, seg_pos={0 0}, upload=196880]: Done uploading piece. err: ecclient: upload failed (node:12nZwQ4Poz9M14ijFV6qpy5SCWEMeyHQpxN17uSBTH1ts1CFtVS, address:104.36.82.3:4796): protocol: read tcp 188.166.225.170:33536->104.36.82.3:4796: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:41:09.619928152 (189h15m4.051008274s) [noise=true , address=77.90.31.12:4593 , piece_id=BARZUGVNKOZINZDADT63F23HKAET5JTCXXP4JNWKHZFSRS3UJ4MQ, seg_pos={0 0}, upload=196880]: Done uploading piece. err: ecclient: upload failed (node:1kSxoZb9nmG6oGNytpFTsoiGh3ZPmebVVe8TZxGzANaw7EfdeF, address:77.90.31.12:4593): protocol: read tcp 188.166.225.170:59294->77.90.31.12:4593: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:41:09.619946919 (189h15m4.05102705s) [seg_pos={0 0}, upload=196880]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil> Aug 16 14:41:10.085434638 (189h15m4.516514769s) [upload=196880]: Done uploading segments... Aug 16 14:41:10.085459245 (189h15m4.516539376s) [upload=196880]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9) Duration: 17.08321023s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 4 out of 60 is failed (threshold=3) [.........X...............X............................X....X] duration: 1.873275016s FAIL (2025-08-16 14:41:27 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/1K):: after 5b4e9c59e1b603b8baec6a7847bc559340eec1a18700944fbb5f0cb717754552 != before 08bb3ceaa2907a9c695559f1a049d1db95ca5e65c937769756841140f9b12378 ======LOGS====== Aug 16 14:41:26.568660143 (189h15m20.999740275s) [upload=196903]: Starting upload Aug 16 14:41:26.568720509 (189h15m20.99980064s) [upload=196903]: Done starting upload Aug 16 14:41:26.568786014 (189h15m20.999866155s) [upload=196903]: Uploading segments... Aug 16 14:41:26.568826642 (189h15m20.999906773s) [upload=196903]: Got next segment. Inline: true Aug 16 14:41:26.568831321 (189h15m20.999911452s) [upload=196903]: Waiting for error group managing segments... Aug 16 14:41:26.568870796 (189h15m20.999950957s) [upload=196903]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 14:41:26.568875666 (189h15m20.999955807s) [upload=196903]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 16 14:41:26.568879223 (189h15m20.999959364s) [upload=196903]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 14:41:27.069024505 (189h15m21.500104636s) [upload=196903]: Done uploading segments... Aug 16 14:41:27.069041257 (189h15m21.500121388s) [upload=196903]: Upload finished. err: <nil> Duration:* 1.87329807s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect) antiblip: 1 out of 15 is failed (threshold=6) [.............X.] duration: 859.398206ms remote: "tcp:3.109.243.18:443" response headers: Date=Sat, 16 Aug 2025 14:40:59 GMT Content-Type=text/html Content-Length=164 X-Cluster-Name=ap-south-1-prod-hosting-red target: https://www.storj.io/dcs-satellites FAIL (2025-08-16 14:41:16 UTC) orbiter warning: unexpected status code "504 Gateway Time-out" "<html>\r\n<head><title>504 Gateway Time-out</title></head>\r\n<body>\r\n<center><h1>504 Gateway Time-out</h1></center>\r\n<hr><center>openresty</center>\r\n</body>\r\n</html>\r\n" Duration: 859.431789ms core website
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 5 out of 60 is failed (threshold=3) [.....................................................XX..XXX] duration: 30.000977541s FAIL (2025-08-16 14:41:11 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.001005014s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 8 out of 60 is failed (threshold=3) [....X...X...........X................................X.X.XXX] duration: 1m0.467841951s FAIL (2025-08-16 14:41:10 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 9) ======LOGS====== Aug 16 14:40:10.271545867 (189h14m4.702625998s) [upload=196880]: Starting upload Aug 16 14:40:10.271612004 (189h14m4.702692135s) [upload=196880]: Done starting upload Aug 16 14:40:10.271706525 (189h14m4.702786646s) [upload=196880]: Uploading segments... Aug 16 14:40:10.271816746 (189h14m4.702896877s) [upload=196880]: Got next segment. Inline: false Aug 16 14:40:10.271855299 (189h14m4.70293543s) [upload=196880]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 14:40:10.271860169 (189h14m4.7029403s) [upload=196880]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 14:40:11.050260955 (189h14m5.481341086s) [seg_pos={0 0}, upload=196880]: Begin upload segment... Aug 16 14:40:11.050307854 (189h14m5.481387985s) [seg_pos={0 0}, upload=196880]: Done begin upload segment. Aug 16 14:40:11.050332451 (189h14m5.481412572s) [noise=true , address=104.36.82.18:4396 , piece_id=WKO7CZ7UW7TQ63KZIEQKQDGIX4PO2ACHQ55S5PMTSHMINFJWFJXA, seg_pos={0 0}, upload=196880]: Uploading piece... Aug 16 14:40:11.050366005 (189h14m5.481446136s) [noise=true , address=45.91.116.7:4014 , piece_id=FYDSCQXNMKJ4OLB3CBIDP5JETT2Y3TAQALPDCOATXMXLHHP25TYQ, seg_pos={0 0}, upload=196880]: Uploading piece... Aug 16 14:40:11.050812209 (189h14m5.48189234s) [noise=true , address=104.36.82.5:4847 , piece_id=XPSL4LVA4HYAKH3AAVDFKBVKLNUOXU3ETNGYP76YRMPCI66IY7TQ, seg_pos={0 0}, upload=196880]: Uploading piece... Aug 16 14:40:11.050899216 (189h14m5.481979347s) [noise=true , address=104.36.82.2:4753 , piece_id=AHUD4Q62NH5RYTZ4TMBOIQVPSOGQXNEMEQ64MQDWNEBN7DFB5N4Q, seg_pos={0 0}, upload=196880]: Uploading piece... Aug 16 14:40:11.051100741 (189h14m5.482180872s) [noise=true , address=66.23.193.175:4561 , piece_id=V6UVFCDFYHZZFZ3JSAHIHX3IHV2ACWLCGXIYEAN3S6XCFPOAQRIQ, seg_pos={0 0}, upload=196880]: Uploading piece... Aug 16 14:40:11.051246319 (189h14m5.48232645s) [noise=true , address=45.91.116.10:4110 , piece_id=MJBFOL7WU6U53FNEDCP6RSHWZ7R7QMV5CPSXB5BHLLLHGPP7X22A, seg_pos={0 0}, upload=196880]: Uploading piece... Aug 16 14:40:11.051375637 (189h14m5.482455768s) [noise=true , address=104.36.82.23:4464 , piece_id=EF2Q3MPPE6LN4YHTJ7D54XJIO6NFP3756PPSOWA2OFXPVAL2CSOQ, seg_pos={0 0}, upload=196880]: Uploading piece... Aug 16 14:40:11.051564248 (189h14m5.482644379s) [noise=true , address=104.36.82.58:4271 , piece_id=SQHWRTQHUJSBDDOVKSNNPCZZJTBLF77C54ORF333HRRP27C3FYRQ, seg_pos={0 0}, upload=196880]: Uploading piece... Aug 16 14:40:11.051656875 (189h14m5.482737006s) [noise=true , address=104.36.82.60:4339 , piece_id=W2YLIEIA4NCOADOTSURQXATJLYJX3A56LMKHEE4THPSZDQ2S5KMQ, seg_pos={0 0}, upload=196880]: Uploading piece... Aug 16 14:40:11.051875734 (189h14m5.482955865s) [noise=true , address=77.90.31.12:4593 , piece_id=BARZUGVNKOZINZDADT63F23HKAET5JTCXXP4JNWKHZFSRS3UJ4MQ, seg_pos={0 0}, upload=196880]: Uploading piece... Aug 16 14:40:11.051977859 (189h14m5.48305799s) [noise=true , address=156.146.43.225:4457 , piece_id=DSGCCX7H2WMVO5OBL42RHOODN2C3Y5V5EQ7HMJMMY6SFZSOZNK7Q, seg_pos={0 0}, upload=196880]: Uploading piece... Aug 16 14:40:11.052194885 (189h14m5.483275016s) [noise=true , address=79.127.201.215:4359 , piece_id=ZAASOGPH4FH2JSF37Y3ENYUFYDEZW3YL6LYUOBJIYYIQGMM4HCCQ, seg_pos={0 0}, upload=196880]: Uploading piece... Aug 16 14:40:11.052241494 (189h14m5.483321625s) [noise=true , address=77.90.31.21:4888 , piece_id=23OKO3PQIDSXNKN4RFSHJJQZ3T3EV4HPYVCUPON344C2SKYA6HGQ, seg_pos={0 0}, upload=196880]: Uploading piece... Aug 16 14:40:11.052607734 (189h14m5.483687866s) [noise=true , address=104.36.82.39:4744 , piece_id=KWNX53ZFT3CQ3P55KXCIKNKSEFY2C74HQLJTR5YTJPVBPJRA7DNQ, seg_pos={0 0}, upload=196880]: Uploading piece... Aug 16 14:40:11.053028068 (189h14m5.484108199s) [noise=true , address=104.36.82.44:4881 , piece_id=XQIHTBFY6EQTOP7W4ZLCZ2IULVLUJ22JF3BWKDUDC4WYO3I3V3WA, seg_pos={0 0}, upload=196880]: Uploading piece... Aug 16 14:40:11.053419859 (189h14m5.48449999s) [noise=true , address=45.91.116.8:4052 , piece_id=WTP26Y3RREPFXX5L7J3LRHANZR3WNHDU53FLV76ON7XE7C5R7VLA, seg_pos={0 0}, upload=196880]: Uploading piece... Aug 16 14:40:11.053755751 (189h14m5.484835882s) [noise=true , address=45.91.116.11:4128 , piece_id=R76MH6VVKHYBC3RIJ5NFHTFICWSKDSTYSGTOIIWDVXINI7RZ3Z3Q, seg_pos={0 0}, upload=196880]: Uploading piece... Aug 16 14:40:11.054120639 (189h14m5.48520077s) [noise=true , address=104.36.82.3:4796 , piece_id=24XWWCPAXXD7NTKH5YZKV6ALH6BSW6VFQOOWAO4F2TXEBJS5JHTQ, seg_pos={0 0}, upload=196880]: Uploading piece... Aug 16 14:40:12.229792430 (189h14m6.660872561s) [upload=196880]: Next returned nil segment Aug 16 14:40:12.229828298 (189h14m6.660908419s) [upload=196880]: Waiting for error group managing segments... Aug 16 14:40:12.565875869 (189h14m6.996956s) [noise=true , address=66.23.193.175:4561 , piece_id=V6UVFCDFYHZZFZ3JSAHIHX3IHV2ACWLCGXIYEAN3S6XCFPOAQRIQ, seg_pos={0 0}, upload=196880]: Done uploading piece. err: <nil> Aug 16 14:40:13.041908045 (189h14m7.472988166s) [noise=true , address=156.146.43.225:4457 , piece_id=DSGCCX7H2WMVO5OBL42RHOODN2C3Y5V5EQ7HMJMMY6SFZSOZNK7Q, seg_pos={0 0}, upload=196880]: Done uploading piece. err: <nil> Aug 16 14:40:13.059924989 (189h14m7.49100511s) [noise=true , address=104.36.82.44:4881 , piece_id=XQIHTBFY6EQTOP7W4ZLCZ2IULVLUJ22JF3BWKDUDC4WYO3I3V3WA, seg_pos={0 0}, upload=196880]: Done uploading piece. err: <nil> Aug 16 14:40:13.169755056 (189h14m7.600835187s) [noise=true , address=104.36.82.58:4271 , piece_id=SQHWRTQHUJSBDDOVKSNNPCZZJTBLF77C54ORF333HRRP27C3FYRQ, seg_pos={0 0}, upload=196880]: Done uploading piece. err: <nil> Aug 16 14:40:25.072613187 (189h14m19.503693318s) [noise=true , address=79.127.201.215:4359 , piece_id=ZAASOGPH4FH2JSF37Y3ENYUFYDEZW3YL6LYUOBJIYYIQGMM4HCCQ, seg_pos={0 0}, upload=196880]: Done uploading piece. err: <nil> Aug 16 14:40:26.679539187 (189h14m21.110619308s) [noise=true , address=104.36.82.39:4744 , piece_id=KWNX53ZFT3CQ3P55KXCIKNKSEFY2C74HQLJTR5YTJPVBPJRA7DNQ, seg_pos={0 0}, upload=196880]: Done uploading piece. err: <nil> Aug 16 14:40:27.717184137 (189h14m22.148264268s) [noise=true , address=104.36.82.23:4464 , piece_id=EF2Q3MPPE6LN4YHTJ7D54XJIO6NFP3756PPSOWA2OFXPVAL2CSOQ, seg_pos={0 0}, upload=196880]: Done uploading piece. err: <nil> Aug 16 14:40:36.160947329 (189h14m30.59202745s) [noise=true , address=45.91.116.10:4110 , piece_id=MJBFOL7WU6U53FNEDCP6RSHWZ7R7QMV5CPSXB5BHLLLHGPP7X22A, seg_pos={0 0}, upload=196880]: Done uploading piece. err: <nil> Aug 16 14:41:01.633322263 (189h14m56.064402414s) [noise=true , address=45.91.116.11:4128 , piece_id=R76MH6VVKHYBC3RIJ5NFHTFICWSKDSTYSGTOIIWDVXINI7RZ3Z3Q, seg_pos={0 0}, upload=196880]: Done uploading piece. err: <nil> Aug 16 14:41:09.619455088 (189h15m4.050535219s) [noise=true , address=45.91.116.8:4052 , piece_id=WTP26Y3RREPFXX5L7J3LRHANZR3WNHDU53FLV76ON7XE7C5R7VLA, seg_pos={0 0}, upload=196880]: Done uploading piece. err: ecclient: upload failed (node:1HHEiq4G4TCaHFJAN3DhL8SwXAFqcwRgYteSHmr1gNe2ThcmnS, address:45.91.116.8:4052): protocol: expected piece hash; context deadline exceeded; EOF Aug 16 14:41:09.619762696 (189h15m4.050842827s) [noise=true , address=104.36.82.18:4396 , piece_id=WKO7CZ7UW7TQ63KZIEQKQDGIX4PO2ACHQ55S5PMTSHMINFJWFJXA, seg_pos={0 0}, upload=196880]: Done uploading piece. err: ecclient: upload failed (node:1k5gzJBGWhpARcfiynburgfTqJ2t7sVNs1mAkUfToMG5saxPqx, address:104.36.82.18:4396): protocol: read tcp 188.166.225.170:58918->104.36.82.18:4396: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:41:09.619779719 (189h15m4.05085985s) [noise=true , address=104.36.82.5:4847 , piece_id=XPSL4LVA4HYAKH3AAVDFKBVKLNUOXU3ETNGYP76YRMPCI66IY7TQ, seg_pos={0 0}, upload=196880]: Done uploading piece. err: ecclient: upload failed (node:12aAq1Axcts5JgFR2ve4AmPACyCprGsyzM1PERjTYeNM4xeBH1i, address:104.36.82.5:4847): protocol: write tcp 188.166.225.170:45564->104.36.82.5:4847: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:41:09.619822731 (189h15m4.050902862s) [noise=true , address=104.36.82.2:4753 , piece_id=AHUD4Q62NH5RYTZ4TMBOIQVPSOGQXNEMEQ64MQDWNEBN7DFB5N4Q, seg_pos={0 0}, upload=196880]: Done uploading piece. err: ecclient: upload failed (node:12RvrzrVY3N4FD2iNFBkHYBgitmrfcrLreLN6mJARGobdpbgtGu, address:104.36.82.2:4753): protocol: read tcp 188.166.225.170:43476->104.36.82.2:4753: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:41:09.619861365 (189h15m4.050941496s) [noise=true , address=45.91.116.7:4014 , piece_id=FYDSCQXNMKJ4OLB3CBIDP5JETT2Y3TAQALPDCOATXMXLHHP25TYQ, seg_pos={0 0}, upload=196880]: Done uploading piece. err: ecclient: upload failed (node:1yVYauHmJiuAd6ZE6CfQvqZY2MzJTZnNA69CWLm9caDiX2fjDH, address:45.91.116.7:4014): protocol: write tcp 188.166.225.170:48072->45.91.116.7:4014: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:41:09.619868599 (189h15m4.05094873s) [noise=true , address=104.36.82.60:4339 , piece_id=W2YLIEIA4NCOADOTSURQXATJLYJX3A56LMKHEE4THPSZDQ2S5KMQ, seg_pos={0 0}, upload=196880]: Done uploading piece. err: ecclient: upload failed (node:124ACRNWgJYWgCVsDkXchmT47DiUcgvtZcdGo2MzyVqZkPjyz1R, address:104.36.82.60:4339): protocol: read tcp 188.166.225.170:44952->104.36.82.60:4339: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:41:09.619894288 (189h15m4.050974419s) [noise=true , address=77.90.31.21:4888 , piece_id=23OKO3PQIDSXNKN4RFSHJJQZ3T3EV4HPYVCUPON344C2SKYA6HGQ, seg_pos={0 0}, upload=196880]: Done uploading piece. err: ecclient: upload failed (node:12BiqwcDuvobZabx3jxDhXf27Ei8oLFS6wLp3qbpZCtJ4ZhQEZA, address:77.90.31.21:4888): protocol: read tcp 188.166.225.170:43064->77.90.31.21:4888: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:41:09.619921740 (189h15m4.051001861s) [noise=true , address=104.36.82.3:4796 , piece_id=24XWWCPAXXD7NTKH5YZKV6ALH6BSW6VFQOOWAO4F2TXEBJS5JHTQ, seg_pos={0 0}, upload=196880]: Done uploading piece. err: ecclient: upload failed (node:12nZwQ4Poz9M14ijFV6qpy5SCWEMeyHQpxN17uSBTH1ts1CFtVS, address:104.36.82.3:4796): protocol: read tcp 188.166.225.170:33536->104.36.82.3:4796: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:41:09.619928152 (189h15m4.051008274s) [noise=true , address=77.90.31.12:4593 , piece_id=BARZUGVNKOZINZDADT63F23HKAET5JTCXXP4JNWKHZFSRS3UJ4MQ, seg_pos={0 0}, upload=196880]: Done uploading piece. err: ecclient: upload failed (node:1kSxoZb9nmG6oGNytpFTsoiGh3ZPmebVVe8TZxGzANaw7EfdeF, address:77.90.31.12:4593): protocol: read tcp 188.166.225.170:59294->77.90.31.12:4593: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:41:09.619946919 (189h15m4.05102705s) [seg_pos={0 0}, upload=196880]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil> Aug 16 14:41:10.085434638 (189h15m4.516514769s) [upload=196880]: Done uploading segments... Aug 16 14:41:10.085459245 (189h15m4.516539376s) [upload=196880]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9) Duration: 1m0.467852591s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect) antiblip: 1 out of 15 is failed (threshold=6) [..............X] duration: 20.22467836s remote: "tcp:3.109.243.18:443" response headers: Date=Sat, 16 Aug 2025 14:40:59 GMT Content-Type=text/html Content-Length=164 X-Cluster-Name=ap-south-1-prod-hosting-red target: https://www.storj.io/dcs-satellites FAIL (2025-08-16 14:40:59 UTC) orbiter warning: unexpected status code "504 Gateway Time-out" "<html>\r\n<head><title>504 Gateway Time-out</title></head>\r\n<body>\r\n<center><h1>504 Gateway Time-out</h1></center>\r\n<hr><center>openresty</center>\r\n</body>\r\n</html>\r\n" Duration: 20.224700963s core website
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [...................................................XXX..XXXX] duration: 30.237037452s FAIL (2025-08-16 14:40:58 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 4) ======LOGS====== e... Aug 16 14:40:29.468725491 (189h14m23.899805682s) [noise=true , address=45.91.116.4:4937 , piece_id=HYOON7ERXMTIPU2SBOBT7JQA4GQOPDEHUR2QGOSAG2XUJNQ4DFEQ, seg_pos={0 0}, upload=196886]: Uploading piece... Aug 16 14:40:29.468746050 (189h14m23.899826211s) [noise=true , address=45.91.116.12:4227 , piece_id=CEWVXSN46MQ2ZVQD2PZJXIBQPA6BJNB2U4SWTIBBEAFWUIUR5GTQ, seg_pos={0 0}, upload=196886]: Uploading piece... Aug 16 14:40:29.469299749 (189h14m23.90037987s) [noise=true , address=104.36.82.6:4874 , piece_id=3RO4W7VSMFI62UDRYXARWIXWC7WEHVSFT7SNGJLE4O643AVNZQAQ, seg_pos={0 0}, upload=196886]: Uploading piece... Aug 16 14:40:29.469765871 (189h14m23.900846012s) [noise=true , address=45.91.116.14:4190 , piece_id=OPS252QRO5QER5LDPXCON4MUOZT36ZGXXRSSAJDHQETOEZHIQP3Q, seg_pos={0 0}, upload=196886]: Uploading piece... Aug 16 14:40:29.470191475 (189h14m23.901271616s) [noise=true , address=77.90.31.18:4806 , piece_id=7REDPWDFKCFVNEBSZGEVMG2QF5PVZXF5INUCTSNR5Z6DZWNVUF2Q, seg_pos={0 0}, upload=196886]: Uploading piece... Aug 16 14:40:29.470583456 (189h14m23.901663597s) [noise=true , address=104.36.82.11:4115 , piece_id=HRJ2ZZLK4D6BFH2YOTOOGWPZ6NCVYU2CLCITD4ATKMDQHWVR46PA, seg_pos={0 0}, upload=196886]: Uploading piece... Aug 16 14:40:29.471051311 (189h14m23.902131452s) [noise=true , address=104.36.82.16:4294 , piece_id=BWKFWUNZ63NHRWXZLEVDSRO5V7VX6D6KTNQGYGQPRXDVW3CU7CEQ, seg_pos={0 0}, upload=196886]: Uploading piece... Aug 16 14:40:29.471471575 (189h14m23.902551706s) [noise=true , address=45.91.116.5:4298 , piece_id=WVMCLNWF5LEIAUQCZVAEUBA7MF2LV54JTBDT4HFAEJ446OXR5NHA, seg_pos={0 0}, upload=196886]: Uploading piece... Aug 16 14:40:29.471924180 (189h14m23.903004311s) [noise=true , address=104.36.82.16:4295 , piece_id=7S3MDDRO6ANHLQYNBRRYA42RJZZ3K45JB2AKB22KB3GSPJK5AYJQ, seg_pos={0 0}, upload=196886]: Uploading piece... Aug 16 14:40:29.472377178 (189h14m23.903457309s) [noise=true , address=104.36.82.7:4909 , piece_id=4P632QWBZITDMP4GVRFJJ3TM3KOWSQC75VH6LA3AQACH2GT3G7HA, seg_pos={0 0}, upload=196886]: Uploading piece... Aug 16 14:40:29.472745662 (189h14m23.903825793s) [noise=true , address=104.36.82.9:4007 , piece_id=K7DE6AXXVFTLAFGWOIVVQLP42JLP6CA3DDYGAPILDUROYBRNOQSA, seg_pos={0 0}, upload=196886]: Uploading piece... Aug 16 14:40:29.472973920 (189h14m23.904054051s) [noise=true , address=104.36.82.14:4221 , piece_id=SR7O4PV52H7B2ZGKIWXJGXG7FYBSHIF234QVRAJEVEXE2ORR3FXA, seg_pos={0 0}, upload=196886]: Uploading piece... Aug 16 14:40:29.473731238 (189h14m23.904811369s) [noise=true , address=104.36.82.13:4186 , piece_id=ROIEESTSMGURGFKCQJII4UCIV3HCCRACR2GII6DPETU4EC2UORAA, seg_pos={0 0}, upload=196886]: Uploading piece... Aug 16 14:40:31.644741351 (189h14m26.075821483s) [upload=196886]: Next returned nil segment Aug 16 14:40:31.644762120 (189h14m26.075842252s) [upload=196886]: Waiting for error group managing segments... Aug 16 14:40:31.973244386 (189h14m26.404324507s) [noise=true , address=104.36.82.6:4874 , piece_id=3RO4W7VSMFI62UDRYXARWIXWC7WEHVSFT7SNGJLE4O643AVNZQAQ, seg_pos={0 0}, upload=196886]: Done uploading piece. err: <nil> Aug 16 14:40:41.873833786 (189h14m36.304913917s) [noise=true , address=104.36.82.13:4186 , piece_id=ROIEESTSMGURGFKCQJII4UCIV3HCCRACR2GII6DPETU4EC2UORAA, seg_pos={0 0}, upload=196886]: Done uploading piece. err: <nil> Aug 16 14:40:44.304388117 (189h14m38.735468248s) [noise=true , address=104.36.82.14:4221 , piece_id=SR7O4PV52H7B2ZGKIWXJGXG7FYBSHIF234QVRAJEVEXE2ORR3FXA, seg_pos={0 0}, upload=196886]: Done uploading piece. err: <nil> Aug 16 14:40:45.752675549 (189h14m40.18375568s) [noise=true , address=104.36.82.11:4115 , piece_id=HRJ2ZZLK4D6BFH2YOTOOGWPZ6NCVYU2CLCITD4ATKMDQHWVR46PA, seg_pos={0 0}, upload=196886]: Done uploading piece. err: <nil> Aug 16 14:40:58.540396148 (189h14m52.971476279s) [noise=true , address=104.36.82.9:4007 , piece_id=K7DE6AXXVFTLAFGWOIVVQLP42JLP6CA3DDYGAPILDUROYBRNOQSA, seg_pos={0 0}, upload=196886]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:104.36.82.9:4007): protocol: expected piece hash; context deadline exceeded; EOF Aug 16 14:40:58.540454079 (189h14m52.971534211s) [noise=true , address=104.36.82.2:4731 , piece_id=LAGAHN2BIQE4H5SGSQDKL2JG45O7HRMY6VHG672I3XSCRHZSZX4Q, seg_pos={0 0}, upload=196886]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:104.36.82.2:4731): protocol: expected piece hash; context deadline exceeded; EOF Aug 16 14:40:58.540492783 (189h14m52.971572914s) [noise=true , address=77.90.31.18:4806 , piece_id=7REDPWDFKCFVNEBSZGEVMG2QF5PVZXF5INUCTSNR5Z6DZWNVUF2Q, seg_pos={0 0}, upload=196886]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: write tcp 188.166.225.170:33040->77.90.31.18:4806: use of closed network connection; write tcp 188.166.225.170:33040->77.90.31.18:4806: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33040->77.90.31.18:4806: use of closed network connection Aug 16 14:40:58.540549972 (189h14m52.971630104s) [noise=true , address=45.91.116.5:4298 , piece_id=WVMCLNWF5LEIAUQCZVAEUBA7MF2LV54JTBDT4HFAEJ446OXR5NHA, seg_pos={0 0}, upload=196886]: Done uploading piece. err: ecclient: upload failed (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, address:45.91.116.5:4298): protocol: write tcp 188.166.225.170:42798->45.91.116.5:4298: use of closed network connection; write tcp 188.166.225.170:42798->45.91.116.5:4298: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42798->45.91.116.5:4298: use of closed network connection Aug 16 14:40:58.540639443 (189h14m52.971719574s) [noise=true , address=104.36.82.13:4185 , piece_id=NLCN3QEYDZPVW6BERCOKGYJNON2V3UXMLJLPJPBY22N5QFY2M24Q, seg_pos={0 0}, upload=196886]: Done uploading piece. err: ecclient: upload failed (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN, address:104.36.82.13:4185): protocol: rpc: dial tcp 104.36.82.13:4185: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:40:58.540660364 (189h14m52.971740495s) [noise=true , address=45.91.116.4:4937 , piece_id=HYOON7ERXMTIPU2SBOBT7JQA4GQOPDEHUR2QGOSAG2XUJNQ4DFEQ, seg_pos={0 0}, upload=196886]: Done uploading piece. err: ecclient: upload failed (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq, address:45.91.116.4:4937): protocol: read tcp 188.166.225.170:58536->45.91.116.4:4937: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:40:58.540700902 (189h14m52.971781033s) [noise=true , address=45.91.116.14:4190 , piece_id=OPS252QRO5QER5LDPXCON4MUOZT36ZGXXRSSAJDHQETOEZHIQP3Q, seg_pos={0 0}, upload=196886]: Done uploading piece. err: ecclient: upload failed (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, address:45.91.116.14:4190): protocol: read tcp 188.166.225.170:33676->45.91.116.14:4190: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:40:58.540715439 (189h14m52.97179557s) [noise=true , address=77.90.31.14:4663 , piece_id=TUO5WJSWELITUQGNQZUQDXZOA6ELD272NVF37JNLQUR2HLOYGHCQ, seg_pos={0 0}, upload=196886]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: read tcp 188.166.225.170:59726->77.90.31.14:4663: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:40:58.540737301 (189h14m52.971817422s) [noise=true , address=104.36.82.16:4295 , piece_id=7S3MDDRO6ANHLQYNBRRYA42RJZZ3K45JB2AKB22KB3GSPJK5AYJQ, seg_pos={0 0}, upload=196886]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:104.36.82.16:4295): protocol: read tcp 188.166.225.170:40058->104.36.82.16:4295: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:40:58.540857220 (189h14m52.971937351s) [noise=true , address=104.36.82.7:4909 , piece_id=4P632QWBZITDMP4GVRFJJ3TM3KOWSQC75VH6LA3AQACH2GT3G7HA, seg_pos={0 0}, upload=196886]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:104.36.82.7:4909): protocol: read tcp 188.166.225.170:35898->104.36.82.7:4909: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:40:58.540893150 (189h14m52.971973291s) [noise=true , address=104.36.82.16:4294 , piece_id=BWKFWUNZ63NHRWXZLEVDSRO5V7VX6D6KTNQGYGQPRXDVW3CU7CEQ, seg_pos={0 0}, upload=196886]: Done uploading piece. err: ecclient: upload failed (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, address:104.36.82.16:4294): protocol: read tcp 188.166.225.170:48946->104.36.82.16:4294: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:40:58.540931202 (189h14m52.972011333s) [noise=true , address=45.91.116.12:4227 , piece_id=CEWVXSN46MQ2ZVQD2PZJXIBQPA6BJNB2U4SWTIBBEAFWUIUR5GTQ, seg_pos={0 0}, upload=196886]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:4227): protocol: read tcp 188.166.225.170:36914->45.91.116.12:4227: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:40:58.540970847 (189h14m52.972050978s) [seg_pos={0 0}, upload=196886]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil> Aug 16 14:40:58.775250768 (189h14m53.206330899s) [upload=196886]: Done uploading segments... Aug 16 14:40:58.775275225 (189h14m53.206355356s) [upload=196886]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4) Duration: 30.237049173s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [.....................................................X.X.XXX] duration: 1m0.230450354s FAIL (2025-08-16 14:40:53 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 9) ======LOGS====== ess=104.36.82.44:4884 , piece_id=2VREOXCLUGM2FUX7VRQRHZID7CSMAJZ6JRWLOJ4GKREJ4OHUMZOA, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.166352764 (189h13m48.597432895s) [noise=true , address=77.90.31.10:4541 , piece_id=HAOVML65R5HZ5HGAFUDBUGB7S3QKJ2DHF3X7AQTMGEXVVFZC3RYA, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.166371479 (189h13m48.59745161s) [noise=true , address=77.90.31.21:4902 , piece_id=WUTMD3HBQ3OUMVOIRSDYOWEQAVV7MIEOONMWDDCKXRXVEZ7UH6WQ, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.166641586 (189h13m48.597721707s) [noise=true , address=104.36.82.23:4463 , piece_id=B2BOEACKYAULFJIJZFCOJQ7C36SOJCKKVCQIP3QYNSBE6YZS66WQ, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.166717972 (189h13m48.597798093s) [noise=true , address=77.90.31.18:4835 , piece_id=XURSIUMVEHTCRCXBWEIZ2ZJCSKTQXCVOQIKXWISDAYMRW7M4NOTQ, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.166907264 (189h13m48.597987395s) [noise=true , address=45.91.116.12:4240 , piece_id=5O3C47NABFGQ2E45ZL67PGJC5F67XNFDBQ7GYKGACMN6ZPAL62XA, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.167135330 (189h13m48.598215461s) [noise=true , address=104.36.82.58:4299 , piece_id=KSQRSQVB6EGT37PKX3YV7QUQTJZSJKEUD3CKCFPCAD6S4NJAL7QA, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.167357465 (189h13m48.598437586s) [noise=true , address=79.127.201.215:4350 , piece_id=6MWZ4MNZXE6KVJ4RLOTRWM24DER64OZW7CMCUPP7RYJQR7LWCTJQ, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.167565724 (189h13m48.598645855s) [noise=true , address=45.91.116.6:4991 , piece_id=6MTNZT65LSI6PXSVMUJHT64GQVMU6CWOVUH2KGOINQIAUSKSDQRQ, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.167779904 (189h13m48.598860035s) [noise=true , address=79.127.220.101:4061 , piece_id=UURYVARKKAUJAAVMCDWQWRB6NKVOHVWT4GEG7B72RZDJ2CI3MLLQ, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.167991478 (189h13m48.599071609s) [noise=true , address=79.127.220.98:4401 , piece_id=6NZAXKDQ6HSBIWHHARLGLUL3QI2NBH6EMGIOYMNUFKJPOBKWDH6A, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.168183566 (189h13m48.599263697s) [noise=true , address=104.36.82.14:4247 , piece_id=G56NHF7M7FCPXDXRXEEADHQ2EZKINRBRED7Q4XI3662EOYODPWAQ, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.168419999 (189h13m48.59950012s) [noise=true , address=45.91.116.7:4017 , piece_id=IO6CYVMTFTQG2SAFKJIWEVJH72MYZA34ZGCT6WZR6PWXPICIQGNQ, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.168687240 (189h13m48.599767371s) [noise=true , address=45.91.116.13:4159 , piece_id=33U4IEPB3Z5GEDQOHFWTAR5ATFJXN236X5Y66R6QNZ2FMK7KTO4A, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.168925556 (189h13m48.600005677s) [noise=true , address=64.49.13.169:4706 , piece_id=5VF6GL7HRPMPQT4FKJ2KQOWPUM6B5LEQZ2SSG62B6LPCAVGBTLBQ, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:54.169130949 (189h13m48.60021107s) [noise=true , address=104.36.82.60:4348 , piece_id=ZJXZKILKMDTWN7ZFW67F3CGZIQHA7OYXQIONNW3CG72WX57ZIQQQ, seg_pos={0 0}, upload=196876]: Uploading piece... Aug 16 14:39:55.311744797 (189h13m49.742824938s) [upload=196876]: Next returned nil segment Aug 16 14:39:55.311772059 (189h13m49.74285219s) [upload=196876]: Waiting for error group managing segments... Aug 16 14:39:55.601879518 (189h13m50.032959639s) [noise=true , address=64.49.13.169:4706 , piece_id=5VF6GL7HRPMPQT4FKJ2KQOWPUM6B5LEQZ2SSG62B6LPCAVGBTLBQ, seg_pos={0 0}, upload=196876]: Done uploading piece. err: <nil> Aug 16 14:39:56.004918829 (189h13m50.435998971s) [noise=true , address=79.127.220.101:4061 , piece_id=UURYVARKKAUJAAVMCDWQWRB6NKVOHVWT4GEG7B72RZDJ2CI3MLLQ, seg_pos={0 0}, upload=196876]: Done uploading piece. err: <nil> Aug 16 14:39:56.027693798 (189h13m50.458773929s) [noise=true , address=79.127.220.98:4401 , piece_id=6NZAXKDQ6HSBIWHHARLGLUL3QI2NBH6EMGIOYMNUFKJPOBKWDH6A, seg_pos={0 0}, upload=196876]: Done uploading piece. err: <nil> Aug 16 14:39:57.572686072 (189h13m52.003766203s) [noise=true , address=45.91.116.12:4240 , piece_id=5O3C47NABFGQ2E45ZL67PGJC5F67XNFDBQ7GYKGACMN6ZPAL62XA, seg_pos={0 0}, upload=196876]: Done uploading piece. err: <nil> Aug 16 14:39:58.673547510 (189h13m53.104627681s) [noise=true , address=77.90.31.10:4541 , piece_id=HAOVML65R5HZ5HGAFUDBUGB7S3QKJ2DHF3X7AQTMGEXVVFZC3RYA, seg_pos={0 0}, upload=196876]: Done uploading piece. err: <nil> Aug 16 14:40:26.090374323 (189h14m20.521454444s) [noise=true , address=79.127.201.215:4350 , piece_id=6MWZ4MNZXE6KVJ4RLOTRWM24DER64OZW7CMCUPP7RYJQR7LWCTJQ, seg_pos={0 0}, upload=196876]: Done uploading piece. err: <nil> Aug 16 14:40:32.402083903 (189h14m26.833164034s) [noise=true , address=104.36.82.14:4247 , piece_id=G56NHF7M7FCPXDXRXEEADHQ2EZKINRBRED7Q4XI3662EOYODPWAQ, seg_pos={0 0}, upload=196876]: Done uploading piece. err: <nil> Aug 16 14:40:38.857957123 (189h14m33.289037244s) [noise=true , address=104.36.82.58:4299 , piece_id=KSQRSQVB6EGT37PKX3YV7QUQTJZSJKEUD3CKCFPCAD6S4NJAL7QA, seg_pos={0 0}, upload=196876]: Done uploading piece. err: <nil> Aug 16 14:40:49.164615516 (189h14m43.595695647s) [noise=true , address=77.90.31.18:4835 , piece_id=XURSIUMVEHTCRCXBWEIZ2ZJCSKTQXCVOQIKXWISDAYMRW7M4NOTQ, seg_pos={0 0}, upload=196876]: Done uploading piece. err: <nil> Aug 16 14:40:53.061942988 (189h14m47.493023119s) [noise=true , address=104.36.82.23:4463 , piece_id=B2BOEACKYAULFJIJZFCOJQ7C36SOJCKKVCQIP3QYNSBE6YZS66WQ, seg_pos={0 0}, upload=196876]: Done uploading piece. err: ecclient: upload failed (node:12d3ACC8cWirsfXo2XNPTtr6sULutTpdEpvFDweeiQuPQ3NiZyL, address:104.36.82.23:4463): protocol: expected piece hash; context deadline exceeded; EOF Aug 16 14:40:53.062241600 (189h14m47.493321731s) [noise=true , address=104.36.82.59:4310 , piece_id=64KHK5PWAB57RCI7FU4U26XDZKH6WXOEJ3NUFLTRRTHKS2UCPF2A, seg_pos={0 0}, upload=196876]: Done uploading piece. err: ecclient: upload failed (node:12uYJ3ag6xFbLiu39t7jqwvtbo1QLwHw7k9PmfrMWBZBappp8MX, address:104.36.82.59:4310): protocol: rpc: dial tcp 104.36.82.59:4310: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:40:53.062279993 (189h14m47.493360124s) [noise=true , address=45.91.116.5:4307 , piece_id=Z2JSIUQPBMW2JSYVMSZVBMCQXW3L2C2NEWPSRF6FUIJQRMEFHSKA, seg_pos={0 0}, upload=196876]: Done uploading piece. err: ecclient: upload failed (node:1ATcyutSAE1A1uN2QcwYFdY1ZvUCZTKiybzXt66NgnqrFYoWWC, address:45.91.116.5:4307): protocol: read tcp 188.166.225.170:57434->45.91.116.5:4307: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:40:53.062286144 (189h14m47.493366275s) [noise=true , address=45.91.116.6:4991 , piece_id=6MTNZT65LSI6PXSVMUJHT64GQVMU6CWOVUH2KGOINQIAUSKSDQRQ, seg_pos={0 0}, upload=196876]: Done uploading piece. err: ecclient: upload failed (node:12muT2wQRGoRoDtpsPGS3EDjQ9sLq2qTS3jRSHWfuV1h9itzMbb, address:45.91.116.6:4991): protocol: expected piece hash; write tcp 188.166.225.170:42596->45.91.116.6:4991: use of closed network connection; write tcp 188.166.225.170:42596->45.91.116.6:4991: use of closed network connection; EOF Aug 16 14:40:53.062367069 (189h14m47.49344719s) [noise=true , address=45.91.116.13:4159 , piece_id=33U4IEPB3Z5GEDQOHFWTAR5ATFJXN236X5Y66R6QNZ2FMK7KTO4A, seg_pos={0 0}, upload=196876]: Done uploading piece. err: ecclient: upload failed (node:128V9AYe3XZKQ9o2cbdeSNYoK55MHFJ4q1KbhjGXWzKrv17Hv3t, address:45.91.116.13:4159): protocol: rpc: dial tcp 45.91.116.13:4159: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:40:53.062469385 (189h14m47.493549516s) [noise=true , address=45.91.116.7:4017 , piece_id=IO6CYVMTFTQG2SAFKJIWEVJH72MYZA34ZGCT6WZR6PWXPICIQGNQ, seg_pos={0 0}, upload=196876]: Done uploading piece. err: ecclient: upload failed (node:1XLSYete7Dj3VNt24BbP5szLp35H6F6urWr8tRAdYy1Ms1Gpj7, address:45.91.116.7:4017): protocol: rpc: dial tcp 45.91.116.7:4017: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:40:53.062564447 (189h14m47.493644578s) [noise=true , address=104.36.82.60:4348 , piece_id=ZJXZKILKMDTWN7ZFW67F3CGZIQHA7OYXQIONNW3CG72WX57ZIQQQ, seg_pos={0 0}, upload=196876]: Done uploading piece. err: ecclient: upload failed (node:12ezboNAWPbaK9UDENYoBb3iHjNU2ahn7mikiteN7y3Aj6uw4Nx, address:104.36.82.60:4348): protocol: read tcp 188.166.225.170:45622->104.36.82.60:4348: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:40:53.062589825 (189h14m47.493669956s) [noise=true , address=77.90.31.21:4902 , piece_id=WUTMD3HBQ3OUMVOIRSDYOWEQAVV7MIEOONMWDDCKXRXVEZ7UH6WQ, seg_pos={0 0}, upload=196876]: Done uploading piece. err: ecclient: upload failed (node:12sD3HFEaYLm7KV6uRvjbojtU9eQGMSy3Aewr4pX5Bxjy4jMCHx, address:77.90.31.21:4902): protocol: read tcp 188.166.225.170:59354->77.90.31.21:4902: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:40:53.062605967 (189h14m47.493686098s) [noise=true , address=104.36.82.44:4884 , piece_id=2VREOXCLUGM2FUX7VRQRHZID7CSMAJZ6JRWLOJ4GKREJ4OHUMZOA, seg_pos={0 0}, upload=196876]: Done uploading piece. err: ecclient: upload failed (node:1qGXdnm83P4eMnCtZdRt2x5yAWP7pQafWwoPG72daZJVR9pyKG, address:104.36.82.44:4884): protocol: read tcp 188.166.225.170:54416->104.36.82.44:4884: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:40:53.062642737 (189h14m47.493722868s) [seg_pos={0 0}, upload=196876]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil> Aug 16 14:40:53.290772077 (189h14m47.721852218s) [upload=196876]: Done uploading segments... Aug 16 14:40:53.290798057 (189h14m47.721878198s) [upload=196876]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9) Duration: 1m0.230464321s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "http://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt", ExpectedStatus: 308, ExpectedRedirect: "https://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt"} failure (antiblip window 3/60): (http.us_select_1.linksharing) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 2.207596047s target: http://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt FAIL (2025-08-16 14:40:46 UTC) orbiter warning: Get "http://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt": dial tcp 45.38.24.2:80: i/o timeout Duration: 2.20762421s edge linksharing us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 3 out of 60 is failed (threshold=3) [..........X...............X............................X....] duration: 4.53928596s FAIL (2025-08-16 14:40:39 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/1K):: after 49dfafbb94c33c8117bc19860e46bb56246facc41c7f02daffe0cf8e29c76acb != before 1be7850c2cc74e399df94f373bb6443b6a49bf63dc3bd4b7e451763927467f44 ======LOGS====== Aug 16 14:37:37.968738630 (189h11m32.399818761s) [upload=196837]: Starting upload Aug 16 14:37:37.968781391 (189h11m32.399861513s) [upload=196837]: Done starting upload Aug 16 14:37:37.968833571 (189h11m32.399913702s) [upload=196837]: Uploading segments... Aug 16 14:37:37.968862807 (189h11m32.399942938s) [upload=196837]: Got next segment. Inline: true Aug 16 14:37:37.968866314 (189h11m32.399946445s) [upload=196837]: Waiting for error group managing segments... Aug 16 14:37:37.968908495 (189h11m32.399988646s) [upload=196837]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 14:37:37.968914897 (189h11m32.399995038s) [upload=196837]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 16 14:37:37.968919596 (189h11m32.399999747s) [upload=196837]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 14:37:41.759696736 (189h11m36.190776867s) [upload=196837]: Done uploading segments... Aug 16 14:37:41.759716192 (189h11m36.190796323s) [upload=196837]: Upload finished. err: <nil> Duration:* 4.539313502s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [..........................................X........XXX..XXXX] duration: 30.0012937s FAIL (2025-08-16 14:40:37 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.001313017s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [....................................................XXX..XXX] duration: 30.23849065s FAIL (2025-08-16 14:40:28 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 89h13m53.373109211s) [seg_pos={0 0}, upload=196877]: Begin upload segment... Aug 16 14:39:58.942069308 (189h13m53.373149439s) [seg_pos={0 0}, upload=196877]: Done begin upload segment. Aug 16 14:39:58.942091379 (189h13m53.37317151s) [noise=true , address=45.91.116.6:4973 , piece_id=6HY67CETYAY37KSJXL2JTUA4NUVWY7ER7CG443KCN7V7742OEE2A, seg_pos={0 0}, upload=196877]: Uploading piece... Aug 16 14:39:58.942134201 (189h13m53.373214333s) [noise=true , address=104.36.82.15:4257 , piece_id=7MNNGYUDCZRMD2FILYSAYST7IQWB4QBLMSJMI2NVIE6JVU2N6RAQ, seg_pos={0 0}, upload=196877]: Uploading piece... Aug 16 14:39:58.942463572 (189h13m53.373543703s) [noise=true , address=104.36.82.15:4259 , piece_id=MI6KPDPFOKTX3AOWQMN62EAKAWEHBA3YJCIJWTTT2DLW3EWOJNNQ, seg_pos={0 0}, upload=196877]: Uploading piece... Aug 16 14:39:58.942598961 (189h13m53.373679092s) [noise=true , address=104.36.82.12:4149 , piece_id=PLRT6NYEGIKJ3BYVGAPQJ56JGHXDALZP5KMC7WVPCF5PXUQMSN4A, seg_pos={0 0}, upload=196877]: Uploading piece... Aug 16 14:39:58.942739780 (189h13m53.373819911s) [noise=true , address=104.36.82.13:4186 , piece_id=422V7ZKHUUH7X4RXV77QMSGCTHWXJDB5WAGWIQC3RJ4CEYU7D7RA, seg_pos={0 0}, upload=196877]: Uploading piece... Aug 16 14:39:58.942881361 (189h13m53.373961492s) [noise=true , address=104.36.82.10:4079 , piece_id=U5Q4UYB4CNIDH4SSICFOQPMZVLPES52LDLX5PADGBOPZFKUP45SA, seg_pos={0 0}, upload=196877]: Uploading piece... Aug 16 14:39:58.943013003 (189h13m53.374093134s) [noise=true , address=45.91.116.4:4940 , piece_id=QA2HCGNMSHIJ6Z33JVHEK7GN3MH64YBJIZ7LNEBUXB6C2PEB53GA, seg_pos={0 0}, upload=196877]: Uploading piece... Aug 16 14:39:58.943275606 (189h13m53.374355727s) [noise=true , address=45.91.116.8:4045 , piece_id=H7TG4EOULVPYN2XIR3U37KVA24WL6PUT7RZYBMBPKN6G7RHCLVJQ, seg_pos={0 0}, upload=196877]: Uploading piece... Aug 16 14:39:58.943301676 (189h13m53.374381807s) [noise=true , address=45.91.116.10:4081 , piece_id=MQOGDQHUS3RP3Z4IE6XSWUJMJHHIFTY5LI4NPXK3PVI3QIPHYJ5A, seg_pos={0 0}, upload=196877]: Uploading piece... Aug 16 14:39:58.943532397 (189h13m53.374612528s) [noise=true , address=104.36.82.3:4767 , piece_id=GKYG2RYEEYE4OJQ7BEGQT3NA7QPM47YBFFSZTI3ZQGYIKSTSCPWA, seg_pos={0 0}, upload=196877]: Uploading piece... Aug 16 14:39:58.943679889 (189h13m53.37476002s) [noise=true , address=45.91.116.7:4009 , piece_id=X6SHRJMFMDP3MVEFMS74Q4O3LN7BB5DYBXK32IZUAKTMT34TMI5Q, seg_pos={0 0}, upload=196877]: Uploading piece... Aug 16 14:39:58.943819205 (189h13m53.374899336s) [noise=true , address=104.36.82.16:4293 , piece_id=BZSIAQDI2MGTASJ5EFS3OWAMN7NCAIEJWF7C5IODW2BJPFKVEWQA, seg_pos={0 0}, upload=196877]: Uploading piece... Aug 16 14:39:58.943938314 (189h13m53.375018445s) [noise=true , address=104.36.82.6:4874 , piece_id=IRCTJOV6TEH3ESXG6E3R3PYGKCZNFO2IDEIHUJML2QWLKGO2UOSA, seg_pos={0 0}, upload=196877]: Uploading piece... Aug 16 14:39:58.944058012 (189h13m53.375138143s) [noise=true , address=45.91.116.5:4297 , piece_id=SZ5263PY5QCSRKXO5HI4CKP2IUK6S6L5LRZICFJKBBJIKSOE65GA, seg_pos={0 0}, upload=196877]: Uploading piece... Aug 16 14:39:58.944175948 (189h13m53.375256069s) [noise=true , address=104.36.82.2:4729 , piece_id=ZNLJBXP2SS4PVG33WHSPXZOHSOEIS37TR3SQF47TW7EN3Z4IQ44Q, seg_pos={0 0}, upload=196877]: Uploading piece... Aug 16 14:39:58.944429924 (189h13m53.375510055s) [noise=true , address=104.36.82.12:4151 , piece_id=E4GGLRPK23PNA7SEHVE6FS3R7AYVEBMFQUQO27TO2I3FEAHCFJLA, seg_pos={0 0}, upload=196877]: Uploading piece... Aug 16 14:40:00.797580116 (189h13m55.228660267s) [upload=196877]: Next returned nil segment Aug 16 14:40:00.797599313 (189h13m55.228679464s) [upload=196877]: Waiting for error group managing segments... Aug 16 14:40:01.104595767 (189h13m55.535675898s) [noise=true , address=104.36.82.15:4257 , piece_id=7MNNGYUDCZRMD2FILYSAYST7IQWB4QBLMSJMI2NVIE6JVU2N6RAQ, seg_pos={0 0}, upload=196877]: Done uploading piece. err: <nil> Aug 16 14:40:01.278551643 (189h13m55.709631774s) [noise=true , address=104.36.82.10:4079 , piece_id=U5Q4UYB4CNIDH4SSICFOQPMZVLPES52LDLX5PADGBOPZFKUP45SA, seg_pos={0 0}, upload=196877]: Done uploading piece. err: <nil> Aug 16 14:40:02.047930636 (189h13m56.479010767s) [noise=true , address=45.91.116.8:4045 , piece_id=H7TG4EOULVPYN2XIR3U37KVA24WL6PUT7RZYBMBPKN6G7RHCLVJQ, seg_pos={0 0}, upload=196877]: Done uploading piece. err: <nil> Aug 16 14:40:03.177026356 (189h13m57.608106488s) [noise=true , address=104.36.82.3:4767 , piece_id=GKYG2RYEEYE4OJQ7BEGQT3NA7QPM47YBFFSZTI3ZQGYIKSTSCPWA, seg_pos={0 0}, upload=196877]: Done uploading piece. err: <nil> Aug 16 14:40:07.867235450 (189h14m2.298315581s) [noise=true , address=104.36.82.13:4186 , piece_id=422V7ZKHUUH7X4RXV77QMSGCTHWXJDB5WAGWIQC3RJ4CEYU7D7RA, seg_pos={0 0}, upload=196877]: Done uploading piece. err: <nil> Aug 16 14:40:12.353236667 (189h14m6.784316788s) [noise=true , address=104.36.82.12:4149 , piece_id=PLRT6NYEGIKJ3BYVGAPQJ56JGHXDALZP5KMC7WVPCF5PXUQMSN4A, seg_pos={0 0}, upload=196877]: Done uploading piece. err: <nil> Aug 16 14:40:28.060379785 (189h14m22.491459916s) [noise=true , address=45.91.116.4:4940 , piece_id=QA2HCGNMSHIJ6Z33JVHEK7GN3MH64YBJIZ7LNEBUXB6C2PEB53GA, seg_pos={0 0}, upload=196877]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: rpc: dial tcp 45.91.116.4:4940: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:40:28.060498471 (189h14m22.491578602s) [noise=true , address=104.36.82.2:4729 , piece_id=ZNLJBXP2SS4PVG33WHSPXZOHSOEIS37TR3SQF47TW7EN3Z4IQ44Q, seg_pos={0 0}, upload=196877]: Done uploading piece. err: ecclient: upload failed (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR, address:104.36.82.2:4729): protocol: read tcp 188.166.225.170:51278->104.36.82.2:4729: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:40:28.060535673 (189h14m22.491615804s) [noise=true , address=45.91.116.5:4297 , piece_id=SZ5263PY5QCSRKXO5HI4CKP2IUK6S6L5LRZICFJKBBJIKSOE65GA, seg_pos={0 0}, upload=196877]: Done uploading piece. err: ecclient: upload failed (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW, address:45.91.116.5:4297): protocol: read tcp 188.166.225.170:46522->45.91.116.5:4297: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:40:28.060567985 (189h14m22.491648116s) [noise=true , address=45.91.116.10:4081 , piece_id=MQOGDQHUS3RP3Z4IE6XSWUJMJHHIFTY5LI4NPXK3PVI3QIPHYJ5A, seg_pos={0 0}, upload=196877]: Done uploading piece. err: ecclient: upload failed (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG, address:45.91.116.10:4081): protocol: read tcp 188.166.225.170:49578->45.91.116.10:4081: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:40:28.060660011 (189h14m22.491740132s) [noise=true , address=104.36.82.15:4259 , piece_id=MI6KPDPFOKTX3AOWQMN62EAKAWEHBA3YJCIJWTTT2DLW3EWOJNNQ, seg_pos={0 0}, upload=196877]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:104.36.82.15:4259): protocol: rpc: dial tcp 104.36.82.15:4259: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:40:28.060756496 (189h14m22.491836627s) [noise=true , address=45.91.116.6:4973 , piece_id=6HY67CETYAY37KSJXL2JTUA4NUVWY7ER7CG443KCN7V7742OEE2A, seg_pos={0 0}, upload=196877]: Done uploading piece. err: ecclient: upload failed (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, address:45.91.116.6:4973): protocol: rpc: dial tcp 45.91.116.6:4973: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:40:28.060768017 (189h14m22.491848178s) [noise=true , address=104.36.82.16:4293 , piece_id=BZSIAQDI2MGTASJ5EFS3OWAMN7NCAIEJWF7C5IODW2BJPFKVEWQA, seg_pos={0 0}, upload=196877]: Done uploading piece. err: ecclient: upload failed (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, address:104.36.82.16:4293): protocol: rpc: dial tcp 104.36.82.16:4293: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:40:28.060794408 (189h14m22.491874539s) [noise=true , address=45.91.116.7:4009 , piece_id=X6SHRJMFMDP3MVEFMS74Q4O3LN7BB5DYBXK32IZUAKTMT34TMI5Q, seg_pos={0 0}, upload=196877]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: rpc: dial tcp 45.91.116.7:4009: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:40:28.060829515 (189h14m22.491909646s) [noise=true , address=104.36.82.12:4151 , piece_id=E4GGLRPK23PNA7SEHVE6FS3R7AYVEBMFQUQO27TO2I3FEAHCFJLA, seg_pos={0 0}, upload=196877]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:104.36.82.12:4151): protocol: read tcp 188.166.225.170:45500->104.36.82.12:4151: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:40:28.060853260 (189h14m22.491933451s) [noise=true , address=104.36.82.6:4874 , piece_id=IRCTJOV6TEH3ESXG6E3R3PYGKCZNFO2IDEIHUJML2QWLKGO2UOSA, seg_pos={0 0}, upload=196877]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:104.36.82.6:4874): protocol: rpc: dial tcp 104.36.82.6:4874: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:40:28.060897555 (189h14m22.491977747s) [seg_pos={0 0}, upload=196877]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Aug 16 14:40:28.296560310 (189h14m22.727640441s) [upload=196877]: Done uploading segments... Aug 16 14:40:28.296587242 (189h14m22.727667373s) [upload=196877]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 30.238517862s core uplink us-select-1
checkers.HTTP{"GET" "http://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt", ExpectedStatus: 308, ExpectedRedirect: "https://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt"} failure (antiblip window 3/60): (http.us_select_1.linksharing) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 507.1047ms target: http://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt FAIL (2025-08-16 14:40:26 UTC) orbiter warning: Get "http://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt": dial tcp 45.38.24.2:80: i/o timeout Duration: 507.130259ms edge linksharing us-select-1
checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 2.547574295s FAIL (2025-08-16 14:40:24 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": dial tcp 45.38.24.2:443: i/o timeout Duration: 2.54759725s edge authservice http us-select-1
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway) antiblip: 4 out of 60 is failed (threshold=3) [......................................................X.XXX.] duration: 1.166585134s target: https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410 FAIL (2025-08-16 14:40:23 UTC) orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": dial tcp 45.38.24.2:443: i/o timeout Duration: 1.166626493s edge gateway vhost us-select-1
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway) antiblip: 4 out of 60 is failed (threshold=3) [.......................................................X.XXX] duration: 30.001054771s target: https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410 FAIL (2025-08-16 14:40:13 UTC) orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": dial tcp 45.38.24.2:443: i/o timeout Duration: 30.001082815s edge gateway vhost us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 4 out of 60 is failed (threshold=3) [......................................................XX..XX] duration: 30.001140934s FAIL (2025-08-16 14:40:10 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": context deadline exceeded Duration: 30.001169629s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 7 out of 60 is failed (threshold=3) [.....X...X...........X................................X.X.XX] duration: 1m1.227789938s FAIL (2025-08-16 14:40:09 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 5) ======LOGS====== Aug 16 14:39:08.830925420 (189h13m3.262005551s) [upload=196862]: Starting upload Aug 16 14:39:08.830974163 (189h13m3.262054294s) [upload=196862]: Done starting upload Aug 16 14:39:08.831060909 (189h13m3.26214104s) [upload=196862]: Uploading segments... Aug 16 14:39:08.831153196 (189h13m3.262233327s) [upload=196862]: Got next segment. Inline: false Aug 16 14:39:08.831182683 (189h13m3.262262814s) [upload=196862]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 14:39:08.831187432 (189h13m3.262267563s) [upload=196862]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 14:39:09.347812678 (189h13m3.778892809s) [seg_pos={0 0}, upload=196862]: Begin upload segment... Aug 16 14:39:09.347873193 (189h13m3.778953394s) [seg_pos={0 0}, upload=196862]: Done begin upload segment. Aug 16 14:39:09.347903501 (189h13m3.778983622s) [noise=true , address=66.23.193.175:4564 , piece_id=IIWWCYUJDZ45LINT6E4SFBQXCY6QFQQZ35Y5JI24C5Q3IUVUWNVQ, seg_pos={0 0}, upload=196862]: Uploading piece... Aug 16 14:39:09.347970620 (189h13m3.779050751s) [noise=true , address=77.90.31.15:4702 , piece_id=2I3EE2BIYOFOATKL2XEPVGHHKYB7QETSR223IYMSWDD7EZZZTMUQ, seg_pos={0 0}, upload=196862]: Uploading piece... Aug 16 14:39:09.348275874 (189h13m3.779356005s) [noise=true , address=45.91.116.10:4110 , piece_id=2VWLBYN2S2BHTIE6MVYSA7L2ON6LHM76OTRM6DI5GAKLE7VYHS4Q, seg_pos={0 0}, upload=196862]: Uploading piece... Aug 16 14:39:09.348517156 (189h13m3.779597287s) [noise=true , address=79.127.220.101:4060 , piece_id=RBL3DY4XPUZ33YLFOPCDVVBL6YGKNDSSWFFZ5UNCP2V7X7SXIWNA, seg_pos={0 0}, upload=196862]: Uploading piece... Aug 16 14:39:09.348616575 (189h13m3.779696697s) [noise=true , address=104.36.82.44:4902 , piece_id=2MYLTYTPNKMRADH6SFNIEMIAKCJ5A6QK3VHXYD2GLATUWJP7GHIQ, seg_pos={0 0}, upload=196862]: Uploading piece... Aug 16 14:39:09.348825255 (189h13m3.779905386s) [noise=true , address=104.36.82.5:4845 , piece_id=DZASULCEPBND3CURAPITVHSNRJLWD43O6VGK6MTFNH5NMXMMFIAA, seg_pos={0 0}, upload=196862]: Uploading piece... Aug 16 14:39:09.348902694 (189h13m3.779982825s) [noise=true , address=77.90.31.13:4648 , piece_id=WHED6EV7X6QK2FQBZFEJ5ZR4DWCZA6A7SLM7HTYKNXSSP73MZIFA, seg_pos={0 0}, upload=196862]: Uploading piece... Aug 16 14:39:09.349103958 (189h13m3.780184089s) [noise=true , address=104.36.82.34:4564 , piece_id=O6UI5YSIPR2OJOCJ6C2HI7HEDLNGGF2CCJLFTMWVTUYAJDCXQ5CQ, seg_pos={0 0}, upload=196862]: Uploading piece... Aug 16 14:39:09.349170666 (189h13m3.780250798s) [noise=true , address=104.36.82.59:4303 , piece_id=4FV7QFDBJIJYQOOJUIYJKDHVXO46RE4R4J7HJUOUCTG65TROFFBA, seg_pos={0 0}, upload=196862]: Uploading piece... Aug 16 14:39:09.349385136 (189h13m3.780465267s) [noise=true , address=45.91.116.9:4294 , piece_id=4JPQ6G6MK4A6PNG7QURZSGXMQQTKD543DPCV6ZSY6CXJZNJU4TLQ, seg_pos={0 0}, upload=196862]: Uploading piece... Aug 16 14:39:09.349468706 (189h13m3.780548837s) [noise=true , address=104.36.82.60:4342 , piece_id=6SBL4REFUTGRBTBGMSJC24CNGDHT7JT2IIKPQOEJQWKM7HQFOHDA, seg_pos={0 0}, upload=196862]: Uploading piece... Aug 16 14:39:09.349663770 (189h13m3.780743891s) [noise=true , address=104.36.82.39:4745 , piece_id=HY24SDFBNUYWPAT6S6ZMIGEDYM5OYXH7F42VPTD2MUSOHWL3LL7A, seg_pos={0 0}, upload=196862]: Uploading piece... Aug 16 14:39:09.349746328 (189h13m3.780826469s) [noise=true , address=79.127.201.214:4320 , piece_id=R53WNWUG63L4MV7SW66G2YUVPYSHGZFEMQVYE4EC2ECYTEIPIYAA, seg_pos={0 0}, upload=196862]: Uploading piece... Aug 16 14:39:09.349924208 (189h13m3.781004339s) [noise=true , address=45.91.116.11:4124 , piece_id=USCD3E3WX5KT3TZ3SL7SPPKLXRY3QPP327LEESA3ZS7RRHSIIK4Q, seg_pos={0 0}, upload=196862]: Uploading piece... Aug 16 14:39:09.349982199 (189h13m3.78106232s) [noise=true , address=104.36.82.58:4279 , piece_id=A3OZ465KFZ32YTE3UOP44ITAJ3MMJVSFLH352HNZR5VBRQ4R6J7Q, seg_pos={0 0}, upload=196862]: Uploading piece... Aug 16 14:39:09.350134601 (189h13m3.781214732s) [noise=true , address=79.127.201.215:4353 , piece_id=36QK7WFZ5NRTKMQOTQHNYFRPOPMEXOXEDRJYVHOEC35GLVRS7V6Q, seg_pos={0 0}, upload=196862]: Uploading piece... Aug 16 14:39:09.350190007 (189h13m3.781270128s) [noise=true , address=104.36.82.13:4211 , piece_id=A2SUYLKWREM2CU7QCCZGFHRZ7DFBLGEVVPUTB2UMZZECEVIZYZLQ, seg_pos={0 0}, upload=196862]: Uploading piece... Aug 16 14:39:09.350343630 (189h13m3.781423761s) [noise=true , address=77.90.31.11:4578 , piece_id=UAXKT4V4P4OLGD4D3FPOGV4QLIUH2WJOAFK2NTPXF6RPSO5LFYZQ, seg_pos={0 0}, upload=196862]: Uploading piece... Aug 16 14:39:10.502315521 (189h13m4.933395652s) [upload=196862]: Next returned nil segment Aug 16 14:39:10.502330590 (189h13m4.933410721s) [upload=196862]: Waiting for error group managing segments... Aug 16 14:39:10.822239080 (189h13m5.253319212s) [noise=true , address=66.23.193.175:4564 , piece_id=IIWWCYUJDZ45LINT6E4SFBQXCY6QFQQZ35Y5JI24C5Q3IUVUWNVQ, seg_pos={0 0}, upload=196862]: Done uploading piece. err: <nil> Aug 16 14:39:11.192685930 (189h13m5.623766061s) [noise=true , address=79.127.220.101:4060 , piece_id=RBL3DY4XPUZ33YLFOPCDVVBL6YGKNDSSWFFZ5UNCP2V7X7SXIWNA, seg_pos={0 0}, upload=196862]: Done uploading piece. err: <nil> Aug 16 14:39:12.484403270 (189h13m6.915483401s) [noise=true , address=77.90.31.11:4578 , piece_id=UAXKT4V4P4OLGD4D3FPOGV4QLIUH2WJOAFK2NTPXF6RPSO5LFYZQ, seg_pos={0 0}, upload=196862]: Done uploading piece. err: <nil> Aug 16 14:39:50.447261798 (189h13m44.878341929s) [noise=true , address=79.127.201.214:4320 , piece_id=R53WNWUG63L4MV7SW66G2YUVPYSHGZFEMQVYE4EC2ECYTEIPIYAA, seg_pos={0 0}, upload=196862]: Done uploading piece. err: <nil> Aug 16 14:39:53.895976127 (189h13m48.327056258s) [noise=true , address=79.127.201.215:4353 , piece_id=36QK7WFZ5NRTKMQOTQHNYFRPOPMEXOXEDRJYVHOEC35GLVRS7V6Q, seg_pos={0 0}, upload=196862]: Done uploading piece. err: <nil> Aug 16 14:40:08.173765963 (189h14m2.604846094s) [noise=true , address=77.90.31.15:4702 , piece_id=2I3EE2BIYOFOATKL2XEPVGHHKYB7QETSR223IYMSWDD7EZZZTMUQ, seg_pos={0 0}, upload=196862]: Done uploading piece. err: ecclient: upload failed (node:1ucSAZVBqRRJU7WZDKs7Cs6yKV73haX7px76sfukA6b8Q8DDsW, address:77.90.31.15:4702): protocol: rpc: dial tcp 77.90.31.15:4702: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:40:08.173864752 (189h14m2.604944883s) [noise=true , address=77.90.31.13:4648 , piece_id=WHED6EV7X6QK2FQBZFEJ5ZR4DWCZA6A7SLM7HTYKNXSSP73MZIFA, seg_pos={0 0}, upload=196862]: Done uploading piece. err: ecclient: upload failed (node:1niTsTaHjA6LoMMrkEsSnLcRDhZtVAjxsbXUXVeXzhUp3FPbWh, address:77.90.31.13:4648): protocol: rpc: dial tcp 77.90.31.13:4648: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:40:08.173917472 (189h14m2.604997603s) [noise=true , address=104.36.82.34:4564 , piece_id=O6UI5YSIPR2OJOCJ6C2HI7HEDLNGGF2CCJLFTMWVTUYAJDCXQ5CQ, seg_pos={0 0}, upload=196862]: Done uploading piece. err: ecclient: upload failed (node:121WW4uBQe8bGQKkasQQKT3k4EAdzncsG1ebNvgKf8S5SyNttCc, address:104.36.82.34:4564): protocol: rpc: dial tcp 104.36.82.34:4564: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:40:08.174042913 (189h14m2.605123044s) [noise=true , address=45.91.116.11:4124 , piece_id=USCD3E3WX5KT3TZ3SL7SPPKLXRY3QPP327LEESA3ZS7RRHSIIK4Q, seg_pos={0 0}, upload=196862]: Done uploading piece. err: ecclient: upload failed (node:1vSQgozKbq7XjSYP5dUJjkrnTkWX4EFgBkruqUmdV3ehnoPfkr, address:45.91.116.11:4124): protocol: rpc: dial tcp 45.91.116.11:4124: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:40:08.174095022 (189h14m2.605175153s) [noise=true , address=104.36.82.13:4211 , piece_id=A2SUYLKWREM2CU7QCCZGFHRZ7DFBLGEVVPUTB2UMZZECEVIZYZLQ, seg_pos={0 0}, upload=196862]: Done uploading piece. err: ecclient: upload failed (node:12dANoYTygpcpWfCcAJtsnVBFWSekLhDjheAXnsaYMChRYAm2qc, address:104.36.82.13:4211): protocol: write tcp 188.166.225.170:41340->104.36.82.13:4211: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:40:08.174148094 (189h14m2.605228215s) [noise=true , address=104.36.82.39:4745 , piece_id=HY24SDFBNUYWPAT6S6ZMIGEDYM5OYXH7F42VPTD2MUSOHWL3LL7A, seg_pos={0 0}, upload=196862]: Done uploading piece. err: ecclient: upload failed (node:1ujPZUMvwigdNuqXMzvgaxXdiVQzEkfsSDNQY7BujAnHLHkXks, address:104.36.82.39:4745): protocol: read tcp 188.166.225.170:36506->104.36.82.39:4745: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:40:08.174152001 (189h14m2.605232132s) [noise=true , address=104.36.82.5:4845 , piece_id=DZASULCEPBND3CURAPITVHSNRJLWD43O6VGK6MTFNH5NMXMMFIAA, seg_pos={0 0}, upload=196862]: Done uploading piece. err: ecclient: upload failed (node:1nChJBXktwFs6z7cSK2Wh6N3jkAZwTZ7zhEUt393Nnx4eB6ube, address:104.36.82.5:4845): protocol: rpc: dial tcp 104.36.82.5:4845: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:40:08.174234058 (189h14m2.605314219s) [noise=true , address=104.36.82.59:4303 , piece_id=4FV7QFDBJIJYQOOJUIYJKDHVXO46RE4R4J7HJUOUCTG65TROFFBA, seg_pos={0 0}, upload=196862]: Done uploading piece. err: ecclient: upload failed (node:12gJKrCBLJWN1SBKibfeqBtcqqiv4veKTGmtnycjZyb3VN9kwoT, address:104.36.82.59:4303): protocol: rpc: dial tcp 104.36.82.59:4303: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:40:08.174251442 (189h14m2.605331573s) [noise=true , address=104.36.82.58:4279 , piece_id=A3OZ465KFZ32YTE3UOP44ITAJ3MMJVSFLH352HNZR5VBRQ4R6J7Q, seg_pos={0 0}, upload=196862]: Done uploading piece. err: ecclient: upload failed (node:12LtRviHk2GinJGa3L53MCHCAfKrqUsey5JhQRHbYQTuSUNKoNC, address:104.36.82.58:4279): protocol: rpc: dial tcp 104.36.82.58:4279: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:40:08.174329842 (189h14m2.605409973s) [noise=true , address=104.36.82.60:4342 , piece_id=6SBL4REFUTGRBTBGMSJC24CNGDHT7JT2IIKPQOEJQWKM7HQFOHDA, seg_pos={0 0}, upload=196862]: Done uploading piece. err: ecclient: upload failed (node:1pYxAYKuf8jtEBtsH5cvx8hmTfqrMv5Z4c6taF9hVgRDmx8KoB, address:104.36.82.60:4342): protocol: rpc: dial tcp 104.36.82.60:4342: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:40:08.174405376 (189h14m2.605485507s) [noise=true , address=104.36.82.44:4902 , piece_id=2MYLTYTPNKMRADH6SFNIEMIAKCJ5A6QK3VHXYD2GLATUWJP7GHIQ, seg_pos={0 0}, upload=196862]: Done uploading piece. err: ecclient: upload failed (node:1Be9T5kXUwLU3C6wBeKFV3Q7aZFbZDTNG1oaEft8V6rmySUcEE, address:104.36.82.44:4902): protocol: read tcp 188.166.225.170:47466->104.36.82.44:4902: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:40:08.174410335 (189h14m2.605490466s) [noise=true , address=45.91.116.10:4110 , piece_id=2VWLBYN2S2BHTIE6MVYSA7L2ON6LHM76OTRM6DI5GAKLE7VYHS4Q, seg_pos={0 0}, upload=196862]: Done uploading piece. err: ecclient: upload failed (node:123KwusATu1ZKtNNVHZZZBE67ZEnVb8puyfQidBi3fEWREqdf1h, address:45.91.116.10:4110): protocol: read tcp 188.166.225.170:51156->45.91.116.10:4110: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:40:08.174453618 (189h14m2.605533769s) [noise=true , address=45.91.116.9:4294 , piece_id=4JPQ6G6MK4A6PNG7QURZSGXMQQTKD543DPCV6ZSY6CXJZNJU4TLQ, seg_pos={0 0}, upload=196862]: Done uploading piece. err: ecclient: upload failed (node:12d7KuNgPYL5Ca2MPKQQCueQNSsYQifHXMhQ87VEXUhrKpLjuks, address:45.91.116.9:4294): protocol: read tcp 188.166.225.170:33756->45.91.116.9:4294: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:40:08.174491500 (189h14m2.605571662s) [seg_pos={0 0}, upload=196862]: Done waiting for segment. successful: 5 optimal: 14 errs: <nil> Aug 16 14:40:09.399215598 (189h14m3.830295749s) [upload=196862]: Done uploading segments... Aug 16 14:40:09.399252178 (189h14m3.830332309s) [upload=196862]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 5) Duration: 1m1.227817s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 3 out of 60 is failed (threshold=3) [...........X...............X............................X...] duration: 6.266620145s FAIL (2025-08-16 14:40:09 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/1K):: after 49dfafbb94c33c8117bc19860e46bb56246facc41c7f02daffe0cf8e29c76acb != before 1be7850c2cc74e399df94f373bb6443b6a49bf63dc3bd4b7e451763927467f44 ======LOGS====== Aug 16 14:37:37.968738630 (189h11m32.399818761s) [upload=196837]: Starting upload Aug 16 14:37:37.968781391 (189h11m32.399861513s) [upload=196837]: Done starting upload Aug 16 14:37:37.968833571 (189h11m32.399913702s) [upload=196837]: Uploading segments... Aug 16 14:37:37.968862807 (189h11m32.399942938s) [upload=196837]: Got next segment. Inline: true Aug 16 14:37:37.968866314 (189h11m32.399946445s) [upload=196837]: Waiting for error group managing segments... Aug 16 14:37:37.968908495 (189h11m32.399988646s) [upload=196837]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 14:37:37.968914897 (189h11m32.399995038s) [upload=196837]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 16 14:37:37.968919596 (189h11m32.399999747s) [upload=196837]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 14:37:41.759696736 (189h11m36.190776867s) [upload=196837]: Done uploading segments... Aug 16 14:37:41.759716192 (189h11m36.190796323s) [upload=196837]: Upload finished. err: <nil> Duration:* 6.266648458s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.HTTP{"GET" "http://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt", ExpectedStatus: 308, ExpectedRedirect: "https://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt"} failure (antiblip window 3/60): (http.us_select_1.linksharing) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 30.000683406s target: http://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt FAIL (2025-08-16 14:39:58 UTC) orbiter warning: Get "http://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt": dial tcp 45.38.24.2:80: i/o timeout Duration: 30.000697432s edge linksharing us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [......................................................X.X.XX] duration: 1m0.228369794s FAIL (2025-08-16 14:39:52 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 6) ======LOGS====== 5GXTE4E2ZAUPUCMDOLZDHQAKAXZOZPHQ, seg_pos={0 0}, upload=196857]: Uploading piece... Aug 16 14:38:53.720381011 (189h12m48.151461142s) [noise=true , address=104.36.82.17:4346 , piece_id=AAB5NDH2OSGLAE7QWUAIJSMWCAUEUTP4IARY5AMPIHL4PDY365NQ, seg_pos={0 0}, upload=196857]: Uploading piece... Aug 16 14:38:53.720849959 (189h12m48.15193008s) [noise=true , address=79.127.220.101:4076 , piece_id=5CV4QZKRWRLK7UKHJ7RWM7DKEKL333EU6KATUBS66U2SC35AZFGQ, seg_pos={0 0}, upload=196857]: Uploading piece... Aug 16 14:38:53.720893342 (189h12m48.151973483s) [noise=true , address=77.90.31.11:4565 , piece_id=6VQDNOSJLJUKVB7IKMYULVFVZHDUR36R4FJFAAZFFVFY6SPZVSVQ, seg_pos={0 0}, upload=196857]: Uploading piece... Aug 16 14:38:53.721221239 (189h12m48.15230138s) [noise=true , address=77.90.31.15:4719 , piece_id=RORR374XL6TCTNYTDMDEEUJ4VFMOBUZNOGMEYFCE3NX3RPL53VBA, seg_pos={0 0}, upload=196857]: Uploading piece... Aug 16 14:38:53.721352912 (189h12m48.152433043s) [noise=true , address=104.36.82.60:4345 , piece_id=SRDCP7N7X2I3L76HKGJQXTEK22HO2KIRMVSKKZ5EKTNWYVWWCFOQ, seg_pos={0 0}, upload=196857]: Uploading piece... Aug 16 14:38:53.721765561 (189h12m48.152845692s) [noise=true , address=104.36.82.5:4871 , piece_id=2RKZXW25PTXKEHA32FR5D62V5VBMUXOUZ23A3TAYL2LHYXI7UTZQ, seg_pos={0 0}, upload=196857]: Uploading piece... Aug 16 14:38:53.722196666 (189h12m48.153276797s) [noise=true , address=77.90.31.21:4899 , piece_id=6UUNM2TQGCRGMUSZU6JFK5IAAT4432RFQ4OI43E6YNTTUH5YVL2Q, seg_pos={0 0}, upload=196857]: Uploading piece... Aug 16 14:38:53.722615467 (189h12m48.153695598s) [noise=true , address=104.36.82.59:4334 , piece_id=DANFZEPDCRN27IPYTRAVJKIANFNRPESNG7XTYADMCM7M24CMV3KA, seg_pos={0 0}, upload=196857]: Uploading piece... Aug 16 14:38:53.722987959 (189h12m48.15406809s) [noise=true , address=77.90.31.19:4855 , piece_id=4YSZLADQ3RXAFSUPLTOHZNJ3CVIE2SXAU53OMXQJAINF7UZLXJ4A, seg_pos={0 0}, upload=196857]: Uploading piece... Aug 16 14:38:53.723213892 (189h12m48.154294034s) [noise=true , address=77.90.31.21:4902 , piece_id=VD2RTIO6W7EAE4BLPRSX3ASHSGP5JMW5BQBJ73PLLUU633WEY5AA, seg_pos={0 0}, upload=196857]: Uploading piece... Aug 16 14:38:53.723511853 (189h12m48.154591984s) [noise=true , address=79.127.220.98:4401 , piece_id=YLFCW6S2UWUAGMPSZFBOHXXDTNFXMQIFIAI5KXSNUDOMRP4OYHCQ, seg_pos={0 0}, upload=196857]: Uploading piece... Aug 16 14:38:53.723807077 (189h12m48.154887208s) [noise=true , address=66.23.193.175:4559 , piece_id=A26T4COZ4WURGDKIVMGOI2AKXN3WFWU6QIICC5PEZAHP2W4PSYNA, seg_pos={0 0}, upload=196857]: Uploading piece... Aug 16 14:38:54.913138394 (189h12m49.344218525s) [upload=196857]: Next returned nil segment Aug 16 14:38:54.913160125 (189h12m49.344240256s) [upload=196857]: Waiting for error group managing segments... Aug 16 14:38:55.284357701 (189h12m49.715437822s) [noise=true , address=79.127.205.235:4680 , piece_id=ZDK7YJIR34MI4UYXJUDUTNVJ4IIC3VJN273T5VAPP4ONFDN3MXWA, seg_pos={0 0}, upload=196857]: Done uploading piece. err: <nil> Aug 16 14:38:55.290490025 (189h12m49.721570166s) [noise=true , address=66.23.193.175:4559 , piece_id=A26T4COZ4WURGDKIVMGOI2AKXN3WFWU6QIICC5PEZAHP2W4PSYNA, seg_pos={0 0}, upload=196857]: Done uploading piece. err: <nil> Aug 16 14:38:55.562012053 (189h12m49.993092174s) [noise=true , address=79.127.220.98:4401 , piece_id=YLFCW6S2UWUAGMPSZFBOHXXDTNFXMQIFIAI5KXSNUDOMRP4OYHCQ, seg_pos={0 0}, upload=196857]: Done uploading piece. err: <nil> Aug 16 14:38:55.613030000 (189h12m50.044110121s) [noise=true , address=79.127.220.101:4076 , piece_id=5CV4QZKRWRLK7UKHJ7RWM7DKEKL333EU6KATUBS66U2SC35AZFGQ, seg_pos={0 0}, upload=196857]: Done uploading piece. err: <nil> Aug 16 14:38:56.223897889 (189h12m50.65497802s) [noise=true , address=77.90.31.18:4840 , piece_id=JEVIKL3VCAIJ3KFE6MOFIIWTST3Q4PSSIW7GNP6ZHQRLDF7776GA, seg_pos={0 0}, upload=196857]: Done uploading piece. err: <nil> Aug 16 14:39:02.434800794 (189h12m56.865880925s) [noise=true , address=104.36.82.60:4345 , piece_id=SRDCP7N7X2I3L76HKGJQXTEK22HO2KIRMVSKKZ5EKTNWYVWWCFOQ, seg_pos={0 0}, upload=196857]: Done uploading piece. err: <nil> Aug 16 14:39:52.612997116 (189h13m47.044077247s) [noise=true , address=104.36.82.17:4346 , piece_id=AAB5NDH2OSGLAE7QWUAIJSMWCAUEUTP4IARY5AMPIHL4PDY365NQ, seg_pos={0 0}, upload=196857]: Done uploading piece. err: ecclient: upload failed (node:12kuxXNRsBsP6izniQ2AcT4DJRW2RYWdJurSPYh5h8TF32pwzza, address:104.36.82.17:4346): protocol: rpc: dial tcp 104.36.82.17:4346: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:39:52.613074544 (189h13m47.044154675s) [noise=true , address=77.90.31.15:4719 , piece_id=RORR374XL6TCTNYTDMDEEUJ4VFMOBUZNOGMEYFCE3NX3RPL53VBA, seg_pos={0 0}, upload=196857]: Done uploading piece. err: ecclient: upload failed (node:1qyhppjd6m6he7eHVZLuhK9BYKXq1DY3GuKBNfyb97JVaz9wyk, address:77.90.31.15:4719): protocol: rpc: dial tcp 77.90.31.15:4719: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:39:52.613143196 (189h13m47.044223327s) [noise=true , address=79.127.201.215:4360 , piece_id=BRUA2C2RJVDRIUQGQNNCCAN2N5JBBWJFMUJBQVV76PKVD7PSFJHA, seg_pos={0 0}, upload=196857]: Done uploading piece. err: ecclient: upload failed (node:1y7dgLXYLTGGFXzHQ5ZYJ4f4kWYCoyiBhGesd1xqbTbnYjxNzG, address:79.127.201.215:4360): protocol: write tcp 188.166.225.170:59584->79.127.201.215:4360: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:39:52.613405698 (189h13m47.044485829s) [noise=true , address=104.36.82.5:4871 , piece_id=2RKZXW25PTXKEHA32FR5D62V5VBMUXOUZ23A3TAYL2LHYXI7UTZQ, seg_pos={0 0}, upload=196857]: Done uploading piece. err: ecclient: upload failed (node:1NEVGFZtdgMqREnYBPuBbYqwGRxExenRwncwCBLDFhuwdFyWdb, address:104.36.82.5:4871): protocol: rpc: dial tcp 104.36.82.5:4871: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:39:52.613521399 (189h13m47.04460153s) [noise=true , address=77.90.31.21:4902 , piece_id=VD2RTIO6W7EAE4BLPRSX3ASHSGP5JMW5BQBJ73PLLUU633WEY5AA, seg_pos={0 0}, upload=196857]: Done uploading piece. err: ecclient: upload failed (node:12sD3HFEaYLm7KV6uRvjbojtU9eQGMSy3Aewr4pX5Bxjy4jMCHx, address:77.90.31.21:4902): protocol: write tcp 188.166.225.170:44696->77.90.31.21:4902: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:39:52.613568650 (189h13m47.044648781s) [noise=true , address=104.36.82.23:4480 , piece_id=Y54LYHUREIZ4P4QXGAPS5GXTE4E2ZAUPUCMDOLZDHQAKAXZOZPHQ, seg_pos={0 0}, upload=196857]: Done uploading piece. err: ecclient: upload failed (node:1qQhWAdN83i1jMTYByNQ8SWGXQaXRgBGezkLTf1BTwLyKskjjU, address:104.36.82.23:4480): protocol: rpc: dial tcp 104.36.82.23:4480: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:39:52.613661226 (189h13m47.044741359s) [noise=true , address=104.36.82.59:4334 , piece_id=DANFZEPDCRN27IPYTRAVJKIANFNRPESNG7XTYADMCM7M24CMV3KA, seg_pos={0 0}, upload=196857]: Done uploading piece. err: ecclient: upload failed (node:12MaiBbEtrHEaXrMy5hhbYDQsGfTCZuT6C62aFxUnXPE5vRW3kg, address:104.36.82.59:4334): protocol: rpc: dial tcp 104.36.82.59:4334: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:39:52.613733164 (189h13m47.044813296s) [noise=true , address=104.36.82.44:4871 , piece_id=YFVTCL2QQR6SHQALA6C62NQKREYEOPEVZDPRT43SUNCSXNAOS44A, seg_pos={0 0}, upload=196857]: Done uploading piece. err: ecclient: upload failed (node:12t3N84zSSMtzRBs1H47Szc2xAHwqmyr9szJahRkxMYDLto8DAx, address:104.36.82.44:4871): protocol: rpc: dial tcp 104.36.82.44:4871: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:39:52.613786747 (189h13m47.044866878s) [noise=true , address=77.90.31.21:4899 , piece_id=6UUNM2TQGCRGMUSZU6JFK5IAAT4432RFQ4OI43E6YNTTUH5YVL2Q, seg_pos={0 0}, upload=196857]: Done uploading piece. err: ecclient: upload failed (node:12UjFkFn2UEvr57LDorcijDCvkBkkKitgZEFx6Yqfk5gMStQvJG, address:77.90.31.21:4899): protocol: read tcp 188.166.225.170:46556->77.90.31.21:4899: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:39:52.613850388 (189h13m47.04493051s) [noise=true , address=77.90.31.19:4855 , piece_id=4YSZLADQ3RXAFSUPLTOHZNJ3CVIE2SXAU53OMXQJAINF7UZLXJ4A, seg_pos={0 0}, upload=196857]: Done uploading piece. err: ecclient: upload failed (node:19LE2Lq4e6HojnMp9xQ3ySLkRe5PemXTwiQZThN5jT46bQhN6K, address:77.90.31.19:4855): protocol: read tcp 188.166.225.170:54106->77.90.31.19:4855: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:39:52.613881659 (189h13m47.04496179s) [noise=true , address=77.90.31.11:4565 , piece_id=6VQDNOSJLJUKVB7IKMYULVFVZHDUR36R4FJFAAZFFVFY6SPZVSVQ, seg_pos={0 0}, upload=196857]: Done uploading piece. err: ecclient: upload failed (node:1xnwFcUEdmpLjmx8NXZZcpiVQwTySXDRYbchuz4oREBq1yP6wh, address:77.90.31.11:4565): protocol: rpc: dial tcp 77.90.31.11:4565: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:39:52.613965309 (189h13m47.04504543s) [noise=true , address=104.36.82.34:4554 , piece_id=PF4YCVD5XOD4OZ5GZHXXZYGKIOBQ3MRC72QA7FDK54S6BUW7SSMQ, seg_pos={0 0}, upload=196857]: Done uploading piece. err: ecclient: upload failed (node:12rSDkPXm47w73hry9W6BWwAiwNpB3cJuwedNt1JuSEEtd3Sup7, address:104.36.82.34:4554): protocol: read tcp 188.166.225.170:41242->104.36.82.34:4554: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:39:52.613991680 (189h13m47.045071811s) [seg_pos={0 0}, upload=196857]: Done waiting for segment. successful: 6 optimal: 14 errs: <nil> Aug 16 14:39:52.840167622 (189h13m47.271247743s) [upload=196857]: Done uploading segments... Aug 16 14:39:52.840195957 (189h13m47.271276088s) [upload=196857]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 6) Duration: 1m0.228383461s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [...........................................X........XXX..XXX] duration: 30.000293861s FAIL (2025-08-16 14:39:41 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": dial tcp 45.38.24.2:443: i/o timeout Duration: 30.000312125s edge gateway us-select-1
checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 30.000477318s FAIL (2025-08-16 14:39:37 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": dial tcp 45.38.24.2:443: i/o timeout Duration: 30.000505934s edge authservice http us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 3 out of 60 is failed (threshold=3) [............X...............X............................X..] duration: 1.82436788s FAIL (2025-08-16 14:39:29 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/1K):: after 49dfafbb94c33c8117bc19860e46bb56246facc41c7f02daffe0cf8e29c76acb != before 1be7850c2cc74e399df94f373bb6443b6a49bf63dc3bd4b7e451763927467f44 ======LOGS====== Aug 16 14:37:37.968738630 (189h11m32.399818761s) [upload=196837]: Starting upload Aug 16 14:37:37.968781391 (189h11m32.399861513s) [upload=196837]: Done starting upload Aug 16 14:37:37.968833571 (189h11m32.399913702s) [upload=196837]: Uploading segments... Aug 16 14:37:37.968862807 (189h11m32.399942938s) [upload=196837]: Got next segment. Inline: true Aug 16 14:37:37.968866314 (189h11m32.399946445s) [upload=196837]: Waiting for error group managing segments... Aug 16 14:37:37.968908495 (189h11m32.399988646s) [upload=196837]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 14:37:37.968914897 (189h11m32.399995038s) [upload=196837]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 16 14:37:37.968919596 (189h11m32.399999747s) [upload=196837]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 14:37:41.759696736 (189h11m36.190776867s) [upload=196837]: Done uploading segments... Aug 16 14:37:41.759716192 (189h11m36.190796323s) [upload=196837]: Upload finished. err: <nil> Duration:* 1.82438413s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.....................................................XXX..XX] duration: 30.225873027s FAIL (2025-08-16 14:39:28 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 2) ======LOGS====== 58.837004627 (189h12m53.268084748s) [noise=true , address=104.36.82.11:4115 , piece_id=L6BBN3LXH7JQSMLAUUBLJZTWV5TEEG3B7VKPMGNVUC2RG5MKDEBQ, seg_pos={0 0}, upload=196859]: Uploading piece... Aug 16 14:38:58.837256408 (189h12m53.268336529s) [noise=true , address=104.36.82.10:4079 , piece_id=S5OW3P54VHUHZTYAPUXEKCPR3G5Q6ASAJ7BFVLGQ65UE5VEDT3RQ, seg_pos={0 0}, upload=196859]: Uploading piece... Aug 16 14:38:58.837477842 (189h12m53.268557973s) [noise=true , address=104.36.82.13:4187 , piece_id=HJ7YZRO422DXRHAPEV3RQ3LU5O6CIWY5RCNHWKKEOXC4YAPXUMLA, seg_pos={0 0}, upload=196859]: Uploading piece... Aug 16 14:38:58.837736777 (189h12m53.268816908s) [noise=true , address=77.90.31.10:4519 , piece_id=GIIAIWO2CLAZ6OWUMJVKFFLQTN4GGB3GYRTZJKEP5KPURSXPLSEQ, seg_pos={0 0}, upload=196859]: Uploading piece... Aug 16 14:38:58.837886464 (189h12m53.268966595s) [noise=true , address=104.36.82.8:4971 , piece_id=Y3JBTLY6AIYSJWTFU4EBWXVCO27YWT555YCWVJDBXDXPV6H7IT6A, seg_pos={0 0}, upload=196859]: Uploading piece... Aug 16 14:38:58.838420225 (189h12m53.269500356s) [noise=true , address=104.36.82.15:4258 , piece_id=7UV55IXSO4FGL2OP5GWQNLVSWAUC7PR2J2TYGXBC6MQ37TLKYOOQ, seg_pos={0 0}, upload=196859]: Uploading piece... Aug 16 14:38:58.838433771 (189h12m53.269513902s) [noise=true , address=104.36.82.14:4222 , piece_id=5VYI6YLESBTEOHWDBQ5Q67YSXXY4VGXHXBN4HO7WSFKUG3GTK7RA, seg_pos={0 0}, upload=196859]: Uploading piece... Aug 16 14:38:58.839011426 (189h12m53.270091557s) [noise=true , address=104.36.82.6:4873 , piece_id=GM6QQE7M2ZXSMYSMQHS4S77W6PWEMIB3WIRNQ7SAYWP74HEEYCDQ, seg_pos={0 0}, upload=196859]: Uploading piece... Aug 16 14:38:58.839708480 (189h12m53.270788611s) [noise=true , address=45.91.116.10:4082 , piece_id=7XELIZWK4XW5E6O5HVWCPLFPIRGHF4R47ZU56S5SOVV5KHPZLEZQ, seg_pos={0 0}, upload=196859]: Uploading piece... Aug 16 14:38:58.840424069 (189h12m53.2715042s) [noise=true , address=104.36.82.2:4729 , piece_id=LRL25FR42O7G4UNBVEMV5MKRACHBPCVKOSY2PIKCDMOXI62Q5IXQ, seg_pos={0 0}, upload=196859]: Uploading piece... Aug 16 14:38:58.841035189 (189h12m53.27211531s) [noise=true , address=77.90.31.15:4697 , piece_id=YBVNE4POO22WD24BB7U5KBOGEP37HH6N5HVVZU6AOE5EUJNVKGJQ, seg_pos={0 0}, upload=196859]: Uploading piece... Aug 16 14:38:58.841360742 (189h12m53.272440873s) [noise=true , address=104.36.82.10:4077 , piece_id=JUWKSRALHSRTEDRZP3S3Y7MH4DKJKPTFGR7GCUHYXLT5QBFZXJDQ, seg_pos={0 0}, upload=196859]: Uploading piece... Aug 16 14:38:58.841851351 (189h12m53.272931472s) [noise=true , address=104.36.82.12:4149 , piece_id=D4CUFEPTZIG3DLPSGE3BOM6DTAE2OCBDQ5NAMC7IIMDV7QWWAG5Q, seg_pos={0 0}, upload=196859]: Uploading piece... Aug 16 14:39:01.494200350 (189h12m55.925280481s) [upload=196859]: Next returned nil segment Aug 16 14:39:01.494217032 (189h12m55.925297163s) [upload=196859]: Waiting for error group managing segments... Aug 16 14:39:01.882160970 (189h12m56.313241111s) [noise=true , address=104.36.82.4:4803 , piece_id=4YCJQDUTOPCCOVLSUONV4HULSULVFR5KRPZUQ26EVVEXMV63PYEQ, seg_pos={0 0}, upload=196859]: Done uploading piece. err: <nil> Aug 16 14:39:25.459729150 (189h13m19.890809281s) [noise=true , address=104.36.82.7:4912 , piece_id=BU67HU3CFJNT654XTV6MGF4R6GVM5UMKRLVE5Y3CHULRRSFLEJWQ, seg_pos={0 0}, upload=196859]: Done uploading piece. err: <nil> Aug 16 14:39:27.961074051 (189h13m22.392154172s) [noise=true , address=104.36.82.12:4149 , piece_id=D4CUFEPTZIG3DLPSGE3BOM6DTAE2OCBDQ5NAMC7IIMDV7QWWAG5Q, seg_pos={0 0}, upload=196859]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:104.36.82.12:4149): protocol: read tcp 188.166.225.170:59060->104.36.82.12:4149: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:39:27.961178280 (189h13m22.392258401s) [noise=true , address=45.91.116.10:4082 , piece_id=7XELIZWK4XW5E6O5HVWCPLFPIRGHF4R47ZU56S5SOVV5KHPZLEZQ, seg_pos={0 0}, upload=196859]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: rpc: dial tcp 45.91.116.10:4082: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:39:27.961216393 (189h13m22.392296524s) [noise=true , address=104.36.82.2:4729 , piece_id=LRL25FR42O7G4UNBVEMV5MKRACHBPCVKOSY2PIKCDMOXI62Q5IXQ, seg_pos={0 0}, upload=196859]: Done uploading piece. err: ecclient: upload failed (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR, address:104.36.82.2:4729): protocol: rpc: dial tcp 104.36.82.2:4729: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:39:27.961313459 (189h13m22.39239359s) [noise=true , address=104.36.82.10:4077 , piece_id=JUWKSRALHSRTEDRZP3S3Y7MH4DKJKPTFGR7GCUHYXLT5QBFZXJDQ, seg_pos={0 0}, upload=196859]: Done uploading piece. err: ecclient: upload failed (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk, address:104.36.82.10:4077): protocol: rpc: dial tcp 104.36.82.10:4077: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:39:27.961322767 (189h13m22.392402908s) [noise=true , address=104.36.82.11:4115 , piece_id=L6BBN3LXH7JQSMLAUUBLJZTWV5TEEG3B7VKPMGNVUC2RG5MKDEBQ, seg_pos={0 0}, upload=196859]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:104.36.82.11:4115): protocol: rpc: dial tcp 104.36.82.11:4115: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:39:27.961368384 (189h13m22.392448515s) [noise=true , address=104.36.82.13:4187 , piece_id=HJ7YZRO422DXRHAPEV3RQ3LU5O6CIWY5RCNHWKKEOXC4YAPXUMLA, seg_pos={0 0}, upload=196859]: Done uploading piece. err: ecclient: upload failed (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, address:104.36.82.13:4187): protocol: rpc: dial tcp 104.36.82.13:4187: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:39:27.961377321 (189h13m22.392457452s) [noise=true , address=104.36.82.10:4079 , piece_id=S5OW3P54VHUHZTYAPUXEKCPR3G5Q6ASAJ7BFVLGQ65UE5VEDT3RQ, seg_pos={0 0}, upload=196859]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:104.36.82.10:4079): protocol: rpc: dial tcp 104.36.82.10:4079: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:39:27.961409913 (189h13m22.392490044s) [noise=true , address=104.36.82.8:4971 , piece_id=Y3JBTLY6AIYSJWTFU4EBWXVCO27YWT555YCWVJDBXDXPV6H7IT6A, seg_pos={0 0}, upload=196859]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:104.36.82.8:4971): protocol: rpc: dial tcp 104.36.82.8:4971: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:39:27.961419762 (189h13m22.392499883s) [noise=true , address=104.36.82.6:4873 , piece_id=GM6QQE7M2ZXSMYSMQHS4S77W6PWEMIB3WIRNQ7SAYWP74HEEYCDQ, seg_pos={0 0}, upload=196859]: Done uploading piece. err: ecclient: upload failed (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3, address:104.36.82.6:4873): protocol: rpc: dial tcp 104.36.82.6:4873: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:39:27.961467503 (189h13m22.392547634s) [noise=true , address=104.36.82.14:4222 , piece_id=5VYI6YLESBTEOHWDBQ5Q67YSXXY4VGXHXBN4HO7WSFKUG3GTK7RA, seg_pos={0 0}, upload=196859]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: rpc: dial tcp 104.36.82.14:4222: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:39:27.961476731 (189h13m22.392556863s) [noise=true , address=77.90.31.10:4519 , piece_id=GIIAIWO2CLAZ6OWUMJVKFFLQTN4GGB3GYRTZJKEP5KPURSXPLSEQ, seg_pos={0 0}, upload=196859]: Done uploading piece. err: ecclient: upload failed (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, address:77.90.31.10:4519): protocol: rpc: dial tcp 77.90.31.10:4519: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:39:27.961508642 (189h13m22.392588763s) [noise=true , address=77.90.31.15:4697 , piece_id=YBVNE4POO22WD24BB7U5KBOGEP37HH6N5HVVZU6AOE5EUJNVKGJQ, seg_pos={0 0}, upload=196859]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: rpc: dial tcp 77.90.31.15:4697: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:39:27.961526115 (189h13m22.392606236s) [noise=true , address=104.36.82.15:4258 , piece_id=7UV55IXSO4FGL2OP5GWQNLVSWAUC7PR2J2TYGXBC6MQ37TLKYOOQ, seg_pos={0 0}, upload=196859]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:104.36.82.15:4258): protocol: rpc: dial tcp 104.36.82.15:4258: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:39:27.961572204 (189h13m22.392652325s) [noise=true , address=45.91.116.14:4189 , piece_id=MGIEKL3AX6IZBA5MGYCK5CP3ARRTE7BO4OO36NGDCJSKXF2GRRTA, seg_pos={0 0}, upload=196859]: Done uploading piece. err: ecclient: upload failed (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, address:45.91.116.14:4189): protocol: write tcp 188.166.225.170:49734->45.91.116.14:4189: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:39:27.961591481 (189h13m22.392671612s) [seg_pos={0 0}, upload=196859]: Done waiting for segment. successful: 2 optimal: 13 errs: <nil> Aug 16 14:39:28.184232199 (189h13m22.61531233s) [upload=196859]: Done uploading segments... Aug 16 14:39:28.184253119 (189h13m22.61533325s) [upload=196859]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 2) Duration: 30.225886131s core uplink us-select-1
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway) antiblip: 3 out of 60 is failed (threshold=3) [........................................................X.XX] duration: 30.000367079s target: https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410 FAIL (2025-08-16 14:39:17 UTC) orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": dial tcp 45.38.24.2:443: i/o timeout Duration: 30.000383731s edge gateway vhost us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 3 out of 60 is failed (threshold=3) [.......................................................XX..X] duration: 30.000975176s FAIL (2025-08-16 14:39:13 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": context deadline exceeded Duration: 30.001006235s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [............................................X........XXX..XX] duration: 30.001616987s FAIL (2025-08-16 14:39:11 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.001629742s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 6 out of 60 is failed (threshold=3) [......X...X...........X................................X.X.X] duration: 1m0.449924511s FAIL (2025-08-16 14:39:07 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== Aug 16 14:38:08.151003134 (189h12m2.582083265s) [upload=196845]: Starting upload Aug 16 14:38:08.151054983 (189h12m2.582135114s) [upload=196845]: Done starting upload Aug 16 14:38:08.151241059 (189h12m2.58232123s) [upload=196845]: Uploading segments... Aug 16 14:38:08.151818785 (189h12m2.582898906s) [upload=196845]: Got next segment. Inline: false Aug 16 14:38:08.151877266 (189h12m2.582957397s) [upload=196845]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 14:38:08.151882046 (189h12m2.582962177s) [upload=196845]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 14:38:08.901910885 (189h12m3.332991006s) [seg_pos={0 0}, upload=196845]: Begin upload segment... Aug 16 14:38:08.901952885 (189h12m3.333033016s) [seg_pos={0 0}, upload=196845]: Done begin upload segment. Aug 16 14:38:08.901974676 (189h12m3.333054808s) [noise=true , address=104.36.82.9:4008 , piece_id=KMZGEG63XWFPHMGRYPDAKYLYLVFMSNPNFZUF7N24SLQK4WN3WBYA, seg_pos={0 0}, upload=196845]: Uploading piece... Aug 16 14:38:08.901993072 (189h12m3.333073203s) [noise=true , address=104.36.82.60:4349 , piece_id=5D5MEJ2UYVYNWR4LG2F3OVGTEMZ4UAM533S7PDVL4LSAIQB3ACAA, seg_pos={0 0}, upload=196845]: Uploading piece... Aug 16 14:38:08.902331770 (189h12m3.333411901s) [noise=true , address=45.91.116.5:4329 , piece_id=Z6WWFRDGJG5R3AKLXVKQ5PEAERTPVH5AM7ZDIXOCGPHZJZRYOOTQ, seg_pos={0 0}, upload=196845]: Uploading piece... Aug 16 14:38:08.902343513 (189h12m3.333423644s) [noise=true , address=104.36.82.5:4862 , piece_id=X3FNG7RMX2W4H4XOVBQPJYZ4W4WN4JKPYWWCMK76RNBGKDJRLEKQ, seg_pos={0 0}, upload=196845]: Uploading piece... Aug 16 14:38:08.902676259 (189h12m3.33375638s) [noise=true , address=66.23.193.175:4557 , piece_id=SERURIXEIMXUX6CIXCPIKTQPNYA632YI25YAJ4GMNP2UHYRBHSPQ, seg_pos={0 0}, upload=196845]: Uploading piece... Aug 16 14:38:08.902693752 (189h12m3.333773883s) [noise=true , address=45.91.116.12:4231 , piece_id=IB467SQ5Y7EVPF5LVUD2UIYYJPDAQ4VBQGPWUGKMXXMOG7EGRUOA, seg_pos={0 0}, upload=196845]: Uploading piece... Aug 16 14:38:08.902983557 (189h12m3.334063688s) [noise=true , address=104.36.82.59:4315 , piece_id=O7PBBCEOZF7NBFH5NXYQLRVWEUKGB6DASOS3B7FFFOELLMNNUUYA, seg_pos={0 0}, upload=196845]: Uploading piece... Aug 16 14:38:08.903007944 (189h12m3.334088065s) [noise=true , address=104.36.82.4:4808 , piece_id=KZVSTTAD5J35NNT4XKZLOYQ7LWA3IVKJ6LLAMS5H5JHTIFDSQ62Q, seg_pos={0 0}, upload=196845]: Uploading piece... Aug 16 14:38:08.903278522 (189h12m3.334358653s) [noise=true , address=77.90.31.16:4745 , piece_id=4Z2A4WPT6QIJJ2EDCKJMO2RFJ7CN2ZEFCWTHZG2YV46K33N4HEGA, seg_pos={0 0}, upload=196845]: Uploading piece... Aug 16 14:38:08.903294963 (189h12m3.334375094s) [noise=true , address=104.36.82.35:4612 , piece_id=HAP6IT3LYPF27PLUA3CGL2FJSDXSNIIN7YU746MUFKX3GKUSLBNQ, seg_pos={0 0}, upload=196845]: Uploading piece... Aug 16 14:38:08.903537948 (189h12m3.334618079s) [noise=true , address=77.90.31.21:4907 , piece_id=NHFTTJ5AQUPM2XGD66GLMYBVXJ7NM4AIKFNCZO2YHDJAY2Q6BFSQ, seg_pos={0 0}, upload=196845]: Uploading piece... Aug 16 14:38:08.903572333 (189h12m3.334652464s) [noise=true , address=104.36.82.44:4901 , piece_id=ZRHCK3ZZ4KXNXXDBDY5P3HSHN52CNQFL2BS2DRDTXKLHAGM3SR4Q, seg_pos={0 0}, upload=196845]: Uploading piece... Aug 16 14:38:08.903744513 (189h12m3.334824634s) [noise=true , address=79.127.201.215:4354 , piece_id=YHK7PNPMNF65SIP6GMKCNGPAZMCA5X4BAJSX44M3DFVRHCP3CQAQ, seg_pos={0 0}, upload=196845]: Uploading piece... Aug 16 14:38:08.903800490 (189h12m3.334880621s) [noise=true , address=77.90.31.15:4726 , piece_id=WOIGX55IVDXCO4E3QPDETOAYHE6QSWLQPPLJNOQ74IGDO24DCYDA, seg_pos={0 0}, upload=196845]: Uploading piece... Aug 16 14:38:08.903948814 (189h12m3.335028945s) [noise=true , address=104.36.82.23:4462 , piece_id=OOGM4OTCRU7H4SXOBULAGDYFZJ5XC74NK7SLPDIUB4ZT6ZOALGTA, seg_pos={0 0}, upload=196845]: Uploading piece... Aug 16 14:38:08.904035379 (189h12m3.33511551s) [noise=true , address=79.127.220.98:4401 , piece_id=OC2BYK3RFYNQHSJHIW4KUUJEUHZSX4VSRHVHU3CJ27VKQR3ZKAYQ, seg_pos={0 0}, upload=196845]: Uploading piece... Aug 16 14:38:08.904157163 (189h12m3.335237284s) [noise=true , address=104.36.82.38:4706 , piece_id=ZDHGGG526SRLY4DJ43HQOXXEI5NMJUJXRYDZOV5FTIJ3T5T4B7LQ, seg_pos={0 0}, upload=196845]: Uploading piece... Aug 16 14:38:08.904389427 (189h12m3.335469558s) [noise=true , address=79.127.220.101:4071 , piece_id=QY4FTEIVHWAS6SLFJTUZZ4AHX7WLLGMCMLPE6GIFRWX2FQV725BA, seg_pos={0 0}, upload=196845]: Uploading piece... Aug 16 14:38:10.064224826 (189h12m4.495304957s) [upload=196845]: Next returned nil segment Aug 16 14:38:10.064600194 (189h12m4.495680315s) [upload=196845]: Waiting for error group managing segments... Aug 16 14:38:10.359465336 (189h12m4.790545457s) [noise=true , address=66.23.193.175:4557 , piece_id=SERURIXEIMXUX6CIXCPIKTQPNYA632YI25YAJ4GMNP2UHYRBHSPQ, seg_pos={0 0}, upload=196845]: Done uploading piece. err: <nil> Aug 16 14:38:10.748440177 (189h12m5.179520298s) [noise=true , address=79.127.220.101:4071 , piece_id=QY4FTEIVHWAS6SLFJTUZZ4AHX7WLLGMCMLPE6GIFRWX2FQV725BA, seg_pos={0 0}, upload=196845]: Done uploading piece. err: <nil> Aug 16 14:38:10.827568591 (189h12m5.258648732s) [noise=true , address=79.127.220.98:4401 , piece_id=OC2BYK3RFYNQHSJHIW4KUUJEUHZSX4VSRHVHU3CJ27VKQR3ZKAYQ, seg_pos={0 0}, upload=196845]: Done uploading piece. err: <nil> Aug 16 14:38:10.881388199 (189h12m5.31246833s) [noise=true , address=79.127.201.215:4354 , piece_id=YHK7PNPMNF65SIP6GMKCNGPAZMCA5X4BAJSX44M3DFVRHCP3CQAQ, seg_pos={0 0}, upload=196845]: Done uploading piece. err: <nil> Aug 16 14:38:11.130665564 (189h12m5.561745695s) [noise=true , address=104.36.82.59:4315 , piece_id=O7PBBCEOZF7NBFH5NXYQLRVWEUKGB6DASOS3B7FFFOELLMNNUUYA, seg_pos={0 0}, upload=196845]: Done uploading piece. err: <nil> Aug 16 14:38:11.186702865 (189h12m5.617782996s) [noise=true , address=104.36.82.38:4706 , piece_id=ZDHGGG526SRLY4DJ43HQOXXEI5NMJUJXRYDZOV5FTIJ3T5T4B7LQ, seg_pos={0 0}, upload=196845]: Done uploading piece. err: <nil> Aug 16 14:38:14.859654005 (189h12m9.290734136s) [noise=true , address=104.36.82.4:4808 , piece_id=KZVSTTAD5J35NNT4XKZLOYQ7LWA3IVKJ6LLAMS5H5JHTIFDSQ62Q, seg_pos={0 0}, upload=196845]: Done uploading piece. err: <nil> Aug 16 14:38:15.220659810 (189h12m9.651739952s) [noise=true , address=104.36.82.44:4901 , piece_id=ZRHCK3ZZ4KXNXXDBDY5P3HSHN52CNQFL2BS2DRDTXKLHAGM3SR4Q, seg_pos={0 0}, upload=196845]: Done uploading piece. err: <nil> Aug 16 14:38:15.683697862 (189h12m10.114777993s) [noise=true , address=45.91.116.5:4329 , piece_id=Z6WWFRDGJG5R3AKLXVKQ5PEAERTPVH5AM7ZDIXOCGPHZJZRYOOTQ, seg_pos={0 0}, upload=196845]: Done uploading piece. err: <nil> Aug 16 14:38:22.464847704 (189h12m16.895927825s) [noise=true , address=104.36.82.9:4008 , piece_id=KMZGEG63XWFPHMGRYPDAKYLYLVFMSNPNFZUF7N24SLQK4WN3WBYA, seg_pos={0 0}, upload=196845]: Done uploading piece. err: <nil> Aug 16 14:38:27.950218202 (189h12m22.381298333s) [noise=true , address=104.36.82.35:4612 , piece_id=HAP6IT3LYPF27PLUA3CGL2FJSDXSNIIN7YU746MUFKX3GKUSLBNQ, seg_pos={0 0}, upload=196845]: Done uploading piece. err: <nil> Aug 16 14:38:36.637099148 (189h12m31.068179279s) [noise=true , address=77.90.31.21:4907 , piece_id=NHFTTJ5AQUPM2XGD66GLMYBVXJ7NM4AIKFNCZO2YHDJAY2Q6BFSQ, seg_pos={0 0}, upload=196845]: Done uploading piece. err: <nil> Aug 16 14:39:07.492440376 (189h13m1.923520497s) [noise=true , address=77.90.31.16:4745 , piece_id=4Z2A4WPT6QIJJ2EDCKJMO2RFJ7CN2ZEFCWTHZG2YV46K33N4HEGA, seg_pos={0 0}, upload=196845]: Done uploading piece. err: ecclient: upload failed (node:12sFKft9P2avVAwrZC4fcCRwuWbFT7q9bCwLmumHNmSiqxHDR7r, address:77.90.31.16:4745): protocol: write tcp 188.166.225.170:49774->77.90.31.16:4745: use of closed network connection; write tcp 188.166.225.170:49774->77.90.31.16:4745: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49774->77.90.31.16:4745: use of closed network connection Aug 16 14:39:07.492635248 (189h13m1.923715369s) [noise=true , address=104.36.82.60:4349 , piece_id=5D5MEJ2UYVYNWR4LG2F3OVGTEMZ4UAM533S7PDVL4LSAIQB3ACAA, seg_pos={0 0}, upload=196845]: Done uploading piece. err: ecclient: upload failed (node:12CrzYmDPm95TgyE7SHJsFVbL1JUTdwMmZVge4uEkSCbeP8iBEk, address:104.36.82.60:4349): protocol: write tcp 188.166.225.170:42588->104.36.82.60:4349: use of closed network connection; write tcp 188.166.225.170:42588->104.36.82.60:4349: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42588->104.36.82.60:4349: use of closed network connection Aug 16 14:39:07.492728317 (189h13m1.923808448s) [noise=true , address=104.36.82.23:4462 , piece_id=OOGM4OTCRU7H4SXOBULAGDYFZJ5XC74NK7SLPDIUB4ZT6ZOALGTA, seg_pos={0 0}, upload=196845]: Done uploading piece. err: ecclient: upload failed (node:1YirC8WVSoE1qrWGMmayNmCsDLxjUhggrgXL5unKtQjsfUkbMu, address:104.36.82.23:4462): protocol: write tcp 188.166.225.170:55050->104.36.82.23:4462: use of closed network connection; write tcp 188.166.225.170:55050->104.36.82.23:4462: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55050->104.36.82.23:4462: use of closed network connection Aug 16 14:39:07.492765348 (189h13m1.923845479s) [noise=true , address=77.90.31.15:4726 , piece_id=WOIGX55IVDXCO4E3QPDETOAYHE6QSWLQPPLJNOQ74IGDO24DCYDA, seg_pos={0 0}, upload=196845]: Done uploading piece. err: ecclient: upload failed (node:12kMaukCkCJSc3ukAZu3thLgzXGGzNq44dGyEjCPqoY3PSYVEm3, address:77.90.31.15:4726): protocol: read tcp 188.166.225.170:53996->77.90.31.15:4726: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:39:07.492798821 (189h13m1.923878952s) [noise=true , address=45.91.116.12:4231 , piece_id=IB467SQ5Y7EVPF5LVUD2UIYYJPDAQ4VBQGPWUGKMXXMOG7EGRUOA, seg_pos={0 0}, upload=196845]: Done uploading piece. err: ecclient: upload failed (node:1NmgZdLWKjrDMmNcgmnzH475akX4WtA9zGTLxZgVW7JpaecBjq, address:45.91.116.12:4231): protocol: read tcp 188.166.225.170:36228->45.91.116.12:4231: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:39:07.492831795 (189h13m1.923911926s) [noise=true , address=104.36.82.5:4862 , piece_id=X3FNG7RMX2W4H4XOVBQPJYZ4W4WN4JKPYWWCMK76RNBGKDJRLEKQ, seg_pos={0 0}, upload=196845]: Done uploading piece. err: ecclient: upload failed (node:12sUT7LBYFqWMUhx7MMjEmyiFDnbJnjSVndNbTaHioaVLLSXYue, address:104.36.82.5:4862): protocol: read tcp 188.166.225.170:53704->104.36.82.5:4862: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:39:07.492845260 (189h13m1.923925382s) [seg_pos={0 0}, upload=196845]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Aug 16 14:39:07.939954375 (189h13m2.371034506s) [upload=196845]: Done uploading segments... Aug 16 14:39:07.939979994 (189h13m2.371060125s) [upload=196845]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 1m0.449948176s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [......................................................XXX..X] duration: 30.230871159s FAIL (2025-08-16 14:38:57 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 5) ======LOGS====== 61 , piece_id=TEV7WFGPMNWXZ5U3OJQO5UFSEQ2KLXKZSMT3M4FJCBYYSYISU6FQ, seg_pos={0 0}, upload=196851]: Uploading piece... Aug 16 14:38:28.607304946 (189h12m23.038385077s) [noise=true , address=104.36.82.11:4114 , piece_id=4MDYK7SFDVNOP64CQ4TT5JZCGVMBWFPIJ3RF723GEE6RSMBAK2YA, seg_pos={0 0}, upload=196851]: Uploading piece... Aug 16 14:38:28.607373857 (189h12m23.038453988s) [noise=true , address=45.91.116.11:4118 , piece_id=TIVE47XVPMFDP3ALBEJ2ICZ772BJ56ALBAY7QYPK7BYKRBYGBJQA, seg_pos={0 0}, upload=196851]: Uploading piece... Aug 16 14:38:28.607639055 (189h12m23.038719176s) [noise=true , address=104.36.82.10:4079 , piece_id=PZZ2GRNWIE54I4VZ64GSZY4ALU67KWRMARPXIEQJDOFBTQ7BOHCA, seg_pos={0 0}, upload=196851]: Uploading piece... Aug 16 14:38:28.607829690 (189h12m23.038909821s) [noise=true , address=77.90.31.15:4698 , piece_id=OLQWRAQXT3NMBEOLOOTBYLBORHMFTQCJXD5EM6VKM27R3XZ422SA, seg_pos={0 0}, upload=196851]: Uploading piece... Aug 16 14:38:28.607912889 (189h12m23.03899301s) [noise=true , address=45.91.116.14:4190 , piece_id=J3W4F4DEDJRMO2N6HHIV6E7QWOMKTKZCT2XGR273CB7BXMZVZJJA, seg_pos={0 0}, upload=196851]: Uploading piece... Aug 16 14:38:28.608170201 (189h12m23.039250332s) [noise=true , address=104.36.82.7:4912 , piece_id=ILEBNMPU7AQWGJ2WFLEM5ZJ3XLMS2ASEL7PR6EVBTY573LX4QJXA, seg_pos={0 0}, upload=196851]: Uploading piece... Aug 16 14:38:28.608420611 (189h12m23.039500782s) [noise=true , address=104.36.82.12:4151 , piece_id=D4NQQ5DXBGW4MZM2IQJ2D5HBO4MCHDPMRSISZ5QZH5M6CANQOI5A, seg_pos={0 0}, upload=196851]: Uploading piece... Aug 16 14:38:28.608477329 (189h12m23.03955745s) [noise=true , address=104.36.82.15:4257 , piece_id=LIPHLPFATOFWJZF4J76AINAUIS5VSHIIVDSX6KPLRI6OVCHHCYDA, seg_pos={0 0}, upload=196851]: Uploading piece... Aug 16 14:38:28.608686008 (189h12m23.039766139s) [noise=true , address=104.36.82.5:4839 , piece_id=HRK5K7DHNEAHKWT5K253NZGXY5WFJGUQC5FBIC5WA7HRYFUPJVJQ, seg_pos={0 0}, upload=196851]: Uploading piece... Aug 16 14:38:28.608763657 (189h12m23.039843788s) [noise=true , address=104.36.82.16:4294 , piece_id=STGMNSHI25GDZXT3ERUVDAXDYA663DUSGDWJRS3YYZ2IFOR6FBZA, seg_pos={0 0}, upload=196851]: Uploading piece... Aug 16 14:38:28.608887634 (189h12m23.039967755s) [noise=true , address=104.36.82.14:4222 , piece_id=Y4AW3OASTCMTWWJNIRWHOE7KY5V6NU3VVC2Z3SZO4J42AECIEMIA, seg_pos={0 0}, upload=196851]: Uploading piece... Aug 16 14:38:28.609100952 (189h12m23.040181083s) [noise=true , address=104.36.82.8:4972 , piece_id=B64SC5HA2QAD56PVIJIYWP5MOMRHQKQZOFQH64K64WR5ZPELY24A, seg_pos={0 0}, upload=196851]: Uploading piece... Aug 16 14:38:28.609302728 (189h12m23.040382859s) [noise=true , address=104.36.82.8:4969 , piece_id=AE7G2AEJOXLIJW6HJNECX4QHO5HSXZZCEK4DUJWKEC2LZNUKV64Q, seg_pos={0 0}, upload=196851]: Uploading piece... Aug 16 14:38:30.469886435 (189h12m24.900966566s) [upload=196851]: Next returned nil segment Aug 16 14:38:30.469903978 (189h12m24.900984109s) [upload=196851]: Waiting for error group managing segments... Aug 16 14:38:30.768921506 (189h12m25.200001637s) [noise=true , address=104.36.82.15:4257 , piece_id=LIPHLPFATOFWJZF4J76AINAUIS5VSHIIVDSX6KPLRI6OVCHHCYDA, seg_pos={0 0}, upload=196851]: Done uploading piece. err: <nil> Aug 16 14:38:32.325150942 (189h12m26.756231073s) [noise=true , address=77.90.31.15:4698 , piece_id=OLQWRAQXT3NMBEOLOOTBYLBORHMFTQCJXD5EM6VKM27R3XZ422SA, seg_pos={0 0}, upload=196851]: Done uploading piece. err: <nil> Aug 16 14:38:32.847922188 (189h12m27.279002309s) [noise=true , address=45.91.116.9:4261 , piece_id=TEV7WFGPMNWXZ5U3OJQO5UFSEQ2KLXKZSMT3M4FJCBYYSYISU6FQ, seg_pos={0 0}, upload=196851]: Done uploading piece. err: <nil> Aug 16 14:38:40.834000912 (189h12m35.265081043s) [noise=true , address=104.36.82.11:4114 , piece_id=4MDYK7SFDVNOP64CQ4TT5JZCGVMBWFPIJ3RF723GEE6RSMBAK2YA, seg_pos={0 0}, upload=196851]: Done uploading piece. err: <nil> Aug 16 14:38:43.340894372 (189h12m37.771974493s) [noise=true , address=104.36.82.14:4222 , piece_id=Y4AW3OASTCMTWWJNIRWHOE7KY5V6NU3VVC2Z3SZO4J42AECIEMIA, seg_pos={0 0}, upload=196851]: Done uploading piece. err: <nil> Aug 16 14:38:57.502769641 (189h12m51.933849772s) [noise=true , address=104.36.82.8:4972 , piece_id=B64SC5HA2QAD56PVIJIYWP5MOMRHQKQZOFQH64K64WR5ZPELY24A, seg_pos={0 0}, upload=196851]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:104.36.82.8:4972): protocol: read tcp 188.166.225.170:33492->104.36.82.8:4972: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:38:57.503458340 (189h12m51.934538471s) [noise=true , address=104.36.82.8:4969 , piece_id=AE7G2AEJOXLIJW6HJNECX4QHO5HSXZZCEK4DUJWKEC2LZNUKV64Q, seg_pos={0 0}, upload=196851]: Done uploading piece. err: ecclient: upload failed (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, address:104.36.82.8:4969): protocol: read tcp 188.166.225.170:50964->104.36.82.8:4969: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:38:57.503494539 (189h12m51.93457467s) [noise=true , address=104.36.82.13:4187 , piece_id=WXQA75RU3R44PT5YK43A2KUKCMHGJIQVAX7VWHMUADHGR3O3L4XQ, seg_pos={0 0}, upload=196851]: Done uploading piece. err: ecclient: upload failed (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, address:104.36.82.13:4187): protocol: read tcp 188.166.225.170:34918->104.36.82.13:4187: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:38:57.503523775 (189h12m51.934603906s) [noise=true , address=104.36.82.10:4079 , piece_id=PZZ2GRNWIE54I4VZ64GSZY4ALU67KWRMARPXIEQJDOFBTQ7BOHCA, seg_pos={0 0}, upload=196851]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:104.36.82.10:4079): protocol: read tcp 188.166.225.170:42862->104.36.82.10:4079: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:38:57.503553251 (189h12m51.934633382s) [noise=true , address=45.91.116.14:4190 , piece_id=J3W4F4DEDJRMO2N6HHIV6E7QWOMKTKZCT2XGR273CB7BXMZVZJJA, seg_pos={0 0}, upload=196851]: Done uploading piece. err: ecclient: upload failed (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, address:45.91.116.14:4190): protocol: read tcp 188.166.225.170:60582->45.91.116.14:4190: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:38:57.503582858 (189h12m51.934662979s) [noise=true , address=104.36.82.5:4839 , piece_id=HRK5K7DHNEAHKWT5K253NZGXY5WFJGUQC5FBIC5WA7HRYFUPJVJQ, seg_pos={0 0}, upload=196851]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:104.36.82.5:4839): protocol: read tcp 188.166.225.170:55072->104.36.82.5:4839: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:38:57.503611853 (189h12m51.934691984s) [noise=true , address=104.36.82.7:4912 , piece_id=ILEBNMPU7AQWGJ2WFLEM5ZJ3XLMS2ASEL7PR6EVBTY573LX4QJXA, seg_pos={0 0}, upload=196851]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:104.36.82.7:4912): protocol: read tcp 188.166.225.170:42580->104.36.82.7:4912: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:38:57.503641780 (189h12m51.934721911s) [noise=true , address=104.36.82.16:4294 , piece_id=STGMNSHI25GDZXT3ERUVDAXDYA663DUSGDWJRS3YYZ2IFOR6FBZA, seg_pos={0 0}, upload=196851]: Done uploading piece. err: ecclient: upload failed (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, address:104.36.82.16:4294): protocol: read tcp 188.166.225.170:51914->104.36.82.16:4294: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:38:57.503720110 (189h12m51.934800241s) [noise=true , address=45.91.116.11:4118 , piece_id=TIVE47XVPMFDP3ALBEJ2ICZ772BJ56ALBAY7QYPK7BYKRBYGBJQA, seg_pos={0 0}, upload=196851]: Done uploading piece. err: ecclient: upload failed (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, address:45.91.116.11:4118): protocol: write tcp 188.166.225.170:60026->45.91.116.11:4118: use of closed network connection; write tcp 188.166.225.170:60026->45.91.116.11:4118: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60026->45.91.116.11:4118: use of closed network connection Aug 16 14:38:57.503784724 (189h12m51.934864855s) [noise=true , address=104.36.82.12:4151 , piece_id=D4NQQ5DXBGW4MZM2IQJ2D5HBO4MCHDPMRSISZ5QZH5M6CANQOI5A, seg_pos={0 0}, upload=196851]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:104.36.82.12:4151): protocol: write tcp 188.166.225.170:43720->104.36.82.12:4151: use of closed network connection; write tcp 188.166.225.170:43720->104.36.82.12:4151: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43720->104.36.82.12:4151: use of closed network connection Aug 16 14:38:57.503899344 (189h12m51.934979475s) [noise=true , address=104.36.82.12:4150 , piece_id=J76RKL4JTHD4QLPYZYLJVYCK5PXJEZCTU2EDIXHVUTYHI54LIJXQ, seg_pos={0 0}, upload=196851]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:104.36.82.12:4150): protocol: context deadline exceeded; context deadline exceeded Aug 16 14:38:57.503920814 (189h12m51.935000935s) [seg_pos={0 0}, upload=196851]: Done waiting for segment. successful: 5 optimal: 13 errs: <nil> Aug 16 14:38:57.732081575 (189h12m52.163161706s) [upload=196851]: Done uploading segments... Aug 16 14:38:57.732106292 (189h12m52.163186424s) [upload=196851]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 5) Duration: 30.230887942s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [.......................................................X.X.X] duration: 1m0.231828146s FAIL (2025-08-16 14:38:52 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 9) ======LOGS====== 596928 (189h11m47.683677059s) [noise=true , address=104.36.82.4:4809 , piece_id=NNXWZXAQZRY2XQ4XSPYNFQMKN2BP7LIJSRJDPFIMMZWR5QU5Y7EQ, seg_pos={0 0}, upload=196840]: Uploading piece... Aug 16 14:37:53.252702400 (189h11m47.683782521s) [noise=true , address=104.36.82.58:4298 , piece_id=DZIAHLCPRWCHJY6YLXJ27R52V5QH232NK7Q3ZK27VJOVCASOOOZQ, seg_pos={0 0}, upload=196840]: Uploading piece... Aug 16 14:37:53.252893385 (189h11m47.683973506s) [noise=true , address=104.36.82.39:4754 , piece_id=PHK3X4GRDU66OINBECGGEVP2S6UER5W4HODYMHWKC3A2JPO3YPRA, seg_pos={0 0}, upload=196840]: Uploading piece... Aug 16 14:37:53.252963189 (189h11m47.68404332s) [noise=true , address=104.36.82.23:4476 , piece_id=5T42PWSGTXMKXI5HNT47K2ZAWA6K47JV3FI23Z7TWLOY7MYQPRCA, seg_pos={0 0}, upload=196840]: Uploading piece... Aug 16 14:37:53.253159545 (189h11m47.684239676s) [noise=true , address=45.91.116.10:4107 , piece_id=PQE22NFIZUFM4MSWTNF4KKNTYXOXPY2H6Y5CSBL4G3GZVJCVBZYA, seg_pos={0 0}, upload=196840]: Uploading piece... Aug 16 14:37:53.253235110 (189h11m47.684315231s) [noise=true , address=104.36.82.5:4863 , piece_id=GUKXQW4WKA3NBYFWBN5VU5USJ5GJHGTVCMBUMISVFJZEWJ33OR5A, seg_pos={0 0}, upload=196840]: Uploading piece... Aug 16 14:37:53.253427598 (189h11m47.684507719s) [noise=true , address=45.91.116.5:4300 , piece_id=UXOWJ3SOJKKW56O4BE3JQPYAPFNNELQLQLKJG6DXECAS5UAUXSHA, seg_pos={0 0}, upload=196840]: Uploading piece... Aug 16 14:37:53.253493484 (189h11m47.684573615s) [noise=true , address=77.90.31.19:4845 , piece_id=VOVPPIYLWCVKXRRH6SILENB5ZPYK7PL274CAOOS2S5YGO2ZK3BCA, seg_pos={0 0}, upload=196840]: Uploading piece... Aug 16 14:37:53.253662697 (189h11m47.684742828s) [noise=true , address=64.49.13.27:4957 , piece_id=GAR2RVVKA65EICURRKEHASEL7R5GF57JQ34O7AG7BFEBZRREDNRA, seg_pos={0 0}, upload=196840]: Uploading piece... Aug 16 14:37:53.253696582 (189h11m47.684776703s) [noise=true , address=104.36.82.44:4889 , piece_id=P4SO6FM4YAKS4ETAICGUV2UQSP2I5ZQH7XDEO6INOXNTFS3VKSCA, seg_pos={0 0}, upload=196840]: Uploading piece... Aug 16 14:37:53.253927855 (189h11m47.685007986s) [noise=true , address=79.127.201.215:4355 , piece_id=JQZ7LAECBG4AG7BR7EIIX4XPK6GGW7EZ6O2YLAORZIPQUKJ7IZGQ, seg_pos={0 0}, upload=196840]: Uploading piece... Aug 16 14:37:53.253931301 (189h11m47.685011433s) [noise=true , address=104.36.82.60:4344 , piece_id=UXFEVKB7PMIYGFOP43KHQSB7FWTUBCMX2JOPKZOYCSKXRW2PGAKA, seg_pos={0 0}, upload=196840]: Uploading piece... Aug 16 14:37:53.253963553 (189h11m47.685043674s) [noise=true , address=77.90.31.14:4689 , piece_id=74GUX7DCA2NY3O2B7C2WBBVAK4Q7BSPC3Q7UGUP63HHC5JU4U24A, seg_pos={0 0}, upload=196840]: Uploading piece... Aug 16 14:37:53.254177262 (189h11m47.685257393s) [noise=true , address=45.91.116.7:4016 , piece_id=OXIRHX36DTUQ3QGNMP4IRZPXL3XEI3D6XYRIABNVKLC2ECMIH3PQ, seg_pos={0 0}, upload=196840]: Uploading piece... Aug 16 14:37:53.254420648 (189h11m47.685500769s) [noise=true , address=77.90.31.16:4756 , piece_id=KTJ5YSYMK3UK7UEOFFGAJI4DQ3PUI5YDJCZPATTEUG2TNVDG5OSA, seg_pos={0 0}, upload=196840]: Uploading piece... Aug 16 14:37:54.367352702 (189h11m48.798432833s) [upload=196840]: Next returned nil segment Aug 16 14:37:54.367378431 (189h11m48.798458612s) [upload=196840]: Waiting for error group managing segments... Aug 16 14:37:54.673019622 (189h11m49.104099753s) [noise=true , address=64.49.13.27:4957 , piece_id=GAR2RVVKA65EICURRKEHASEL7R5GF57JQ34O7AG7BFEBZRREDNRA, seg_pos={0 0}, upload=196840]: Done uploading piece. err: <nil> Aug 16 14:37:54.682636463 (189h11m49.113716594s) [noise=true , address=66.23.193.175:4562 , piece_id=6QDXYAXM7O4MSFZXLBFA36M774DSEQFS663YTCG7UBD22DN42BBA, seg_pos={0 0}, upload=196840]: Done uploading piece. err: <nil> Aug 16 14:37:54.692050687 (189h11m49.123130818s) [noise=true , address=79.127.205.235:4680 , piece_id=I5OJ6U45RV43DKGKZF2P7AIRKJZJWKCQVRJCEIGKVSF4XMUXLFRA, seg_pos={0 0}, upload=196840]: Done uploading piece. err: <nil> Aug 16 14:37:55.381936968 (189h11m49.813017099s) [noise=true , address=79.127.201.215:4355 , piece_id=JQZ7LAECBG4AG7BR7EIIX4XPK6GGW7EZ6O2YLAORZIPQUKJ7IZGQ, seg_pos={0 0}, upload=196840]: Done uploading piece. err: <nil> Aug 16 14:37:57.017107398 (189h11m51.448187529s) [noise=true , address=104.36.82.5:4863 , piece_id=GUKXQW4WKA3NBYFWBN5VU5USJ5GJHGTVCMBUMISVFJZEWJ33OR5A, seg_pos={0 0}, upload=196840]: Done uploading piece. err: <nil> Aug 16 14:38:01.645860903 (189h11m56.076941034s) [noise=true , address=45.91.116.5:4300 , piece_id=UXOWJ3SOJKKW56O4BE3JQPYAPFNNELQLQLKJG6DXECAS5UAUXSHA, seg_pos={0 0}, upload=196840]: Done uploading piece. err: <nil> Aug 16 14:38:02.068115531 (189h11m56.499195662s) [noise=true , address=77.90.31.16:4756 , piece_id=KTJ5YSYMK3UK7UEOFFGAJI4DQ3PUI5YDJCZPATTEUG2TNVDG5OSA, seg_pos={0 0}, upload=196840]: Done uploading piece. err: <nil> Aug 16 14:38:03.414398135 (189h11m57.845478266s) [noise=true , address=45.91.116.10:4107 , piece_id=PQE22NFIZUFM4MSWTNF4KKNTYXOXPY2H6Y5CSBL4G3GZVJCVBZYA, seg_pos={0 0}, upload=196840]: Done uploading piece. err: <nil> Aug 16 14:38:07.705200098 (189h12m2.136280219s) [noise=true , address=104.36.82.44:4889 , piece_id=P4SO6FM4YAKS4ETAICGUV2UQSP2I5ZQH7XDEO6INOXNTFS3VKSCA, seg_pos={0 0}, upload=196840]: Done uploading piece. err: <nil> Aug 16 14:38:52.152866513 (189h12m46.583946644s) [noise=true , address=104.36.82.39:4754 , piece_id=PHK3X4GRDU66OINBECGGEVP2S6UER5W4HODYMHWKC3A2JPO3YPRA, seg_pos={0 0}, upload=196840]: Done uploading piece. err: ecclient: upload failed (node:1HB1v3ZURPMSYsbHBjJs1neE7HQtLS1KmeLwbKK2sTKAZcXPhi, address:104.36.82.39:4754): protocol: read tcp 188.166.225.170:53794->104.36.82.39:4754: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:38:52.152930194 (189h12m46.584010325s) [noise=true , address=45.91.116.7:4016 , piece_id=OXIRHX36DTUQ3QGNMP4IRZPXL3XEI3D6XYRIABNVKLC2ECMIH3PQ, seg_pos={0 0}, upload=196840]: Done uploading piece. err: ecclient: upload failed (node:1GeydXGpXPZzL65aqsaGiRQjz7NRzL9BZCUb5MEUWCAzNPMRPW, address:45.91.116.7:4016): protocol: read tcp 188.166.225.170:45274->45.91.116.7:4016: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:38:52.152974028 (189h12m46.584054169s) [noise=true , address=104.36.82.23:4476 , piece_id=5T42PWSGTXMKXI5HNT47K2ZAWA6K47JV3FI23Z7TWLOY7MYQPRCA, seg_pos={0 0}, upload=196840]: Done uploading piece. err: ecclient: upload failed (node:1foWwyGzFenUicfY7cTU5dmdDJsvXGR3YXv1rf3QRXMUco59TV, address:104.36.82.23:4476): protocol: read tcp 188.166.225.170:38786->104.36.82.23:4476: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:38:52.153016519 (189h12m46.58409666s) [noise=true , address=77.90.31.19:4845 , piece_id=VOVPPIYLWCVKXRRH6SILENB5ZPYK7PL274CAOOS2S5YGO2ZK3BCA, seg_pos={0 0}, upload=196840]: Done uploading piece. err: ecclient: upload failed (node:1ysgMdZQcnmzQ5Vfv3RadZN8widTZKrqDnDSthxkYEUXKgRxxF, address:77.90.31.19:4845): protocol: read tcp 188.166.225.170:50836->77.90.31.19:4845: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:38:52.153062367 (189h12m46.584142498s) [noise=true , address=77.90.31.14:4689 , piece_id=74GUX7DCA2NY3O2B7C2WBBVAK4Q7BSPC3Q7UGUP63HHC5JU4U24A, seg_pos={0 0}, upload=196840]: Done uploading piece. err: ecclient: upload failed (node:1bzvSPximqxjmXA4xtXZzD8Xyp27KURJ3oH1zCvbPmPwaLkuhM, address:77.90.31.14:4689): protocol: read tcp 188.166.225.170:36968->77.90.31.14:4689: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:38:52.153104217 (189h12m46.584184358s) [noise=true , address=104.36.82.4:4809 , piece_id=NNXWZXAQZRY2XQ4XSPYNFQMKN2BP7LIJSRJDPFIMMZWR5QU5Y7EQ, seg_pos={0 0}, upload=196840]: Done uploading piece. err: ecclient: upload failed (node:17LrYeCkrPWdiL9gGZ568PvfFct6cLVRQE4Qa92ykLgqXSfBL9, address:104.36.82.4:4809): protocol: read tcp 188.166.225.170:38082->104.36.82.4:4809: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:38:52.153144314 (189h12m46.584224445s) [noise=true , address=104.36.82.60:4344 , piece_id=UXFEVKB7PMIYGFOP43KHQSB7FWTUBCMX2JOPKZOYCSKXRW2PGAKA, seg_pos={0 0}, upload=196840]: Done uploading piece. err: ecclient: upload failed (node:127WWD5hr1ZBkRLCHz3jb7kpqqnxPS6VuZ1MyQDmitFbhTSbJNK, address:104.36.82.60:4344): protocol: read tcp 188.166.225.170:56238->104.36.82.60:4344: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:38:52.153182988 (189h12m46.584263119s) [noise=true , address=77.90.31.11:4582 , piece_id=QLEM3I4QDOC4P5ALXRELK7HYJXLL3LQH3IPXOLGH22U5BQNLE5WA, seg_pos={0 0}, upload=196840]: Done uploading piece. err: ecclient: upload failed (node:17VVqv2A8JgKbRYuqxsdnPhomFxdBaznCHMDV21XFsCpuAFBNx, address:77.90.31.11:4582): protocol: read tcp 188.166.225.170:36122->77.90.31.11:4582: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:38:52.153220901 (189h12m46.584301032s) [noise=true , address=104.36.82.58:4298 , piece_id=DZIAHLCPRWCHJY6YLXJ27R52V5QH232NK7Q3ZK27VJOVCASOOOZQ, seg_pos={0 0}, upload=196840]: Done uploading piece. err: ecclient: upload failed (node:1MWsyGhKkDhjngUDDRvxmhQddbNnRZ1A3mpzEtjLs9ZcRC5LBz, address:104.36.82.58:4298): protocol: read tcp 188.166.225.170:40164->104.36.82.58:4298: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 14:38:52.153241419 (189h12m46.58432155s) [seg_pos={0 0}, upload=196840]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil> Aug 16 14:38:52.382302142 (189h12m46.813382263s) [upload=196840]: Done uploading segments... Aug 16 14:38:52.382329385 (189h12m46.813409516s) [upload=196840]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9) Duration: 1m0.231848315s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect)

checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1)

checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1)

checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway)

checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1)

checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1)

checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway)

checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)