Orbiter History

checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect) antiblip: 1 out of 15 is failed (threshold=6) [.......X.......] duration: 297.174564ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/dcs-satellites trace: https://jaeger.dev.storj.io/trace/6aea9f2e04a3cb62 FAIL (2025-11-09 17:04:28 UTC) orbiter warning: Get "https://www.storj.io/dcs-satellites": read tcp [2400:6180:0:d0::f70:8001]:39466->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 297.205271ms core website
checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect) antiblip: 1 out of 15 is failed (threshold=6) [........X......] duration: 334.67884ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/dcs-satellites trace: https://jaeger.dev.storj.io/trace/6aea9f2e04a3cb62 FAIL (2025-11-09 17:03:28 UTC) orbiter warning: Get "https://www.storj.io/dcs-satellites": read tcp [2400:6180:0:d0::f70:8001]:39466->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 334.710479ms core website
checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect) antiblip: 1 out of 15 is failed (threshold=6) [.........X.....] duration: 353.510698ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/dcs-satellites trace: https://jaeger.dev.storj.io/trace/6aea9f2e04a3cb62 FAIL (2025-11-09 17:03:06 UTC) orbiter warning: Get "https://www.storj.io/dcs-satellites": read tcp [2400:6180:0:d0::f70:8001]:39466->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 353.541256ms core website
checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect) antiblip: 2 out of 15 is failed (threshold=6) [X.........X....] duration: 320.052546ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/dcs-satellites trace: https://jaeger.dev.storj.io/trace/6aea9f2e04a3cb62 FAIL (2025-11-09 17:02:18 UTC) orbiter warning: Get "https://www.storj.io/dcs-satellites": read tcp [2400:6180:0:d0::f70:8001]:39466->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 320.111309ms core website
checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect) antiblip: 2 out of 15 is failed (threshold=6) [.X.........X...] duration: 332.412525ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/dcs-satellites trace: https://jaeger.dev.storj.io/trace/6aea9f2e04a3cb62 FAIL (2025-11-09 17:01:47 UTC) orbiter warning: Get "https://www.storj.io/dcs-satellites": read tcp [2400:6180:0:d0::f70:8001]:39466->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 332.472759ms core website
checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect) antiblip: 2 out of 15 is failed (threshold=6) [..X.........X..] duration: 332.184811ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/dcs-satellites trace: https://jaeger.dev.storj.io/trace/6aea9f2e04a3cb62 FAIL (2025-11-09 17:00:41 UTC) orbiter warning: Get "https://www.storj.io/dcs-satellites": read tcp [2400:6180:0:d0::f70:8001]:39466->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 332.292475ms core website
checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect) antiblip: 2 out of 15 is failed (threshold=6) [...X.........X.] duration: 452.997424ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/dcs-satellites trace: https://jaeger.dev.storj.io/trace/6aea9f2e04a3cb62 FAIL (2025-11-09 16:59:49 UTC) orbiter warning: Get "https://www.storj.io/dcs-satellites": read tcp [2400:6180:0:d0::f70:8001]:39466->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 453.02673ms core website
checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect) antiblip: 2 out of 15 is failed (threshold=6) [....X.........X] duration: 233.70829ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/dcs-satellites trace: https://jaeger.dev.storj.io/trace/6aea9f2e04a3cb62 FAIL (2025-11-09 16:59:27 UTC) orbiter warning: Get "https://www.storj.io/dcs-satellites": read tcp [2400:6180:0:d0::f70:8001]:39466->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 233.737335ms core website
checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect) antiblip: 1 out of 15 is failed (threshold=6) [.....X.........] duration: 296.85264ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/dcs-satellites trace: https://jaeger.dev.storj.io/trace/6c05943a06037fce FAIL (2025-11-09 16:58:47 UTC) orbiter warning: Get "https://www.storj.io/dcs-satellites": read tcp [2400:6180:0:d0::f70:8001]:55606->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 296.879031ms core website
checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect) antiblip: 1 out of 15 is failed (threshold=6) [......X........] duration: 299.044607ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/dcs-satellites trace: https://jaeger.dev.storj.io/trace/6c05943a06037fce FAIL (2025-11-09 16:58:10 UTC) orbiter warning: Get "https://www.storj.io/dcs-satellites": read tcp [2400:6180:0:d0::f70:8001]:55606->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 299.064224ms core website
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 1 out of 15 is failed (threshold=6) [X..............] duration: 64.918067ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/5a29ab99611218c4 FAIL (2025-11-09 16:57:21 UTC) orbiter warning: Get "https://www.storj.io/": read tcp [2400:6180:0:d0::f70:8001]:39302->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 64.947635ms core website
checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect) antiblip: 1 out of 15 is failed (threshold=6) [.......X.......] duration: 348.893477ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/dcs-satellites trace: https://jaeger.dev.storj.io/trace/6c05943a06037fce FAIL (2025-11-09 16:57:21 UTC) orbiter warning: Get "https://www.storj.io/dcs-satellites": read tcp [2400:6180:0:d0::f70:8001]:55606->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 348.92142ms core website
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 1 out of 15 is failed (threshold=6) [.X.............] duration: 67.31929ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/5a29ab99611218c4 FAIL (2025-11-09 16:56:45 UTC) orbiter warning: Get "https://www.storj.io/": read tcp [2400:6180:0:d0::f70:8001]:39302->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 67.351291ms core website
checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect) antiblip: 1 out of 15 is failed (threshold=6) [........X......] duration: 304.34711ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/dcs-satellites trace: https://jaeger.dev.storj.io/trace/6c05943a06037fce FAIL (2025-11-09 16:56:36 UTC) orbiter warning: Get "https://www.storj.io/dcs-satellites": read tcp [2400:6180:0:d0::f70:8001]:55606->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 304.414509ms core website
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 1 out of 15 is failed (threshold=6) [..X............] duration: 61.352447ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/5a29ab99611218c4 FAIL (2025-11-09 16:56:06 UTC) orbiter warning: Get "https://www.storj.io/": read tcp [2400:6180:0:d0::f70:8001]:39302->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 61.384168ms core website
checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect) antiblip: 1 out of 15 is failed (threshold=6) [.........X.....] duration: 332.732774ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/dcs-satellites trace: https://jaeger.dev.storj.io/trace/6c05943a06037fce FAIL (2025-11-09 16:55:43 UTC) orbiter warning: Get "https://www.storj.io/dcs-satellites": read tcp [2400:6180:0:d0::f70:8001]:55606->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 332.761277ms core website
checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect) antiblip: 1 out of 15 is failed (threshold=6) [..........X....] duration: 324.882131ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/dcs-satellites trace: https://jaeger.dev.storj.io/trace/6c05943a06037fce FAIL (2025-11-09 16:55:14 UTC) orbiter warning: Get "https://www.storj.io/dcs-satellites": read tcp [2400:6180:0:d0::f70:8001]:55606->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 324.917279ms core website
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 1 out of 15 is failed (threshold=6) [...X...........] duration: 79.627358ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/5a29ab99611218c4 FAIL (2025-11-09 16:54:59 UTC) orbiter warning: Get "https://www.storj.io/": read tcp [2400:6180:0:d0::f70:8001]:39302->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 79.658296ms core website
checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect) antiblip: 1 out of 15 is failed (threshold=6) [...........X...] duration: 297.33533ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/dcs-satellites trace: https://jaeger.dev.storj.io/trace/6c05943a06037fce FAIL (2025-11-09 16:54:45 UTC) orbiter warning: Get "https://www.storj.io/dcs-satellites": read tcp [2400:6180:0:d0::f70:8001]:55606->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 297.349997ms core website
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 1 out of 15 is failed (threshold=6) [....X..........] duration: 56.525997ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/5a29ab99611218c4 FAIL (2025-11-09 16:54:42 UTC) orbiter warning: Get "https://www.storj.io/": read tcp [2400:6180:0:d0::f70:8001]:39302->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 56.591983ms core website
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 1 out of 15 is failed (threshold=6) [.....X.........] duration: 73.538313ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/5a29ab99611218c4 FAIL (2025-11-09 16:54:23 UTC) orbiter warning: Get "https://www.storj.io/": read tcp [2400:6180:0:d0::f70:8001]:39302->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 73.566746ms core website
checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect) antiblip: 1 out of 15 is failed (threshold=6) [............X..] duration: 307.264417ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/dcs-satellites trace: https://jaeger.dev.storj.io/trace/6c05943a06037fce FAIL (2025-11-09 16:53:46 UTC) orbiter warning: Get "https://www.storj.io/dcs-satellites": read tcp [2400:6180:0:d0::f70:8001]:55606->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 307.289535ms core website
checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect) antiblip: 1 out of 15 is failed (threshold=6) [.............X.] duration: 338.767035ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/dcs-satellites trace: https://jaeger.dev.storj.io/trace/6c05943a06037fce FAIL (2025-11-09 16:53:31 UTC) orbiter warning: Get "https://www.storj.io/dcs-satellites": read tcp [2400:6180:0:d0::f70:8001]:55606->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 338.797944ms core website
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 1 out of 15 is failed (threshold=6) [......X........] duration: 90.851141ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/5a29ab99611218c4 FAIL (2025-11-09 16:53:18 UTC) orbiter warning: Get "https://www.storj.io/": read tcp [2400:6180:0:d0::f70:8001]:39302->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 90.87737ms core website
checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect) antiblip: 1 out of 15 is failed (threshold=6) [..............X] duration: 637.943288ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/dcs-satellites trace: https://jaeger.dev.storj.io/trace/6c05943a06037fce FAIL (2025-11-09 16:52:24 UTC) orbiter warning: Get "https://www.storj.io/dcs-satellites": read tcp [2400:6180:0:d0::f70:8001]:55606->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 637.96566ms core website
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 1 out of 15 is failed (threshold=6) [.......X.......] duration: 55.163112ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/5a29ab99611218c4 FAIL (2025-11-09 16:52:07 UTC) orbiter warning: Get "https://www.storj.io/": read tcp [2400:6180:0:d0::f70:8001]:39302->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 55.185996ms core website
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 1 out of 15 is failed (threshold=6) [........X......] duration: 55.59247ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/5a29ab99611218c4 FAIL (2025-11-09 16:51:12 UTC) orbiter warning: Get "https://www.storj.io/": read tcp [2400:6180:0:d0::f70:8001]:39302->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 55.620744ms core website
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 1 out of 15 is failed (threshold=6) [.........X.....] duration: 77.026334ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/5a29ab99611218c4 FAIL (2025-11-09 16:50:30 UTC) orbiter warning: Get "https://www.storj.io/": read tcp [2400:6180:0:d0::f70:8001]:39302->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 77.099784ms core website
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 1 out of 15 is failed (threshold=6) [..........X....] duration: 62.00495ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/5a29ab99611218c4 FAIL (2025-11-09 16:49:49 UTC) orbiter warning: Get "https://www.storj.io/": read tcp [2400:6180:0:d0::f70:8001]:39302->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 62.031552ms core website
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 1 out of 15 is failed (threshold=6) [...........X...] duration: 82.14809ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/5a29ab99611218c4 FAIL (2025-11-09 16:49:27 UTC) orbiter warning: Get "https://www.storj.io/": read tcp [2400:6180:0:d0::f70:8001]:39302->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 82.181863ms core website
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 1 out of 15 is failed (threshold=6) [............X..] duration: 58.874661ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/5a29ab99611218c4 FAIL (2025-11-09 16:49:09 UTC) orbiter warning: Get "https://www.storj.io/": read tcp [2400:6180:0:d0::f70:8001]:39302->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 58.90081ms core website
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 1 out of 15 is failed (threshold=6) [.............X.] duration: 76.214218ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/5a29ab99611218c4 FAIL (2025-11-09 16:48:51 UTC) orbiter warning: Get "https://www.storj.io/": read tcp [2400:6180:0:d0::f70:8001]:39302->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 76.24136ms core website
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 1 out of 15 is failed (threshold=6) [..............X] duration: 220.661076ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/5a29ab99611218c4 FAIL (2025-11-09 16:47:48 UTC) orbiter warning: Get "https://www.storj.io/": read tcp [2400:6180:0:d0::f70:8001]:39302->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 220.677507ms core website
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 8.917835506s trace: https://jaeger.dev.storj.io/trace/3846516a5cecd778 FAIL (2025-11-09 16:26:55 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 9 15:40:06.410809146 (7h36m4.962279607s) [upload=6714]: Starting upload Nov 9 15:40:06.411046298 (7h36m4.962516769s) [upload=6714]: Done starting upload Nov 9 15:40:06.411314729 (7h36m4.96278519s) [upload=6714]: Uploading segments... Nov 9 15:40:06.411410882 (7h36m4.962881383s) [upload=6714]: Got next segment. Inline: false Nov 9 15:40:06.411464574 (7h36m4.962935035s) [upload=6714]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 9 15:40:06.411475064 (7h36m4.962945555s) [upload=6714]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 9 15:40:06.664789367 (7h36m5.216259828s) [seg_pos={0 0}, upload=6714]: Begin upload segment... Nov 9 15:40:06.664851705 (7h36m5.216322166s) [seg_pos={0 0}, upload=6714]: Done begin upload segment. Nov 9 15:40:06.664891662 (7h36m5.216362123s) [noise=true , address=145.79.227.32:4835 , piece_id=EYIBNSVPQPKLKOBHKZI2VZHVCDKW3C3YJSFEWP4G62UTBXX2YRZA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.664996933 (7h36m5.216467394s) [noise=true , address=45.91.116.14:4190 , piece_id=XSIDZL5HMLZTQE7ZAPURAAUGQLPP3QOZK5BIE5PHRMZPSTN5APSA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.665698789 (7h36m5.21716925s) [noise=true , address=77.90.31.14:4662 , piece_id=HHSJDZPYKGCHURAZV46OQSGBU7D4TFPXREDBFPGTYF7OAUMT2X3A, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.666456111 (7h36m5.217926572s) [noise=true , address=145.79.227.25:4619 , piece_id=23D3MCZRVNPI5KB43ROAHKUALDGXLZBZU5E6EIU6U2FY3YQOVPGA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.666642225 (7h36m5.218112676s) [noise=true , address=145.79.227.17:4971 , piece_id=VTUI377UCF44ICILORHEKUO33ICRXM2FECO5EXS7RUDH4VGV7FJA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.667027539 (7h36m5.218498s) [noise=true , address=145.79.227.9:4366 , piece_id=BT3GWQSXIMMSFPUYJC5EQ2LX2DS4BW5DAMW6ILLNVVRUUHZMOQUA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.667041675 (7h36m5.218512126s) [noise=true , address=145.79.227.17:4972 , piece_id=AI3KVNUJKJUD33YVU2Z472R72SKQKRYQG65JPKFZ5WBJLKRLW6MA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.667502794 (7h36m5.218973255s) [noise=true , address=145.79.227.13:4803 , piece_id=V37DZKBC4CHRGISXEQHTQAGKZFNXQKWXSWBSMHNRQTSJ32WYI2OQ, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.667805441 (7h36m5.219275902s) [noise=true , address=145.79.227.9:4365 , piece_id=7BK6E45EY6Y3SV6PY3A4EVCS42PC4HYLXU3J7MN4HXRLDEFUEFCA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.667895652 (7h36m5.219366113s) [noise=true , address=145.79.227.17:4970 , piece_id=5ZP5DDC3XAYQ5UFYJVQV6IZHB5KQQGNG54MTYW3UUYPN2IAK5QVQ, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.668240909 (7h36m5.219711361s) [noise=true , address=77.90.31.17:4772 , piece_id=CT6BNOYZGDBUEFVGM2RDS4ICMWPYS6N4IOUDA5V7MJQ3IKELNW2Q, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.668671640 (7h36m5.220142101s) [noise=true , address=145.79.227.31:4797 , piece_id=VQEEGJTLJY2IMCDHL2GMVGYEX4VL34TMMXKDYNERFYK3HRV7TC6A, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.669093554 (7h36m5.220564015s) [noise=true , address=145.79.227.21:4474 , piece_id=J5XE2MTCEW4ZHQDDZ4KCVYFK5EX6RQZARKNY4QCH4HUHGJO3SXDQ, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.669510097 (7h36m5.220980568s) [noise=true , address=77.90.31.18:4807 , piece_id=HDJAJYXNZNZN3LVAQSYVJEOQ3MU5H3BXCCW3NA2GLKPT7ATZHSJQ, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.671936887 (7h36m5.223407348s) [noise=true , address=145.79.227.20:4439 , piece_id=PPOVWSRAU7Y4CZAKAXV6ZP74EEPHONUC35O6XM57VE47NVJL5WZA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.672494549 (7h36m5.22396502s) [noise=true , address=77.90.31.21:4877 , piece_id=3V3Z6GNF32Q66EOS66WV3AGJNA7IXT5NHVAPKNUGVYU3R46IKKKA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:08.438197970 (7h36m6.989668451s) [upload=6714]: Next returned nil segment Nov 9 15:40:08.438219050 (7h36m6.989689521s) [upload=6714]: Waiting for error group managing segments... Nov 9 15:40:08.829503696 (7h36m7.380974147s) [noise=true , address=77.90.31.18:4807 , piece_id=HDJAJYXNZNZN3LVAQSYVJEOQ3MU5H3BXCCW3NA2GLKPT7ATZHSJQ, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:08.846923116 (7h36m7.398393577s) [noise=true , address=45.91.116.14:4190 , piece_id=XSIDZL5HMLZTQE7ZAPURAAUGQLPP3QOZK5BIE5PHRMZPSTN5APSA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:08.867005608 (7h36m7.418476069s) [noise=true , address=77.90.31.21:4877 , piece_id=3V3Z6GNF32Q66EOS66WV3AGJNA7IXT5NHVAPKNUGVYU3R46IKKKA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:08.888352807 (7h36m7.439823268s) [noise=true , address=77.90.31.14:4662 , piece_id=HHSJDZPYKGCHURAZV46OQSGBU7D4TFPXREDBFPGTYF7OAUMT2X3A, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:08.904680649 (7h36m7.45615111s) [noise=true , address=77.90.31.17:4772 , piece_id=CT6BNOYZGDBUEFVGM2RDS4ICMWPYS6N4IOUDA5V7MJQ3IKELNW2Q, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:18.663082162 (7h36m17.214552623s) [noise=true , address=145.79.227.17:4971 , piece_id=VTUI377UCF44ICILORHEKUO33ICRXM2FECO5EXS7RUDH4VGV7FJA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:20.077194714 (7h36m18.628665185s) [noise=true , address=145.79.227.13:4803 , piece_id=V37DZKBC4CHRGISXEQHTQAGKZFNXQKWXSWBSMHNRQTSJ32WYI2OQ, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:28.517900114 (7h36m27.069370575s) [noise=true , address=145.79.227.32:4835 , piece_id=EYIBNSVPQPKLKOBHKZI2VZHVCDKW3C3YJSFEWP4G62UTBXX2YRZA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:28.567571990 (7h36m27.119042451s) [noise=true , address=145.79.227.31:4797 , piece_id=VQEEGJTLJY2IMCDHL2GMVGYEX4VL34TMMXKDYNERFYK3HRV7TC6A, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:33.651382593 (7h36m32.202853044s) [noise=true , address=145.79.227.20:4439 , piece_id=PPOVWSRAU7Y4CZAKAXV6ZP74EEPHONUC35O6XM57VE47NVJL5WZA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:34.842221635 (7h36m33.393692096s) [noise=true , address=145.79.227.9:4365 , piece_id=7BK6E45EY6Y3SV6PY3A4EVCS42PC4HYLXU3J7MN4HXRLDEFUEFCA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:35.601406517 (7h36m34.152877008s) [noise=true , address=145.79.227.25:4619 , piece_id=23D3MCZRVNPI5KB43ROAHKUALDGXLZBZU5E6EIU6U2FY3YQOVPGA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:35.697082303 (7h36m34.248552774s) [noise=true , address=145.79.227.17:4970 , piece_id=5ZP5DDC3XAYQ5UFYJVQV6IZHB5KQQGNG54MTYW3UUYPN2IAK5QVQ, seg_pos={0 0}, upload=6714]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: expected piece hash; context deadline exceeded; EOF Nov 9 15:40:35.697252518 (7h36m34.248722969s) [noise=true , address=145.79.227.17:4972 , piece_id=AI3KVNUJKJUD33YVU2Z472R72SKQKRYQG65JPKFZ5WBJLKRLW6MA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:145.79.227.17:4972): protocol: expected piece hash; context deadline exceeded; EOF Nov 9 15:40:35.697302322 (7h36m34.248772773s) [noise=true , address=145.79.227.21:4474 , piece_id=J5XE2MTCEW4ZHQDDZ4KCVYFK5EX6RQZARKNY4QCH4HUHGJO3SXDQ, seg_pos={0 0}, upload=6714]: Done uploading piece. err: ecclient: upload failed (node:1rL2DnjSgHP36Md3bCgy5padqTAhv3iQCeURbLMuuuMBxtUNZH, address:145.79.227.21:4474): protocol: expected piece hash; context deadline exceeded; EOF Nov 9 15:40:35.697472546 (7h36m34.248943007s) [noise=true , address=145.79.227.9:4366 , piece_id=BT3GWQSXIMMSFPUYJC5EQ2LX2DS4BW5DAMW6ILLNVVRUUHZMOQUA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: expected piece hash; context deadline exceeded; EOF Nov 9 15:40:35.697514686 (7h36m34.248985147s) [seg_pos={0 0}, upload=6714]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 9 15:40:35.950211122 (7h36m34.501681583s) [upload=6714]: Done uploading segments... Nov 9 15:40:35.950416643 (7h36m34.501887104s) [upload=6714]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 8.91786436s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 11.764389384s trace: https://jaeger.dev.storj.io/trace/3846516a5cecd778 FAIL (2025-11-09 16:26:35 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 9 15:40:06.410809146 (7h36m4.962279607s) [upload=6714]: Starting upload Nov 9 15:40:06.411046298 (7h36m4.962516769s) [upload=6714]: Done starting upload Nov 9 15:40:06.411314729 (7h36m4.96278519s) [upload=6714]: Uploading segments... Nov 9 15:40:06.411410882 (7h36m4.962881383s) [upload=6714]: Got next segment. Inline: false Nov 9 15:40:06.411464574 (7h36m4.962935035s) [upload=6714]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 9 15:40:06.411475064 (7h36m4.962945555s) [upload=6714]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 9 15:40:06.664789367 (7h36m5.216259828s) [seg_pos={0 0}, upload=6714]: Begin upload segment... Nov 9 15:40:06.664851705 (7h36m5.216322166s) [seg_pos={0 0}, upload=6714]: Done begin upload segment. Nov 9 15:40:06.664891662 (7h36m5.216362123s) [noise=true , address=145.79.227.32:4835 , piece_id=EYIBNSVPQPKLKOBHKZI2VZHVCDKW3C3YJSFEWP4G62UTBXX2YRZA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.664996933 (7h36m5.216467394s) [noise=true , address=45.91.116.14:4190 , piece_id=XSIDZL5HMLZTQE7ZAPURAAUGQLPP3QOZK5BIE5PHRMZPSTN5APSA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.665698789 (7h36m5.21716925s) [noise=true , address=77.90.31.14:4662 , piece_id=HHSJDZPYKGCHURAZV46OQSGBU7D4TFPXREDBFPGTYF7OAUMT2X3A, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.666456111 (7h36m5.217926572s) [noise=true , address=145.79.227.25:4619 , piece_id=23D3MCZRVNPI5KB43ROAHKUALDGXLZBZU5E6EIU6U2FY3YQOVPGA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.666642225 (7h36m5.218112676s) [noise=true , address=145.79.227.17:4971 , piece_id=VTUI377UCF44ICILORHEKUO33ICRXM2FECO5EXS7RUDH4VGV7FJA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.667027539 (7h36m5.218498s) [noise=true , address=145.79.227.9:4366 , piece_id=BT3GWQSXIMMSFPUYJC5EQ2LX2DS4BW5DAMW6ILLNVVRUUHZMOQUA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.667041675 (7h36m5.218512126s) [noise=true , address=145.79.227.17:4972 , piece_id=AI3KVNUJKJUD33YVU2Z472R72SKQKRYQG65JPKFZ5WBJLKRLW6MA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.667502794 (7h36m5.218973255s) [noise=true , address=145.79.227.13:4803 , piece_id=V37DZKBC4CHRGISXEQHTQAGKZFNXQKWXSWBSMHNRQTSJ32WYI2OQ, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.667805441 (7h36m5.219275902s) [noise=true , address=145.79.227.9:4365 , piece_id=7BK6E45EY6Y3SV6PY3A4EVCS42PC4HYLXU3J7MN4HXRLDEFUEFCA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.667895652 (7h36m5.219366113s) [noise=true , address=145.79.227.17:4970 , piece_id=5ZP5DDC3XAYQ5UFYJVQV6IZHB5KQQGNG54MTYW3UUYPN2IAK5QVQ, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.668240909 (7h36m5.219711361s) [noise=true , address=77.90.31.17:4772 , piece_id=CT6BNOYZGDBUEFVGM2RDS4ICMWPYS6N4IOUDA5V7MJQ3IKELNW2Q, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.668671640 (7h36m5.220142101s) [noise=true , address=145.79.227.31:4797 , piece_id=VQEEGJTLJY2IMCDHL2GMVGYEX4VL34TMMXKDYNERFYK3HRV7TC6A, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.669093554 (7h36m5.220564015s) [noise=true , address=145.79.227.21:4474 , piece_id=J5XE2MTCEW4ZHQDDZ4KCVYFK5EX6RQZARKNY4QCH4HUHGJO3SXDQ, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.669510097 (7h36m5.220980568s) [noise=true , address=77.90.31.18:4807 , piece_id=HDJAJYXNZNZN3LVAQSYVJEOQ3MU5H3BXCCW3NA2GLKPT7ATZHSJQ, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.671936887 (7h36m5.223407348s) [noise=true , address=145.79.227.20:4439 , piece_id=PPOVWSRAU7Y4CZAKAXV6ZP74EEPHONUC35O6XM57VE47NVJL5WZA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.672494549 (7h36m5.22396502s) [noise=true , address=77.90.31.21:4877 , piece_id=3V3Z6GNF32Q66EOS66WV3AGJNA7IXT5NHVAPKNUGVYU3R46IKKKA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:08.438197970 (7h36m6.989668451s) [upload=6714]: Next returned nil segment Nov 9 15:40:08.438219050 (7h36m6.989689521s) [upload=6714]: Waiting for error group managing segments... Nov 9 15:40:08.829503696 (7h36m7.380974147s) [noise=true , address=77.90.31.18:4807 , piece_id=HDJAJYXNZNZN3LVAQSYVJEOQ3MU5H3BXCCW3NA2GLKPT7ATZHSJQ, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:08.846923116 (7h36m7.398393577s) [noise=true , address=45.91.116.14:4190 , piece_id=XSIDZL5HMLZTQE7ZAPURAAUGQLPP3QOZK5BIE5PHRMZPSTN5APSA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:08.867005608 (7h36m7.418476069s) [noise=true , address=77.90.31.21:4877 , piece_id=3V3Z6GNF32Q66EOS66WV3AGJNA7IXT5NHVAPKNUGVYU3R46IKKKA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:08.888352807 (7h36m7.439823268s) [noise=true , address=77.90.31.14:4662 , piece_id=HHSJDZPYKGCHURAZV46OQSGBU7D4TFPXREDBFPGTYF7OAUMT2X3A, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:08.904680649 (7h36m7.45615111s) [noise=true , address=77.90.31.17:4772 , piece_id=CT6BNOYZGDBUEFVGM2RDS4ICMWPYS6N4IOUDA5V7MJQ3IKELNW2Q, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:18.663082162 (7h36m17.214552623s) [noise=true , address=145.79.227.17:4971 , piece_id=VTUI377UCF44ICILORHEKUO33ICRXM2FECO5EXS7RUDH4VGV7FJA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:20.077194714 (7h36m18.628665185s) [noise=true , address=145.79.227.13:4803 , piece_id=V37DZKBC4CHRGISXEQHTQAGKZFNXQKWXSWBSMHNRQTSJ32WYI2OQ, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:28.517900114 (7h36m27.069370575s) [noise=true , address=145.79.227.32:4835 , piece_id=EYIBNSVPQPKLKOBHKZI2VZHVCDKW3C3YJSFEWP4G62UTBXX2YRZA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:28.567571990 (7h36m27.119042451s) [noise=true , address=145.79.227.31:4797 , piece_id=VQEEGJTLJY2IMCDHL2GMVGYEX4VL34TMMXKDYNERFYK3HRV7TC6A, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:33.651382593 (7h36m32.202853044s) [noise=true , address=145.79.227.20:4439 , piece_id=PPOVWSRAU7Y4CZAKAXV6ZP74EEPHONUC35O6XM57VE47NVJL5WZA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:34.842221635 (7h36m33.393692096s) [noise=true , address=145.79.227.9:4365 , piece_id=7BK6E45EY6Y3SV6PY3A4EVCS42PC4HYLXU3J7MN4HXRLDEFUEFCA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:35.601406517 (7h36m34.152877008s) [noise=true , address=145.79.227.25:4619 , piece_id=23D3MCZRVNPI5KB43ROAHKUALDGXLZBZU5E6EIU6U2FY3YQOVPGA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:35.697082303 (7h36m34.248552774s) [noise=true , address=145.79.227.17:4970 , piece_id=5ZP5DDC3XAYQ5UFYJVQV6IZHB5KQQGNG54MTYW3UUYPN2IAK5QVQ, seg_pos={0 0}, upload=6714]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: expected piece hash; context deadline exceeded; EOF Nov 9 15:40:35.697252518 (7h36m34.248722969s) [noise=true , address=145.79.227.17:4972 , piece_id=AI3KVNUJKJUD33YVU2Z472R72SKQKRYQG65JPKFZ5WBJLKRLW6MA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:145.79.227.17:4972): protocol: expected piece hash; context deadline exceeded; EOF Nov 9 15:40:35.697302322 (7h36m34.248772773s) [noise=true , address=145.79.227.21:4474 , piece_id=J5XE2MTCEW4ZHQDDZ4KCVYFK5EX6RQZARKNY4QCH4HUHGJO3SXDQ, seg_pos={0 0}, upload=6714]: Done uploading piece. err: ecclient: upload failed (node:1rL2DnjSgHP36Md3bCgy5padqTAhv3iQCeURbLMuuuMBxtUNZH, address:145.79.227.21:4474): protocol: expected piece hash; context deadline exceeded; EOF Nov 9 15:40:35.697472546 (7h36m34.248943007s) [noise=true , address=145.79.227.9:4366 , piece_id=BT3GWQSXIMMSFPUYJC5EQ2LX2DS4BW5DAMW6ILLNVVRUUHZMOQUA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: expected piece hash; context deadline exceeded; EOF Nov 9 15:40:35.697514686 (7h36m34.248985147s) [seg_pos={0 0}, upload=6714]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 9 15:40:35.950211122 (7h36m34.501681583s) [upload=6714]: Done uploading segments... Nov 9 15:40:35.950416643 (7h36m34.501887104s) [upload=6714]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 11.764418699s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 9.478093028s trace: https://jaeger.dev.storj.io/trace/3846516a5cecd778 FAIL (2025-11-09 16:26:09 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 9 15:40:06.410809146 (7h36m4.962279607s) [upload=6714]: Starting upload Nov 9 15:40:06.411046298 (7h36m4.962516769s) [upload=6714]: Done starting upload Nov 9 15:40:06.411314729 (7h36m4.96278519s) [upload=6714]: Uploading segments... Nov 9 15:40:06.411410882 (7h36m4.962881383s) [upload=6714]: Got next segment. Inline: false Nov 9 15:40:06.411464574 (7h36m4.962935035s) [upload=6714]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 9 15:40:06.411475064 (7h36m4.962945555s) [upload=6714]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 9 15:40:06.664789367 (7h36m5.216259828s) [seg_pos={0 0}, upload=6714]: Begin upload segment... Nov 9 15:40:06.664851705 (7h36m5.216322166s) [seg_pos={0 0}, upload=6714]: Done begin upload segment. Nov 9 15:40:06.664891662 (7h36m5.216362123s) [noise=true , address=145.79.227.32:4835 , piece_id=EYIBNSVPQPKLKOBHKZI2VZHVCDKW3C3YJSFEWP4G62UTBXX2YRZA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.664996933 (7h36m5.216467394s) [noise=true , address=45.91.116.14:4190 , piece_id=XSIDZL5HMLZTQE7ZAPURAAUGQLPP3QOZK5BIE5PHRMZPSTN5APSA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.665698789 (7h36m5.21716925s) [noise=true , address=77.90.31.14:4662 , piece_id=HHSJDZPYKGCHURAZV46OQSGBU7D4TFPXREDBFPGTYF7OAUMT2X3A, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.666456111 (7h36m5.217926572s) [noise=true , address=145.79.227.25:4619 , piece_id=23D3MCZRVNPI5KB43ROAHKUALDGXLZBZU5E6EIU6U2FY3YQOVPGA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.666642225 (7h36m5.218112676s) [noise=true , address=145.79.227.17:4971 , piece_id=VTUI377UCF44ICILORHEKUO33ICRXM2FECO5EXS7RUDH4VGV7FJA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.667027539 (7h36m5.218498s) [noise=true , address=145.79.227.9:4366 , piece_id=BT3GWQSXIMMSFPUYJC5EQ2LX2DS4BW5DAMW6ILLNVVRUUHZMOQUA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.667041675 (7h36m5.218512126s) [noise=true , address=145.79.227.17:4972 , piece_id=AI3KVNUJKJUD33YVU2Z472R72SKQKRYQG65JPKFZ5WBJLKRLW6MA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.667502794 (7h36m5.218973255s) [noise=true , address=145.79.227.13:4803 , piece_id=V37DZKBC4CHRGISXEQHTQAGKZFNXQKWXSWBSMHNRQTSJ32WYI2OQ, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.667805441 (7h36m5.219275902s) [noise=true , address=145.79.227.9:4365 , piece_id=7BK6E45EY6Y3SV6PY3A4EVCS42PC4HYLXU3J7MN4HXRLDEFUEFCA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.667895652 (7h36m5.219366113s) [noise=true , address=145.79.227.17:4970 , piece_id=5ZP5DDC3XAYQ5UFYJVQV6IZHB5KQQGNG54MTYW3UUYPN2IAK5QVQ, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.668240909 (7h36m5.219711361s) [noise=true , address=77.90.31.17:4772 , piece_id=CT6BNOYZGDBUEFVGM2RDS4ICMWPYS6N4IOUDA5V7MJQ3IKELNW2Q, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.668671640 (7h36m5.220142101s) [noise=true , address=145.79.227.31:4797 , piece_id=VQEEGJTLJY2IMCDHL2GMVGYEX4VL34TMMXKDYNERFYK3HRV7TC6A, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.669093554 (7h36m5.220564015s) [noise=true , address=145.79.227.21:4474 , piece_id=J5XE2MTCEW4ZHQDDZ4KCVYFK5EX6RQZARKNY4QCH4HUHGJO3SXDQ, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.669510097 (7h36m5.220980568s) [noise=true , address=77.90.31.18:4807 , piece_id=HDJAJYXNZNZN3LVAQSYVJEOQ3MU5H3BXCCW3NA2GLKPT7ATZHSJQ, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.671936887 (7h36m5.223407348s) [noise=true , address=145.79.227.20:4439 , piece_id=PPOVWSRAU7Y4CZAKAXV6ZP74EEPHONUC35O6XM57VE47NVJL5WZA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.672494549 (7h36m5.22396502s) [noise=true , address=77.90.31.21:4877 , piece_id=3V3Z6GNF32Q66EOS66WV3AGJNA7IXT5NHVAPKNUGVYU3R46IKKKA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:08.438197970 (7h36m6.989668451s) [upload=6714]: Next returned nil segment Nov 9 15:40:08.438219050 (7h36m6.989689521s) [upload=6714]: Waiting for error group managing segments... Nov 9 15:40:08.829503696 (7h36m7.380974147s) [noise=true , address=77.90.31.18:4807 , piece_id=HDJAJYXNZNZN3LVAQSYVJEOQ3MU5H3BXCCW3NA2GLKPT7ATZHSJQ, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:08.846923116 (7h36m7.398393577s) [noise=true , address=45.91.116.14:4190 , piece_id=XSIDZL5HMLZTQE7ZAPURAAUGQLPP3QOZK5BIE5PHRMZPSTN5APSA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:08.867005608 (7h36m7.418476069s) [noise=true , address=77.90.31.21:4877 , piece_id=3V3Z6GNF32Q66EOS66WV3AGJNA7IXT5NHVAPKNUGVYU3R46IKKKA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:08.888352807 (7h36m7.439823268s) [noise=true , address=77.90.31.14:4662 , piece_id=HHSJDZPYKGCHURAZV46OQSGBU7D4TFPXREDBFPGTYF7OAUMT2X3A, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:08.904680649 (7h36m7.45615111s) [noise=true , address=77.90.31.17:4772 , piece_id=CT6BNOYZGDBUEFVGM2RDS4ICMWPYS6N4IOUDA5V7MJQ3IKELNW2Q, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:18.663082162 (7h36m17.214552623s) [noise=true , address=145.79.227.17:4971 , piece_id=VTUI377UCF44ICILORHEKUO33ICRXM2FECO5EXS7RUDH4VGV7FJA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:20.077194714 (7h36m18.628665185s) [noise=true , address=145.79.227.13:4803 , piece_id=V37DZKBC4CHRGISXEQHTQAGKZFNXQKWXSWBSMHNRQTSJ32WYI2OQ, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:28.517900114 (7h36m27.069370575s) [noise=true , address=145.79.227.32:4835 , piece_id=EYIBNSVPQPKLKOBHKZI2VZHVCDKW3C3YJSFEWP4G62UTBXX2YRZA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:28.567571990 (7h36m27.119042451s) [noise=true , address=145.79.227.31:4797 , piece_id=VQEEGJTLJY2IMCDHL2GMVGYEX4VL34TMMXKDYNERFYK3HRV7TC6A, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:33.651382593 (7h36m32.202853044s) [noise=true , address=145.79.227.20:4439 , piece_id=PPOVWSRAU7Y4CZAKAXV6ZP74EEPHONUC35O6XM57VE47NVJL5WZA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:34.842221635 (7h36m33.393692096s) [noise=true , address=145.79.227.9:4365 , piece_id=7BK6E45EY6Y3SV6PY3A4EVCS42PC4HYLXU3J7MN4HXRLDEFUEFCA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:35.601406517 (7h36m34.152877008s) [noise=true , address=145.79.227.25:4619 , piece_id=23D3MCZRVNPI5KB43ROAHKUALDGXLZBZU5E6EIU6U2FY3YQOVPGA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:35.697082303 (7h36m34.248552774s) [noise=true , address=145.79.227.17:4970 , piece_id=5ZP5DDC3XAYQ5UFYJVQV6IZHB5KQQGNG54MTYW3UUYPN2IAK5QVQ, seg_pos={0 0}, upload=6714]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: expected piece hash; context deadline exceeded; EOF Nov 9 15:40:35.697252518 (7h36m34.248722969s) [noise=true , address=145.79.227.17:4972 , piece_id=AI3KVNUJKJUD33YVU2Z472R72SKQKRYQG65JPKFZ5WBJLKRLW6MA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:145.79.227.17:4972): protocol: expected piece hash; context deadline exceeded; EOF Nov 9 15:40:35.697302322 (7h36m34.248772773s) [noise=true , address=145.79.227.21:4474 , piece_id=J5XE2MTCEW4ZHQDDZ4KCVYFK5EX6RQZARKNY4QCH4HUHGJO3SXDQ, seg_pos={0 0}, upload=6714]: Done uploading piece. err: ecclient: upload failed (node:1rL2DnjSgHP36Md3bCgy5padqTAhv3iQCeURbLMuuuMBxtUNZH, address:145.79.227.21:4474): protocol: expected piece hash; context deadline exceeded; EOF Nov 9 15:40:35.697472546 (7h36m34.248943007s) [noise=true , address=145.79.227.9:4366 , piece_id=BT3GWQSXIMMSFPUYJC5EQ2LX2DS4BW5DAMW6ILLNVVRUUHZMOQUA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: expected piece hash; context deadline exceeded; EOF Nov 9 15:40:35.697514686 (7h36m34.248985147s) [seg_pos={0 0}, upload=6714]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 9 15:40:35.950211122 (7h36m34.501681583s) [upload=6714]: Done uploading segments... Nov 9 15:40:35.950416643 (7h36m34.501887104s) [upload=6714]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 9.47812557s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 11.008290787s trace: https://jaeger.dev.storj.io/trace/3846516a5cecd778 FAIL (2025-11-09 16:25:23 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 9 15:40:06.410809146 (7h36m4.962279607s) [upload=6714]: Starting upload Nov 9 15:40:06.411046298 (7h36m4.962516769s) [upload=6714]: Done starting upload Nov 9 15:40:06.411314729 (7h36m4.96278519s) [upload=6714]: Uploading segments... Nov 9 15:40:06.411410882 (7h36m4.962881383s) [upload=6714]: Got next segment. Inline: false Nov 9 15:40:06.411464574 (7h36m4.962935035s) [upload=6714]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 9 15:40:06.411475064 (7h36m4.962945555s) [upload=6714]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 9 15:40:06.664789367 (7h36m5.216259828s) [seg_pos={0 0}, upload=6714]: Begin upload segment... Nov 9 15:40:06.664851705 (7h36m5.216322166s) [seg_pos={0 0}, upload=6714]: Done begin upload segment. Nov 9 15:40:06.664891662 (7h36m5.216362123s) [noise=true , address=145.79.227.32:4835 , piece_id=EYIBNSVPQPKLKOBHKZI2VZHVCDKW3C3YJSFEWP4G62UTBXX2YRZA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.664996933 (7h36m5.216467394s) [noise=true , address=45.91.116.14:4190 , piece_id=XSIDZL5HMLZTQE7ZAPURAAUGQLPP3QOZK5BIE5PHRMZPSTN5APSA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.665698789 (7h36m5.21716925s) [noise=true , address=77.90.31.14:4662 , piece_id=HHSJDZPYKGCHURAZV46OQSGBU7D4TFPXREDBFPGTYF7OAUMT2X3A, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.666456111 (7h36m5.217926572s) [noise=true , address=145.79.227.25:4619 , piece_id=23D3MCZRVNPI5KB43ROAHKUALDGXLZBZU5E6EIU6U2FY3YQOVPGA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.666642225 (7h36m5.218112676s) [noise=true , address=145.79.227.17:4971 , piece_id=VTUI377UCF44ICILORHEKUO33ICRXM2FECO5EXS7RUDH4VGV7FJA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.667027539 (7h36m5.218498s) [noise=true , address=145.79.227.9:4366 , piece_id=BT3GWQSXIMMSFPUYJC5EQ2LX2DS4BW5DAMW6ILLNVVRUUHZMOQUA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.667041675 (7h36m5.218512126s) [noise=true , address=145.79.227.17:4972 , piece_id=AI3KVNUJKJUD33YVU2Z472R72SKQKRYQG65JPKFZ5WBJLKRLW6MA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.667502794 (7h36m5.218973255s) [noise=true , address=145.79.227.13:4803 , piece_id=V37DZKBC4CHRGISXEQHTQAGKZFNXQKWXSWBSMHNRQTSJ32WYI2OQ, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.667805441 (7h36m5.219275902s) [noise=true , address=145.79.227.9:4365 , piece_id=7BK6E45EY6Y3SV6PY3A4EVCS42PC4HYLXU3J7MN4HXRLDEFUEFCA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.667895652 (7h36m5.219366113s) [noise=true , address=145.79.227.17:4970 , piece_id=5ZP5DDC3XAYQ5UFYJVQV6IZHB5KQQGNG54MTYW3UUYPN2IAK5QVQ, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.668240909 (7h36m5.219711361s) [noise=true , address=77.90.31.17:4772 , piece_id=CT6BNOYZGDBUEFVGM2RDS4ICMWPYS6N4IOUDA5V7MJQ3IKELNW2Q, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.668671640 (7h36m5.220142101s) [noise=true , address=145.79.227.31:4797 , piece_id=VQEEGJTLJY2IMCDHL2GMVGYEX4VL34TMMXKDYNERFYK3HRV7TC6A, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.669093554 (7h36m5.220564015s) [noise=true , address=145.79.227.21:4474 , piece_id=J5XE2MTCEW4ZHQDDZ4KCVYFK5EX6RQZARKNY4QCH4HUHGJO3SXDQ, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.669510097 (7h36m5.220980568s) [noise=true , address=77.90.31.18:4807 , piece_id=HDJAJYXNZNZN3LVAQSYVJEOQ3MU5H3BXCCW3NA2GLKPT7ATZHSJQ, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.671936887 (7h36m5.223407348s) [noise=true , address=145.79.227.20:4439 , piece_id=PPOVWSRAU7Y4CZAKAXV6ZP74EEPHONUC35O6XM57VE47NVJL5WZA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.672494549 (7h36m5.22396502s) [noise=true , address=77.90.31.21:4877 , piece_id=3V3Z6GNF32Q66EOS66WV3AGJNA7IXT5NHVAPKNUGVYU3R46IKKKA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:08.438197970 (7h36m6.989668451s) [upload=6714]: Next returned nil segment Nov 9 15:40:08.438219050 (7h36m6.989689521s) [upload=6714]: Waiting for error group managing segments... Nov 9 15:40:08.829503696 (7h36m7.380974147s) [noise=true , address=77.90.31.18:4807 , piece_id=HDJAJYXNZNZN3LVAQSYVJEOQ3MU5H3BXCCW3NA2GLKPT7ATZHSJQ, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:08.846923116 (7h36m7.398393577s) [noise=true , address=45.91.116.14:4190 , piece_id=XSIDZL5HMLZTQE7ZAPURAAUGQLPP3QOZK5BIE5PHRMZPSTN5APSA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:08.867005608 (7h36m7.418476069s) [noise=true , address=77.90.31.21:4877 , piece_id=3V3Z6GNF32Q66EOS66WV3AGJNA7IXT5NHVAPKNUGVYU3R46IKKKA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:08.888352807 (7h36m7.439823268s) [noise=true , address=77.90.31.14:4662 , piece_id=HHSJDZPYKGCHURAZV46OQSGBU7D4TFPXREDBFPGTYF7OAUMT2X3A, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:08.904680649 (7h36m7.45615111s) [noise=true , address=77.90.31.17:4772 , piece_id=CT6BNOYZGDBUEFVGM2RDS4ICMWPYS6N4IOUDA5V7MJQ3IKELNW2Q, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:18.663082162 (7h36m17.214552623s) [noise=true , address=145.79.227.17:4971 , piece_id=VTUI377UCF44ICILORHEKUO33ICRXM2FECO5EXS7RUDH4VGV7FJA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:20.077194714 (7h36m18.628665185s) [noise=true , address=145.79.227.13:4803 , piece_id=V37DZKBC4CHRGISXEQHTQAGKZFNXQKWXSWBSMHNRQTSJ32WYI2OQ, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:28.517900114 (7h36m27.069370575s) [noise=true , address=145.79.227.32:4835 , piece_id=EYIBNSVPQPKLKOBHKZI2VZHVCDKW3C3YJSFEWP4G62UTBXX2YRZA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:28.567571990 (7h36m27.119042451s) [noise=true , address=145.79.227.31:4797 , piece_id=VQEEGJTLJY2IMCDHL2GMVGYEX4VL34TMMXKDYNERFYK3HRV7TC6A, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:33.651382593 (7h36m32.202853044s) [noise=true , address=145.79.227.20:4439 , piece_id=PPOVWSRAU7Y4CZAKAXV6ZP74EEPHONUC35O6XM57VE47NVJL5WZA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:34.842221635 (7h36m33.393692096s) [noise=true , address=145.79.227.9:4365 , piece_id=7BK6E45EY6Y3SV6PY3A4EVCS42PC4HYLXU3J7MN4HXRLDEFUEFCA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:35.601406517 (7h36m34.152877008s) [noise=true , address=145.79.227.25:4619 , piece_id=23D3MCZRVNPI5KB43ROAHKUALDGXLZBZU5E6EIU6U2FY3YQOVPGA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:35.697082303 (7h36m34.248552774s) [noise=true , address=145.79.227.17:4970 , piece_id=5ZP5DDC3XAYQ5UFYJVQV6IZHB5KQQGNG54MTYW3UUYPN2IAK5QVQ, seg_pos={0 0}, upload=6714]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: expected piece hash; context deadline exceeded; EOF Nov 9 15:40:35.697252518 (7h36m34.248722969s) [noise=true , address=145.79.227.17:4972 , piece_id=AI3KVNUJKJUD33YVU2Z472R72SKQKRYQG65JPKFZ5WBJLKRLW6MA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:145.79.227.17:4972): protocol: expected piece hash; context deadline exceeded; EOF Nov 9 15:40:35.697302322 (7h36m34.248772773s) [noise=true , address=145.79.227.21:4474 , piece_id=J5XE2MTCEW4ZHQDDZ4KCVYFK5EX6RQZARKNY4QCH4HUHGJO3SXDQ, seg_pos={0 0}, upload=6714]: Done uploading piece. err: ecclient: upload failed (node:1rL2DnjSgHP36Md3bCgy5padqTAhv3iQCeURbLMuuuMBxtUNZH, address:145.79.227.21:4474): protocol: expected piece hash; context deadline exceeded; EOF Nov 9 15:40:35.697472546 (7h36m34.248943007s) [noise=true , address=145.79.227.9:4366 , piece_id=BT3GWQSXIMMSFPUYJC5EQ2LX2DS4BW5DAMW6ILLNVVRUUHZMOQUA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: expected piece hash; context deadline exceeded; EOF Nov 9 15:40:35.697514686 (7h36m34.248985147s) [seg_pos={0 0}, upload=6714]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 9 15:40:35.950211122 (7h36m34.501681583s) [upload=6714]: Done uploading segments... Nov 9 15:40:35.950416643 (7h36m34.501887104s) [upload=6714]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 11.00831957s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 9.83845542s trace: https://jaeger.dev.storj.io/trace/3846516a5cecd778 FAIL (2025-11-09 16:24:21 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 9 15:40:06.410809146 (7h36m4.962279607s) [upload=6714]: Starting upload Nov 9 15:40:06.411046298 (7h36m4.962516769s) [upload=6714]: Done starting upload Nov 9 15:40:06.411314729 (7h36m4.96278519s) [upload=6714]: Uploading segments... Nov 9 15:40:06.411410882 (7h36m4.962881383s) [upload=6714]: Got next segment. Inline: false Nov 9 15:40:06.411464574 (7h36m4.962935035s) [upload=6714]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 9 15:40:06.411475064 (7h36m4.962945555s) [upload=6714]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 9 15:40:06.664789367 (7h36m5.216259828s) [seg_pos={0 0}, upload=6714]: Begin upload segment... Nov 9 15:40:06.664851705 (7h36m5.216322166s) [seg_pos={0 0}, upload=6714]: Done begin upload segment. Nov 9 15:40:06.664891662 (7h36m5.216362123s) [noise=true , address=145.79.227.32:4835 , piece_id=EYIBNSVPQPKLKOBHKZI2VZHVCDKW3C3YJSFEWP4G62UTBXX2YRZA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.664996933 (7h36m5.216467394s) [noise=true , address=45.91.116.14:4190 , piece_id=XSIDZL5HMLZTQE7ZAPURAAUGQLPP3QOZK5BIE5PHRMZPSTN5APSA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.665698789 (7h36m5.21716925s) [noise=true , address=77.90.31.14:4662 , piece_id=HHSJDZPYKGCHURAZV46OQSGBU7D4TFPXREDBFPGTYF7OAUMT2X3A, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.666456111 (7h36m5.217926572s) [noise=true , address=145.79.227.25:4619 , piece_id=23D3MCZRVNPI5KB43ROAHKUALDGXLZBZU5E6EIU6U2FY3YQOVPGA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.666642225 (7h36m5.218112676s) [noise=true , address=145.79.227.17:4971 , piece_id=VTUI377UCF44ICILORHEKUO33ICRXM2FECO5EXS7RUDH4VGV7FJA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.667027539 (7h36m5.218498s) [noise=true , address=145.79.227.9:4366 , piece_id=BT3GWQSXIMMSFPUYJC5EQ2LX2DS4BW5DAMW6ILLNVVRUUHZMOQUA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.667041675 (7h36m5.218512126s) [noise=true , address=145.79.227.17:4972 , piece_id=AI3KVNUJKJUD33YVU2Z472R72SKQKRYQG65JPKFZ5WBJLKRLW6MA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.667502794 (7h36m5.218973255s) [noise=true , address=145.79.227.13:4803 , piece_id=V37DZKBC4CHRGISXEQHTQAGKZFNXQKWXSWBSMHNRQTSJ32WYI2OQ, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.667805441 (7h36m5.219275902s) [noise=true , address=145.79.227.9:4365 , piece_id=7BK6E45EY6Y3SV6PY3A4EVCS42PC4HYLXU3J7MN4HXRLDEFUEFCA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.667895652 (7h36m5.219366113s) [noise=true , address=145.79.227.17:4970 , piece_id=5ZP5DDC3XAYQ5UFYJVQV6IZHB5KQQGNG54MTYW3UUYPN2IAK5QVQ, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.668240909 (7h36m5.219711361s) [noise=true , address=77.90.31.17:4772 , piece_id=CT6BNOYZGDBUEFVGM2RDS4ICMWPYS6N4IOUDA5V7MJQ3IKELNW2Q, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.668671640 (7h36m5.220142101s) [noise=true , address=145.79.227.31:4797 , piece_id=VQEEGJTLJY2IMCDHL2GMVGYEX4VL34TMMXKDYNERFYK3HRV7TC6A, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.669093554 (7h36m5.220564015s) [noise=true , address=145.79.227.21:4474 , piece_id=J5XE2MTCEW4ZHQDDZ4KCVYFK5EX6RQZARKNY4QCH4HUHGJO3SXDQ, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.669510097 (7h36m5.220980568s) [noise=true , address=77.90.31.18:4807 , piece_id=HDJAJYXNZNZN3LVAQSYVJEOQ3MU5H3BXCCW3NA2GLKPT7ATZHSJQ, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.671936887 (7h36m5.223407348s) [noise=true , address=145.79.227.20:4439 , piece_id=PPOVWSRAU7Y4CZAKAXV6ZP74EEPHONUC35O6XM57VE47NVJL5WZA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.672494549 (7h36m5.22396502s) [noise=true , address=77.90.31.21:4877 , piece_id=3V3Z6GNF32Q66EOS66WV3AGJNA7IXT5NHVAPKNUGVYU3R46IKKKA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:08.438197970 (7h36m6.989668451s) [upload=6714]: Next returned nil segment Nov 9 15:40:08.438219050 (7h36m6.989689521s) [upload=6714]: Waiting for error group managing segments... Nov 9 15:40:08.829503696 (7h36m7.380974147s) [noise=true , address=77.90.31.18:4807 , piece_id=HDJAJYXNZNZN3LVAQSYVJEOQ3MU5H3BXCCW3NA2GLKPT7ATZHSJQ, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:08.846923116 (7h36m7.398393577s) [noise=true , address=45.91.116.14:4190 , piece_id=XSIDZL5HMLZTQE7ZAPURAAUGQLPP3QOZK5BIE5PHRMZPSTN5APSA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:08.867005608 (7h36m7.418476069s) [noise=true , address=77.90.31.21:4877 , piece_id=3V3Z6GNF32Q66EOS66WV3AGJNA7IXT5NHVAPKNUGVYU3R46IKKKA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:08.888352807 (7h36m7.439823268s) [noise=true , address=77.90.31.14:4662 , piece_id=HHSJDZPYKGCHURAZV46OQSGBU7D4TFPXREDBFPGTYF7OAUMT2X3A, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:08.904680649 (7h36m7.45615111s) [noise=true , address=77.90.31.17:4772 , piece_id=CT6BNOYZGDBUEFVGM2RDS4ICMWPYS6N4IOUDA5V7MJQ3IKELNW2Q, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:18.663082162 (7h36m17.214552623s) [noise=true , address=145.79.227.17:4971 , piece_id=VTUI377UCF44ICILORHEKUO33ICRXM2FECO5EXS7RUDH4VGV7FJA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:20.077194714 (7h36m18.628665185s) [noise=true , address=145.79.227.13:4803 , piece_id=V37DZKBC4CHRGISXEQHTQAGKZFNXQKWXSWBSMHNRQTSJ32WYI2OQ, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:28.517900114 (7h36m27.069370575s) [noise=true , address=145.79.227.32:4835 , piece_id=EYIBNSVPQPKLKOBHKZI2VZHVCDKW3C3YJSFEWP4G62UTBXX2YRZA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:28.567571990 (7h36m27.119042451s) [noise=true , address=145.79.227.31:4797 , piece_id=VQEEGJTLJY2IMCDHL2GMVGYEX4VL34TMMXKDYNERFYK3HRV7TC6A, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:33.651382593 (7h36m32.202853044s) [noise=true , address=145.79.227.20:4439 , piece_id=PPOVWSRAU7Y4CZAKAXV6ZP74EEPHONUC35O6XM57VE47NVJL5WZA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:34.842221635 (7h36m33.393692096s) [noise=true , address=145.79.227.9:4365 , piece_id=7BK6E45EY6Y3SV6PY3A4EVCS42PC4HYLXU3J7MN4HXRLDEFUEFCA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:35.601406517 (7h36m34.152877008s) [noise=true , address=145.79.227.25:4619 , piece_id=23D3MCZRVNPI5KB43ROAHKUALDGXLZBZU5E6EIU6U2FY3YQOVPGA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:35.697082303 (7h36m34.248552774s) [noise=true , address=145.79.227.17:4970 , piece_id=5ZP5DDC3XAYQ5UFYJVQV6IZHB5KQQGNG54MTYW3UUYPN2IAK5QVQ, seg_pos={0 0}, upload=6714]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: expected piece hash; context deadline exceeded; EOF Nov 9 15:40:35.697252518 (7h36m34.248722969s) [noise=true , address=145.79.227.17:4972 , piece_id=AI3KVNUJKJUD33YVU2Z472R72SKQKRYQG65JPKFZ5WBJLKRLW6MA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:145.79.227.17:4972): protocol: expected piece hash; context deadline exceeded; EOF Nov 9 15:40:35.697302322 (7h36m34.248772773s) [noise=true , address=145.79.227.21:4474 , piece_id=J5XE2MTCEW4ZHQDDZ4KCVYFK5EX6RQZARKNY4QCH4HUHGJO3SXDQ, seg_pos={0 0}, upload=6714]: Done uploading piece. err: ecclient: upload failed (node:1rL2DnjSgHP36Md3bCgy5padqTAhv3iQCeURbLMuuuMBxtUNZH, address:145.79.227.21:4474): protocol: expected piece hash; context deadline exceeded; EOF Nov 9 15:40:35.697472546 (7h36m34.248943007s) [noise=true , address=145.79.227.9:4366 , piece_id=BT3GWQSXIMMSFPUYJC5EQ2LX2DS4BW5DAMW6ILLNVVRUUHZMOQUA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: expected piece hash; context deadline exceeded; EOF Nov 9 15:40:35.697514686 (7h36m34.248985147s) [seg_pos={0 0}, upload=6714]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 9 15:40:35.950211122 (7h36m34.501681583s) [upload=6714]: Done uploading segments... Nov 9 15:40:35.950416643 (7h36m34.501887104s) [upload=6714]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 9.838485969s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 10.505202984s trace: https://jaeger.dev.storj.io/trace/3846516a5cecd778 FAIL (2025-11-09 16:23:40 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 9 15:40:06.410809146 (7h36m4.962279607s) [upload=6714]: Starting upload Nov 9 15:40:06.411046298 (7h36m4.962516769s) [upload=6714]: Done starting upload Nov 9 15:40:06.411314729 (7h36m4.96278519s) [upload=6714]: Uploading segments... Nov 9 15:40:06.411410882 (7h36m4.962881383s) [upload=6714]: Got next segment. Inline: false Nov 9 15:40:06.411464574 (7h36m4.962935035s) [upload=6714]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 9 15:40:06.411475064 (7h36m4.962945555s) [upload=6714]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 9 15:40:06.664789367 (7h36m5.216259828s) [seg_pos={0 0}, upload=6714]: Begin upload segment... Nov 9 15:40:06.664851705 (7h36m5.216322166s) [seg_pos={0 0}, upload=6714]: Done begin upload segment. Nov 9 15:40:06.664891662 (7h36m5.216362123s) [noise=true , address=145.79.227.32:4835 , piece_id=EYIBNSVPQPKLKOBHKZI2VZHVCDKW3C3YJSFEWP4G62UTBXX2YRZA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.664996933 (7h36m5.216467394s) [noise=true , address=45.91.116.14:4190 , piece_id=XSIDZL5HMLZTQE7ZAPURAAUGQLPP3QOZK5BIE5PHRMZPSTN5APSA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.665698789 (7h36m5.21716925s) [noise=true , address=77.90.31.14:4662 , piece_id=HHSJDZPYKGCHURAZV46OQSGBU7D4TFPXREDBFPGTYF7OAUMT2X3A, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.666456111 (7h36m5.217926572s) [noise=true , address=145.79.227.25:4619 , piece_id=23D3MCZRVNPI5KB43ROAHKUALDGXLZBZU5E6EIU6U2FY3YQOVPGA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.666642225 (7h36m5.218112676s) [noise=true , address=145.79.227.17:4971 , piece_id=VTUI377UCF44ICILORHEKUO33ICRXM2FECO5EXS7RUDH4VGV7FJA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.667027539 (7h36m5.218498s) [noise=true , address=145.79.227.9:4366 , piece_id=BT3GWQSXIMMSFPUYJC5EQ2LX2DS4BW5DAMW6ILLNVVRUUHZMOQUA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.667041675 (7h36m5.218512126s) [noise=true , address=145.79.227.17:4972 , piece_id=AI3KVNUJKJUD33YVU2Z472R72SKQKRYQG65JPKFZ5WBJLKRLW6MA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.667502794 (7h36m5.218973255s) [noise=true , address=145.79.227.13:4803 , piece_id=V37DZKBC4CHRGISXEQHTQAGKZFNXQKWXSWBSMHNRQTSJ32WYI2OQ, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.667805441 (7h36m5.219275902s) [noise=true , address=145.79.227.9:4365 , piece_id=7BK6E45EY6Y3SV6PY3A4EVCS42PC4HYLXU3J7MN4HXRLDEFUEFCA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.667895652 (7h36m5.219366113s) [noise=true , address=145.79.227.17:4970 , piece_id=5ZP5DDC3XAYQ5UFYJVQV6IZHB5KQQGNG54MTYW3UUYPN2IAK5QVQ, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.668240909 (7h36m5.219711361s) [noise=true , address=77.90.31.17:4772 , piece_id=CT6BNOYZGDBUEFVGM2RDS4ICMWPYS6N4IOUDA5V7MJQ3IKELNW2Q, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.668671640 (7h36m5.220142101s) [noise=true , address=145.79.227.31:4797 , piece_id=VQEEGJTLJY2IMCDHL2GMVGYEX4VL34TMMXKDYNERFYK3HRV7TC6A, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.669093554 (7h36m5.220564015s) [noise=true , address=145.79.227.21:4474 , piece_id=J5XE2MTCEW4ZHQDDZ4KCVYFK5EX6RQZARKNY4QCH4HUHGJO3SXDQ, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.669510097 (7h36m5.220980568s) [noise=true , address=77.90.31.18:4807 , piece_id=HDJAJYXNZNZN3LVAQSYVJEOQ3MU5H3BXCCW3NA2GLKPT7ATZHSJQ, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.671936887 (7h36m5.223407348s) [noise=true , address=145.79.227.20:4439 , piece_id=PPOVWSRAU7Y4CZAKAXV6ZP74EEPHONUC35O6XM57VE47NVJL5WZA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.672494549 (7h36m5.22396502s) [noise=true , address=77.90.31.21:4877 , piece_id=3V3Z6GNF32Q66EOS66WV3AGJNA7IXT5NHVAPKNUGVYU3R46IKKKA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:08.438197970 (7h36m6.989668451s) [upload=6714]: Next returned nil segment Nov 9 15:40:08.438219050 (7h36m6.989689521s) [upload=6714]: Waiting for error group managing segments... Nov 9 15:40:08.829503696 (7h36m7.380974147s) [noise=true , address=77.90.31.18:4807 , piece_id=HDJAJYXNZNZN3LVAQSYVJEOQ3MU5H3BXCCW3NA2GLKPT7ATZHSJQ, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:08.846923116 (7h36m7.398393577s) [noise=true , address=45.91.116.14:4190 , piece_id=XSIDZL5HMLZTQE7ZAPURAAUGQLPP3QOZK5BIE5PHRMZPSTN5APSA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:08.867005608 (7h36m7.418476069s) [noise=true , address=77.90.31.21:4877 , piece_id=3V3Z6GNF32Q66EOS66WV3AGJNA7IXT5NHVAPKNUGVYU3R46IKKKA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:08.888352807 (7h36m7.439823268s) [noise=true , address=77.90.31.14:4662 , piece_id=HHSJDZPYKGCHURAZV46OQSGBU7D4TFPXREDBFPGTYF7OAUMT2X3A, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:08.904680649 (7h36m7.45615111s) [noise=true , address=77.90.31.17:4772 , piece_id=CT6BNOYZGDBUEFVGM2RDS4ICMWPYS6N4IOUDA5V7MJQ3IKELNW2Q, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:18.663082162 (7h36m17.214552623s) [noise=true , address=145.79.227.17:4971 , piece_id=VTUI377UCF44ICILORHEKUO33ICRXM2FECO5EXS7RUDH4VGV7FJA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:20.077194714 (7h36m18.628665185s) [noise=true , address=145.79.227.13:4803 , piece_id=V37DZKBC4CHRGISXEQHTQAGKZFNXQKWXSWBSMHNRQTSJ32WYI2OQ, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:28.517900114 (7h36m27.069370575s) [noise=true , address=145.79.227.32:4835 , piece_id=EYIBNSVPQPKLKOBHKZI2VZHVCDKW3C3YJSFEWP4G62UTBXX2YRZA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:28.567571990 (7h36m27.119042451s) [noise=true , address=145.79.227.31:4797 , piece_id=VQEEGJTLJY2IMCDHL2GMVGYEX4VL34TMMXKDYNERFYK3HRV7TC6A, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:33.651382593 (7h36m32.202853044s) [noise=true , address=145.79.227.20:4439 , piece_id=PPOVWSRAU7Y4CZAKAXV6ZP74EEPHONUC35O6XM57VE47NVJL5WZA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:34.842221635 (7h36m33.393692096s) [noise=true , address=145.79.227.9:4365 , piece_id=7BK6E45EY6Y3SV6PY3A4EVCS42PC4HYLXU3J7MN4HXRLDEFUEFCA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:35.601406517 (7h36m34.152877008s) [noise=true , address=145.79.227.25:4619 , piece_id=23D3MCZRVNPI5KB43ROAHKUALDGXLZBZU5E6EIU6U2FY3YQOVPGA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:35.697082303 (7h36m34.248552774s) [noise=true , address=145.79.227.17:4970 , piece_id=5ZP5DDC3XAYQ5UFYJVQV6IZHB5KQQGNG54MTYW3UUYPN2IAK5QVQ, seg_pos={0 0}, upload=6714]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: expected piece hash; context deadline exceeded; EOF Nov 9 15:40:35.697252518 (7h36m34.248722969s) [noise=true , address=145.79.227.17:4972 , piece_id=AI3KVNUJKJUD33YVU2Z472R72SKQKRYQG65JPKFZ5WBJLKRLW6MA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:145.79.227.17:4972): protocol: expected piece hash; context deadline exceeded; EOF Nov 9 15:40:35.697302322 (7h36m34.248772773s) [noise=true , address=145.79.227.21:4474 , piece_id=J5XE2MTCEW4ZHQDDZ4KCVYFK5EX6RQZARKNY4QCH4HUHGJO3SXDQ, seg_pos={0 0}, upload=6714]: Done uploading piece. err: ecclient: upload failed (node:1rL2DnjSgHP36Md3bCgy5padqTAhv3iQCeURbLMuuuMBxtUNZH, address:145.79.227.21:4474): protocol: expected piece hash; context deadline exceeded; EOF Nov 9 15:40:35.697472546 (7h36m34.248943007s) [noise=true , address=145.79.227.9:4366 , piece_id=BT3GWQSXIMMSFPUYJC5EQ2LX2DS4BW5DAMW6ILLNVVRUUHZMOQUA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: expected piece hash; context deadline exceeded; EOF Nov 9 15:40:35.697514686 (7h36m34.248985147s) [seg_pos={0 0}, upload=6714]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 9 15:40:35.950211122 (7h36m34.501681583s) [upload=6714]: Done uploading segments... Nov 9 15:40:35.950416643 (7h36m34.501887104s) [upload=6714]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 10.505326438s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 9.078779223s trace: https://jaeger.dev.storj.io/trace/3846516a5cecd778 FAIL (2025-11-09 16:22:52 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 9 15:40:06.410809146 (7h36m4.962279607s) [upload=6714]: Starting upload Nov 9 15:40:06.411046298 (7h36m4.962516769s) [upload=6714]: Done starting upload Nov 9 15:40:06.411314729 (7h36m4.96278519s) [upload=6714]: Uploading segments... Nov 9 15:40:06.411410882 (7h36m4.962881383s) [upload=6714]: Got next segment. Inline: false Nov 9 15:40:06.411464574 (7h36m4.962935035s) [upload=6714]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 9 15:40:06.411475064 (7h36m4.962945555s) [upload=6714]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 9 15:40:06.664789367 (7h36m5.216259828s) [seg_pos={0 0}, upload=6714]: Begin upload segment... Nov 9 15:40:06.664851705 (7h36m5.216322166s) [seg_pos={0 0}, upload=6714]: Done begin upload segment. Nov 9 15:40:06.664891662 (7h36m5.216362123s) [noise=true , address=145.79.227.32:4835 , piece_id=EYIBNSVPQPKLKOBHKZI2VZHVCDKW3C3YJSFEWP4G62UTBXX2YRZA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.664996933 (7h36m5.216467394s) [noise=true , address=45.91.116.14:4190 , piece_id=XSIDZL5HMLZTQE7ZAPURAAUGQLPP3QOZK5BIE5PHRMZPSTN5APSA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.665698789 (7h36m5.21716925s) [noise=true , address=77.90.31.14:4662 , piece_id=HHSJDZPYKGCHURAZV46OQSGBU7D4TFPXREDBFPGTYF7OAUMT2X3A, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.666456111 (7h36m5.217926572s) [noise=true , address=145.79.227.25:4619 , piece_id=23D3MCZRVNPI5KB43ROAHKUALDGXLZBZU5E6EIU6U2FY3YQOVPGA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.666642225 (7h36m5.218112676s) [noise=true , address=145.79.227.17:4971 , piece_id=VTUI377UCF44ICILORHEKUO33ICRXM2FECO5EXS7RUDH4VGV7FJA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.667027539 (7h36m5.218498s) [noise=true , address=145.79.227.9:4366 , piece_id=BT3GWQSXIMMSFPUYJC5EQ2LX2DS4BW5DAMW6ILLNVVRUUHZMOQUA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.667041675 (7h36m5.218512126s) [noise=true , address=145.79.227.17:4972 , piece_id=AI3KVNUJKJUD33YVU2Z472R72SKQKRYQG65JPKFZ5WBJLKRLW6MA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.667502794 (7h36m5.218973255s) [noise=true , address=145.79.227.13:4803 , piece_id=V37DZKBC4CHRGISXEQHTQAGKZFNXQKWXSWBSMHNRQTSJ32WYI2OQ, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.667805441 (7h36m5.219275902s) [noise=true , address=145.79.227.9:4365 , piece_id=7BK6E45EY6Y3SV6PY3A4EVCS42PC4HYLXU3J7MN4HXRLDEFUEFCA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.667895652 (7h36m5.219366113s) [noise=true , address=145.79.227.17:4970 , piece_id=5ZP5DDC3XAYQ5UFYJVQV6IZHB5KQQGNG54MTYW3UUYPN2IAK5QVQ, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.668240909 (7h36m5.219711361s) [noise=true , address=77.90.31.17:4772 , piece_id=CT6BNOYZGDBUEFVGM2RDS4ICMWPYS6N4IOUDA5V7MJQ3IKELNW2Q, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.668671640 (7h36m5.220142101s) [noise=true , address=145.79.227.31:4797 , piece_id=VQEEGJTLJY2IMCDHL2GMVGYEX4VL34TMMXKDYNERFYK3HRV7TC6A, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.669093554 (7h36m5.220564015s) [noise=true , address=145.79.227.21:4474 , piece_id=J5XE2MTCEW4ZHQDDZ4KCVYFK5EX6RQZARKNY4QCH4HUHGJO3SXDQ, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.669510097 (7h36m5.220980568s) [noise=true , address=77.90.31.18:4807 , piece_id=HDJAJYXNZNZN3LVAQSYVJEOQ3MU5H3BXCCW3NA2GLKPT7ATZHSJQ, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.671936887 (7h36m5.223407348s) [noise=true , address=145.79.227.20:4439 , piece_id=PPOVWSRAU7Y4CZAKAXV6ZP74EEPHONUC35O6XM57VE47NVJL5WZA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.672494549 (7h36m5.22396502s) [noise=true , address=77.90.31.21:4877 , piece_id=3V3Z6GNF32Q66EOS66WV3AGJNA7IXT5NHVAPKNUGVYU3R46IKKKA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:08.438197970 (7h36m6.989668451s) [upload=6714]: Next returned nil segment Nov 9 15:40:08.438219050 (7h36m6.989689521s) [upload=6714]: Waiting for error group managing segments... Nov 9 15:40:08.829503696 (7h36m7.380974147s) [noise=true , address=77.90.31.18:4807 , piece_id=HDJAJYXNZNZN3LVAQSYVJEOQ3MU5H3BXCCW3NA2GLKPT7ATZHSJQ, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:08.846923116 (7h36m7.398393577s) [noise=true , address=45.91.116.14:4190 , piece_id=XSIDZL5HMLZTQE7ZAPURAAUGQLPP3QOZK5BIE5PHRMZPSTN5APSA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:08.867005608 (7h36m7.418476069s) [noise=true , address=77.90.31.21:4877 , piece_id=3V3Z6GNF32Q66EOS66WV3AGJNA7IXT5NHVAPKNUGVYU3R46IKKKA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:08.888352807 (7h36m7.439823268s) [noise=true , address=77.90.31.14:4662 , piece_id=HHSJDZPYKGCHURAZV46OQSGBU7D4TFPXREDBFPGTYF7OAUMT2X3A, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:08.904680649 (7h36m7.45615111s) [noise=true , address=77.90.31.17:4772 , piece_id=CT6BNOYZGDBUEFVGM2RDS4ICMWPYS6N4IOUDA5V7MJQ3IKELNW2Q, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:18.663082162 (7h36m17.214552623s) [noise=true , address=145.79.227.17:4971 , piece_id=VTUI377UCF44ICILORHEKUO33ICRXM2FECO5EXS7RUDH4VGV7FJA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:20.077194714 (7h36m18.628665185s) [noise=true , address=145.79.227.13:4803 , piece_id=V37DZKBC4CHRGISXEQHTQAGKZFNXQKWXSWBSMHNRQTSJ32WYI2OQ, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:28.517900114 (7h36m27.069370575s) [noise=true , address=145.79.227.32:4835 , piece_id=EYIBNSVPQPKLKOBHKZI2VZHVCDKW3C3YJSFEWP4G62UTBXX2YRZA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:28.567571990 (7h36m27.119042451s) [noise=true , address=145.79.227.31:4797 , piece_id=VQEEGJTLJY2IMCDHL2GMVGYEX4VL34TMMXKDYNERFYK3HRV7TC6A, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:33.651382593 (7h36m32.202853044s) [noise=true , address=145.79.227.20:4439 , piece_id=PPOVWSRAU7Y4CZAKAXV6ZP74EEPHONUC35O6XM57VE47NVJL5WZA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:34.842221635 (7h36m33.393692096s) [noise=true , address=145.79.227.9:4365 , piece_id=7BK6E45EY6Y3SV6PY3A4EVCS42PC4HYLXU3J7MN4HXRLDEFUEFCA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:35.601406517 (7h36m34.152877008s) [noise=true , address=145.79.227.25:4619 , piece_id=23D3MCZRVNPI5KB43ROAHKUALDGXLZBZU5E6EIU6U2FY3YQOVPGA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:35.697082303 (7h36m34.248552774s) [noise=true , address=145.79.227.17:4970 , piece_id=5ZP5DDC3XAYQ5UFYJVQV6IZHB5KQQGNG54MTYW3UUYPN2IAK5QVQ, seg_pos={0 0}, upload=6714]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: expected piece hash; context deadline exceeded; EOF Nov 9 15:40:35.697252518 (7h36m34.248722969s) [noise=true , address=145.79.227.17:4972 , piece_id=AI3KVNUJKJUD33YVU2Z472R72SKQKRYQG65JPKFZ5WBJLKRLW6MA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:145.79.227.17:4972): protocol: expected piece hash; context deadline exceeded; EOF Nov 9 15:40:35.697302322 (7h36m34.248772773s) [noise=true , address=145.79.227.21:4474 , piece_id=J5XE2MTCEW4ZHQDDZ4KCVYFK5EX6RQZARKNY4QCH4HUHGJO3SXDQ, seg_pos={0 0}, upload=6714]: Done uploading piece. err: ecclient: upload failed (node:1rL2DnjSgHP36Md3bCgy5padqTAhv3iQCeURbLMuuuMBxtUNZH, address:145.79.227.21:4474): protocol: expected piece hash; context deadline exceeded; EOF Nov 9 15:40:35.697472546 (7h36m34.248943007s) [noise=true , address=145.79.227.9:4366 , piece_id=BT3GWQSXIMMSFPUYJC5EQ2LX2DS4BW5DAMW6ILLNVVRUUHZMOQUA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: expected piece hash; context deadline exceeded; EOF Nov 9 15:40:35.697514686 (7h36m34.248985147s) [seg_pos={0 0}, upload=6714]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 9 15:40:35.950211122 (7h36m34.501681583s) [upload=6714]: Done uploading segments... Nov 9 15:40:35.950416643 (7h36m34.501887104s) [upload=6714]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 9.07881451s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 10.656548401s trace: https://jaeger.dev.storj.io/trace/3846516a5cecd778 FAIL (2025-11-09 16:22:30 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 9 15:40:06.410809146 (7h36m4.962279607s) [upload=6714]: Starting upload Nov 9 15:40:06.411046298 (7h36m4.962516769s) [upload=6714]: Done starting upload Nov 9 15:40:06.411314729 (7h36m4.96278519s) [upload=6714]: Uploading segments... Nov 9 15:40:06.411410882 (7h36m4.962881383s) [upload=6714]: Got next segment. Inline: false Nov 9 15:40:06.411464574 (7h36m4.962935035s) [upload=6714]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 9 15:40:06.411475064 (7h36m4.962945555s) [upload=6714]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 9 15:40:06.664789367 (7h36m5.216259828s) [seg_pos={0 0}, upload=6714]: Begin upload segment... Nov 9 15:40:06.664851705 (7h36m5.216322166s) [seg_pos={0 0}, upload=6714]: Done begin upload segment. Nov 9 15:40:06.664891662 (7h36m5.216362123s) [noise=true , address=145.79.227.32:4835 , piece_id=EYIBNSVPQPKLKOBHKZI2VZHVCDKW3C3YJSFEWP4G62UTBXX2YRZA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.664996933 (7h36m5.216467394s) [noise=true , address=45.91.116.14:4190 , piece_id=XSIDZL5HMLZTQE7ZAPURAAUGQLPP3QOZK5BIE5PHRMZPSTN5APSA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.665698789 (7h36m5.21716925s) [noise=true , address=77.90.31.14:4662 , piece_id=HHSJDZPYKGCHURAZV46OQSGBU7D4TFPXREDBFPGTYF7OAUMT2X3A, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.666456111 (7h36m5.217926572s) [noise=true , address=145.79.227.25:4619 , piece_id=23D3MCZRVNPI5KB43ROAHKUALDGXLZBZU5E6EIU6U2FY3YQOVPGA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.666642225 (7h36m5.218112676s) [noise=true , address=145.79.227.17:4971 , piece_id=VTUI377UCF44ICILORHEKUO33ICRXM2FECO5EXS7RUDH4VGV7FJA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.667027539 (7h36m5.218498s) [noise=true , address=145.79.227.9:4366 , piece_id=BT3GWQSXIMMSFPUYJC5EQ2LX2DS4BW5DAMW6ILLNVVRUUHZMOQUA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.667041675 (7h36m5.218512126s) [noise=true , address=145.79.227.17:4972 , piece_id=AI3KVNUJKJUD33YVU2Z472R72SKQKRYQG65JPKFZ5WBJLKRLW6MA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.667502794 (7h36m5.218973255s) [noise=true , address=145.79.227.13:4803 , piece_id=V37DZKBC4CHRGISXEQHTQAGKZFNXQKWXSWBSMHNRQTSJ32WYI2OQ, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.667805441 (7h36m5.219275902s) [noise=true , address=145.79.227.9:4365 , piece_id=7BK6E45EY6Y3SV6PY3A4EVCS42PC4HYLXU3J7MN4HXRLDEFUEFCA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.667895652 (7h36m5.219366113s) [noise=true , address=145.79.227.17:4970 , piece_id=5ZP5DDC3XAYQ5UFYJVQV6IZHB5KQQGNG54MTYW3UUYPN2IAK5QVQ, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.668240909 (7h36m5.219711361s) [noise=true , address=77.90.31.17:4772 , piece_id=CT6BNOYZGDBUEFVGM2RDS4ICMWPYS6N4IOUDA5V7MJQ3IKELNW2Q, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.668671640 (7h36m5.220142101s) [noise=true , address=145.79.227.31:4797 , piece_id=VQEEGJTLJY2IMCDHL2GMVGYEX4VL34TMMXKDYNERFYK3HRV7TC6A, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.669093554 (7h36m5.220564015s) [noise=true , address=145.79.227.21:4474 , piece_id=J5XE2MTCEW4ZHQDDZ4KCVYFK5EX6RQZARKNY4QCH4HUHGJO3SXDQ, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.669510097 (7h36m5.220980568s) [noise=true , address=77.90.31.18:4807 , piece_id=HDJAJYXNZNZN3LVAQSYVJEOQ3MU5H3BXCCW3NA2GLKPT7ATZHSJQ, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.671936887 (7h36m5.223407348s) [noise=true , address=145.79.227.20:4439 , piece_id=PPOVWSRAU7Y4CZAKAXV6ZP74EEPHONUC35O6XM57VE47NVJL5WZA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.672494549 (7h36m5.22396502s) [noise=true , address=77.90.31.21:4877 , piece_id=3V3Z6GNF32Q66EOS66WV3AGJNA7IXT5NHVAPKNUGVYU3R46IKKKA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:08.438197970 (7h36m6.989668451s) [upload=6714]: Next returned nil segment Nov 9 15:40:08.438219050 (7h36m6.989689521s) [upload=6714]: Waiting for error group managing segments... Nov 9 15:40:08.829503696 (7h36m7.380974147s) [noise=true , address=77.90.31.18:4807 , piece_id=HDJAJYXNZNZN3LVAQSYVJEOQ3MU5H3BXCCW3NA2GLKPT7ATZHSJQ, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:08.846923116 (7h36m7.398393577s) [noise=true , address=45.91.116.14:4190 , piece_id=XSIDZL5HMLZTQE7ZAPURAAUGQLPP3QOZK5BIE5PHRMZPSTN5APSA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:08.867005608 (7h36m7.418476069s) [noise=true , address=77.90.31.21:4877 , piece_id=3V3Z6GNF32Q66EOS66WV3AGJNA7IXT5NHVAPKNUGVYU3R46IKKKA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:08.888352807 (7h36m7.439823268s) [noise=true , address=77.90.31.14:4662 , piece_id=HHSJDZPYKGCHURAZV46OQSGBU7D4TFPXREDBFPGTYF7OAUMT2X3A, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:08.904680649 (7h36m7.45615111s) [noise=true , address=77.90.31.17:4772 , piece_id=CT6BNOYZGDBUEFVGM2RDS4ICMWPYS6N4IOUDA5V7MJQ3IKELNW2Q, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:18.663082162 (7h36m17.214552623s) [noise=true , address=145.79.227.17:4971 , piece_id=VTUI377UCF44ICILORHEKUO33ICRXM2FECO5EXS7RUDH4VGV7FJA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:20.077194714 (7h36m18.628665185s) [noise=true , address=145.79.227.13:4803 , piece_id=V37DZKBC4CHRGISXEQHTQAGKZFNXQKWXSWBSMHNRQTSJ32WYI2OQ, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:28.517900114 (7h36m27.069370575s) [noise=true , address=145.79.227.32:4835 , piece_id=EYIBNSVPQPKLKOBHKZI2VZHVCDKW3C3YJSFEWP4G62UTBXX2YRZA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:28.567571990 (7h36m27.119042451s) [noise=true , address=145.79.227.31:4797 , piece_id=VQEEGJTLJY2IMCDHL2GMVGYEX4VL34TMMXKDYNERFYK3HRV7TC6A, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:33.651382593 (7h36m32.202853044s) [noise=true , address=145.79.227.20:4439 , piece_id=PPOVWSRAU7Y4CZAKAXV6ZP74EEPHONUC35O6XM57VE47NVJL5WZA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:34.842221635 (7h36m33.393692096s) [noise=true , address=145.79.227.9:4365 , piece_id=7BK6E45EY6Y3SV6PY3A4EVCS42PC4HYLXU3J7MN4HXRLDEFUEFCA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:35.601406517 (7h36m34.152877008s) [noise=true , address=145.79.227.25:4619 , piece_id=23D3MCZRVNPI5KB43ROAHKUALDGXLZBZU5E6EIU6U2FY3YQOVPGA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:35.697082303 (7h36m34.248552774s) [noise=true , address=145.79.227.17:4970 , piece_id=5ZP5DDC3XAYQ5UFYJVQV6IZHB5KQQGNG54MTYW3UUYPN2IAK5QVQ, seg_pos={0 0}, upload=6714]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: expected piece hash; context deadline exceeded; EOF Nov 9 15:40:35.697252518 (7h36m34.248722969s) [noise=true , address=145.79.227.17:4972 , piece_id=AI3KVNUJKJUD33YVU2Z472R72SKQKRYQG65JPKFZ5WBJLKRLW6MA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:145.79.227.17:4972): protocol: expected piece hash; context deadline exceeded; EOF Nov 9 15:40:35.697302322 (7h36m34.248772773s) [noise=true , address=145.79.227.21:4474 , piece_id=J5XE2MTCEW4ZHQDDZ4KCVYFK5EX6RQZARKNY4QCH4HUHGJO3SXDQ, seg_pos={0 0}, upload=6714]: Done uploading piece. err: ecclient: upload failed (node:1rL2DnjSgHP36Md3bCgy5padqTAhv3iQCeURbLMuuuMBxtUNZH, address:145.79.227.21:4474): protocol: expected piece hash; context deadline exceeded; EOF Nov 9 15:40:35.697472546 (7h36m34.248943007s) [noise=true , address=145.79.227.9:4366 , piece_id=BT3GWQSXIMMSFPUYJC5EQ2LX2DS4BW5DAMW6ILLNVVRUUHZMOQUA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: expected piece hash; context deadline exceeded; EOF Nov 9 15:40:35.697514686 (7h36m34.248985147s) [seg_pos={0 0}, upload=6714]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 9 15:40:35.950211122 (7h36m34.501681583s) [upload=6714]: Done uploading segments... Nov 9 15:40:35.950416643 (7h36m34.501887104s) [upload=6714]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 10.656592014s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 11.004915591s trace: https://jaeger.dev.storj.io/trace/3846516a5cecd778 FAIL (2025-11-09 16:21:55 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 9 15:40:06.410809146 (7h36m4.962279607s) [upload=6714]: Starting upload Nov 9 15:40:06.411046298 (7h36m4.962516769s) [upload=6714]: Done starting upload Nov 9 15:40:06.411314729 (7h36m4.96278519s) [upload=6714]: Uploading segments... Nov 9 15:40:06.411410882 (7h36m4.962881383s) [upload=6714]: Got next segment. Inline: false Nov 9 15:40:06.411464574 (7h36m4.962935035s) [upload=6714]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 9 15:40:06.411475064 (7h36m4.962945555s) [upload=6714]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 9 15:40:06.664789367 (7h36m5.216259828s) [seg_pos={0 0}, upload=6714]: Begin upload segment... Nov 9 15:40:06.664851705 (7h36m5.216322166s) [seg_pos={0 0}, upload=6714]: Done begin upload segment. Nov 9 15:40:06.664891662 (7h36m5.216362123s) [noise=true , address=145.79.227.32:4835 , piece_id=EYIBNSVPQPKLKOBHKZI2VZHVCDKW3C3YJSFEWP4G62UTBXX2YRZA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.664996933 (7h36m5.216467394s) [noise=true , address=45.91.116.14:4190 , piece_id=XSIDZL5HMLZTQE7ZAPURAAUGQLPP3QOZK5BIE5PHRMZPSTN5APSA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.665698789 (7h36m5.21716925s) [noise=true , address=77.90.31.14:4662 , piece_id=HHSJDZPYKGCHURAZV46OQSGBU7D4TFPXREDBFPGTYF7OAUMT2X3A, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.666456111 (7h36m5.217926572s) [noise=true , address=145.79.227.25:4619 , piece_id=23D3MCZRVNPI5KB43ROAHKUALDGXLZBZU5E6EIU6U2FY3YQOVPGA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.666642225 (7h36m5.218112676s) [noise=true , address=145.79.227.17:4971 , piece_id=VTUI377UCF44ICILORHEKUO33ICRXM2FECO5EXS7RUDH4VGV7FJA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.667027539 (7h36m5.218498s) [noise=true , address=145.79.227.9:4366 , piece_id=BT3GWQSXIMMSFPUYJC5EQ2LX2DS4BW5DAMW6ILLNVVRUUHZMOQUA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.667041675 (7h36m5.218512126s) [noise=true , address=145.79.227.17:4972 , piece_id=AI3KVNUJKJUD33YVU2Z472R72SKQKRYQG65JPKFZ5WBJLKRLW6MA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.667502794 (7h36m5.218973255s) [noise=true , address=145.79.227.13:4803 , piece_id=V37DZKBC4CHRGISXEQHTQAGKZFNXQKWXSWBSMHNRQTSJ32WYI2OQ, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.667805441 (7h36m5.219275902s) [noise=true , address=145.79.227.9:4365 , piece_id=7BK6E45EY6Y3SV6PY3A4EVCS42PC4HYLXU3J7MN4HXRLDEFUEFCA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.667895652 (7h36m5.219366113s) [noise=true , address=145.79.227.17:4970 , piece_id=5ZP5DDC3XAYQ5UFYJVQV6IZHB5KQQGNG54MTYW3UUYPN2IAK5QVQ, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.668240909 (7h36m5.219711361s) [noise=true , address=77.90.31.17:4772 , piece_id=CT6BNOYZGDBUEFVGM2RDS4ICMWPYS6N4IOUDA5V7MJQ3IKELNW2Q, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.668671640 (7h36m5.220142101s) [noise=true , address=145.79.227.31:4797 , piece_id=VQEEGJTLJY2IMCDHL2GMVGYEX4VL34TMMXKDYNERFYK3HRV7TC6A, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.669093554 (7h36m5.220564015s) [noise=true , address=145.79.227.21:4474 , piece_id=J5XE2MTCEW4ZHQDDZ4KCVYFK5EX6RQZARKNY4QCH4HUHGJO3SXDQ, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.669510097 (7h36m5.220980568s) [noise=true , address=77.90.31.18:4807 , piece_id=HDJAJYXNZNZN3LVAQSYVJEOQ3MU5H3BXCCW3NA2GLKPT7ATZHSJQ, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.671936887 (7h36m5.223407348s) [noise=true , address=145.79.227.20:4439 , piece_id=PPOVWSRAU7Y4CZAKAXV6ZP74EEPHONUC35O6XM57VE47NVJL5WZA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.672494549 (7h36m5.22396502s) [noise=true , address=77.90.31.21:4877 , piece_id=3V3Z6GNF32Q66EOS66WV3AGJNA7IXT5NHVAPKNUGVYU3R46IKKKA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:08.438197970 (7h36m6.989668451s) [upload=6714]: Next returned nil segment Nov 9 15:40:08.438219050 (7h36m6.989689521s) [upload=6714]: Waiting for error group managing segments... Nov 9 15:40:08.829503696 (7h36m7.380974147s) [noise=true , address=77.90.31.18:4807 , piece_id=HDJAJYXNZNZN3LVAQSYVJEOQ3MU5H3BXCCW3NA2GLKPT7ATZHSJQ, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:08.846923116 (7h36m7.398393577s) [noise=true , address=45.91.116.14:4190 , piece_id=XSIDZL5HMLZTQE7ZAPURAAUGQLPP3QOZK5BIE5PHRMZPSTN5APSA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:08.867005608 (7h36m7.418476069s) [noise=true , address=77.90.31.21:4877 , piece_id=3V3Z6GNF32Q66EOS66WV3AGJNA7IXT5NHVAPKNUGVYU3R46IKKKA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:08.888352807 (7h36m7.439823268s) [noise=true , address=77.90.31.14:4662 , piece_id=HHSJDZPYKGCHURAZV46OQSGBU7D4TFPXREDBFPGTYF7OAUMT2X3A, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:08.904680649 (7h36m7.45615111s) [noise=true , address=77.90.31.17:4772 , piece_id=CT6BNOYZGDBUEFVGM2RDS4ICMWPYS6N4IOUDA5V7MJQ3IKELNW2Q, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:18.663082162 (7h36m17.214552623s) [noise=true , address=145.79.227.17:4971 , piece_id=VTUI377UCF44ICILORHEKUO33ICRXM2FECO5EXS7RUDH4VGV7FJA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:20.077194714 (7h36m18.628665185s) [noise=true , address=145.79.227.13:4803 , piece_id=V37DZKBC4CHRGISXEQHTQAGKZFNXQKWXSWBSMHNRQTSJ32WYI2OQ, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:28.517900114 (7h36m27.069370575s) [noise=true , address=145.79.227.32:4835 , piece_id=EYIBNSVPQPKLKOBHKZI2VZHVCDKW3C3YJSFEWP4G62UTBXX2YRZA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:28.567571990 (7h36m27.119042451s) [noise=true , address=145.79.227.31:4797 , piece_id=VQEEGJTLJY2IMCDHL2GMVGYEX4VL34TMMXKDYNERFYK3HRV7TC6A, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:33.651382593 (7h36m32.202853044s) [noise=true , address=145.79.227.20:4439 , piece_id=PPOVWSRAU7Y4CZAKAXV6ZP74EEPHONUC35O6XM57VE47NVJL5WZA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:34.842221635 (7h36m33.393692096s) [noise=true , address=145.79.227.9:4365 , piece_id=7BK6E45EY6Y3SV6PY3A4EVCS42PC4HYLXU3J7MN4HXRLDEFUEFCA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:35.601406517 (7h36m34.152877008s) [noise=true , address=145.79.227.25:4619 , piece_id=23D3MCZRVNPI5KB43ROAHKUALDGXLZBZU5E6EIU6U2FY3YQOVPGA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:35.697082303 (7h36m34.248552774s) [noise=true , address=145.79.227.17:4970 , piece_id=5ZP5DDC3XAYQ5UFYJVQV6IZHB5KQQGNG54MTYW3UUYPN2IAK5QVQ, seg_pos={0 0}, upload=6714]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: expected piece hash; context deadline exceeded; EOF Nov 9 15:40:35.697252518 (7h36m34.248722969s) [noise=true , address=145.79.227.17:4972 , piece_id=AI3KVNUJKJUD33YVU2Z472R72SKQKRYQG65JPKFZ5WBJLKRLW6MA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:145.79.227.17:4972): protocol: expected piece hash; context deadline exceeded; EOF Nov 9 15:40:35.697302322 (7h36m34.248772773s) [noise=true , address=145.79.227.21:4474 , piece_id=J5XE2MTCEW4ZHQDDZ4KCVYFK5EX6RQZARKNY4QCH4HUHGJO3SXDQ, seg_pos={0 0}, upload=6714]: Done uploading piece. err: ecclient: upload failed (node:1rL2DnjSgHP36Md3bCgy5padqTAhv3iQCeURbLMuuuMBxtUNZH, address:145.79.227.21:4474): protocol: expected piece hash; context deadline exceeded; EOF Nov 9 15:40:35.697472546 (7h36m34.248943007s) [noise=true , address=145.79.227.9:4366 , piece_id=BT3GWQSXIMMSFPUYJC5EQ2LX2DS4BW5DAMW6ILLNVVRUUHZMOQUA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: expected piece hash; context deadline exceeded; EOF Nov 9 15:40:35.697514686 (7h36m34.248985147s) [seg_pos={0 0}, upload=6714]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 9 15:40:35.950211122 (7h36m34.501681583s) [upload=6714]: Done uploading segments... Nov 9 15:40:35.950416643 (7h36m34.501887104s) [upload=6714]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 11.004942682s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 10.501591392s trace: https://jaeger.dev.storj.io/trace/3846516a5cecd778 FAIL (2025-11-09 16:21:26 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 9 15:40:06.410809146 (7h36m4.962279607s) [upload=6714]: Starting upload Nov 9 15:40:06.411046298 (7h36m4.962516769s) [upload=6714]: Done starting upload Nov 9 15:40:06.411314729 (7h36m4.96278519s) [upload=6714]: Uploading segments... Nov 9 15:40:06.411410882 (7h36m4.962881383s) [upload=6714]: Got next segment. Inline: false Nov 9 15:40:06.411464574 (7h36m4.962935035s) [upload=6714]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 9 15:40:06.411475064 (7h36m4.962945555s) [upload=6714]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 9 15:40:06.664789367 (7h36m5.216259828s) [seg_pos={0 0}, upload=6714]: Begin upload segment... Nov 9 15:40:06.664851705 (7h36m5.216322166s) [seg_pos={0 0}, upload=6714]: Done begin upload segment. Nov 9 15:40:06.664891662 (7h36m5.216362123s) [noise=true , address=145.79.227.32:4835 , piece_id=EYIBNSVPQPKLKOBHKZI2VZHVCDKW3C3YJSFEWP4G62UTBXX2YRZA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.664996933 (7h36m5.216467394s) [noise=true , address=45.91.116.14:4190 , piece_id=XSIDZL5HMLZTQE7ZAPURAAUGQLPP3QOZK5BIE5PHRMZPSTN5APSA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.665698789 (7h36m5.21716925s) [noise=true , address=77.90.31.14:4662 , piece_id=HHSJDZPYKGCHURAZV46OQSGBU7D4TFPXREDBFPGTYF7OAUMT2X3A, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.666456111 (7h36m5.217926572s) [noise=true , address=145.79.227.25:4619 , piece_id=23D3MCZRVNPI5KB43ROAHKUALDGXLZBZU5E6EIU6U2FY3YQOVPGA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.666642225 (7h36m5.218112676s) [noise=true , address=145.79.227.17:4971 , piece_id=VTUI377UCF44ICILORHEKUO33ICRXM2FECO5EXS7RUDH4VGV7FJA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.667027539 (7h36m5.218498s) [noise=true , address=145.79.227.9:4366 , piece_id=BT3GWQSXIMMSFPUYJC5EQ2LX2DS4BW5DAMW6ILLNVVRUUHZMOQUA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.667041675 (7h36m5.218512126s) [noise=true , address=145.79.227.17:4972 , piece_id=AI3KVNUJKJUD33YVU2Z472R72SKQKRYQG65JPKFZ5WBJLKRLW6MA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.667502794 (7h36m5.218973255s) [noise=true , address=145.79.227.13:4803 , piece_id=V37DZKBC4CHRGISXEQHTQAGKZFNXQKWXSWBSMHNRQTSJ32WYI2OQ, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.667805441 (7h36m5.219275902s) [noise=true , address=145.79.227.9:4365 , piece_id=7BK6E45EY6Y3SV6PY3A4EVCS42PC4HYLXU3J7MN4HXRLDEFUEFCA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.667895652 (7h36m5.219366113s) [noise=true , address=145.79.227.17:4970 , piece_id=5ZP5DDC3XAYQ5UFYJVQV6IZHB5KQQGNG54MTYW3UUYPN2IAK5QVQ, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.668240909 (7h36m5.219711361s) [noise=true , address=77.90.31.17:4772 , piece_id=CT6BNOYZGDBUEFVGM2RDS4ICMWPYS6N4IOUDA5V7MJQ3IKELNW2Q, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.668671640 (7h36m5.220142101s) [noise=true , address=145.79.227.31:4797 , piece_id=VQEEGJTLJY2IMCDHL2GMVGYEX4VL34TMMXKDYNERFYK3HRV7TC6A, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.669093554 (7h36m5.220564015s) [noise=true , address=145.79.227.21:4474 , piece_id=J5XE2MTCEW4ZHQDDZ4KCVYFK5EX6RQZARKNY4QCH4HUHGJO3SXDQ, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.669510097 (7h36m5.220980568s) [noise=true , address=77.90.31.18:4807 , piece_id=HDJAJYXNZNZN3LVAQSYVJEOQ3MU5H3BXCCW3NA2GLKPT7ATZHSJQ, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.671936887 (7h36m5.223407348s) [noise=true , address=145.79.227.20:4439 , piece_id=PPOVWSRAU7Y4CZAKAXV6ZP74EEPHONUC35O6XM57VE47NVJL5WZA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.672494549 (7h36m5.22396502s) [noise=true , address=77.90.31.21:4877 , piece_id=3V3Z6GNF32Q66EOS66WV3AGJNA7IXT5NHVAPKNUGVYU3R46IKKKA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:08.438197970 (7h36m6.989668451s) [upload=6714]: Next returned nil segment Nov 9 15:40:08.438219050 (7h36m6.989689521s) [upload=6714]: Waiting for error group managing segments... Nov 9 15:40:08.829503696 (7h36m7.380974147s) [noise=true , address=77.90.31.18:4807 , piece_id=HDJAJYXNZNZN3LVAQSYVJEOQ3MU5H3BXCCW3NA2GLKPT7ATZHSJQ, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:08.846923116 (7h36m7.398393577s) [noise=true , address=45.91.116.14:4190 , piece_id=XSIDZL5HMLZTQE7ZAPURAAUGQLPP3QOZK5BIE5PHRMZPSTN5APSA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:08.867005608 (7h36m7.418476069s) [noise=true , address=77.90.31.21:4877 , piece_id=3V3Z6GNF32Q66EOS66WV3AGJNA7IXT5NHVAPKNUGVYU3R46IKKKA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:08.888352807 (7h36m7.439823268s) [noise=true , address=77.90.31.14:4662 , piece_id=HHSJDZPYKGCHURAZV46OQSGBU7D4TFPXREDBFPGTYF7OAUMT2X3A, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:08.904680649 (7h36m7.45615111s) [noise=true , address=77.90.31.17:4772 , piece_id=CT6BNOYZGDBUEFVGM2RDS4ICMWPYS6N4IOUDA5V7MJQ3IKELNW2Q, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:18.663082162 (7h36m17.214552623s) [noise=true , address=145.79.227.17:4971 , piece_id=VTUI377UCF44ICILORHEKUO33ICRXM2FECO5EXS7RUDH4VGV7FJA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:20.077194714 (7h36m18.628665185s) [noise=true , address=145.79.227.13:4803 , piece_id=V37DZKBC4CHRGISXEQHTQAGKZFNXQKWXSWBSMHNRQTSJ32WYI2OQ, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:28.517900114 (7h36m27.069370575s) [noise=true , address=145.79.227.32:4835 , piece_id=EYIBNSVPQPKLKOBHKZI2VZHVCDKW3C3YJSFEWP4G62UTBXX2YRZA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:28.567571990 (7h36m27.119042451s) [noise=true , address=145.79.227.31:4797 , piece_id=VQEEGJTLJY2IMCDHL2GMVGYEX4VL34TMMXKDYNERFYK3HRV7TC6A, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:33.651382593 (7h36m32.202853044s) [noise=true , address=145.79.227.20:4439 , piece_id=PPOVWSRAU7Y4CZAKAXV6ZP74EEPHONUC35O6XM57VE47NVJL5WZA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:34.842221635 (7h36m33.393692096s) [noise=true , address=145.79.227.9:4365 , piece_id=7BK6E45EY6Y3SV6PY3A4EVCS42PC4HYLXU3J7MN4HXRLDEFUEFCA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:35.601406517 (7h36m34.152877008s) [noise=true , address=145.79.227.25:4619 , piece_id=23D3MCZRVNPI5KB43ROAHKUALDGXLZBZU5E6EIU6U2FY3YQOVPGA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:35.697082303 (7h36m34.248552774s) [noise=true , address=145.79.227.17:4970 , piece_id=5ZP5DDC3XAYQ5UFYJVQV6IZHB5KQQGNG54MTYW3UUYPN2IAK5QVQ, seg_pos={0 0}, upload=6714]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: expected piece hash; context deadline exceeded; EOF Nov 9 15:40:35.697252518 (7h36m34.248722969s) [noise=true , address=145.79.227.17:4972 , piece_id=AI3KVNUJKJUD33YVU2Z472R72SKQKRYQG65JPKFZ5WBJLKRLW6MA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:145.79.227.17:4972): protocol: expected piece hash; context deadline exceeded; EOF Nov 9 15:40:35.697302322 (7h36m34.248772773s) [noise=true , address=145.79.227.21:4474 , piece_id=J5XE2MTCEW4ZHQDDZ4KCVYFK5EX6RQZARKNY4QCH4HUHGJO3SXDQ, seg_pos={0 0}, upload=6714]: Done uploading piece. err: ecclient: upload failed (node:1rL2DnjSgHP36Md3bCgy5padqTAhv3iQCeURbLMuuuMBxtUNZH, address:145.79.227.21:4474): protocol: expected piece hash; context deadline exceeded; EOF Nov 9 15:40:35.697472546 (7h36m34.248943007s) [noise=true , address=145.79.227.9:4366 , piece_id=BT3GWQSXIMMSFPUYJC5EQ2LX2DS4BW5DAMW6ILLNVVRUUHZMOQUA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: expected piece hash; context deadline exceeded; EOF Nov 9 15:40:35.697514686 (7h36m34.248985147s) [seg_pos={0 0}, upload=6714]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 9 15:40:35.950211122 (7h36m34.501681583s) [upload=6714]: Done uploading segments... Nov 9 15:40:35.950416643 (7h36m34.501887104s) [upload=6714]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 10.501611351s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 10.014582034s trace: https://jaeger.dev.storj.io/trace/3846516a5cecd778 FAIL (2025-11-09 16:20:43 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 9 15:40:06.410809146 (7h36m4.962279607s) [upload=6714]: Starting upload Nov 9 15:40:06.411046298 (7h36m4.962516769s) [upload=6714]: Done starting upload Nov 9 15:40:06.411314729 (7h36m4.96278519s) [upload=6714]: Uploading segments... Nov 9 15:40:06.411410882 (7h36m4.962881383s) [upload=6714]: Got next segment. Inline: false Nov 9 15:40:06.411464574 (7h36m4.962935035s) [upload=6714]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 9 15:40:06.411475064 (7h36m4.962945555s) [upload=6714]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 9 15:40:06.664789367 (7h36m5.216259828s) [seg_pos={0 0}, upload=6714]: Begin upload segment... Nov 9 15:40:06.664851705 (7h36m5.216322166s) [seg_pos={0 0}, upload=6714]: Done begin upload segment. Nov 9 15:40:06.664891662 (7h36m5.216362123s) [noise=true , address=145.79.227.32:4835 , piece_id=EYIBNSVPQPKLKOBHKZI2VZHVCDKW3C3YJSFEWP4G62UTBXX2YRZA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.664996933 (7h36m5.216467394s) [noise=true , address=45.91.116.14:4190 , piece_id=XSIDZL5HMLZTQE7ZAPURAAUGQLPP3QOZK5BIE5PHRMZPSTN5APSA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.665698789 (7h36m5.21716925s) [noise=true , address=77.90.31.14:4662 , piece_id=HHSJDZPYKGCHURAZV46OQSGBU7D4TFPXREDBFPGTYF7OAUMT2X3A, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.666456111 (7h36m5.217926572s) [noise=true , address=145.79.227.25:4619 , piece_id=23D3MCZRVNPI5KB43ROAHKUALDGXLZBZU5E6EIU6U2FY3YQOVPGA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.666642225 (7h36m5.218112676s) [noise=true , address=145.79.227.17:4971 , piece_id=VTUI377UCF44ICILORHEKUO33ICRXM2FECO5EXS7RUDH4VGV7FJA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.667027539 (7h36m5.218498s) [noise=true , address=145.79.227.9:4366 , piece_id=BT3GWQSXIMMSFPUYJC5EQ2LX2DS4BW5DAMW6ILLNVVRUUHZMOQUA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.667041675 (7h36m5.218512126s) [noise=true , address=145.79.227.17:4972 , piece_id=AI3KVNUJKJUD33YVU2Z472R72SKQKRYQG65JPKFZ5WBJLKRLW6MA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.667502794 (7h36m5.218973255s) [noise=true , address=145.79.227.13:4803 , piece_id=V37DZKBC4CHRGISXEQHTQAGKZFNXQKWXSWBSMHNRQTSJ32WYI2OQ, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.667805441 (7h36m5.219275902s) [noise=true , address=145.79.227.9:4365 , piece_id=7BK6E45EY6Y3SV6PY3A4EVCS42PC4HYLXU3J7MN4HXRLDEFUEFCA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.667895652 (7h36m5.219366113s) [noise=true , address=145.79.227.17:4970 , piece_id=5ZP5DDC3XAYQ5UFYJVQV6IZHB5KQQGNG54MTYW3UUYPN2IAK5QVQ, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.668240909 (7h36m5.219711361s) [noise=true , address=77.90.31.17:4772 , piece_id=CT6BNOYZGDBUEFVGM2RDS4ICMWPYS6N4IOUDA5V7MJQ3IKELNW2Q, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.668671640 (7h36m5.220142101s) [noise=true , address=145.79.227.31:4797 , piece_id=VQEEGJTLJY2IMCDHL2GMVGYEX4VL34TMMXKDYNERFYK3HRV7TC6A, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.669093554 (7h36m5.220564015s) [noise=true , address=145.79.227.21:4474 , piece_id=J5XE2MTCEW4ZHQDDZ4KCVYFK5EX6RQZARKNY4QCH4HUHGJO3SXDQ, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.669510097 (7h36m5.220980568s) [noise=true , address=77.90.31.18:4807 , piece_id=HDJAJYXNZNZN3LVAQSYVJEOQ3MU5H3BXCCW3NA2GLKPT7ATZHSJQ, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.671936887 (7h36m5.223407348s) [noise=true , address=145.79.227.20:4439 , piece_id=PPOVWSRAU7Y4CZAKAXV6ZP74EEPHONUC35O6XM57VE47NVJL5WZA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.672494549 (7h36m5.22396502s) [noise=true , address=77.90.31.21:4877 , piece_id=3V3Z6GNF32Q66EOS66WV3AGJNA7IXT5NHVAPKNUGVYU3R46IKKKA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:08.438197970 (7h36m6.989668451s) [upload=6714]: Next returned nil segment Nov 9 15:40:08.438219050 (7h36m6.989689521s) [upload=6714]: Waiting for error group managing segments... Nov 9 15:40:08.829503696 (7h36m7.380974147s) [noise=true , address=77.90.31.18:4807 , piece_id=HDJAJYXNZNZN3LVAQSYVJEOQ3MU5H3BXCCW3NA2GLKPT7ATZHSJQ, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:08.846923116 (7h36m7.398393577s) [noise=true , address=45.91.116.14:4190 , piece_id=XSIDZL5HMLZTQE7ZAPURAAUGQLPP3QOZK5BIE5PHRMZPSTN5APSA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:08.867005608 (7h36m7.418476069s) [noise=true , address=77.90.31.21:4877 , piece_id=3V3Z6GNF32Q66EOS66WV3AGJNA7IXT5NHVAPKNUGVYU3R46IKKKA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:08.888352807 (7h36m7.439823268s) [noise=true , address=77.90.31.14:4662 , piece_id=HHSJDZPYKGCHURAZV46OQSGBU7D4TFPXREDBFPGTYF7OAUMT2X3A, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:08.904680649 (7h36m7.45615111s) [noise=true , address=77.90.31.17:4772 , piece_id=CT6BNOYZGDBUEFVGM2RDS4ICMWPYS6N4IOUDA5V7MJQ3IKELNW2Q, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:18.663082162 (7h36m17.214552623s) [noise=true , address=145.79.227.17:4971 , piece_id=VTUI377UCF44ICILORHEKUO33ICRXM2FECO5EXS7RUDH4VGV7FJA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:20.077194714 (7h36m18.628665185s) [noise=true , address=145.79.227.13:4803 , piece_id=V37DZKBC4CHRGISXEQHTQAGKZFNXQKWXSWBSMHNRQTSJ32WYI2OQ, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:28.517900114 (7h36m27.069370575s) [noise=true , address=145.79.227.32:4835 , piece_id=EYIBNSVPQPKLKOBHKZI2VZHVCDKW3C3YJSFEWP4G62UTBXX2YRZA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:28.567571990 (7h36m27.119042451s) [noise=true , address=145.79.227.31:4797 , piece_id=VQEEGJTLJY2IMCDHL2GMVGYEX4VL34TMMXKDYNERFYK3HRV7TC6A, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:33.651382593 (7h36m32.202853044s) [noise=true , address=145.79.227.20:4439 , piece_id=PPOVWSRAU7Y4CZAKAXV6ZP74EEPHONUC35O6XM57VE47NVJL5WZA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:34.842221635 (7h36m33.393692096s) [noise=true , address=145.79.227.9:4365 , piece_id=7BK6E45EY6Y3SV6PY3A4EVCS42PC4HYLXU3J7MN4HXRLDEFUEFCA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:35.601406517 (7h36m34.152877008s) [noise=true , address=145.79.227.25:4619 , piece_id=23D3MCZRVNPI5KB43ROAHKUALDGXLZBZU5E6EIU6U2FY3YQOVPGA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:35.697082303 (7h36m34.248552774s) [noise=true , address=145.79.227.17:4970 , piece_id=5ZP5DDC3XAYQ5UFYJVQV6IZHB5KQQGNG54MTYW3UUYPN2IAK5QVQ, seg_pos={0 0}, upload=6714]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: expected piece hash; context deadline exceeded; EOF Nov 9 15:40:35.697252518 (7h36m34.248722969s) [noise=true , address=145.79.227.17:4972 , piece_id=AI3KVNUJKJUD33YVU2Z472R72SKQKRYQG65JPKFZ5WBJLKRLW6MA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:145.79.227.17:4972): protocol: expected piece hash; context deadline exceeded; EOF Nov 9 15:40:35.697302322 (7h36m34.248772773s) [noise=true , address=145.79.227.21:4474 , piece_id=J5XE2MTCEW4ZHQDDZ4KCVYFK5EX6RQZARKNY4QCH4HUHGJO3SXDQ, seg_pos={0 0}, upload=6714]: Done uploading piece. err: ecclient: upload failed (node:1rL2DnjSgHP36Md3bCgy5padqTAhv3iQCeURbLMuuuMBxtUNZH, address:145.79.227.21:4474): protocol: expected piece hash; context deadline exceeded; EOF Nov 9 15:40:35.697472546 (7h36m34.248943007s) [noise=true , address=145.79.227.9:4366 , piece_id=BT3GWQSXIMMSFPUYJC5EQ2LX2DS4BW5DAMW6ILLNVVRUUHZMOQUA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: expected piece hash; context deadline exceeded; EOF Nov 9 15:40:35.697514686 (7h36m34.248985147s) [seg_pos={0 0}, upload=6714]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 9 15:40:35.950211122 (7h36m34.501681583s) [upload=6714]: Done uploading segments... Nov 9 15:40:35.950416643 (7h36m34.501887104s) [upload=6714]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 10.014628632s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 9.059059112s trace: https://jaeger.dev.storj.io/trace/3846516a5cecd778 FAIL (2025-11-09 16:20:04 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 9 15:40:06.410809146 (7h36m4.962279607s) [upload=6714]: Starting upload Nov 9 15:40:06.411046298 (7h36m4.962516769s) [upload=6714]: Done starting upload Nov 9 15:40:06.411314729 (7h36m4.96278519s) [upload=6714]: Uploading segments... Nov 9 15:40:06.411410882 (7h36m4.962881383s) [upload=6714]: Got next segment. Inline: false Nov 9 15:40:06.411464574 (7h36m4.962935035s) [upload=6714]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 9 15:40:06.411475064 (7h36m4.962945555s) [upload=6714]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 9 15:40:06.664789367 (7h36m5.216259828s) [seg_pos={0 0}, upload=6714]: Begin upload segment... Nov 9 15:40:06.664851705 (7h36m5.216322166s) [seg_pos={0 0}, upload=6714]: Done begin upload segment. Nov 9 15:40:06.664891662 (7h36m5.216362123s) [noise=true , address=145.79.227.32:4835 , piece_id=EYIBNSVPQPKLKOBHKZI2VZHVCDKW3C3YJSFEWP4G62UTBXX2YRZA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.664996933 (7h36m5.216467394s) [noise=true , address=45.91.116.14:4190 , piece_id=XSIDZL5HMLZTQE7ZAPURAAUGQLPP3QOZK5BIE5PHRMZPSTN5APSA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.665698789 (7h36m5.21716925s) [noise=true , address=77.90.31.14:4662 , piece_id=HHSJDZPYKGCHURAZV46OQSGBU7D4TFPXREDBFPGTYF7OAUMT2X3A, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.666456111 (7h36m5.217926572s) [noise=true , address=145.79.227.25:4619 , piece_id=23D3MCZRVNPI5KB43ROAHKUALDGXLZBZU5E6EIU6U2FY3YQOVPGA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.666642225 (7h36m5.218112676s) [noise=true , address=145.79.227.17:4971 , piece_id=VTUI377UCF44ICILORHEKUO33ICRXM2FECO5EXS7RUDH4VGV7FJA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.667027539 (7h36m5.218498s) [noise=true , address=145.79.227.9:4366 , piece_id=BT3GWQSXIMMSFPUYJC5EQ2LX2DS4BW5DAMW6ILLNVVRUUHZMOQUA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.667041675 (7h36m5.218512126s) [noise=true , address=145.79.227.17:4972 , piece_id=AI3KVNUJKJUD33YVU2Z472R72SKQKRYQG65JPKFZ5WBJLKRLW6MA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.667502794 (7h36m5.218973255s) [noise=true , address=145.79.227.13:4803 , piece_id=V37DZKBC4CHRGISXEQHTQAGKZFNXQKWXSWBSMHNRQTSJ32WYI2OQ, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.667805441 (7h36m5.219275902s) [noise=true , address=145.79.227.9:4365 , piece_id=7BK6E45EY6Y3SV6PY3A4EVCS42PC4HYLXU3J7MN4HXRLDEFUEFCA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.667895652 (7h36m5.219366113s) [noise=true , address=145.79.227.17:4970 , piece_id=5ZP5DDC3XAYQ5UFYJVQV6IZHB5KQQGNG54MTYW3UUYPN2IAK5QVQ, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.668240909 (7h36m5.219711361s) [noise=true , address=77.90.31.17:4772 , piece_id=CT6BNOYZGDBUEFVGM2RDS4ICMWPYS6N4IOUDA5V7MJQ3IKELNW2Q, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.668671640 (7h36m5.220142101s) [noise=true , address=145.79.227.31:4797 , piece_id=VQEEGJTLJY2IMCDHL2GMVGYEX4VL34TMMXKDYNERFYK3HRV7TC6A, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.669093554 (7h36m5.220564015s) [noise=true , address=145.79.227.21:4474 , piece_id=J5XE2MTCEW4ZHQDDZ4KCVYFK5EX6RQZARKNY4QCH4HUHGJO3SXDQ, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.669510097 (7h36m5.220980568s) [noise=true , address=77.90.31.18:4807 , piece_id=HDJAJYXNZNZN3LVAQSYVJEOQ3MU5H3BXCCW3NA2GLKPT7ATZHSJQ, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.671936887 (7h36m5.223407348s) [noise=true , address=145.79.227.20:4439 , piece_id=PPOVWSRAU7Y4CZAKAXV6ZP74EEPHONUC35O6XM57VE47NVJL5WZA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.672494549 (7h36m5.22396502s) [noise=true , address=77.90.31.21:4877 , piece_id=3V3Z6GNF32Q66EOS66WV3AGJNA7IXT5NHVAPKNUGVYU3R46IKKKA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:08.438197970 (7h36m6.989668451s) [upload=6714]: Next returned nil segment Nov 9 15:40:08.438219050 (7h36m6.989689521s) [upload=6714]: Waiting for error group managing segments... Nov 9 15:40:08.829503696 (7h36m7.380974147s) [noise=true , address=77.90.31.18:4807 , piece_id=HDJAJYXNZNZN3LVAQSYVJEOQ3MU5H3BXCCW3NA2GLKPT7ATZHSJQ, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:08.846923116 (7h36m7.398393577s) [noise=true , address=45.91.116.14:4190 , piece_id=XSIDZL5HMLZTQE7ZAPURAAUGQLPP3QOZK5BIE5PHRMZPSTN5APSA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:08.867005608 (7h36m7.418476069s) [noise=true , address=77.90.31.21:4877 , piece_id=3V3Z6GNF32Q66EOS66WV3AGJNA7IXT5NHVAPKNUGVYU3R46IKKKA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:08.888352807 (7h36m7.439823268s) [noise=true , address=77.90.31.14:4662 , piece_id=HHSJDZPYKGCHURAZV46OQSGBU7D4TFPXREDBFPGTYF7OAUMT2X3A, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:08.904680649 (7h36m7.45615111s) [noise=true , address=77.90.31.17:4772 , piece_id=CT6BNOYZGDBUEFVGM2RDS4ICMWPYS6N4IOUDA5V7MJQ3IKELNW2Q, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:18.663082162 (7h36m17.214552623s) [noise=true , address=145.79.227.17:4971 , piece_id=VTUI377UCF44ICILORHEKUO33ICRXM2FECO5EXS7RUDH4VGV7FJA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:20.077194714 (7h36m18.628665185s) [noise=true , address=145.79.227.13:4803 , piece_id=V37DZKBC4CHRGISXEQHTQAGKZFNXQKWXSWBSMHNRQTSJ32WYI2OQ, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:28.517900114 (7h36m27.069370575s) [noise=true , address=145.79.227.32:4835 , piece_id=EYIBNSVPQPKLKOBHKZI2VZHVCDKW3C3YJSFEWP4G62UTBXX2YRZA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:28.567571990 (7h36m27.119042451s) [noise=true , address=145.79.227.31:4797 , piece_id=VQEEGJTLJY2IMCDHL2GMVGYEX4VL34TMMXKDYNERFYK3HRV7TC6A, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:33.651382593 (7h36m32.202853044s) [noise=true , address=145.79.227.20:4439 , piece_id=PPOVWSRAU7Y4CZAKAXV6ZP74EEPHONUC35O6XM57VE47NVJL5WZA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:34.842221635 (7h36m33.393692096s) [noise=true , address=145.79.227.9:4365 , piece_id=7BK6E45EY6Y3SV6PY3A4EVCS42PC4HYLXU3J7MN4HXRLDEFUEFCA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:35.601406517 (7h36m34.152877008s) [noise=true , address=145.79.227.25:4619 , piece_id=23D3MCZRVNPI5KB43ROAHKUALDGXLZBZU5E6EIU6U2FY3YQOVPGA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:35.697082303 (7h36m34.248552774s) [noise=true , address=145.79.227.17:4970 , piece_id=5ZP5DDC3XAYQ5UFYJVQV6IZHB5KQQGNG54MTYW3UUYPN2IAK5QVQ, seg_pos={0 0}, upload=6714]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: expected piece hash; context deadline exceeded; EOF Nov 9 15:40:35.697252518 (7h36m34.248722969s) [noise=true , address=145.79.227.17:4972 , piece_id=AI3KVNUJKJUD33YVU2Z472R72SKQKRYQG65JPKFZ5WBJLKRLW6MA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:145.79.227.17:4972): protocol: expected piece hash; context deadline exceeded; EOF Nov 9 15:40:35.697302322 (7h36m34.248772773s) [noise=true , address=145.79.227.21:4474 , piece_id=J5XE2MTCEW4ZHQDDZ4KCVYFK5EX6RQZARKNY4QCH4HUHGJO3SXDQ, seg_pos={0 0}, upload=6714]: Done uploading piece. err: ecclient: upload failed (node:1rL2DnjSgHP36Md3bCgy5padqTAhv3iQCeURbLMuuuMBxtUNZH, address:145.79.227.21:4474): protocol: expected piece hash; context deadline exceeded; EOF Nov 9 15:40:35.697472546 (7h36m34.248943007s) [noise=true , address=145.79.227.9:4366 , piece_id=BT3GWQSXIMMSFPUYJC5EQ2LX2DS4BW5DAMW6ILLNVVRUUHZMOQUA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: expected piece hash; context deadline exceeded; EOF Nov 9 15:40:35.697514686 (7h36m34.248985147s) [seg_pos={0 0}, upload=6714]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 9 15:40:35.950211122 (7h36m34.501681583s) [upload=6714]: Done uploading segments... Nov 9 15:40:35.950416643 (7h36m34.501887104s) [upload=6714]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 9.059088457s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 9.33443425s trace: https://jaeger.dev.storj.io/trace/3846516a5cecd778 FAIL (2025-11-09 16:19:06 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 9 15:40:06.410809146 (7h36m4.962279607s) [upload=6714]: Starting upload Nov 9 15:40:06.411046298 (7h36m4.962516769s) [upload=6714]: Done starting upload Nov 9 15:40:06.411314729 (7h36m4.96278519s) [upload=6714]: Uploading segments... Nov 9 15:40:06.411410882 (7h36m4.962881383s) [upload=6714]: Got next segment. Inline: false Nov 9 15:40:06.411464574 (7h36m4.962935035s) [upload=6714]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 9 15:40:06.411475064 (7h36m4.962945555s) [upload=6714]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 9 15:40:06.664789367 (7h36m5.216259828s) [seg_pos={0 0}, upload=6714]: Begin upload segment... Nov 9 15:40:06.664851705 (7h36m5.216322166s) [seg_pos={0 0}, upload=6714]: Done begin upload segment. Nov 9 15:40:06.664891662 (7h36m5.216362123s) [noise=true , address=145.79.227.32:4835 , piece_id=EYIBNSVPQPKLKOBHKZI2VZHVCDKW3C3YJSFEWP4G62UTBXX2YRZA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.664996933 (7h36m5.216467394s) [noise=true , address=45.91.116.14:4190 , piece_id=XSIDZL5HMLZTQE7ZAPURAAUGQLPP3QOZK5BIE5PHRMZPSTN5APSA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.665698789 (7h36m5.21716925s) [noise=true , address=77.90.31.14:4662 , piece_id=HHSJDZPYKGCHURAZV46OQSGBU7D4TFPXREDBFPGTYF7OAUMT2X3A, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.666456111 (7h36m5.217926572s) [noise=true , address=145.79.227.25:4619 , piece_id=23D3MCZRVNPI5KB43ROAHKUALDGXLZBZU5E6EIU6U2FY3YQOVPGA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.666642225 (7h36m5.218112676s) [noise=true , address=145.79.227.17:4971 , piece_id=VTUI377UCF44ICILORHEKUO33ICRXM2FECO5EXS7RUDH4VGV7FJA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.667027539 (7h36m5.218498s) [noise=true , address=145.79.227.9:4366 , piece_id=BT3GWQSXIMMSFPUYJC5EQ2LX2DS4BW5DAMW6ILLNVVRUUHZMOQUA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.667041675 (7h36m5.218512126s) [noise=true , address=145.79.227.17:4972 , piece_id=AI3KVNUJKJUD33YVU2Z472R72SKQKRYQG65JPKFZ5WBJLKRLW6MA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.667502794 (7h36m5.218973255s) [noise=true , address=145.79.227.13:4803 , piece_id=V37DZKBC4CHRGISXEQHTQAGKZFNXQKWXSWBSMHNRQTSJ32WYI2OQ, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.667805441 (7h36m5.219275902s) [noise=true , address=145.79.227.9:4365 , piece_id=7BK6E45EY6Y3SV6PY3A4EVCS42PC4HYLXU3J7MN4HXRLDEFUEFCA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.667895652 (7h36m5.219366113s) [noise=true , address=145.79.227.17:4970 , piece_id=5ZP5DDC3XAYQ5UFYJVQV6IZHB5KQQGNG54MTYW3UUYPN2IAK5QVQ, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.668240909 (7h36m5.219711361s) [noise=true , address=77.90.31.17:4772 , piece_id=CT6BNOYZGDBUEFVGM2RDS4ICMWPYS6N4IOUDA5V7MJQ3IKELNW2Q, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.668671640 (7h36m5.220142101s) [noise=true , address=145.79.227.31:4797 , piece_id=VQEEGJTLJY2IMCDHL2GMVGYEX4VL34TMMXKDYNERFYK3HRV7TC6A, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.669093554 (7h36m5.220564015s) [noise=true , address=145.79.227.21:4474 , piece_id=J5XE2MTCEW4ZHQDDZ4KCVYFK5EX6RQZARKNY4QCH4HUHGJO3SXDQ, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.669510097 (7h36m5.220980568s) [noise=true , address=77.90.31.18:4807 , piece_id=HDJAJYXNZNZN3LVAQSYVJEOQ3MU5H3BXCCW3NA2GLKPT7ATZHSJQ, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.671936887 (7h36m5.223407348s) [noise=true , address=145.79.227.20:4439 , piece_id=PPOVWSRAU7Y4CZAKAXV6ZP74EEPHONUC35O6XM57VE47NVJL5WZA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.672494549 (7h36m5.22396502s) [noise=true , address=77.90.31.21:4877 , piece_id=3V3Z6GNF32Q66EOS66WV3AGJNA7IXT5NHVAPKNUGVYU3R46IKKKA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:08.438197970 (7h36m6.989668451s) [upload=6714]: Next returned nil segment Nov 9 15:40:08.438219050 (7h36m6.989689521s) [upload=6714]: Waiting for error group managing segments... Nov 9 15:40:08.829503696 (7h36m7.380974147s) [noise=true , address=77.90.31.18:4807 , piece_id=HDJAJYXNZNZN3LVAQSYVJEOQ3MU5H3BXCCW3NA2GLKPT7ATZHSJQ, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:08.846923116 (7h36m7.398393577s) [noise=true , address=45.91.116.14:4190 , piece_id=XSIDZL5HMLZTQE7ZAPURAAUGQLPP3QOZK5BIE5PHRMZPSTN5APSA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:08.867005608 (7h36m7.418476069s) [noise=true , address=77.90.31.21:4877 , piece_id=3V3Z6GNF32Q66EOS66WV3AGJNA7IXT5NHVAPKNUGVYU3R46IKKKA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:08.888352807 (7h36m7.439823268s) [noise=true , address=77.90.31.14:4662 , piece_id=HHSJDZPYKGCHURAZV46OQSGBU7D4TFPXREDBFPGTYF7OAUMT2X3A, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:08.904680649 (7h36m7.45615111s) [noise=true , address=77.90.31.17:4772 , piece_id=CT6BNOYZGDBUEFVGM2RDS4ICMWPYS6N4IOUDA5V7MJQ3IKELNW2Q, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:18.663082162 (7h36m17.214552623s) [noise=true , address=145.79.227.17:4971 , piece_id=VTUI377UCF44ICILORHEKUO33ICRXM2FECO5EXS7RUDH4VGV7FJA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:20.077194714 (7h36m18.628665185s) [noise=true , address=145.79.227.13:4803 , piece_id=V37DZKBC4CHRGISXEQHTQAGKZFNXQKWXSWBSMHNRQTSJ32WYI2OQ, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:28.517900114 (7h36m27.069370575s) [noise=true , address=145.79.227.32:4835 , piece_id=EYIBNSVPQPKLKOBHKZI2VZHVCDKW3C3YJSFEWP4G62UTBXX2YRZA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:28.567571990 (7h36m27.119042451s) [noise=true , address=145.79.227.31:4797 , piece_id=VQEEGJTLJY2IMCDHL2GMVGYEX4VL34TMMXKDYNERFYK3HRV7TC6A, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:33.651382593 (7h36m32.202853044s) [noise=true , address=145.79.227.20:4439 , piece_id=PPOVWSRAU7Y4CZAKAXV6ZP74EEPHONUC35O6XM57VE47NVJL5WZA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:34.842221635 (7h36m33.393692096s) [noise=true , address=145.79.227.9:4365 , piece_id=7BK6E45EY6Y3SV6PY3A4EVCS42PC4HYLXU3J7MN4HXRLDEFUEFCA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:35.601406517 (7h36m34.152877008s) [noise=true , address=145.79.227.25:4619 , piece_id=23D3MCZRVNPI5KB43ROAHKUALDGXLZBZU5E6EIU6U2FY3YQOVPGA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:35.697082303 (7h36m34.248552774s) [noise=true , address=145.79.227.17:4970 , piece_id=5ZP5DDC3XAYQ5UFYJVQV6IZHB5KQQGNG54MTYW3UUYPN2IAK5QVQ, seg_pos={0 0}, upload=6714]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: expected piece hash; context deadline exceeded; EOF Nov 9 15:40:35.697252518 (7h36m34.248722969s) [noise=true , address=145.79.227.17:4972 , piece_id=AI3KVNUJKJUD33YVU2Z472R72SKQKRYQG65JPKFZ5WBJLKRLW6MA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:145.79.227.17:4972): protocol: expected piece hash; context deadline exceeded; EOF Nov 9 15:40:35.697302322 (7h36m34.248772773s) [noise=true , address=145.79.227.21:4474 , piece_id=J5XE2MTCEW4ZHQDDZ4KCVYFK5EX6RQZARKNY4QCH4HUHGJO3SXDQ, seg_pos={0 0}, upload=6714]: Done uploading piece. err: ecclient: upload failed (node:1rL2DnjSgHP36Md3bCgy5padqTAhv3iQCeURbLMuuuMBxtUNZH, address:145.79.227.21:4474): protocol: expected piece hash; context deadline exceeded; EOF Nov 9 15:40:35.697472546 (7h36m34.248943007s) [noise=true , address=145.79.227.9:4366 , piece_id=BT3GWQSXIMMSFPUYJC5EQ2LX2DS4BW5DAMW6ILLNVVRUUHZMOQUA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: expected piece hash; context deadline exceeded; EOF Nov 9 15:40:35.697514686 (7h36m34.248985147s) [seg_pos={0 0}, upload=6714]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 9 15:40:35.950211122 (7h36m34.501681583s) [upload=6714]: Done uploading segments... Nov 9 15:40:35.950416643 (7h36m34.501887104s) [upload=6714]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 9.334487812s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 9.288419632s trace: https://jaeger.dev.storj.io/trace/3846516a5cecd778 FAIL (2025-11-09 16:18:18 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 9 15:40:06.410809146 (7h36m4.962279607s) [upload=6714]: Starting upload Nov 9 15:40:06.411046298 (7h36m4.962516769s) [upload=6714]: Done starting upload Nov 9 15:40:06.411314729 (7h36m4.96278519s) [upload=6714]: Uploading segments... Nov 9 15:40:06.411410882 (7h36m4.962881383s) [upload=6714]: Got next segment. Inline: false Nov 9 15:40:06.411464574 (7h36m4.962935035s) [upload=6714]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 9 15:40:06.411475064 (7h36m4.962945555s) [upload=6714]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 9 15:40:06.664789367 (7h36m5.216259828s) [seg_pos={0 0}, upload=6714]: Begin upload segment... Nov 9 15:40:06.664851705 (7h36m5.216322166s) [seg_pos={0 0}, upload=6714]: Done begin upload segment. Nov 9 15:40:06.664891662 (7h36m5.216362123s) [noise=true , address=145.79.227.32:4835 , piece_id=EYIBNSVPQPKLKOBHKZI2VZHVCDKW3C3YJSFEWP4G62UTBXX2YRZA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.664996933 (7h36m5.216467394s) [noise=true , address=45.91.116.14:4190 , piece_id=XSIDZL5HMLZTQE7ZAPURAAUGQLPP3QOZK5BIE5PHRMZPSTN5APSA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.665698789 (7h36m5.21716925s) [noise=true , address=77.90.31.14:4662 , piece_id=HHSJDZPYKGCHURAZV46OQSGBU7D4TFPXREDBFPGTYF7OAUMT2X3A, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.666456111 (7h36m5.217926572s) [noise=true , address=145.79.227.25:4619 , piece_id=23D3MCZRVNPI5KB43ROAHKUALDGXLZBZU5E6EIU6U2FY3YQOVPGA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.666642225 (7h36m5.218112676s) [noise=true , address=145.79.227.17:4971 , piece_id=VTUI377UCF44ICILORHEKUO33ICRXM2FECO5EXS7RUDH4VGV7FJA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.667027539 (7h36m5.218498s) [noise=true , address=145.79.227.9:4366 , piece_id=BT3GWQSXIMMSFPUYJC5EQ2LX2DS4BW5DAMW6ILLNVVRUUHZMOQUA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.667041675 (7h36m5.218512126s) [noise=true , address=145.79.227.17:4972 , piece_id=AI3KVNUJKJUD33YVU2Z472R72SKQKRYQG65JPKFZ5WBJLKRLW6MA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.667502794 (7h36m5.218973255s) [noise=true , address=145.79.227.13:4803 , piece_id=V37DZKBC4CHRGISXEQHTQAGKZFNXQKWXSWBSMHNRQTSJ32WYI2OQ, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.667805441 (7h36m5.219275902s) [noise=true , address=145.79.227.9:4365 , piece_id=7BK6E45EY6Y3SV6PY3A4EVCS42PC4HYLXU3J7MN4HXRLDEFUEFCA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.667895652 (7h36m5.219366113s) [noise=true , address=145.79.227.17:4970 , piece_id=5ZP5DDC3XAYQ5UFYJVQV6IZHB5KQQGNG54MTYW3UUYPN2IAK5QVQ, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.668240909 (7h36m5.219711361s) [noise=true , address=77.90.31.17:4772 , piece_id=CT6BNOYZGDBUEFVGM2RDS4ICMWPYS6N4IOUDA5V7MJQ3IKELNW2Q, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.668671640 (7h36m5.220142101s) [noise=true , address=145.79.227.31:4797 , piece_id=VQEEGJTLJY2IMCDHL2GMVGYEX4VL34TMMXKDYNERFYK3HRV7TC6A, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.669093554 (7h36m5.220564015s) [noise=true , address=145.79.227.21:4474 , piece_id=J5XE2MTCEW4ZHQDDZ4KCVYFK5EX6RQZARKNY4QCH4HUHGJO3SXDQ, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.669510097 (7h36m5.220980568s) [noise=true , address=77.90.31.18:4807 , piece_id=HDJAJYXNZNZN3LVAQSYVJEOQ3MU5H3BXCCW3NA2GLKPT7ATZHSJQ, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.671936887 (7h36m5.223407348s) [noise=true , address=145.79.227.20:4439 , piece_id=PPOVWSRAU7Y4CZAKAXV6ZP74EEPHONUC35O6XM57VE47NVJL5WZA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.672494549 (7h36m5.22396502s) [noise=true , address=77.90.31.21:4877 , piece_id=3V3Z6GNF32Q66EOS66WV3AGJNA7IXT5NHVAPKNUGVYU3R46IKKKA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:08.438197970 (7h36m6.989668451s) [upload=6714]: Next returned nil segment Nov 9 15:40:08.438219050 (7h36m6.989689521s) [upload=6714]: Waiting for error group managing segments... Nov 9 15:40:08.829503696 (7h36m7.380974147s) [noise=true , address=77.90.31.18:4807 , piece_id=HDJAJYXNZNZN3LVAQSYVJEOQ3MU5H3BXCCW3NA2GLKPT7ATZHSJQ, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:08.846923116 (7h36m7.398393577s) [noise=true , address=45.91.116.14:4190 , piece_id=XSIDZL5HMLZTQE7ZAPURAAUGQLPP3QOZK5BIE5PHRMZPSTN5APSA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:08.867005608 (7h36m7.418476069s) [noise=true , address=77.90.31.21:4877 , piece_id=3V3Z6GNF32Q66EOS66WV3AGJNA7IXT5NHVAPKNUGVYU3R46IKKKA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:08.888352807 (7h36m7.439823268s) [noise=true , address=77.90.31.14:4662 , piece_id=HHSJDZPYKGCHURAZV46OQSGBU7D4TFPXREDBFPGTYF7OAUMT2X3A, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:08.904680649 (7h36m7.45615111s) [noise=true , address=77.90.31.17:4772 , piece_id=CT6BNOYZGDBUEFVGM2RDS4ICMWPYS6N4IOUDA5V7MJQ3IKELNW2Q, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:18.663082162 (7h36m17.214552623s) [noise=true , address=145.79.227.17:4971 , piece_id=VTUI377UCF44ICILORHEKUO33ICRXM2FECO5EXS7RUDH4VGV7FJA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:20.077194714 (7h36m18.628665185s) [noise=true , address=145.79.227.13:4803 , piece_id=V37DZKBC4CHRGISXEQHTQAGKZFNXQKWXSWBSMHNRQTSJ32WYI2OQ, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:28.517900114 (7h36m27.069370575s) [noise=true , address=145.79.227.32:4835 , piece_id=EYIBNSVPQPKLKOBHKZI2VZHVCDKW3C3YJSFEWP4G62UTBXX2YRZA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:28.567571990 (7h36m27.119042451s) [noise=true , address=145.79.227.31:4797 , piece_id=VQEEGJTLJY2IMCDHL2GMVGYEX4VL34TMMXKDYNERFYK3HRV7TC6A, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:33.651382593 (7h36m32.202853044s) [noise=true , address=145.79.227.20:4439 , piece_id=PPOVWSRAU7Y4CZAKAXV6ZP74EEPHONUC35O6XM57VE47NVJL5WZA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:34.842221635 (7h36m33.393692096s) [noise=true , address=145.79.227.9:4365 , piece_id=7BK6E45EY6Y3SV6PY3A4EVCS42PC4HYLXU3J7MN4HXRLDEFUEFCA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:35.601406517 (7h36m34.152877008s) [noise=true , address=145.79.227.25:4619 , piece_id=23D3MCZRVNPI5KB43ROAHKUALDGXLZBZU5E6EIU6U2FY3YQOVPGA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:35.697082303 (7h36m34.248552774s) [noise=true , address=145.79.227.17:4970 , piece_id=5ZP5DDC3XAYQ5UFYJVQV6IZHB5KQQGNG54MTYW3UUYPN2IAK5QVQ, seg_pos={0 0}, upload=6714]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: expected piece hash; context deadline exceeded; EOF Nov 9 15:40:35.697252518 (7h36m34.248722969s) [noise=true , address=145.79.227.17:4972 , piece_id=AI3KVNUJKJUD33YVU2Z472R72SKQKRYQG65JPKFZ5WBJLKRLW6MA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:145.79.227.17:4972): protocol: expected piece hash; context deadline exceeded; EOF Nov 9 15:40:35.697302322 (7h36m34.248772773s) [noise=true , address=145.79.227.21:4474 , piece_id=J5XE2MTCEW4ZHQDDZ4KCVYFK5EX6RQZARKNY4QCH4HUHGJO3SXDQ, seg_pos={0 0}, upload=6714]: Done uploading piece. err: ecclient: upload failed (node:1rL2DnjSgHP36Md3bCgy5padqTAhv3iQCeURbLMuuuMBxtUNZH, address:145.79.227.21:4474): protocol: expected piece hash; context deadline exceeded; EOF Nov 9 15:40:35.697472546 (7h36m34.248943007s) [noise=true , address=145.79.227.9:4366 , piece_id=BT3GWQSXIMMSFPUYJC5EQ2LX2DS4BW5DAMW6ILLNVVRUUHZMOQUA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: expected piece hash; context deadline exceeded; EOF Nov 9 15:40:35.697514686 (7h36m34.248985147s) [seg_pos={0 0}, upload=6714]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 9 15:40:35.950211122 (7h36m34.501681583s) [upload=6714]: Done uploading segments... Nov 9 15:40:35.950416643 (7h36m34.501887104s) [upload=6714]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 9.28844971s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 8.696369363s trace: https://jaeger.dev.storj.io/trace/3846516a5cecd778 FAIL (2025-11-09 16:17:59 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 9 15:40:06.410809146 (7h36m4.962279607s) [upload=6714]: Starting upload Nov 9 15:40:06.411046298 (7h36m4.962516769s) [upload=6714]: Done starting upload Nov 9 15:40:06.411314729 (7h36m4.96278519s) [upload=6714]: Uploading segments... Nov 9 15:40:06.411410882 (7h36m4.962881383s) [upload=6714]: Got next segment. Inline: false Nov 9 15:40:06.411464574 (7h36m4.962935035s) [upload=6714]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 9 15:40:06.411475064 (7h36m4.962945555s) [upload=6714]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 9 15:40:06.664789367 (7h36m5.216259828s) [seg_pos={0 0}, upload=6714]: Begin upload segment... Nov 9 15:40:06.664851705 (7h36m5.216322166s) [seg_pos={0 0}, upload=6714]: Done begin upload segment. Nov 9 15:40:06.664891662 (7h36m5.216362123s) [noise=true , address=145.79.227.32:4835 , piece_id=EYIBNSVPQPKLKOBHKZI2VZHVCDKW3C3YJSFEWP4G62UTBXX2YRZA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.664996933 (7h36m5.216467394s) [noise=true , address=45.91.116.14:4190 , piece_id=XSIDZL5HMLZTQE7ZAPURAAUGQLPP3QOZK5BIE5PHRMZPSTN5APSA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.665698789 (7h36m5.21716925s) [noise=true , address=77.90.31.14:4662 , piece_id=HHSJDZPYKGCHURAZV46OQSGBU7D4TFPXREDBFPGTYF7OAUMT2X3A, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.666456111 (7h36m5.217926572s) [noise=true , address=145.79.227.25:4619 , piece_id=23D3MCZRVNPI5KB43ROAHKUALDGXLZBZU5E6EIU6U2FY3YQOVPGA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.666642225 (7h36m5.218112676s) [noise=true , address=145.79.227.17:4971 , piece_id=VTUI377UCF44ICILORHEKUO33ICRXM2FECO5EXS7RUDH4VGV7FJA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.667027539 (7h36m5.218498s) [noise=true , address=145.79.227.9:4366 , piece_id=BT3GWQSXIMMSFPUYJC5EQ2LX2DS4BW5DAMW6ILLNVVRUUHZMOQUA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.667041675 (7h36m5.218512126s) [noise=true , address=145.79.227.17:4972 , piece_id=AI3KVNUJKJUD33YVU2Z472R72SKQKRYQG65JPKFZ5WBJLKRLW6MA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.667502794 (7h36m5.218973255s) [noise=true , address=145.79.227.13:4803 , piece_id=V37DZKBC4CHRGISXEQHTQAGKZFNXQKWXSWBSMHNRQTSJ32WYI2OQ, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.667805441 (7h36m5.219275902s) [noise=true , address=145.79.227.9:4365 , piece_id=7BK6E45EY6Y3SV6PY3A4EVCS42PC4HYLXU3J7MN4HXRLDEFUEFCA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.667895652 (7h36m5.219366113s) [noise=true , address=145.79.227.17:4970 , piece_id=5ZP5DDC3XAYQ5UFYJVQV6IZHB5KQQGNG54MTYW3UUYPN2IAK5QVQ, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.668240909 (7h36m5.219711361s) [noise=true , address=77.90.31.17:4772 , piece_id=CT6BNOYZGDBUEFVGM2RDS4ICMWPYS6N4IOUDA5V7MJQ3IKELNW2Q, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.668671640 (7h36m5.220142101s) [noise=true , address=145.79.227.31:4797 , piece_id=VQEEGJTLJY2IMCDHL2GMVGYEX4VL34TMMXKDYNERFYK3HRV7TC6A, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.669093554 (7h36m5.220564015s) [noise=true , address=145.79.227.21:4474 , piece_id=J5XE2MTCEW4ZHQDDZ4KCVYFK5EX6RQZARKNY4QCH4HUHGJO3SXDQ, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.669510097 (7h36m5.220980568s) [noise=true , address=77.90.31.18:4807 , piece_id=HDJAJYXNZNZN3LVAQSYVJEOQ3MU5H3BXCCW3NA2GLKPT7ATZHSJQ, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.671936887 (7h36m5.223407348s) [noise=true , address=145.79.227.20:4439 , piece_id=PPOVWSRAU7Y4CZAKAXV6ZP74EEPHONUC35O6XM57VE47NVJL5WZA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.672494549 (7h36m5.22396502s) [noise=true , address=77.90.31.21:4877 , piece_id=3V3Z6GNF32Q66EOS66WV3AGJNA7IXT5NHVAPKNUGVYU3R46IKKKA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:08.438197970 (7h36m6.989668451s) [upload=6714]: Next returned nil segment Nov 9 15:40:08.438219050 (7h36m6.989689521s) [upload=6714]: Waiting for error group managing segments... Nov 9 15:40:08.829503696 (7h36m7.380974147s) [noise=true , address=77.90.31.18:4807 , piece_id=HDJAJYXNZNZN3LVAQSYVJEOQ3MU5H3BXCCW3NA2GLKPT7ATZHSJQ, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:08.846923116 (7h36m7.398393577s) [noise=true , address=45.91.116.14:4190 , piece_id=XSIDZL5HMLZTQE7ZAPURAAUGQLPP3QOZK5BIE5PHRMZPSTN5APSA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:08.867005608 (7h36m7.418476069s) [noise=true , address=77.90.31.21:4877 , piece_id=3V3Z6GNF32Q66EOS66WV3AGJNA7IXT5NHVAPKNUGVYU3R46IKKKA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:08.888352807 (7h36m7.439823268s) [noise=true , address=77.90.31.14:4662 , piece_id=HHSJDZPYKGCHURAZV46OQSGBU7D4TFPXREDBFPGTYF7OAUMT2X3A, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:08.904680649 (7h36m7.45615111s) [noise=true , address=77.90.31.17:4772 , piece_id=CT6BNOYZGDBUEFVGM2RDS4ICMWPYS6N4IOUDA5V7MJQ3IKELNW2Q, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:18.663082162 (7h36m17.214552623s) [noise=true , address=145.79.227.17:4971 , piece_id=VTUI377UCF44ICILORHEKUO33ICRXM2FECO5EXS7RUDH4VGV7FJA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:20.077194714 (7h36m18.628665185s) [noise=true , address=145.79.227.13:4803 , piece_id=V37DZKBC4CHRGISXEQHTQAGKZFNXQKWXSWBSMHNRQTSJ32WYI2OQ, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:28.517900114 (7h36m27.069370575s) [noise=true , address=145.79.227.32:4835 , piece_id=EYIBNSVPQPKLKOBHKZI2VZHVCDKW3C3YJSFEWP4G62UTBXX2YRZA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:28.567571990 (7h36m27.119042451s) [noise=true , address=145.79.227.31:4797 , piece_id=VQEEGJTLJY2IMCDHL2GMVGYEX4VL34TMMXKDYNERFYK3HRV7TC6A, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:33.651382593 (7h36m32.202853044s) [noise=true , address=145.79.227.20:4439 , piece_id=PPOVWSRAU7Y4CZAKAXV6ZP74EEPHONUC35O6XM57VE47NVJL5WZA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:34.842221635 (7h36m33.393692096s) [noise=true , address=145.79.227.9:4365 , piece_id=7BK6E45EY6Y3SV6PY3A4EVCS42PC4HYLXU3J7MN4HXRLDEFUEFCA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:35.601406517 (7h36m34.152877008s) [noise=true , address=145.79.227.25:4619 , piece_id=23D3MCZRVNPI5KB43ROAHKUALDGXLZBZU5E6EIU6U2FY3YQOVPGA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:35.697082303 (7h36m34.248552774s) [noise=true , address=145.79.227.17:4970 , piece_id=5ZP5DDC3XAYQ5UFYJVQV6IZHB5KQQGNG54MTYW3UUYPN2IAK5QVQ, seg_pos={0 0}, upload=6714]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: expected piece hash; context deadline exceeded; EOF Nov 9 15:40:35.697252518 (7h36m34.248722969s) [noise=true , address=145.79.227.17:4972 , piece_id=AI3KVNUJKJUD33YVU2Z472R72SKQKRYQG65JPKFZ5WBJLKRLW6MA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:145.79.227.17:4972): protocol: expected piece hash; context deadline exceeded; EOF Nov 9 15:40:35.697302322 (7h36m34.248772773s) [noise=true , address=145.79.227.21:4474 , piece_id=J5XE2MTCEW4ZHQDDZ4KCVYFK5EX6RQZARKNY4QCH4HUHGJO3SXDQ, seg_pos={0 0}, upload=6714]: Done uploading piece. err: ecclient: upload failed (node:1rL2DnjSgHP36Md3bCgy5padqTAhv3iQCeURbLMuuuMBxtUNZH, address:145.79.227.21:4474): protocol: expected piece hash; context deadline exceeded; EOF Nov 9 15:40:35.697472546 (7h36m34.248943007s) [noise=true , address=145.79.227.9:4366 , piece_id=BT3GWQSXIMMSFPUYJC5EQ2LX2DS4BW5DAMW6ILLNVVRUUHZMOQUA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: expected piece hash; context deadline exceeded; EOF Nov 9 15:40:35.697514686 (7h36m34.248985147s) [seg_pos={0 0}, upload=6714]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 9 15:40:35.950211122 (7h36m34.501681583s) [upload=6714]: Done uploading segments... Nov 9 15:40:35.950416643 (7h36m34.501887104s) [upload=6714]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 8.69640956s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 9.534400382s trace: https://jaeger.dev.storj.io/trace/3846516a5cecd778 FAIL (2025-11-09 16:16:50 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 9 15:40:06.410809146 (7h36m4.962279607s) [upload=6714]: Starting upload Nov 9 15:40:06.411046298 (7h36m4.962516769s) [upload=6714]: Done starting upload Nov 9 15:40:06.411314729 (7h36m4.96278519s) [upload=6714]: Uploading segments... Nov 9 15:40:06.411410882 (7h36m4.962881383s) [upload=6714]: Got next segment. Inline: false Nov 9 15:40:06.411464574 (7h36m4.962935035s) [upload=6714]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 9 15:40:06.411475064 (7h36m4.962945555s) [upload=6714]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 9 15:40:06.664789367 (7h36m5.216259828s) [seg_pos={0 0}, upload=6714]: Begin upload segment... Nov 9 15:40:06.664851705 (7h36m5.216322166s) [seg_pos={0 0}, upload=6714]: Done begin upload segment. Nov 9 15:40:06.664891662 (7h36m5.216362123s) [noise=true , address=145.79.227.32:4835 , piece_id=EYIBNSVPQPKLKOBHKZI2VZHVCDKW3C3YJSFEWP4G62UTBXX2YRZA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.664996933 (7h36m5.216467394s) [noise=true , address=45.91.116.14:4190 , piece_id=XSIDZL5HMLZTQE7ZAPURAAUGQLPP3QOZK5BIE5PHRMZPSTN5APSA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.665698789 (7h36m5.21716925s) [noise=true , address=77.90.31.14:4662 , piece_id=HHSJDZPYKGCHURAZV46OQSGBU7D4TFPXREDBFPGTYF7OAUMT2X3A, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.666456111 (7h36m5.217926572s) [noise=true , address=145.79.227.25:4619 , piece_id=23D3MCZRVNPI5KB43ROAHKUALDGXLZBZU5E6EIU6U2FY3YQOVPGA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.666642225 (7h36m5.218112676s) [noise=true , address=145.79.227.17:4971 , piece_id=VTUI377UCF44ICILORHEKUO33ICRXM2FECO5EXS7RUDH4VGV7FJA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.667027539 (7h36m5.218498s) [noise=true , address=145.79.227.9:4366 , piece_id=BT3GWQSXIMMSFPUYJC5EQ2LX2DS4BW5DAMW6ILLNVVRUUHZMOQUA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.667041675 (7h36m5.218512126s) [noise=true , address=145.79.227.17:4972 , piece_id=AI3KVNUJKJUD33YVU2Z472R72SKQKRYQG65JPKFZ5WBJLKRLW6MA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.667502794 (7h36m5.218973255s) [noise=true , address=145.79.227.13:4803 , piece_id=V37DZKBC4CHRGISXEQHTQAGKZFNXQKWXSWBSMHNRQTSJ32WYI2OQ, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.667805441 (7h36m5.219275902s) [noise=true , address=145.79.227.9:4365 , piece_id=7BK6E45EY6Y3SV6PY3A4EVCS42PC4HYLXU3J7MN4HXRLDEFUEFCA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.667895652 (7h36m5.219366113s) [noise=true , address=145.79.227.17:4970 , piece_id=5ZP5DDC3XAYQ5UFYJVQV6IZHB5KQQGNG54MTYW3UUYPN2IAK5QVQ, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.668240909 (7h36m5.219711361s) [noise=true , address=77.90.31.17:4772 , piece_id=CT6BNOYZGDBUEFVGM2RDS4ICMWPYS6N4IOUDA5V7MJQ3IKELNW2Q, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.668671640 (7h36m5.220142101s) [noise=true , address=145.79.227.31:4797 , piece_id=VQEEGJTLJY2IMCDHL2GMVGYEX4VL34TMMXKDYNERFYK3HRV7TC6A, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.669093554 (7h36m5.220564015s) [noise=true , address=145.79.227.21:4474 , piece_id=J5XE2MTCEW4ZHQDDZ4KCVYFK5EX6RQZARKNY4QCH4HUHGJO3SXDQ, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.669510097 (7h36m5.220980568s) [noise=true , address=77.90.31.18:4807 , piece_id=HDJAJYXNZNZN3LVAQSYVJEOQ3MU5H3BXCCW3NA2GLKPT7ATZHSJQ, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.671936887 (7h36m5.223407348s) [noise=true , address=145.79.227.20:4439 , piece_id=PPOVWSRAU7Y4CZAKAXV6ZP74EEPHONUC35O6XM57VE47NVJL5WZA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.672494549 (7h36m5.22396502s) [noise=true , address=77.90.31.21:4877 , piece_id=3V3Z6GNF32Q66EOS66WV3AGJNA7IXT5NHVAPKNUGVYU3R46IKKKA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:08.438197970 (7h36m6.989668451s) [upload=6714]: Next returned nil segment Nov 9 15:40:08.438219050 (7h36m6.989689521s) [upload=6714]: Waiting for error group managing segments... Nov 9 15:40:08.829503696 (7h36m7.380974147s) [noise=true , address=77.90.31.18:4807 , piece_id=HDJAJYXNZNZN3LVAQSYVJEOQ3MU5H3BXCCW3NA2GLKPT7ATZHSJQ, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:08.846923116 (7h36m7.398393577s) [noise=true , address=45.91.116.14:4190 , piece_id=XSIDZL5HMLZTQE7ZAPURAAUGQLPP3QOZK5BIE5PHRMZPSTN5APSA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:08.867005608 (7h36m7.418476069s) [noise=true , address=77.90.31.21:4877 , piece_id=3V3Z6GNF32Q66EOS66WV3AGJNA7IXT5NHVAPKNUGVYU3R46IKKKA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:08.888352807 (7h36m7.439823268s) [noise=true , address=77.90.31.14:4662 , piece_id=HHSJDZPYKGCHURAZV46OQSGBU7D4TFPXREDBFPGTYF7OAUMT2X3A, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:08.904680649 (7h36m7.45615111s) [noise=true , address=77.90.31.17:4772 , piece_id=CT6BNOYZGDBUEFVGM2RDS4ICMWPYS6N4IOUDA5V7MJQ3IKELNW2Q, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:18.663082162 (7h36m17.214552623s) [noise=true , address=145.79.227.17:4971 , piece_id=VTUI377UCF44ICILORHEKUO33ICRXM2FECO5EXS7RUDH4VGV7FJA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:20.077194714 (7h36m18.628665185s) [noise=true , address=145.79.227.13:4803 , piece_id=V37DZKBC4CHRGISXEQHTQAGKZFNXQKWXSWBSMHNRQTSJ32WYI2OQ, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:28.517900114 (7h36m27.069370575s) [noise=true , address=145.79.227.32:4835 , piece_id=EYIBNSVPQPKLKOBHKZI2VZHVCDKW3C3YJSFEWP4G62UTBXX2YRZA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:28.567571990 (7h36m27.119042451s) [noise=true , address=145.79.227.31:4797 , piece_id=VQEEGJTLJY2IMCDHL2GMVGYEX4VL34TMMXKDYNERFYK3HRV7TC6A, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:33.651382593 (7h36m32.202853044s) [noise=true , address=145.79.227.20:4439 , piece_id=PPOVWSRAU7Y4CZAKAXV6ZP74EEPHONUC35O6XM57VE47NVJL5WZA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:34.842221635 (7h36m33.393692096s) [noise=true , address=145.79.227.9:4365 , piece_id=7BK6E45EY6Y3SV6PY3A4EVCS42PC4HYLXU3J7MN4HXRLDEFUEFCA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:35.601406517 (7h36m34.152877008s) [noise=true , address=145.79.227.25:4619 , piece_id=23D3MCZRVNPI5KB43ROAHKUALDGXLZBZU5E6EIU6U2FY3YQOVPGA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:35.697082303 (7h36m34.248552774s) [noise=true , address=145.79.227.17:4970 , piece_id=5ZP5DDC3XAYQ5UFYJVQV6IZHB5KQQGNG54MTYW3UUYPN2IAK5QVQ, seg_pos={0 0}, upload=6714]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: expected piece hash; context deadline exceeded; EOF Nov 9 15:40:35.697252518 (7h36m34.248722969s) [noise=true , address=145.79.227.17:4972 , piece_id=AI3KVNUJKJUD33YVU2Z472R72SKQKRYQG65JPKFZ5WBJLKRLW6MA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:145.79.227.17:4972): protocol: expected piece hash; context deadline exceeded; EOF Nov 9 15:40:35.697302322 (7h36m34.248772773s) [noise=true , address=145.79.227.21:4474 , piece_id=J5XE2MTCEW4ZHQDDZ4KCVYFK5EX6RQZARKNY4QCH4HUHGJO3SXDQ, seg_pos={0 0}, upload=6714]: Done uploading piece. err: ecclient: upload failed (node:1rL2DnjSgHP36Md3bCgy5padqTAhv3iQCeURbLMuuuMBxtUNZH, address:145.79.227.21:4474): protocol: expected piece hash; context deadline exceeded; EOF Nov 9 15:40:35.697472546 (7h36m34.248943007s) [noise=true , address=145.79.227.9:4366 , piece_id=BT3GWQSXIMMSFPUYJC5EQ2LX2DS4BW5DAMW6ILLNVVRUUHZMOQUA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: expected piece hash; context deadline exceeded; EOF Nov 9 15:40:35.697514686 (7h36m34.248985147s) [seg_pos={0 0}, upload=6714]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 9 15:40:35.950211122 (7h36m34.501681583s) [upload=6714]: Done uploading segments... Nov 9 15:40:35.950416643 (7h36m34.501887104s) [upload=6714]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 9.534433848s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 9.542953782s trace: https://jaeger.dev.storj.io/trace/3846516a5cecd778 FAIL (2025-11-09 16:15:56 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 9 15:40:06.410809146 (7h36m4.962279607s) [upload=6714]: Starting upload Nov 9 15:40:06.411046298 (7h36m4.962516769s) [upload=6714]: Done starting upload Nov 9 15:40:06.411314729 (7h36m4.96278519s) [upload=6714]: Uploading segments... Nov 9 15:40:06.411410882 (7h36m4.962881383s) [upload=6714]: Got next segment. Inline: false Nov 9 15:40:06.411464574 (7h36m4.962935035s) [upload=6714]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 9 15:40:06.411475064 (7h36m4.962945555s) [upload=6714]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 9 15:40:06.664789367 (7h36m5.216259828s) [seg_pos={0 0}, upload=6714]: Begin upload segment... Nov 9 15:40:06.664851705 (7h36m5.216322166s) [seg_pos={0 0}, upload=6714]: Done begin upload segment. Nov 9 15:40:06.664891662 (7h36m5.216362123s) [noise=true , address=145.79.227.32:4835 , piece_id=EYIBNSVPQPKLKOBHKZI2VZHVCDKW3C3YJSFEWP4G62UTBXX2YRZA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.664996933 (7h36m5.216467394s) [noise=true , address=45.91.116.14:4190 , piece_id=XSIDZL5HMLZTQE7ZAPURAAUGQLPP3QOZK5BIE5PHRMZPSTN5APSA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.665698789 (7h36m5.21716925s) [noise=true , address=77.90.31.14:4662 , piece_id=HHSJDZPYKGCHURAZV46OQSGBU7D4TFPXREDBFPGTYF7OAUMT2X3A, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.666456111 (7h36m5.217926572s) [noise=true , address=145.79.227.25:4619 , piece_id=23D3MCZRVNPI5KB43ROAHKUALDGXLZBZU5E6EIU6U2FY3YQOVPGA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.666642225 (7h36m5.218112676s) [noise=true , address=145.79.227.17:4971 , piece_id=VTUI377UCF44ICILORHEKUO33ICRXM2FECO5EXS7RUDH4VGV7FJA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.667027539 (7h36m5.218498s) [noise=true , address=145.79.227.9:4366 , piece_id=BT3GWQSXIMMSFPUYJC5EQ2LX2DS4BW5DAMW6ILLNVVRUUHZMOQUA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.667041675 (7h36m5.218512126s) [noise=true , address=145.79.227.17:4972 , piece_id=AI3KVNUJKJUD33YVU2Z472R72SKQKRYQG65JPKFZ5WBJLKRLW6MA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.667502794 (7h36m5.218973255s) [noise=true , address=145.79.227.13:4803 , piece_id=V37DZKBC4CHRGISXEQHTQAGKZFNXQKWXSWBSMHNRQTSJ32WYI2OQ, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.667805441 (7h36m5.219275902s) [noise=true , address=145.79.227.9:4365 , piece_id=7BK6E45EY6Y3SV6PY3A4EVCS42PC4HYLXU3J7MN4HXRLDEFUEFCA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.667895652 (7h36m5.219366113s) [noise=true , address=145.79.227.17:4970 , piece_id=5ZP5DDC3XAYQ5UFYJVQV6IZHB5KQQGNG54MTYW3UUYPN2IAK5QVQ, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.668240909 (7h36m5.219711361s) [noise=true , address=77.90.31.17:4772 , piece_id=CT6BNOYZGDBUEFVGM2RDS4ICMWPYS6N4IOUDA5V7MJQ3IKELNW2Q, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.668671640 (7h36m5.220142101s) [noise=true , address=145.79.227.31:4797 , piece_id=VQEEGJTLJY2IMCDHL2GMVGYEX4VL34TMMXKDYNERFYK3HRV7TC6A, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.669093554 (7h36m5.220564015s) [noise=true , address=145.79.227.21:4474 , piece_id=J5XE2MTCEW4ZHQDDZ4KCVYFK5EX6RQZARKNY4QCH4HUHGJO3SXDQ, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.669510097 (7h36m5.220980568s) [noise=true , address=77.90.31.18:4807 , piece_id=HDJAJYXNZNZN3LVAQSYVJEOQ3MU5H3BXCCW3NA2GLKPT7ATZHSJQ, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.671936887 (7h36m5.223407348s) [noise=true , address=145.79.227.20:4439 , piece_id=PPOVWSRAU7Y4CZAKAXV6ZP74EEPHONUC35O6XM57VE47NVJL5WZA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.672494549 (7h36m5.22396502s) [noise=true , address=77.90.31.21:4877 , piece_id=3V3Z6GNF32Q66EOS66WV3AGJNA7IXT5NHVAPKNUGVYU3R46IKKKA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:08.438197970 (7h36m6.989668451s) [upload=6714]: Next returned nil segment Nov 9 15:40:08.438219050 (7h36m6.989689521s) [upload=6714]: Waiting for error group managing segments... Nov 9 15:40:08.829503696 (7h36m7.380974147s) [noise=true , address=77.90.31.18:4807 , piece_id=HDJAJYXNZNZN3LVAQSYVJEOQ3MU5H3BXCCW3NA2GLKPT7ATZHSJQ, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:08.846923116 (7h36m7.398393577s) [noise=true , address=45.91.116.14:4190 , piece_id=XSIDZL5HMLZTQE7ZAPURAAUGQLPP3QOZK5BIE5PHRMZPSTN5APSA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:08.867005608 (7h36m7.418476069s) [noise=true , address=77.90.31.21:4877 , piece_id=3V3Z6GNF32Q66EOS66WV3AGJNA7IXT5NHVAPKNUGVYU3R46IKKKA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:08.888352807 (7h36m7.439823268s) [noise=true , address=77.90.31.14:4662 , piece_id=HHSJDZPYKGCHURAZV46OQSGBU7D4TFPXREDBFPGTYF7OAUMT2X3A, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:08.904680649 (7h36m7.45615111s) [noise=true , address=77.90.31.17:4772 , piece_id=CT6BNOYZGDBUEFVGM2RDS4ICMWPYS6N4IOUDA5V7MJQ3IKELNW2Q, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:18.663082162 (7h36m17.214552623s) [noise=true , address=145.79.227.17:4971 , piece_id=VTUI377UCF44ICILORHEKUO33ICRXM2FECO5EXS7RUDH4VGV7FJA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:20.077194714 (7h36m18.628665185s) [noise=true , address=145.79.227.13:4803 , piece_id=V37DZKBC4CHRGISXEQHTQAGKZFNXQKWXSWBSMHNRQTSJ32WYI2OQ, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:28.517900114 (7h36m27.069370575s) [noise=true , address=145.79.227.32:4835 , piece_id=EYIBNSVPQPKLKOBHKZI2VZHVCDKW3C3YJSFEWP4G62UTBXX2YRZA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:28.567571990 (7h36m27.119042451s) [noise=true , address=145.79.227.31:4797 , piece_id=VQEEGJTLJY2IMCDHL2GMVGYEX4VL34TMMXKDYNERFYK3HRV7TC6A, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:33.651382593 (7h36m32.202853044s) [noise=true , address=145.79.227.20:4439 , piece_id=PPOVWSRAU7Y4CZAKAXV6ZP74EEPHONUC35O6XM57VE47NVJL5WZA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:34.842221635 (7h36m33.393692096s) [noise=true , address=145.79.227.9:4365 , piece_id=7BK6E45EY6Y3SV6PY3A4EVCS42PC4HYLXU3J7MN4HXRLDEFUEFCA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:35.601406517 (7h36m34.152877008s) [noise=true , address=145.79.227.25:4619 , piece_id=23D3MCZRVNPI5KB43ROAHKUALDGXLZBZU5E6EIU6U2FY3YQOVPGA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:35.697082303 (7h36m34.248552774s) [noise=true , address=145.79.227.17:4970 , piece_id=5ZP5DDC3XAYQ5UFYJVQV6IZHB5KQQGNG54MTYW3UUYPN2IAK5QVQ, seg_pos={0 0}, upload=6714]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: expected piece hash; context deadline exceeded; EOF Nov 9 15:40:35.697252518 (7h36m34.248722969s) [noise=true , address=145.79.227.17:4972 , piece_id=AI3KVNUJKJUD33YVU2Z472R72SKQKRYQG65JPKFZ5WBJLKRLW6MA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:145.79.227.17:4972): protocol: expected piece hash; context deadline exceeded; EOF Nov 9 15:40:35.697302322 (7h36m34.248772773s) [noise=true , address=145.79.227.21:4474 , piece_id=J5XE2MTCEW4ZHQDDZ4KCVYFK5EX6RQZARKNY4QCH4HUHGJO3SXDQ, seg_pos={0 0}, upload=6714]: Done uploading piece. err: ecclient: upload failed (node:1rL2DnjSgHP36Md3bCgy5padqTAhv3iQCeURbLMuuuMBxtUNZH, address:145.79.227.21:4474): protocol: expected piece hash; context deadline exceeded; EOF Nov 9 15:40:35.697472546 (7h36m34.248943007s) [noise=true , address=145.79.227.9:4366 , piece_id=BT3GWQSXIMMSFPUYJC5EQ2LX2DS4BW5DAMW6ILLNVVRUUHZMOQUA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: expected piece hash; context deadline exceeded; EOF Nov 9 15:40:35.697514686 (7h36m34.248985147s) [seg_pos={0 0}, upload=6714]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 9 15:40:35.950211122 (7h36m34.501681583s) [upload=6714]: Done uploading segments... Nov 9 15:40:35.950416643 (7h36m34.501887104s) [upload=6714]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 9.54301048s core uplink us-select-1

Orbiter Check History

checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect)

checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect)

checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect)

checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect)

checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect)

checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect)

checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect)

checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect)

checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect)

checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect)

checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect)

checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)