Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 7.741522403s trace: https://jaeger.dev.storj.io/trace/78ef362f5bf25321 FAIL (2025-11-08 17:28:10 UTC) orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Duration: 7.741553892s core uplink us-select-1
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 701.624437ms target: https://us1.storj.io/login trace: https://jaeger.dev.storj.io/trace/241dc17381c83708 FAIL (2025-11-08 17:28:05 UTC) orbiter warning: Get "https://us1.storj.io/login": dial tcp 35.212.10.183:443: i/o timeout Duration: 701.713015ms core satellite us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 6.329211634s trace: https://jaeger.dev.storj.io/trace/6f3df2181993dff FAIL (2025-11-08 17:28:04 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 6.329247473s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 2.145458123s trace: https://jaeger.dev.storj.io/trace/610e55e4abb6caeb FAIL (2025-11-08 17:27:55 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 2.145498521s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 8.688893089s trace: https://jaeger.dev.storj.io/trace/78ef362f5bf25321 FAIL (2025-11-08 17:27:53 UTC) orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Duration: 8.688917224s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 1.992815792s trace: https://jaeger.dev.storj.io/trace/94b803dc07bcb3c FAIL (2025-11-08 17:27:47 UTC) orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Duration: 1.992862971s core uplink us-select-1
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 696.433813ms target: https://us1.storj.io/login trace: https://jaeger.dev.storj.io/trace/241dc17381c83708 FAIL (2025-11-08 17:27:36 UTC) orbiter warning: Get "https://us1.storj.io/login": dial tcp 35.212.10.183:443: i/o timeout Duration: 696.466925ms core satellite us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 2.006406501s trace: https://jaeger.dev.storj.io/trace/610e55e4abb6caeb FAIL (2025-11-08 17:27:30 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 2.006428073s core uplink us1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 6.081665424s trace: https://jaeger.dev.storj.io/trace/3f5ea518e559e87b FAIL (2025-11-08 17:27:29 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.081695943s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 8.438316815s trace: https://jaeger.dev.storj.io/trace/78ef362f5bf25321 FAIL (2025-11-08 17:27:27 UTC) orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Duration: 8.438341582s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 7.033267477s trace: https://jaeger.dev.storj.io/trace/65365d1628b4a08a FAIL (2025-11-08 17:27:24 UTC) orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Duration: 7.033285751s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 5.968976498s trace: https://jaeger.dev.storj.io/trace/6f3df2181993dff FAIL (2025-11-08 17:27:18 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 5.969009821s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 1m0.001079018s trace: https://jaeger.dev.storj.io/trace/65365d1628b4a08a FAIL (2025-11-08 17:27:17 UTC) orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Duration: 1m0.001106562s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 30.001403375s target: https://us1.storj.io/login trace: https://jaeger.dev.storj.io/trace/241dc17381c83708 FAIL (2025-11-08 17:27:09 UTC) orbiter warning: Get "https://us1.storj.io/login": dial tcp 35.212.10.183:443: i/o timeout Duration: 30.001431658s core satellite us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 2.218023825s trace: https://jaeger.dev.storj.io/trace/94b803dc07bcb3c FAIL (2025-11-08 17:27:08 UTC) orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Duration: 2.218086496s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 30.001546274s trace: https://jaeger.dev.storj.io/trace/610e55e4abb6caeb FAIL (2025-11-08 17:27:04 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 30.001575691s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 5.839609119s trace: https://jaeger.dev.storj.io/trace/6f3df2181993dff FAIL (2025-11-08 17:26:55 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 5.839629819s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 2.185376326s trace: https://jaeger.dev.storj.io/trace/94b803dc07bcb3c FAIL (2025-11-08 17:26:51 UTC) orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Duration: 2.1853941s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 30.001429475s trace: https://jaeger.dev.storj.io/trace/6f3df2181993dff FAIL (2025-11-08 17:26:49 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 30.001453621s core uplink us1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 6.316062495s trace: https://jaeger.dev.storj.io/trace/3f5ea518e559e87b FAIL (2025-11-08 17:26:32 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.316085318s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 7.995655107s trace: https://jaeger.dev.storj.io/trace/78ef362f5bf25321 FAIL (2025-11-08 17:26:27 UTC) orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Duration: 7.995690213s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 2.071791059s trace: https://jaeger.dev.storj.io/trace/94b803dc07bcb3c FAIL (2025-11-08 17:26:14 UTC) orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Duration: 2.071818311s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 7.45426256s trace: https://jaeger.dev.storj.io/trace/78ef362f5bf25321 FAIL (2025-11-08 17:25:38 UTC) orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Duration: 7.454298097s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 30.001509595s trace: https://jaeger.dev.storj.io/trace/78ef362f5bf25321 FAIL (2025-11-08 17:25:30 UTC) orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Duration: 30.001538441s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 6.243057079s trace: https://jaeger.dev.storj.io/trace/3f5ea518e559e87b FAIL (2025-11-08 17:25:17 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.243089078s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 2.240504514s trace: https://jaeger.dev.storj.io/trace/94b803dc07bcb3c FAIL (2025-11-08 17:25:00 UTC) orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Duration: 2.240532878s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 30.000888288s trace: https://jaeger.dev.storj.io/trace/94b803dc07bcb3c FAIL (2025-11-08 17:24:45 UTC) orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Duration: 30.00091546s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 30.003570636s trace: https://jaeger.dev.storj.io/trace/3f5ea518e559e87b FAIL (2025-11-08 17:24:44 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.003586135s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 7.468295688s trace: https://jaeger.dev.storj.io/trace/3f377bb0d96d25c FAIL (2025-11-08 15:10:51 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Nov 8 14:26:07.882330547 (218h45m39.5781793s) [upload=192678]: Starting upload Nov 8 14:26:07.882433824 (218h45m39.578282567s) [upload=192678]: Done starting upload Nov 8 14:26:07.882492927 (218h45m39.57834167s) [upload=192678]: Uploading segments... Nov 8 14:26:07.882645367 (218h45m39.57849411s) [upload=192678]: Got next segment. Inline: false Nov 8 14:26:07.882698468 (218h45m39.578547221s) [upload=192678]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 8 14:26:07.882704229 (218h45m39.578552982s) [upload=192678]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 8 14:26:08.131091114 (218h45m39.826939897s) [seg_pos={0 0}, upload=192678]: Begin upload segment... Nov 8 14:26:08.131165496 (218h45m39.827014239s) [seg_pos={0 0}, upload=192678]: Done begin upload segment. Nov 8 14:26:08.131220160 (218h45m39.827068913s) [noise=true , address=145.79.227.16:4911 , piece_id=QKITSRMSCHOZLCQ7X6GARTE3GS24G3CZFXJIBKDM3VWOVQLMLEHA, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.131304631 (218h45m39.827153384s) [noise=true , address=145.79.227.27:4689 , piece_id=JEEKOLHURPGNFSR4YCWL3UD6BU2MT7VW4S7BU3YKE7BJSUEYLPGQ, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.132190408 (218h45m39.828039161s) [noise=true , address=145.79.227.7:4295 , piece_id=5OY4NKICVFGAWR7JPOOYIYRXR442IIUDDWTN3MT6KAR4ZHXTKPNQ, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.132569360 (218h45m39.828418113s) [noise=true , address=145.79.227.31:4798 , piece_id=5Q7NYNXM2IMV7YUCB464FQMBI4L476Y6VKPPOPW7RKNCC5XFDJ2A, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.133276446 (218h45m39.829125199s) [noise=true , address=145.79.227.26:4653 , piece_id=UIV3WFYLH4JZSIIHZUBL56THDPAYK7DMRUQG5TVTHHEF3XGN6NKQ, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.134167893 (218h45m39.830016646s) [noise=true , address=145.79.227.26:4655 , piece_id=GP4F5L6HH6S4ZEYWS7XRG5KJSYFGOX5JXQBTQEOXFDFSNUBHZJ6A, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.134314192 (218h45m39.830162945s) [noise=true , address=145.79.227.20:4437 , piece_id=G5RTYGERAVOTQDZGCG6NEKRCQAUOVJNYXO4LEBJ7XHJZ6FDXTULQ, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.135016950 (218h45m39.830865703s) [noise=true , address=145.79.227.10:4402 , piece_id=IRNQEX73D4AEXNYXEWWKCKEMYFC3YXIPX6HSDYYBNDKBM77ROBEA, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.135235305 (218h45m39.831084059s) [noise=true , address=145.79.227.22:4509 , piece_id=6RJJCCT2Z5U3JOWWOBLRIDK5CTWKXU7NR4OIREAQ2JDE2KYDJHOQ, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.135440236 (218h45m39.831288979s) [noise=true , address=145.79.227.2:4113 , piece_id=Q5E3IXSHGFKIFG7EFAPW2XWYP63TO3BC5C7PZJDPXXD2UKDQDJLA, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.136611887 (218h45m39.83246064s) [noise=true , address=77.90.31.15:4699 , piece_id=Z4PXJFTM7XMK62ORTMHCWEJ7OQF6PIYQCMAWJC3DT5RZ7CEH4KXA, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.137027559 (218h45m39.832876312s) [noise=true , address=145.79.227.8:4329 , piece_id=JJFSJ7USPZFTPXVUZVLMXRAHYLARBQZLWWU7ITXFNESYVVL4QBMQ, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.137896133 (218h45m39.833744886s) [noise=true , address=77.90.31.17:4771 , piece_id=LUZ4KXA4YUVTQYKXVEPTEXZ4MFQ5IWFTY3C2LUUVQ3N6DOVO6ZXA, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.139107940 (218h45m39.834956693s) [noise=true , address=145.79.227.21:4475 , piece_id=6F7IZNQLORMTKUF576FJA6L7FBG66INEUFPQYHL77WLC6DYF32GQ, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.139872657 (218h45m39.83572141s) [noise=true , address=145.79.227.9:4366 , piece_id=3BINODFKOOG7BYULCICW6S64YZOU5D2Q5MFZYGPSSA34XWM762BQ, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.140005960 (218h45m39.835854713s) [noise=true , address=77.90.31.16:4734 , piece_id=EKYBQDVNAYTGEPWU443RFLUX7OKCTWBPXA5UMCLZ22M2PBXO3E3Q, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:10.015794232 (218h45m41.711642985s) [upload=192678]: Next returned nil segment Nov 8 14:26:10.015816414 (218h45m41.711665167s) [upload=192678]: Waiting for error group managing segments... Nov 8 14:26:10.314091526 (218h45m42.009940279s) [noise=true , address=77.90.31.16:4734 , piece_id=EKYBQDVNAYTGEPWU443RFLUX7OKCTWBPXA5UMCLZ22M2PBXO3E3Q, seg_pos={0 0}, upload=192678]: Done uploading piece. err: <nil> Nov 8 14:26:10.388439672 (218h45m42.084288465s) [noise=true , address=77.90.31.15:4699 , piece_id=Z4PXJFTM7XMK62ORTMHCWEJ7OQF6PIYQCMAWJC3DT5RZ7CEH4KXA, seg_pos={0 0}, upload=192678]: Done uploading piece. err: <nil> Nov 8 14:26:11.119863095 (218h45m42.815711838s) [noise=true , address=77.90.31.17:4771 , piece_id=LUZ4KXA4YUVTQYKXVEPTEXZ4MFQ5IWFTY3C2LUUVQ3N6DOVO6ZXA, seg_pos={0 0}, upload=192678]: Done uploading piece. err: <nil> Nov 8 14:26:14.806769255 (218h45m46.502618008s) [noise=true , address=145.79.227.2:4113 , piece_id=Q5E3IXSHGFKIFG7EFAPW2XWYP63TO3BC5C7PZJDPXXD2UKDQDJLA, seg_pos={0 0}, upload=192678]: Done uploading piece. err: <nil> Nov 8 14:26:15.197752745 (218h45m46.893601498s) [noise=true , address=145.79.227.26:4655 , piece_id=GP4F5L6HH6S4ZEYWS7XRG5KJSYFGOX5JXQBTQEOXFDFSNUBHZJ6A, seg_pos={0 0}, upload=192678]: Done uploading piece. err: <nil> Nov 8 14:26:17.425396997 (218h45m49.12124575s) [noise=true , address=145.79.227.22:4509 , piece_id=6RJJCCT2Z5U3JOWWOBLRIDK5CTWKXU7NR4OIREAQ2JDE2KYDJHOQ, seg_pos={0 0}, upload=192678]: Done uploading piece. err: <nil> Nov 8 14:26:21.856283285 (218h45m53.552132048s) [noise=true , address=145.79.227.31:4798 , piece_id=5Q7NYNXM2IMV7YUCB464FQMBI4L476Y6VKPPOPW7RKNCC5XFDJ2A, seg_pos={0 0}, upload=192678]: Done uploading piece. err: <nil> Nov 8 14:26:23.486413928 (218h45m55.182262681s) [noise=true , address=145.79.227.20:4437 , piece_id=G5RTYGERAVOTQDZGCG6NEKRCQAUOVJNYXO4LEBJ7XHJZ6FDXTULQ, seg_pos={0 0}, upload=192678]: Done uploading piece. err: <nil> Nov 8 14:26:24.659870720 (218h45m56.355719473s) [noise=true , address=145.79.227.26:4653 , piece_id=UIV3WFYLH4JZSIIHZUBL56THDPAYK7DMRUQG5TVTHHEF3XGN6NKQ, seg_pos={0 0}, upload=192678]: Done uploading piece. err: <nil> Nov 8 14:26:34.701060537 (218h46m6.3969093s) [noise=true , address=145.79.227.8:4329 , piece_id=JJFSJ7USPZFTPXVUZVLMXRAHYLARBQZLWWU7ITXFNESYVVL4QBMQ, seg_pos={0 0}, upload=192678]: Done uploading piece. err: <nil> Nov 8 14:26:37.183628509 (218h46m8.879477262s) [noise=true , address=145.79.227.10:4402 , piece_id=IRNQEX73D4AEXNYXEWWKCKEMYFC3YXIPX6HSDYYBNDKBM77ROBEA, seg_pos={0 0}, upload=192678]: Done uploading piece. err: <nil> Nov 8 14:26:37.186238899 (218h46m8.882087662s) [noise=true , address=145.79.227.16:4911 , piece_id=QKITSRMSCHOZLCQ7X6GARTE3GS24G3CZFXJIBKDM3VWOVQLMLEHA, seg_pos={0 0}, upload=192678]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:145.79.227.16:4911): protocol: write tcp 188.166.225.170:56440->145.79.227.16:4911: use of closed network connection; write tcp 188.166.225.170:56440->145.79.227.16:4911: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56440->145.79.227.16:4911: use of closed network connection Nov 8 14:26:37.186462625 (218h46m8.882311378s) [noise=true , address=145.79.227.21:4475 , piece_id=6F7IZNQLORMTKUF576FJA6L7FBG66INEUFPQYHL77WLC6DYF32GQ, seg_pos={0 0}, upload=192678]: Done uploading piece. err: ecclient: upload failed (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk, address:145.79.227.21:4475): protocol: expected piece hash; context deadline exceeded; EOF Nov 8 14:26:37.186739442 (218h46m8.882588195s) [noise=true , address=145.79.227.27:4689 , piece_id=JEEKOLHURPGNFSR4YCWL3UD6BU2MT7VW4S7BU3YKE7BJSUEYLPGQ, seg_pos={0 0}, upload=192678]: Done uploading piece. err: ecclient: upload failed (node:1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, address:145.79.227.27:4689): protocol: write tcp 188.166.225.170:37312->145.79.227.27:4689: use of closed network connection; write tcp 188.166.225.170:37312->145.79.227.27:4689: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37312->145.79.227.27:4689: use of closed network connection Nov 8 14:26:37.186915056 (218h46m8.88276381s) [noise=true , address=145.79.227.7:4295 , piece_id=5OY4NKICVFGAWR7JPOOYIYRXR442IIUDDWTN3MT6KAR4ZHXTKPNQ, seg_pos={0 0}, upload=192678]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:145.79.227.7:4295): protocol: expected piece hash; context deadline exceeded; EOF Nov 8 14:26:37.187072807 (218h46m8.88292156s) [noise=true , address=145.79.227.9:4366 , piece_id=3BINODFKOOG7BYULCICW6S64YZOU5D2Q5MFZYGPSSA34XWM762BQ, seg_pos={0 0}, upload=192678]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: expected piece hash; context deadline exceeded; EOF Nov 8 14:26:37.187107163 (218h46m8.882955916s) [seg_pos={0 0}, upload=192678]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 8 14:26:37.433482306 (218h46m9.129331059s) [upload=192678]: Done uploading segments... Nov 8 14:26:37.433526460 (218h46m9.129375213s) [upload=192678]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 7.468316328s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 12.652893212s trace: https://jaeger.dev.storj.io/trace/3f377bb0d96d25c FAIL (2025-11-08 15:10:11 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Nov 8 14:26:07.882330547 (218h45m39.5781793s) [upload=192678]: Starting upload Nov 8 14:26:07.882433824 (218h45m39.578282567s) [upload=192678]: Done starting upload Nov 8 14:26:07.882492927 (218h45m39.57834167s) [upload=192678]: Uploading segments... Nov 8 14:26:07.882645367 (218h45m39.57849411s) [upload=192678]: Got next segment. Inline: false Nov 8 14:26:07.882698468 (218h45m39.578547221s) [upload=192678]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 8 14:26:07.882704229 (218h45m39.578552982s) [upload=192678]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 8 14:26:08.131091114 (218h45m39.826939897s) [seg_pos={0 0}, upload=192678]: Begin upload segment... Nov 8 14:26:08.131165496 (218h45m39.827014239s) [seg_pos={0 0}, upload=192678]: Done begin upload segment. Nov 8 14:26:08.131220160 (218h45m39.827068913s) [noise=true , address=145.79.227.16:4911 , piece_id=QKITSRMSCHOZLCQ7X6GARTE3GS24G3CZFXJIBKDM3VWOVQLMLEHA, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.131304631 (218h45m39.827153384s) [noise=true , address=145.79.227.27:4689 , piece_id=JEEKOLHURPGNFSR4YCWL3UD6BU2MT7VW4S7BU3YKE7BJSUEYLPGQ, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.132190408 (218h45m39.828039161s) [noise=true , address=145.79.227.7:4295 , piece_id=5OY4NKICVFGAWR7JPOOYIYRXR442IIUDDWTN3MT6KAR4ZHXTKPNQ, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.132569360 (218h45m39.828418113s) [noise=true , address=145.79.227.31:4798 , piece_id=5Q7NYNXM2IMV7YUCB464FQMBI4L476Y6VKPPOPW7RKNCC5XFDJ2A, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.133276446 (218h45m39.829125199s) [noise=true , address=145.79.227.26:4653 , piece_id=UIV3WFYLH4JZSIIHZUBL56THDPAYK7DMRUQG5TVTHHEF3XGN6NKQ, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.134167893 (218h45m39.830016646s) [noise=true , address=145.79.227.26:4655 , piece_id=GP4F5L6HH6S4ZEYWS7XRG5KJSYFGOX5JXQBTQEOXFDFSNUBHZJ6A, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.134314192 (218h45m39.830162945s) [noise=true , address=145.79.227.20:4437 , piece_id=G5RTYGERAVOTQDZGCG6NEKRCQAUOVJNYXO4LEBJ7XHJZ6FDXTULQ, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.135016950 (218h45m39.830865703s) [noise=true , address=145.79.227.10:4402 , piece_id=IRNQEX73D4AEXNYXEWWKCKEMYFC3YXIPX6HSDYYBNDKBM77ROBEA, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.135235305 (218h45m39.831084059s) [noise=true , address=145.79.227.22:4509 , piece_id=6RJJCCT2Z5U3JOWWOBLRIDK5CTWKXU7NR4OIREAQ2JDE2KYDJHOQ, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.135440236 (218h45m39.831288979s) [noise=true , address=145.79.227.2:4113 , piece_id=Q5E3IXSHGFKIFG7EFAPW2XWYP63TO3BC5C7PZJDPXXD2UKDQDJLA, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.136611887 (218h45m39.83246064s) [noise=true , address=77.90.31.15:4699 , piece_id=Z4PXJFTM7XMK62ORTMHCWEJ7OQF6PIYQCMAWJC3DT5RZ7CEH4KXA, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.137027559 (218h45m39.832876312s) [noise=true , address=145.79.227.8:4329 , piece_id=JJFSJ7USPZFTPXVUZVLMXRAHYLARBQZLWWU7ITXFNESYVVL4QBMQ, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.137896133 (218h45m39.833744886s) [noise=true , address=77.90.31.17:4771 , piece_id=LUZ4KXA4YUVTQYKXVEPTEXZ4MFQ5IWFTY3C2LUUVQ3N6DOVO6ZXA, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.139107940 (218h45m39.834956693s) [noise=true , address=145.79.227.21:4475 , piece_id=6F7IZNQLORMTKUF576FJA6L7FBG66INEUFPQYHL77WLC6DYF32GQ, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.139872657 (218h45m39.83572141s) [noise=true , address=145.79.227.9:4366 , piece_id=3BINODFKOOG7BYULCICW6S64YZOU5D2Q5MFZYGPSSA34XWM762BQ, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.140005960 (218h45m39.835854713s) [noise=true , address=77.90.31.16:4734 , piece_id=EKYBQDVNAYTGEPWU443RFLUX7OKCTWBPXA5UMCLZ22M2PBXO3E3Q, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:10.015794232 (218h45m41.711642985s) [upload=192678]: Next returned nil segment Nov 8 14:26:10.015816414 (218h45m41.711665167s) [upload=192678]: Waiting for error group managing segments... Nov 8 14:26:10.314091526 (218h45m42.009940279s) [noise=true , address=77.90.31.16:4734 , piece_id=EKYBQDVNAYTGEPWU443RFLUX7OKCTWBPXA5UMCLZ22M2PBXO3E3Q, seg_pos={0 0}, upload=192678]: Done uploading piece. err: <nil> Nov 8 14:26:10.388439672 (218h45m42.084288465s) [noise=true , address=77.90.31.15:4699 , piece_id=Z4PXJFTM7XMK62ORTMHCWEJ7OQF6PIYQCMAWJC3DT5RZ7CEH4KXA, seg_pos={0 0}, upload=192678]: Done uploading piece. err: <nil> Nov 8 14:26:11.119863095 (218h45m42.815711838s) [noise=true , address=77.90.31.17:4771 , piece_id=LUZ4KXA4YUVTQYKXVEPTEXZ4MFQ5IWFTY3C2LUUVQ3N6DOVO6ZXA, seg_pos={0 0}, upload=192678]: Done uploading piece. err: <nil> Nov 8 14:26:14.806769255 (218h45m46.502618008s) [noise=true , address=145.79.227.2:4113 , piece_id=Q5E3IXSHGFKIFG7EFAPW2XWYP63TO3BC5C7PZJDPXXD2UKDQDJLA, seg_pos={0 0}, upload=192678]: Done uploading piece. err: <nil> Nov 8 14:26:15.197752745 (218h45m46.893601498s) [noise=true , address=145.79.227.26:4655 , piece_id=GP4F5L6HH6S4ZEYWS7XRG5KJSYFGOX5JXQBTQEOXFDFSNUBHZJ6A, seg_pos={0 0}, upload=192678]: Done uploading piece. err: <nil> Nov 8 14:26:17.425396997 (218h45m49.12124575s) [noise=true , address=145.79.227.22:4509 , piece_id=6RJJCCT2Z5U3JOWWOBLRIDK5CTWKXU7NR4OIREAQ2JDE2KYDJHOQ, seg_pos={0 0}, upload=192678]: Done uploading piece. err: <nil> Nov 8 14:26:21.856283285 (218h45m53.552132048s) [noise=true , address=145.79.227.31:4798 , piece_id=5Q7NYNXM2IMV7YUCB464FQMBI4L476Y6VKPPOPW7RKNCC5XFDJ2A, seg_pos={0 0}, upload=192678]: Done uploading piece. err: <nil> Nov 8 14:26:23.486413928 (218h45m55.182262681s) [noise=true , address=145.79.227.20:4437 , piece_id=G5RTYGERAVOTQDZGCG6NEKRCQAUOVJNYXO4LEBJ7XHJZ6FDXTULQ, seg_pos={0 0}, upload=192678]: Done uploading piece. err: <nil> Nov 8 14:26:24.659870720 (218h45m56.355719473s) [noise=true , address=145.79.227.26:4653 , piece_id=UIV3WFYLH4JZSIIHZUBL56THDPAYK7DMRUQG5TVTHHEF3XGN6NKQ, seg_pos={0 0}, upload=192678]: Done uploading piece. err: <nil> Nov 8 14:26:34.701060537 (218h46m6.3969093s) [noise=true , address=145.79.227.8:4329 , piece_id=JJFSJ7USPZFTPXVUZVLMXRAHYLARBQZLWWU7ITXFNESYVVL4QBMQ, seg_pos={0 0}, upload=192678]: Done uploading piece. err: <nil> Nov 8 14:26:37.183628509 (218h46m8.879477262s) [noise=true , address=145.79.227.10:4402 , piece_id=IRNQEX73D4AEXNYXEWWKCKEMYFC3YXIPX6HSDYYBNDKBM77ROBEA, seg_pos={0 0}, upload=192678]: Done uploading piece. err: <nil> Nov 8 14:26:37.186238899 (218h46m8.882087662s) [noise=true , address=145.79.227.16:4911 , piece_id=QKITSRMSCHOZLCQ7X6GARTE3GS24G3CZFXJIBKDM3VWOVQLMLEHA, seg_pos={0 0}, upload=192678]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:145.79.227.16:4911): protocol: write tcp 188.166.225.170:56440->145.79.227.16:4911: use of closed network connection; write tcp 188.166.225.170:56440->145.79.227.16:4911: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56440->145.79.227.16:4911: use of closed network connection Nov 8 14:26:37.186462625 (218h46m8.882311378s) [noise=true , address=145.79.227.21:4475 , piece_id=6F7IZNQLORMTKUF576FJA6L7FBG66INEUFPQYHL77WLC6DYF32GQ, seg_pos={0 0}, upload=192678]: Done uploading piece. err: ecclient: upload failed (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk, address:145.79.227.21:4475): protocol: expected piece hash; context deadline exceeded; EOF Nov 8 14:26:37.186739442 (218h46m8.882588195s) [noise=true , address=145.79.227.27:4689 , piece_id=JEEKOLHURPGNFSR4YCWL3UD6BU2MT7VW4S7BU3YKE7BJSUEYLPGQ, seg_pos={0 0}, upload=192678]: Done uploading piece. err: ecclient: upload failed (node:1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, address:145.79.227.27:4689): protocol: write tcp 188.166.225.170:37312->145.79.227.27:4689: use of closed network connection; write tcp 188.166.225.170:37312->145.79.227.27:4689: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37312->145.79.227.27:4689: use of closed network connection Nov 8 14:26:37.186915056 (218h46m8.88276381s) [noise=true , address=145.79.227.7:4295 , piece_id=5OY4NKICVFGAWR7JPOOYIYRXR442IIUDDWTN3MT6KAR4ZHXTKPNQ, seg_pos={0 0}, upload=192678]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:145.79.227.7:4295): protocol: expected piece hash; context deadline exceeded; EOF Nov 8 14:26:37.187072807 (218h46m8.88292156s) [noise=true , address=145.79.227.9:4366 , piece_id=3BINODFKOOG7BYULCICW6S64YZOU5D2Q5MFZYGPSSA34XWM762BQ, seg_pos={0 0}, upload=192678]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: expected piece hash; context deadline exceeded; EOF Nov 8 14:26:37.187107163 (218h46m8.882955916s) [seg_pos={0 0}, upload=192678]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 8 14:26:37.433482306 (218h46m9.129331059s) [upload=192678]: Done uploading segments... Nov 8 14:26:37.433526460 (218h46m9.129375213s) [upload=192678]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 12.652924703s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 7.823528365s trace: https://jaeger.dev.storj.io/trace/3f377bb0d96d25c FAIL (2025-11-08 15:08:57 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Nov 8 14:26:07.882330547 (218h45m39.5781793s) [upload=192678]: Starting upload Nov 8 14:26:07.882433824 (218h45m39.578282567s) [upload=192678]: Done starting upload Nov 8 14:26:07.882492927 (218h45m39.57834167s) [upload=192678]: Uploading segments... Nov 8 14:26:07.882645367 (218h45m39.57849411s) [upload=192678]: Got next segment. Inline: false Nov 8 14:26:07.882698468 (218h45m39.578547221s) [upload=192678]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 8 14:26:07.882704229 (218h45m39.578552982s) [upload=192678]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 8 14:26:08.131091114 (218h45m39.826939897s) [seg_pos={0 0}, upload=192678]: Begin upload segment... Nov 8 14:26:08.131165496 (218h45m39.827014239s) [seg_pos={0 0}, upload=192678]: Done begin upload segment. Nov 8 14:26:08.131220160 (218h45m39.827068913s) [noise=true , address=145.79.227.16:4911 , piece_id=QKITSRMSCHOZLCQ7X6GARTE3GS24G3CZFXJIBKDM3VWOVQLMLEHA, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.131304631 (218h45m39.827153384s) [noise=true , address=145.79.227.27:4689 , piece_id=JEEKOLHURPGNFSR4YCWL3UD6BU2MT7VW4S7BU3YKE7BJSUEYLPGQ, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.132190408 (218h45m39.828039161s) [noise=true , address=145.79.227.7:4295 , piece_id=5OY4NKICVFGAWR7JPOOYIYRXR442IIUDDWTN3MT6KAR4ZHXTKPNQ, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.132569360 (218h45m39.828418113s) [noise=true , address=145.79.227.31:4798 , piece_id=5Q7NYNXM2IMV7YUCB464FQMBI4L476Y6VKPPOPW7RKNCC5XFDJ2A, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.133276446 (218h45m39.829125199s) [noise=true , address=145.79.227.26:4653 , piece_id=UIV3WFYLH4JZSIIHZUBL56THDPAYK7DMRUQG5TVTHHEF3XGN6NKQ, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.134167893 (218h45m39.830016646s) [noise=true , address=145.79.227.26:4655 , piece_id=GP4F5L6HH6S4ZEYWS7XRG5KJSYFGOX5JXQBTQEOXFDFSNUBHZJ6A, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.134314192 (218h45m39.830162945s) [noise=true , address=145.79.227.20:4437 , piece_id=G5RTYGERAVOTQDZGCG6NEKRCQAUOVJNYXO4LEBJ7XHJZ6FDXTULQ, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.135016950 (218h45m39.830865703s) [noise=true , address=145.79.227.10:4402 , piece_id=IRNQEX73D4AEXNYXEWWKCKEMYFC3YXIPX6HSDYYBNDKBM77ROBEA, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.135235305 (218h45m39.831084059s) [noise=true , address=145.79.227.22:4509 , piece_id=6RJJCCT2Z5U3JOWWOBLRIDK5CTWKXU7NR4OIREAQ2JDE2KYDJHOQ, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.135440236 (218h45m39.831288979s) [noise=true , address=145.79.227.2:4113 , piece_id=Q5E3IXSHGFKIFG7EFAPW2XWYP63TO3BC5C7PZJDPXXD2UKDQDJLA, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.136611887 (218h45m39.83246064s) [noise=true , address=77.90.31.15:4699 , piece_id=Z4PXJFTM7XMK62ORTMHCWEJ7OQF6PIYQCMAWJC3DT5RZ7CEH4KXA, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.137027559 (218h45m39.832876312s) [noise=true , address=145.79.227.8:4329 , piece_id=JJFSJ7USPZFTPXVUZVLMXRAHYLARBQZLWWU7ITXFNESYVVL4QBMQ, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.137896133 (218h45m39.833744886s) [noise=true , address=77.90.31.17:4771 , piece_id=LUZ4KXA4YUVTQYKXVEPTEXZ4MFQ5IWFTY3C2LUUVQ3N6DOVO6ZXA, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.139107940 (218h45m39.834956693s) [noise=true , address=145.79.227.21:4475 , piece_id=6F7IZNQLORMTKUF576FJA6L7FBG66INEUFPQYHL77WLC6DYF32GQ, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.139872657 (218h45m39.83572141s) [noise=true , address=145.79.227.9:4366 , piece_id=3BINODFKOOG7BYULCICW6S64YZOU5D2Q5MFZYGPSSA34XWM762BQ, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.140005960 (218h45m39.835854713s) [noise=true , address=77.90.31.16:4734 , piece_id=EKYBQDVNAYTGEPWU443RFLUX7OKCTWBPXA5UMCLZ22M2PBXO3E3Q, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:10.015794232 (218h45m41.711642985s) [upload=192678]: Next returned nil segment Nov 8 14:26:10.015816414 (218h45m41.711665167s) [upload=192678]: Waiting for error group managing segments... Nov 8 14:26:10.314091526 (218h45m42.009940279s) [noise=true , address=77.90.31.16:4734 , piece_id=EKYBQDVNAYTGEPWU443RFLUX7OKCTWBPXA5UMCLZ22M2PBXO3E3Q, seg_pos={0 0}, upload=192678]: Done uploading piece. err: <nil> Nov 8 14:26:10.388439672 (218h45m42.084288465s) [noise=true , address=77.90.31.15:4699 , piece_id=Z4PXJFTM7XMK62ORTMHCWEJ7OQF6PIYQCMAWJC3DT5RZ7CEH4KXA, seg_pos={0 0}, upload=192678]: Done uploading piece. err: <nil> Nov 8 14:26:11.119863095 (218h45m42.815711838s) [noise=true , address=77.90.31.17:4771 , piece_id=LUZ4KXA4YUVTQYKXVEPTEXZ4MFQ5IWFTY3C2LUUVQ3N6DOVO6ZXA, seg_pos={0 0}, upload=192678]: Done uploading piece. err: <nil> Nov 8 14:26:14.806769255 (218h45m46.502618008s) [noise=true , address=145.79.227.2:4113 , piece_id=Q5E3IXSHGFKIFG7EFAPW2XWYP63TO3BC5C7PZJDPXXD2UKDQDJLA, seg_pos={0 0}, upload=192678]: Done uploading piece. err: <nil> Nov 8 14:26:15.197752745 (218h45m46.893601498s) [noise=true , address=145.79.227.26:4655 , piece_id=GP4F5L6HH6S4ZEYWS7XRG5KJSYFGOX5JXQBTQEOXFDFSNUBHZJ6A, seg_pos={0 0}, upload=192678]: Done uploading piece. err: <nil> Nov 8 14:26:17.425396997 (218h45m49.12124575s) [noise=true , address=145.79.227.22:4509 , piece_id=6RJJCCT2Z5U3JOWWOBLRIDK5CTWKXU7NR4OIREAQ2JDE2KYDJHOQ, seg_pos={0 0}, upload=192678]: Done uploading piece. err: <nil> Nov 8 14:26:21.856283285 (218h45m53.552132048s) [noise=true , address=145.79.227.31:4798 , piece_id=5Q7NYNXM2IMV7YUCB464FQMBI4L476Y6VKPPOPW7RKNCC5XFDJ2A, seg_pos={0 0}, upload=192678]: Done uploading piece. err: <nil> Nov 8 14:26:23.486413928 (218h45m55.182262681s) [noise=true , address=145.79.227.20:4437 , piece_id=G5RTYGERAVOTQDZGCG6NEKRCQAUOVJNYXO4LEBJ7XHJZ6FDXTULQ, seg_pos={0 0}, upload=192678]: Done uploading piece. err: <nil> Nov 8 14:26:24.659870720 (218h45m56.355719473s) [noise=true , address=145.79.227.26:4653 , piece_id=UIV3WFYLH4JZSIIHZUBL56THDPAYK7DMRUQG5TVTHHEF3XGN6NKQ, seg_pos={0 0}, upload=192678]: Done uploading piece. err: <nil> Nov 8 14:26:34.701060537 (218h46m6.3969093s) [noise=true , address=145.79.227.8:4329 , piece_id=JJFSJ7USPZFTPXVUZVLMXRAHYLARBQZLWWU7ITXFNESYVVL4QBMQ, seg_pos={0 0}, upload=192678]: Done uploading piece. err: <nil> Nov 8 14:26:37.183628509 (218h46m8.879477262s) [noise=true , address=145.79.227.10:4402 , piece_id=IRNQEX73D4AEXNYXEWWKCKEMYFC3YXIPX6HSDYYBNDKBM77ROBEA, seg_pos={0 0}, upload=192678]: Done uploading piece. err: <nil> Nov 8 14:26:37.186238899 (218h46m8.882087662s) [noise=true , address=145.79.227.16:4911 , piece_id=QKITSRMSCHOZLCQ7X6GARTE3GS24G3CZFXJIBKDM3VWOVQLMLEHA, seg_pos={0 0}, upload=192678]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:145.79.227.16:4911): protocol: write tcp 188.166.225.170:56440->145.79.227.16:4911: use of closed network connection; write tcp 188.166.225.170:56440->145.79.227.16:4911: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56440->145.79.227.16:4911: use of closed network connection Nov 8 14:26:37.186462625 (218h46m8.882311378s) [noise=true , address=145.79.227.21:4475 , piece_id=6F7IZNQLORMTKUF576FJA6L7FBG66INEUFPQYHL77WLC6DYF32GQ, seg_pos={0 0}, upload=192678]: Done uploading piece. err: ecclient: upload failed (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk, address:145.79.227.21:4475): protocol: expected piece hash; context deadline exceeded; EOF Nov 8 14:26:37.186739442 (218h46m8.882588195s) [noise=true , address=145.79.227.27:4689 , piece_id=JEEKOLHURPGNFSR4YCWL3UD6BU2MT7VW4S7BU3YKE7BJSUEYLPGQ, seg_pos={0 0}, upload=192678]: Done uploading piece. err: ecclient: upload failed (node:1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, address:145.79.227.27:4689): protocol: write tcp 188.166.225.170:37312->145.79.227.27:4689: use of closed network connection; write tcp 188.166.225.170:37312->145.79.227.27:4689: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37312->145.79.227.27:4689: use of closed network connection Nov 8 14:26:37.186915056 (218h46m8.88276381s) [noise=true , address=145.79.227.7:4295 , piece_id=5OY4NKICVFGAWR7JPOOYIYRXR442IIUDDWTN3MT6KAR4ZHXTKPNQ, seg_pos={0 0}, upload=192678]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:145.79.227.7:4295): protocol: expected piece hash; context deadline exceeded; EOF Nov 8 14:26:37.187072807 (218h46m8.88292156s) [noise=true , address=145.79.227.9:4366 , piece_id=3BINODFKOOG7BYULCICW6S64YZOU5D2Q5MFZYGPSSA34XWM762BQ, seg_pos={0 0}, upload=192678]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: expected piece hash; context deadline exceeded; EOF Nov 8 14:26:37.187107163 (218h46m8.882955916s) [seg_pos={0 0}, upload=192678]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 8 14:26:37.433482306 (218h46m9.129331059s) [upload=192678]: Done uploading segments... Nov 8 14:26:37.433526460 (218h46m9.129375213s) [upload=192678]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 7.823576036s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 8.949779746s trace: https://jaeger.dev.storj.io/trace/3f377bb0d96d25c FAIL (2025-11-08 15:08:01 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Nov 8 14:26:07.882330547 (218h45m39.5781793s) [upload=192678]: Starting upload Nov 8 14:26:07.882433824 (218h45m39.578282567s) [upload=192678]: Done starting upload Nov 8 14:26:07.882492927 (218h45m39.57834167s) [upload=192678]: Uploading segments... Nov 8 14:26:07.882645367 (218h45m39.57849411s) [upload=192678]: Got next segment. Inline: false Nov 8 14:26:07.882698468 (218h45m39.578547221s) [upload=192678]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 8 14:26:07.882704229 (218h45m39.578552982s) [upload=192678]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 8 14:26:08.131091114 (218h45m39.826939897s) [seg_pos={0 0}, upload=192678]: Begin upload segment... Nov 8 14:26:08.131165496 (218h45m39.827014239s) [seg_pos={0 0}, upload=192678]: Done begin upload segment. Nov 8 14:26:08.131220160 (218h45m39.827068913s) [noise=true , address=145.79.227.16:4911 , piece_id=QKITSRMSCHOZLCQ7X6GARTE3GS24G3CZFXJIBKDM3VWOVQLMLEHA, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.131304631 (218h45m39.827153384s) [noise=true , address=145.79.227.27:4689 , piece_id=JEEKOLHURPGNFSR4YCWL3UD6BU2MT7VW4S7BU3YKE7BJSUEYLPGQ, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.132190408 (218h45m39.828039161s) [noise=true , address=145.79.227.7:4295 , piece_id=5OY4NKICVFGAWR7JPOOYIYRXR442IIUDDWTN3MT6KAR4ZHXTKPNQ, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.132569360 (218h45m39.828418113s) [noise=true , address=145.79.227.31:4798 , piece_id=5Q7NYNXM2IMV7YUCB464FQMBI4L476Y6VKPPOPW7RKNCC5XFDJ2A, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.133276446 (218h45m39.829125199s) [noise=true , address=145.79.227.26:4653 , piece_id=UIV3WFYLH4JZSIIHZUBL56THDPAYK7DMRUQG5TVTHHEF3XGN6NKQ, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.134167893 (218h45m39.830016646s) [noise=true , address=145.79.227.26:4655 , piece_id=GP4F5L6HH6S4ZEYWS7XRG5KJSYFGOX5JXQBTQEOXFDFSNUBHZJ6A, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.134314192 (218h45m39.830162945s) [noise=true , address=145.79.227.20:4437 , piece_id=G5RTYGERAVOTQDZGCG6NEKRCQAUOVJNYXO4LEBJ7XHJZ6FDXTULQ, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.135016950 (218h45m39.830865703s) [noise=true , address=145.79.227.10:4402 , piece_id=IRNQEX73D4AEXNYXEWWKCKEMYFC3YXIPX6HSDYYBNDKBM77ROBEA, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.135235305 (218h45m39.831084059s) [noise=true , address=145.79.227.22:4509 , piece_id=6RJJCCT2Z5U3JOWWOBLRIDK5CTWKXU7NR4OIREAQ2JDE2KYDJHOQ, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.135440236 (218h45m39.831288979s) [noise=true , address=145.79.227.2:4113 , piece_id=Q5E3IXSHGFKIFG7EFAPW2XWYP63TO3BC5C7PZJDPXXD2UKDQDJLA, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.136611887 (218h45m39.83246064s) [noise=true , address=77.90.31.15:4699 , piece_id=Z4PXJFTM7XMK62ORTMHCWEJ7OQF6PIYQCMAWJC3DT5RZ7CEH4KXA, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.137027559 (218h45m39.832876312s) [noise=true , address=145.79.227.8:4329 , piece_id=JJFSJ7USPZFTPXVUZVLMXRAHYLARBQZLWWU7ITXFNESYVVL4QBMQ, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.137896133 (218h45m39.833744886s) [noise=true , address=77.90.31.17:4771 , piece_id=LUZ4KXA4YUVTQYKXVEPTEXZ4MFQ5IWFTY3C2LUUVQ3N6DOVO6ZXA, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.139107940 (218h45m39.834956693s) [noise=true , address=145.79.227.21:4475 , piece_id=6F7IZNQLORMTKUF576FJA6L7FBG66INEUFPQYHL77WLC6DYF32GQ, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.139872657 (218h45m39.83572141s) [noise=true , address=145.79.227.9:4366 , piece_id=3BINODFKOOG7BYULCICW6S64YZOU5D2Q5MFZYGPSSA34XWM762BQ, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.140005960 (218h45m39.835854713s) [noise=true , address=77.90.31.16:4734 , piece_id=EKYBQDVNAYTGEPWU443RFLUX7OKCTWBPXA5UMCLZ22M2PBXO3E3Q, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:10.015794232 (218h45m41.711642985s) [upload=192678]: Next returned nil segment Nov 8 14:26:10.015816414 (218h45m41.711665167s) [upload=192678]: Waiting for error group managing segments... Nov 8 14:26:10.314091526 (218h45m42.009940279s) [noise=true , address=77.90.31.16:4734 , piece_id=EKYBQDVNAYTGEPWU443RFLUX7OKCTWBPXA5UMCLZ22M2PBXO3E3Q, seg_pos={0 0}, upload=192678]: Done uploading piece. err: <nil> Nov 8 14:26:10.388439672 (218h45m42.084288465s) [noise=true , address=77.90.31.15:4699 , piece_id=Z4PXJFTM7XMK62ORTMHCWEJ7OQF6PIYQCMAWJC3DT5RZ7CEH4KXA, seg_pos={0 0}, upload=192678]: Done uploading piece. err: <nil> Nov 8 14:26:11.119863095 (218h45m42.815711838s) [noise=true , address=77.90.31.17:4771 , piece_id=LUZ4KXA4YUVTQYKXVEPTEXZ4MFQ5IWFTY3C2LUUVQ3N6DOVO6ZXA, seg_pos={0 0}, upload=192678]: Done uploading piece. err: <nil> Nov 8 14:26:14.806769255 (218h45m46.502618008s) [noise=true , address=145.79.227.2:4113 , piece_id=Q5E3IXSHGFKIFG7EFAPW2XWYP63TO3BC5C7PZJDPXXD2UKDQDJLA, seg_pos={0 0}, upload=192678]: Done uploading piece. err: <nil> Nov 8 14:26:15.197752745 (218h45m46.893601498s) [noise=true , address=145.79.227.26:4655 , piece_id=GP4F5L6HH6S4ZEYWS7XRG5KJSYFGOX5JXQBTQEOXFDFSNUBHZJ6A, seg_pos={0 0}, upload=192678]: Done uploading piece. err: <nil> Nov 8 14:26:17.425396997 (218h45m49.12124575s) [noise=true , address=145.79.227.22:4509 , piece_id=6RJJCCT2Z5U3JOWWOBLRIDK5CTWKXU7NR4OIREAQ2JDE2KYDJHOQ, seg_pos={0 0}, upload=192678]: Done uploading piece. err: <nil> Nov 8 14:26:21.856283285 (218h45m53.552132048s) [noise=true , address=145.79.227.31:4798 , piece_id=5Q7NYNXM2IMV7YUCB464FQMBI4L476Y6VKPPOPW7RKNCC5XFDJ2A, seg_pos={0 0}, upload=192678]: Done uploading piece. err: <nil> Nov 8 14:26:23.486413928 (218h45m55.182262681s) [noise=true , address=145.79.227.20:4437 , piece_id=G5RTYGERAVOTQDZGCG6NEKRCQAUOVJNYXO4LEBJ7XHJZ6FDXTULQ, seg_pos={0 0}, upload=192678]: Done uploading piece. err: <nil> Nov 8 14:26:24.659870720 (218h45m56.355719473s) [noise=true , address=145.79.227.26:4653 , piece_id=UIV3WFYLH4JZSIIHZUBL56THDPAYK7DMRUQG5TVTHHEF3XGN6NKQ, seg_pos={0 0}, upload=192678]: Done uploading piece. err: <nil> Nov 8 14:26:34.701060537 (218h46m6.3969093s) [noise=true , address=145.79.227.8:4329 , piece_id=JJFSJ7USPZFTPXVUZVLMXRAHYLARBQZLWWU7ITXFNESYVVL4QBMQ, seg_pos={0 0}, upload=192678]: Done uploading piece. err: <nil> Nov 8 14:26:37.183628509 (218h46m8.879477262s) [noise=true , address=145.79.227.10:4402 , piece_id=IRNQEX73D4AEXNYXEWWKCKEMYFC3YXIPX6HSDYYBNDKBM77ROBEA, seg_pos={0 0}, upload=192678]: Done uploading piece. err: <nil> Nov 8 14:26:37.186238899 (218h46m8.882087662s) [noise=true , address=145.79.227.16:4911 , piece_id=QKITSRMSCHOZLCQ7X6GARTE3GS24G3CZFXJIBKDM3VWOVQLMLEHA, seg_pos={0 0}, upload=192678]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:145.79.227.16:4911): protocol: write tcp 188.166.225.170:56440->145.79.227.16:4911: use of closed network connection; write tcp 188.166.225.170:56440->145.79.227.16:4911: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56440->145.79.227.16:4911: use of closed network connection Nov 8 14:26:37.186462625 (218h46m8.882311378s) [noise=true , address=145.79.227.21:4475 , piece_id=6F7IZNQLORMTKUF576FJA6L7FBG66INEUFPQYHL77WLC6DYF32GQ, seg_pos={0 0}, upload=192678]: Done uploading piece. err: ecclient: upload failed (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk, address:145.79.227.21:4475): protocol: expected piece hash; context deadline exceeded; EOF Nov 8 14:26:37.186739442 (218h46m8.882588195s) [noise=true , address=145.79.227.27:4689 , piece_id=JEEKOLHURPGNFSR4YCWL3UD6BU2MT7VW4S7BU3YKE7BJSUEYLPGQ, seg_pos={0 0}, upload=192678]: Done uploading piece. err: ecclient: upload failed (node:1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, address:145.79.227.27:4689): protocol: write tcp 188.166.225.170:37312->145.79.227.27:4689: use of closed network connection; write tcp 188.166.225.170:37312->145.79.227.27:4689: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37312->145.79.227.27:4689: use of closed network connection Nov 8 14:26:37.186915056 (218h46m8.88276381s) [noise=true , address=145.79.227.7:4295 , piece_id=5OY4NKICVFGAWR7JPOOYIYRXR442IIUDDWTN3MT6KAR4ZHXTKPNQ, seg_pos={0 0}, upload=192678]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:145.79.227.7:4295): protocol: expected piece hash; context deadline exceeded; EOF Nov 8 14:26:37.187072807 (218h46m8.88292156s) [noise=true , address=145.79.227.9:4366 , piece_id=3BINODFKOOG7BYULCICW6S64YZOU5D2Q5MFZYGPSSA34XWM762BQ, seg_pos={0 0}, upload=192678]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: expected piece hash; context deadline exceeded; EOF Nov 8 14:26:37.187107163 (218h46m8.882955916s) [seg_pos={0 0}, upload=192678]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 8 14:26:37.433482306 (218h46m9.129331059s) [upload=192678]: Done uploading segments... Nov 8 14:26:37.433526460 (218h46m9.129375213s) [upload=192678]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 8.94980791s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 8.320007111s trace: https://jaeger.dev.storj.io/trace/3f377bb0d96d25c FAIL (2025-11-08 15:07:02 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Nov 8 14:26:07.882330547 (218h45m39.5781793s) [upload=192678]: Starting upload Nov 8 14:26:07.882433824 (218h45m39.578282567s) [upload=192678]: Done starting upload Nov 8 14:26:07.882492927 (218h45m39.57834167s) [upload=192678]: Uploading segments... Nov 8 14:26:07.882645367 (218h45m39.57849411s) [upload=192678]: Got next segment. Inline: false Nov 8 14:26:07.882698468 (218h45m39.578547221s) [upload=192678]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 8 14:26:07.882704229 (218h45m39.578552982s) [upload=192678]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 8 14:26:08.131091114 (218h45m39.826939897s) [seg_pos={0 0}, upload=192678]: Begin upload segment... Nov 8 14:26:08.131165496 (218h45m39.827014239s) [seg_pos={0 0}, upload=192678]: Done begin upload segment. Nov 8 14:26:08.131220160 (218h45m39.827068913s) [noise=true , address=145.79.227.16:4911 , piece_id=QKITSRMSCHOZLCQ7X6GARTE3GS24G3CZFXJIBKDM3VWOVQLMLEHA, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.131304631 (218h45m39.827153384s) [noise=true , address=145.79.227.27:4689 , piece_id=JEEKOLHURPGNFSR4YCWL3UD6BU2MT7VW4S7BU3YKE7BJSUEYLPGQ, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.132190408 (218h45m39.828039161s) [noise=true , address=145.79.227.7:4295 , piece_id=5OY4NKICVFGAWR7JPOOYIYRXR442IIUDDWTN3MT6KAR4ZHXTKPNQ, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.132569360 (218h45m39.828418113s) [noise=true , address=145.79.227.31:4798 , piece_id=5Q7NYNXM2IMV7YUCB464FQMBI4L476Y6VKPPOPW7RKNCC5XFDJ2A, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.133276446 (218h45m39.829125199s) [noise=true , address=145.79.227.26:4653 , piece_id=UIV3WFYLH4JZSIIHZUBL56THDPAYK7DMRUQG5TVTHHEF3XGN6NKQ, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.134167893 (218h45m39.830016646s) [noise=true , address=145.79.227.26:4655 , piece_id=GP4F5L6HH6S4ZEYWS7XRG5KJSYFGOX5JXQBTQEOXFDFSNUBHZJ6A, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.134314192 (218h45m39.830162945s) [noise=true , address=145.79.227.20:4437 , piece_id=G5RTYGERAVOTQDZGCG6NEKRCQAUOVJNYXO4LEBJ7XHJZ6FDXTULQ, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.135016950 (218h45m39.830865703s) [noise=true , address=145.79.227.10:4402 , piece_id=IRNQEX73D4AEXNYXEWWKCKEMYFC3YXIPX6HSDYYBNDKBM77ROBEA, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.135235305 (218h45m39.831084059s) [noise=true , address=145.79.227.22:4509 , piece_id=6RJJCCT2Z5U3JOWWOBLRIDK5CTWKXU7NR4OIREAQ2JDE2KYDJHOQ, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.135440236 (218h45m39.831288979s) [noise=true , address=145.79.227.2:4113 , piece_id=Q5E3IXSHGFKIFG7EFAPW2XWYP63TO3BC5C7PZJDPXXD2UKDQDJLA, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.136611887 (218h45m39.83246064s) [noise=true , address=77.90.31.15:4699 , piece_id=Z4PXJFTM7XMK62ORTMHCWEJ7OQF6PIYQCMAWJC3DT5RZ7CEH4KXA, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.137027559 (218h45m39.832876312s) [noise=true , address=145.79.227.8:4329 , piece_id=JJFSJ7USPZFTPXVUZVLMXRAHYLARBQZLWWU7ITXFNESYVVL4QBMQ, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.137896133 (218h45m39.833744886s) [noise=true , address=77.90.31.17:4771 , piece_id=LUZ4KXA4YUVTQYKXVEPTEXZ4MFQ5IWFTY3C2LUUVQ3N6DOVO6ZXA, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.139107940 (218h45m39.834956693s) [noise=true , address=145.79.227.21:4475 , piece_id=6F7IZNQLORMTKUF576FJA6L7FBG66INEUFPQYHL77WLC6DYF32GQ, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.139872657 (218h45m39.83572141s) [noise=true , address=145.79.227.9:4366 , piece_id=3BINODFKOOG7BYULCICW6S64YZOU5D2Q5MFZYGPSSA34XWM762BQ, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.140005960 (218h45m39.835854713s) [noise=true , address=77.90.31.16:4734 , piece_id=EKYBQDVNAYTGEPWU443RFLUX7OKCTWBPXA5UMCLZ22M2PBXO3E3Q, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:10.015794232 (218h45m41.711642985s) [upload=192678]: Next returned nil segment Nov 8 14:26:10.015816414 (218h45m41.711665167s) [upload=192678]: Waiting for error group managing segments... Nov 8 14:26:10.314091526 (218h45m42.009940279s) [noise=true , address=77.90.31.16:4734 , piece_id=EKYBQDVNAYTGEPWU443RFLUX7OKCTWBPXA5UMCLZ22M2PBXO3E3Q, seg_pos={0 0}, upload=192678]: Done uploading piece. err: <nil> Nov 8 14:26:10.388439672 (218h45m42.084288465s) [noise=true , address=77.90.31.15:4699 , piece_id=Z4PXJFTM7XMK62ORTMHCWEJ7OQF6PIYQCMAWJC3DT5RZ7CEH4KXA, seg_pos={0 0}, upload=192678]: Done uploading piece. err: <nil> Nov 8 14:26:11.119863095 (218h45m42.815711838s) [noise=true , address=77.90.31.17:4771 , piece_id=LUZ4KXA4YUVTQYKXVEPTEXZ4MFQ5IWFTY3C2LUUVQ3N6DOVO6ZXA, seg_pos={0 0}, upload=192678]: Done uploading piece. err: <nil> Nov 8 14:26:14.806769255 (218h45m46.502618008s) [noise=true , address=145.79.227.2:4113 , piece_id=Q5E3IXSHGFKIFG7EFAPW2XWYP63TO3BC5C7PZJDPXXD2UKDQDJLA, seg_pos={0 0}, upload=192678]: Done uploading piece. err: <nil> Nov 8 14:26:15.197752745 (218h45m46.893601498s) [noise=true , address=145.79.227.26:4655 , piece_id=GP4F5L6HH6S4ZEYWS7XRG5KJSYFGOX5JXQBTQEOXFDFSNUBHZJ6A, seg_pos={0 0}, upload=192678]: Done uploading piece. err: <nil> Nov 8 14:26:17.425396997 (218h45m49.12124575s) [noise=true , address=145.79.227.22:4509 , piece_id=6RJJCCT2Z5U3JOWWOBLRIDK5CTWKXU7NR4OIREAQ2JDE2KYDJHOQ, seg_pos={0 0}, upload=192678]: Done uploading piece. err: <nil> Nov 8 14:26:21.856283285 (218h45m53.552132048s) [noise=true , address=145.79.227.31:4798 , piece_id=5Q7NYNXM2IMV7YUCB464FQMBI4L476Y6VKPPOPW7RKNCC5XFDJ2A, seg_pos={0 0}, upload=192678]: Done uploading piece. err: <nil> Nov 8 14:26:23.486413928 (218h45m55.182262681s) [noise=true , address=145.79.227.20:4437 , piece_id=G5RTYGERAVOTQDZGCG6NEKRCQAUOVJNYXO4LEBJ7XHJZ6FDXTULQ, seg_pos={0 0}, upload=192678]: Done uploading piece. err: <nil> Nov 8 14:26:24.659870720 (218h45m56.355719473s) [noise=true , address=145.79.227.26:4653 , piece_id=UIV3WFYLH4JZSIIHZUBL56THDPAYK7DMRUQG5TVTHHEF3XGN6NKQ, seg_pos={0 0}, upload=192678]: Done uploading piece. err: <nil> Nov 8 14:26:34.701060537 (218h46m6.3969093s) [noise=true , address=145.79.227.8:4329 , piece_id=JJFSJ7USPZFTPXVUZVLMXRAHYLARBQZLWWU7ITXFNESYVVL4QBMQ, seg_pos={0 0}, upload=192678]: Done uploading piece. err: <nil> Nov 8 14:26:37.183628509 (218h46m8.879477262s) [noise=true , address=145.79.227.10:4402 , piece_id=IRNQEX73D4AEXNYXEWWKCKEMYFC3YXIPX6HSDYYBNDKBM77ROBEA, seg_pos={0 0}, upload=192678]: Done uploading piece. err: <nil> Nov 8 14:26:37.186238899 (218h46m8.882087662s) [noise=true , address=145.79.227.16:4911 , piece_id=QKITSRMSCHOZLCQ7X6GARTE3GS24G3CZFXJIBKDM3VWOVQLMLEHA, seg_pos={0 0}, upload=192678]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:145.79.227.16:4911): protocol: write tcp 188.166.225.170:56440->145.79.227.16:4911: use of closed network connection; write tcp 188.166.225.170:56440->145.79.227.16:4911: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56440->145.79.227.16:4911: use of closed network connection Nov 8 14:26:37.186462625 (218h46m8.882311378s) [noise=true , address=145.79.227.21:4475 , piece_id=6F7IZNQLORMTKUF576FJA6L7FBG66INEUFPQYHL77WLC6DYF32GQ, seg_pos={0 0}, upload=192678]: Done uploading piece. err: ecclient: upload failed (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk, address:145.79.227.21:4475): protocol: expected piece hash; context deadline exceeded; EOF Nov 8 14:26:37.186739442 (218h46m8.882588195s) [noise=true , address=145.79.227.27:4689 , piece_id=JEEKOLHURPGNFSR4YCWL3UD6BU2MT7VW4S7BU3YKE7BJSUEYLPGQ, seg_pos={0 0}, upload=192678]: Done uploading piece. err: ecclient: upload failed (node:1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, address:145.79.227.27:4689): protocol: write tcp 188.166.225.170:37312->145.79.227.27:4689: use of closed network connection; write tcp 188.166.225.170:37312->145.79.227.27:4689: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37312->145.79.227.27:4689: use of closed network connection Nov 8 14:26:37.186915056 (218h46m8.88276381s) [noise=true , address=145.79.227.7:4295 , piece_id=5OY4NKICVFGAWR7JPOOYIYRXR442IIUDDWTN3MT6KAR4ZHXTKPNQ, seg_pos={0 0}, upload=192678]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:145.79.227.7:4295): protocol: expected piece hash; context deadline exceeded; EOF Nov 8 14:26:37.187072807 (218h46m8.88292156s) [noise=true , address=145.79.227.9:4366 , piece_id=3BINODFKOOG7BYULCICW6S64YZOU5D2Q5MFZYGPSSA34XWM762BQ, seg_pos={0 0}, upload=192678]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: expected piece hash; context deadline exceeded; EOF Nov 8 14:26:37.187107163 (218h46m8.882955916s) [seg_pos={0 0}, upload=192678]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 8 14:26:37.433482306 (218h46m9.129331059s) [upload=192678]: Done uploading segments... Nov 8 14:26:37.433526460 (218h46m9.129375213s) [upload=192678]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 8.320045453s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 7.340676301s trace: https://jaeger.dev.storj.io/trace/3f377bb0d96d25c FAIL (2025-11-08 15:06:42 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Nov 8 14:26:07.882330547 (218h45m39.5781793s) [upload=192678]: Starting upload Nov 8 14:26:07.882433824 (218h45m39.578282567s) [upload=192678]: Done starting upload Nov 8 14:26:07.882492927 (218h45m39.57834167s) [upload=192678]: Uploading segments... Nov 8 14:26:07.882645367 (218h45m39.57849411s) [upload=192678]: Got next segment. Inline: false Nov 8 14:26:07.882698468 (218h45m39.578547221s) [upload=192678]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 8 14:26:07.882704229 (218h45m39.578552982s) [upload=192678]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 8 14:26:08.131091114 (218h45m39.826939897s) [seg_pos={0 0}, upload=192678]: Begin upload segment... Nov 8 14:26:08.131165496 (218h45m39.827014239s) [seg_pos={0 0}, upload=192678]: Done begin upload segment. Nov 8 14:26:08.131220160 (218h45m39.827068913s) [noise=true , address=145.79.227.16:4911 , piece_id=QKITSRMSCHOZLCQ7X6GARTE3GS24G3CZFXJIBKDM3VWOVQLMLEHA, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.131304631 (218h45m39.827153384s) [noise=true , address=145.79.227.27:4689 , piece_id=JEEKOLHURPGNFSR4YCWL3UD6BU2MT7VW4S7BU3YKE7BJSUEYLPGQ, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.132190408 (218h45m39.828039161s) [noise=true , address=145.79.227.7:4295 , piece_id=5OY4NKICVFGAWR7JPOOYIYRXR442IIUDDWTN3MT6KAR4ZHXTKPNQ, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.132569360 (218h45m39.828418113s) [noise=true , address=145.79.227.31:4798 , piece_id=5Q7NYNXM2IMV7YUCB464FQMBI4L476Y6VKPPOPW7RKNCC5XFDJ2A, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.133276446 (218h45m39.829125199s) [noise=true , address=145.79.227.26:4653 , piece_id=UIV3WFYLH4JZSIIHZUBL56THDPAYK7DMRUQG5TVTHHEF3XGN6NKQ, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.134167893 (218h45m39.830016646s) [noise=true , address=145.79.227.26:4655 , piece_id=GP4F5L6HH6S4ZEYWS7XRG5KJSYFGOX5JXQBTQEOXFDFSNUBHZJ6A, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.134314192 (218h45m39.830162945s) [noise=true , address=145.79.227.20:4437 , piece_id=G5RTYGERAVOTQDZGCG6NEKRCQAUOVJNYXO4LEBJ7XHJZ6FDXTULQ, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.135016950 (218h45m39.830865703s) [noise=true , address=145.79.227.10:4402 , piece_id=IRNQEX73D4AEXNYXEWWKCKEMYFC3YXIPX6HSDYYBNDKBM77ROBEA, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.135235305 (218h45m39.831084059s) [noise=true , address=145.79.227.22:4509 , piece_id=6RJJCCT2Z5U3JOWWOBLRIDK5CTWKXU7NR4OIREAQ2JDE2KYDJHOQ, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.135440236 (218h45m39.831288979s) [noise=true , address=145.79.227.2:4113 , piece_id=Q5E3IXSHGFKIFG7EFAPW2XWYP63TO3BC5C7PZJDPXXD2UKDQDJLA, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.136611887 (218h45m39.83246064s) [noise=true , address=77.90.31.15:4699 , piece_id=Z4PXJFTM7XMK62ORTMHCWEJ7OQF6PIYQCMAWJC3DT5RZ7CEH4KXA, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.137027559 (218h45m39.832876312s) [noise=true , address=145.79.227.8:4329 , piece_id=JJFSJ7USPZFTPXVUZVLMXRAHYLARBQZLWWU7ITXFNESYVVL4QBMQ, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.137896133 (218h45m39.833744886s) [noise=true , address=77.90.31.17:4771 , piece_id=LUZ4KXA4YUVTQYKXVEPTEXZ4MFQ5IWFTY3C2LUUVQ3N6DOVO6ZXA, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.139107940 (218h45m39.834956693s) [noise=true , address=145.79.227.21:4475 , piece_id=6F7IZNQLORMTKUF576FJA6L7FBG66INEUFPQYHL77WLC6DYF32GQ, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.139872657 (218h45m39.83572141s) [noise=true , address=145.79.227.9:4366 , piece_id=3BINODFKOOG7BYULCICW6S64YZOU5D2Q5MFZYGPSSA34XWM762BQ, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.140005960 (218h45m39.835854713s) [noise=true , address=77.90.31.16:4734 , piece_id=EKYBQDVNAYTGEPWU443RFLUX7OKCTWBPXA5UMCLZ22M2PBXO3E3Q, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:10.015794232 (218h45m41.711642985s) [upload=192678]: Next returned nil segment Nov 8 14:26:10.015816414 (218h45m41.711665167s) [upload=192678]: Waiting for error group managing segments... Nov 8 14:26:10.314091526 (218h45m42.009940279s) [noise=true , address=77.90.31.16:4734 , piece_id=EKYBQDVNAYTGEPWU443RFLUX7OKCTWBPXA5UMCLZ22M2PBXO3E3Q, seg_pos={0 0}, upload=192678]: Done uploading piece. err: <nil> Nov 8 14:26:10.388439672 (218h45m42.084288465s) [noise=true , address=77.90.31.15:4699 , piece_id=Z4PXJFTM7XMK62ORTMHCWEJ7OQF6PIYQCMAWJC3DT5RZ7CEH4KXA, seg_pos={0 0}, upload=192678]: Done uploading piece. err: <nil> Nov 8 14:26:11.119863095 (218h45m42.815711838s) [noise=true , address=77.90.31.17:4771 , piece_id=LUZ4KXA4YUVTQYKXVEPTEXZ4MFQ5IWFTY3C2LUUVQ3N6DOVO6ZXA, seg_pos={0 0}, upload=192678]: Done uploading piece. err: <nil> Nov 8 14:26:14.806769255 (218h45m46.502618008s) [noise=true , address=145.79.227.2:4113 , piece_id=Q5E3IXSHGFKIFG7EFAPW2XWYP63TO3BC5C7PZJDPXXD2UKDQDJLA, seg_pos={0 0}, upload=192678]: Done uploading piece. err: <nil> Nov 8 14:26:15.197752745 (218h45m46.893601498s) [noise=true , address=145.79.227.26:4655 , piece_id=GP4F5L6HH6S4ZEYWS7XRG5KJSYFGOX5JXQBTQEOXFDFSNUBHZJ6A, seg_pos={0 0}, upload=192678]: Done uploading piece. err: <nil> Nov 8 14:26:17.425396997 (218h45m49.12124575s) [noise=true , address=145.79.227.22:4509 , piece_id=6RJJCCT2Z5U3JOWWOBLRIDK5CTWKXU7NR4OIREAQ2JDE2KYDJHOQ, seg_pos={0 0}, upload=192678]: Done uploading piece. err: <nil> Nov 8 14:26:21.856283285 (218h45m53.552132048s) [noise=true , address=145.79.227.31:4798 , piece_id=5Q7NYNXM2IMV7YUCB464FQMBI4L476Y6VKPPOPW7RKNCC5XFDJ2A, seg_pos={0 0}, upload=192678]: Done uploading piece. err: <nil> Nov 8 14:26:23.486413928 (218h45m55.182262681s) [noise=true , address=145.79.227.20:4437 , piece_id=G5RTYGERAVOTQDZGCG6NEKRCQAUOVJNYXO4LEBJ7XHJZ6FDXTULQ, seg_pos={0 0}, upload=192678]: Done uploading piece. err: <nil> Nov 8 14:26:24.659870720 (218h45m56.355719473s) [noise=true , address=145.79.227.26:4653 , piece_id=UIV3WFYLH4JZSIIHZUBL56THDPAYK7DMRUQG5TVTHHEF3XGN6NKQ, seg_pos={0 0}, upload=192678]: Done uploading piece. err: <nil> Nov 8 14:26:34.701060537 (218h46m6.3969093s) [noise=true , address=145.79.227.8:4329 , piece_id=JJFSJ7USPZFTPXVUZVLMXRAHYLARBQZLWWU7ITXFNESYVVL4QBMQ, seg_pos={0 0}, upload=192678]: Done uploading piece. err: <nil> Nov 8 14:26:37.183628509 (218h46m8.879477262s) [noise=true , address=145.79.227.10:4402 , piece_id=IRNQEX73D4AEXNYXEWWKCKEMYFC3YXIPX6HSDYYBNDKBM77ROBEA, seg_pos={0 0}, upload=192678]: Done uploading piece. err: <nil> Nov 8 14:26:37.186238899 (218h46m8.882087662s) [noise=true , address=145.79.227.16:4911 , piece_id=QKITSRMSCHOZLCQ7X6GARTE3GS24G3CZFXJIBKDM3VWOVQLMLEHA, seg_pos={0 0}, upload=192678]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:145.79.227.16:4911): protocol: write tcp 188.166.225.170:56440->145.79.227.16:4911: use of closed network connection; write tcp 188.166.225.170:56440->145.79.227.16:4911: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56440->145.79.227.16:4911: use of closed network connection Nov 8 14:26:37.186462625 (218h46m8.882311378s) [noise=true , address=145.79.227.21:4475 , piece_id=6F7IZNQLORMTKUF576FJA6L7FBG66INEUFPQYHL77WLC6DYF32GQ, seg_pos={0 0}, upload=192678]: Done uploading piece. err: ecclient: upload failed (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk, address:145.79.227.21:4475): protocol: expected piece hash; context deadline exceeded; EOF Nov 8 14:26:37.186739442 (218h46m8.882588195s) [noise=true , address=145.79.227.27:4689 , piece_id=JEEKOLHURPGNFSR4YCWL3UD6BU2MT7VW4S7BU3YKE7BJSUEYLPGQ, seg_pos={0 0}, upload=192678]: Done uploading piece. err: ecclient: upload failed (node:1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, address:145.79.227.27:4689): protocol: write tcp 188.166.225.170:37312->145.79.227.27:4689: use of closed network connection; write tcp 188.166.225.170:37312->145.79.227.27:4689: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37312->145.79.227.27:4689: use of closed network connection Nov 8 14:26:37.186915056 (218h46m8.88276381s) [noise=true , address=145.79.227.7:4295 , piece_id=5OY4NKICVFGAWR7JPOOYIYRXR442IIUDDWTN3MT6KAR4ZHXTKPNQ, seg_pos={0 0}, upload=192678]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:145.79.227.7:4295): protocol: expected piece hash; context deadline exceeded; EOF Nov 8 14:26:37.187072807 (218h46m8.88292156s) [noise=true , address=145.79.227.9:4366 , piece_id=3BINODFKOOG7BYULCICW6S64YZOU5D2Q5MFZYGPSSA34XWM762BQ, seg_pos={0 0}, upload=192678]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: expected piece hash; context deadline exceeded; EOF Nov 8 14:26:37.187107163 (218h46m8.882955916s) [seg_pos={0 0}, upload=192678]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 8 14:26:37.433482306 (218h46m9.129331059s) [upload=192678]: Done uploading segments... Nov 8 14:26:37.433526460 (218h46m9.129375213s) [upload=192678]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 7.340704616s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 11.77592024s trace: https://jaeger.dev.storj.io/trace/3f377bb0d96d25c FAIL (2025-11-08 15:06:00 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Nov 8 14:26:07.882330547 (218h45m39.5781793s) [upload=192678]: Starting upload Nov 8 14:26:07.882433824 (218h45m39.578282567s) [upload=192678]: Done starting upload Nov 8 14:26:07.882492927 (218h45m39.57834167s) [upload=192678]: Uploading segments... Nov 8 14:26:07.882645367 (218h45m39.57849411s) [upload=192678]: Got next segment. Inline: false Nov 8 14:26:07.882698468 (218h45m39.578547221s) [upload=192678]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 8 14:26:07.882704229 (218h45m39.578552982s) [upload=192678]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 8 14:26:08.131091114 (218h45m39.826939897s) [seg_pos={0 0}, upload=192678]: Begin upload segment... Nov 8 14:26:08.131165496 (218h45m39.827014239s) [seg_pos={0 0}, upload=192678]: Done begin upload segment. Nov 8 14:26:08.131220160 (218h45m39.827068913s) [noise=true , address=145.79.227.16:4911 , piece_id=QKITSRMSCHOZLCQ7X6GARTE3GS24G3CZFXJIBKDM3VWOVQLMLEHA, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.131304631 (218h45m39.827153384s) [noise=true , address=145.79.227.27:4689 , piece_id=JEEKOLHURPGNFSR4YCWL3UD6BU2MT7VW4S7BU3YKE7BJSUEYLPGQ, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.132190408 (218h45m39.828039161s) [noise=true , address=145.79.227.7:4295 , piece_id=5OY4NKICVFGAWR7JPOOYIYRXR442IIUDDWTN3MT6KAR4ZHXTKPNQ, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.132569360 (218h45m39.828418113s) [noise=true , address=145.79.227.31:4798 , piece_id=5Q7NYNXM2IMV7YUCB464FQMBI4L476Y6VKPPOPW7RKNCC5XFDJ2A, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.133276446 (218h45m39.829125199s) [noise=true , address=145.79.227.26:4653 , piece_id=UIV3WFYLH4JZSIIHZUBL56THDPAYK7DMRUQG5TVTHHEF3XGN6NKQ, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.134167893 (218h45m39.830016646s) [noise=true , address=145.79.227.26:4655 , piece_id=GP4F5L6HH6S4ZEYWS7XRG5KJSYFGOX5JXQBTQEOXFDFSNUBHZJ6A, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.134314192 (218h45m39.830162945s) [noise=true , address=145.79.227.20:4437 , piece_id=G5RTYGERAVOTQDZGCG6NEKRCQAUOVJNYXO4LEBJ7XHJZ6FDXTULQ, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.135016950 (218h45m39.830865703s) [noise=true , address=145.79.227.10:4402 , piece_id=IRNQEX73D4AEXNYXEWWKCKEMYFC3YXIPX6HSDYYBNDKBM77ROBEA, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.135235305 (218h45m39.831084059s) [noise=true , address=145.79.227.22:4509 , piece_id=6RJJCCT2Z5U3JOWWOBLRIDK5CTWKXU7NR4OIREAQ2JDE2KYDJHOQ, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.135440236 (218h45m39.831288979s) [noise=true , address=145.79.227.2:4113 , piece_id=Q5E3IXSHGFKIFG7EFAPW2XWYP63TO3BC5C7PZJDPXXD2UKDQDJLA, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.136611887 (218h45m39.83246064s) [noise=true , address=77.90.31.15:4699 , piece_id=Z4PXJFTM7XMK62ORTMHCWEJ7OQF6PIYQCMAWJC3DT5RZ7CEH4KXA, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.137027559 (218h45m39.832876312s) [noise=true , address=145.79.227.8:4329 , piece_id=JJFSJ7USPZFTPXVUZVLMXRAHYLARBQZLWWU7ITXFNESYVVL4QBMQ, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.137896133 (218h45m39.833744886s) [noise=true , address=77.90.31.17:4771 , piece_id=LUZ4KXA4YUVTQYKXVEPTEXZ4MFQ5IWFTY3C2LUUVQ3N6DOVO6ZXA, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.139107940 (218h45m39.834956693s) [noise=true , address=145.79.227.21:4475 , piece_id=6F7IZNQLORMTKUF576FJA6L7FBG66INEUFPQYHL77WLC6DYF32GQ, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.139872657 (218h45m39.83572141s) [noise=true , address=145.79.227.9:4366 , piece_id=3BINODFKOOG7BYULCICW6S64YZOU5D2Q5MFZYGPSSA34XWM762BQ, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.140005960 (218h45m39.835854713s) [noise=true , address=77.90.31.16:4734 , piece_id=EKYBQDVNAYTGEPWU443RFLUX7OKCTWBPXA5UMCLZ22M2PBXO3E3Q, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:10.015794232 (218h45m41.711642985s) [upload=192678]: Next returned nil segment Nov 8 14:26:10.015816414 (218h45m41.711665167s) [upload=192678]: Waiting for error group managing segments... Nov 8 14:26:10.314091526 (218h45m42.009940279s) [noise=true , address=77.90.31.16:4734 , piece_id=EKYBQDVNAYTGEPWU443RFLUX7OKCTWBPXA5UMCLZ22M2PBXO3E3Q, seg_pos={0 0}, upload=192678]: Done uploading piece. err: <nil> Nov 8 14:26:10.388439672 (218h45m42.084288465s) [noise=true , address=77.90.31.15:4699 , piece_id=Z4PXJFTM7XMK62ORTMHCWEJ7OQF6PIYQCMAWJC3DT5RZ7CEH4KXA, seg_pos={0 0}, upload=192678]: Done uploading piece. err: <nil> Nov 8 14:26:11.119863095 (218h45m42.815711838s) [noise=true , address=77.90.31.17:4771 , piece_id=LUZ4KXA4YUVTQYKXVEPTEXZ4MFQ5IWFTY3C2LUUVQ3N6DOVO6ZXA, seg_pos={0 0}, upload=192678]: Done uploading piece. err: <nil> Nov 8 14:26:14.806769255 (218h45m46.502618008s) [noise=true , address=145.79.227.2:4113 , piece_id=Q5E3IXSHGFKIFG7EFAPW2XWYP63TO3BC5C7PZJDPXXD2UKDQDJLA, seg_pos={0 0}, upload=192678]: Done uploading piece. err: <nil> Nov 8 14:26:15.197752745 (218h45m46.893601498s) [noise=true , address=145.79.227.26:4655 , piece_id=GP4F5L6HH6S4ZEYWS7XRG5KJSYFGOX5JXQBTQEOXFDFSNUBHZJ6A, seg_pos={0 0}, upload=192678]: Done uploading piece. err: <nil> Nov 8 14:26:17.425396997 (218h45m49.12124575s) [noise=true , address=145.79.227.22:4509 , piece_id=6RJJCCT2Z5U3JOWWOBLRIDK5CTWKXU7NR4OIREAQ2JDE2KYDJHOQ, seg_pos={0 0}, upload=192678]: Done uploading piece. err: <nil> Nov 8 14:26:21.856283285 (218h45m53.552132048s) [noise=true , address=145.79.227.31:4798 , piece_id=5Q7NYNXM2IMV7YUCB464FQMBI4L476Y6VKPPOPW7RKNCC5XFDJ2A, seg_pos={0 0}, upload=192678]: Done uploading piece. err: <nil> Nov 8 14:26:23.486413928 (218h45m55.182262681s) [noise=true , address=145.79.227.20:4437 , piece_id=G5RTYGERAVOTQDZGCG6NEKRCQAUOVJNYXO4LEBJ7XHJZ6FDXTULQ, seg_pos={0 0}, upload=192678]: Done uploading piece. err: <nil> Nov 8 14:26:24.659870720 (218h45m56.355719473s) [noise=true , address=145.79.227.26:4653 , piece_id=UIV3WFYLH4JZSIIHZUBL56THDPAYK7DMRUQG5TVTHHEF3XGN6NKQ, seg_pos={0 0}, upload=192678]: Done uploading piece. err: <nil> Nov 8 14:26:34.701060537 (218h46m6.3969093s) [noise=true , address=145.79.227.8:4329 , piece_id=JJFSJ7USPZFTPXVUZVLMXRAHYLARBQZLWWU7ITXFNESYVVL4QBMQ, seg_pos={0 0}, upload=192678]: Done uploading piece. err: <nil> Nov 8 14:26:37.183628509 (218h46m8.879477262s) [noise=true , address=145.79.227.10:4402 , piece_id=IRNQEX73D4AEXNYXEWWKCKEMYFC3YXIPX6HSDYYBNDKBM77ROBEA, seg_pos={0 0}, upload=192678]: Done uploading piece. err: <nil> Nov 8 14:26:37.186238899 (218h46m8.882087662s) [noise=true , address=145.79.227.16:4911 , piece_id=QKITSRMSCHOZLCQ7X6GARTE3GS24G3CZFXJIBKDM3VWOVQLMLEHA, seg_pos={0 0}, upload=192678]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:145.79.227.16:4911): protocol: write tcp 188.166.225.170:56440->145.79.227.16:4911: use of closed network connection; write tcp 188.166.225.170:56440->145.79.227.16:4911: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56440->145.79.227.16:4911: use of closed network connection Nov 8 14:26:37.186462625 (218h46m8.882311378s) [noise=true , address=145.79.227.21:4475 , piece_id=6F7IZNQLORMTKUF576FJA6L7FBG66INEUFPQYHL77WLC6DYF32GQ, seg_pos={0 0}, upload=192678]: Done uploading piece. err: ecclient: upload failed (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk, address:145.79.227.21:4475): protocol: expected piece hash; context deadline exceeded; EOF Nov 8 14:26:37.186739442 (218h46m8.882588195s) [noise=true , address=145.79.227.27:4689 , piece_id=JEEKOLHURPGNFSR4YCWL3UD6BU2MT7VW4S7BU3YKE7BJSUEYLPGQ, seg_pos={0 0}, upload=192678]: Done uploading piece. err: ecclient: upload failed (node:1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, address:145.79.227.27:4689): protocol: write tcp 188.166.225.170:37312->145.79.227.27:4689: use of closed network connection; write tcp 188.166.225.170:37312->145.79.227.27:4689: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37312->145.79.227.27:4689: use of closed network connection Nov 8 14:26:37.186915056 (218h46m8.88276381s) [noise=true , address=145.79.227.7:4295 , piece_id=5OY4NKICVFGAWR7JPOOYIYRXR442IIUDDWTN3MT6KAR4ZHXTKPNQ, seg_pos={0 0}, upload=192678]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:145.79.227.7:4295): protocol: expected piece hash; context deadline exceeded; EOF Nov 8 14:26:37.187072807 (218h46m8.88292156s) [noise=true , address=145.79.227.9:4366 , piece_id=3BINODFKOOG7BYULCICW6S64YZOU5D2Q5MFZYGPSSA34XWM762BQ, seg_pos={0 0}, upload=192678]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: expected piece hash; context deadline exceeded; EOF Nov 8 14:26:37.187107163 (218h46m8.882955916s) [seg_pos={0 0}, upload=192678]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 8 14:26:37.433482306 (218h46m9.129331059s) [upload=192678]: Done uploading segments... Nov 8 14:26:37.433526460 (218h46m9.129375213s) [upload=192678]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 11.775981947s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 12.382320418s trace: https://jaeger.dev.storj.io/trace/3f377bb0d96d25c FAIL (2025-11-08 15:04:53 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Nov 8 14:26:07.882330547 (218h45m39.5781793s) [upload=192678]: Starting upload Nov 8 14:26:07.882433824 (218h45m39.578282567s) [upload=192678]: Done starting upload Nov 8 14:26:07.882492927 (218h45m39.57834167s) [upload=192678]: Uploading segments... Nov 8 14:26:07.882645367 (218h45m39.57849411s) [upload=192678]: Got next segment. Inline: false Nov 8 14:26:07.882698468 (218h45m39.578547221s) [upload=192678]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 8 14:26:07.882704229 (218h45m39.578552982s) [upload=192678]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 8 14:26:08.131091114 (218h45m39.826939897s) [seg_pos={0 0}, upload=192678]: Begin upload segment... Nov 8 14:26:08.131165496 (218h45m39.827014239s) [seg_pos={0 0}, upload=192678]: Done begin upload segment. Nov 8 14:26:08.131220160 (218h45m39.827068913s) [noise=true , address=145.79.227.16:4911 , piece_id=QKITSRMSCHOZLCQ7X6GARTE3GS24G3CZFXJIBKDM3VWOVQLMLEHA, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.131304631 (218h45m39.827153384s) [noise=true , address=145.79.227.27:4689 , piece_id=JEEKOLHURPGNFSR4YCWL3UD6BU2MT7VW4S7BU3YKE7BJSUEYLPGQ, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.132190408 (218h45m39.828039161s) [noise=true , address=145.79.227.7:4295 , piece_id=5OY4NKICVFGAWR7JPOOYIYRXR442IIUDDWTN3MT6KAR4ZHXTKPNQ, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.132569360 (218h45m39.828418113s) [noise=true , address=145.79.227.31:4798 , piece_id=5Q7NYNXM2IMV7YUCB464FQMBI4L476Y6VKPPOPW7RKNCC5XFDJ2A, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.133276446 (218h45m39.829125199s) [noise=true , address=145.79.227.26:4653 , piece_id=UIV3WFYLH4JZSIIHZUBL56THDPAYK7DMRUQG5TVTHHEF3XGN6NKQ, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.134167893 (218h45m39.830016646s) [noise=true , address=145.79.227.26:4655 , piece_id=GP4F5L6HH6S4ZEYWS7XRG5KJSYFGOX5JXQBTQEOXFDFSNUBHZJ6A, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.134314192 (218h45m39.830162945s) [noise=true , address=145.79.227.20:4437 , piece_id=G5RTYGERAVOTQDZGCG6NEKRCQAUOVJNYXO4LEBJ7XHJZ6FDXTULQ, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.135016950 (218h45m39.830865703s) [noise=true , address=145.79.227.10:4402 , piece_id=IRNQEX73D4AEXNYXEWWKCKEMYFC3YXIPX6HSDYYBNDKBM77ROBEA, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.135235305 (218h45m39.831084059s) [noise=true , address=145.79.227.22:4509 , piece_id=6RJJCCT2Z5U3JOWWOBLRIDK5CTWKXU7NR4OIREAQ2JDE2KYDJHOQ, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.135440236 (218h45m39.831288979s) [noise=true , address=145.79.227.2:4113 , piece_id=Q5E3IXSHGFKIFG7EFAPW2XWYP63TO3BC5C7PZJDPXXD2UKDQDJLA, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.136611887 (218h45m39.83246064s) [noise=true , address=77.90.31.15:4699 , piece_id=Z4PXJFTM7XMK62ORTMHCWEJ7OQF6PIYQCMAWJC3DT5RZ7CEH4KXA, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.137027559 (218h45m39.832876312s) [noise=true , address=145.79.227.8:4329 , piece_id=JJFSJ7USPZFTPXVUZVLMXRAHYLARBQZLWWU7ITXFNESYVVL4QBMQ, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.137896133 (218h45m39.833744886s) [noise=true , address=77.90.31.17:4771 , piece_id=LUZ4KXA4YUVTQYKXVEPTEXZ4MFQ5IWFTY3C2LUUVQ3N6DOVO6ZXA, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.139107940 (218h45m39.834956693s) [noise=true , address=145.79.227.21:4475 , piece_id=6F7IZNQLORMTKUF576FJA6L7FBG66INEUFPQYHL77WLC6DYF32GQ, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.139872657 (218h45m39.83572141s) [noise=true , address=145.79.227.9:4366 , piece_id=3BINODFKOOG7BYULCICW6S64YZOU5D2Q5MFZYGPSSA34XWM762BQ, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.140005960 (218h45m39.835854713s) [noise=true , address=77.90.31.16:4734 , piece_id=EKYBQDVNAYTGEPWU443RFLUX7OKCTWBPXA5UMCLZ22M2PBXO3E3Q, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:10.015794232 (218h45m41.711642985s) [upload=192678]: Next returned nil segment Nov 8 14:26:10.015816414 (218h45m41.711665167s) [upload=192678]: Waiting for error group managing segments... Nov 8 14:26:10.314091526 (218h45m42.009940279s) [noise=true , address=77.90.31.16:4734 , piece_id=EKYBQDVNAYTGEPWU443RFLUX7OKCTWBPXA5UMCLZ22M2PBXO3E3Q, seg_pos={0 0}, upload=192678]: Done uploading piece. err: <nil> Nov 8 14:26:10.388439672 (218h45m42.084288465s) [noise=true , address=77.90.31.15:4699 , piece_id=Z4PXJFTM7XMK62ORTMHCWEJ7OQF6PIYQCMAWJC3DT5RZ7CEH4KXA, seg_pos={0 0}, upload=192678]: Done uploading piece. err: <nil> Nov 8 14:26:11.119863095 (218h45m42.815711838s) [noise=true , address=77.90.31.17:4771 , piece_id=LUZ4KXA4YUVTQYKXVEPTEXZ4MFQ5IWFTY3C2LUUVQ3N6DOVO6ZXA, seg_pos={0 0}, upload=192678]: Done uploading piece. err: <nil> Nov 8 14:26:14.806769255 (218h45m46.502618008s) [noise=true , address=145.79.227.2:4113 , piece_id=Q5E3IXSHGFKIFG7EFAPW2XWYP63TO3BC5C7PZJDPXXD2UKDQDJLA, seg_pos={0 0}, upload=192678]: Done uploading piece. err: <nil> Nov 8 14:26:15.197752745 (218h45m46.893601498s) [noise=true , address=145.79.227.26:4655 , piece_id=GP4F5L6HH6S4ZEYWS7XRG5KJSYFGOX5JXQBTQEOXFDFSNUBHZJ6A, seg_pos={0 0}, upload=192678]: Done uploading piece. err: <nil> Nov 8 14:26:17.425396997 (218h45m49.12124575s) [noise=true , address=145.79.227.22:4509 , piece_id=6RJJCCT2Z5U3JOWWOBLRIDK5CTWKXU7NR4OIREAQ2JDE2KYDJHOQ, seg_pos={0 0}, upload=192678]: Done uploading piece. err: <nil> Nov 8 14:26:21.856283285 (218h45m53.552132048s) [noise=true , address=145.79.227.31:4798 , piece_id=5Q7NYNXM2IMV7YUCB464FQMBI4L476Y6VKPPOPW7RKNCC5XFDJ2A, seg_pos={0 0}, upload=192678]: Done uploading piece. err: <nil> Nov 8 14:26:23.486413928 (218h45m55.182262681s) [noise=true , address=145.79.227.20:4437 , piece_id=G5RTYGERAVOTQDZGCG6NEKRCQAUOVJNYXO4LEBJ7XHJZ6FDXTULQ, seg_pos={0 0}, upload=192678]: Done uploading piece. err: <nil> Nov 8 14:26:24.659870720 (218h45m56.355719473s) [noise=true , address=145.79.227.26:4653 , piece_id=UIV3WFYLH4JZSIIHZUBL56THDPAYK7DMRUQG5TVTHHEF3XGN6NKQ, seg_pos={0 0}, upload=192678]: Done uploading piece. err: <nil> Nov 8 14:26:34.701060537 (218h46m6.3969093s) [noise=true , address=145.79.227.8:4329 , piece_id=JJFSJ7USPZFTPXVUZVLMXRAHYLARBQZLWWU7ITXFNESYVVL4QBMQ, seg_pos={0 0}, upload=192678]: Done uploading piece. err: <nil> Nov 8 14:26:37.183628509 (218h46m8.879477262s) [noise=true , address=145.79.227.10:4402 , piece_id=IRNQEX73D4AEXNYXEWWKCKEMYFC3YXIPX6HSDYYBNDKBM77ROBEA, seg_pos={0 0}, upload=192678]: Done uploading piece. err: <nil> Nov 8 14:26:37.186238899 (218h46m8.882087662s) [noise=true , address=145.79.227.16:4911 , piece_id=QKITSRMSCHOZLCQ7X6GARTE3GS24G3CZFXJIBKDM3VWOVQLMLEHA, seg_pos={0 0}, upload=192678]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:145.79.227.16:4911): protocol: write tcp 188.166.225.170:56440->145.79.227.16:4911: use of closed network connection; write tcp 188.166.225.170:56440->145.79.227.16:4911: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56440->145.79.227.16:4911: use of closed network connection Nov 8 14:26:37.186462625 (218h46m8.882311378s) [noise=true , address=145.79.227.21:4475 , piece_id=6F7IZNQLORMTKUF576FJA6L7FBG66INEUFPQYHL77WLC6DYF32GQ, seg_pos={0 0}, upload=192678]: Done uploading piece. err: ecclient: upload failed (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk, address:145.79.227.21:4475): protocol: expected piece hash; context deadline exceeded; EOF Nov 8 14:26:37.186739442 (218h46m8.882588195s) [noise=true , address=145.79.227.27:4689 , piece_id=JEEKOLHURPGNFSR4YCWL3UD6BU2MT7VW4S7BU3YKE7BJSUEYLPGQ, seg_pos={0 0}, upload=192678]: Done uploading piece. err: ecclient: upload failed (node:1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, address:145.79.227.27:4689): protocol: write tcp 188.166.225.170:37312->145.79.227.27:4689: use of closed network connection; write tcp 188.166.225.170:37312->145.79.227.27:4689: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37312->145.79.227.27:4689: use of closed network connection Nov 8 14:26:37.186915056 (218h46m8.88276381s) [noise=true , address=145.79.227.7:4295 , piece_id=5OY4NKICVFGAWR7JPOOYIYRXR442IIUDDWTN3MT6KAR4ZHXTKPNQ, seg_pos={0 0}, upload=192678]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:145.79.227.7:4295): protocol: expected piece hash; context deadline exceeded; EOF Nov 8 14:26:37.187072807 (218h46m8.88292156s) [noise=true , address=145.79.227.9:4366 , piece_id=3BINODFKOOG7BYULCICW6S64YZOU5D2Q5MFZYGPSSA34XWM762BQ, seg_pos={0 0}, upload=192678]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: expected piece hash; context deadline exceeded; EOF Nov 8 14:26:37.187107163 (218h46m8.882955916s) [seg_pos={0 0}, upload=192678]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 8 14:26:37.433482306 (218h46m9.129331059s) [upload=192678]: Done uploading segments... Nov 8 14:26:37.433526460 (218h46m9.129375213s) [upload=192678]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 12.382352871s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 9.691619476s trace: https://jaeger.dev.storj.io/trace/3f377bb0d96d25c FAIL (2025-11-08 15:04:34 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Nov 8 14:26:07.882330547 (218h45m39.5781793s) [upload=192678]: Starting upload Nov 8 14:26:07.882433824 (218h45m39.578282567s) [upload=192678]: Done starting upload Nov 8 14:26:07.882492927 (218h45m39.57834167s) [upload=192678]: Uploading segments... Nov 8 14:26:07.882645367 (218h45m39.57849411s) [upload=192678]: Got next segment. Inline: false Nov 8 14:26:07.882698468 (218h45m39.578547221s) [upload=192678]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 8 14:26:07.882704229 (218h45m39.578552982s) [upload=192678]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 8 14:26:08.131091114 (218h45m39.826939897s) [seg_pos={0 0}, upload=192678]: Begin upload segment... Nov 8 14:26:08.131165496 (218h45m39.827014239s) [seg_pos={0 0}, upload=192678]: Done begin upload segment. Nov 8 14:26:08.131220160 (218h45m39.827068913s) [noise=true , address=145.79.227.16:4911 , piece_id=QKITSRMSCHOZLCQ7X6GARTE3GS24G3CZFXJIBKDM3VWOVQLMLEHA, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.131304631 (218h45m39.827153384s) [noise=true , address=145.79.227.27:4689 , piece_id=JEEKOLHURPGNFSR4YCWL3UD6BU2MT7VW4S7BU3YKE7BJSUEYLPGQ, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.132190408 (218h45m39.828039161s) [noise=true , address=145.79.227.7:4295 , piece_id=5OY4NKICVFGAWR7JPOOYIYRXR442IIUDDWTN3MT6KAR4ZHXTKPNQ, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.132569360 (218h45m39.828418113s) [noise=true , address=145.79.227.31:4798 , piece_id=5Q7NYNXM2IMV7YUCB464FQMBI4L476Y6VKPPOPW7RKNCC5XFDJ2A, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.133276446 (218h45m39.829125199s) [noise=true , address=145.79.227.26:4653 , piece_id=UIV3WFYLH4JZSIIHZUBL56THDPAYK7DMRUQG5TVTHHEF3XGN6NKQ, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.134167893 (218h45m39.830016646s) [noise=true , address=145.79.227.26:4655 , piece_id=GP4F5L6HH6S4ZEYWS7XRG5KJSYFGOX5JXQBTQEOXFDFSNUBHZJ6A, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.134314192 (218h45m39.830162945s) [noise=true , address=145.79.227.20:4437 , piece_id=G5RTYGERAVOTQDZGCG6NEKRCQAUOVJNYXO4LEBJ7XHJZ6FDXTULQ, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.135016950 (218h45m39.830865703s) [noise=true , address=145.79.227.10:4402 , piece_id=IRNQEX73D4AEXNYXEWWKCKEMYFC3YXIPX6HSDYYBNDKBM77ROBEA, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.135235305 (218h45m39.831084059s) [noise=true , address=145.79.227.22:4509 , piece_id=6RJJCCT2Z5U3JOWWOBLRIDK5CTWKXU7NR4OIREAQ2JDE2KYDJHOQ, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.135440236 (218h45m39.831288979s) [noise=true , address=145.79.227.2:4113 , piece_id=Q5E3IXSHGFKIFG7EFAPW2XWYP63TO3BC5C7PZJDPXXD2UKDQDJLA, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.136611887 (218h45m39.83246064s) [noise=true , address=77.90.31.15:4699 , piece_id=Z4PXJFTM7XMK62ORTMHCWEJ7OQF6PIYQCMAWJC3DT5RZ7CEH4KXA, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.137027559 (218h45m39.832876312s) [noise=true , address=145.79.227.8:4329 , piece_id=JJFSJ7USPZFTPXVUZVLMXRAHYLARBQZLWWU7ITXFNESYVVL4QBMQ, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.137896133 (218h45m39.833744886s) [noise=true , address=77.90.31.17:4771 , piece_id=LUZ4KXA4YUVTQYKXVEPTEXZ4MFQ5IWFTY3C2LUUVQ3N6DOVO6ZXA, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.139107940 (218h45m39.834956693s) [noise=true , address=145.79.227.21:4475 , piece_id=6F7IZNQLORMTKUF576FJA6L7FBG66INEUFPQYHL77WLC6DYF32GQ, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.139872657 (218h45m39.83572141s) [noise=true , address=145.79.227.9:4366 , piece_id=3BINODFKOOG7BYULCICW6S64YZOU5D2Q5MFZYGPSSA34XWM762BQ, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.140005960 (218h45m39.835854713s) [noise=true , address=77.90.31.16:4734 , piece_id=EKYBQDVNAYTGEPWU443RFLUX7OKCTWBPXA5UMCLZ22M2PBXO3E3Q, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:10.015794232 (218h45m41.711642985s) [upload=192678]: Next returned nil segment Nov 8 14:26:10.015816414 (218h45m41.711665167s) [upload=192678]: Waiting for error group managing segments... Nov 8 14:26:10.314091526 (218h45m42.009940279s) [noise=true , address=77.90.31.16:4734 , piece_id=EKYBQDVNAYTGEPWU443RFLUX7OKCTWBPXA5UMCLZ22M2PBXO3E3Q, seg_pos={0 0}, upload=192678]: Done uploading piece. err: <nil> Nov 8 14:26:10.388439672 (218h45m42.084288465s) [noise=true , address=77.90.31.15:4699 , piece_id=Z4PXJFTM7XMK62ORTMHCWEJ7OQF6PIYQCMAWJC3DT5RZ7CEH4KXA, seg_pos={0 0}, upload=192678]: Done uploading piece. err: <nil> Nov 8 14:26:11.119863095 (218h45m42.815711838s) [noise=true , address=77.90.31.17:4771 , piece_id=LUZ4KXA4YUVTQYKXVEPTEXZ4MFQ5IWFTY3C2LUUVQ3N6DOVO6ZXA, seg_pos={0 0}, upload=192678]: Done uploading piece. err: <nil> Nov 8 14:26:14.806769255 (218h45m46.502618008s) [noise=true , address=145.79.227.2:4113 , piece_id=Q5E3IXSHGFKIFG7EFAPW2XWYP63TO3BC5C7PZJDPXXD2UKDQDJLA, seg_pos={0 0}, upload=192678]: Done uploading piece. err: <nil> Nov 8 14:26:15.197752745 (218h45m46.893601498s) [noise=true , address=145.79.227.26:4655 , piece_id=GP4F5L6HH6S4ZEYWS7XRG5KJSYFGOX5JXQBTQEOXFDFSNUBHZJ6A, seg_pos={0 0}, upload=192678]: Done uploading piece. err: <nil> Nov 8 14:26:17.425396997 (218h45m49.12124575s) [noise=true , address=145.79.227.22:4509 , piece_id=6RJJCCT2Z5U3JOWWOBLRIDK5CTWKXU7NR4OIREAQ2JDE2KYDJHOQ, seg_pos={0 0}, upload=192678]: Done uploading piece. err: <nil> Nov 8 14:26:21.856283285 (218h45m53.552132048s) [noise=true , address=145.79.227.31:4798 , piece_id=5Q7NYNXM2IMV7YUCB464FQMBI4L476Y6VKPPOPW7RKNCC5XFDJ2A, seg_pos={0 0}, upload=192678]: Done uploading piece. err: <nil> Nov 8 14:26:23.486413928 (218h45m55.182262681s) [noise=true , address=145.79.227.20:4437 , piece_id=G5RTYGERAVOTQDZGCG6NEKRCQAUOVJNYXO4LEBJ7XHJZ6FDXTULQ, seg_pos={0 0}, upload=192678]: Done uploading piece. err: <nil> Nov 8 14:26:24.659870720 (218h45m56.355719473s) [noise=true , address=145.79.227.26:4653 , piece_id=UIV3WFYLH4JZSIIHZUBL56THDPAYK7DMRUQG5TVTHHEF3XGN6NKQ, seg_pos={0 0}, upload=192678]: Done uploading piece. err: <nil> Nov 8 14:26:34.701060537 (218h46m6.3969093s) [noise=true , address=145.79.227.8:4329 , piece_id=JJFSJ7USPZFTPXVUZVLMXRAHYLARBQZLWWU7ITXFNESYVVL4QBMQ, seg_pos={0 0}, upload=192678]: Done uploading piece. err: <nil> Nov 8 14:26:37.183628509 (218h46m8.879477262s) [noise=true , address=145.79.227.10:4402 , piece_id=IRNQEX73D4AEXNYXEWWKCKEMYFC3YXIPX6HSDYYBNDKBM77ROBEA, seg_pos={0 0}, upload=192678]: Done uploading piece. err: <nil> Nov 8 14:26:37.186238899 (218h46m8.882087662s) [noise=true , address=145.79.227.16:4911 , piece_id=QKITSRMSCHOZLCQ7X6GARTE3GS24G3CZFXJIBKDM3VWOVQLMLEHA, seg_pos={0 0}, upload=192678]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:145.79.227.16:4911): protocol: write tcp 188.166.225.170:56440->145.79.227.16:4911: use of closed network connection; write tcp 188.166.225.170:56440->145.79.227.16:4911: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56440->145.79.227.16:4911: use of closed network connection Nov 8 14:26:37.186462625 (218h46m8.882311378s) [noise=true , address=145.79.227.21:4475 , piece_id=6F7IZNQLORMTKUF576FJA6L7FBG66INEUFPQYHL77WLC6DYF32GQ, seg_pos={0 0}, upload=192678]: Done uploading piece. err: ecclient: upload failed (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk, address:145.79.227.21:4475): protocol: expected piece hash; context deadline exceeded; EOF Nov 8 14:26:37.186739442 (218h46m8.882588195s) [noise=true , address=145.79.227.27:4689 , piece_id=JEEKOLHURPGNFSR4YCWL3UD6BU2MT7VW4S7BU3YKE7BJSUEYLPGQ, seg_pos={0 0}, upload=192678]: Done uploading piece. err: ecclient: upload failed (node:1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, address:145.79.227.27:4689): protocol: write tcp 188.166.225.170:37312->145.79.227.27:4689: use of closed network connection; write tcp 188.166.225.170:37312->145.79.227.27:4689: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37312->145.79.227.27:4689: use of closed network connection Nov 8 14:26:37.186915056 (218h46m8.88276381s) [noise=true , address=145.79.227.7:4295 , piece_id=5OY4NKICVFGAWR7JPOOYIYRXR442IIUDDWTN3MT6KAR4ZHXTKPNQ, seg_pos={0 0}, upload=192678]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:145.79.227.7:4295): protocol: expected piece hash; context deadline exceeded; EOF Nov 8 14:26:37.187072807 (218h46m8.88292156s) [noise=true , address=145.79.227.9:4366 , piece_id=3BINODFKOOG7BYULCICW6S64YZOU5D2Q5MFZYGPSSA34XWM762BQ, seg_pos={0 0}, upload=192678]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: expected piece hash; context deadline exceeded; EOF Nov 8 14:26:37.187107163 (218h46m8.882955916s) [seg_pos={0 0}, upload=192678]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 8 14:26:37.433482306 (218h46m9.129331059s) [upload=192678]: Done uploading segments... Nov 8 14:26:37.433526460 (218h46m9.129375213s) [upload=192678]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 9.691651937s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 7.748270978s trace: https://jaeger.dev.storj.io/trace/3f377bb0d96d25c FAIL (2025-11-08 15:03:31 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Nov 8 14:26:07.882330547 (218h45m39.5781793s) [upload=192678]: Starting upload Nov 8 14:26:07.882433824 (218h45m39.578282567s) [upload=192678]: Done starting upload Nov 8 14:26:07.882492927 (218h45m39.57834167s) [upload=192678]: Uploading segments... Nov 8 14:26:07.882645367 (218h45m39.57849411s) [upload=192678]: Got next segment. Inline: false Nov 8 14:26:07.882698468 (218h45m39.578547221s) [upload=192678]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 8 14:26:07.882704229 (218h45m39.578552982s) [upload=192678]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 8 14:26:08.131091114 (218h45m39.826939897s) [seg_pos={0 0}, upload=192678]: Begin upload segment... Nov 8 14:26:08.131165496 (218h45m39.827014239s) [seg_pos={0 0}, upload=192678]: Done begin upload segment. Nov 8 14:26:08.131220160 (218h45m39.827068913s) [noise=true , address=145.79.227.16:4911 , piece_id=QKITSRMSCHOZLCQ7X6GARTE3GS24G3CZFXJIBKDM3VWOVQLMLEHA, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.131304631 (218h45m39.827153384s) [noise=true , address=145.79.227.27:4689 , piece_id=JEEKOLHURPGNFSR4YCWL3UD6BU2MT7VW4S7BU3YKE7BJSUEYLPGQ, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.132190408 (218h45m39.828039161s) [noise=true , address=145.79.227.7:4295 , piece_id=5OY4NKICVFGAWR7JPOOYIYRXR442IIUDDWTN3MT6KAR4ZHXTKPNQ, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.132569360 (218h45m39.828418113s) [noise=true , address=145.79.227.31:4798 , piece_id=5Q7NYNXM2IMV7YUCB464FQMBI4L476Y6VKPPOPW7RKNCC5XFDJ2A, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.133276446 (218h45m39.829125199s) [noise=true , address=145.79.227.26:4653 , piece_id=UIV3WFYLH4JZSIIHZUBL56THDPAYK7DMRUQG5TVTHHEF3XGN6NKQ, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.134167893 (218h45m39.830016646s) [noise=true , address=145.79.227.26:4655 , piece_id=GP4F5L6HH6S4ZEYWS7XRG5KJSYFGOX5JXQBTQEOXFDFSNUBHZJ6A, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.134314192 (218h45m39.830162945s) [noise=true , address=145.79.227.20:4437 , piece_id=G5RTYGERAVOTQDZGCG6NEKRCQAUOVJNYXO4LEBJ7XHJZ6FDXTULQ, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.135016950 (218h45m39.830865703s) [noise=true , address=145.79.227.10:4402 , piece_id=IRNQEX73D4AEXNYXEWWKCKEMYFC3YXIPX6HSDYYBNDKBM77ROBEA, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.135235305 (218h45m39.831084059s) [noise=true , address=145.79.227.22:4509 , piece_id=6RJJCCT2Z5U3JOWWOBLRIDK5CTWKXU7NR4OIREAQ2JDE2KYDJHOQ, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.135440236 (218h45m39.831288979s) [noise=true , address=145.79.227.2:4113 , piece_id=Q5E3IXSHGFKIFG7EFAPW2XWYP63TO3BC5C7PZJDPXXD2UKDQDJLA, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.136611887 (218h45m39.83246064s) [noise=true , address=77.90.31.15:4699 , piece_id=Z4PXJFTM7XMK62ORTMHCWEJ7OQF6PIYQCMAWJC3DT5RZ7CEH4KXA, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.137027559 (218h45m39.832876312s) [noise=true , address=145.79.227.8:4329 , piece_id=JJFSJ7USPZFTPXVUZVLMXRAHYLARBQZLWWU7ITXFNESYVVL4QBMQ, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.137896133 (218h45m39.833744886s) [noise=true , address=77.90.31.17:4771 , piece_id=LUZ4KXA4YUVTQYKXVEPTEXZ4MFQ5IWFTY3C2LUUVQ3N6DOVO6ZXA, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.139107940 (218h45m39.834956693s) [noise=true , address=145.79.227.21:4475 , piece_id=6F7IZNQLORMTKUF576FJA6L7FBG66INEUFPQYHL77WLC6DYF32GQ, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.139872657 (218h45m39.83572141s) [noise=true , address=145.79.227.9:4366 , piece_id=3BINODFKOOG7BYULCICW6S64YZOU5D2Q5MFZYGPSSA34XWM762BQ, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.140005960 (218h45m39.835854713s) [noise=true , address=77.90.31.16:4734 , piece_id=EKYBQDVNAYTGEPWU443RFLUX7OKCTWBPXA5UMCLZ22M2PBXO3E3Q, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:10.015794232 (218h45m41.711642985s) [upload=192678]: Next returned nil segment Nov 8 14:26:10.015816414 (218h45m41.711665167s) [upload=192678]: Waiting for error group managing segments... Nov 8 14:26:10.314091526 (218h45m42.009940279s) [noise=true , address=77.90.31.16:4734 , piece_id=EKYBQDVNAYTGEPWU443RFLUX7OKCTWBPXA5UMCLZ22M2PBXO3E3Q, seg_pos={0 0}, upload=192678]: Done uploading piece. err: <nil> Nov 8 14:26:10.388439672 (218h45m42.084288465s) [noise=true , address=77.90.31.15:4699 , piece_id=Z4PXJFTM7XMK62ORTMHCWEJ7OQF6PIYQCMAWJC3DT5RZ7CEH4KXA, seg_pos={0 0}, upload=192678]: Done uploading piece. err: <nil> Nov 8 14:26:11.119863095 (218h45m42.815711838s) [noise=true , address=77.90.31.17:4771 , piece_id=LUZ4KXA4YUVTQYKXVEPTEXZ4MFQ5IWFTY3C2LUUVQ3N6DOVO6ZXA, seg_pos={0 0}, upload=192678]: Done uploading piece. err: <nil> Nov 8 14:26:14.806769255 (218h45m46.502618008s) [noise=true , address=145.79.227.2:4113 , piece_id=Q5E3IXSHGFKIFG7EFAPW2XWYP63TO3BC5C7PZJDPXXD2UKDQDJLA, seg_pos={0 0}, upload=192678]: Done uploading piece. err: <nil> Nov 8 14:26:15.197752745 (218h45m46.893601498s) [noise=true , address=145.79.227.26:4655 , piece_id=GP4F5L6HH6S4ZEYWS7XRG5KJSYFGOX5JXQBTQEOXFDFSNUBHZJ6A, seg_pos={0 0}, upload=192678]: Done uploading piece. err: <nil> Nov 8 14:26:17.425396997 (218h45m49.12124575s) [noise=true , address=145.79.227.22:4509 , piece_id=6RJJCCT2Z5U3JOWWOBLRIDK5CTWKXU7NR4OIREAQ2JDE2KYDJHOQ, seg_pos={0 0}, upload=192678]: Done uploading piece. err: <nil> Nov 8 14:26:21.856283285 (218h45m53.552132048s) [noise=true , address=145.79.227.31:4798 , piece_id=5Q7NYNXM2IMV7YUCB464FQMBI4L476Y6VKPPOPW7RKNCC5XFDJ2A, seg_pos={0 0}, upload=192678]: Done uploading piece. err: <nil> Nov 8 14:26:23.486413928 (218h45m55.182262681s) [noise=true , address=145.79.227.20:4437 , piece_id=G5RTYGERAVOTQDZGCG6NEKRCQAUOVJNYXO4LEBJ7XHJZ6FDXTULQ, seg_pos={0 0}, upload=192678]: Done uploading piece. err: <nil> Nov 8 14:26:24.659870720 (218h45m56.355719473s) [noise=true , address=145.79.227.26:4653 , piece_id=UIV3WFYLH4JZSIIHZUBL56THDPAYK7DMRUQG5TVTHHEF3XGN6NKQ, seg_pos={0 0}, upload=192678]: Done uploading piece. err: <nil> Nov 8 14:26:34.701060537 (218h46m6.3969093s) [noise=true , address=145.79.227.8:4329 , piece_id=JJFSJ7USPZFTPXVUZVLMXRAHYLARBQZLWWU7ITXFNESYVVL4QBMQ, seg_pos={0 0}, upload=192678]: Done uploading piece. err: <nil> Nov 8 14:26:37.183628509 (218h46m8.879477262s) [noise=true , address=145.79.227.10:4402 , piece_id=IRNQEX73D4AEXNYXEWWKCKEMYFC3YXIPX6HSDYYBNDKBM77ROBEA, seg_pos={0 0}, upload=192678]: Done uploading piece. err: <nil> Nov 8 14:26:37.186238899 (218h46m8.882087662s) [noise=true , address=145.79.227.16:4911 , piece_id=QKITSRMSCHOZLCQ7X6GARTE3GS24G3CZFXJIBKDM3VWOVQLMLEHA, seg_pos={0 0}, upload=192678]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:145.79.227.16:4911): protocol: write tcp 188.166.225.170:56440->145.79.227.16:4911: use of closed network connection; write tcp 188.166.225.170:56440->145.79.227.16:4911: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56440->145.79.227.16:4911: use of closed network connection Nov 8 14:26:37.186462625 (218h46m8.882311378s) [noise=true , address=145.79.227.21:4475 , piece_id=6F7IZNQLORMTKUF576FJA6L7FBG66INEUFPQYHL77WLC6DYF32GQ, seg_pos={0 0}, upload=192678]: Done uploading piece. err: ecclient: upload failed (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk, address:145.79.227.21:4475): protocol: expected piece hash; context deadline exceeded; EOF Nov 8 14:26:37.186739442 (218h46m8.882588195s) [noise=true , address=145.79.227.27:4689 , piece_id=JEEKOLHURPGNFSR4YCWL3UD6BU2MT7VW4S7BU3YKE7BJSUEYLPGQ, seg_pos={0 0}, upload=192678]: Done uploading piece. err: ecclient: upload failed (node:1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, address:145.79.227.27:4689): protocol: write tcp 188.166.225.170:37312->145.79.227.27:4689: use of closed network connection; write tcp 188.166.225.170:37312->145.79.227.27:4689: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37312->145.79.227.27:4689: use of closed network connection Nov 8 14:26:37.186915056 (218h46m8.88276381s) [noise=true , address=145.79.227.7:4295 , piece_id=5OY4NKICVFGAWR7JPOOYIYRXR442IIUDDWTN3MT6KAR4ZHXTKPNQ, seg_pos={0 0}, upload=192678]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:145.79.227.7:4295): protocol: expected piece hash; context deadline exceeded; EOF Nov 8 14:26:37.187072807 (218h46m8.88292156s) [noise=true , address=145.79.227.9:4366 , piece_id=3BINODFKOOG7BYULCICW6S64YZOU5D2Q5MFZYGPSSA34XWM762BQ, seg_pos={0 0}, upload=192678]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: expected piece hash; context deadline exceeded; EOF Nov 8 14:26:37.187107163 (218h46m8.882955916s) [seg_pos={0 0}, upload=192678]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 8 14:26:37.433482306 (218h46m9.129331059s) [upload=192678]: Done uploading segments... Nov 8 14:26:37.433526460 (218h46m9.129375213s) [upload=192678]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 7.748299182s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 7.61350373s trace: https://jaeger.dev.storj.io/trace/3f377bb0d96d25c FAIL (2025-11-08 15:02:27 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Nov 8 14:26:07.882330547 (218h45m39.5781793s) [upload=192678]: Starting upload Nov 8 14:26:07.882433824 (218h45m39.578282567s) [upload=192678]: Done starting upload Nov 8 14:26:07.882492927 (218h45m39.57834167s) [upload=192678]: Uploading segments... Nov 8 14:26:07.882645367 (218h45m39.57849411s) [upload=192678]: Got next segment. Inline: false Nov 8 14:26:07.882698468 (218h45m39.578547221s) [upload=192678]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 8 14:26:07.882704229 (218h45m39.578552982s) [upload=192678]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 8 14:26:08.131091114 (218h45m39.826939897s) [seg_pos={0 0}, upload=192678]: Begin upload segment... Nov 8 14:26:08.131165496 (218h45m39.827014239s) [seg_pos={0 0}, upload=192678]: Done begin upload segment. Nov 8 14:26:08.131220160 (218h45m39.827068913s) [noise=true , address=145.79.227.16:4911 , piece_id=QKITSRMSCHOZLCQ7X6GARTE3GS24G3CZFXJIBKDM3VWOVQLMLEHA, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.131304631 (218h45m39.827153384s) [noise=true , address=145.79.227.27:4689 , piece_id=JEEKOLHURPGNFSR4YCWL3UD6BU2MT7VW4S7BU3YKE7BJSUEYLPGQ, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.132190408 (218h45m39.828039161s) [noise=true , address=145.79.227.7:4295 , piece_id=5OY4NKICVFGAWR7JPOOYIYRXR442IIUDDWTN3MT6KAR4ZHXTKPNQ, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.132569360 (218h45m39.828418113s) [noise=true , address=145.79.227.31:4798 , piece_id=5Q7NYNXM2IMV7YUCB464FQMBI4L476Y6VKPPOPW7RKNCC5XFDJ2A, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.133276446 (218h45m39.829125199s) [noise=true , address=145.79.227.26:4653 , piece_id=UIV3WFYLH4JZSIIHZUBL56THDPAYK7DMRUQG5TVTHHEF3XGN6NKQ, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.134167893 (218h45m39.830016646s) [noise=true , address=145.79.227.26:4655 , piece_id=GP4F5L6HH6S4ZEYWS7XRG5KJSYFGOX5JXQBTQEOXFDFSNUBHZJ6A, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.134314192 (218h45m39.830162945s) [noise=true , address=145.79.227.20:4437 , piece_id=G5RTYGERAVOTQDZGCG6NEKRCQAUOVJNYXO4LEBJ7XHJZ6FDXTULQ, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.135016950 (218h45m39.830865703s) [noise=true , address=145.79.227.10:4402 , piece_id=IRNQEX73D4AEXNYXEWWKCKEMYFC3YXIPX6HSDYYBNDKBM77ROBEA, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.135235305 (218h45m39.831084059s) [noise=true , address=145.79.227.22:4509 , piece_id=6RJJCCT2Z5U3JOWWOBLRIDK5CTWKXU7NR4OIREAQ2JDE2KYDJHOQ, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.135440236 (218h45m39.831288979s) [noise=true , address=145.79.227.2:4113 , piece_id=Q5E3IXSHGFKIFG7EFAPW2XWYP63TO3BC5C7PZJDPXXD2UKDQDJLA, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.136611887 (218h45m39.83246064s) [noise=true , address=77.90.31.15:4699 , piece_id=Z4PXJFTM7XMK62ORTMHCWEJ7OQF6PIYQCMAWJC3DT5RZ7CEH4KXA, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.137027559 (218h45m39.832876312s) [noise=true , address=145.79.227.8:4329 , piece_id=JJFSJ7USPZFTPXVUZVLMXRAHYLARBQZLWWU7ITXFNESYVVL4QBMQ, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.137896133 (218h45m39.833744886s) [noise=true , address=77.90.31.17:4771 , piece_id=LUZ4KXA4YUVTQYKXVEPTEXZ4MFQ5IWFTY3C2LUUVQ3N6DOVO6ZXA, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.139107940 (218h45m39.834956693s) [noise=true , address=145.79.227.21:4475 , piece_id=6F7IZNQLORMTKUF576FJA6L7FBG66INEUFPQYHL77WLC6DYF32GQ, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.139872657 (218h45m39.83572141s) [noise=true , address=145.79.227.9:4366 , piece_id=3BINODFKOOG7BYULCICW6S64YZOU5D2Q5MFZYGPSSA34XWM762BQ, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.140005960 (218h45m39.835854713s) [noise=true , address=77.90.31.16:4734 , piece_id=EKYBQDVNAYTGEPWU443RFLUX7OKCTWBPXA5UMCLZ22M2PBXO3E3Q, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:10.015794232 (218h45m41.711642985s) [upload=192678]: Next returned nil segment Nov 8 14:26:10.015816414 (218h45m41.711665167s) [upload=192678]: Waiting for error group managing segments... Nov 8 14:26:10.314091526 (218h45m42.009940279s) [noise=true , address=77.90.31.16:4734 , piece_id=EKYBQDVNAYTGEPWU443RFLUX7OKCTWBPXA5UMCLZ22M2PBXO3E3Q, seg_pos={0 0}, upload=192678]: Done uploading piece. err: <nil> Nov 8 14:26:10.388439672 (218h45m42.084288465s) [noise=true , address=77.90.31.15:4699 , piece_id=Z4PXJFTM7XMK62ORTMHCWEJ7OQF6PIYQCMAWJC3DT5RZ7CEH4KXA, seg_pos={0 0}, upload=192678]: Done uploading piece. err: <nil> Nov 8 14:26:11.119863095 (218h45m42.815711838s) [noise=true , address=77.90.31.17:4771 , piece_id=LUZ4KXA4YUVTQYKXVEPTEXZ4MFQ5IWFTY3C2LUUVQ3N6DOVO6ZXA, seg_pos={0 0}, upload=192678]: Done uploading piece. err: <nil> Nov 8 14:26:14.806769255 (218h45m46.502618008s) [noise=true , address=145.79.227.2:4113 , piece_id=Q5E3IXSHGFKIFG7EFAPW2XWYP63TO3BC5C7PZJDPXXD2UKDQDJLA, seg_pos={0 0}, upload=192678]: Done uploading piece. err: <nil> Nov 8 14:26:15.197752745 (218h45m46.893601498s) [noise=true , address=145.79.227.26:4655 , piece_id=GP4F5L6HH6S4ZEYWS7XRG5KJSYFGOX5JXQBTQEOXFDFSNUBHZJ6A, seg_pos={0 0}, upload=192678]: Done uploading piece. err: <nil> Nov 8 14:26:17.425396997 (218h45m49.12124575s) [noise=true , address=145.79.227.22:4509 , piece_id=6RJJCCT2Z5U3JOWWOBLRIDK5CTWKXU7NR4OIREAQ2JDE2KYDJHOQ, seg_pos={0 0}, upload=192678]: Done uploading piece. err: <nil> Nov 8 14:26:21.856283285 (218h45m53.552132048s) [noise=true , address=145.79.227.31:4798 , piece_id=5Q7NYNXM2IMV7YUCB464FQMBI4L476Y6VKPPOPW7RKNCC5XFDJ2A, seg_pos={0 0}, upload=192678]: Done uploading piece. err: <nil> Nov 8 14:26:23.486413928 (218h45m55.182262681s) [noise=true , address=145.79.227.20:4437 , piece_id=G5RTYGERAVOTQDZGCG6NEKRCQAUOVJNYXO4LEBJ7XHJZ6FDXTULQ, seg_pos={0 0}, upload=192678]: Done uploading piece. err: <nil> Nov 8 14:26:24.659870720 (218h45m56.355719473s) [noise=true , address=145.79.227.26:4653 , piece_id=UIV3WFYLH4JZSIIHZUBL56THDPAYK7DMRUQG5TVTHHEF3XGN6NKQ, seg_pos={0 0}, upload=192678]: Done uploading piece. err: <nil> Nov 8 14:26:34.701060537 (218h46m6.3969093s) [noise=true , address=145.79.227.8:4329 , piece_id=JJFSJ7USPZFTPXVUZVLMXRAHYLARBQZLWWU7ITXFNESYVVL4QBMQ, seg_pos={0 0}, upload=192678]: Done uploading piece. err: <nil> Nov 8 14:26:37.183628509 (218h46m8.879477262s) [noise=true , address=145.79.227.10:4402 , piece_id=IRNQEX73D4AEXNYXEWWKCKEMYFC3YXIPX6HSDYYBNDKBM77ROBEA, seg_pos={0 0}, upload=192678]: Done uploading piece. err: <nil> Nov 8 14:26:37.186238899 (218h46m8.882087662s) [noise=true , address=145.79.227.16:4911 , piece_id=QKITSRMSCHOZLCQ7X6GARTE3GS24G3CZFXJIBKDM3VWOVQLMLEHA, seg_pos={0 0}, upload=192678]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:145.79.227.16:4911): protocol: write tcp 188.166.225.170:56440->145.79.227.16:4911: use of closed network connection; write tcp 188.166.225.170:56440->145.79.227.16:4911: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56440->145.79.227.16:4911: use of closed network connection Nov 8 14:26:37.186462625 (218h46m8.882311378s) [noise=true , address=145.79.227.21:4475 , piece_id=6F7IZNQLORMTKUF576FJA6L7FBG66INEUFPQYHL77WLC6DYF32GQ, seg_pos={0 0}, upload=192678]: Done uploading piece. err: ecclient: upload failed (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk, address:145.79.227.21:4475): protocol: expected piece hash; context deadline exceeded; EOF Nov 8 14:26:37.186739442 (218h46m8.882588195s) [noise=true , address=145.79.227.27:4689 , piece_id=JEEKOLHURPGNFSR4YCWL3UD6BU2MT7VW4S7BU3YKE7BJSUEYLPGQ, seg_pos={0 0}, upload=192678]: Done uploading piece. err: ecclient: upload failed (node:1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, address:145.79.227.27:4689): protocol: write tcp 188.166.225.170:37312->145.79.227.27:4689: use of closed network connection; write tcp 188.166.225.170:37312->145.79.227.27:4689: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37312->145.79.227.27:4689: use of closed network connection Nov 8 14:26:37.186915056 (218h46m8.88276381s) [noise=true , address=145.79.227.7:4295 , piece_id=5OY4NKICVFGAWR7JPOOYIYRXR442IIUDDWTN3MT6KAR4ZHXTKPNQ, seg_pos={0 0}, upload=192678]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:145.79.227.7:4295): protocol: expected piece hash; context deadline exceeded; EOF Nov 8 14:26:37.187072807 (218h46m8.88292156s) [noise=true , address=145.79.227.9:4366 , piece_id=3BINODFKOOG7BYULCICW6S64YZOU5D2Q5MFZYGPSSA34XWM762BQ, seg_pos={0 0}, upload=192678]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: expected piece hash; context deadline exceeded; EOF Nov 8 14:26:37.187107163 (218h46m8.882955916s) [seg_pos={0 0}, upload=192678]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 8 14:26:37.433482306 (218h46m9.129331059s) [upload=192678]: Done uploading segments... Nov 8 14:26:37.433526460 (218h46m9.129375213s) [upload=192678]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 7.613524149s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 8.532058334s trace: https://jaeger.dev.storj.io/trace/3f377bb0d96d25c FAIL (2025-11-08 15:01:33 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Nov 8 14:26:07.882330547 (218h45m39.5781793s) [upload=192678]: Starting upload Nov 8 14:26:07.882433824 (218h45m39.578282567s) [upload=192678]: Done starting upload Nov 8 14:26:07.882492927 (218h45m39.57834167s) [upload=192678]: Uploading segments... Nov 8 14:26:07.882645367 (218h45m39.57849411s) [upload=192678]: Got next segment. Inline: false Nov 8 14:26:07.882698468 (218h45m39.578547221s) [upload=192678]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 8 14:26:07.882704229 (218h45m39.578552982s) [upload=192678]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 8 14:26:08.131091114 (218h45m39.826939897s) [seg_pos={0 0}, upload=192678]: Begin upload segment... Nov 8 14:26:08.131165496 (218h45m39.827014239s) [seg_pos={0 0}, upload=192678]: Done begin upload segment. Nov 8 14:26:08.131220160 (218h45m39.827068913s) [noise=true , address=145.79.227.16:4911 , piece_id=QKITSRMSCHOZLCQ7X6GARTE3GS24G3CZFXJIBKDM3VWOVQLMLEHA, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.131304631 (218h45m39.827153384s) [noise=true , address=145.79.227.27:4689 , piece_id=JEEKOLHURPGNFSR4YCWL3UD6BU2MT7VW4S7BU3YKE7BJSUEYLPGQ, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.132190408 (218h45m39.828039161s) [noise=true , address=145.79.227.7:4295 , piece_id=5OY4NKICVFGAWR7JPOOYIYRXR442IIUDDWTN3MT6KAR4ZHXTKPNQ, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.132569360 (218h45m39.828418113s) [noise=true , address=145.79.227.31:4798 , piece_id=5Q7NYNXM2IMV7YUCB464FQMBI4L476Y6VKPPOPW7RKNCC5XFDJ2A, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.133276446 (218h45m39.829125199s) [noise=true , address=145.79.227.26:4653 , piece_id=UIV3WFYLH4JZSIIHZUBL56THDPAYK7DMRUQG5TVTHHEF3XGN6NKQ, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.134167893 (218h45m39.830016646s) [noise=true , address=145.79.227.26:4655 , piece_id=GP4F5L6HH6S4ZEYWS7XRG5KJSYFGOX5JXQBTQEOXFDFSNUBHZJ6A, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.134314192 (218h45m39.830162945s) [noise=true , address=145.79.227.20:4437 , piece_id=G5RTYGERAVOTQDZGCG6NEKRCQAUOVJNYXO4LEBJ7XHJZ6FDXTULQ, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.135016950 (218h45m39.830865703s) [noise=true , address=145.79.227.10:4402 , piece_id=IRNQEX73D4AEXNYXEWWKCKEMYFC3YXIPX6HSDYYBNDKBM77ROBEA, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.135235305 (218h45m39.831084059s) [noise=true , address=145.79.227.22:4509 , piece_id=6RJJCCT2Z5U3JOWWOBLRIDK5CTWKXU7NR4OIREAQ2JDE2KYDJHOQ, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.135440236 (218h45m39.831288979s) [noise=true , address=145.79.227.2:4113 , piece_id=Q5E3IXSHGFKIFG7EFAPW2XWYP63TO3BC5C7PZJDPXXD2UKDQDJLA, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.136611887 (218h45m39.83246064s) [noise=true , address=77.90.31.15:4699 , piece_id=Z4PXJFTM7XMK62ORTMHCWEJ7OQF6PIYQCMAWJC3DT5RZ7CEH4KXA, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.137027559 (218h45m39.832876312s) [noise=true , address=145.79.227.8:4329 , piece_id=JJFSJ7USPZFTPXVUZVLMXRAHYLARBQZLWWU7ITXFNESYVVL4QBMQ, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.137896133 (218h45m39.833744886s) [noise=true , address=77.90.31.17:4771 , piece_id=LUZ4KXA4YUVTQYKXVEPTEXZ4MFQ5IWFTY3C2LUUVQ3N6DOVO6ZXA, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.139107940 (218h45m39.834956693s) [noise=true , address=145.79.227.21:4475 , piece_id=6F7IZNQLORMTKUF576FJA6L7FBG66INEUFPQYHL77WLC6DYF32GQ, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.139872657 (218h45m39.83572141s) [noise=true , address=145.79.227.9:4366 , piece_id=3BINODFKOOG7BYULCICW6S64YZOU5D2Q5MFZYGPSSA34XWM762BQ, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.140005960 (218h45m39.835854713s) [noise=true , address=77.90.31.16:4734 , piece_id=EKYBQDVNAYTGEPWU443RFLUX7OKCTWBPXA5UMCLZ22M2PBXO3E3Q, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:10.015794232 (218h45m41.711642985s) [upload=192678]: Next returned nil segment Nov 8 14:26:10.015816414 (218h45m41.711665167s) [upload=192678]: Waiting for error group managing segments... Nov 8 14:26:10.314091526 (218h45m42.009940279s) [noise=true , address=77.90.31.16:4734 , piece_id=EKYBQDVNAYTGEPWU443RFLUX7OKCTWBPXA5UMCLZ22M2PBXO3E3Q, seg_pos={0 0}, upload=192678]: Done uploading piece. err: <nil> Nov 8 14:26:10.388439672 (218h45m42.084288465s) [noise=true , address=77.90.31.15:4699 , piece_id=Z4PXJFTM7XMK62ORTMHCWEJ7OQF6PIYQCMAWJC3DT5RZ7CEH4KXA, seg_pos={0 0}, upload=192678]: Done uploading piece. err: <nil> Nov 8 14:26:11.119863095 (218h45m42.815711838s) [noise=true , address=77.90.31.17:4771 , piece_id=LUZ4KXA4YUVTQYKXVEPTEXZ4MFQ5IWFTY3C2LUUVQ3N6DOVO6ZXA, seg_pos={0 0}, upload=192678]: Done uploading piece. err: <nil> Nov 8 14:26:14.806769255 (218h45m46.502618008s) [noise=true , address=145.79.227.2:4113 , piece_id=Q5E3IXSHGFKIFG7EFAPW2XWYP63TO3BC5C7PZJDPXXD2UKDQDJLA, seg_pos={0 0}, upload=192678]: Done uploading piece. err: <nil> Nov 8 14:26:15.197752745 (218h45m46.893601498s) [noise=true , address=145.79.227.26:4655 , piece_id=GP4F5L6HH6S4ZEYWS7XRG5KJSYFGOX5JXQBTQEOXFDFSNUBHZJ6A, seg_pos={0 0}, upload=192678]: Done uploading piece. err: <nil> Nov 8 14:26:17.425396997 (218h45m49.12124575s) [noise=true , address=145.79.227.22:4509 , piece_id=6RJJCCT2Z5U3JOWWOBLRIDK5CTWKXU7NR4OIREAQ2JDE2KYDJHOQ, seg_pos={0 0}, upload=192678]: Done uploading piece. err: <nil> Nov 8 14:26:21.856283285 (218h45m53.552132048s) [noise=true , address=145.79.227.31:4798 , piece_id=5Q7NYNXM2IMV7YUCB464FQMBI4L476Y6VKPPOPW7RKNCC5XFDJ2A, seg_pos={0 0}, upload=192678]: Done uploading piece. err: <nil> Nov 8 14:26:23.486413928 (218h45m55.182262681s) [noise=true , address=145.79.227.20:4437 , piece_id=G5RTYGERAVOTQDZGCG6NEKRCQAUOVJNYXO4LEBJ7XHJZ6FDXTULQ, seg_pos={0 0}, upload=192678]: Done uploading piece. err: <nil> Nov 8 14:26:24.659870720 (218h45m56.355719473s) [noise=true , address=145.79.227.26:4653 , piece_id=UIV3WFYLH4JZSIIHZUBL56THDPAYK7DMRUQG5TVTHHEF3XGN6NKQ, seg_pos={0 0}, upload=192678]: Done uploading piece. err: <nil> Nov 8 14:26:34.701060537 (218h46m6.3969093s) [noise=true , address=145.79.227.8:4329 , piece_id=JJFSJ7USPZFTPXVUZVLMXRAHYLARBQZLWWU7ITXFNESYVVL4QBMQ, seg_pos={0 0}, upload=192678]: Done uploading piece. err: <nil> Nov 8 14:26:37.183628509 (218h46m8.879477262s) [noise=true , address=145.79.227.10:4402 , piece_id=IRNQEX73D4AEXNYXEWWKCKEMYFC3YXIPX6HSDYYBNDKBM77ROBEA, seg_pos={0 0}, upload=192678]: Done uploading piece. err: <nil> Nov 8 14:26:37.186238899 (218h46m8.882087662s) [noise=true , address=145.79.227.16:4911 , piece_id=QKITSRMSCHOZLCQ7X6GARTE3GS24G3CZFXJIBKDM3VWOVQLMLEHA, seg_pos={0 0}, upload=192678]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:145.79.227.16:4911): protocol: write tcp 188.166.225.170:56440->145.79.227.16:4911: use of closed network connection; write tcp 188.166.225.170:56440->145.79.227.16:4911: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56440->145.79.227.16:4911: use of closed network connection Nov 8 14:26:37.186462625 (218h46m8.882311378s) [noise=true , address=145.79.227.21:4475 , piece_id=6F7IZNQLORMTKUF576FJA6L7FBG66INEUFPQYHL77WLC6DYF32GQ, seg_pos={0 0}, upload=192678]: Done uploading piece. err: ecclient: upload failed (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk, address:145.79.227.21:4475): protocol: expected piece hash; context deadline exceeded; EOF Nov 8 14:26:37.186739442 (218h46m8.882588195s) [noise=true , address=145.79.227.27:4689 , piece_id=JEEKOLHURPGNFSR4YCWL3UD6BU2MT7VW4S7BU3YKE7BJSUEYLPGQ, seg_pos={0 0}, upload=192678]: Done uploading piece. err: ecclient: upload failed (node:1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, address:145.79.227.27:4689): protocol: write tcp 188.166.225.170:37312->145.79.227.27:4689: use of closed network connection; write tcp 188.166.225.170:37312->145.79.227.27:4689: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37312->145.79.227.27:4689: use of closed network connection Nov 8 14:26:37.186915056 (218h46m8.88276381s) [noise=true , address=145.79.227.7:4295 , piece_id=5OY4NKICVFGAWR7JPOOYIYRXR442IIUDDWTN3MT6KAR4ZHXTKPNQ, seg_pos={0 0}, upload=192678]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:145.79.227.7:4295): protocol: expected piece hash; context deadline exceeded; EOF Nov 8 14:26:37.187072807 (218h46m8.88292156s) [noise=true , address=145.79.227.9:4366 , piece_id=3BINODFKOOG7BYULCICW6S64YZOU5D2Q5MFZYGPSSA34XWM762BQ, seg_pos={0 0}, upload=192678]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: expected piece hash; context deadline exceeded; EOF Nov 8 14:26:37.187107163 (218h46m8.882955916s) [seg_pos={0 0}, upload=192678]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 8 14:26:37.433482306 (218h46m9.129331059s) [upload=192678]: Done uploading segments... Nov 8 14:26:37.433526460 (218h46m9.129375213s) [upload=192678]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 8.532089513s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 10.484602006s trace: https://jaeger.dev.storj.io/trace/3f377bb0d96d25c FAIL (2025-11-08 15:00:37 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Nov 8 14:26:07.882330547 (218h45m39.5781793s) [upload=192678]: Starting upload Nov 8 14:26:07.882433824 (218h45m39.578282567s) [upload=192678]: Done starting upload Nov 8 14:26:07.882492927 (218h45m39.57834167s) [upload=192678]: Uploading segments... Nov 8 14:26:07.882645367 (218h45m39.57849411s) [upload=192678]: Got next segment. Inline: false Nov 8 14:26:07.882698468 (218h45m39.578547221s) [upload=192678]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 8 14:26:07.882704229 (218h45m39.578552982s) [upload=192678]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 8 14:26:08.131091114 (218h45m39.826939897s) [seg_pos={0 0}, upload=192678]: Begin upload segment... Nov 8 14:26:08.131165496 (218h45m39.827014239s) [seg_pos={0 0}, upload=192678]: Done begin upload segment. Nov 8 14:26:08.131220160 (218h45m39.827068913s) [noise=true , address=145.79.227.16:4911 , piece_id=QKITSRMSCHOZLCQ7X6GARTE3GS24G3CZFXJIBKDM3VWOVQLMLEHA, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.131304631 (218h45m39.827153384s) [noise=true , address=145.79.227.27:4689 , piece_id=JEEKOLHURPGNFSR4YCWL3UD6BU2MT7VW4S7BU3YKE7BJSUEYLPGQ, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.132190408 (218h45m39.828039161s) [noise=true , address=145.79.227.7:4295 , piece_id=5OY4NKICVFGAWR7JPOOYIYRXR442IIUDDWTN3MT6KAR4ZHXTKPNQ, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.132569360 (218h45m39.828418113s) [noise=true , address=145.79.227.31:4798 , piece_id=5Q7NYNXM2IMV7YUCB464FQMBI4L476Y6VKPPOPW7RKNCC5XFDJ2A, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.133276446 (218h45m39.829125199s) [noise=true , address=145.79.227.26:4653 , piece_id=UIV3WFYLH4JZSIIHZUBL56THDPAYK7DMRUQG5TVTHHEF3XGN6NKQ, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.134167893 (218h45m39.830016646s) [noise=true , address=145.79.227.26:4655 , piece_id=GP4F5L6HH6S4ZEYWS7XRG5KJSYFGOX5JXQBTQEOXFDFSNUBHZJ6A, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.134314192 (218h45m39.830162945s) [noise=true , address=145.79.227.20:4437 , piece_id=G5RTYGERAVOTQDZGCG6NEKRCQAUOVJNYXO4LEBJ7XHJZ6FDXTULQ, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.135016950 (218h45m39.830865703s) [noise=true , address=145.79.227.10:4402 , piece_id=IRNQEX73D4AEXNYXEWWKCKEMYFC3YXIPX6HSDYYBNDKBM77ROBEA, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.135235305 (218h45m39.831084059s) [noise=true , address=145.79.227.22:4509 , piece_id=6RJJCCT2Z5U3JOWWOBLRIDK5CTWKXU7NR4OIREAQ2JDE2KYDJHOQ, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.135440236 (218h45m39.831288979s) [noise=true , address=145.79.227.2:4113 , piece_id=Q5E3IXSHGFKIFG7EFAPW2XWYP63TO3BC5C7PZJDPXXD2UKDQDJLA, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.136611887 (218h45m39.83246064s) [noise=true , address=77.90.31.15:4699 , piece_id=Z4PXJFTM7XMK62ORTMHCWEJ7OQF6PIYQCMAWJC3DT5RZ7CEH4KXA, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.137027559 (218h45m39.832876312s) [noise=true , address=145.79.227.8:4329 , piece_id=JJFSJ7USPZFTPXVUZVLMXRAHYLARBQZLWWU7ITXFNESYVVL4QBMQ, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.137896133 (218h45m39.833744886s) [noise=true , address=77.90.31.17:4771 , piece_id=LUZ4KXA4YUVTQYKXVEPTEXZ4MFQ5IWFTY3C2LUUVQ3N6DOVO6ZXA, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.139107940 (218h45m39.834956693s) [noise=true , address=145.79.227.21:4475 , piece_id=6F7IZNQLORMTKUF576FJA6L7FBG66INEUFPQYHL77WLC6DYF32GQ, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.139872657 (218h45m39.83572141s) [noise=true , address=145.79.227.9:4366 , piece_id=3BINODFKOOG7BYULCICW6S64YZOU5D2Q5MFZYGPSSA34XWM762BQ, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.140005960 (218h45m39.835854713s) [noise=true , address=77.90.31.16:4734 , piece_id=EKYBQDVNAYTGEPWU443RFLUX7OKCTWBPXA5UMCLZ22M2PBXO3E3Q, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:10.015794232 (218h45m41.711642985s) [upload=192678]: Next returned nil segment Nov 8 14:26:10.015816414 (218h45m41.711665167s) [upload=192678]: Waiting for error group managing segments... Nov 8 14:26:10.314091526 (218h45m42.009940279s) [noise=true , address=77.90.31.16:4734 , piece_id=EKYBQDVNAYTGEPWU443RFLUX7OKCTWBPXA5UMCLZ22M2PBXO3E3Q, seg_pos={0 0}, upload=192678]: Done uploading piece. err: <nil> Nov 8 14:26:10.388439672 (218h45m42.084288465s) [noise=true , address=77.90.31.15:4699 , piece_id=Z4PXJFTM7XMK62ORTMHCWEJ7OQF6PIYQCMAWJC3DT5RZ7CEH4KXA, seg_pos={0 0}, upload=192678]: Done uploading piece. err: <nil> Nov 8 14:26:11.119863095 (218h45m42.815711838s) [noise=true , address=77.90.31.17:4771 , piece_id=LUZ4KXA4YUVTQYKXVEPTEXZ4MFQ5IWFTY3C2LUUVQ3N6DOVO6ZXA, seg_pos={0 0}, upload=192678]: Done uploading piece. err: <nil> Nov 8 14:26:14.806769255 (218h45m46.502618008s) [noise=true , address=145.79.227.2:4113 , piece_id=Q5E3IXSHGFKIFG7EFAPW2XWYP63TO3BC5C7PZJDPXXD2UKDQDJLA, seg_pos={0 0}, upload=192678]: Done uploading piece. err: <nil> Nov 8 14:26:15.197752745 (218h45m46.893601498s) [noise=true , address=145.79.227.26:4655 , piece_id=GP4F5L6HH6S4ZEYWS7XRG5KJSYFGOX5JXQBTQEOXFDFSNUBHZJ6A, seg_pos={0 0}, upload=192678]: Done uploading piece. err: <nil> Nov 8 14:26:17.425396997 (218h45m49.12124575s) [noise=true , address=145.79.227.22:4509 , piece_id=6RJJCCT2Z5U3JOWWOBLRIDK5CTWKXU7NR4OIREAQ2JDE2KYDJHOQ, seg_pos={0 0}, upload=192678]: Done uploading piece. err: <nil> Nov 8 14:26:21.856283285 (218h45m53.552132048s) [noise=true , address=145.79.227.31:4798 , piece_id=5Q7NYNXM2IMV7YUCB464FQMBI4L476Y6VKPPOPW7RKNCC5XFDJ2A, seg_pos={0 0}, upload=192678]: Done uploading piece. err: <nil> Nov 8 14:26:23.486413928 (218h45m55.182262681s) [noise=true , address=145.79.227.20:4437 , piece_id=G5RTYGERAVOTQDZGCG6NEKRCQAUOVJNYXO4LEBJ7XHJZ6FDXTULQ, seg_pos={0 0}, upload=192678]: Done uploading piece. err: <nil> Nov 8 14:26:24.659870720 (218h45m56.355719473s) [noise=true , address=145.79.227.26:4653 , piece_id=UIV3WFYLH4JZSIIHZUBL56THDPAYK7DMRUQG5TVTHHEF3XGN6NKQ, seg_pos={0 0}, upload=192678]: Done uploading piece. err: <nil> Nov 8 14:26:34.701060537 (218h46m6.3969093s) [noise=true , address=145.79.227.8:4329 , piece_id=JJFSJ7USPZFTPXVUZVLMXRAHYLARBQZLWWU7ITXFNESYVVL4QBMQ, seg_pos={0 0}, upload=192678]: Done uploading piece. err: <nil> Nov 8 14:26:37.183628509 (218h46m8.879477262s) [noise=true , address=145.79.227.10:4402 , piece_id=IRNQEX73D4AEXNYXEWWKCKEMYFC3YXIPX6HSDYYBNDKBM77ROBEA, seg_pos={0 0}, upload=192678]: Done uploading piece. err: <nil> Nov 8 14:26:37.186238899 (218h46m8.882087662s) [noise=true , address=145.79.227.16:4911 , piece_id=QKITSRMSCHOZLCQ7X6GARTE3GS24G3CZFXJIBKDM3VWOVQLMLEHA, seg_pos={0 0}, upload=192678]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:145.79.227.16:4911): protocol: write tcp 188.166.225.170:56440->145.79.227.16:4911: use of closed network connection; write tcp 188.166.225.170:56440->145.79.227.16:4911: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56440->145.79.227.16:4911: use of closed network connection Nov 8 14:26:37.186462625 (218h46m8.882311378s) [noise=true , address=145.79.227.21:4475 , piece_id=6F7IZNQLORMTKUF576FJA6L7FBG66INEUFPQYHL77WLC6DYF32GQ, seg_pos={0 0}, upload=192678]: Done uploading piece. err: ecclient: upload failed (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk, address:145.79.227.21:4475): protocol: expected piece hash; context deadline exceeded; EOF Nov 8 14:26:37.186739442 (218h46m8.882588195s) [noise=true , address=145.79.227.27:4689 , piece_id=JEEKOLHURPGNFSR4YCWL3UD6BU2MT7VW4S7BU3YKE7BJSUEYLPGQ, seg_pos={0 0}, upload=192678]: Done uploading piece. err: ecclient: upload failed (node:1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, address:145.79.227.27:4689): protocol: write tcp 188.166.225.170:37312->145.79.227.27:4689: use of closed network connection; write tcp 188.166.225.170:37312->145.79.227.27:4689: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37312->145.79.227.27:4689: use of closed network connection Nov 8 14:26:37.186915056 (218h46m8.88276381s) [noise=true , address=145.79.227.7:4295 , piece_id=5OY4NKICVFGAWR7JPOOYIYRXR442IIUDDWTN3MT6KAR4ZHXTKPNQ, seg_pos={0 0}, upload=192678]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:145.79.227.7:4295): protocol: expected piece hash; context deadline exceeded; EOF Nov 8 14:26:37.187072807 (218h46m8.88292156s) [noise=true , address=145.79.227.9:4366 , piece_id=3BINODFKOOG7BYULCICW6S64YZOU5D2Q5MFZYGPSSA34XWM762BQ, seg_pos={0 0}, upload=192678]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: expected piece hash; context deadline exceeded; EOF Nov 8 14:26:37.187107163 (218h46m8.882955916s) [seg_pos={0 0}, upload=192678]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 8 14:26:37.433482306 (218h46m9.129331059s) [upload=192678]: Done uploading segments... Nov 8 14:26:37.433526460 (218h46m9.129375213s) [upload=192678]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 10.48463627s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 8.936358859s trace: https://jaeger.dev.storj.io/trace/3f377bb0d96d25c FAIL (2025-11-08 15:00:06 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Nov 8 14:26:07.882330547 (218h45m39.5781793s) [upload=192678]: Starting upload Nov 8 14:26:07.882433824 (218h45m39.578282567s) [upload=192678]: Done starting upload Nov 8 14:26:07.882492927 (218h45m39.57834167s) [upload=192678]: Uploading segments... Nov 8 14:26:07.882645367 (218h45m39.57849411s) [upload=192678]: Got next segment. Inline: false Nov 8 14:26:07.882698468 (218h45m39.578547221s) [upload=192678]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 8 14:26:07.882704229 (218h45m39.578552982s) [upload=192678]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 8 14:26:08.131091114 (218h45m39.826939897s) [seg_pos={0 0}, upload=192678]: Begin upload segment... Nov 8 14:26:08.131165496 (218h45m39.827014239s) [seg_pos={0 0}, upload=192678]: Done begin upload segment. Nov 8 14:26:08.131220160 (218h45m39.827068913s) [noise=true , address=145.79.227.16:4911 , piece_id=QKITSRMSCHOZLCQ7X6GARTE3GS24G3CZFXJIBKDM3VWOVQLMLEHA, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.131304631 (218h45m39.827153384s) [noise=true , address=145.79.227.27:4689 , piece_id=JEEKOLHURPGNFSR4YCWL3UD6BU2MT7VW4S7BU3YKE7BJSUEYLPGQ, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.132190408 (218h45m39.828039161s) [noise=true , address=145.79.227.7:4295 , piece_id=5OY4NKICVFGAWR7JPOOYIYRXR442IIUDDWTN3MT6KAR4ZHXTKPNQ, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.132569360 (218h45m39.828418113s) [noise=true , address=145.79.227.31:4798 , piece_id=5Q7NYNXM2IMV7YUCB464FQMBI4L476Y6VKPPOPW7RKNCC5XFDJ2A, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.133276446 (218h45m39.829125199s) [noise=true , address=145.79.227.26:4653 , piece_id=UIV3WFYLH4JZSIIHZUBL56THDPAYK7DMRUQG5TVTHHEF3XGN6NKQ, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.134167893 (218h45m39.830016646s) [noise=true , address=145.79.227.26:4655 , piece_id=GP4F5L6HH6S4ZEYWS7XRG5KJSYFGOX5JXQBTQEOXFDFSNUBHZJ6A, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.134314192 (218h45m39.830162945s) [noise=true , address=145.79.227.20:4437 , piece_id=G5RTYGERAVOTQDZGCG6NEKRCQAUOVJNYXO4LEBJ7XHJZ6FDXTULQ, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.135016950 (218h45m39.830865703s) [noise=true , address=145.79.227.10:4402 , piece_id=IRNQEX73D4AEXNYXEWWKCKEMYFC3YXIPX6HSDYYBNDKBM77ROBEA, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.135235305 (218h45m39.831084059s) [noise=true , address=145.79.227.22:4509 , piece_id=6RJJCCT2Z5U3JOWWOBLRIDK5CTWKXU7NR4OIREAQ2JDE2KYDJHOQ, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.135440236 (218h45m39.831288979s) [noise=true , address=145.79.227.2:4113 , piece_id=Q5E3IXSHGFKIFG7EFAPW2XWYP63TO3BC5C7PZJDPXXD2UKDQDJLA, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.136611887 (218h45m39.83246064s) [noise=true , address=77.90.31.15:4699 , piece_id=Z4PXJFTM7XMK62ORTMHCWEJ7OQF6PIYQCMAWJC3DT5RZ7CEH4KXA, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.137027559 (218h45m39.832876312s) [noise=true , address=145.79.227.8:4329 , piece_id=JJFSJ7USPZFTPXVUZVLMXRAHYLARBQZLWWU7ITXFNESYVVL4QBMQ, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.137896133 (218h45m39.833744886s) [noise=true , address=77.90.31.17:4771 , piece_id=LUZ4KXA4YUVTQYKXVEPTEXZ4MFQ5IWFTY3C2LUUVQ3N6DOVO6ZXA, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.139107940 (218h45m39.834956693s) [noise=true , address=145.79.227.21:4475 , piece_id=6F7IZNQLORMTKUF576FJA6L7FBG66INEUFPQYHL77WLC6DYF32GQ, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.139872657 (218h45m39.83572141s) [noise=true , address=145.79.227.9:4366 , piece_id=3BINODFKOOG7BYULCICW6S64YZOU5D2Q5MFZYGPSSA34XWM762BQ, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.140005960 (218h45m39.835854713s) [noise=true , address=77.90.31.16:4734 , piece_id=EKYBQDVNAYTGEPWU443RFLUX7OKCTWBPXA5UMCLZ22M2PBXO3E3Q, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:10.015794232 (218h45m41.711642985s) [upload=192678]: Next returned nil segment Nov 8 14:26:10.015816414 (218h45m41.711665167s) [upload=192678]: Waiting for error group managing segments... Nov 8 14:26:10.314091526 (218h45m42.009940279s) [noise=true , address=77.90.31.16:4734 , piece_id=EKYBQDVNAYTGEPWU443RFLUX7OKCTWBPXA5UMCLZ22M2PBXO3E3Q, seg_pos={0 0}, upload=192678]: Done uploading piece. err: <nil> Nov 8 14:26:10.388439672 (218h45m42.084288465s) [noise=true , address=77.90.31.15:4699 , piece_id=Z4PXJFTM7XMK62ORTMHCWEJ7OQF6PIYQCMAWJC3DT5RZ7CEH4KXA, seg_pos={0 0}, upload=192678]: Done uploading piece. err: <nil> Nov 8 14:26:11.119863095 (218h45m42.815711838s) [noise=true , address=77.90.31.17:4771 , piece_id=LUZ4KXA4YUVTQYKXVEPTEXZ4MFQ5IWFTY3C2LUUVQ3N6DOVO6ZXA, seg_pos={0 0}, upload=192678]: Done uploading piece. err: <nil> Nov 8 14:26:14.806769255 (218h45m46.502618008s) [noise=true , address=145.79.227.2:4113 , piece_id=Q5E3IXSHGFKIFG7EFAPW2XWYP63TO3BC5C7PZJDPXXD2UKDQDJLA, seg_pos={0 0}, upload=192678]: Done uploading piece. err: <nil> Nov 8 14:26:15.197752745 (218h45m46.893601498s) [noise=true , address=145.79.227.26:4655 , piece_id=GP4F5L6HH6S4ZEYWS7XRG5KJSYFGOX5JXQBTQEOXFDFSNUBHZJ6A, seg_pos={0 0}, upload=192678]: Done uploading piece. err: <nil> Nov 8 14:26:17.425396997 (218h45m49.12124575s) [noise=true , address=145.79.227.22:4509 , piece_id=6RJJCCT2Z5U3JOWWOBLRIDK5CTWKXU7NR4OIREAQ2JDE2KYDJHOQ, seg_pos={0 0}, upload=192678]: Done uploading piece. err: <nil> Nov 8 14:26:21.856283285 (218h45m53.552132048s) [noise=true , address=145.79.227.31:4798 , piece_id=5Q7NYNXM2IMV7YUCB464FQMBI4L476Y6VKPPOPW7RKNCC5XFDJ2A, seg_pos={0 0}, upload=192678]: Done uploading piece. err: <nil> Nov 8 14:26:23.486413928 (218h45m55.182262681s) [noise=true , address=145.79.227.20:4437 , piece_id=G5RTYGERAVOTQDZGCG6NEKRCQAUOVJNYXO4LEBJ7XHJZ6FDXTULQ, seg_pos={0 0}, upload=192678]: Done uploading piece. err: <nil> Nov 8 14:26:24.659870720 (218h45m56.355719473s) [noise=true , address=145.79.227.26:4653 , piece_id=UIV3WFYLH4JZSIIHZUBL56THDPAYK7DMRUQG5TVTHHEF3XGN6NKQ, seg_pos={0 0}, upload=192678]: Done uploading piece. err: <nil> Nov 8 14:26:34.701060537 (218h46m6.3969093s) [noise=true , address=145.79.227.8:4329 , piece_id=JJFSJ7USPZFTPXVUZVLMXRAHYLARBQZLWWU7ITXFNESYVVL4QBMQ, seg_pos={0 0}, upload=192678]: Done uploading piece. err: <nil> Nov 8 14:26:37.183628509 (218h46m8.879477262s) [noise=true , address=145.79.227.10:4402 , piece_id=IRNQEX73D4AEXNYXEWWKCKEMYFC3YXIPX6HSDYYBNDKBM77ROBEA, seg_pos={0 0}, upload=192678]: Done uploading piece. err: <nil> Nov 8 14:26:37.186238899 (218h46m8.882087662s) [noise=true , address=145.79.227.16:4911 , piece_id=QKITSRMSCHOZLCQ7X6GARTE3GS24G3CZFXJIBKDM3VWOVQLMLEHA, seg_pos={0 0}, upload=192678]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:145.79.227.16:4911): protocol: write tcp 188.166.225.170:56440->145.79.227.16:4911: use of closed network connection; write tcp 188.166.225.170:56440->145.79.227.16:4911: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56440->145.79.227.16:4911: use of closed network connection Nov 8 14:26:37.186462625 (218h46m8.882311378s) [noise=true , address=145.79.227.21:4475 , piece_id=6F7IZNQLORMTKUF576FJA6L7FBG66INEUFPQYHL77WLC6DYF32GQ, seg_pos={0 0}, upload=192678]: Done uploading piece. err: ecclient: upload failed (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk, address:145.79.227.21:4475): protocol: expected piece hash; context deadline exceeded; EOF Nov 8 14:26:37.186739442 (218h46m8.882588195s) [noise=true , address=145.79.227.27:4689 , piece_id=JEEKOLHURPGNFSR4YCWL3UD6BU2MT7VW4S7BU3YKE7BJSUEYLPGQ, seg_pos={0 0}, upload=192678]: Done uploading piece. err: ecclient: upload failed (node:1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, address:145.79.227.27:4689): protocol: write tcp 188.166.225.170:37312->145.79.227.27:4689: use of closed network connection; write tcp 188.166.225.170:37312->145.79.227.27:4689: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37312->145.79.227.27:4689: use of closed network connection Nov 8 14:26:37.186915056 (218h46m8.88276381s) [noise=true , address=145.79.227.7:4295 , piece_id=5OY4NKICVFGAWR7JPOOYIYRXR442IIUDDWTN3MT6KAR4ZHXTKPNQ, seg_pos={0 0}, upload=192678]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:145.79.227.7:4295): protocol: expected piece hash; context deadline exceeded; EOF Nov 8 14:26:37.187072807 (218h46m8.88292156s) [noise=true , address=145.79.227.9:4366 , piece_id=3BINODFKOOG7BYULCICW6S64YZOU5D2Q5MFZYGPSSA34XWM762BQ, seg_pos={0 0}, upload=192678]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: expected piece hash; context deadline exceeded; EOF Nov 8 14:26:37.187107163 (218h46m8.882955916s) [seg_pos={0 0}, upload=192678]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 8 14:26:37.433482306 (218h46m9.129331059s) [upload=192678]: Done uploading segments... Nov 8 14:26:37.433526460 (218h46m9.129375213s) [upload=192678]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 8.936399206s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 6.916806563s trace: https://jaeger.dev.storj.io/trace/3f377bb0d96d25c FAIL (2025-11-08 14:58:54 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Nov 8 14:26:07.882330547 (218h45m39.5781793s) [upload=192678]: Starting upload Nov 8 14:26:07.882433824 (218h45m39.578282567s) [upload=192678]: Done starting upload Nov 8 14:26:07.882492927 (218h45m39.57834167s) [upload=192678]: Uploading segments... Nov 8 14:26:07.882645367 (218h45m39.57849411s) [upload=192678]: Got next segment. Inline: false Nov 8 14:26:07.882698468 (218h45m39.578547221s) [upload=192678]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 8 14:26:07.882704229 (218h45m39.578552982s) [upload=192678]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 8 14:26:08.131091114 (218h45m39.826939897s) [seg_pos={0 0}, upload=192678]: Begin upload segment... Nov 8 14:26:08.131165496 (218h45m39.827014239s) [seg_pos={0 0}, upload=192678]: Done begin upload segment. Nov 8 14:26:08.131220160 (218h45m39.827068913s) [noise=true , address=145.79.227.16:4911 , piece_id=QKITSRMSCHOZLCQ7X6GARTE3GS24G3CZFXJIBKDM3VWOVQLMLEHA, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.131304631 (218h45m39.827153384s) [noise=true , address=145.79.227.27:4689 , piece_id=JEEKOLHURPGNFSR4YCWL3UD6BU2MT7VW4S7BU3YKE7BJSUEYLPGQ, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.132190408 (218h45m39.828039161s) [noise=true , address=145.79.227.7:4295 , piece_id=5OY4NKICVFGAWR7JPOOYIYRXR442IIUDDWTN3MT6KAR4ZHXTKPNQ, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.132569360 (218h45m39.828418113s) [noise=true , address=145.79.227.31:4798 , piece_id=5Q7NYNXM2IMV7YUCB464FQMBI4L476Y6VKPPOPW7RKNCC5XFDJ2A, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.133276446 (218h45m39.829125199s) [noise=true , address=145.79.227.26:4653 , piece_id=UIV3WFYLH4JZSIIHZUBL56THDPAYK7DMRUQG5TVTHHEF3XGN6NKQ, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.134167893 (218h45m39.830016646s) [noise=true , address=145.79.227.26:4655 , piece_id=GP4F5L6HH6S4ZEYWS7XRG5KJSYFGOX5JXQBTQEOXFDFSNUBHZJ6A, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.134314192 (218h45m39.830162945s) [noise=true , address=145.79.227.20:4437 , piece_id=G5RTYGERAVOTQDZGCG6NEKRCQAUOVJNYXO4LEBJ7XHJZ6FDXTULQ, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.135016950 (218h45m39.830865703s) [noise=true , address=145.79.227.10:4402 , piece_id=IRNQEX73D4AEXNYXEWWKCKEMYFC3YXIPX6HSDYYBNDKBM77ROBEA, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.135235305 (218h45m39.831084059s) [noise=true , address=145.79.227.22:4509 , piece_id=6RJJCCT2Z5U3JOWWOBLRIDK5CTWKXU7NR4OIREAQ2JDE2KYDJHOQ, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.135440236 (218h45m39.831288979s) [noise=true , address=145.79.227.2:4113 , piece_id=Q5E3IXSHGFKIFG7EFAPW2XWYP63TO3BC5C7PZJDPXXD2UKDQDJLA, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.136611887 (218h45m39.83246064s) [noise=true , address=77.90.31.15:4699 , piece_id=Z4PXJFTM7XMK62ORTMHCWEJ7OQF6PIYQCMAWJC3DT5RZ7CEH4KXA, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.137027559 (218h45m39.832876312s) [noise=true , address=145.79.227.8:4329 , piece_id=JJFSJ7USPZFTPXVUZVLMXRAHYLARBQZLWWU7ITXFNESYVVL4QBMQ, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.137896133 (218h45m39.833744886s) [noise=true , address=77.90.31.17:4771 , piece_id=LUZ4KXA4YUVTQYKXVEPTEXZ4MFQ5IWFTY3C2LUUVQ3N6DOVO6ZXA, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.139107940 (218h45m39.834956693s) [noise=true , address=145.79.227.21:4475 , piece_id=6F7IZNQLORMTKUF576FJA6L7FBG66INEUFPQYHL77WLC6DYF32GQ, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.139872657 (218h45m39.83572141s) [noise=true , address=145.79.227.9:4366 , piece_id=3BINODFKOOG7BYULCICW6S64YZOU5D2Q5MFZYGPSSA34XWM762BQ, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.140005960 (218h45m39.835854713s) [noise=true , address=77.90.31.16:4734 , piece_id=EKYBQDVNAYTGEPWU443RFLUX7OKCTWBPXA5UMCLZ22M2PBXO3E3Q, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:10.015794232 (218h45m41.711642985s) [upload=192678]: Next returned nil segment Nov 8 14:26:10.015816414 (218h45m41.711665167s) [upload=192678]: Waiting for error group managing segments... Nov 8 14:26:10.314091526 (218h45m42.009940279s) [noise=true , address=77.90.31.16:4734 , piece_id=EKYBQDVNAYTGEPWU443RFLUX7OKCTWBPXA5UMCLZ22M2PBXO3E3Q, seg_pos={0 0}, upload=192678]: Done uploading piece. err: <nil> Nov 8 14:26:10.388439672 (218h45m42.084288465s) [noise=true , address=77.90.31.15:4699 , piece_id=Z4PXJFTM7XMK62ORTMHCWEJ7OQF6PIYQCMAWJC3DT5RZ7CEH4KXA, seg_pos={0 0}, upload=192678]: Done uploading piece. err: <nil> Nov 8 14:26:11.119863095 (218h45m42.815711838s) [noise=true , address=77.90.31.17:4771 , piece_id=LUZ4KXA4YUVTQYKXVEPTEXZ4MFQ5IWFTY3C2LUUVQ3N6DOVO6ZXA, seg_pos={0 0}, upload=192678]: Done uploading piece. err: <nil> Nov 8 14:26:14.806769255 (218h45m46.502618008s) [noise=true , address=145.79.227.2:4113 , piece_id=Q5E3IXSHGFKIFG7EFAPW2XWYP63TO3BC5C7PZJDPXXD2UKDQDJLA, seg_pos={0 0}, upload=192678]: Done uploading piece. err: <nil> Nov 8 14:26:15.197752745 (218h45m46.893601498s) [noise=true , address=145.79.227.26:4655 , piece_id=GP4F5L6HH6S4ZEYWS7XRG5KJSYFGOX5JXQBTQEOXFDFSNUBHZJ6A, seg_pos={0 0}, upload=192678]: Done uploading piece. err: <nil> Nov 8 14:26:17.425396997 (218h45m49.12124575s) [noise=true , address=145.79.227.22:4509 , piece_id=6RJJCCT2Z5U3JOWWOBLRIDK5CTWKXU7NR4OIREAQ2JDE2KYDJHOQ, seg_pos={0 0}, upload=192678]: Done uploading piece. err: <nil> Nov 8 14:26:21.856283285 (218h45m53.552132048s) [noise=true , address=145.79.227.31:4798 , piece_id=5Q7NYNXM2IMV7YUCB464FQMBI4L476Y6VKPPOPW7RKNCC5XFDJ2A, seg_pos={0 0}, upload=192678]: Done uploading piece. err: <nil> Nov 8 14:26:23.486413928 (218h45m55.182262681s) [noise=true , address=145.79.227.20:4437 , piece_id=G5RTYGERAVOTQDZGCG6NEKRCQAUOVJNYXO4LEBJ7XHJZ6FDXTULQ, seg_pos={0 0}, upload=192678]: Done uploading piece. err: <nil> Nov 8 14:26:24.659870720 (218h45m56.355719473s) [noise=true , address=145.79.227.26:4653 , piece_id=UIV3WFYLH4JZSIIHZUBL56THDPAYK7DMRUQG5TVTHHEF3XGN6NKQ, seg_pos={0 0}, upload=192678]: Done uploading piece. err: <nil> Nov 8 14:26:34.701060537 (218h46m6.3969093s) [noise=true , address=145.79.227.8:4329 , piece_id=JJFSJ7USPZFTPXVUZVLMXRAHYLARBQZLWWU7ITXFNESYVVL4QBMQ, seg_pos={0 0}, upload=192678]: Done uploading piece. err: <nil> Nov 8 14:26:37.183628509 (218h46m8.879477262s) [noise=true , address=145.79.227.10:4402 , piece_id=IRNQEX73D4AEXNYXEWWKCKEMYFC3YXIPX6HSDYYBNDKBM77ROBEA, seg_pos={0 0}, upload=192678]: Done uploading piece. err: <nil> Nov 8 14:26:37.186238899 (218h46m8.882087662s) [noise=true , address=145.79.227.16:4911 , piece_id=QKITSRMSCHOZLCQ7X6GARTE3GS24G3CZFXJIBKDM3VWOVQLMLEHA, seg_pos={0 0}, upload=192678]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:145.79.227.16:4911): protocol: write tcp 188.166.225.170:56440->145.79.227.16:4911: use of closed network connection; write tcp 188.166.225.170:56440->145.79.227.16:4911: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56440->145.79.227.16:4911: use of closed network connection Nov 8 14:26:37.186462625 (218h46m8.882311378s) [noise=true , address=145.79.227.21:4475 , piece_id=6F7IZNQLORMTKUF576FJA6L7FBG66INEUFPQYHL77WLC6DYF32GQ, seg_pos={0 0}, upload=192678]: Done uploading piece. err: ecclient: upload failed (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk, address:145.79.227.21:4475): protocol: expected piece hash; context deadline exceeded; EOF Nov 8 14:26:37.186739442 (218h46m8.882588195s) [noise=true , address=145.79.227.27:4689 , piece_id=JEEKOLHURPGNFSR4YCWL3UD6BU2MT7VW4S7BU3YKE7BJSUEYLPGQ, seg_pos={0 0}, upload=192678]: Done uploading piece. err: ecclient: upload failed (node:1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, address:145.79.227.27:4689): protocol: write tcp 188.166.225.170:37312->145.79.227.27:4689: use of closed network connection; write tcp 188.166.225.170:37312->145.79.227.27:4689: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37312->145.79.227.27:4689: use of closed network connection Nov 8 14:26:37.186915056 (218h46m8.88276381s) [noise=true , address=145.79.227.7:4295 , piece_id=5OY4NKICVFGAWR7JPOOYIYRXR442IIUDDWTN3MT6KAR4ZHXTKPNQ, seg_pos={0 0}, upload=192678]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:145.79.227.7:4295): protocol: expected piece hash; context deadline exceeded; EOF Nov 8 14:26:37.187072807 (218h46m8.88292156s) [noise=true , address=145.79.227.9:4366 , piece_id=3BINODFKOOG7BYULCICW6S64YZOU5D2Q5MFZYGPSSA34XWM762BQ, seg_pos={0 0}, upload=192678]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: expected piece hash; context deadline exceeded; EOF Nov 8 14:26:37.187107163 (218h46m8.882955916s) [seg_pos={0 0}, upload=192678]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 8 14:26:37.433482306 (218h46m9.129331059s) [upload=192678]: Done uploading segments... Nov 8 14:26:37.433526460 (218h46m9.129375213s) [upload=192678]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 6.916838263s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 10.026773435s trace: https://jaeger.dev.storj.io/trace/3f377bb0d96d25c FAIL (2025-11-08 14:57:59 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Nov 8 14:26:07.882330547 (218h45m39.5781793s) [upload=192678]: Starting upload Nov 8 14:26:07.882433824 (218h45m39.578282567s) [upload=192678]: Done starting upload Nov 8 14:26:07.882492927 (218h45m39.57834167s) [upload=192678]: Uploading segments... Nov 8 14:26:07.882645367 (218h45m39.57849411s) [upload=192678]: Got next segment. Inline: false Nov 8 14:26:07.882698468 (218h45m39.578547221s) [upload=192678]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 8 14:26:07.882704229 (218h45m39.578552982s) [upload=192678]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 8 14:26:08.131091114 (218h45m39.826939897s) [seg_pos={0 0}, upload=192678]: Begin upload segment... Nov 8 14:26:08.131165496 (218h45m39.827014239s) [seg_pos={0 0}, upload=192678]: Done begin upload segment. Nov 8 14:26:08.131220160 (218h45m39.827068913s) [noise=true , address=145.79.227.16:4911 , piece_id=QKITSRMSCHOZLCQ7X6GARTE3GS24G3CZFXJIBKDM3VWOVQLMLEHA, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.131304631 (218h45m39.827153384s) [noise=true , address=145.79.227.27:4689 , piece_id=JEEKOLHURPGNFSR4YCWL3UD6BU2MT7VW4S7BU3YKE7BJSUEYLPGQ, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.132190408 (218h45m39.828039161s) [noise=true , address=145.79.227.7:4295 , piece_id=5OY4NKICVFGAWR7JPOOYIYRXR442IIUDDWTN3MT6KAR4ZHXTKPNQ, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.132569360 (218h45m39.828418113s) [noise=true , address=145.79.227.31:4798 , piece_id=5Q7NYNXM2IMV7YUCB464FQMBI4L476Y6VKPPOPW7RKNCC5XFDJ2A, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.133276446 (218h45m39.829125199s) [noise=true , address=145.79.227.26:4653 , piece_id=UIV3WFYLH4JZSIIHZUBL56THDPAYK7DMRUQG5TVTHHEF3XGN6NKQ, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.134167893 (218h45m39.830016646s) [noise=true , address=145.79.227.26:4655 , piece_id=GP4F5L6HH6S4ZEYWS7XRG5KJSYFGOX5JXQBTQEOXFDFSNUBHZJ6A, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.134314192 (218h45m39.830162945s) [noise=true , address=145.79.227.20:4437 , piece_id=G5RTYGERAVOTQDZGCG6NEKRCQAUOVJNYXO4LEBJ7XHJZ6FDXTULQ, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.135016950 (218h45m39.830865703s) [noise=true , address=145.79.227.10:4402 , piece_id=IRNQEX73D4AEXNYXEWWKCKEMYFC3YXIPX6HSDYYBNDKBM77ROBEA, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.135235305 (218h45m39.831084059s) [noise=true , address=145.79.227.22:4509 , piece_id=6RJJCCT2Z5U3JOWWOBLRIDK5CTWKXU7NR4OIREAQ2JDE2KYDJHOQ, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.135440236 (218h45m39.831288979s) [noise=true , address=145.79.227.2:4113 , piece_id=Q5E3IXSHGFKIFG7EFAPW2XWYP63TO3BC5C7PZJDPXXD2UKDQDJLA, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.136611887 (218h45m39.83246064s) [noise=true , address=77.90.31.15:4699 , piece_id=Z4PXJFTM7XMK62ORTMHCWEJ7OQF6PIYQCMAWJC3DT5RZ7CEH4KXA, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.137027559 (218h45m39.832876312s) [noise=true , address=145.79.227.8:4329 , piece_id=JJFSJ7USPZFTPXVUZVLMXRAHYLARBQZLWWU7ITXFNESYVVL4QBMQ, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.137896133 (218h45m39.833744886s) [noise=true , address=77.90.31.17:4771 , piece_id=LUZ4KXA4YUVTQYKXVEPTEXZ4MFQ5IWFTY3C2LUUVQ3N6DOVO6ZXA, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.139107940 (218h45m39.834956693s) [noise=true , address=145.79.227.21:4475 , piece_id=6F7IZNQLORMTKUF576FJA6L7FBG66INEUFPQYHL77WLC6DYF32GQ, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.139872657 (218h45m39.83572141s) [noise=true , address=145.79.227.9:4366 , piece_id=3BINODFKOOG7BYULCICW6S64YZOU5D2Q5MFZYGPSSA34XWM762BQ, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.140005960 (218h45m39.835854713s) [noise=true , address=77.90.31.16:4734 , piece_id=EKYBQDVNAYTGEPWU443RFLUX7OKCTWBPXA5UMCLZ22M2PBXO3E3Q, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:10.015794232 (218h45m41.711642985s) [upload=192678]: Next returned nil segment Nov 8 14:26:10.015816414 (218h45m41.711665167s) [upload=192678]: Waiting for error group managing segments... Nov 8 14:26:10.314091526 (218h45m42.009940279s) [noise=true , address=77.90.31.16:4734 , piece_id=EKYBQDVNAYTGEPWU443RFLUX7OKCTWBPXA5UMCLZ22M2PBXO3E3Q, seg_pos={0 0}, upload=192678]: Done uploading piece. err: <nil> Nov 8 14:26:10.388439672 (218h45m42.084288465s) [noise=true , address=77.90.31.15:4699 , piece_id=Z4PXJFTM7XMK62ORTMHCWEJ7OQF6PIYQCMAWJC3DT5RZ7CEH4KXA, seg_pos={0 0}, upload=192678]: Done uploading piece. err: <nil> Nov 8 14:26:11.119863095 (218h45m42.815711838s) [noise=true , address=77.90.31.17:4771 , piece_id=LUZ4KXA4YUVTQYKXVEPTEXZ4MFQ5IWFTY3C2LUUVQ3N6DOVO6ZXA, seg_pos={0 0}, upload=192678]: Done uploading piece. err: <nil> Nov 8 14:26:14.806769255 (218h45m46.502618008s) [noise=true , address=145.79.227.2:4113 , piece_id=Q5E3IXSHGFKIFG7EFAPW2XWYP63TO3BC5C7PZJDPXXD2UKDQDJLA, seg_pos={0 0}, upload=192678]: Done uploading piece. err: <nil> Nov 8 14:26:15.197752745 (218h45m46.893601498s) [noise=true , address=145.79.227.26:4655 , piece_id=GP4F5L6HH6S4ZEYWS7XRG5KJSYFGOX5JXQBTQEOXFDFSNUBHZJ6A, seg_pos={0 0}, upload=192678]: Done uploading piece. err: <nil> Nov 8 14:26:17.425396997 (218h45m49.12124575s) [noise=true , address=145.79.227.22:4509 , piece_id=6RJJCCT2Z5U3JOWWOBLRIDK5CTWKXU7NR4OIREAQ2JDE2KYDJHOQ, seg_pos={0 0}, upload=192678]: Done uploading piece. err: <nil> Nov 8 14:26:21.856283285 (218h45m53.552132048s) [noise=true , address=145.79.227.31:4798 , piece_id=5Q7NYNXM2IMV7YUCB464FQMBI4L476Y6VKPPOPW7RKNCC5XFDJ2A, seg_pos={0 0}, upload=192678]: Done uploading piece. err: <nil> Nov 8 14:26:23.486413928 (218h45m55.182262681s) [noise=true , address=145.79.227.20:4437 , piece_id=G5RTYGERAVOTQDZGCG6NEKRCQAUOVJNYXO4LEBJ7XHJZ6FDXTULQ, seg_pos={0 0}, upload=192678]: Done uploading piece. err: <nil> Nov 8 14:26:24.659870720 (218h45m56.355719473s) [noise=true , address=145.79.227.26:4653 , piece_id=UIV3WFYLH4JZSIIHZUBL56THDPAYK7DMRUQG5TVTHHEF3XGN6NKQ, seg_pos={0 0}, upload=192678]: Done uploading piece. err: <nil> Nov 8 14:26:34.701060537 (218h46m6.3969093s) [noise=true , address=145.79.227.8:4329 , piece_id=JJFSJ7USPZFTPXVUZVLMXRAHYLARBQZLWWU7ITXFNESYVVL4QBMQ, seg_pos={0 0}, upload=192678]: Done uploading piece. err: <nil> Nov 8 14:26:37.183628509 (218h46m8.879477262s) [noise=true , address=145.79.227.10:4402 , piece_id=IRNQEX73D4AEXNYXEWWKCKEMYFC3YXIPX6HSDYYBNDKBM77ROBEA, seg_pos={0 0}, upload=192678]: Done uploading piece. err: <nil> Nov 8 14:26:37.186238899 (218h46m8.882087662s) [noise=true , address=145.79.227.16:4911 , piece_id=QKITSRMSCHOZLCQ7X6GARTE3GS24G3CZFXJIBKDM3VWOVQLMLEHA, seg_pos={0 0}, upload=192678]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:145.79.227.16:4911): protocol: write tcp 188.166.225.170:56440->145.79.227.16:4911: use of closed network connection; write tcp 188.166.225.170:56440->145.79.227.16:4911: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56440->145.79.227.16:4911: use of closed network connection Nov 8 14:26:37.186462625 (218h46m8.882311378s) [noise=true , address=145.79.227.21:4475 , piece_id=6F7IZNQLORMTKUF576FJA6L7FBG66INEUFPQYHL77WLC6DYF32GQ, seg_pos={0 0}, upload=192678]: Done uploading piece. err: ecclient: upload failed (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk, address:145.79.227.21:4475): protocol: expected piece hash; context deadline exceeded; EOF Nov 8 14:26:37.186739442 (218h46m8.882588195s) [noise=true , address=145.79.227.27:4689 , piece_id=JEEKOLHURPGNFSR4YCWL3UD6BU2MT7VW4S7BU3YKE7BJSUEYLPGQ, seg_pos={0 0}, upload=192678]: Done uploading piece. err: ecclient: upload failed (node:1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, address:145.79.227.27:4689): protocol: write tcp 188.166.225.170:37312->145.79.227.27:4689: use of closed network connection; write tcp 188.166.225.170:37312->145.79.227.27:4689: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37312->145.79.227.27:4689: use of closed network connection Nov 8 14:26:37.186915056 (218h46m8.88276381s) [noise=true , address=145.79.227.7:4295 , piece_id=5OY4NKICVFGAWR7JPOOYIYRXR442IIUDDWTN3MT6KAR4ZHXTKPNQ, seg_pos={0 0}, upload=192678]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:145.79.227.7:4295): protocol: expected piece hash; context deadline exceeded; EOF Nov 8 14:26:37.187072807 (218h46m8.88292156s) [noise=true , address=145.79.227.9:4366 , piece_id=3BINODFKOOG7BYULCICW6S64YZOU5D2Q5MFZYGPSSA34XWM762BQ, seg_pos={0 0}, upload=192678]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: expected piece hash; context deadline exceeded; EOF Nov 8 14:26:37.187107163 (218h46m8.882955916s) [seg_pos={0 0}, upload=192678]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 8 14:26:37.433482306 (218h46m9.129331059s) [upload=192678]: Done uploading segments... Nov 8 14:26:37.433526460 (218h46m9.129375213s) [upload=192678]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 10.02682324s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 9.047290751s trace: https://jaeger.dev.storj.io/trace/3f377bb0d96d25c FAIL (2025-11-08 14:56:55 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Nov 8 14:26:07.882330547 (218h45m39.5781793s) [upload=192678]: Starting upload Nov 8 14:26:07.882433824 (218h45m39.578282567s) [upload=192678]: Done starting upload Nov 8 14:26:07.882492927 (218h45m39.57834167s) [upload=192678]: Uploading segments... Nov 8 14:26:07.882645367 (218h45m39.57849411s) [upload=192678]: Got next segment. Inline: false Nov 8 14:26:07.882698468 (218h45m39.578547221s) [upload=192678]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 8 14:26:07.882704229 (218h45m39.578552982s) [upload=192678]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 8 14:26:08.131091114 (218h45m39.826939897s) [seg_pos={0 0}, upload=192678]: Begin upload segment... Nov 8 14:26:08.131165496 (218h45m39.827014239s) [seg_pos={0 0}, upload=192678]: Done begin upload segment. Nov 8 14:26:08.131220160 (218h45m39.827068913s) [noise=true , address=145.79.227.16:4911 , piece_id=QKITSRMSCHOZLCQ7X6GARTE3GS24G3CZFXJIBKDM3VWOVQLMLEHA, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.131304631 (218h45m39.827153384s) [noise=true , address=145.79.227.27:4689 , piece_id=JEEKOLHURPGNFSR4YCWL3UD6BU2MT7VW4S7BU3YKE7BJSUEYLPGQ, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.132190408 (218h45m39.828039161s) [noise=true , address=145.79.227.7:4295 , piece_id=5OY4NKICVFGAWR7JPOOYIYRXR442IIUDDWTN3MT6KAR4ZHXTKPNQ, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.132569360 (218h45m39.828418113s) [noise=true , address=145.79.227.31:4798 , piece_id=5Q7NYNXM2IMV7YUCB464FQMBI4L476Y6VKPPOPW7RKNCC5XFDJ2A, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.133276446 (218h45m39.829125199s) [noise=true , address=145.79.227.26:4653 , piece_id=UIV3WFYLH4JZSIIHZUBL56THDPAYK7DMRUQG5TVTHHEF3XGN6NKQ, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.134167893 (218h45m39.830016646s) [noise=true , address=145.79.227.26:4655 , piece_id=GP4F5L6HH6S4ZEYWS7XRG5KJSYFGOX5JXQBTQEOXFDFSNUBHZJ6A, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.134314192 (218h45m39.830162945s) [noise=true , address=145.79.227.20:4437 , piece_id=G5RTYGERAVOTQDZGCG6NEKRCQAUOVJNYXO4LEBJ7XHJZ6FDXTULQ, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.135016950 (218h45m39.830865703s) [noise=true , address=145.79.227.10:4402 , piece_id=IRNQEX73D4AEXNYXEWWKCKEMYFC3YXIPX6HSDYYBNDKBM77ROBEA, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.135235305 (218h45m39.831084059s) [noise=true , address=145.79.227.22:4509 , piece_id=6RJJCCT2Z5U3JOWWOBLRIDK5CTWKXU7NR4OIREAQ2JDE2KYDJHOQ, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.135440236 (218h45m39.831288979s) [noise=true , address=145.79.227.2:4113 , piece_id=Q5E3IXSHGFKIFG7EFAPW2XWYP63TO3BC5C7PZJDPXXD2UKDQDJLA, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.136611887 (218h45m39.83246064s) [noise=true , address=77.90.31.15:4699 , piece_id=Z4PXJFTM7XMK62ORTMHCWEJ7OQF6PIYQCMAWJC3DT5RZ7CEH4KXA, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.137027559 (218h45m39.832876312s) [noise=true , address=145.79.227.8:4329 , piece_id=JJFSJ7USPZFTPXVUZVLMXRAHYLARBQZLWWU7ITXFNESYVVL4QBMQ, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.137896133 (218h45m39.833744886s) [noise=true , address=77.90.31.17:4771 , piece_id=LUZ4KXA4YUVTQYKXVEPTEXZ4MFQ5IWFTY3C2LUUVQ3N6DOVO6ZXA, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.139107940 (218h45m39.834956693s) [noise=true , address=145.79.227.21:4475 , piece_id=6F7IZNQLORMTKUF576FJA6L7FBG66INEUFPQYHL77WLC6DYF32GQ, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.139872657 (218h45m39.83572141s) [noise=true , address=145.79.227.9:4366 , piece_id=3BINODFKOOG7BYULCICW6S64YZOU5D2Q5MFZYGPSSA34XWM762BQ, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.140005960 (218h45m39.835854713s) [noise=true , address=77.90.31.16:4734 , piece_id=EKYBQDVNAYTGEPWU443RFLUX7OKCTWBPXA5UMCLZ22M2PBXO3E3Q, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:10.015794232 (218h45m41.711642985s) [upload=192678]: Next returned nil segment Nov 8 14:26:10.015816414 (218h45m41.711665167s) [upload=192678]: Waiting for error group managing segments... Nov 8 14:26:10.314091526 (218h45m42.009940279s) [noise=true , address=77.90.31.16:4734 , piece_id=EKYBQDVNAYTGEPWU443RFLUX7OKCTWBPXA5UMCLZ22M2PBXO3E3Q, seg_pos={0 0}, upload=192678]: Done uploading piece. err: <nil> Nov 8 14:26:10.388439672 (218h45m42.084288465s) [noise=true , address=77.90.31.15:4699 , piece_id=Z4PXJFTM7XMK62ORTMHCWEJ7OQF6PIYQCMAWJC3DT5RZ7CEH4KXA, seg_pos={0 0}, upload=192678]: Done uploading piece. err: <nil> Nov 8 14:26:11.119863095 (218h45m42.815711838s) [noise=true , address=77.90.31.17:4771 , piece_id=LUZ4KXA4YUVTQYKXVEPTEXZ4MFQ5IWFTY3C2LUUVQ3N6DOVO6ZXA, seg_pos={0 0}, upload=192678]: Done uploading piece. err: <nil> Nov 8 14:26:14.806769255 (218h45m46.502618008s) [noise=true , address=145.79.227.2:4113 , piece_id=Q5E3IXSHGFKIFG7EFAPW2XWYP63TO3BC5C7PZJDPXXD2UKDQDJLA, seg_pos={0 0}, upload=192678]: Done uploading piece. err: <nil> Nov 8 14:26:15.197752745 (218h45m46.893601498s) [noise=true , address=145.79.227.26:4655 , piece_id=GP4F5L6HH6S4ZEYWS7XRG5KJSYFGOX5JXQBTQEOXFDFSNUBHZJ6A, seg_pos={0 0}, upload=192678]: Done uploading piece. err: <nil> Nov 8 14:26:17.425396997 (218h45m49.12124575s) [noise=true , address=145.79.227.22:4509 , piece_id=6RJJCCT2Z5U3JOWWOBLRIDK5CTWKXU7NR4OIREAQ2JDE2KYDJHOQ, seg_pos={0 0}, upload=192678]: Done uploading piece. err: <nil> Nov 8 14:26:21.856283285 (218h45m53.552132048s) [noise=true , address=145.79.227.31:4798 , piece_id=5Q7NYNXM2IMV7YUCB464FQMBI4L476Y6VKPPOPW7RKNCC5XFDJ2A, seg_pos={0 0}, upload=192678]: Done uploading piece. err: <nil> Nov 8 14:26:23.486413928 (218h45m55.182262681s) [noise=true , address=145.79.227.20:4437 , piece_id=G5RTYGERAVOTQDZGCG6NEKRCQAUOVJNYXO4LEBJ7XHJZ6FDXTULQ, seg_pos={0 0}, upload=192678]: Done uploading piece. err: <nil> Nov 8 14:26:24.659870720 (218h45m56.355719473s) [noise=true , address=145.79.227.26:4653 , piece_id=UIV3WFYLH4JZSIIHZUBL56THDPAYK7DMRUQG5TVTHHEF3XGN6NKQ, seg_pos={0 0}, upload=192678]: Done uploading piece. err: <nil> Nov 8 14:26:34.701060537 (218h46m6.3969093s) [noise=true , address=145.79.227.8:4329 , piece_id=JJFSJ7USPZFTPXVUZVLMXRAHYLARBQZLWWU7ITXFNESYVVL4QBMQ, seg_pos={0 0}, upload=192678]: Done uploading piece. err: <nil> Nov 8 14:26:37.183628509 (218h46m8.879477262s) [noise=true , address=145.79.227.10:4402 , piece_id=IRNQEX73D4AEXNYXEWWKCKEMYFC3YXIPX6HSDYYBNDKBM77ROBEA, seg_pos={0 0}, upload=192678]: Done uploading piece. err: <nil> Nov 8 14:26:37.186238899 (218h46m8.882087662s) [noise=true , address=145.79.227.16:4911 , piece_id=QKITSRMSCHOZLCQ7X6GARTE3GS24G3CZFXJIBKDM3VWOVQLMLEHA, seg_pos={0 0}, upload=192678]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:145.79.227.16:4911): protocol: write tcp 188.166.225.170:56440->145.79.227.16:4911: use of closed network connection; write tcp 188.166.225.170:56440->145.79.227.16:4911: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56440->145.79.227.16:4911: use of closed network connection Nov 8 14:26:37.186462625 (218h46m8.882311378s) [noise=true , address=145.79.227.21:4475 , piece_id=6F7IZNQLORMTKUF576FJA6L7FBG66INEUFPQYHL77WLC6DYF32GQ, seg_pos={0 0}, upload=192678]: Done uploading piece. err: ecclient: upload failed (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk, address:145.79.227.21:4475): protocol: expected piece hash; context deadline exceeded; EOF Nov 8 14:26:37.186739442 (218h46m8.882588195s) [noise=true , address=145.79.227.27:4689 , piece_id=JEEKOLHURPGNFSR4YCWL3UD6BU2MT7VW4S7BU3YKE7BJSUEYLPGQ, seg_pos={0 0}, upload=192678]: Done uploading piece. err: ecclient: upload failed (node:1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, address:145.79.227.27:4689): protocol: write tcp 188.166.225.170:37312->145.79.227.27:4689: use of closed network connection; write tcp 188.166.225.170:37312->145.79.227.27:4689: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37312->145.79.227.27:4689: use of closed network connection Nov 8 14:26:37.186915056 (218h46m8.88276381s) [noise=true , address=145.79.227.7:4295 , piece_id=5OY4NKICVFGAWR7JPOOYIYRXR442IIUDDWTN3MT6KAR4ZHXTKPNQ, seg_pos={0 0}, upload=192678]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:145.79.227.7:4295): protocol: expected piece hash; context deadline exceeded; EOF Nov 8 14:26:37.187072807 (218h46m8.88292156s) [noise=true , address=145.79.227.9:4366 , piece_id=3BINODFKOOG7BYULCICW6S64YZOU5D2Q5MFZYGPSSA34XWM762BQ, seg_pos={0 0}, upload=192678]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: expected piece hash; context deadline exceeded; EOF Nov 8 14:26:37.187107163 (218h46m8.882955916s) [seg_pos={0 0}, upload=192678]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 8 14:26:37.433482306 (218h46m9.129331059s) [upload=192678]: Done uploading segments... Nov 8 14:26:37.433526460 (218h46m9.129375213s) [upload=192678]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 9.047349042s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 8.186397034s trace: https://jaeger.dev.storj.io/trace/3f377bb0d96d25c FAIL (2025-11-08 14:56:06 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Nov 8 14:26:07.882330547 (218h45m39.5781793s) [upload=192678]: Starting upload Nov 8 14:26:07.882433824 (218h45m39.578282567s) [upload=192678]: Done starting upload Nov 8 14:26:07.882492927 (218h45m39.57834167s) [upload=192678]: Uploading segments... Nov 8 14:26:07.882645367 (218h45m39.57849411s) [upload=192678]: Got next segment. Inline: false Nov 8 14:26:07.882698468 (218h45m39.578547221s) [upload=192678]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 8 14:26:07.882704229 (218h45m39.578552982s) [upload=192678]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 8 14:26:08.131091114 (218h45m39.826939897s) [seg_pos={0 0}, upload=192678]: Begin upload segment... Nov 8 14:26:08.131165496 (218h45m39.827014239s) [seg_pos={0 0}, upload=192678]: Done begin upload segment. Nov 8 14:26:08.131220160 (218h45m39.827068913s) [noise=true , address=145.79.227.16:4911 , piece_id=QKITSRMSCHOZLCQ7X6GARTE3GS24G3CZFXJIBKDM3VWOVQLMLEHA, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.131304631 (218h45m39.827153384s) [noise=true , address=145.79.227.27:4689 , piece_id=JEEKOLHURPGNFSR4YCWL3UD6BU2MT7VW4S7BU3YKE7BJSUEYLPGQ, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.132190408 (218h45m39.828039161s) [noise=true , address=145.79.227.7:4295 , piece_id=5OY4NKICVFGAWR7JPOOYIYRXR442IIUDDWTN3MT6KAR4ZHXTKPNQ, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.132569360 (218h45m39.828418113s) [noise=true , address=145.79.227.31:4798 , piece_id=5Q7NYNXM2IMV7YUCB464FQMBI4L476Y6VKPPOPW7RKNCC5XFDJ2A, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.133276446 (218h45m39.829125199s) [noise=true , address=145.79.227.26:4653 , piece_id=UIV3WFYLH4JZSIIHZUBL56THDPAYK7DMRUQG5TVTHHEF3XGN6NKQ, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.134167893 (218h45m39.830016646s) [noise=true , address=145.79.227.26:4655 , piece_id=GP4F5L6HH6S4ZEYWS7XRG5KJSYFGOX5JXQBTQEOXFDFSNUBHZJ6A, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.134314192 (218h45m39.830162945s) [noise=true , address=145.79.227.20:4437 , piece_id=G5RTYGERAVOTQDZGCG6NEKRCQAUOVJNYXO4LEBJ7XHJZ6FDXTULQ, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.135016950 (218h45m39.830865703s) [noise=true , address=145.79.227.10:4402 , piece_id=IRNQEX73D4AEXNYXEWWKCKEMYFC3YXIPX6HSDYYBNDKBM77ROBEA, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.135235305 (218h45m39.831084059s) [noise=true , address=145.79.227.22:4509 , piece_id=6RJJCCT2Z5U3JOWWOBLRIDK5CTWKXU7NR4OIREAQ2JDE2KYDJHOQ, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.135440236 (218h45m39.831288979s) [noise=true , address=145.79.227.2:4113 , piece_id=Q5E3IXSHGFKIFG7EFAPW2XWYP63TO3BC5C7PZJDPXXD2UKDQDJLA, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.136611887 (218h45m39.83246064s) [noise=true , address=77.90.31.15:4699 , piece_id=Z4PXJFTM7XMK62ORTMHCWEJ7OQF6PIYQCMAWJC3DT5RZ7CEH4KXA, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.137027559 (218h45m39.832876312s) [noise=true , address=145.79.227.8:4329 , piece_id=JJFSJ7USPZFTPXVUZVLMXRAHYLARBQZLWWU7ITXFNESYVVL4QBMQ, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.137896133 (218h45m39.833744886s) [noise=true , address=77.90.31.17:4771 , piece_id=LUZ4KXA4YUVTQYKXVEPTEXZ4MFQ5IWFTY3C2LUUVQ3N6DOVO6ZXA, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.139107940 (218h45m39.834956693s) [noise=true , address=145.79.227.21:4475 , piece_id=6F7IZNQLORMTKUF576FJA6L7FBG66INEUFPQYHL77WLC6DYF32GQ, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.139872657 (218h45m39.83572141s) [noise=true , address=145.79.227.9:4366 , piece_id=3BINODFKOOG7BYULCICW6S64YZOU5D2Q5MFZYGPSSA34XWM762BQ, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.140005960 (218h45m39.835854713s) [noise=true , address=77.90.31.16:4734 , piece_id=EKYBQDVNAYTGEPWU443RFLUX7OKCTWBPXA5UMCLZ22M2PBXO3E3Q, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:10.015794232 (218h45m41.711642985s) [upload=192678]: Next returned nil segment Nov 8 14:26:10.015816414 (218h45m41.711665167s) [upload=192678]: Waiting for error group managing segments... Nov 8 14:26:10.314091526 (218h45m42.009940279s) [noise=true , address=77.90.31.16:4734 , piece_id=EKYBQDVNAYTGEPWU443RFLUX7OKCTWBPXA5UMCLZ22M2PBXO3E3Q, seg_pos={0 0}, upload=192678]: Done uploading piece. err: <nil> Nov 8 14:26:10.388439672 (218h45m42.084288465s) [noise=true , address=77.90.31.15:4699 , piece_id=Z4PXJFTM7XMK62ORTMHCWEJ7OQF6PIYQCMAWJC3DT5RZ7CEH4KXA, seg_pos={0 0}, upload=192678]: Done uploading piece. err: <nil> Nov 8 14:26:11.119863095 (218h45m42.815711838s) [noise=true , address=77.90.31.17:4771 , piece_id=LUZ4KXA4YUVTQYKXVEPTEXZ4MFQ5IWFTY3C2LUUVQ3N6DOVO6ZXA, seg_pos={0 0}, upload=192678]: Done uploading piece. err: <nil> Nov 8 14:26:14.806769255 (218h45m46.502618008s) [noise=true , address=145.79.227.2:4113 , piece_id=Q5E3IXSHGFKIFG7EFAPW2XWYP63TO3BC5C7PZJDPXXD2UKDQDJLA, seg_pos={0 0}, upload=192678]: Done uploading piece. err: <nil> Nov 8 14:26:15.197752745 (218h45m46.893601498s) [noise=true , address=145.79.227.26:4655 , piece_id=GP4F5L6HH6S4ZEYWS7XRG5KJSYFGOX5JXQBTQEOXFDFSNUBHZJ6A, seg_pos={0 0}, upload=192678]: Done uploading piece. err: <nil> Nov 8 14:26:17.425396997 (218h45m49.12124575s) [noise=true , address=145.79.227.22:4509 , piece_id=6RJJCCT2Z5U3JOWWOBLRIDK5CTWKXU7NR4OIREAQ2JDE2KYDJHOQ, seg_pos={0 0}, upload=192678]: Done uploading piece. err: <nil> Nov 8 14:26:21.856283285 (218h45m53.552132048s) [noise=true , address=145.79.227.31:4798 , piece_id=5Q7NYNXM2IMV7YUCB464FQMBI4L476Y6VKPPOPW7RKNCC5XFDJ2A, seg_pos={0 0}, upload=192678]: Done uploading piece. err: <nil> Nov 8 14:26:23.486413928 (218h45m55.182262681s) [noise=true , address=145.79.227.20:4437 , piece_id=G5RTYGERAVOTQDZGCG6NEKRCQAUOVJNYXO4LEBJ7XHJZ6FDXTULQ, seg_pos={0 0}, upload=192678]: Done uploading piece. err: <nil> Nov 8 14:26:24.659870720 (218h45m56.355719473s) [noise=true , address=145.79.227.26:4653 , piece_id=UIV3WFYLH4JZSIIHZUBL56THDPAYK7DMRUQG5TVTHHEF3XGN6NKQ, seg_pos={0 0}, upload=192678]: Done uploading piece. err: <nil> Nov 8 14:26:34.701060537 (218h46m6.3969093s) [noise=true , address=145.79.227.8:4329 , piece_id=JJFSJ7USPZFTPXVUZVLMXRAHYLARBQZLWWU7ITXFNESYVVL4QBMQ, seg_pos={0 0}, upload=192678]: Done uploading piece. err: <nil> Nov 8 14:26:37.183628509 (218h46m8.879477262s) [noise=true , address=145.79.227.10:4402 , piece_id=IRNQEX73D4AEXNYXEWWKCKEMYFC3YXIPX6HSDYYBNDKBM77ROBEA, seg_pos={0 0}, upload=192678]: Done uploading piece. err: <nil> Nov 8 14:26:37.186238899 (218h46m8.882087662s) [noise=true , address=145.79.227.16:4911 , piece_id=QKITSRMSCHOZLCQ7X6GARTE3GS24G3CZFXJIBKDM3VWOVQLMLEHA, seg_pos={0 0}, upload=192678]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:145.79.227.16:4911): protocol: write tcp 188.166.225.170:56440->145.79.227.16:4911: use of closed network connection; write tcp 188.166.225.170:56440->145.79.227.16:4911: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56440->145.79.227.16:4911: use of closed network connection Nov 8 14:26:37.186462625 (218h46m8.882311378s) [noise=true , address=145.79.227.21:4475 , piece_id=6F7IZNQLORMTKUF576FJA6L7FBG66INEUFPQYHL77WLC6DYF32GQ, seg_pos={0 0}, upload=192678]: Done uploading piece. err: ecclient: upload failed (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk, address:145.79.227.21:4475): protocol: expected piece hash; context deadline exceeded; EOF Nov 8 14:26:37.186739442 (218h46m8.882588195s) [noise=true , address=145.79.227.27:4689 , piece_id=JEEKOLHURPGNFSR4YCWL3UD6BU2MT7VW4S7BU3YKE7BJSUEYLPGQ, seg_pos={0 0}, upload=192678]: Done uploading piece. err: ecclient: upload failed (node:1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, address:145.79.227.27:4689): protocol: write tcp 188.166.225.170:37312->145.79.227.27:4689: use of closed network connection; write tcp 188.166.225.170:37312->145.79.227.27:4689: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37312->145.79.227.27:4689: use of closed network connection Nov 8 14:26:37.186915056 (218h46m8.88276381s) [noise=true , address=145.79.227.7:4295 , piece_id=5OY4NKICVFGAWR7JPOOYIYRXR442IIUDDWTN3MT6KAR4ZHXTKPNQ, seg_pos={0 0}, upload=192678]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:145.79.227.7:4295): protocol: expected piece hash; context deadline exceeded; EOF Nov 8 14:26:37.187072807 (218h46m8.88292156s) [noise=true , address=145.79.227.9:4366 , piece_id=3BINODFKOOG7BYULCICW6S64YZOU5D2Q5MFZYGPSSA34XWM762BQ, seg_pos={0 0}, upload=192678]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: expected piece hash; context deadline exceeded; EOF Nov 8 14:26:37.187107163 (218h46m8.882955916s) [seg_pos={0 0}, upload=192678]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 8 14:26:37.433482306 (218h46m9.129331059s) [upload=192678]: Done uploading segments... Nov 8 14:26:37.433526460 (218h46m9.129375213s) [upload=192678]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 8.186438463s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 14.312816976s trace: https://jaeger.dev.storj.io/trace/3f377bb0d96d25c FAIL (2025-11-08 14:55:37 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Nov 8 14:26:07.882330547 (218h45m39.5781793s) [upload=192678]: Starting upload Nov 8 14:26:07.882433824 (218h45m39.578282567s) [upload=192678]: Done starting upload Nov 8 14:26:07.882492927 (218h45m39.57834167s) [upload=192678]: Uploading segments... Nov 8 14:26:07.882645367 (218h45m39.57849411s) [upload=192678]: Got next segment. Inline: false Nov 8 14:26:07.882698468 (218h45m39.578547221s) [upload=192678]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 8 14:26:07.882704229 (218h45m39.578552982s) [upload=192678]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 8 14:26:08.131091114 (218h45m39.826939897s) [seg_pos={0 0}, upload=192678]: Begin upload segment... Nov 8 14:26:08.131165496 (218h45m39.827014239s) [seg_pos={0 0}, upload=192678]: Done begin upload segment. Nov 8 14:26:08.131220160 (218h45m39.827068913s) [noise=true , address=145.79.227.16:4911 , piece_id=QKITSRMSCHOZLCQ7X6GARTE3GS24G3CZFXJIBKDM3VWOVQLMLEHA, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.131304631 (218h45m39.827153384s) [noise=true , address=145.79.227.27:4689 , piece_id=JEEKOLHURPGNFSR4YCWL3UD6BU2MT7VW4S7BU3YKE7BJSUEYLPGQ, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.132190408 (218h45m39.828039161s) [noise=true , address=145.79.227.7:4295 , piece_id=5OY4NKICVFGAWR7JPOOYIYRXR442IIUDDWTN3MT6KAR4ZHXTKPNQ, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.132569360 (218h45m39.828418113s) [noise=true , address=145.79.227.31:4798 , piece_id=5Q7NYNXM2IMV7YUCB464FQMBI4L476Y6VKPPOPW7RKNCC5XFDJ2A, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.133276446 (218h45m39.829125199s) [noise=true , address=145.79.227.26:4653 , piece_id=UIV3WFYLH4JZSIIHZUBL56THDPAYK7DMRUQG5TVTHHEF3XGN6NKQ, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.134167893 (218h45m39.830016646s) [noise=true , address=145.79.227.26:4655 , piece_id=GP4F5L6HH6S4ZEYWS7XRG5KJSYFGOX5JXQBTQEOXFDFSNUBHZJ6A, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.134314192 (218h45m39.830162945s) [noise=true , address=145.79.227.20:4437 , piece_id=G5RTYGERAVOTQDZGCG6NEKRCQAUOVJNYXO4LEBJ7XHJZ6FDXTULQ, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.135016950 (218h45m39.830865703s) [noise=true , address=145.79.227.10:4402 , piece_id=IRNQEX73D4AEXNYXEWWKCKEMYFC3YXIPX6HSDYYBNDKBM77ROBEA, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.135235305 (218h45m39.831084059s) [noise=true , address=145.79.227.22:4509 , piece_id=6RJJCCT2Z5U3JOWWOBLRIDK5CTWKXU7NR4OIREAQ2JDE2KYDJHOQ, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.135440236 (218h45m39.831288979s) [noise=true , address=145.79.227.2:4113 , piece_id=Q5E3IXSHGFKIFG7EFAPW2XWYP63TO3BC5C7PZJDPXXD2UKDQDJLA, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.136611887 (218h45m39.83246064s) [noise=true , address=77.90.31.15:4699 , piece_id=Z4PXJFTM7XMK62ORTMHCWEJ7OQF6PIYQCMAWJC3DT5RZ7CEH4KXA, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.137027559 (218h45m39.832876312s) [noise=true , address=145.79.227.8:4329 , piece_id=JJFSJ7USPZFTPXVUZVLMXRAHYLARBQZLWWU7ITXFNESYVVL4QBMQ, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.137896133 (218h45m39.833744886s) [noise=true , address=77.90.31.17:4771 , piece_id=LUZ4KXA4YUVTQYKXVEPTEXZ4MFQ5IWFTY3C2LUUVQ3N6DOVO6ZXA, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.139107940 (218h45m39.834956693s) [noise=true , address=145.79.227.21:4475 , piece_id=6F7IZNQLORMTKUF576FJA6L7FBG66INEUFPQYHL77WLC6DYF32GQ, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.139872657 (218h45m39.83572141s) [noise=true , address=145.79.227.9:4366 , piece_id=3BINODFKOOG7BYULCICW6S64YZOU5D2Q5MFZYGPSSA34XWM762BQ, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.140005960 (218h45m39.835854713s) [noise=true , address=77.90.31.16:4734 , piece_id=EKYBQDVNAYTGEPWU443RFLUX7OKCTWBPXA5UMCLZ22M2PBXO3E3Q, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:10.015794232 (218h45m41.711642985s) [upload=192678]: Next returned nil segment Nov 8 14:26:10.015816414 (218h45m41.711665167s) [upload=192678]: Waiting for error group managing segments... Nov 8 14:26:10.314091526 (218h45m42.009940279s) [noise=true , address=77.90.31.16:4734 , piece_id=EKYBQDVNAYTGEPWU443RFLUX7OKCTWBPXA5UMCLZ22M2PBXO3E3Q, seg_pos={0 0}, upload=192678]: Done uploading piece. err: <nil> Nov 8 14:26:10.388439672 (218h45m42.084288465s) [noise=true , address=77.90.31.15:4699 , piece_id=Z4PXJFTM7XMK62ORTMHCWEJ7OQF6PIYQCMAWJC3DT5RZ7CEH4KXA, seg_pos={0 0}, upload=192678]: Done uploading piece. err: <nil> Nov 8 14:26:11.119863095 (218h45m42.815711838s) [noise=true , address=77.90.31.17:4771 , piece_id=LUZ4KXA4YUVTQYKXVEPTEXZ4MFQ5IWFTY3C2LUUVQ3N6DOVO6ZXA, seg_pos={0 0}, upload=192678]: Done uploading piece. err: <nil> Nov 8 14:26:14.806769255 (218h45m46.502618008s) [noise=true , address=145.79.227.2:4113 , piece_id=Q5E3IXSHGFKIFG7EFAPW2XWYP63TO3BC5C7PZJDPXXD2UKDQDJLA, seg_pos={0 0}, upload=192678]: Done uploading piece. err: <nil> Nov 8 14:26:15.197752745 (218h45m46.893601498s) [noise=true , address=145.79.227.26:4655 , piece_id=GP4F5L6HH6S4ZEYWS7XRG5KJSYFGOX5JXQBTQEOXFDFSNUBHZJ6A, seg_pos={0 0}, upload=192678]: Done uploading piece. err: <nil> Nov 8 14:26:17.425396997 (218h45m49.12124575s) [noise=true , address=145.79.227.22:4509 , piece_id=6RJJCCT2Z5U3JOWWOBLRIDK5CTWKXU7NR4OIREAQ2JDE2KYDJHOQ, seg_pos={0 0}, upload=192678]: Done uploading piece. err: <nil> Nov 8 14:26:21.856283285 (218h45m53.552132048s) [noise=true , address=145.79.227.31:4798 , piece_id=5Q7NYNXM2IMV7YUCB464FQMBI4L476Y6VKPPOPW7RKNCC5XFDJ2A, seg_pos={0 0}, upload=192678]: Done uploading piece. err: <nil> Nov 8 14:26:23.486413928 (218h45m55.182262681s) [noise=true , address=145.79.227.20:4437 , piece_id=G5RTYGERAVOTQDZGCG6NEKRCQAUOVJNYXO4LEBJ7XHJZ6FDXTULQ, seg_pos={0 0}, upload=192678]: Done uploading piece. err: <nil> Nov 8 14:26:24.659870720 (218h45m56.355719473s) [noise=true , address=145.79.227.26:4653 , piece_id=UIV3WFYLH4JZSIIHZUBL56THDPAYK7DMRUQG5TVTHHEF3XGN6NKQ, seg_pos={0 0}, upload=192678]: Done uploading piece. err: <nil> Nov 8 14:26:34.701060537 (218h46m6.3969093s) [noise=true , address=145.79.227.8:4329 , piece_id=JJFSJ7USPZFTPXVUZVLMXRAHYLARBQZLWWU7ITXFNESYVVL4QBMQ, seg_pos={0 0}, upload=192678]: Done uploading piece. err: <nil> Nov 8 14:26:37.183628509 (218h46m8.879477262s) [noise=true , address=145.79.227.10:4402 , piece_id=IRNQEX73D4AEXNYXEWWKCKEMYFC3YXIPX6HSDYYBNDKBM77ROBEA, seg_pos={0 0}, upload=192678]: Done uploading piece. err: <nil> Nov 8 14:26:37.186238899 (218h46m8.882087662s) [noise=true , address=145.79.227.16:4911 , piece_id=QKITSRMSCHOZLCQ7X6GARTE3GS24G3CZFXJIBKDM3VWOVQLMLEHA, seg_pos={0 0}, upload=192678]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:145.79.227.16:4911): protocol: write tcp 188.166.225.170:56440->145.79.227.16:4911: use of closed network connection; write tcp 188.166.225.170:56440->145.79.227.16:4911: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56440->145.79.227.16:4911: use of closed network connection Nov 8 14:26:37.186462625 (218h46m8.882311378s) [noise=true , address=145.79.227.21:4475 , piece_id=6F7IZNQLORMTKUF576FJA6L7FBG66INEUFPQYHL77WLC6DYF32GQ, seg_pos={0 0}, upload=192678]: Done uploading piece. err: ecclient: upload failed (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk, address:145.79.227.21:4475): protocol: expected piece hash; context deadline exceeded; EOF Nov 8 14:26:37.186739442 (218h46m8.882588195s) [noise=true , address=145.79.227.27:4689 , piece_id=JEEKOLHURPGNFSR4YCWL3UD6BU2MT7VW4S7BU3YKE7BJSUEYLPGQ, seg_pos={0 0}, upload=192678]: Done uploading piece. err: ecclient: upload failed (node:1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, address:145.79.227.27:4689): protocol: write tcp 188.166.225.170:37312->145.79.227.27:4689: use of closed network connection; write tcp 188.166.225.170:37312->145.79.227.27:4689: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37312->145.79.227.27:4689: use of closed network connection Nov 8 14:26:37.186915056 (218h46m8.88276381s) [noise=true , address=145.79.227.7:4295 , piece_id=5OY4NKICVFGAWR7JPOOYIYRXR442IIUDDWTN3MT6KAR4ZHXTKPNQ, seg_pos={0 0}, upload=192678]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:145.79.227.7:4295): protocol: expected piece hash; context deadline exceeded; EOF Nov 8 14:26:37.187072807 (218h46m8.88292156s) [noise=true , address=145.79.227.9:4366 , piece_id=3BINODFKOOG7BYULCICW6S64YZOU5D2Q5MFZYGPSSA34XWM762BQ, seg_pos={0 0}, upload=192678]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: expected piece hash; context deadline exceeded; EOF Nov 8 14:26:37.187107163 (218h46m8.882955916s) [seg_pos={0 0}, upload=192678]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 8 14:26:37.433482306 (218h46m9.129331059s) [upload=192678]: Done uploading segments... Nov 8 14:26:37.433526460 (218h46m9.129375213s) [upload=192678]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 14.312849407s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 11.196086268s trace: https://jaeger.dev.storj.io/trace/3f377bb0d96d25c FAIL (2025-11-08 14:54:20 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Nov 8 14:26:07.882330547 (218h45m39.5781793s) [upload=192678]: Starting upload Nov 8 14:26:07.882433824 (218h45m39.578282567s) [upload=192678]: Done starting upload Nov 8 14:26:07.882492927 (218h45m39.57834167s) [upload=192678]: Uploading segments... Nov 8 14:26:07.882645367 (218h45m39.57849411s) [upload=192678]: Got next segment. Inline: false Nov 8 14:26:07.882698468 (218h45m39.578547221s) [upload=192678]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 8 14:26:07.882704229 (218h45m39.578552982s) [upload=192678]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 8 14:26:08.131091114 (218h45m39.826939897s) [seg_pos={0 0}, upload=192678]: Begin upload segment... Nov 8 14:26:08.131165496 (218h45m39.827014239s) [seg_pos={0 0}, upload=192678]: Done begin upload segment. Nov 8 14:26:08.131220160 (218h45m39.827068913s) [noise=true , address=145.79.227.16:4911 , piece_id=QKITSRMSCHOZLCQ7X6GARTE3GS24G3CZFXJIBKDM3VWOVQLMLEHA, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.131304631 (218h45m39.827153384s) [noise=true , address=145.79.227.27:4689 , piece_id=JEEKOLHURPGNFSR4YCWL3UD6BU2MT7VW4S7BU3YKE7BJSUEYLPGQ, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.132190408 (218h45m39.828039161s) [noise=true , address=145.79.227.7:4295 , piece_id=5OY4NKICVFGAWR7JPOOYIYRXR442IIUDDWTN3MT6KAR4ZHXTKPNQ, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.132569360 (218h45m39.828418113s) [noise=true , address=145.79.227.31:4798 , piece_id=5Q7NYNXM2IMV7YUCB464FQMBI4L476Y6VKPPOPW7RKNCC5XFDJ2A, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.133276446 (218h45m39.829125199s) [noise=true , address=145.79.227.26:4653 , piece_id=UIV3WFYLH4JZSIIHZUBL56THDPAYK7DMRUQG5TVTHHEF3XGN6NKQ, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.134167893 (218h45m39.830016646s) [noise=true , address=145.79.227.26:4655 , piece_id=GP4F5L6HH6S4ZEYWS7XRG5KJSYFGOX5JXQBTQEOXFDFSNUBHZJ6A, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.134314192 (218h45m39.830162945s) [noise=true , address=145.79.227.20:4437 , piece_id=G5RTYGERAVOTQDZGCG6NEKRCQAUOVJNYXO4LEBJ7XHJZ6FDXTULQ, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.135016950 (218h45m39.830865703s) [noise=true , address=145.79.227.10:4402 , piece_id=IRNQEX73D4AEXNYXEWWKCKEMYFC3YXIPX6HSDYYBNDKBM77ROBEA, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.135235305 (218h45m39.831084059s) [noise=true , address=145.79.227.22:4509 , piece_id=6RJJCCT2Z5U3JOWWOBLRIDK5CTWKXU7NR4OIREAQ2JDE2KYDJHOQ, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.135440236 (218h45m39.831288979s) [noise=true , address=145.79.227.2:4113 , piece_id=Q5E3IXSHGFKIFG7EFAPW2XWYP63TO3BC5C7PZJDPXXD2UKDQDJLA, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.136611887 (218h45m39.83246064s) [noise=true , address=77.90.31.15:4699 , piece_id=Z4PXJFTM7XMK62ORTMHCWEJ7OQF6PIYQCMAWJC3DT5RZ7CEH4KXA, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.137027559 (218h45m39.832876312s) [noise=true , address=145.79.227.8:4329 , piece_id=JJFSJ7USPZFTPXVUZVLMXRAHYLARBQZLWWU7ITXFNESYVVL4QBMQ, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.137896133 (218h45m39.833744886s) [noise=true , address=77.90.31.17:4771 , piece_id=LUZ4KXA4YUVTQYKXVEPTEXZ4MFQ5IWFTY3C2LUUVQ3N6DOVO6ZXA, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.139107940 (218h45m39.834956693s) [noise=true , address=145.79.227.21:4475 , piece_id=6F7IZNQLORMTKUF576FJA6L7FBG66INEUFPQYHL77WLC6DYF32GQ, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.139872657 (218h45m39.83572141s) [noise=true , address=145.79.227.9:4366 , piece_id=3BINODFKOOG7BYULCICW6S64YZOU5D2Q5MFZYGPSSA34XWM762BQ, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.140005960 (218h45m39.835854713s) [noise=true , address=77.90.31.16:4734 , piece_id=EKYBQDVNAYTGEPWU443RFLUX7OKCTWBPXA5UMCLZ22M2PBXO3E3Q, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:10.015794232 (218h45m41.711642985s) [upload=192678]: Next returned nil segment Nov 8 14:26:10.015816414 (218h45m41.711665167s) [upload=192678]: Waiting for error group managing segments... Nov 8 14:26:10.314091526 (218h45m42.009940279s) [noise=true , address=77.90.31.16:4734 , piece_id=EKYBQDVNAYTGEPWU443RFLUX7OKCTWBPXA5UMCLZ22M2PBXO3E3Q, seg_pos={0 0}, upload=192678]: Done uploading piece. err: <nil> Nov 8 14:26:10.388439672 (218h45m42.084288465s) [noise=true , address=77.90.31.15:4699 , piece_id=Z4PXJFTM7XMK62ORTMHCWEJ7OQF6PIYQCMAWJC3DT5RZ7CEH4KXA, seg_pos={0 0}, upload=192678]: Done uploading piece. err: <nil> Nov 8 14:26:11.119863095 (218h45m42.815711838s) [noise=true , address=77.90.31.17:4771 , piece_id=LUZ4KXA4YUVTQYKXVEPTEXZ4MFQ5IWFTY3C2LUUVQ3N6DOVO6ZXA, seg_pos={0 0}, upload=192678]: Done uploading piece. err: <nil> Nov 8 14:26:14.806769255 (218h45m46.502618008s) [noise=true , address=145.79.227.2:4113 , piece_id=Q5E3IXSHGFKIFG7EFAPW2XWYP63TO3BC5C7PZJDPXXD2UKDQDJLA, seg_pos={0 0}, upload=192678]: Done uploading piece. err: <nil> Nov 8 14:26:15.197752745 (218h45m46.893601498s) [noise=true , address=145.79.227.26:4655 , piece_id=GP4F5L6HH6S4ZEYWS7XRG5KJSYFGOX5JXQBTQEOXFDFSNUBHZJ6A, seg_pos={0 0}, upload=192678]: Done uploading piece. err: <nil> Nov 8 14:26:17.425396997 (218h45m49.12124575s) [noise=true , address=145.79.227.22:4509 , piece_id=6RJJCCT2Z5U3JOWWOBLRIDK5CTWKXU7NR4OIREAQ2JDE2KYDJHOQ, seg_pos={0 0}, upload=192678]: Done uploading piece. err: <nil> Nov 8 14:26:21.856283285 (218h45m53.552132048s) [noise=true , address=145.79.227.31:4798 , piece_id=5Q7NYNXM2IMV7YUCB464FQMBI4L476Y6VKPPOPW7RKNCC5XFDJ2A, seg_pos={0 0}, upload=192678]: Done uploading piece. err: <nil> Nov 8 14:26:23.486413928 (218h45m55.182262681s) [noise=true , address=145.79.227.20:4437 , piece_id=G5RTYGERAVOTQDZGCG6NEKRCQAUOVJNYXO4LEBJ7XHJZ6FDXTULQ, seg_pos={0 0}, upload=192678]: Done uploading piece. err: <nil> Nov 8 14:26:24.659870720 (218h45m56.355719473s) [noise=true , address=145.79.227.26:4653 , piece_id=UIV3WFYLH4JZSIIHZUBL56THDPAYK7DMRUQG5TVTHHEF3XGN6NKQ, seg_pos={0 0}, upload=192678]: Done uploading piece. err: <nil> Nov 8 14:26:34.701060537 (218h46m6.3969093s) [noise=true , address=145.79.227.8:4329 , piece_id=JJFSJ7USPZFTPXVUZVLMXRAHYLARBQZLWWU7ITXFNESYVVL4QBMQ, seg_pos={0 0}, upload=192678]: Done uploading piece. err: <nil> Nov 8 14:26:37.183628509 (218h46m8.879477262s) [noise=true , address=145.79.227.10:4402 , piece_id=IRNQEX73D4AEXNYXEWWKCKEMYFC3YXIPX6HSDYYBNDKBM77ROBEA, seg_pos={0 0}, upload=192678]: Done uploading piece. err: <nil> Nov 8 14:26:37.186238899 (218h46m8.882087662s) [noise=true , address=145.79.227.16:4911 , piece_id=QKITSRMSCHOZLCQ7X6GARTE3GS24G3CZFXJIBKDM3VWOVQLMLEHA, seg_pos={0 0}, upload=192678]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:145.79.227.16:4911): protocol: write tcp 188.166.225.170:56440->145.79.227.16:4911: use of closed network connection; write tcp 188.166.225.170:56440->145.79.227.16:4911: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56440->145.79.227.16:4911: use of closed network connection Nov 8 14:26:37.186462625 (218h46m8.882311378s) [noise=true , address=145.79.227.21:4475 , piece_id=6F7IZNQLORMTKUF576FJA6L7FBG66INEUFPQYHL77WLC6DYF32GQ, seg_pos={0 0}, upload=192678]: Done uploading piece. err: ecclient: upload failed (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk, address:145.79.227.21:4475): protocol: expected piece hash; context deadline exceeded; EOF Nov 8 14:26:37.186739442 (218h46m8.882588195s) [noise=true , address=145.79.227.27:4689 , piece_id=JEEKOLHURPGNFSR4YCWL3UD6BU2MT7VW4S7BU3YKE7BJSUEYLPGQ, seg_pos={0 0}, upload=192678]: Done uploading piece. err: ecclient: upload failed (node:1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, address:145.79.227.27:4689): protocol: write tcp 188.166.225.170:37312->145.79.227.27:4689: use of closed network connection; write tcp 188.166.225.170:37312->145.79.227.27:4689: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37312->145.79.227.27:4689: use of closed network connection Nov 8 14:26:37.186915056 (218h46m8.88276381s) [noise=true , address=145.79.227.7:4295 , piece_id=5OY4NKICVFGAWR7JPOOYIYRXR442IIUDDWTN3MT6KAR4ZHXTKPNQ, seg_pos={0 0}, upload=192678]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:145.79.227.7:4295): protocol: expected piece hash; context deadline exceeded; EOF Nov 8 14:26:37.187072807 (218h46m8.88292156s) [noise=true , address=145.79.227.9:4366 , piece_id=3BINODFKOOG7BYULCICW6S64YZOU5D2Q5MFZYGPSSA34XWM762BQ, seg_pos={0 0}, upload=192678]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: expected piece hash; context deadline exceeded; EOF Nov 8 14:26:37.187107163 (218h46m8.882955916s) [seg_pos={0 0}, upload=192678]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 8 14:26:37.433482306 (218h46m9.129331059s) [upload=192678]: Done uploading segments... Nov 8 14:26:37.433526460 (218h46m9.129375213s) [upload=192678]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 11.196120572s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 7.276676887s trace: https://jaeger.dev.storj.io/trace/3f377bb0d96d25c FAIL (2025-11-08 14:53:50 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Nov 8 14:26:07.882330547 (218h45m39.5781793s) [upload=192678]: Starting upload Nov 8 14:26:07.882433824 (218h45m39.578282567s) [upload=192678]: Done starting upload Nov 8 14:26:07.882492927 (218h45m39.57834167s) [upload=192678]: Uploading segments... Nov 8 14:26:07.882645367 (218h45m39.57849411s) [upload=192678]: Got next segment. Inline: false Nov 8 14:26:07.882698468 (218h45m39.578547221s) [upload=192678]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 8 14:26:07.882704229 (218h45m39.578552982s) [upload=192678]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 8 14:26:08.131091114 (218h45m39.826939897s) [seg_pos={0 0}, upload=192678]: Begin upload segment... Nov 8 14:26:08.131165496 (218h45m39.827014239s) [seg_pos={0 0}, upload=192678]: Done begin upload segment. Nov 8 14:26:08.131220160 (218h45m39.827068913s) [noise=true , address=145.79.227.16:4911 , piece_id=QKITSRMSCHOZLCQ7X6GARTE3GS24G3CZFXJIBKDM3VWOVQLMLEHA, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.131304631 (218h45m39.827153384s) [noise=true , address=145.79.227.27:4689 , piece_id=JEEKOLHURPGNFSR4YCWL3UD6BU2MT7VW4S7BU3YKE7BJSUEYLPGQ, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.132190408 (218h45m39.828039161s) [noise=true , address=145.79.227.7:4295 , piece_id=5OY4NKICVFGAWR7JPOOYIYRXR442IIUDDWTN3MT6KAR4ZHXTKPNQ, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.132569360 (218h45m39.828418113s) [noise=true , address=145.79.227.31:4798 , piece_id=5Q7NYNXM2IMV7YUCB464FQMBI4L476Y6VKPPOPW7RKNCC5XFDJ2A, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.133276446 (218h45m39.829125199s) [noise=true , address=145.79.227.26:4653 , piece_id=UIV3WFYLH4JZSIIHZUBL56THDPAYK7DMRUQG5TVTHHEF3XGN6NKQ, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.134167893 (218h45m39.830016646s) [noise=true , address=145.79.227.26:4655 , piece_id=GP4F5L6HH6S4ZEYWS7XRG5KJSYFGOX5JXQBTQEOXFDFSNUBHZJ6A, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.134314192 (218h45m39.830162945s) [noise=true , address=145.79.227.20:4437 , piece_id=G5RTYGERAVOTQDZGCG6NEKRCQAUOVJNYXO4LEBJ7XHJZ6FDXTULQ, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.135016950 (218h45m39.830865703s) [noise=true , address=145.79.227.10:4402 , piece_id=IRNQEX73D4AEXNYXEWWKCKEMYFC3YXIPX6HSDYYBNDKBM77ROBEA, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.135235305 (218h45m39.831084059s) [noise=true , address=145.79.227.22:4509 , piece_id=6RJJCCT2Z5U3JOWWOBLRIDK5CTWKXU7NR4OIREAQ2JDE2KYDJHOQ, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.135440236 (218h45m39.831288979s) [noise=true , address=145.79.227.2:4113 , piece_id=Q5E3IXSHGFKIFG7EFAPW2XWYP63TO3BC5C7PZJDPXXD2UKDQDJLA, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.136611887 (218h45m39.83246064s) [noise=true , address=77.90.31.15:4699 , piece_id=Z4PXJFTM7XMK62ORTMHCWEJ7OQF6PIYQCMAWJC3DT5RZ7CEH4KXA, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.137027559 (218h45m39.832876312s) [noise=true , address=145.79.227.8:4329 , piece_id=JJFSJ7USPZFTPXVUZVLMXRAHYLARBQZLWWU7ITXFNESYVVL4QBMQ, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.137896133 (218h45m39.833744886s) [noise=true , address=77.90.31.17:4771 , piece_id=LUZ4KXA4YUVTQYKXVEPTEXZ4MFQ5IWFTY3C2LUUVQ3N6DOVO6ZXA, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.139107940 (218h45m39.834956693s) [noise=true , address=145.79.227.21:4475 , piece_id=6F7IZNQLORMTKUF576FJA6L7FBG66INEUFPQYHL77WLC6DYF32GQ, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.139872657 (218h45m39.83572141s) [noise=true , address=145.79.227.9:4366 , piece_id=3BINODFKOOG7BYULCICW6S64YZOU5D2Q5MFZYGPSSA34XWM762BQ, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.140005960 (218h45m39.835854713s) [noise=true , address=77.90.31.16:4734 , piece_id=EKYBQDVNAYTGEPWU443RFLUX7OKCTWBPXA5UMCLZ22M2PBXO3E3Q, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:10.015794232 (218h45m41.711642985s) [upload=192678]: Next returned nil segment Nov 8 14:26:10.015816414 (218h45m41.711665167s) [upload=192678]: Waiting for error group managing segments... Nov 8 14:26:10.314091526 (218h45m42.009940279s) [noise=true , address=77.90.31.16:4734 , piece_id=EKYBQDVNAYTGEPWU443RFLUX7OKCTWBPXA5UMCLZ22M2PBXO3E3Q, seg_pos={0 0}, upload=192678]: Done uploading piece. err: <nil> Nov 8 14:26:10.388439672 (218h45m42.084288465s) [noise=true , address=77.90.31.15:4699 , piece_id=Z4PXJFTM7XMK62ORTMHCWEJ7OQF6PIYQCMAWJC3DT5RZ7CEH4KXA, seg_pos={0 0}, upload=192678]: Done uploading piece. err: <nil> Nov 8 14:26:11.119863095 (218h45m42.815711838s) [noise=true , address=77.90.31.17:4771 , piece_id=LUZ4KXA4YUVTQYKXVEPTEXZ4MFQ5IWFTY3C2LUUVQ3N6DOVO6ZXA, seg_pos={0 0}, upload=192678]: Done uploading piece. err: <nil> Nov 8 14:26:14.806769255 (218h45m46.502618008s) [noise=true , address=145.79.227.2:4113 , piece_id=Q5E3IXSHGFKIFG7EFAPW2XWYP63TO3BC5C7PZJDPXXD2UKDQDJLA, seg_pos={0 0}, upload=192678]: Done uploading piece. err: <nil> Nov 8 14:26:15.197752745 (218h45m46.893601498s) [noise=true , address=145.79.227.26:4655 , piece_id=GP4F5L6HH6S4ZEYWS7XRG5KJSYFGOX5JXQBTQEOXFDFSNUBHZJ6A, seg_pos={0 0}, upload=192678]: Done uploading piece. err: <nil> Nov 8 14:26:17.425396997 (218h45m49.12124575s) [noise=true , address=145.79.227.22:4509 , piece_id=6RJJCCT2Z5U3JOWWOBLRIDK5CTWKXU7NR4OIREAQ2JDE2KYDJHOQ, seg_pos={0 0}, upload=192678]: Done uploading piece. err: <nil> Nov 8 14:26:21.856283285 (218h45m53.552132048s) [noise=true , address=145.79.227.31:4798 , piece_id=5Q7NYNXM2IMV7YUCB464FQMBI4L476Y6VKPPOPW7RKNCC5XFDJ2A, seg_pos={0 0}, upload=192678]: Done uploading piece. err: <nil> Nov 8 14:26:23.486413928 (218h45m55.182262681s) [noise=true , address=145.79.227.20:4437 , piece_id=G5RTYGERAVOTQDZGCG6NEKRCQAUOVJNYXO4LEBJ7XHJZ6FDXTULQ, seg_pos={0 0}, upload=192678]: Done uploading piece. err: <nil> Nov 8 14:26:24.659870720 (218h45m56.355719473s) [noise=true , address=145.79.227.26:4653 , piece_id=UIV3WFYLH4JZSIIHZUBL56THDPAYK7DMRUQG5TVTHHEF3XGN6NKQ, seg_pos={0 0}, upload=192678]: Done uploading piece. err: <nil> Nov 8 14:26:34.701060537 (218h46m6.3969093s) [noise=true , address=145.79.227.8:4329 , piece_id=JJFSJ7USPZFTPXVUZVLMXRAHYLARBQZLWWU7ITXFNESYVVL4QBMQ, seg_pos={0 0}, upload=192678]: Done uploading piece. err: <nil> Nov 8 14:26:37.183628509 (218h46m8.879477262s) [noise=true , address=145.79.227.10:4402 , piece_id=IRNQEX73D4AEXNYXEWWKCKEMYFC3YXIPX6HSDYYBNDKBM77ROBEA, seg_pos={0 0}, upload=192678]: Done uploading piece. err: <nil> Nov 8 14:26:37.186238899 (218h46m8.882087662s) [noise=true , address=145.79.227.16:4911 , piece_id=QKITSRMSCHOZLCQ7X6GARTE3GS24G3CZFXJIBKDM3VWOVQLMLEHA, seg_pos={0 0}, upload=192678]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:145.79.227.16:4911): protocol: write tcp 188.166.225.170:56440->145.79.227.16:4911: use of closed network connection; write tcp 188.166.225.170:56440->145.79.227.16:4911: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56440->145.79.227.16:4911: use of closed network connection Nov 8 14:26:37.186462625 (218h46m8.882311378s) [noise=true , address=145.79.227.21:4475 , piece_id=6F7IZNQLORMTKUF576FJA6L7FBG66INEUFPQYHL77WLC6DYF32GQ, seg_pos={0 0}, upload=192678]: Done uploading piece. err: ecclient: upload failed (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk, address:145.79.227.21:4475): protocol: expected piece hash; context deadline exceeded; EOF Nov 8 14:26:37.186739442 (218h46m8.882588195s) [noise=true , address=145.79.227.27:4689 , piece_id=JEEKOLHURPGNFSR4YCWL3UD6BU2MT7VW4S7BU3YKE7BJSUEYLPGQ, seg_pos={0 0}, upload=192678]: Done uploading piece. err: ecclient: upload failed (node:1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, address:145.79.227.27:4689): protocol: write tcp 188.166.225.170:37312->145.79.227.27:4689: use of closed network connection; write tcp 188.166.225.170:37312->145.79.227.27:4689: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37312->145.79.227.27:4689: use of closed network connection Nov 8 14:26:37.186915056 (218h46m8.88276381s) [noise=true , address=145.79.227.7:4295 , piece_id=5OY4NKICVFGAWR7JPOOYIYRXR442IIUDDWTN3MT6KAR4ZHXTKPNQ, seg_pos={0 0}, upload=192678]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:145.79.227.7:4295): protocol: expected piece hash; context deadline exceeded; EOF Nov 8 14:26:37.187072807 (218h46m8.88292156s) [noise=true , address=145.79.227.9:4366 , piece_id=3BINODFKOOG7BYULCICW6S64YZOU5D2Q5MFZYGPSSA34XWM762BQ, seg_pos={0 0}, upload=192678]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: expected piece hash; context deadline exceeded; EOF Nov 8 14:26:37.187107163 (218h46m8.882955916s) [seg_pos={0 0}, upload=192678]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 8 14:26:37.433482306 (218h46m9.129331059s) [upload=192678]: Done uploading segments... Nov 8 14:26:37.433526460 (218h46m9.129375213s) [upload=192678]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 7.276710371s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 7.746852676s trace: https://jaeger.dev.storj.io/trace/3f377bb0d96d25c FAIL (2025-11-08 14:53:05 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Nov 8 14:26:07.882330547 (218h45m39.5781793s) [upload=192678]: Starting upload Nov 8 14:26:07.882433824 (218h45m39.578282567s) [upload=192678]: Done starting upload Nov 8 14:26:07.882492927 (218h45m39.57834167s) [upload=192678]: Uploading segments... Nov 8 14:26:07.882645367 (218h45m39.57849411s) [upload=192678]: Got next segment. Inline: false Nov 8 14:26:07.882698468 (218h45m39.578547221s) [upload=192678]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 8 14:26:07.882704229 (218h45m39.578552982s) [upload=192678]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 8 14:26:08.131091114 (218h45m39.826939897s) [seg_pos={0 0}, upload=192678]: Begin upload segment... Nov 8 14:26:08.131165496 (218h45m39.827014239s) [seg_pos={0 0}, upload=192678]: Done begin upload segment. Nov 8 14:26:08.131220160 (218h45m39.827068913s) [noise=true , address=145.79.227.16:4911 , piece_id=QKITSRMSCHOZLCQ7X6GARTE3GS24G3CZFXJIBKDM3VWOVQLMLEHA, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.131304631 (218h45m39.827153384s) [noise=true , address=145.79.227.27:4689 , piece_id=JEEKOLHURPGNFSR4YCWL3UD6BU2MT7VW4S7BU3YKE7BJSUEYLPGQ, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.132190408 (218h45m39.828039161s) [noise=true , address=145.79.227.7:4295 , piece_id=5OY4NKICVFGAWR7JPOOYIYRXR442IIUDDWTN3MT6KAR4ZHXTKPNQ, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.132569360 (218h45m39.828418113s) [noise=true , address=145.79.227.31:4798 , piece_id=5Q7NYNXM2IMV7YUCB464FQMBI4L476Y6VKPPOPW7RKNCC5XFDJ2A, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.133276446 (218h45m39.829125199s) [noise=true , address=145.79.227.26:4653 , piece_id=UIV3WFYLH4JZSIIHZUBL56THDPAYK7DMRUQG5TVTHHEF3XGN6NKQ, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.134167893 (218h45m39.830016646s) [noise=true , address=145.79.227.26:4655 , piece_id=GP4F5L6HH6S4ZEYWS7XRG5KJSYFGOX5JXQBTQEOXFDFSNUBHZJ6A, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.134314192 (218h45m39.830162945s) [noise=true , address=145.79.227.20:4437 , piece_id=G5RTYGERAVOTQDZGCG6NEKRCQAUOVJNYXO4LEBJ7XHJZ6FDXTULQ, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.135016950 (218h45m39.830865703s) [noise=true , address=145.79.227.10:4402 , piece_id=IRNQEX73D4AEXNYXEWWKCKEMYFC3YXIPX6HSDYYBNDKBM77ROBEA, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.135235305 (218h45m39.831084059s) [noise=true , address=145.79.227.22:4509 , piece_id=6RJJCCT2Z5U3JOWWOBLRIDK5CTWKXU7NR4OIREAQ2JDE2KYDJHOQ, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.135440236 (218h45m39.831288979s) [noise=true , address=145.79.227.2:4113 , piece_id=Q5E3IXSHGFKIFG7EFAPW2XWYP63TO3BC5C7PZJDPXXD2UKDQDJLA, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.136611887 (218h45m39.83246064s) [noise=true , address=77.90.31.15:4699 , piece_id=Z4PXJFTM7XMK62ORTMHCWEJ7OQF6PIYQCMAWJC3DT5RZ7CEH4KXA, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.137027559 (218h45m39.832876312s) [noise=true , address=145.79.227.8:4329 , piece_id=JJFSJ7USPZFTPXVUZVLMXRAHYLARBQZLWWU7ITXFNESYVVL4QBMQ, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.137896133 (218h45m39.833744886s) [noise=true , address=77.90.31.17:4771 , piece_id=LUZ4KXA4YUVTQYKXVEPTEXZ4MFQ5IWFTY3C2LUUVQ3N6DOVO6ZXA, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.139107940 (218h45m39.834956693s) [noise=true , address=145.79.227.21:4475 , piece_id=6F7IZNQLORMTKUF576FJA6L7FBG66INEUFPQYHL77WLC6DYF32GQ, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.139872657 (218h45m39.83572141s) [noise=true , address=145.79.227.9:4366 , piece_id=3BINODFKOOG7BYULCICW6S64YZOU5D2Q5MFZYGPSSA34XWM762BQ, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:08.140005960 (218h45m39.835854713s) [noise=true , address=77.90.31.16:4734 , piece_id=EKYBQDVNAYTGEPWU443RFLUX7OKCTWBPXA5UMCLZ22M2PBXO3E3Q, seg_pos={0 0}, upload=192678]: Uploading piece... Nov 8 14:26:10.015794232 (218h45m41.711642985s) [upload=192678]: Next returned nil segment Nov 8 14:26:10.015816414 (218h45m41.711665167s) [upload=192678]: Waiting for error group managing segments... Nov 8 14:26:10.314091526 (218h45m42.009940279s) [noise=true , address=77.90.31.16:4734 , piece_id=EKYBQDVNAYTGEPWU443RFLUX7OKCTWBPXA5UMCLZ22M2PBXO3E3Q, seg_pos={0 0}, upload=192678]: Done uploading piece. err: <nil> Nov 8 14:26:10.388439672 (218h45m42.084288465s) [noise=true , address=77.90.31.15:4699 , piece_id=Z4PXJFTM7XMK62ORTMHCWEJ7OQF6PIYQCMAWJC3DT5RZ7CEH4KXA, seg_pos={0 0}, upload=192678]: Done uploading piece. err: <nil> Nov 8 14:26:11.119863095 (218h45m42.815711838s) [noise=true , address=77.90.31.17:4771 , piece_id=LUZ4KXA4YUVTQYKXVEPTEXZ4MFQ5IWFTY3C2LUUVQ3N6DOVO6ZXA, seg_pos={0 0}, upload=192678]: Done uploading piece. err: <nil> Nov 8 14:26:14.806769255 (218h45m46.502618008s) [noise=true , address=145.79.227.2:4113 , piece_id=Q5E3IXSHGFKIFG7EFAPW2XWYP63TO3BC5C7PZJDPXXD2UKDQDJLA, seg_pos={0 0}, upload=192678]: Done uploading piece. err: <nil> Nov 8 14:26:15.197752745 (218h45m46.893601498s) [noise=true , address=145.79.227.26:4655 , piece_id=GP4F5L6HH6S4ZEYWS7XRG5KJSYFGOX5JXQBTQEOXFDFSNUBHZJ6A, seg_pos={0 0}, upload=192678]: Done uploading piece. err: <nil> Nov 8 14:26:17.425396997 (218h45m49.12124575s) [noise=true , address=145.79.227.22:4509 , piece_id=6RJJCCT2Z5U3JOWWOBLRIDK5CTWKXU7NR4OIREAQ2JDE2KYDJHOQ, seg_pos={0 0}, upload=192678]: Done uploading piece. err: <nil> Nov 8 14:26:21.856283285 (218h45m53.552132048s) [noise=true , address=145.79.227.31:4798 , piece_id=5Q7NYNXM2IMV7YUCB464FQMBI4L476Y6VKPPOPW7RKNCC5XFDJ2A, seg_pos={0 0}, upload=192678]: Done uploading piece. err: <nil> Nov 8 14:26:23.486413928 (218h45m55.182262681s) [noise=true , address=145.79.227.20:4437 , piece_id=G5RTYGERAVOTQDZGCG6NEKRCQAUOVJNYXO4LEBJ7XHJZ6FDXTULQ, seg_pos={0 0}, upload=192678]: Done uploading piece. err: <nil> Nov 8 14:26:24.659870720 (218h45m56.355719473s) [noise=true , address=145.79.227.26:4653 , piece_id=UIV3WFYLH4JZSIIHZUBL56THDPAYK7DMRUQG5TVTHHEF3XGN6NKQ, seg_pos={0 0}, upload=192678]: Done uploading piece. err: <nil> Nov 8 14:26:34.701060537 (218h46m6.3969093s) [noise=true , address=145.79.227.8:4329 , piece_id=JJFSJ7USPZFTPXVUZVLMXRAHYLARBQZLWWU7ITXFNESYVVL4QBMQ, seg_pos={0 0}, upload=192678]: Done uploading piece. err: <nil> Nov 8 14:26:37.183628509 (218h46m8.879477262s) [noise=true , address=145.79.227.10:4402 , piece_id=IRNQEX73D4AEXNYXEWWKCKEMYFC3YXIPX6HSDYYBNDKBM77ROBEA, seg_pos={0 0}, upload=192678]: Done uploading piece. err: <nil> Nov 8 14:26:37.186238899 (218h46m8.882087662s) [noise=true , address=145.79.227.16:4911 , piece_id=QKITSRMSCHOZLCQ7X6GARTE3GS24G3CZFXJIBKDM3VWOVQLMLEHA, seg_pos={0 0}, upload=192678]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:145.79.227.16:4911): protocol: write tcp 188.166.225.170:56440->145.79.227.16:4911: use of closed network connection; write tcp 188.166.225.170:56440->145.79.227.16:4911: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56440->145.79.227.16:4911: use of closed network connection Nov 8 14:26:37.186462625 (218h46m8.882311378s) [noise=true , address=145.79.227.21:4475 , piece_id=6F7IZNQLORMTKUF576FJA6L7FBG66INEUFPQYHL77WLC6DYF32GQ, seg_pos={0 0}, upload=192678]: Done uploading piece. err: ecclient: upload failed (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk, address:145.79.227.21:4475): protocol: expected piece hash; context deadline exceeded; EOF Nov 8 14:26:37.186739442 (218h46m8.882588195s) [noise=true , address=145.79.227.27:4689 , piece_id=JEEKOLHURPGNFSR4YCWL3UD6BU2MT7VW4S7BU3YKE7BJSUEYLPGQ, seg_pos={0 0}, upload=192678]: Done uploading piece. err: ecclient: upload failed (node:1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, address:145.79.227.27:4689): protocol: write tcp 188.166.225.170:37312->145.79.227.27:4689: use of closed network connection; write tcp 188.166.225.170:37312->145.79.227.27:4689: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37312->145.79.227.27:4689: use of closed network connection Nov 8 14:26:37.186915056 (218h46m8.88276381s) [noise=true , address=145.79.227.7:4295 , piece_id=5OY4NKICVFGAWR7JPOOYIYRXR442IIUDDWTN3MT6KAR4ZHXTKPNQ, seg_pos={0 0}, upload=192678]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:145.79.227.7:4295): protocol: expected piece hash; context deadline exceeded; EOF Nov 8 14:26:37.187072807 (218h46m8.88292156s) [noise=true , address=145.79.227.9:4366 , piece_id=3BINODFKOOG7BYULCICW6S64YZOU5D2Q5MFZYGPSSA34XWM762BQ, seg_pos={0 0}, upload=192678]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: expected piece hash; context deadline exceeded; EOF Nov 8 14:26:37.187107163 (218h46m8.882955916s) [seg_pos={0 0}, upload=192678]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 8 14:26:37.433482306 (218h46m9.129331059s) [upload=192678]: Done uploading segments... Nov 8 14:26:37.433526460 (218h46m9.129375213s) [upload=192678]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 7.746893685s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)