Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [.....................X................X.............X.......] duration: 3.619774914s trace: https://jaeger.dev.storj.io/trace/740267b1692fc58 FAIL (2025-12-15 11:30:09 UTC) orbiter warning: context deadline exceeded Duration: 3.619813398s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [......................X................X.............X......] duration: 3.426021364s trace: https://jaeger.dev.storj.io/trace/740267b1692fc58 FAIL (2025-12-15 11:29:10 UTC) orbiter warning: context deadline exceeded Duration: 3.426052643s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [.......................X................X.............X.....] duration: 5.001213833s trace: https://jaeger.dev.storj.io/trace/740267b1692fc58 FAIL (2025-12-15 11:28:26 UTC) orbiter warning: context deadline exceeded Duration: 5.001239553s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [........................X................X.............X....] duration: 4.574666353s trace: https://jaeger.dev.storj.io/trace/740267b1692fc58 FAIL (2025-12-15 11:27:49 UTC) orbiter warning: context deadline exceeded Duration: 4.574727341s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [.........................X................X.............X...] duration: 5.000948893s trace: https://jaeger.dev.storj.io/trace/740267b1692fc58 FAIL (2025-12-15 11:26:55 UTC) orbiter warning: context deadline exceeded Duration: 5.000974693s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [..........................X................X.............X..] duration: 4.298618037s trace: https://jaeger.dev.storj.io/trace/740267b1692fc58 FAIL (2025-12-15 11:25:42 UTC) orbiter warning: context deadline exceeded Duration: 4.298652012s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [...........................X................X.............X.] duration: 4.7336176s trace: https://jaeger.dev.storj.io/trace/740267b1692fc58 FAIL (2025-12-15 11:24:40 UTC) orbiter warning: context deadline exceeded Duration: 4.733667984s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [............................X................X.............X] duration: 5.00138813s trace: https://jaeger.dev.storj.io/trace/740267b1692fc58 FAIL (2025-12-15 11:24:14 UTC) orbiter warning: context deadline exceeded Duration: 5.001409933s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.............................X................X.............] duration: 3.735461503s trace: https://jaeger.dev.storj.io/trace/5e0346406182c7d1 FAIL (2025-12-15 11:23:52 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.735515346s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [..............................X................X............] duration: 5.001774664s trace: https://jaeger.dev.storj.io/trace/5e0346406182c7d1 FAIL (2025-12-15 11:23:08 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001802979s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [...............................X................X...........] duration: 4.816289374s trace: https://jaeger.dev.storj.io/trace/5e0346406182c7d1 FAIL (2025-12-15 11:22:33 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.816320934s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [................................X................X..........] duration: 5.00211309s trace: https://jaeger.dev.storj.io/trace/5e0346406182c7d1 FAIL (2025-12-15 11:21:46 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.002149598s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.................................X................X.........] duration: 3.444854247s trace: https://jaeger.dev.storj.io/trace/5e0346406182c7d1 FAIL (2025-12-15 11:21:21 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.444885356s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [..................................X................X........] duration: 5.001094039s trace: https://jaeger.dev.storj.io/trace/5e0346406182c7d1 FAIL (2025-12-15 11:20:37 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001118966s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [...................................X................X.......] duration: 5.000511159s trace: https://jaeger.dev.storj.io/trace/5e0346406182c7d1 FAIL (2025-12-15 11:19:24 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000603605s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [....................................X................X......] duration: 5.000423773s trace: https://jaeger.dev.storj.io/trace/5e0346406182c7d1 FAIL (2025-12-15 11:18:54 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000571214s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.....................................X................X.....] duration: 3.524478459s trace: https://jaeger.dev.storj.io/trace/5e0346406182c7d1 FAIL (2025-12-15 11:17:48 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.524502356s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [......................................X................X....] duration: 4.962551605s trace: https://jaeger.dev.storj.io/trace/5e0346406182c7d1 FAIL (2025-12-15 11:17:33 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.962579518s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.......................................X................X...] duration: 4.546635938s trace: https://jaeger.dev.storj.io/trace/5e0346406182c7d1 FAIL (2025-12-15 11:17:02 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.546665997s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [........................................X................X..] duration: 4.817606905s trace: https://jaeger.dev.storj.io/trace/5e0346406182c7d1 FAIL (2025-12-15 11:15:58 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.817639566s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.........................................X................X.] duration: 4.27534105s trace: https://jaeger.dev.storj.io/trace/5e0346406182c7d1 FAIL (2025-12-15 11:15:14 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.275359836s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [..........................................X................X] duration: 5.001334283s trace: https://jaeger.dev.storj.io/trace/5e0346406182c7d1 FAIL (2025-12-15 11:14:22 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.00136355s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...........................................X................] duration: 4.79526538s trace: https://jaeger.dev.storj.io/trace/55752dded9459114 FAIL (2025-12-15 11:13:46 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.795294014s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [............................................X...............] duration: 4.294349604s trace: https://jaeger.dev.storj.io/trace/55752dded9459114 FAIL (2025-12-15 11:13:04 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.294378379s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.............................................X..............] duration: 3.523545602s trace: https://jaeger.dev.storj.io/trace/55752dded9459114 FAIL (2025-12-15 11:12:00 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.523606798s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..............................................X.............] duration: 4.112633789s trace: https://jaeger.dev.storj.io/trace/55752dded9459114 FAIL (2025-12-15 11:10:45 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.112664258s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...............................................X............] duration: 3.404955993s trace: https://jaeger.dev.storj.io/trace/55752dded9459114 FAIL (2025-12-15 11:09:58 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.40498646s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [................................................X...........] duration: 4.111915656s trace: https://jaeger.dev.storj.io/trace/55752dded9459114 FAIL (2025-12-15 11:09:02 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.111956683s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.................................................X..........] duration: 4.227955208s trace: https://jaeger.dev.storj.io/trace/55752dded9459114 FAIL (2025-12-15 11:08:33 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.227986067s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..................................................X.........] duration: 3.467304764s trace: https://jaeger.dev.storj.io/trace/55752dded9459114 FAIL (2025-12-15 11:07:34 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.467335213s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...................................................X........] duration: 4.344147323s trace: https://jaeger.dev.storj.io/trace/55752dded9459114 FAIL (2025-12-15 11:06:56 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.344176479s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 8.792407629s trace: https://jaeger.dev.storj.io/trace/4a708d72fc2c52e9 FAIL (2025-12-15 11:06:38 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, Piece ID: EDSOWIAZ4FOKDNU33OIAOTCHZ4ATYSUFABGRXWZIXBLX45ZHXYIQ) context deadline exceeded; piecestore: (Node ID: 138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, Piece ID: 5LZCGCHJVFRE3F4AVYDQAPADEMZBEHGISIBE6UJH6UUIHGR6QOUQ) context deadline exceeded ======LOGS====== Dec 15 10:20:49.101615376 (234h51m48.972698411s) [upload=205925]: Starting upload Dec 15 10:20:49.101674809 (234h51m48.972757854s) [upload=205925]: Done starting upload Dec 15 10:20:49.101724754 (234h51m48.972807809s) [upload=205925]: Uploading segments... Dec 15 10:20:49.101919515 (234h51m48.97300255s) [upload=205925]: Got next segment. Inline: false Dec 15 10:20:49.101952237 (234h51m48.973035272s) [upload=205925]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 15 10:20:49.101960182 (234h51m48.973043217s) [upload=205925]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 15 10:20:49.424060615 (234h51m49.29514369s) [seg_pos={0 0}, upload=205925]: Begin upload segment... Dec 15 10:20:49.424121009 (234h51m49.295204045s) [seg_pos={0 0}, upload=205925]: Done begin upload segment. Dec 15 10:20:49.424161688 (234h51m49.295244733s) [noise=true , address=145.79.227.32:4834 , piece_id=3SYBLOFKGOUJDTQEBKUQ5SATA2HDXBCTBLMX67AFPBOPLASI4U6Q, seg_pos={0 0}, upload=205925]: Uploading piece... Dec 15 10:20:49.424254584 (234h51m49.295337629s) [noise=true , address=145.79.227.42:4122 , piece_id=ALJ2YXWR7HT4M53UBTDNYO2EXNOOJCU7KL5NQZCQ6S3PYGSQ3AUQ, seg_pos={0 0}, upload=205925]: Uploading piece... Dec 15 10:20:49.424905434 (234h51m49.295988479s) [noise=true , address=77.90.31.16:4735 , piece_id=EDSOWIAZ4FOKDNU33OIAOTCHZ4ATYSUFABGRXWZIXBLX45ZHXYIQ, seg_pos={0 0}, upload=205925]: Uploading piece... Dec 15 10:20:49.424915713 (234h51m49.295998758s) [noise=true , address=145.79.227.51:4410 , piece_id=RB5N6MONVOFKSF3GGEQVRL6EGGPMC5UJ77NVPRGEJV7UYUP3XNFA, seg_pos={0 0}, upload=205925]: Uploading piece... Dec 15 10:20:49.424964086 (234h51m49.296047131s) [noise=true , address=77.90.31.14:4662 , piece_id=5LZCGCHJVFRE3F4AVYDQAPADEMZBEHGISIBE6UJH6UUIHGR6QOUQ, seg_pos={0 0}, upload=205925]: Uploading piece... Dec 15 10:20:49.425378545 (234h51m49.29646159s) [noise=true , address=145.79.227.40:4050 , piece_id=LYBFCWNG53LIAPARIFUQPQBVO2EFCD6DXBR5B6FAWDIFPA7QRPHQ, seg_pos={0 0}, upload=205925]: Uploading piece... Dec 15 10:20:49.425429762 (234h51m49.296512807s) [noise=true , address=145.79.227.27:4690 , piece_id=YNTNQDEKK4HEPHCDKTWTKGBAIBUW63LZ2X3VUMFW6U75L24CFBHA, seg_pos={0 0}, upload=205925]: Uploading piece... Dec 15 10:20:49.425828811 (234h51m49.296911846s) [noise=true , address=145.79.227.8:4329 , piece_id=GIHDKALXLKHSUFFLQSOKSLHXMGMYPOPSXFNWZCI6A7GHJ2KQHEKQ, seg_pos={0 0}, upload=205925]: Uploading piece... Dec 15 10:20:49.425977345 (234h51m49.29706039s) [noise=true , address=145.79.227.28:4725 , piece_id=FQ6CLMT5DAYQJEQMYBBI2U63353PWPLJEV4RFHGZCZ7PVB3G4N3Q, seg_pos={0 0}, upload=205925]: Uploading piece... Dec 15 10:20:49.426452569 (234h51m49.297535604s) [noise=true , address=145.79.227.23:4546 , piece_id=OIXWEY7NSIW4WK4QOGKGKYS2YSI6YLFRVHRDCICXUP5CST73PMQQ, seg_pos={0 0}, upload=205925]: Uploading piece... Dec 15 10:20:49.427024318 (234h51m49.298107364s) [noise=true , address=145.79.227.35:4483 , piece_id=PGLX25C2FYVOGJ5RNV6H7J32UIJZVHH46YJWPHM2RLBGHE3AOD6A, seg_pos={0 0}, upload=205925]: Uploading piece... Dec 15 10:20:49.427498542 (234h51m49.298581587s) [noise=true , address=145.79.227.50:4373 , piece_id=WZFK2WQVD5PBH5YZQZJXVZNL5LHQXNAEE6ODWMJKEHW6QKHPYDWQ, seg_pos={0 0}, upload=205925]: Uploading piece... Dec 15 10:20:49.427934261 (234h51m49.299017296s) [noise=true , address=145.79.227.31:4797 , piece_id=LJTUYNDSGEOLP5GC67MTTHIECLSV3UY6WAT6FBXAEDPWWGMFB7CA, seg_pos={0 0}, upload=205925]: Uploading piece... Dec 15 10:20:49.428816932 (234h51m49.299899977s) [noise=true , address=145.79.227.10:4401 , piece_id=RLN5U2PG3IEEGHHHPD3EHAUJKMIQLLMKPGSAH7QADMWSQ3A67F4A, seg_pos={0 0}, upload=205925]: Uploading piece... Dec 15 10:20:49.430214363 (234h51m49.301297408s) [noise=true , address=145.79.227.25:4619 , piece_id=LIP5BYLRZW7E4V2JE3UQNRGLP7VSF5HWUQJAGBX2Z7O5IRAAXXEQ, seg_pos={0 0}, upload=205925]: Uploading piece... Dec 15 10:20:49.434382891 (234h51m49.305465936s) [noise=true , address=145.79.227.38:4977 , piece_id=6RYP2ACTJ3HEN3C22WGMXEVLUMBG4HEPC7BYDFWEPNHK5WCKM3HQ, seg_pos={0 0}, upload=205925]: Uploading piece... Dec 15 10:20:51.173093833 (234h51m51.044176878s) [upload=205925]: Next returned nil segment Dec 15 10:20:51.173109864 (234h51m51.044192909s) [upload=205925]: Waiting for error group managing segments... Dec 15 10:20:51.700311661 (234h51m51.571394706s) [noise=true , address=77.90.31.16:4735 , piece_id=EDSOWIAZ4FOKDNU33OIAOTCHZ4ATYSUFABGRXWZIXBLX45ZHXYIQ, seg_pos={0 0}, upload=205925]: Done uploading piece. err: <nil> Dec 15 10:20:51.865652042 (234h51m51.736735077s) [noise=true , address=77.90.31.14:4662 , piece_id=5LZCGCHJVFRE3F4AVYDQAPADEMZBEHGISIBE6UJH6UUIHGR6QOUQ, seg_pos={0 0}, upload=205925]: Done uploading piece. err: <nil> Dec 15 10:20:52.041004417 (234h51m51.912087453s) [noise=true , address=145.79.227.32:4834 , piece_id=3SYBLOFKGOUJDTQEBKUQ5SATA2HDXBCTBLMX67AFPBOPLASI4U6Q, seg_pos={0 0}, upload=205925]: Done uploading piece. err: <nil> Dec 15 10:20:52.041882921 (234h51m51.912965966s) [noise=true , address=145.79.227.10:4401 , piece_id=RLN5U2PG3IEEGHHHPD3EHAUJKMIQLLMKPGSAH7QADMWSQ3A67F4A, seg_pos={0 0}, upload=205925]: Done uploading piece. err: <nil> Dec 15 10:20:52.204300587 (234h51m52.075383622s) [noise=true , address=145.79.227.50:4373 , piece_id=WZFK2WQVD5PBH5YZQZJXVZNL5LHQXNAEE6ODWMJKEHW6QKHPYDWQ, seg_pos={0 0}, upload=205925]: Done uploading piece. err: <nil> Dec 15 10:20:52.215017794 (234h51m52.086100839s) [noise=true , address=145.79.227.25:4619 , piece_id=LIP5BYLRZW7E4V2JE3UQNRGLP7VSF5HWUQJAGBX2Z7O5IRAAXXEQ, seg_pos={0 0}, upload=205925]: Done uploading piece. err: <nil> Dec 15 10:20:52.215343334 (234h51m52.086426369s) [noise=true , address=145.79.227.51:4410 , piece_id=RB5N6MONVOFKSF3GGEQVRL6EGGPMC5UJ77NVPRGEJV7UYUP3XNFA, seg_pos={0 0}, upload=205925]: Done uploading piece. err: <nil> Dec 15 10:20:52.229549165 (234h51m52.10063221s) [noise=true , address=145.79.227.28:4725 , piece_id=FQ6CLMT5DAYQJEQMYBBI2U63353PWPLJEV4RFHGZCZ7PVB3G4N3Q, seg_pos={0 0}, upload=205925]: Done uploading piece. err: <nil> Dec 15 10:20:52.449423590 (234h51m52.320506635s) [noise=true , address=145.79.227.42:4122 , piece_id=ALJ2YXWR7HT4M53UBTDNYO2EXNOOJCU7KL5NQZCQ6S3PYGSQ3AUQ, seg_pos={0 0}, upload=205925]: Done uploading piece. err: <nil> Dec 15 10:20:52.455660495 (234h51m52.326743531s) [noise=true , address=145.79.227.40:4050 , piece_id=LYBFCWNG53LIAPARIFUQPQBVO2EFCD6DXBR5B6FAWDIFPA7QRPHQ, seg_pos={0 0}, upload=205925]: Done uploading piece. err: <nil> Dec 15 10:20:52.458375175 (234h51m52.32945822s) [noise=true , address=145.79.227.27:4690 , piece_id=YNTNQDEKK4HEPHCDKTWTKGBAIBUW63LZ2X3VUMFW6U75L24CFBHA, seg_pos={0 0}, upload=205925]: Done uploading piece. err: <nil> Dec 15 10:20:52.459049809 (234h51m52.330132854s) [noise=true , address=145.79.227.35:4483 , piece_id=PGLX25C2FYVOGJ5RNV6H7J32UIJZVHH46YJWPHM2RLBGHE3AOD6A, seg_pos={0 0}, upload=205925]: Done uploading piece. err: <nil> Dec 15 10:20:52.479342080 (234h51m52.350425115s) [noise=true , address=145.79.227.8:4329 , piece_id=GIHDKALXLKHSUFFLQSOKSLHXMGMYPOPSXFNWZCI6A7GHJ2KQHEKQ, seg_pos={0 0}, upload=205925]: Done uploading piece. err: <nil> Dec 15 10:20:52.479365885 (234h51m52.35044893s) [seg_pos={0 0}, upload=205925]: Segment reached optimal threshold of 13 pieces. Dec 15 10:20:52.479574001 (234h51m52.350657036s) [noise=true , address=145.79.227.23:4546 , piece_id=OIXWEY7NSIW4WK4QOGKGKYS2YSI6YLFRVHRDCICXUP5CST73PMQQ, seg_pos={0 0}, upload=205925]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZMTFpqfMjeVWXJuvsGdbGkMtGQn6axUZFoU1N9sK6fnbPcMy5): protocol: expected piece hash; context canceled; EOF Dec 15 10:20:52.479584632 (234h51m52.350667667s) [noise=true , address=145.79.227.23:4546 , piece_id=OIXWEY7NSIW4WK4QOGKGKYS2YSI6YLFRVHRDCICXUP5CST73PMQQ, seg_pos={0 0}, upload=205925]: Long tail cancellation due to optimal threshold Dec 15 10:20:52.479595793 (234h51m52.350678838s) [noise=true , address=145.79.227.38:4977 , piece_id=6RYP2ACTJ3HEN3C22WGMXEVLUMBG4HEPC7BYDFWEPNHK5WCKM3HQ, seg_pos={0 0}, upload=205925]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125dJwj1tvBkFrgDCoj3FhpH553yGgjxPuSTGxbrkGeDwdKk6M8): protocol: expected piece hash; context canceled; EOF Dec 15 10:20:52.479606773 (234h51m52.350689809s) [noise=true , address=145.79.227.38:4977 , piece_id=6RYP2ACTJ3HEN3C22WGMXEVLUMBG4HEPC7BYDFWEPNHK5WCKM3HQ, seg_pos={0 0}, upload=205925]: Long tail cancellation due to optimal threshold Dec 15 10:20:52.479787528 (234h51m52.350870563s) [noise=true , address=145.79.227.31:4797 , piece_id=LJTUYNDSGEOLP5GC67MTTHIECLSV3UY6WAT6FBXAEDPWWGMFB7CA, seg_pos={0 0}, upload=205925]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9): protocol: expected piece hash; context canceled; EOF Dec 15 10:20:52.479795373 (234h51m52.350878408s) [noise=true , address=145.79.227.31:4797 , piece_id=LJTUYNDSGEOLP5GC67MTTHIECLSV3UY6WAT6FBXAEDPWWGMFB7CA, seg_pos={0 0}, upload=205925]: Long tail cancellation due to optimal threshold Dec 15 10:20:52.479819700 (234h51m52.350902735s) [seg_pos={0 0}, upload=205925]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 15 10:20:52.479903930 (234h51m52.350986975s) [upload=205925]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 15 10:20:52.479909911 (234h51m52.350992956s) [upload=205925]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 15 10:20:52.884169543 (234h51m52.755252618s) [upload=205925]: Done uploading segments... Dec 15 10:20:52.884199039 (234h51m52.755282104s) [upload=205925]: Upload finished. err: <nil> Duration: 8.792440111s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [....................................................X.......] duration: 3.454382153s trace: https://jaeger.dev.storj.io/trace/55752dded9459114 FAIL (2025-12-15 11:06:10 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.454408813s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 8.472289323s trace: https://jaeger.dev.storj.io/trace/4a708d72fc2c52e9 FAIL (2025-12-15 11:05:34 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, Piece ID: EDSOWIAZ4FOKDNU33OIAOTCHZ4ATYSUFABGRXWZIXBLX45ZHXYIQ) context deadline exceeded; piecestore: (Node ID: 138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, Piece ID: 5LZCGCHJVFRE3F4AVYDQAPADEMZBEHGISIBE6UJH6UUIHGR6QOUQ) context deadline exceeded ======LOGS====== Dec 15 10:20:49.101615376 (234h51m48.972698411s) [upload=205925]: Starting upload Dec 15 10:20:49.101674809 (234h51m48.972757854s) [upload=205925]: Done starting upload Dec 15 10:20:49.101724754 (234h51m48.972807809s) [upload=205925]: Uploading segments... Dec 15 10:20:49.101919515 (234h51m48.97300255s) [upload=205925]: Got next segment. Inline: false Dec 15 10:20:49.101952237 (234h51m48.973035272s) [upload=205925]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 15 10:20:49.101960182 (234h51m48.973043217s) [upload=205925]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 15 10:20:49.424060615 (234h51m49.29514369s) [seg_pos={0 0}, upload=205925]: Begin upload segment... Dec 15 10:20:49.424121009 (234h51m49.295204045s) [seg_pos={0 0}, upload=205925]: Done begin upload segment. Dec 15 10:20:49.424161688 (234h51m49.295244733s) [noise=true , address=145.79.227.32:4834 , piece_id=3SYBLOFKGOUJDTQEBKUQ5SATA2HDXBCTBLMX67AFPBOPLASI4U6Q, seg_pos={0 0}, upload=205925]: Uploading piece... Dec 15 10:20:49.424254584 (234h51m49.295337629s) [noise=true , address=145.79.227.42:4122 , piece_id=ALJ2YXWR7HT4M53UBTDNYO2EXNOOJCU7KL5NQZCQ6S3PYGSQ3AUQ, seg_pos={0 0}, upload=205925]: Uploading piece... Dec 15 10:20:49.424905434 (234h51m49.295988479s) [noise=true , address=77.90.31.16:4735 , piece_id=EDSOWIAZ4FOKDNU33OIAOTCHZ4ATYSUFABGRXWZIXBLX45ZHXYIQ, seg_pos={0 0}, upload=205925]: Uploading piece... Dec 15 10:20:49.424915713 (234h51m49.295998758s) [noise=true , address=145.79.227.51:4410 , piece_id=RB5N6MONVOFKSF3GGEQVRL6EGGPMC5UJ77NVPRGEJV7UYUP3XNFA, seg_pos={0 0}, upload=205925]: Uploading piece... Dec 15 10:20:49.424964086 (234h51m49.296047131s) [noise=true , address=77.90.31.14:4662 , piece_id=5LZCGCHJVFRE3F4AVYDQAPADEMZBEHGISIBE6UJH6UUIHGR6QOUQ, seg_pos={0 0}, upload=205925]: Uploading piece... Dec 15 10:20:49.425378545 (234h51m49.29646159s) [noise=true , address=145.79.227.40:4050 , piece_id=LYBFCWNG53LIAPARIFUQPQBVO2EFCD6DXBR5B6FAWDIFPA7QRPHQ, seg_pos={0 0}, upload=205925]: Uploading piece... Dec 15 10:20:49.425429762 (234h51m49.296512807s) [noise=true , address=145.79.227.27:4690 , piece_id=YNTNQDEKK4HEPHCDKTWTKGBAIBUW63LZ2X3VUMFW6U75L24CFBHA, seg_pos={0 0}, upload=205925]: Uploading piece... Dec 15 10:20:49.425828811 (234h51m49.296911846s) [noise=true , address=145.79.227.8:4329 , piece_id=GIHDKALXLKHSUFFLQSOKSLHXMGMYPOPSXFNWZCI6A7GHJ2KQHEKQ, seg_pos={0 0}, upload=205925]: Uploading piece... Dec 15 10:20:49.425977345 (234h51m49.29706039s) [noise=true , address=145.79.227.28:4725 , piece_id=FQ6CLMT5DAYQJEQMYBBI2U63353PWPLJEV4RFHGZCZ7PVB3G4N3Q, seg_pos={0 0}, upload=205925]: Uploading piece... Dec 15 10:20:49.426452569 (234h51m49.297535604s) [noise=true , address=145.79.227.23:4546 , piece_id=OIXWEY7NSIW4WK4QOGKGKYS2YSI6YLFRVHRDCICXUP5CST73PMQQ, seg_pos={0 0}, upload=205925]: Uploading piece... Dec 15 10:20:49.427024318 (234h51m49.298107364s) [noise=true , address=145.79.227.35:4483 , piece_id=PGLX25C2FYVOGJ5RNV6H7J32UIJZVHH46YJWPHM2RLBGHE3AOD6A, seg_pos={0 0}, upload=205925]: Uploading piece... Dec 15 10:20:49.427498542 (234h51m49.298581587s) [noise=true , address=145.79.227.50:4373 , piece_id=WZFK2WQVD5PBH5YZQZJXVZNL5LHQXNAEE6ODWMJKEHW6QKHPYDWQ, seg_pos={0 0}, upload=205925]: Uploading piece... Dec 15 10:20:49.427934261 (234h51m49.299017296s) [noise=true , address=145.79.227.31:4797 , piece_id=LJTUYNDSGEOLP5GC67MTTHIECLSV3UY6WAT6FBXAEDPWWGMFB7CA, seg_pos={0 0}, upload=205925]: Uploading piece... Dec 15 10:20:49.428816932 (234h51m49.299899977s) [noise=true , address=145.79.227.10:4401 , piece_id=RLN5U2PG3IEEGHHHPD3EHAUJKMIQLLMKPGSAH7QADMWSQ3A67F4A, seg_pos={0 0}, upload=205925]: Uploading piece... Dec 15 10:20:49.430214363 (234h51m49.301297408s) [noise=true , address=145.79.227.25:4619 , piece_id=LIP5BYLRZW7E4V2JE3UQNRGLP7VSF5HWUQJAGBX2Z7O5IRAAXXEQ, seg_pos={0 0}, upload=205925]: Uploading piece... Dec 15 10:20:49.434382891 (234h51m49.305465936s) [noise=true , address=145.79.227.38:4977 , piece_id=6RYP2ACTJ3HEN3C22WGMXEVLUMBG4HEPC7BYDFWEPNHK5WCKM3HQ, seg_pos={0 0}, upload=205925]: Uploading piece... Dec 15 10:20:51.173093833 (234h51m51.044176878s) [upload=205925]: Next returned nil segment Dec 15 10:20:51.173109864 (234h51m51.044192909s) [upload=205925]: Waiting for error group managing segments... Dec 15 10:20:51.700311661 (234h51m51.571394706s) [noise=true , address=77.90.31.16:4735 , piece_id=EDSOWIAZ4FOKDNU33OIAOTCHZ4ATYSUFABGRXWZIXBLX45ZHXYIQ, seg_pos={0 0}, upload=205925]: Done uploading piece. err: <nil> Dec 15 10:20:51.865652042 (234h51m51.736735077s) [noise=true , address=77.90.31.14:4662 , piece_id=5LZCGCHJVFRE3F4AVYDQAPADEMZBEHGISIBE6UJH6UUIHGR6QOUQ, seg_pos={0 0}, upload=205925]: Done uploading piece. err: <nil> Dec 15 10:20:52.041004417 (234h51m51.912087453s) [noise=true , address=145.79.227.32:4834 , piece_id=3SYBLOFKGOUJDTQEBKUQ5SATA2HDXBCTBLMX67AFPBOPLASI4U6Q, seg_pos={0 0}, upload=205925]: Done uploading piece. err: <nil> Dec 15 10:20:52.041882921 (234h51m51.912965966s) [noise=true , address=145.79.227.10:4401 , piece_id=RLN5U2PG3IEEGHHHPD3EHAUJKMIQLLMKPGSAH7QADMWSQ3A67F4A, seg_pos={0 0}, upload=205925]: Done uploading piece. err: <nil> Dec 15 10:20:52.204300587 (234h51m52.075383622s) [noise=true , address=145.79.227.50:4373 , piece_id=WZFK2WQVD5PBH5YZQZJXVZNL5LHQXNAEE6ODWMJKEHW6QKHPYDWQ, seg_pos={0 0}, upload=205925]: Done uploading piece. err: <nil> Dec 15 10:20:52.215017794 (234h51m52.086100839s) [noise=true , address=145.79.227.25:4619 , piece_id=LIP5BYLRZW7E4V2JE3UQNRGLP7VSF5HWUQJAGBX2Z7O5IRAAXXEQ, seg_pos={0 0}, upload=205925]: Done uploading piece. err: <nil> Dec 15 10:20:52.215343334 (234h51m52.086426369s) [noise=true , address=145.79.227.51:4410 , piece_id=RB5N6MONVOFKSF3GGEQVRL6EGGPMC5UJ77NVPRGEJV7UYUP3XNFA, seg_pos={0 0}, upload=205925]: Done uploading piece. err: <nil> Dec 15 10:20:52.229549165 (234h51m52.10063221s) [noise=true , address=145.79.227.28:4725 , piece_id=FQ6CLMT5DAYQJEQMYBBI2U63353PWPLJEV4RFHGZCZ7PVB3G4N3Q, seg_pos={0 0}, upload=205925]: Done uploading piece. err: <nil> Dec 15 10:20:52.449423590 (234h51m52.320506635s) [noise=true , address=145.79.227.42:4122 , piece_id=ALJ2YXWR7HT4M53UBTDNYO2EXNOOJCU7KL5NQZCQ6S3PYGSQ3AUQ, seg_pos={0 0}, upload=205925]: Done uploading piece. err: <nil> Dec 15 10:20:52.455660495 (234h51m52.326743531s) [noise=true , address=145.79.227.40:4050 , piece_id=LYBFCWNG53LIAPARIFUQPQBVO2EFCD6DXBR5B6FAWDIFPA7QRPHQ, seg_pos={0 0}, upload=205925]: Done uploading piece. err: <nil> Dec 15 10:20:52.458375175 (234h51m52.32945822s) [noise=true , address=145.79.227.27:4690 , piece_id=YNTNQDEKK4HEPHCDKTWTKGBAIBUW63LZ2X3VUMFW6U75L24CFBHA, seg_pos={0 0}, upload=205925]: Done uploading piece. err: <nil> Dec 15 10:20:52.459049809 (234h51m52.330132854s) [noise=true , address=145.79.227.35:4483 , piece_id=PGLX25C2FYVOGJ5RNV6H7J32UIJZVHH46YJWPHM2RLBGHE3AOD6A, seg_pos={0 0}, upload=205925]: Done uploading piece. err: <nil> Dec 15 10:20:52.479342080 (234h51m52.350425115s) [noise=true , address=145.79.227.8:4329 , piece_id=GIHDKALXLKHSUFFLQSOKSLHXMGMYPOPSXFNWZCI6A7GHJ2KQHEKQ, seg_pos={0 0}, upload=205925]: Done uploading piece. err: <nil> Dec 15 10:20:52.479365885 (234h51m52.35044893s) [seg_pos={0 0}, upload=205925]: Segment reached optimal threshold of 13 pieces. Dec 15 10:20:52.479574001 (234h51m52.350657036s) [noise=true , address=145.79.227.23:4546 , piece_id=OIXWEY7NSIW4WK4QOGKGKYS2YSI6YLFRVHRDCICXUP5CST73PMQQ, seg_pos={0 0}, upload=205925]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZMTFpqfMjeVWXJuvsGdbGkMtGQn6axUZFoU1N9sK6fnbPcMy5): protocol: expected piece hash; context canceled; EOF Dec 15 10:20:52.479584632 (234h51m52.350667667s) [noise=true , address=145.79.227.23:4546 , piece_id=OIXWEY7NSIW4WK4QOGKGKYS2YSI6YLFRVHRDCICXUP5CST73PMQQ, seg_pos={0 0}, upload=205925]: Long tail cancellation due to optimal threshold Dec 15 10:20:52.479595793 (234h51m52.350678838s) [noise=true , address=145.79.227.38:4977 , piece_id=6RYP2ACTJ3HEN3C22WGMXEVLUMBG4HEPC7BYDFWEPNHK5WCKM3HQ, seg_pos={0 0}, upload=205925]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125dJwj1tvBkFrgDCoj3FhpH553yGgjxPuSTGxbrkGeDwdKk6M8): protocol: expected piece hash; context canceled; EOF Dec 15 10:20:52.479606773 (234h51m52.350689809s) [noise=true , address=145.79.227.38:4977 , piece_id=6RYP2ACTJ3HEN3C22WGMXEVLUMBG4HEPC7BYDFWEPNHK5WCKM3HQ, seg_pos={0 0}, upload=205925]: Long tail cancellation due to optimal threshold Dec 15 10:20:52.479787528 (234h51m52.350870563s) [noise=true , address=145.79.227.31:4797 , piece_id=LJTUYNDSGEOLP5GC67MTTHIECLSV3UY6WAT6FBXAEDPWWGMFB7CA, seg_pos={0 0}, upload=205925]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9): protocol: expected piece hash; context canceled; EOF Dec 15 10:20:52.479795373 (234h51m52.350878408s) [noise=true , address=145.79.227.31:4797 , piece_id=LJTUYNDSGEOLP5GC67MTTHIECLSV3UY6WAT6FBXAEDPWWGMFB7CA, seg_pos={0 0}, upload=205925]: Long tail cancellation due to optimal threshold Dec 15 10:20:52.479819700 (234h51m52.350902735s) [seg_pos={0 0}, upload=205925]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 15 10:20:52.479903930 (234h51m52.350986975s) [upload=205925]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 15 10:20:52.479909911 (234h51m52.350992956s) [upload=205925]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 15 10:20:52.884169543 (234h51m52.755252618s) [upload=205925]: Done uploading segments... Dec 15 10:20:52.884199039 (234h51m52.755282104s) [upload=205925]: Upload finished. err: <nil> Duration: 8.472319689s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.....................................................X......] duration: 3.470422151s trace: https://jaeger.dev.storj.io/trace/55752dded9459114 FAIL (2025-12-15 11:05:23 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.470442903s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [......................................................X.....] duration: 4.762095934s trace: https://jaeger.dev.storj.io/trace/55752dded9459114 FAIL (2025-12-15 11:04:47 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.76215679s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 7.391758416s trace: https://jaeger.dev.storj.io/trace/4a708d72fc2c52e9 FAIL (2025-12-15 11:04:44 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, Piece ID: EDSOWIAZ4FOKDNU33OIAOTCHZ4ATYSUFABGRXWZIXBLX45ZHXYIQ) context deadline exceeded; piecestore: (Node ID: 138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, Piece ID: 5LZCGCHJVFRE3F4AVYDQAPADEMZBEHGISIBE6UJH6UUIHGR6QOUQ) context deadline exceeded ======LOGS====== Dec 15 10:20:49.101615376 (234h51m48.972698411s) [upload=205925]: Starting upload Dec 15 10:20:49.101674809 (234h51m48.972757854s) [upload=205925]: Done starting upload Dec 15 10:20:49.101724754 (234h51m48.972807809s) [upload=205925]: Uploading segments... Dec 15 10:20:49.101919515 (234h51m48.97300255s) [upload=205925]: Got next segment. Inline: false Dec 15 10:20:49.101952237 (234h51m48.973035272s) [upload=205925]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 15 10:20:49.101960182 (234h51m48.973043217s) [upload=205925]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 15 10:20:49.424060615 (234h51m49.29514369s) [seg_pos={0 0}, upload=205925]: Begin upload segment... Dec 15 10:20:49.424121009 (234h51m49.295204045s) [seg_pos={0 0}, upload=205925]: Done begin upload segment. Dec 15 10:20:49.424161688 (234h51m49.295244733s) [noise=true , address=145.79.227.32:4834 , piece_id=3SYBLOFKGOUJDTQEBKUQ5SATA2HDXBCTBLMX67AFPBOPLASI4U6Q, seg_pos={0 0}, upload=205925]: Uploading piece... Dec 15 10:20:49.424254584 (234h51m49.295337629s) [noise=true , address=145.79.227.42:4122 , piece_id=ALJ2YXWR7HT4M53UBTDNYO2EXNOOJCU7KL5NQZCQ6S3PYGSQ3AUQ, seg_pos={0 0}, upload=205925]: Uploading piece... Dec 15 10:20:49.424905434 (234h51m49.295988479s) [noise=true , address=77.90.31.16:4735 , piece_id=EDSOWIAZ4FOKDNU33OIAOTCHZ4ATYSUFABGRXWZIXBLX45ZHXYIQ, seg_pos={0 0}, upload=205925]: Uploading piece... Dec 15 10:20:49.424915713 (234h51m49.295998758s) [noise=true , address=145.79.227.51:4410 , piece_id=RB5N6MONVOFKSF3GGEQVRL6EGGPMC5UJ77NVPRGEJV7UYUP3XNFA, seg_pos={0 0}, upload=205925]: Uploading piece... Dec 15 10:20:49.424964086 (234h51m49.296047131s) [noise=true , address=77.90.31.14:4662 , piece_id=5LZCGCHJVFRE3F4AVYDQAPADEMZBEHGISIBE6UJH6UUIHGR6QOUQ, seg_pos={0 0}, upload=205925]: Uploading piece... Dec 15 10:20:49.425378545 (234h51m49.29646159s) [noise=true , address=145.79.227.40:4050 , piece_id=LYBFCWNG53LIAPARIFUQPQBVO2EFCD6DXBR5B6FAWDIFPA7QRPHQ, seg_pos={0 0}, upload=205925]: Uploading piece... Dec 15 10:20:49.425429762 (234h51m49.296512807s) [noise=true , address=145.79.227.27:4690 , piece_id=YNTNQDEKK4HEPHCDKTWTKGBAIBUW63LZ2X3VUMFW6U75L24CFBHA, seg_pos={0 0}, upload=205925]: Uploading piece... Dec 15 10:20:49.425828811 (234h51m49.296911846s) [noise=true , address=145.79.227.8:4329 , piece_id=GIHDKALXLKHSUFFLQSOKSLHXMGMYPOPSXFNWZCI6A7GHJ2KQHEKQ, seg_pos={0 0}, upload=205925]: Uploading piece... Dec 15 10:20:49.425977345 (234h51m49.29706039s) [noise=true , address=145.79.227.28:4725 , piece_id=FQ6CLMT5DAYQJEQMYBBI2U63353PWPLJEV4RFHGZCZ7PVB3G4N3Q, seg_pos={0 0}, upload=205925]: Uploading piece... Dec 15 10:20:49.426452569 (234h51m49.297535604s) [noise=true , address=145.79.227.23:4546 , piece_id=OIXWEY7NSIW4WK4QOGKGKYS2YSI6YLFRVHRDCICXUP5CST73PMQQ, seg_pos={0 0}, upload=205925]: Uploading piece... Dec 15 10:20:49.427024318 (234h51m49.298107364s) [noise=true , address=145.79.227.35:4483 , piece_id=PGLX25C2FYVOGJ5RNV6H7J32UIJZVHH46YJWPHM2RLBGHE3AOD6A, seg_pos={0 0}, upload=205925]: Uploading piece... Dec 15 10:20:49.427498542 (234h51m49.298581587s) [noise=true , address=145.79.227.50:4373 , piece_id=WZFK2WQVD5PBH5YZQZJXVZNL5LHQXNAEE6ODWMJKEHW6QKHPYDWQ, seg_pos={0 0}, upload=205925]: Uploading piece... Dec 15 10:20:49.427934261 (234h51m49.299017296s) [noise=true , address=145.79.227.31:4797 , piece_id=LJTUYNDSGEOLP5GC67MTTHIECLSV3UY6WAT6FBXAEDPWWGMFB7CA, seg_pos={0 0}, upload=205925]: Uploading piece... Dec 15 10:20:49.428816932 (234h51m49.299899977s) [noise=true , address=145.79.227.10:4401 , piece_id=RLN5U2PG3IEEGHHHPD3EHAUJKMIQLLMKPGSAH7QADMWSQ3A67F4A, seg_pos={0 0}, upload=205925]: Uploading piece... Dec 15 10:20:49.430214363 (234h51m49.301297408s) [noise=true , address=145.79.227.25:4619 , piece_id=LIP5BYLRZW7E4V2JE3UQNRGLP7VSF5HWUQJAGBX2Z7O5IRAAXXEQ, seg_pos={0 0}, upload=205925]: Uploading piece... Dec 15 10:20:49.434382891 (234h51m49.305465936s) [noise=true , address=145.79.227.38:4977 , piece_id=6RYP2ACTJ3HEN3C22WGMXEVLUMBG4HEPC7BYDFWEPNHK5WCKM3HQ, seg_pos={0 0}, upload=205925]: Uploading piece... Dec 15 10:20:51.173093833 (234h51m51.044176878s) [upload=205925]: Next returned nil segment Dec 15 10:20:51.173109864 (234h51m51.044192909s) [upload=205925]: Waiting for error group managing segments... Dec 15 10:20:51.700311661 (234h51m51.571394706s) [noise=true , address=77.90.31.16:4735 , piece_id=EDSOWIAZ4FOKDNU33OIAOTCHZ4ATYSUFABGRXWZIXBLX45ZHXYIQ, seg_pos={0 0}, upload=205925]: Done uploading piece. err: <nil> Dec 15 10:20:51.865652042 (234h51m51.736735077s) [noise=true , address=77.90.31.14:4662 , piece_id=5LZCGCHJVFRE3F4AVYDQAPADEMZBEHGISIBE6UJH6UUIHGR6QOUQ, seg_pos={0 0}, upload=205925]: Done uploading piece. err: <nil> Dec 15 10:20:52.041004417 (234h51m51.912087453s) [noise=true , address=145.79.227.32:4834 , piece_id=3SYBLOFKGOUJDTQEBKUQ5SATA2HDXBCTBLMX67AFPBOPLASI4U6Q, seg_pos={0 0}, upload=205925]: Done uploading piece. err: <nil> Dec 15 10:20:52.041882921 (234h51m51.912965966s) [noise=true , address=145.79.227.10:4401 , piece_id=RLN5U2PG3IEEGHHHPD3EHAUJKMIQLLMKPGSAH7QADMWSQ3A67F4A, seg_pos={0 0}, upload=205925]: Done uploading piece. err: <nil> Dec 15 10:20:52.204300587 (234h51m52.075383622s) [noise=true , address=145.79.227.50:4373 , piece_id=WZFK2WQVD5PBH5YZQZJXVZNL5LHQXNAEE6ODWMJKEHW6QKHPYDWQ, seg_pos={0 0}, upload=205925]: Done uploading piece. err: <nil> Dec 15 10:20:52.215017794 (234h51m52.086100839s) [noise=true , address=145.79.227.25:4619 , piece_id=LIP5BYLRZW7E4V2JE3UQNRGLP7VSF5HWUQJAGBX2Z7O5IRAAXXEQ, seg_pos={0 0}, upload=205925]: Done uploading piece. err: <nil> Dec 15 10:20:52.215343334 (234h51m52.086426369s) [noise=true , address=145.79.227.51:4410 , piece_id=RB5N6MONVOFKSF3GGEQVRL6EGGPMC5UJ77NVPRGEJV7UYUP3XNFA, seg_pos={0 0}, upload=205925]: Done uploading piece. err: <nil> Dec 15 10:20:52.229549165 (234h51m52.10063221s) [noise=true , address=145.79.227.28:4725 , piece_id=FQ6CLMT5DAYQJEQMYBBI2U63353PWPLJEV4RFHGZCZ7PVB3G4N3Q, seg_pos={0 0}, upload=205925]: Done uploading piece. err: <nil> Dec 15 10:20:52.449423590 (234h51m52.320506635s) [noise=true , address=145.79.227.42:4122 , piece_id=ALJ2YXWR7HT4M53UBTDNYO2EXNOOJCU7KL5NQZCQ6S3PYGSQ3AUQ, seg_pos={0 0}, upload=205925]: Done uploading piece. err: <nil> Dec 15 10:20:52.455660495 (234h51m52.326743531s) [noise=true , address=145.79.227.40:4050 , piece_id=LYBFCWNG53LIAPARIFUQPQBVO2EFCD6DXBR5B6FAWDIFPA7QRPHQ, seg_pos={0 0}, upload=205925]: Done uploading piece. err: <nil> Dec 15 10:20:52.458375175 (234h51m52.32945822s) [noise=true , address=145.79.227.27:4690 , piece_id=YNTNQDEKK4HEPHCDKTWTKGBAIBUW63LZ2X3VUMFW6U75L24CFBHA, seg_pos={0 0}, upload=205925]: Done uploading piece. err: <nil> Dec 15 10:20:52.459049809 (234h51m52.330132854s) [noise=true , address=145.79.227.35:4483 , piece_id=PGLX25C2FYVOGJ5RNV6H7J32UIJZVHH46YJWPHM2RLBGHE3AOD6A, seg_pos={0 0}, upload=205925]: Done uploading piece. err: <nil> Dec 15 10:20:52.479342080 (234h51m52.350425115s) [noise=true , address=145.79.227.8:4329 , piece_id=GIHDKALXLKHSUFFLQSOKSLHXMGMYPOPSXFNWZCI6A7GHJ2KQHEKQ, seg_pos={0 0}, upload=205925]: Done uploading piece. err: <nil> Dec 15 10:20:52.479365885 (234h51m52.35044893s) [seg_pos={0 0}, upload=205925]: Segment reached optimal threshold of 13 pieces. Dec 15 10:20:52.479574001 (234h51m52.350657036s) [noise=true , address=145.79.227.23:4546 , piece_id=OIXWEY7NSIW4WK4QOGKGKYS2YSI6YLFRVHRDCICXUP5CST73PMQQ, seg_pos={0 0}, upload=205925]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZMTFpqfMjeVWXJuvsGdbGkMtGQn6axUZFoU1N9sK6fnbPcMy5): protocol: expected piece hash; context canceled; EOF Dec 15 10:20:52.479584632 (234h51m52.350667667s) [noise=true , address=145.79.227.23:4546 , piece_id=OIXWEY7NSIW4WK4QOGKGKYS2YSI6YLFRVHRDCICXUP5CST73PMQQ, seg_pos={0 0}, upload=205925]: Long tail cancellation due to optimal threshold Dec 15 10:20:52.479595793 (234h51m52.350678838s) [noise=true , address=145.79.227.38:4977 , piece_id=6RYP2ACTJ3HEN3C22WGMXEVLUMBG4HEPC7BYDFWEPNHK5WCKM3HQ, seg_pos={0 0}, upload=205925]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125dJwj1tvBkFrgDCoj3FhpH553yGgjxPuSTGxbrkGeDwdKk6M8): protocol: expected piece hash; context canceled; EOF Dec 15 10:20:52.479606773 (234h51m52.350689809s) [noise=true , address=145.79.227.38:4977 , piece_id=6RYP2ACTJ3HEN3C22WGMXEVLUMBG4HEPC7BYDFWEPNHK5WCKM3HQ, seg_pos={0 0}, upload=205925]: Long tail cancellation due to optimal threshold Dec 15 10:20:52.479787528 (234h51m52.350870563s) [noise=true , address=145.79.227.31:4797 , piece_id=LJTUYNDSGEOLP5GC67MTTHIECLSV3UY6WAT6FBXAEDPWWGMFB7CA, seg_pos={0 0}, upload=205925]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9): protocol: expected piece hash; context canceled; EOF Dec 15 10:20:52.479795373 (234h51m52.350878408s) [noise=true , address=145.79.227.31:4797 , piece_id=LJTUYNDSGEOLP5GC67MTTHIECLSV3UY6WAT6FBXAEDPWWGMFB7CA, seg_pos={0 0}, upload=205925]: Long tail cancellation due to optimal threshold Dec 15 10:20:52.479819700 (234h51m52.350902735s) [seg_pos={0 0}, upload=205925]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 15 10:20:52.479903930 (234h51m52.350986975s) [upload=205925]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 15 10:20:52.479909911 (234h51m52.350992956s) [upload=205925]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 15 10:20:52.884169543 (234h51m52.755252618s) [upload=205925]: Done uploading segments... Dec 15 10:20:52.884199039 (234h51m52.755282104s) [upload=205925]: Upload finished. err: <nil> Duration: 7.391789505s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 13.25425205s trace: https://jaeger.dev.storj.io/trace/4a708d72fc2c52e9 FAIL (2025-12-15 11:04:16 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, Piece ID: EDSOWIAZ4FOKDNU33OIAOTCHZ4ATYSUFABGRXWZIXBLX45ZHXYIQ) context deadline exceeded; piecestore: (Node ID: 138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, Piece ID: 5LZCGCHJVFRE3F4AVYDQAPADEMZBEHGISIBE6UJH6UUIHGR6QOUQ) context deadline exceeded ======LOGS====== Dec 15 10:20:49.101615376 (234h51m48.972698411s) [upload=205925]: Starting upload Dec 15 10:20:49.101674809 (234h51m48.972757854s) [upload=205925]: Done starting upload Dec 15 10:20:49.101724754 (234h51m48.972807809s) [upload=205925]: Uploading segments... Dec 15 10:20:49.101919515 (234h51m48.97300255s) [upload=205925]: Got next segment. Inline: false Dec 15 10:20:49.101952237 (234h51m48.973035272s) [upload=205925]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 15 10:20:49.101960182 (234h51m48.973043217s) [upload=205925]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 15 10:20:49.424060615 (234h51m49.29514369s) [seg_pos={0 0}, upload=205925]: Begin upload segment... Dec 15 10:20:49.424121009 (234h51m49.295204045s) [seg_pos={0 0}, upload=205925]: Done begin upload segment. Dec 15 10:20:49.424161688 (234h51m49.295244733s) [noise=true , address=145.79.227.32:4834 , piece_id=3SYBLOFKGOUJDTQEBKUQ5SATA2HDXBCTBLMX67AFPBOPLASI4U6Q, seg_pos={0 0}, upload=205925]: Uploading piece... Dec 15 10:20:49.424254584 (234h51m49.295337629s) [noise=true , address=145.79.227.42:4122 , piece_id=ALJ2YXWR7HT4M53UBTDNYO2EXNOOJCU7KL5NQZCQ6S3PYGSQ3AUQ, seg_pos={0 0}, upload=205925]: Uploading piece... Dec 15 10:20:49.424905434 (234h51m49.295988479s) [noise=true , address=77.90.31.16:4735 , piece_id=EDSOWIAZ4FOKDNU33OIAOTCHZ4ATYSUFABGRXWZIXBLX45ZHXYIQ, seg_pos={0 0}, upload=205925]: Uploading piece... Dec 15 10:20:49.424915713 (234h51m49.295998758s) [noise=true , address=145.79.227.51:4410 , piece_id=RB5N6MONVOFKSF3GGEQVRL6EGGPMC5UJ77NVPRGEJV7UYUP3XNFA, seg_pos={0 0}, upload=205925]: Uploading piece... Dec 15 10:20:49.424964086 (234h51m49.296047131s) [noise=true , address=77.90.31.14:4662 , piece_id=5LZCGCHJVFRE3F4AVYDQAPADEMZBEHGISIBE6UJH6UUIHGR6QOUQ, seg_pos={0 0}, upload=205925]: Uploading piece... Dec 15 10:20:49.425378545 (234h51m49.29646159s) [noise=true , address=145.79.227.40:4050 , piece_id=LYBFCWNG53LIAPARIFUQPQBVO2EFCD6DXBR5B6FAWDIFPA7QRPHQ, seg_pos={0 0}, upload=205925]: Uploading piece... Dec 15 10:20:49.425429762 (234h51m49.296512807s) [noise=true , address=145.79.227.27:4690 , piece_id=YNTNQDEKK4HEPHCDKTWTKGBAIBUW63LZ2X3VUMFW6U75L24CFBHA, seg_pos={0 0}, upload=205925]: Uploading piece... Dec 15 10:20:49.425828811 (234h51m49.296911846s) [noise=true , address=145.79.227.8:4329 , piece_id=GIHDKALXLKHSUFFLQSOKSLHXMGMYPOPSXFNWZCI6A7GHJ2KQHEKQ, seg_pos={0 0}, upload=205925]: Uploading piece... Dec 15 10:20:49.425977345 (234h51m49.29706039s) [noise=true , address=145.79.227.28:4725 , piece_id=FQ6CLMT5DAYQJEQMYBBI2U63353PWPLJEV4RFHGZCZ7PVB3G4N3Q, seg_pos={0 0}, upload=205925]: Uploading piece... Dec 15 10:20:49.426452569 (234h51m49.297535604s) [noise=true , address=145.79.227.23:4546 , piece_id=OIXWEY7NSIW4WK4QOGKGKYS2YSI6YLFRVHRDCICXUP5CST73PMQQ, seg_pos={0 0}, upload=205925]: Uploading piece... Dec 15 10:20:49.427024318 (234h51m49.298107364s) [noise=true , address=145.79.227.35:4483 , piece_id=PGLX25C2FYVOGJ5RNV6H7J32UIJZVHH46YJWPHM2RLBGHE3AOD6A, seg_pos={0 0}, upload=205925]: Uploading piece... Dec 15 10:20:49.427498542 (234h51m49.298581587s) [noise=true , address=145.79.227.50:4373 , piece_id=WZFK2WQVD5PBH5YZQZJXVZNL5LHQXNAEE6ODWMJKEHW6QKHPYDWQ, seg_pos={0 0}, upload=205925]: Uploading piece... Dec 15 10:20:49.427934261 (234h51m49.299017296s) [noise=true , address=145.79.227.31:4797 , piece_id=LJTUYNDSGEOLP5GC67MTTHIECLSV3UY6WAT6FBXAEDPWWGMFB7CA, seg_pos={0 0}, upload=205925]: Uploading piece... Dec 15 10:20:49.428816932 (234h51m49.299899977s) [noise=true , address=145.79.227.10:4401 , piece_id=RLN5U2PG3IEEGHHHPD3EHAUJKMIQLLMKPGSAH7QADMWSQ3A67F4A, seg_pos={0 0}, upload=205925]: Uploading piece... Dec 15 10:20:49.430214363 (234h51m49.301297408s) [noise=true , address=145.79.227.25:4619 , piece_id=LIP5BYLRZW7E4V2JE3UQNRGLP7VSF5HWUQJAGBX2Z7O5IRAAXXEQ, seg_pos={0 0}, upload=205925]: Uploading piece... Dec 15 10:20:49.434382891 (234h51m49.305465936s) [noise=true , address=145.79.227.38:4977 , piece_id=6RYP2ACTJ3HEN3C22WGMXEVLUMBG4HEPC7BYDFWEPNHK5WCKM3HQ, seg_pos={0 0}, upload=205925]: Uploading piece... Dec 15 10:20:51.173093833 (234h51m51.044176878s) [upload=205925]: Next returned nil segment Dec 15 10:20:51.173109864 (234h51m51.044192909s) [upload=205925]: Waiting for error group managing segments... Dec 15 10:20:51.700311661 (234h51m51.571394706s) [noise=true , address=77.90.31.16:4735 , piece_id=EDSOWIAZ4FOKDNU33OIAOTCHZ4ATYSUFABGRXWZIXBLX45ZHXYIQ, seg_pos={0 0}, upload=205925]: Done uploading piece. err: <nil> Dec 15 10:20:51.865652042 (234h51m51.736735077s) [noise=true , address=77.90.31.14:4662 , piece_id=5LZCGCHJVFRE3F4AVYDQAPADEMZBEHGISIBE6UJH6UUIHGR6QOUQ, seg_pos={0 0}, upload=205925]: Done uploading piece. err: <nil> Dec 15 10:20:52.041004417 (234h51m51.912087453s) [noise=true , address=145.79.227.32:4834 , piece_id=3SYBLOFKGOUJDTQEBKUQ5SATA2HDXBCTBLMX67AFPBOPLASI4U6Q, seg_pos={0 0}, upload=205925]: Done uploading piece. err: <nil> Dec 15 10:20:52.041882921 (234h51m51.912965966s) [noise=true , address=145.79.227.10:4401 , piece_id=RLN5U2PG3IEEGHHHPD3EHAUJKMIQLLMKPGSAH7QADMWSQ3A67F4A, seg_pos={0 0}, upload=205925]: Done uploading piece. err: <nil> Dec 15 10:20:52.204300587 (234h51m52.075383622s) [noise=true , address=145.79.227.50:4373 , piece_id=WZFK2WQVD5PBH5YZQZJXVZNL5LHQXNAEE6ODWMJKEHW6QKHPYDWQ, seg_pos={0 0}, upload=205925]: Done uploading piece. err: <nil> Dec 15 10:20:52.215017794 (234h51m52.086100839s) [noise=true , address=145.79.227.25:4619 , piece_id=LIP5BYLRZW7E4V2JE3UQNRGLP7VSF5HWUQJAGBX2Z7O5IRAAXXEQ, seg_pos={0 0}, upload=205925]: Done uploading piece. err: <nil> Dec 15 10:20:52.215343334 (234h51m52.086426369s) [noise=true , address=145.79.227.51:4410 , piece_id=RB5N6MONVOFKSF3GGEQVRL6EGGPMC5UJ77NVPRGEJV7UYUP3XNFA, seg_pos={0 0}, upload=205925]: Done uploading piece. err: <nil> Dec 15 10:20:52.229549165 (234h51m52.10063221s) [noise=true , address=145.79.227.28:4725 , piece_id=FQ6CLMT5DAYQJEQMYBBI2U63353PWPLJEV4RFHGZCZ7PVB3G4N3Q, seg_pos={0 0}, upload=205925]: Done uploading piece. err: <nil> Dec 15 10:20:52.449423590 (234h51m52.320506635s) [noise=true , address=145.79.227.42:4122 , piece_id=ALJ2YXWR7HT4M53UBTDNYO2EXNOOJCU7KL5NQZCQ6S3PYGSQ3AUQ, seg_pos={0 0}, upload=205925]: Done uploading piece. err: <nil> Dec 15 10:20:52.455660495 (234h51m52.326743531s) [noise=true , address=145.79.227.40:4050 , piece_id=LYBFCWNG53LIAPARIFUQPQBVO2EFCD6DXBR5B6FAWDIFPA7QRPHQ, seg_pos={0 0}, upload=205925]: Done uploading piece. err: <nil> Dec 15 10:20:52.458375175 (234h51m52.32945822s) [noise=true , address=145.79.227.27:4690 , piece_id=YNTNQDEKK4HEPHCDKTWTKGBAIBUW63LZ2X3VUMFW6U75L24CFBHA, seg_pos={0 0}, upload=205925]: Done uploading piece. err: <nil> Dec 15 10:20:52.459049809 (234h51m52.330132854s) [noise=true , address=145.79.227.35:4483 , piece_id=PGLX25C2FYVOGJ5RNV6H7J32UIJZVHH46YJWPHM2RLBGHE3AOD6A, seg_pos={0 0}, upload=205925]: Done uploading piece. err: <nil> Dec 15 10:20:52.479342080 (234h51m52.350425115s) [noise=true , address=145.79.227.8:4329 , piece_id=GIHDKALXLKHSUFFLQSOKSLHXMGMYPOPSXFNWZCI6A7GHJ2KQHEKQ, seg_pos={0 0}, upload=205925]: Done uploading piece. err: <nil> Dec 15 10:20:52.479365885 (234h51m52.35044893s) [seg_pos={0 0}, upload=205925]: Segment reached optimal threshold of 13 pieces. Dec 15 10:20:52.479574001 (234h51m52.350657036s) [noise=true , address=145.79.227.23:4546 , piece_id=OIXWEY7NSIW4WK4QOGKGKYS2YSI6YLFRVHRDCICXUP5CST73PMQQ, seg_pos={0 0}, upload=205925]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZMTFpqfMjeVWXJuvsGdbGkMtGQn6axUZFoU1N9sK6fnbPcMy5): protocol: expected piece hash; context canceled; EOF Dec 15 10:20:52.479584632 (234h51m52.350667667s) [noise=true , address=145.79.227.23:4546 , piece_id=OIXWEY7NSIW4WK4QOGKGKYS2YSI6YLFRVHRDCICXUP5CST73PMQQ, seg_pos={0 0}, upload=205925]: Long tail cancellation due to optimal threshold Dec 15 10:20:52.479595793 (234h51m52.350678838s) [noise=true , address=145.79.227.38:4977 , piece_id=6RYP2ACTJ3HEN3C22WGMXEVLUMBG4HEPC7BYDFWEPNHK5WCKM3HQ, seg_pos={0 0}, upload=205925]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125dJwj1tvBkFrgDCoj3FhpH553yGgjxPuSTGxbrkGeDwdKk6M8): protocol: expected piece hash; context canceled; EOF Dec 15 10:20:52.479606773 (234h51m52.350689809s) [noise=true , address=145.79.227.38:4977 , piece_id=6RYP2ACTJ3HEN3C22WGMXEVLUMBG4HEPC7BYDFWEPNHK5WCKM3HQ, seg_pos={0 0}, upload=205925]: Long tail cancellation due to optimal threshold Dec 15 10:20:52.479787528 (234h51m52.350870563s) [noise=true , address=145.79.227.31:4797 , piece_id=LJTUYNDSGEOLP5GC67MTTHIECLSV3UY6WAT6FBXAEDPWWGMFB7CA, seg_pos={0 0}, upload=205925]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9): protocol: expected piece hash; context canceled; EOF Dec 15 10:20:52.479795373 (234h51m52.350878408s) [noise=true , address=145.79.227.31:4797 , piece_id=LJTUYNDSGEOLP5GC67MTTHIECLSV3UY6WAT6FBXAEDPWWGMFB7CA, seg_pos={0 0}, upload=205925]: Long tail cancellation due to optimal threshold Dec 15 10:20:52.479819700 (234h51m52.350902735s) [seg_pos={0 0}, upload=205925]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 15 10:20:52.479903930 (234h51m52.350986975s) [upload=205925]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 15 10:20:52.479909911 (234h51m52.350992956s) [upload=205925]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 15 10:20:52.884169543 (234h51m52.755252618s) [upload=205925]: Done uploading segments... Dec 15 10:20:52.884199039 (234h51m52.755282104s) [upload=205925]: Upload finished. err: <nil> Duration: 13.254286356s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 8.350968476s trace: https://jaeger.dev.storj.io/trace/4a708d72fc2c52e9 FAIL (2025-12-15 11:03:50 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, Piece ID: EDSOWIAZ4FOKDNU33OIAOTCHZ4ATYSUFABGRXWZIXBLX45ZHXYIQ) context deadline exceeded; piecestore: (Node ID: 138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, Piece ID: 5LZCGCHJVFRE3F4AVYDQAPADEMZBEHGISIBE6UJH6UUIHGR6QOUQ) context deadline exceeded ======LOGS====== Dec 15 10:20:49.101615376 (234h51m48.972698411s) [upload=205925]: Starting upload Dec 15 10:20:49.101674809 (234h51m48.972757854s) [upload=205925]: Done starting upload Dec 15 10:20:49.101724754 (234h51m48.972807809s) [upload=205925]: Uploading segments... Dec 15 10:20:49.101919515 (234h51m48.97300255s) [upload=205925]: Got next segment. Inline: false Dec 15 10:20:49.101952237 (234h51m48.973035272s) [upload=205925]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 15 10:20:49.101960182 (234h51m48.973043217s) [upload=205925]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 15 10:20:49.424060615 (234h51m49.29514369s) [seg_pos={0 0}, upload=205925]: Begin upload segment... Dec 15 10:20:49.424121009 (234h51m49.295204045s) [seg_pos={0 0}, upload=205925]: Done begin upload segment. Dec 15 10:20:49.424161688 (234h51m49.295244733s) [noise=true , address=145.79.227.32:4834 , piece_id=3SYBLOFKGOUJDTQEBKUQ5SATA2HDXBCTBLMX67AFPBOPLASI4U6Q, seg_pos={0 0}, upload=205925]: Uploading piece... Dec 15 10:20:49.424254584 (234h51m49.295337629s) [noise=true , address=145.79.227.42:4122 , piece_id=ALJ2YXWR7HT4M53UBTDNYO2EXNOOJCU7KL5NQZCQ6S3PYGSQ3AUQ, seg_pos={0 0}, upload=205925]: Uploading piece... Dec 15 10:20:49.424905434 (234h51m49.295988479s) [noise=true , address=77.90.31.16:4735 , piece_id=EDSOWIAZ4FOKDNU33OIAOTCHZ4ATYSUFABGRXWZIXBLX45ZHXYIQ, seg_pos={0 0}, upload=205925]: Uploading piece... Dec 15 10:20:49.424915713 (234h51m49.295998758s) [noise=true , address=145.79.227.51:4410 , piece_id=RB5N6MONVOFKSF3GGEQVRL6EGGPMC5UJ77NVPRGEJV7UYUP3XNFA, seg_pos={0 0}, upload=205925]: Uploading piece... Dec 15 10:20:49.424964086 (234h51m49.296047131s) [noise=true , address=77.90.31.14:4662 , piece_id=5LZCGCHJVFRE3F4AVYDQAPADEMZBEHGISIBE6UJH6UUIHGR6QOUQ, seg_pos={0 0}, upload=205925]: Uploading piece... Dec 15 10:20:49.425378545 (234h51m49.29646159s) [noise=true , address=145.79.227.40:4050 , piece_id=LYBFCWNG53LIAPARIFUQPQBVO2EFCD6DXBR5B6FAWDIFPA7QRPHQ, seg_pos={0 0}, upload=205925]: Uploading piece... Dec 15 10:20:49.425429762 (234h51m49.296512807s) [noise=true , address=145.79.227.27:4690 , piece_id=YNTNQDEKK4HEPHCDKTWTKGBAIBUW63LZ2X3VUMFW6U75L24CFBHA, seg_pos={0 0}, upload=205925]: Uploading piece... Dec 15 10:20:49.425828811 (234h51m49.296911846s) [noise=true , address=145.79.227.8:4329 , piece_id=GIHDKALXLKHSUFFLQSOKSLHXMGMYPOPSXFNWZCI6A7GHJ2KQHEKQ, seg_pos={0 0}, upload=205925]: Uploading piece... Dec 15 10:20:49.425977345 (234h51m49.29706039s) [noise=true , address=145.79.227.28:4725 , piece_id=FQ6CLMT5DAYQJEQMYBBI2U63353PWPLJEV4RFHGZCZ7PVB3G4N3Q, seg_pos={0 0}, upload=205925]: Uploading piece... Dec 15 10:20:49.426452569 (234h51m49.297535604s) [noise=true , address=145.79.227.23:4546 , piece_id=OIXWEY7NSIW4WK4QOGKGKYS2YSI6YLFRVHRDCICXUP5CST73PMQQ, seg_pos={0 0}, upload=205925]: Uploading piece... Dec 15 10:20:49.427024318 (234h51m49.298107364s) [noise=true , address=145.79.227.35:4483 , piece_id=PGLX25C2FYVOGJ5RNV6H7J32UIJZVHH46YJWPHM2RLBGHE3AOD6A, seg_pos={0 0}, upload=205925]: Uploading piece... Dec 15 10:20:49.427498542 (234h51m49.298581587s) [noise=true , address=145.79.227.50:4373 , piece_id=WZFK2WQVD5PBH5YZQZJXVZNL5LHQXNAEE6ODWMJKEHW6QKHPYDWQ, seg_pos={0 0}, upload=205925]: Uploading piece... Dec 15 10:20:49.427934261 (234h51m49.299017296s) [noise=true , address=145.79.227.31:4797 , piece_id=LJTUYNDSGEOLP5GC67MTTHIECLSV3UY6WAT6FBXAEDPWWGMFB7CA, seg_pos={0 0}, upload=205925]: Uploading piece... Dec 15 10:20:49.428816932 (234h51m49.299899977s) [noise=true , address=145.79.227.10:4401 , piece_id=RLN5U2PG3IEEGHHHPD3EHAUJKMIQLLMKPGSAH7QADMWSQ3A67F4A, seg_pos={0 0}, upload=205925]: Uploading piece... Dec 15 10:20:49.430214363 (234h51m49.301297408s) [noise=true , address=145.79.227.25:4619 , piece_id=LIP5BYLRZW7E4V2JE3UQNRGLP7VSF5HWUQJAGBX2Z7O5IRAAXXEQ, seg_pos={0 0}, upload=205925]: Uploading piece... Dec 15 10:20:49.434382891 (234h51m49.305465936s) [noise=true , address=145.79.227.38:4977 , piece_id=6RYP2ACTJ3HEN3C22WGMXEVLUMBG4HEPC7BYDFWEPNHK5WCKM3HQ, seg_pos={0 0}, upload=205925]: Uploading piece... Dec 15 10:20:51.173093833 (234h51m51.044176878s) [upload=205925]: Next returned nil segment Dec 15 10:20:51.173109864 (234h51m51.044192909s) [upload=205925]: Waiting for error group managing segments... Dec 15 10:20:51.700311661 (234h51m51.571394706s) [noise=true , address=77.90.31.16:4735 , piece_id=EDSOWIAZ4FOKDNU33OIAOTCHZ4ATYSUFABGRXWZIXBLX45ZHXYIQ, seg_pos={0 0}, upload=205925]: Done uploading piece. err: <nil> Dec 15 10:20:51.865652042 (234h51m51.736735077s) [noise=true , address=77.90.31.14:4662 , piece_id=5LZCGCHJVFRE3F4AVYDQAPADEMZBEHGISIBE6UJH6UUIHGR6QOUQ, seg_pos={0 0}, upload=205925]: Done uploading piece. err: <nil> Dec 15 10:20:52.041004417 (234h51m51.912087453s) [noise=true , address=145.79.227.32:4834 , piece_id=3SYBLOFKGOUJDTQEBKUQ5SATA2HDXBCTBLMX67AFPBOPLASI4U6Q, seg_pos={0 0}, upload=205925]: Done uploading piece. err: <nil> Dec 15 10:20:52.041882921 (234h51m51.912965966s) [noise=true , address=145.79.227.10:4401 , piece_id=RLN5U2PG3IEEGHHHPD3EHAUJKMIQLLMKPGSAH7QADMWSQ3A67F4A, seg_pos={0 0}, upload=205925]: Done uploading piece. err: <nil> Dec 15 10:20:52.204300587 (234h51m52.075383622s) [noise=true , address=145.79.227.50:4373 , piece_id=WZFK2WQVD5PBH5YZQZJXVZNL5LHQXNAEE6ODWMJKEHW6QKHPYDWQ, seg_pos={0 0}, upload=205925]: Done uploading piece. err: <nil> Dec 15 10:20:52.215017794 (234h51m52.086100839s) [noise=true , address=145.79.227.25:4619 , piece_id=LIP5BYLRZW7E4V2JE3UQNRGLP7VSF5HWUQJAGBX2Z7O5IRAAXXEQ, seg_pos={0 0}, upload=205925]: Done uploading piece. err: <nil> Dec 15 10:20:52.215343334 (234h51m52.086426369s) [noise=true , address=145.79.227.51:4410 , piece_id=RB5N6MONVOFKSF3GGEQVRL6EGGPMC5UJ77NVPRGEJV7UYUP3XNFA, seg_pos={0 0}, upload=205925]: Done uploading piece. err: <nil> Dec 15 10:20:52.229549165 (234h51m52.10063221s) [noise=true , address=145.79.227.28:4725 , piece_id=FQ6CLMT5DAYQJEQMYBBI2U63353PWPLJEV4RFHGZCZ7PVB3G4N3Q, seg_pos={0 0}, upload=205925]: Done uploading piece. err: <nil> Dec 15 10:20:52.449423590 (234h51m52.320506635s) [noise=true , address=145.79.227.42:4122 , piece_id=ALJ2YXWR7HT4M53UBTDNYO2EXNOOJCU7KL5NQZCQ6S3PYGSQ3AUQ, seg_pos={0 0}, upload=205925]: Done uploading piece. err: <nil> Dec 15 10:20:52.455660495 (234h51m52.326743531s) [noise=true , address=145.79.227.40:4050 , piece_id=LYBFCWNG53LIAPARIFUQPQBVO2EFCD6DXBR5B6FAWDIFPA7QRPHQ, seg_pos={0 0}, upload=205925]: Done uploading piece. err: <nil> Dec 15 10:20:52.458375175 (234h51m52.32945822s) [noise=true , address=145.79.227.27:4690 , piece_id=YNTNQDEKK4HEPHCDKTWTKGBAIBUW63LZ2X3VUMFW6U75L24CFBHA, seg_pos={0 0}, upload=205925]: Done uploading piece. err: <nil> Dec 15 10:20:52.459049809 (234h51m52.330132854s) [noise=true , address=145.79.227.35:4483 , piece_id=PGLX25C2FYVOGJ5RNV6H7J32UIJZVHH46YJWPHM2RLBGHE3AOD6A, seg_pos={0 0}, upload=205925]: Done uploading piece. err: <nil> Dec 15 10:20:52.479342080 (234h51m52.350425115s) [noise=true , address=145.79.227.8:4329 , piece_id=GIHDKALXLKHSUFFLQSOKSLHXMGMYPOPSXFNWZCI6A7GHJ2KQHEKQ, seg_pos={0 0}, upload=205925]: Done uploading piece. err: <nil> Dec 15 10:20:52.479365885 (234h51m52.35044893s) [seg_pos={0 0}, upload=205925]: Segment reached optimal threshold of 13 pieces. Dec 15 10:20:52.479574001 (234h51m52.350657036s) [noise=true , address=145.79.227.23:4546 , piece_id=OIXWEY7NSIW4WK4QOGKGKYS2YSI6YLFRVHRDCICXUP5CST73PMQQ, seg_pos={0 0}, upload=205925]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZMTFpqfMjeVWXJuvsGdbGkMtGQn6axUZFoU1N9sK6fnbPcMy5): protocol: expected piece hash; context canceled; EOF Dec 15 10:20:52.479584632 (234h51m52.350667667s) [noise=true , address=145.79.227.23:4546 , piece_id=OIXWEY7NSIW4WK4QOGKGKYS2YSI6YLFRVHRDCICXUP5CST73PMQQ, seg_pos={0 0}, upload=205925]: Long tail cancellation due to optimal threshold Dec 15 10:20:52.479595793 (234h51m52.350678838s) [noise=true , address=145.79.227.38:4977 , piece_id=6RYP2ACTJ3HEN3C22WGMXEVLUMBG4HEPC7BYDFWEPNHK5WCKM3HQ, seg_pos={0 0}, upload=205925]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125dJwj1tvBkFrgDCoj3FhpH553yGgjxPuSTGxbrkGeDwdKk6M8): protocol: expected piece hash; context canceled; EOF Dec 15 10:20:52.479606773 (234h51m52.350689809s) [noise=true , address=145.79.227.38:4977 , piece_id=6RYP2ACTJ3HEN3C22WGMXEVLUMBG4HEPC7BYDFWEPNHK5WCKM3HQ, seg_pos={0 0}, upload=205925]: Long tail cancellation due to optimal threshold Dec 15 10:20:52.479787528 (234h51m52.350870563s) [noise=true , address=145.79.227.31:4797 , piece_id=LJTUYNDSGEOLP5GC67MTTHIECLSV3UY6WAT6FBXAEDPWWGMFB7CA, seg_pos={0 0}, upload=205925]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9): protocol: expected piece hash; context canceled; EOF Dec 15 10:20:52.479795373 (234h51m52.350878408s) [noise=true , address=145.79.227.31:4797 , piece_id=LJTUYNDSGEOLP5GC67MTTHIECLSV3UY6WAT6FBXAEDPWWGMFB7CA, seg_pos={0 0}, upload=205925]: Long tail cancellation due to optimal threshold Dec 15 10:20:52.479819700 (234h51m52.350902735s) [seg_pos={0 0}, upload=205925]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 15 10:20:52.479903930 (234h51m52.350986975s) [upload=205925]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 15 10:20:52.479909911 (234h51m52.350992956s) [upload=205925]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 15 10:20:52.884169543 (234h51m52.755252618s) [upload=205925]: Done uploading segments... Dec 15 10:20:52.884199039 (234h51m52.755282104s) [upload=205925]: Upload finished. err: <nil> Duration: 8.350999335s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.......................................................X....] duration: 3.642411039s trace: https://jaeger.dev.storj.io/trace/55752dded9459114 FAIL (2025-12-15 11:03:44 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.642440446s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 11.471305103s trace: https://jaeger.dev.storj.io/trace/4a708d72fc2c52e9 FAIL (2025-12-15 11:03:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, Piece ID: EDSOWIAZ4FOKDNU33OIAOTCHZ4ATYSUFABGRXWZIXBLX45ZHXYIQ) context deadline exceeded; piecestore: (Node ID: 138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, Piece ID: 5LZCGCHJVFRE3F4AVYDQAPADEMZBEHGISIBE6UJH6UUIHGR6QOUQ) context deadline exceeded ======LOGS====== Dec 15 10:20:49.101615376 (234h51m48.972698411s) [upload=205925]: Starting upload Dec 15 10:20:49.101674809 (234h51m48.972757854s) [upload=205925]: Done starting upload Dec 15 10:20:49.101724754 (234h51m48.972807809s) [upload=205925]: Uploading segments... Dec 15 10:20:49.101919515 (234h51m48.97300255s) [upload=205925]: Got next segment. Inline: false Dec 15 10:20:49.101952237 (234h51m48.973035272s) [upload=205925]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 15 10:20:49.101960182 (234h51m48.973043217s) [upload=205925]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 15 10:20:49.424060615 (234h51m49.29514369s) [seg_pos={0 0}, upload=205925]: Begin upload segment... Dec 15 10:20:49.424121009 (234h51m49.295204045s) [seg_pos={0 0}, upload=205925]: Done begin upload segment. Dec 15 10:20:49.424161688 (234h51m49.295244733s) [noise=true , address=145.79.227.32:4834 , piece_id=3SYBLOFKGOUJDTQEBKUQ5SATA2HDXBCTBLMX67AFPBOPLASI4U6Q, seg_pos={0 0}, upload=205925]: Uploading piece... Dec 15 10:20:49.424254584 (234h51m49.295337629s) [noise=true , address=145.79.227.42:4122 , piece_id=ALJ2YXWR7HT4M53UBTDNYO2EXNOOJCU7KL5NQZCQ6S3PYGSQ3AUQ, seg_pos={0 0}, upload=205925]: Uploading piece... Dec 15 10:20:49.424905434 (234h51m49.295988479s) [noise=true , address=77.90.31.16:4735 , piece_id=EDSOWIAZ4FOKDNU33OIAOTCHZ4ATYSUFABGRXWZIXBLX45ZHXYIQ, seg_pos={0 0}, upload=205925]: Uploading piece... Dec 15 10:20:49.424915713 (234h51m49.295998758s) [noise=true , address=145.79.227.51:4410 , piece_id=RB5N6MONVOFKSF3GGEQVRL6EGGPMC5UJ77NVPRGEJV7UYUP3XNFA, seg_pos={0 0}, upload=205925]: Uploading piece... Dec 15 10:20:49.424964086 (234h51m49.296047131s) [noise=true , address=77.90.31.14:4662 , piece_id=5LZCGCHJVFRE3F4AVYDQAPADEMZBEHGISIBE6UJH6UUIHGR6QOUQ, seg_pos={0 0}, upload=205925]: Uploading piece... Dec 15 10:20:49.425378545 (234h51m49.29646159s) [noise=true , address=145.79.227.40:4050 , piece_id=LYBFCWNG53LIAPARIFUQPQBVO2EFCD6DXBR5B6FAWDIFPA7QRPHQ, seg_pos={0 0}, upload=205925]: Uploading piece... Dec 15 10:20:49.425429762 (234h51m49.296512807s) [noise=true , address=145.79.227.27:4690 , piece_id=YNTNQDEKK4HEPHCDKTWTKGBAIBUW63LZ2X3VUMFW6U75L24CFBHA, seg_pos={0 0}, upload=205925]: Uploading piece... Dec 15 10:20:49.425828811 (234h51m49.296911846s) [noise=true , address=145.79.227.8:4329 , piece_id=GIHDKALXLKHSUFFLQSOKSLHXMGMYPOPSXFNWZCI6A7GHJ2KQHEKQ, seg_pos={0 0}, upload=205925]: Uploading piece... Dec 15 10:20:49.425977345 (234h51m49.29706039s) [noise=true , address=145.79.227.28:4725 , piece_id=FQ6CLMT5DAYQJEQMYBBI2U63353PWPLJEV4RFHGZCZ7PVB3G4N3Q, seg_pos={0 0}, upload=205925]: Uploading piece... Dec 15 10:20:49.426452569 (234h51m49.297535604s) [noise=true , address=145.79.227.23:4546 , piece_id=OIXWEY7NSIW4WK4QOGKGKYS2YSI6YLFRVHRDCICXUP5CST73PMQQ, seg_pos={0 0}, upload=205925]: Uploading piece... Dec 15 10:20:49.427024318 (234h51m49.298107364s) [noise=true , address=145.79.227.35:4483 , piece_id=PGLX25C2FYVOGJ5RNV6H7J32UIJZVHH46YJWPHM2RLBGHE3AOD6A, seg_pos={0 0}, upload=205925]: Uploading piece... Dec 15 10:20:49.427498542 (234h51m49.298581587s) [noise=true , address=145.79.227.50:4373 , piece_id=WZFK2WQVD5PBH5YZQZJXVZNL5LHQXNAEE6ODWMJKEHW6QKHPYDWQ, seg_pos={0 0}, upload=205925]: Uploading piece... Dec 15 10:20:49.427934261 (234h51m49.299017296s) [noise=true , address=145.79.227.31:4797 , piece_id=LJTUYNDSGEOLP5GC67MTTHIECLSV3UY6WAT6FBXAEDPWWGMFB7CA, seg_pos={0 0}, upload=205925]: Uploading piece... Dec 15 10:20:49.428816932 (234h51m49.299899977s) [noise=true , address=145.79.227.10:4401 , piece_id=RLN5U2PG3IEEGHHHPD3EHAUJKMIQLLMKPGSAH7QADMWSQ3A67F4A, seg_pos={0 0}, upload=205925]: Uploading piece... Dec 15 10:20:49.430214363 (234h51m49.301297408s) [noise=true , address=145.79.227.25:4619 , piece_id=LIP5BYLRZW7E4V2JE3UQNRGLP7VSF5HWUQJAGBX2Z7O5IRAAXXEQ, seg_pos={0 0}, upload=205925]: Uploading piece... Dec 15 10:20:49.434382891 (234h51m49.305465936s) [noise=true , address=145.79.227.38:4977 , piece_id=6RYP2ACTJ3HEN3C22WGMXEVLUMBG4HEPC7BYDFWEPNHK5WCKM3HQ, seg_pos={0 0}, upload=205925]: Uploading piece... Dec 15 10:20:51.173093833 (234h51m51.044176878s) [upload=205925]: Next returned nil segment Dec 15 10:20:51.173109864 (234h51m51.044192909s) [upload=205925]: Waiting for error group managing segments... Dec 15 10:20:51.700311661 (234h51m51.571394706s) [noise=true , address=77.90.31.16:4735 , piece_id=EDSOWIAZ4FOKDNU33OIAOTCHZ4ATYSUFABGRXWZIXBLX45ZHXYIQ, seg_pos={0 0}, upload=205925]: Done uploading piece. err: <nil> Dec 15 10:20:51.865652042 (234h51m51.736735077s) [noise=true , address=77.90.31.14:4662 , piece_id=5LZCGCHJVFRE3F4AVYDQAPADEMZBEHGISIBE6UJH6UUIHGR6QOUQ, seg_pos={0 0}, upload=205925]: Done uploading piece. err: <nil> Dec 15 10:20:52.041004417 (234h51m51.912087453s) [noise=true , address=145.79.227.32:4834 , piece_id=3SYBLOFKGOUJDTQEBKUQ5SATA2HDXBCTBLMX67AFPBOPLASI4U6Q, seg_pos={0 0}, upload=205925]: Done uploading piece. err: <nil> Dec 15 10:20:52.041882921 (234h51m51.912965966s) [noise=true , address=145.79.227.10:4401 , piece_id=RLN5U2PG3IEEGHHHPD3EHAUJKMIQLLMKPGSAH7QADMWSQ3A67F4A, seg_pos={0 0}, upload=205925]: Done uploading piece. err: <nil> Dec 15 10:20:52.204300587 (234h51m52.075383622s) [noise=true , address=145.79.227.50:4373 , piece_id=WZFK2WQVD5PBH5YZQZJXVZNL5LHQXNAEE6ODWMJKEHW6QKHPYDWQ, seg_pos={0 0}, upload=205925]: Done uploading piece. err: <nil> Dec 15 10:20:52.215017794 (234h51m52.086100839s) [noise=true , address=145.79.227.25:4619 , piece_id=LIP5BYLRZW7E4V2JE3UQNRGLP7VSF5HWUQJAGBX2Z7O5IRAAXXEQ, seg_pos={0 0}, upload=205925]: Done uploading piece. err: <nil> Dec 15 10:20:52.215343334 (234h51m52.086426369s) [noise=true , address=145.79.227.51:4410 , piece_id=RB5N6MONVOFKSF3GGEQVRL6EGGPMC5UJ77NVPRGEJV7UYUP3XNFA, seg_pos={0 0}, upload=205925]: Done uploading piece. err: <nil> Dec 15 10:20:52.229549165 (234h51m52.10063221s) [noise=true , address=145.79.227.28:4725 , piece_id=FQ6CLMT5DAYQJEQMYBBI2U63353PWPLJEV4RFHGZCZ7PVB3G4N3Q, seg_pos={0 0}, upload=205925]: Done uploading piece. err: <nil> Dec 15 10:20:52.449423590 (234h51m52.320506635s) [noise=true , address=145.79.227.42:4122 , piece_id=ALJ2YXWR7HT4M53UBTDNYO2EXNOOJCU7KL5NQZCQ6S3PYGSQ3AUQ, seg_pos={0 0}, upload=205925]: Done uploading piece. err: <nil> Dec 15 10:20:52.455660495 (234h51m52.326743531s) [noise=true , address=145.79.227.40:4050 , piece_id=LYBFCWNG53LIAPARIFUQPQBVO2EFCD6DXBR5B6FAWDIFPA7QRPHQ, seg_pos={0 0}, upload=205925]: Done uploading piece. err: <nil> Dec 15 10:20:52.458375175 (234h51m52.32945822s) [noise=true , address=145.79.227.27:4690 , piece_id=YNTNQDEKK4HEPHCDKTWTKGBAIBUW63LZ2X3VUMFW6U75L24CFBHA, seg_pos={0 0}, upload=205925]: Done uploading piece. err: <nil> Dec 15 10:20:52.459049809 (234h51m52.330132854s) [noise=true , address=145.79.227.35:4483 , piece_id=PGLX25C2FYVOGJ5RNV6H7J32UIJZVHH46YJWPHM2RLBGHE3AOD6A, seg_pos={0 0}, upload=205925]: Done uploading piece. err: <nil> Dec 15 10:20:52.479342080 (234h51m52.350425115s) [noise=true , address=145.79.227.8:4329 , piece_id=GIHDKALXLKHSUFFLQSOKSLHXMGMYPOPSXFNWZCI6A7GHJ2KQHEKQ, seg_pos={0 0}, upload=205925]: Done uploading piece. err: <nil> Dec 15 10:20:52.479365885 (234h51m52.35044893s) [seg_pos={0 0}, upload=205925]: Segment reached optimal threshold of 13 pieces. Dec 15 10:20:52.479574001 (234h51m52.350657036s) [noise=true , address=145.79.227.23:4546 , piece_id=OIXWEY7NSIW4WK4QOGKGKYS2YSI6YLFRVHRDCICXUP5CST73PMQQ, seg_pos={0 0}, upload=205925]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZMTFpqfMjeVWXJuvsGdbGkMtGQn6axUZFoU1N9sK6fnbPcMy5): protocol: expected piece hash; context canceled; EOF Dec 15 10:20:52.479584632 (234h51m52.350667667s) [noise=true , address=145.79.227.23:4546 , piece_id=OIXWEY7NSIW4WK4QOGKGKYS2YSI6YLFRVHRDCICXUP5CST73PMQQ, seg_pos={0 0}, upload=205925]: Long tail cancellation due to optimal threshold Dec 15 10:20:52.479595793 (234h51m52.350678838s) [noise=true , address=145.79.227.38:4977 , piece_id=6RYP2ACTJ3HEN3C22WGMXEVLUMBG4HEPC7BYDFWEPNHK5WCKM3HQ, seg_pos={0 0}, upload=205925]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125dJwj1tvBkFrgDCoj3FhpH553yGgjxPuSTGxbrkGeDwdKk6M8): protocol: expected piece hash; context canceled; EOF Dec 15 10:20:52.479606773 (234h51m52.350689809s) [noise=true , address=145.79.227.38:4977 , piece_id=6RYP2ACTJ3HEN3C22WGMXEVLUMBG4HEPC7BYDFWEPNHK5WCKM3HQ, seg_pos={0 0}, upload=205925]: Long tail cancellation due to optimal threshold Dec 15 10:20:52.479787528 (234h51m52.350870563s) [noise=true , address=145.79.227.31:4797 , piece_id=LJTUYNDSGEOLP5GC67MTTHIECLSV3UY6WAT6FBXAEDPWWGMFB7CA, seg_pos={0 0}, upload=205925]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9): protocol: expected piece hash; context canceled; EOF Dec 15 10:20:52.479795373 (234h51m52.350878408s) [noise=true , address=145.79.227.31:4797 , piece_id=LJTUYNDSGEOLP5GC67MTTHIECLSV3UY6WAT6FBXAEDPWWGMFB7CA, seg_pos={0 0}, upload=205925]: Long tail cancellation due to optimal threshold Dec 15 10:20:52.479819700 (234h51m52.350902735s) [seg_pos={0 0}, upload=205925]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 15 10:20:52.479903930 (234h51m52.350986975s) [upload=205925]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 15 10:20:52.479909911 (234h51m52.350992956s) [upload=205925]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 15 10:20:52.884169543 (234h51m52.755252618s) [upload=205925]: Done uploading segments... Dec 15 10:20:52.884199039 (234h51m52.755282104s) [upload=205925]: Upload finished. err: <nil> Duration: 11.471333787s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [........................................................X...] duration: 5.000762907s trace: https://jaeger.dev.storj.io/trace/55752dded9459114 FAIL (2025-12-15 11:02:32 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000834583s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 8.584768415s trace: https://jaeger.dev.storj.io/trace/4a708d72fc2c52e9 FAIL (2025-12-15 11:02:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, Piece ID: EDSOWIAZ4FOKDNU33OIAOTCHZ4ATYSUFABGRXWZIXBLX45ZHXYIQ) context deadline exceeded; piecestore: (Node ID: 138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, Piece ID: 5LZCGCHJVFRE3F4AVYDQAPADEMZBEHGISIBE6UJH6UUIHGR6QOUQ) context deadline exceeded ======LOGS====== Dec 15 10:20:49.101615376 (234h51m48.972698411s) [upload=205925]: Starting upload Dec 15 10:20:49.101674809 (234h51m48.972757854s) [upload=205925]: Done starting upload Dec 15 10:20:49.101724754 (234h51m48.972807809s) [upload=205925]: Uploading segments... Dec 15 10:20:49.101919515 (234h51m48.97300255s) [upload=205925]: Got next segment. Inline: false Dec 15 10:20:49.101952237 (234h51m48.973035272s) [upload=205925]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 15 10:20:49.101960182 (234h51m48.973043217s) [upload=205925]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 15 10:20:49.424060615 (234h51m49.29514369s) [seg_pos={0 0}, upload=205925]: Begin upload segment... Dec 15 10:20:49.424121009 (234h51m49.295204045s) [seg_pos={0 0}, upload=205925]: Done begin upload segment. Dec 15 10:20:49.424161688 (234h51m49.295244733s) [noise=true , address=145.79.227.32:4834 , piece_id=3SYBLOFKGOUJDTQEBKUQ5SATA2HDXBCTBLMX67AFPBOPLASI4U6Q, seg_pos={0 0}, upload=205925]: Uploading piece... Dec 15 10:20:49.424254584 (234h51m49.295337629s) [noise=true , address=145.79.227.42:4122 , piece_id=ALJ2YXWR7HT4M53UBTDNYO2EXNOOJCU7KL5NQZCQ6S3PYGSQ3AUQ, seg_pos={0 0}, upload=205925]: Uploading piece... Dec 15 10:20:49.424905434 (234h51m49.295988479s) [noise=true , address=77.90.31.16:4735 , piece_id=EDSOWIAZ4FOKDNU33OIAOTCHZ4ATYSUFABGRXWZIXBLX45ZHXYIQ, seg_pos={0 0}, upload=205925]: Uploading piece... Dec 15 10:20:49.424915713 (234h51m49.295998758s) [noise=true , address=145.79.227.51:4410 , piece_id=RB5N6MONVOFKSF3GGEQVRL6EGGPMC5UJ77NVPRGEJV7UYUP3XNFA, seg_pos={0 0}, upload=205925]: Uploading piece... Dec 15 10:20:49.424964086 (234h51m49.296047131s) [noise=true , address=77.90.31.14:4662 , piece_id=5LZCGCHJVFRE3F4AVYDQAPADEMZBEHGISIBE6UJH6UUIHGR6QOUQ, seg_pos={0 0}, upload=205925]: Uploading piece... Dec 15 10:20:49.425378545 (234h51m49.29646159s) [noise=true , address=145.79.227.40:4050 , piece_id=LYBFCWNG53LIAPARIFUQPQBVO2EFCD6DXBR5B6FAWDIFPA7QRPHQ, seg_pos={0 0}, upload=205925]: Uploading piece... Dec 15 10:20:49.425429762 (234h51m49.296512807s) [noise=true , address=145.79.227.27:4690 , piece_id=YNTNQDEKK4HEPHCDKTWTKGBAIBUW63LZ2X3VUMFW6U75L24CFBHA, seg_pos={0 0}, upload=205925]: Uploading piece... Dec 15 10:20:49.425828811 (234h51m49.296911846s) [noise=true , address=145.79.227.8:4329 , piece_id=GIHDKALXLKHSUFFLQSOKSLHXMGMYPOPSXFNWZCI6A7GHJ2KQHEKQ, seg_pos={0 0}, upload=205925]: Uploading piece... Dec 15 10:20:49.425977345 (234h51m49.29706039s) [noise=true , address=145.79.227.28:4725 , piece_id=FQ6CLMT5DAYQJEQMYBBI2U63353PWPLJEV4RFHGZCZ7PVB3G4N3Q, seg_pos={0 0}, upload=205925]: Uploading piece... Dec 15 10:20:49.426452569 (234h51m49.297535604s) [noise=true , address=145.79.227.23:4546 , piece_id=OIXWEY7NSIW4WK4QOGKGKYS2YSI6YLFRVHRDCICXUP5CST73PMQQ, seg_pos={0 0}, upload=205925]: Uploading piece... Dec 15 10:20:49.427024318 (234h51m49.298107364s) [noise=true , address=145.79.227.35:4483 , piece_id=PGLX25C2FYVOGJ5RNV6H7J32UIJZVHH46YJWPHM2RLBGHE3AOD6A, seg_pos={0 0}, upload=205925]: Uploading piece... Dec 15 10:20:49.427498542 (234h51m49.298581587s) [noise=true , address=145.79.227.50:4373 , piece_id=WZFK2WQVD5PBH5YZQZJXVZNL5LHQXNAEE6ODWMJKEHW6QKHPYDWQ, seg_pos={0 0}, upload=205925]: Uploading piece... Dec 15 10:20:49.427934261 (234h51m49.299017296s) [noise=true , address=145.79.227.31:4797 , piece_id=LJTUYNDSGEOLP5GC67MTTHIECLSV3UY6WAT6FBXAEDPWWGMFB7CA, seg_pos={0 0}, upload=205925]: Uploading piece... Dec 15 10:20:49.428816932 (234h51m49.299899977s) [noise=true , address=145.79.227.10:4401 , piece_id=RLN5U2PG3IEEGHHHPD3EHAUJKMIQLLMKPGSAH7QADMWSQ3A67F4A, seg_pos={0 0}, upload=205925]: Uploading piece... Dec 15 10:20:49.430214363 (234h51m49.301297408s) [noise=true , address=145.79.227.25:4619 , piece_id=LIP5BYLRZW7E4V2JE3UQNRGLP7VSF5HWUQJAGBX2Z7O5IRAAXXEQ, seg_pos={0 0}, upload=205925]: Uploading piece... Dec 15 10:20:49.434382891 (234h51m49.305465936s) [noise=true , address=145.79.227.38:4977 , piece_id=6RYP2ACTJ3HEN3C22WGMXEVLUMBG4HEPC7BYDFWEPNHK5WCKM3HQ, seg_pos={0 0}, upload=205925]: Uploading piece... Dec 15 10:20:51.173093833 (234h51m51.044176878s) [upload=205925]: Next returned nil segment Dec 15 10:20:51.173109864 (234h51m51.044192909s) [upload=205925]: Waiting for error group managing segments... Dec 15 10:20:51.700311661 (234h51m51.571394706s) [noise=true , address=77.90.31.16:4735 , piece_id=EDSOWIAZ4FOKDNU33OIAOTCHZ4ATYSUFABGRXWZIXBLX45ZHXYIQ, seg_pos={0 0}, upload=205925]: Done uploading piece. err: <nil> Dec 15 10:20:51.865652042 (234h51m51.736735077s) [noise=true , address=77.90.31.14:4662 , piece_id=5LZCGCHJVFRE3F4AVYDQAPADEMZBEHGISIBE6UJH6UUIHGR6QOUQ, seg_pos={0 0}, upload=205925]: Done uploading piece. err: <nil> Dec 15 10:20:52.041004417 (234h51m51.912087453s) [noise=true , address=145.79.227.32:4834 , piece_id=3SYBLOFKGOUJDTQEBKUQ5SATA2HDXBCTBLMX67AFPBOPLASI4U6Q, seg_pos={0 0}, upload=205925]: Done uploading piece. err: <nil> Dec 15 10:20:52.041882921 (234h51m51.912965966s) [noise=true , address=145.79.227.10:4401 , piece_id=RLN5U2PG3IEEGHHHPD3EHAUJKMIQLLMKPGSAH7QADMWSQ3A67F4A, seg_pos={0 0}, upload=205925]: Done uploading piece. err: <nil> Dec 15 10:20:52.204300587 (234h51m52.075383622s) [noise=true , address=145.79.227.50:4373 , piece_id=WZFK2WQVD5PBH5YZQZJXVZNL5LHQXNAEE6ODWMJKEHW6QKHPYDWQ, seg_pos={0 0}, upload=205925]: Done uploading piece. err: <nil> Dec 15 10:20:52.215017794 (234h51m52.086100839s) [noise=true , address=145.79.227.25:4619 , piece_id=LIP5BYLRZW7E4V2JE3UQNRGLP7VSF5HWUQJAGBX2Z7O5IRAAXXEQ, seg_pos={0 0}, upload=205925]: Done uploading piece. err: <nil> Dec 15 10:20:52.215343334 (234h51m52.086426369s) [noise=true , address=145.79.227.51:4410 , piece_id=RB5N6MONVOFKSF3GGEQVRL6EGGPMC5UJ77NVPRGEJV7UYUP3XNFA, seg_pos={0 0}, upload=205925]: Done uploading piece. err: <nil> Dec 15 10:20:52.229549165 (234h51m52.10063221s) [noise=true , address=145.79.227.28:4725 , piece_id=FQ6CLMT5DAYQJEQMYBBI2U63353PWPLJEV4RFHGZCZ7PVB3G4N3Q, seg_pos={0 0}, upload=205925]: Done uploading piece. err: <nil> Dec 15 10:20:52.449423590 (234h51m52.320506635s) [noise=true , address=145.79.227.42:4122 , piece_id=ALJ2YXWR7HT4M53UBTDNYO2EXNOOJCU7KL5NQZCQ6S3PYGSQ3AUQ, seg_pos={0 0}, upload=205925]: Done uploading piece. err: <nil> Dec 15 10:20:52.455660495 (234h51m52.326743531s) [noise=true , address=145.79.227.40:4050 , piece_id=LYBFCWNG53LIAPARIFUQPQBVO2EFCD6DXBR5B6FAWDIFPA7QRPHQ, seg_pos={0 0}, upload=205925]: Done uploading piece. err: <nil> Dec 15 10:20:52.458375175 (234h51m52.32945822s) [noise=true , address=145.79.227.27:4690 , piece_id=YNTNQDEKK4HEPHCDKTWTKGBAIBUW63LZ2X3VUMFW6U75L24CFBHA, seg_pos={0 0}, upload=205925]: Done uploading piece. err: <nil> Dec 15 10:20:52.459049809 (234h51m52.330132854s) [noise=true , address=145.79.227.35:4483 , piece_id=PGLX25C2FYVOGJ5RNV6H7J32UIJZVHH46YJWPHM2RLBGHE3AOD6A, seg_pos={0 0}, upload=205925]: Done uploading piece. err: <nil> Dec 15 10:20:52.479342080 (234h51m52.350425115s) [noise=true , address=145.79.227.8:4329 , piece_id=GIHDKALXLKHSUFFLQSOKSLHXMGMYPOPSXFNWZCI6A7GHJ2KQHEKQ, seg_pos={0 0}, upload=205925]: Done uploading piece. err: <nil> Dec 15 10:20:52.479365885 (234h51m52.35044893s) [seg_pos={0 0}, upload=205925]: Segment reached optimal threshold of 13 pieces. Dec 15 10:20:52.479574001 (234h51m52.350657036s) [noise=true , address=145.79.227.23:4546 , piece_id=OIXWEY7NSIW4WK4QOGKGKYS2YSI6YLFRVHRDCICXUP5CST73PMQQ, seg_pos={0 0}, upload=205925]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZMTFpqfMjeVWXJuvsGdbGkMtGQn6axUZFoU1N9sK6fnbPcMy5): protocol: expected piece hash; context canceled; EOF Dec 15 10:20:52.479584632 (234h51m52.350667667s) [noise=true , address=145.79.227.23:4546 , piece_id=OIXWEY7NSIW4WK4QOGKGKYS2YSI6YLFRVHRDCICXUP5CST73PMQQ, seg_pos={0 0}, upload=205925]: Long tail cancellation due to optimal threshold Dec 15 10:20:52.479595793 (234h51m52.350678838s) [noise=true , address=145.79.227.38:4977 , piece_id=6RYP2ACTJ3HEN3C22WGMXEVLUMBG4HEPC7BYDFWEPNHK5WCKM3HQ, seg_pos={0 0}, upload=205925]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125dJwj1tvBkFrgDCoj3FhpH553yGgjxPuSTGxbrkGeDwdKk6M8): protocol: expected piece hash; context canceled; EOF Dec 15 10:20:52.479606773 (234h51m52.350689809s) [noise=true , address=145.79.227.38:4977 , piece_id=6RYP2ACTJ3HEN3C22WGMXEVLUMBG4HEPC7BYDFWEPNHK5WCKM3HQ, seg_pos={0 0}, upload=205925]: Long tail cancellation due to optimal threshold Dec 15 10:20:52.479787528 (234h51m52.350870563s) [noise=true , address=145.79.227.31:4797 , piece_id=LJTUYNDSGEOLP5GC67MTTHIECLSV3UY6WAT6FBXAEDPWWGMFB7CA, seg_pos={0 0}, upload=205925]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9): protocol: expected piece hash; context canceled; EOF Dec 15 10:20:52.479795373 (234h51m52.350878408s) [noise=true , address=145.79.227.31:4797 , piece_id=LJTUYNDSGEOLP5GC67MTTHIECLSV3UY6WAT6FBXAEDPWWGMFB7CA, seg_pos={0 0}, upload=205925]: Long tail cancellation due to optimal threshold Dec 15 10:20:52.479819700 (234h51m52.350902735s) [seg_pos={0 0}, upload=205925]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 15 10:20:52.479903930 (234h51m52.350986975s) [upload=205925]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 15 10:20:52.479909911 (234h51m52.350992956s) [upload=205925]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 15 10:20:52.884169543 (234h51m52.755252618s) [upload=205925]: Done uploading segments... Dec 15 10:20:52.884199039 (234h51m52.755282104s) [upload=205925]: Upload finished. err: <nil> Duration: 8.58479228s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 8.783535296s trace: https://jaeger.dev.storj.io/trace/4a708d72fc2c52e9 FAIL (2025-12-15 11:01:47 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, Piece ID: EDSOWIAZ4FOKDNU33OIAOTCHZ4ATYSUFABGRXWZIXBLX45ZHXYIQ) context deadline exceeded; piecestore: (Node ID: 138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, Piece ID: 5LZCGCHJVFRE3F4AVYDQAPADEMZBEHGISIBE6UJH6UUIHGR6QOUQ) context deadline exceeded ======LOGS====== Dec 15 10:20:49.101615376 (234h51m48.972698411s) [upload=205925]: Starting upload Dec 15 10:20:49.101674809 (234h51m48.972757854s) [upload=205925]: Done starting upload Dec 15 10:20:49.101724754 (234h51m48.972807809s) [upload=205925]: Uploading segments... Dec 15 10:20:49.101919515 (234h51m48.97300255s) [upload=205925]: Got next segment. Inline: false Dec 15 10:20:49.101952237 (234h51m48.973035272s) [upload=205925]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 15 10:20:49.101960182 (234h51m48.973043217s) [upload=205925]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 15 10:20:49.424060615 (234h51m49.29514369s) [seg_pos={0 0}, upload=205925]: Begin upload segment... Dec 15 10:20:49.424121009 (234h51m49.295204045s) [seg_pos={0 0}, upload=205925]: Done begin upload segment. Dec 15 10:20:49.424161688 (234h51m49.295244733s) [noise=true , address=145.79.227.32:4834 , piece_id=3SYBLOFKGOUJDTQEBKUQ5SATA2HDXBCTBLMX67AFPBOPLASI4U6Q, seg_pos={0 0}, upload=205925]: Uploading piece... Dec 15 10:20:49.424254584 (234h51m49.295337629s) [noise=true , address=145.79.227.42:4122 , piece_id=ALJ2YXWR7HT4M53UBTDNYO2EXNOOJCU7KL5NQZCQ6S3PYGSQ3AUQ, seg_pos={0 0}, upload=205925]: Uploading piece... Dec 15 10:20:49.424905434 (234h51m49.295988479s) [noise=true , address=77.90.31.16:4735 , piece_id=EDSOWIAZ4FOKDNU33OIAOTCHZ4ATYSUFABGRXWZIXBLX45ZHXYIQ, seg_pos={0 0}, upload=205925]: Uploading piece... Dec 15 10:20:49.424915713 (234h51m49.295998758s) [noise=true , address=145.79.227.51:4410 , piece_id=RB5N6MONVOFKSF3GGEQVRL6EGGPMC5UJ77NVPRGEJV7UYUP3XNFA, seg_pos={0 0}, upload=205925]: Uploading piece... Dec 15 10:20:49.424964086 (234h51m49.296047131s) [noise=true , address=77.90.31.14:4662 , piece_id=5LZCGCHJVFRE3F4AVYDQAPADEMZBEHGISIBE6UJH6UUIHGR6QOUQ, seg_pos={0 0}, upload=205925]: Uploading piece... Dec 15 10:20:49.425378545 (234h51m49.29646159s) [noise=true , address=145.79.227.40:4050 , piece_id=LYBFCWNG53LIAPARIFUQPQBVO2EFCD6DXBR5B6FAWDIFPA7QRPHQ, seg_pos={0 0}, upload=205925]: Uploading piece... Dec 15 10:20:49.425429762 (234h51m49.296512807s) [noise=true , address=145.79.227.27:4690 , piece_id=YNTNQDEKK4HEPHCDKTWTKGBAIBUW63LZ2X3VUMFW6U75L24CFBHA, seg_pos={0 0}, upload=205925]: Uploading piece... Dec 15 10:20:49.425828811 (234h51m49.296911846s) [noise=true , address=145.79.227.8:4329 , piece_id=GIHDKALXLKHSUFFLQSOKSLHXMGMYPOPSXFNWZCI6A7GHJ2KQHEKQ, seg_pos={0 0}, upload=205925]: Uploading piece... Dec 15 10:20:49.425977345 (234h51m49.29706039s) [noise=true , address=145.79.227.28:4725 , piece_id=FQ6CLMT5DAYQJEQMYBBI2U63353PWPLJEV4RFHGZCZ7PVB3G4N3Q, seg_pos={0 0}, upload=205925]: Uploading piece... Dec 15 10:20:49.426452569 (234h51m49.297535604s) [noise=true , address=145.79.227.23:4546 , piece_id=OIXWEY7NSIW4WK4QOGKGKYS2YSI6YLFRVHRDCICXUP5CST73PMQQ, seg_pos={0 0}, upload=205925]: Uploading piece... Dec 15 10:20:49.427024318 (234h51m49.298107364s) [noise=true , address=145.79.227.35:4483 , piece_id=PGLX25C2FYVOGJ5RNV6H7J32UIJZVHH46YJWPHM2RLBGHE3AOD6A, seg_pos={0 0}, upload=205925]: Uploading piece... Dec 15 10:20:49.427498542 (234h51m49.298581587s) [noise=true , address=145.79.227.50:4373 , piece_id=WZFK2WQVD5PBH5YZQZJXVZNL5LHQXNAEE6ODWMJKEHW6QKHPYDWQ, seg_pos={0 0}, upload=205925]: Uploading piece... Dec 15 10:20:49.427934261 (234h51m49.299017296s) [noise=true , address=145.79.227.31:4797 , piece_id=LJTUYNDSGEOLP5GC67MTTHIECLSV3UY6WAT6FBXAEDPWWGMFB7CA, seg_pos={0 0}, upload=205925]: Uploading piece... Dec 15 10:20:49.428816932 (234h51m49.299899977s) [noise=true , address=145.79.227.10:4401 , piece_id=RLN5U2PG3IEEGHHHPD3EHAUJKMIQLLMKPGSAH7QADMWSQ3A67F4A, seg_pos={0 0}, upload=205925]: Uploading piece... Dec 15 10:20:49.430214363 (234h51m49.301297408s) [noise=true , address=145.79.227.25:4619 , piece_id=LIP5BYLRZW7E4V2JE3UQNRGLP7VSF5HWUQJAGBX2Z7O5IRAAXXEQ, seg_pos={0 0}, upload=205925]: Uploading piece... Dec 15 10:20:49.434382891 (234h51m49.305465936s) [noise=true , address=145.79.227.38:4977 , piece_id=6RYP2ACTJ3HEN3C22WGMXEVLUMBG4HEPC7BYDFWEPNHK5WCKM3HQ, seg_pos={0 0}, upload=205925]: Uploading piece... Dec 15 10:20:51.173093833 (234h51m51.044176878s) [upload=205925]: Next returned nil segment Dec 15 10:20:51.173109864 (234h51m51.044192909s) [upload=205925]: Waiting for error group managing segments... Dec 15 10:20:51.700311661 (234h51m51.571394706s) [noise=true , address=77.90.31.16:4735 , piece_id=EDSOWIAZ4FOKDNU33OIAOTCHZ4ATYSUFABGRXWZIXBLX45ZHXYIQ, seg_pos={0 0}, upload=205925]: Done uploading piece. err: <nil> Dec 15 10:20:51.865652042 (234h51m51.736735077s) [noise=true , address=77.90.31.14:4662 , piece_id=5LZCGCHJVFRE3F4AVYDQAPADEMZBEHGISIBE6UJH6UUIHGR6QOUQ, seg_pos={0 0}, upload=205925]: Done uploading piece. err: <nil> Dec 15 10:20:52.041004417 (234h51m51.912087453s) [noise=true , address=145.79.227.32:4834 , piece_id=3SYBLOFKGOUJDTQEBKUQ5SATA2HDXBCTBLMX67AFPBOPLASI4U6Q, seg_pos={0 0}, upload=205925]: Done uploading piece. err: <nil> Dec 15 10:20:52.041882921 (234h51m51.912965966s) [noise=true , address=145.79.227.10:4401 , piece_id=RLN5U2PG3IEEGHHHPD3EHAUJKMIQLLMKPGSAH7QADMWSQ3A67F4A, seg_pos={0 0}, upload=205925]: Done uploading piece. err: <nil> Dec 15 10:20:52.204300587 (234h51m52.075383622s) [noise=true , address=145.79.227.50:4373 , piece_id=WZFK2WQVD5PBH5YZQZJXVZNL5LHQXNAEE6ODWMJKEHW6QKHPYDWQ, seg_pos={0 0}, upload=205925]: Done uploading piece. err: <nil> Dec 15 10:20:52.215017794 (234h51m52.086100839s) [noise=true , address=145.79.227.25:4619 , piece_id=LIP5BYLRZW7E4V2JE3UQNRGLP7VSF5HWUQJAGBX2Z7O5IRAAXXEQ, seg_pos={0 0}, upload=205925]: Done uploading piece. err: <nil> Dec 15 10:20:52.215343334 (234h51m52.086426369s) [noise=true , address=145.79.227.51:4410 , piece_id=RB5N6MONVOFKSF3GGEQVRL6EGGPMC5UJ77NVPRGEJV7UYUP3XNFA, seg_pos={0 0}, upload=205925]: Done uploading piece. err: <nil> Dec 15 10:20:52.229549165 (234h51m52.10063221s) [noise=true , address=145.79.227.28:4725 , piece_id=FQ6CLMT5DAYQJEQMYBBI2U63353PWPLJEV4RFHGZCZ7PVB3G4N3Q, seg_pos={0 0}, upload=205925]: Done uploading piece. err: <nil> Dec 15 10:20:52.449423590 (234h51m52.320506635s) [noise=true , address=145.79.227.42:4122 , piece_id=ALJ2YXWR7HT4M53UBTDNYO2EXNOOJCU7KL5NQZCQ6S3PYGSQ3AUQ, seg_pos={0 0}, upload=205925]: Done uploading piece. err: <nil> Dec 15 10:20:52.455660495 (234h51m52.326743531s) [noise=true , address=145.79.227.40:4050 , piece_id=LYBFCWNG53LIAPARIFUQPQBVO2EFCD6DXBR5B6FAWDIFPA7QRPHQ, seg_pos={0 0}, upload=205925]: Done uploading piece. err: <nil> Dec 15 10:20:52.458375175 (234h51m52.32945822s) [noise=true , address=145.79.227.27:4690 , piece_id=YNTNQDEKK4HEPHCDKTWTKGBAIBUW63LZ2X3VUMFW6U75L24CFBHA, seg_pos={0 0}, upload=205925]: Done uploading piece. err: <nil> Dec 15 10:20:52.459049809 (234h51m52.330132854s) [noise=true , address=145.79.227.35:4483 , piece_id=PGLX25C2FYVOGJ5RNV6H7J32UIJZVHH46YJWPHM2RLBGHE3AOD6A, seg_pos={0 0}, upload=205925]: Done uploading piece. err: <nil> Dec 15 10:20:52.479342080 (234h51m52.350425115s) [noise=true , address=145.79.227.8:4329 , piece_id=GIHDKALXLKHSUFFLQSOKSLHXMGMYPOPSXFNWZCI6A7GHJ2KQHEKQ, seg_pos={0 0}, upload=205925]: Done uploading piece. err: <nil> Dec 15 10:20:52.479365885 (234h51m52.35044893s) [seg_pos={0 0}, upload=205925]: Segment reached optimal threshold of 13 pieces. Dec 15 10:20:52.479574001 (234h51m52.350657036s) [noise=true , address=145.79.227.23:4546 , piece_id=OIXWEY7NSIW4WK4QOGKGKYS2YSI6YLFRVHRDCICXUP5CST73PMQQ, seg_pos={0 0}, upload=205925]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZMTFpqfMjeVWXJuvsGdbGkMtGQn6axUZFoU1N9sK6fnbPcMy5): protocol: expected piece hash; context canceled; EOF Dec 15 10:20:52.479584632 (234h51m52.350667667s) [noise=true , address=145.79.227.23:4546 , piece_id=OIXWEY7NSIW4WK4QOGKGKYS2YSI6YLFRVHRDCICXUP5CST73PMQQ, seg_pos={0 0}, upload=205925]: Long tail cancellation due to optimal threshold Dec 15 10:20:52.479595793 (234h51m52.350678838s) [noise=true , address=145.79.227.38:4977 , piece_id=6RYP2ACTJ3HEN3C22WGMXEVLUMBG4HEPC7BYDFWEPNHK5WCKM3HQ, seg_pos={0 0}, upload=205925]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125dJwj1tvBkFrgDCoj3FhpH553yGgjxPuSTGxbrkGeDwdKk6M8): protocol: expected piece hash; context canceled; EOF Dec 15 10:20:52.479606773 (234h51m52.350689809s) [noise=true , address=145.79.227.38:4977 , piece_id=6RYP2ACTJ3HEN3C22WGMXEVLUMBG4HEPC7BYDFWEPNHK5WCKM3HQ, seg_pos={0 0}, upload=205925]: Long tail cancellation due to optimal threshold Dec 15 10:20:52.479787528 (234h51m52.350870563s) [noise=true , address=145.79.227.31:4797 , piece_id=LJTUYNDSGEOLP5GC67MTTHIECLSV3UY6WAT6FBXAEDPWWGMFB7CA, seg_pos={0 0}, upload=205925]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9): protocol: expected piece hash; context canceled; EOF Dec 15 10:20:52.479795373 (234h51m52.350878408s) [noise=true , address=145.79.227.31:4797 , piece_id=LJTUYNDSGEOLP5GC67MTTHIECLSV3UY6WAT6FBXAEDPWWGMFB7CA, seg_pos={0 0}, upload=205925]: Long tail cancellation due to optimal threshold Dec 15 10:20:52.479819700 (234h51m52.350902735s) [seg_pos={0 0}, upload=205925]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 15 10:20:52.479903930 (234h51m52.350986975s) [upload=205925]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 15 10:20:52.479909911 (234h51m52.350992956s) [upload=205925]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 15 10:20:52.884169543 (234h51m52.755252618s) [upload=205925]: Done uploading segments... Dec 15 10:20:52.884199039 (234h51m52.755282104s) [upload=205925]: Upload finished. err: <nil> Duration: 8.783565976s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [X........................................................X..] duration: 4.061953318s trace: https://jaeger.dev.storj.io/trace/55752dded9459114 FAIL (2025-12-15 11:01:45 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.062043931s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.X........................................................X.] duration: 4.745940881s trace: https://jaeger.dev.storj.io/trace/55752dded9459114 FAIL (2025-12-15 11:00:45 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.745969798s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 7.733606231s trace: https://jaeger.dev.storj.io/trace/4a708d72fc2c52e9 FAIL (2025-12-15 11:00:39 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, Piece ID: EDSOWIAZ4FOKDNU33OIAOTCHZ4ATYSUFABGRXWZIXBLX45ZHXYIQ) context deadline exceeded; piecestore: (Node ID: 138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, Piece ID: 5LZCGCHJVFRE3F4AVYDQAPADEMZBEHGISIBE6UJH6UUIHGR6QOUQ) context deadline exceeded ======LOGS====== Dec 15 10:20:49.101615376 (234h51m48.972698411s) [upload=205925]: Starting upload Dec 15 10:20:49.101674809 (234h51m48.972757854s) [upload=205925]: Done starting upload Dec 15 10:20:49.101724754 (234h51m48.972807809s) [upload=205925]: Uploading segments... Dec 15 10:20:49.101919515 (234h51m48.97300255s) [upload=205925]: Got next segment. Inline: false Dec 15 10:20:49.101952237 (234h51m48.973035272s) [upload=205925]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 15 10:20:49.101960182 (234h51m48.973043217s) [upload=205925]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 15 10:20:49.424060615 (234h51m49.29514369s) [seg_pos={0 0}, upload=205925]: Begin upload segment... Dec 15 10:20:49.424121009 (234h51m49.295204045s) [seg_pos={0 0}, upload=205925]: Done begin upload segment. Dec 15 10:20:49.424161688 (234h51m49.295244733s) [noise=true , address=145.79.227.32:4834 , piece_id=3SYBLOFKGOUJDTQEBKUQ5SATA2HDXBCTBLMX67AFPBOPLASI4U6Q, seg_pos={0 0}, upload=205925]: Uploading piece... Dec 15 10:20:49.424254584 (234h51m49.295337629s) [noise=true , address=145.79.227.42:4122 , piece_id=ALJ2YXWR7HT4M53UBTDNYO2EXNOOJCU7KL5NQZCQ6S3PYGSQ3AUQ, seg_pos={0 0}, upload=205925]: Uploading piece... Dec 15 10:20:49.424905434 (234h51m49.295988479s) [noise=true , address=77.90.31.16:4735 , piece_id=EDSOWIAZ4FOKDNU33OIAOTCHZ4ATYSUFABGRXWZIXBLX45ZHXYIQ, seg_pos={0 0}, upload=205925]: Uploading piece... Dec 15 10:20:49.424915713 (234h51m49.295998758s) [noise=true , address=145.79.227.51:4410 , piece_id=RB5N6MONVOFKSF3GGEQVRL6EGGPMC5UJ77NVPRGEJV7UYUP3XNFA, seg_pos={0 0}, upload=205925]: Uploading piece... Dec 15 10:20:49.424964086 (234h51m49.296047131s) [noise=true , address=77.90.31.14:4662 , piece_id=5LZCGCHJVFRE3F4AVYDQAPADEMZBEHGISIBE6UJH6UUIHGR6QOUQ, seg_pos={0 0}, upload=205925]: Uploading piece... Dec 15 10:20:49.425378545 (234h51m49.29646159s) [noise=true , address=145.79.227.40:4050 , piece_id=LYBFCWNG53LIAPARIFUQPQBVO2EFCD6DXBR5B6FAWDIFPA7QRPHQ, seg_pos={0 0}, upload=205925]: Uploading piece... Dec 15 10:20:49.425429762 (234h51m49.296512807s) [noise=true , address=145.79.227.27:4690 , piece_id=YNTNQDEKK4HEPHCDKTWTKGBAIBUW63LZ2X3VUMFW6U75L24CFBHA, seg_pos={0 0}, upload=205925]: Uploading piece... Dec 15 10:20:49.425828811 (234h51m49.296911846s) [noise=true , address=145.79.227.8:4329 , piece_id=GIHDKALXLKHSUFFLQSOKSLHXMGMYPOPSXFNWZCI6A7GHJ2KQHEKQ, seg_pos={0 0}, upload=205925]: Uploading piece... Dec 15 10:20:49.425977345 (234h51m49.29706039s) [noise=true , address=145.79.227.28:4725 , piece_id=FQ6CLMT5DAYQJEQMYBBI2U63353PWPLJEV4RFHGZCZ7PVB3G4N3Q, seg_pos={0 0}, upload=205925]: Uploading piece... Dec 15 10:20:49.426452569 (234h51m49.297535604s) [noise=true , address=145.79.227.23:4546 , piece_id=OIXWEY7NSIW4WK4QOGKGKYS2YSI6YLFRVHRDCICXUP5CST73PMQQ, seg_pos={0 0}, upload=205925]: Uploading piece... Dec 15 10:20:49.427024318 (234h51m49.298107364s) [noise=true , address=145.79.227.35:4483 , piece_id=PGLX25C2FYVOGJ5RNV6H7J32UIJZVHH46YJWPHM2RLBGHE3AOD6A, seg_pos={0 0}, upload=205925]: Uploading piece... Dec 15 10:20:49.427498542 (234h51m49.298581587s) [noise=true , address=145.79.227.50:4373 , piece_id=WZFK2WQVD5PBH5YZQZJXVZNL5LHQXNAEE6ODWMJKEHW6QKHPYDWQ, seg_pos={0 0}, upload=205925]: Uploading piece... Dec 15 10:20:49.427934261 (234h51m49.299017296s) [noise=true , address=145.79.227.31:4797 , piece_id=LJTUYNDSGEOLP5GC67MTTHIECLSV3UY6WAT6FBXAEDPWWGMFB7CA, seg_pos={0 0}, upload=205925]: Uploading piece... Dec 15 10:20:49.428816932 (234h51m49.299899977s) [noise=true , address=145.79.227.10:4401 , piece_id=RLN5U2PG3IEEGHHHPD3EHAUJKMIQLLMKPGSAH7QADMWSQ3A67F4A, seg_pos={0 0}, upload=205925]: Uploading piece... Dec 15 10:20:49.430214363 (234h51m49.301297408s) [noise=true , address=145.79.227.25:4619 , piece_id=LIP5BYLRZW7E4V2JE3UQNRGLP7VSF5HWUQJAGBX2Z7O5IRAAXXEQ, seg_pos={0 0}, upload=205925]: Uploading piece... Dec 15 10:20:49.434382891 (234h51m49.305465936s) [noise=true , address=145.79.227.38:4977 , piece_id=6RYP2ACTJ3HEN3C22WGMXEVLUMBG4HEPC7BYDFWEPNHK5WCKM3HQ, seg_pos={0 0}, upload=205925]: Uploading piece... Dec 15 10:20:51.173093833 (234h51m51.044176878s) [upload=205925]: Next returned nil segment Dec 15 10:20:51.173109864 (234h51m51.044192909s) [upload=205925]: Waiting for error group managing segments... Dec 15 10:20:51.700311661 (234h51m51.571394706s) [noise=true , address=77.90.31.16:4735 , piece_id=EDSOWIAZ4FOKDNU33OIAOTCHZ4ATYSUFABGRXWZIXBLX45ZHXYIQ, seg_pos={0 0}, upload=205925]: Done uploading piece. err: <nil> Dec 15 10:20:51.865652042 (234h51m51.736735077s) [noise=true , address=77.90.31.14:4662 , piece_id=5LZCGCHJVFRE3F4AVYDQAPADEMZBEHGISIBE6UJH6UUIHGR6QOUQ, seg_pos={0 0}, upload=205925]: Done uploading piece. err: <nil> Dec 15 10:20:52.041004417 (234h51m51.912087453s) [noise=true , address=145.79.227.32:4834 , piece_id=3SYBLOFKGOUJDTQEBKUQ5SATA2HDXBCTBLMX67AFPBOPLASI4U6Q, seg_pos={0 0}, upload=205925]: Done uploading piece. err: <nil> Dec 15 10:20:52.041882921 (234h51m51.912965966s) [noise=true , address=145.79.227.10:4401 , piece_id=RLN5U2PG3IEEGHHHPD3EHAUJKMIQLLMKPGSAH7QADMWSQ3A67F4A, seg_pos={0 0}, upload=205925]: Done uploading piece. err: <nil> Dec 15 10:20:52.204300587 (234h51m52.075383622s) [noise=true , address=145.79.227.50:4373 , piece_id=WZFK2WQVD5PBH5YZQZJXVZNL5LHQXNAEE6ODWMJKEHW6QKHPYDWQ, seg_pos={0 0}, upload=205925]: Done uploading piece. err: <nil> Dec 15 10:20:52.215017794 (234h51m52.086100839s) [noise=true , address=145.79.227.25:4619 , piece_id=LIP5BYLRZW7E4V2JE3UQNRGLP7VSF5HWUQJAGBX2Z7O5IRAAXXEQ, seg_pos={0 0}, upload=205925]: Done uploading piece. err: <nil> Dec 15 10:20:52.215343334 (234h51m52.086426369s) [noise=true , address=145.79.227.51:4410 , piece_id=RB5N6MONVOFKSF3GGEQVRL6EGGPMC5UJ77NVPRGEJV7UYUP3XNFA, seg_pos={0 0}, upload=205925]: Done uploading piece. err: <nil> Dec 15 10:20:52.229549165 (234h51m52.10063221s) [noise=true , address=145.79.227.28:4725 , piece_id=FQ6CLMT5DAYQJEQMYBBI2U63353PWPLJEV4RFHGZCZ7PVB3G4N3Q, seg_pos={0 0}, upload=205925]: Done uploading piece. err: <nil> Dec 15 10:20:52.449423590 (234h51m52.320506635s) [noise=true , address=145.79.227.42:4122 , piece_id=ALJ2YXWR7HT4M53UBTDNYO2EXNOOJCU7KL5NQZCQ6S3PYGSQ3AUQ, seg_pos={0 0}, upload=205925]: Done uploading piece. err: <nil> Dec 15 10:20:52.455660495 (234h51m52.326743531s) [noise=true , address=145.79.227.40:4050 , piece_id=LYBFCWNG53LIAPARIFUQPQBVO2EFCD6DXBR5B6FAWDIFPA7QRPHQ, seg_pos={0 0}, upload=205925]: Done uploading piece. err: <nil> Dec 15 10:20:52.458375175 (234h51m52.32945822s) [noise=true , address=145.79.227.27:4690 , piece_id=YNTNQDEKK4HEPHCDKTWTKGBAIBUW63LZ2X3VUMFW6U75L24CFBHA, seg_pos={0 0}, upload=205925]: Done uploading piece. err: <nil> Dec 15 10:20:52.459049809 (234h51m52.330132854s) [noise=true , address=145.79.227.35:4483 , piece_id=PGLX25C2FYVOGJ5RNV6H7J32UIJZVHH46YJWPHM2RLBGHE3AOD6A, seg_pos={0 0}, upload=205925]: Done uploading piece. err: <nil> Dec 15 10:20:52.479342080 (234h51m52.350425115s) [noise=true , address=145.79.227.8:4329 , piece_id=GIHDKALXLKHSUFFLQSOKSLHXMGMYPOPSXFNWZCI6A7GHJ2KQHEKQ, seg_pos={0 0}, upload=205925]: Done uploading piece. err: <nil> Dec 15 10:20:52.479365885 (234h51m52.35044893s) [seg_pos={0 0}, upload=205925]: Segment reached optimal threshold of 13 pieces. Dec 15 10:20:52.479574001 (234h51m52.350657036s) [noise=true , address=145.79.227.23:4546 , piece_id=OIXWEY7NSIW4WK4QOGKGKYS2YSI6YLFRVHRDCICXUP5CST73PMQQ, seg_pos={0 0}, upload=205925]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZMTFpqfMjeVWXJuvsGdbGkMtGQn6axUZFoU1N9sK6fnbPcMy5): protocol: expected piece hash; context canceled; EOF Dec 15 10:20:52.479584632 (234h51m52.350667667s) [noise=true , address=145.79.227.23:4546 , piece_id=OIXWEY7NSIW4WK4QOGKGKYS2YSI6YLFRVHRDCICXUP5CST73PMQQ, seg_pos={0 0}, upload=205925]: Long tail cancellation due to optimal threshold Dec 15 10:20:52.479595793 (234h51m52.350678838s) [noise=true , address=145.79.227.38:4977 , piece_id=6RYP2ACTJ3HEN3C22WGMXEVLUMBG4HEPC7BYDFWEPNHK5WCKM3HQ, seg_pos={0 0}, upload=205925]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125dJwj1tvBkFrgDCoj3FhpH553yGgjxPuSTGxbrkGeDwdKk6M8): protocol: expected piece hash; context canceled; EOF Dec 15 10:20:52.479606773 (234h51m52.350689809s) [noise=true , address=145.79.227.38:4977 , piece_id=6RYP2ACTJ3HEN3C22WGMXEVLUMBG4HEPC7BYDFWEPNHK5WCKM3HQ, seg_pos={0 0}, upload=205925]: Long tail cancellation due to optimal threshold Dec 15 10:20:52.479787528 (234h51m52.350870563s) [noise=true , address=145.79.227.31:4797 , piece_id=LJTUYNDSGEOLP5GC67MTTHIECLSV3UY6WAT6FBXAEDPWWGMFB7CA, seg_pos={0 0}, upload=205925]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9): protocol: expected piece hash; context canceled; EOF Dec 15 10:20:52.479795373 (234h51m52.350878408s) [noise=true , address=145.79.227.31:4797 , piece_id=LJTUYNDSGEOLP5GC67MTTHIECLSV3UY6WAT6FBXAEDPWWGMFB7CA, seg_pos={0 0}, upload=205925]: Long tail cancellation due to optimal threshold Dec 15 10:20:52.479819700 (234h51m52.350902735s) [seg_pos={0 0}, upload=205925]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 15 10:20:52.479903930 (234h51m52.350986975s) [upload=205925]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 15 10:20:52.479909911 (234h51m52.350992956s) [upload=205925]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 15 10:20:52.884169543 (234h51m52.755252618s) [upload=205925]: Done uploading segments... Dec 15 10:20:52.884199039 (234h51m52.755282104s) [upload=205925]: Upload finished. err: <nil> Duration: 7.733635756s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [..X........................................................X] duration: 5.001324017s trace: https://jaeger.dev.storj.io/trace/55752dded9459114 FAIL (2025-12-15 11:00:14 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001358623s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...X........................................................] duration: 3.62509717s trace: https://jaeger.dev.storj.io/trace/1b69ac01af256235 FAIL (2025-12-15 10:59:46 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.625130263s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 7.747995058s trace: https://jaeger.dev.storj.io/trace/4a708d72fc2c52e9 FAIL (2025-12-15 10:59:45 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, Piece ID: EDSOWIAZ4FOKDNU33OIAOTCHZ4ATYSUFABGRXWZIXBLX45ZHXYIQ) context deadline exceeded; piecestore: (Node ID: 138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, Piece ID: 5LZCGCHJVFRE3F4AVYDQAPADEMZBEHGISIBE6UJH6UUIHGR6QOUQ) context deadline exceeded ======LOGS====== Dec 15 10:20:49.101615376 (234h51m48.972698411s) [upload=205925]: Starting upload Dec 15 10:20:49.101674809 (234h51m48.972757854s) [upload=205925]: Done starting upload Dec 15 10:20:49.101724754 (234h51m48.972807809s) [upload=205925]: Uploading segments... Dec 15 10:20:49.101919515 (234h51m48.97300255s) [upload=205925]: Got next segment. Inline: false Dec 15 10:20:49.101952237 (234h51m48.973035272s) [upload=205925]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 15 10:20:49.101960182 (234h51m48.973043217s) [upload=205925]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 15 10:20:49.424060615 (234h51m49.29514369s) [seg_pos={0 0}, upload=205925]: Begin upload segment... Dec 15 10:20:49.424121009 (234h51m49.295204045s) [seg_pos={0 0}, upload=205925]: Done begin upload segment. Dec 15 10:20:49.424161688 (234h51m49.295244733s) [noise=true , address=145.79.227.32:4834 , piece_id=3SYBLOFKGOUJDTQEBKUQ5SATA2HDXBCTBLMX67AFPBOPLASI4U6Q, seg_pos={0 0}, upload=205925]: Uploading piece... Dec 15 10:20:49.424254584 (234h51m49.295337629s) [noise=true , address=145.79.227.42:4122 , piece_id=ALJ2YXWR7HT4M53UBTDNYO2EXNOOJCU7KL5NQZCQ6S3PYGSQ3AUQ, seg_pos={0 0}, upload=205925]: Uploading piece... Dec 15 10:20:49.424905434 (234h51m49.295988479s) [noise=true , address=77.90.31.16:4735 , piece_id=EDSOWIAZ4FOKDNU33OIAOTCHZ4ATYSUFABGRXWZIXBLX45ZHXYIQ, seg_pos={0 0}, upload=205925]: Uploading piece... Dec 15 10:20:49.424915713 (234h51m49.295998758s) [noise=true , address=145.79.227.51:4410 , piece_id=RB5N6MONVOFKSF3GGEQVRL6EGGPMC5UJ77NVPRGEJV7UYUP3XNFA, seg_pos={0 0}, upload=205925]: Uploading piece... Dec 15 10:20:49.424964086 (234h51m49.296047131s) [noise=true , address=77.90.31.14:4662 , piece_id=5LZCGCHJVFRE3F4AVYDQAPADEMZBEHGISIBE6UJH6UUIHGR6QOUQ, seg_pos={0 0}, upload=205925]: Uploading piece... Dec 15 10:20:49.425378545 (234h51m49.29646159s) [noise=true , address=145.79.227.40:4050 , piece_id=LYBFCWNG53LIAPARIFUQPQBVO2EFCD6DXBR5B6FAWDIFPA7QRPHQ, seg_pos={0 0}, upload=205925]: Uploading piece... Dec 15 10:20:49.425429762 (234h51m49.296512807s) [noise=true , address=145.79.227.27:4690 , piece_id=YNTNQDEKK4HEPHCDKTWTKGBAIBUW63LZ2X3VUMFW6U75L24CFBHA, seg_pos={0 0}, upload=205925]: Uploading piece... Dec 15 10:20:49.425828811 (234h51m49.296911846s) [noise=true , address=145.79.227.8:4329 , piece_id=GIHDKALXLKHSUFFLQSOKSLHXMGMYPOPSXFNWZCI6A7GHJ2KQHEKQ, seg_pos={0 0}, upload=205925]: Uploading piece... Dec 15 10:20:49.425977345 (234h51m49.29706039s) [noise=true , address=145.79.227.28:4725 , piece_id=FQ6CLMT5DAYQJEQMYBBI2U63353PWPLJEV4RFHGZCZ7PVB3G4N3Q, seg_pos={0 0}, upload=205925]: Uploading piece... Dec 15 10:20:49.426452569 (234h51m49.297535604s) [noise=true , address=145.79.227.23:4546 , piece_id=OIXWEY7NSIW4WK4QOGKGKYS2YSI6YLFRVHRDCICXUP5CST73PMQQ, seg_pos={0 0}, upload=205925]: Uploading piece... Dec 15 10:20:49.427024318 (234h51m49.298107364s) [noise=true , address=145.79.227.35:4483 , piece_id=PGLX25C2FYVOGJ5RNV6H7J32UIJZVHH46YJWPHM2RLBGHE3AOD6A, seg_pos={0 0}, upload=205925]: Uploading piece... Dec 15 10:20:49.427498542 (234h51m49.298581587s) [noise=true , address=145.79.227.50:4373 , piece_id=WZFK2WQVD5PBH5YZQZJXVZNL5LHQXNAEE6ODWMJKEHW6QKHPYDWQ, seg_pos={0 0}, upload=205925]: Uploading piece... Dec 15 10:20:49.427934261 (234h51m49.299017296s) [noise=true , address=145.79.227.31:4797 , piece_id=LJTUYNDSGEOLP5GC67MTTHIECLSV3UY6WAT6FBXAEDPWWGMFB7CA, seg_pos={0 0}, upload=205925]: Uploading piece... Dec 15 10:20:49.428816932 (234h51m49.299899977s) [noise=true , address=145.79.227.10:4401 , piece_id=RLN5U2PG3IEEGHHHPD3EHAUJKMIQLLMKPGSAH7QADMWSQ3A67F4A, seg_pos={0 0}, upload=205925]: Uploading piece... Dec 15 10:20:49.430214363 (234h51m49.301297408s) [noise=true , address=145.79.227.25:4619 , piece_id=LIP5BYLRZW7E4V2JE3UQNRGLP7VSF5HWUQJAGBX2Z7O5IRAAXXEQ, seg_pos={0 0}, upload=205925]: Uploading piece... Dec 15 10:20:49.434382891 (234h51m49.305465936s) [noise=true , address=145.79.227.38:4977 , piece_id=6RYP2ACTJ3HEN3C22WGMXEVLUMBG4HEPC7BYDFWEPNHK5WCKM3HQ, seg_pos={0 0}, upload=205925]: Uploading piece... Dec 15 10:20:51.173093833 (234h51m51.044176878s) [upload=205925]: Next returned nil segment Dec 15 10:20:51.173109864 (234h51m51.044192909s) [upload=205925]: Waiting for error group managing segments... Dec 15 10:20:51.700311661 (234h51m51.571394706s) [noise=true , address=77.90.31.16:4735 , piece_id=EDSOWIAZ4FOKDNU33OIAOTCHZ4ATYSUFABGRXWZIXBLX45ZHXYIQ, seg_pos={0 0}, upload=205925]: Done uploading piece. err: <nil> Dec 15 10:20:51.865652042 (234h51m51.736735077s) [noise=true , address=77.90.31.14:4662 , piece_id=5LZCGCHJVFRE3F4AVYDQAPADEMZBEHGISIBE6UJH6UUIHGR6QOUQ, seg_pos={0 0}, upload=205925]: Done uploading piece. err: <nil> Dec 15 10:20:52.041004417 (234h51m51.912087453s) [noise=true , address=145.79.227.32:4834 , piece_id=3SYBLOFKGOUJDTQEBKUQ5SATA2HDXBCTBLMX67AFPBOPLASI4U6Q, seg_pos={0 0}, upload=205925]: Done uploading piece. err: <nil> Dec 15 10:20:52.041882921 (234h51m51.912965966s) [noise=true , address=145.79.227.10:4401 , piece_id=RLN5U2PG3IEEGHHHPD3EHAUJKMIQLLMKPGSAH7QADMWSQ3A67F4A, seg_pos={0 0}, upload=205925]: Done uploading piece. err: <nil> Dec 15 10:20:52.204300587 (234h51m52.075383622s) [noise=true , address=145.79.227.50:4373 , piece_id=WZFK2WQVD5PBH5YZQZJXVZNL5LHQXNAEE6ODWMJKEHW6QKHPYDWQ, seg_pos={0 0}, upload=205925]: Done uploading piece. err: <nil> Dec 15 10:20:52.215017794 (234h51m52.086100839s) [noise=true , address=145.79.227.25:4619 , piece_id=LIP5BYLRZW7E4V2JE3UQNRGLP7VSF5HWUQJAGBX2Z7O5IRAAXXEQ, seg_pos={0 0}, upload=205925]: Done uploading piece. err: <nil> Dec 15 10:20:52.215343334 (234h51m52.086426369s) [noise=true , address=145.79.227.51:4410 , piece_id=RB5N6MONVOFKSF3GGEQVRL6EGGPMC5UJ77NVPRGEJV7UYUP3XNFA, seg_pos={0 0}, upload=205925]: Done uploading piece. err: <nil> Dec 15 10:20:52.229549165 (234h51m52.10063221s) [noise=true , address=145.79.227.28:4725 , piece_id=FQ6CLMT5DAYQJEQMYBBI2U63353PWPLJEV4RFHGZCZ7PVB3G4N3Q, seg_pos={0 0}, upload=205925]: Done uploading piece. err: <nil> Dec 15 10:20:52.449423590 (234h51m52.320506635s) [noise=true , address=145.79.227.42:4122 , piece_id=ALJ2YXWR7HT4M53UBTDNYO2EXNOOJCU7KL5NQZCQ6S3PYGSQ3AUQ, seg_pos={0 0}, upload=205925]: Done uploading piece. err: <nil> Dec 15 10:20:52.455660495 (234h51m52.326743531s) [noise=true , address=145.79.227.40:4050 , piece_id=LYBFCWNG53LIAPARIFUQPQBVO2EFCD6DXBR5B6FAWDIFPA7QRPHQ, seg_pos={0 0}, upload=205925]: Done uploading piece. err: <nil> Dec 15 10:20:52.458375175 (234h51m52.32945822s) [noise=true , address=145.79.227.27:4690 , piece_id=YNTNQDEKK4HEPHCDKTWTKGBAIBUW63LZ2X3VUMFW6U75L24CFBHA, seg_pos={0 0}, upload=205925]: Done uploading piece. err: <nil> Dec 15 10:20:52.459049809 (234h51m52.330132854s) [noise=true , address=145.79.227.35:4483 , piece_id=PGLX25C2FYVOGJ5RNV6H7J32UIJZVHH46YJWPHM2RLBGHE3AOD6A, seg_pos={0 0}, upload=205925]: Done uploading piece. err: <nil> Dec 15 10:20:52.479342080 (234h51m52.350425115s) [noise=true , address=145.79.227.8:4329 , piece_id=GIHDKALXLKHSUFFLQSOKSLHXMGMYPOPSXFNWZCI6A7GHJ2KQHEKQ, seg_pos={0 0}, upload=205925]: Done uploading piece. err: <nil> Dec 15 10:20:52.479365885 (234h51m52.35044893s) [seg_pos={0 0}, upload=205925]: Segment reached optimal threshold of 13 pieces. Dec 15 10:20:52.479574001 (234h51m52.350657036s) [noise=true , address=145.79.227.23:4546 , piece_id=OIXWEY7NSIW4WK4QOGKGKYS2YSI6YLFRVHRDCICXUP5CST73PMQQ, seg_pos={0 0}, upload=205925]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZMTFpqfMjeVWXJuvsGdbGkMtGQn6axUZFoU1N9sK6fnbPcMy5): protocol: expected piece hash; context canceled; EOF Dec 15 10:20:52.479584632 (234h51m52.350667667s) [noise=true , address=145.79.227.23:4546 , piece_id=OIXWEY7NSIW4WK4QOGKGKYS2YSI6YLFRVHRDCICXUP5CST73PMQQ, seg_pos={0 0}, upload=205925]: Long tail cancellation due to optimal threshold Dec 15 10:20:52.479595793 (234h51m52.350678838s) [noise=true , address=145.79.227.38:4977 , piece_id=6RYP2ACTJ3HEN3C22WGMXEVLUMBG4HEPC7BYDFWEPNHK5WCKM3HQ, seg_pos={0 0}, upload=205925]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125dJwj1tvBkFrgDCoj3FhpH553yGgjxPuSTGxbrkGeDwdKk6M8): protocol: expected piece hash; context canceled; EOF Dec 15 10:20:52.479606773 (234h51m52.350689809s) [noise=true , address=145.79.227.38:4977 , piece_id=6RYP2ACTJ3HEN3C22WGMXEVLUMBG4HEPC7BYDFWEPNHK5WCKM3HQ, seg_pos={0 0}, upload=205925]: Long tail cancellation due to optimal threshold Dec 15 10:20:52.479787528 (234h51m52.350870563s) [noise=true , address=145.79.227.31:4797 , piece_id=LJTUYNDSGEOLP5GC67MTTHIECLSV3UY6WAT6FBXAEDPWWGMFB7CA, seg_pos={0 0}, upload=205925]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9): protocol: expected piece hash; context canceled; EOF Dec 15 10:20:52.479795373 (234h51m52.350878408s) [noise=true , address=145.79.227.31:4797 , piece_id=LJTUYNDSGEOLP5GC67MTTHIECLSV3UY6WAT6FBXAEDPWWGMFB7CA, seg_pos={0 0}, upload=205925]: Long tail cancellation due to optimal threshold Dec 15 10:20:52.479819700 (234h51m52.350902735s) [seg_pos={0 0}, upload=205925]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 15 10:20:52.479903930 (234h51m52.350986975s) [upload=205925]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 15 10:20:52.479909911 (234h51m52.350992956s) [upload=205925]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 15 10:20:52.884169543 (234h51m52.755252618s) [upload=205925]: Done uploading segments... Dec 15 10:20:52.884199039 (234h51m52.755282104s) [upload=205925]: Upload finished. err: <nil> Duration: 7.748033171s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)