Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 5 out of 60 is failed (threshold=6) [............X.................X..............X.X.X..........] duration: 3.50848529s trace: https://jaeger.dev.storj.io/trace/6e59f4ebc82d0fcc FAIL (2025-11-22 13:42:16 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.508517452s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 5 out of 60 is failed (threshold=6) [.............X.................X..............X.X.X.........] duration: 3.531189438s trace: https://jaeger.dev.storj.io/trace/6e59f4ebc82d0fcc FAIL (2025-11-22 13:41:42 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.531207193s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 5 out of 60 is failed (threshold=6) [..............X.................X..............X.X.X........] duration: 3.509169821s trace: https://jaeger.dev.storj.io/trace/6e59f4ebc82d0fcc FAIL (2025-11-22 13:40:40 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.509292825s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 5 out of 60 is failed (threshold=6) [...............X.................X..............X.X.X.......] duration: 3.472439669s trace: https://jaeger.dev.storj.io/trace/6e59f4ebc82d0fcc FAIL (2025-11-22 13:39:51 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.472467654s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 5 out of 60 is failed (threshold=6) [................X.................X..............X.X.X......] duration: 3.528807494s trace: https://jaeger.dev.storj.io/trace/6e59f4ebc82d0fcc FAIL (2025-11-22 13:39:06 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.528840427s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 5 out of 60 is failed (threshold=6) [.................X.................X..............X.X.X.....] duration: 3.49650338s trace: https://jaeger.dev.storj.io/trace/6e59f4ebc82d0fcc FAIL (2025-11-22 13:38:14 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.496570508s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 5 out of 60 is failed (threshold=6) [..................X.................X..............X.X.X....] duration: 3.407914232s trace: https://jaeger.dev.storj.io/trace/6e59f4ebc82d0fcc FAIL (2025-11-22 13:37:31 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.407952745s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 5 out of 60 is failed (threshold=6) [...................X.................X..............X.X.X...] duration: 3.520340932s trace: https://jaeger.dev.storj.io/trace/6e59f4ebc82d0fcc FAIL (2025-11-22 13:36:32 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.520355259s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 5 out of 60 is failed (threshold=6) [....................X.................X..............X.X.X..] duration: 3.505973283s trace: https://jaeger.dev.storj.io/trace/6e59f4ebc82d0fcc FAIL (2025-11-22 13:35:37 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.506006366s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 5 out of 60 is failed (threshold=6) [.....................X.................X..............X.X.X.] duration: 3.361479622s trace: https://jaeger.dev.storj.io/trace/6e59f4ebc82d0fcc FAIL (2025-11-22 13:34:22 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.361506424s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 5 out of 60 is failed (threshold=6) [......................X.................X..............X.X.X] duration: 5.001988527s trace: https://jaeger.dev.storj.io/trace/6e59f4ebc82d0fcc FAIL (2025-11-22 13:34:06 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.002012792s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [.......................X.................X..............X.X.] duration: 3.418297603s trace: https://jaeger.dev.storj.io/trace/58878b04e97973c5 FAIL (2025-11-22 13:33:40 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.418360153s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [........................X.................X..............X.X] duration: 5.00152859s trace: https://jaeger.dev.storj.io/trace/58878b04e97973c5 FAIL (2025-11-22 13:32:46 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001556494s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [.........................X.................X..............X.] duration: 3.454921041s trace: https://jaeger.dev.storj.io/trace/16bff4f3a89abfb8 FAIL (2025-11-22 13:32:23 UTC) orbiter warning: context deadline exceeded Duration: 3.454954123s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [..........................X.................X..............X] duration: 5.002565776s trace: https://jaeger.dev.storj.io/trace/16bff4f3a89abfb8 FAIL (2025-11-22 13:32:05 UTC) orbiter warning: context deadline exceeded Duration: 5.002589519s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [...........................X.................X..............] duration: 3.80909089s trace: https://jaeger.dev.storj.io/trace/53a9e28ab5efa5ae FAIL (2025-11-22 13:31:34 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.80912185s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [............................X.................X.............] duration: 3.532336824s trace: https://jaeger.dev.storj.io/trace/53a9e28ab5efa5ae FAIL (2025-11-22 13:30:52 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.532364066s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.............................X.................X............] duration: 3.501952758s trace: https://jaeger.dev.storj.io/trace/53a9e28ab5efa5ae FAIL (2025-11-22 13:29:41 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.501979499s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [..............................X.................X...........] duration: 5.001156682s trace: https://jaeger.dev.storj.io/trace/53a9e28ab5efa5ae FAIL (2025-11-22 13:29:21 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001181689s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [...............................X.................X..........] duration: 3.460552194s trace: https://jaeger.dev.storj.io/trace/53a9e28ab5efa5ae FAIL (2025-11-22 13:28:42 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.460580547s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [................................X.................X.........] duration: 3.440017943s trace: https://jaeger.dev.storj.io/trace/53a9e28ab5efa5ae FAIL (2025-11-22 13:27:47 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.440045214s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.................................X.................X........] duration: 3.553820814s trace: https://jaeger.dev.storj.io/trace/53a9e28ab5efa5ae FAIL (2025-11-22 13:26:58 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.553847454s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [..................................X.................X.......] duration: 3.437015446s trace: https://jaeger.dev.storj.io/trace/53a9e28ab5efa5ae FAIL (2025-11-22 13:26:42 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.437245955s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [...................................X.................X......] duration: 3.492178917s trace: https://jaeger.dev.storj.io/trace/53a9e28ab5efa5ae FAIL (2025-11-22 13:25:52 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.492209085s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [....................................X.................X.....] duration: 4.769196021s trace: https://jaeger.dev.storj.io/trace/53a9e28ab5efa5ae FAIL (2025-11-22 13:24:41 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.769230126s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.....................................X.................X....] duration: 5.001129603s trace: https://jaeger.dev.storj.io/trace/53a9e28ab5efa5ae FAIL (2025-11-22 13:24:25 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001161414s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [......................................X.................X...] duration: 3.387913782s trace: https://jaeger.dev.storj.io/trace/53a9e28ab5efa5ae FAIL (2025-11-22 13:23:37 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.388014604s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.......................................X.................X..] duration: 4.671787438s trace: https://jaeger.dev.storj.io/trace/53a9e28ab5efa5ae FAIL (2025-11-22 13:23:10 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.671827616s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [........................................X.................X.] duration: 3.409643519s trace: https://jaeger.dev.storj.io/trace/53a9e28ab5efa5ae FAIL (2025-11-22 13:22:00 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.409689756s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.........................................X.................X] duration: 5.001285729s trace: https://jaeger.dev.storj.io/trace/53a9e28ab5efa5ae FAIL (2025-11-22 13:21:04 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001309474s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..........................................X.................] duration: 3.562994255s trace: https://jaeger.dev.storj.io/trace/38596674a32f39c1 FAIL (2025-11-22 13:20:10 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.563049921s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...........................................X................] duration: 4.417149746s trace: https://jaeger.dev.storj.io/trace/38596674a32f39c1 FAIL (2025-11-22 13:19:50 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.417184251s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [............................................X...............] duration: 4.875914591s trace: https://jaeger.dev.storj.io/trace/38596674a32f39c1 FAIL (2025-11-22 13:19:27 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.87659644s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.............................................X..............] duration: 3.614522474s trace: https://jaeger.dev.storj.io/trace/38596674a32f39c1 FAIL (2025-11-22 13:18:13 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.614559194s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 5.907455833s trace: https://jaeger.dev.storj.io/trace/6c800b4c3a84b58c FAIL (2025-11-22 13:17:44 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 5) ======LOGS====== N5RGYNBMYFXXBJDBTUCN37EAT7WBRNTX2YQA, seg_pos={0 0}, upload=278731]: Uploading piece... Nov 22 12:33:11.890374814 (316h29m10.441845275s) [noise=true , address=145.79.227.32:4858 , piece_id=K5L3LEUKZWHS36XJHNNY4TYRXCP4DG5UAKCZXPJJTSG26CKUI4HQ, seg_pos={0 0}, upload=278731]: Uploading piece... Nov 22 12:33:11.890535981 (316h29m10.442006442s) [noise=true , address=77.90.31.21:4890 , piece_id=DUEFPJUTYMD564Z4III63LZM6L4IL2ZTGM3QDZ6CG7WV5APHBNRA, seg_pos={0 0}, upload=278731]: Uploading piece... Nov 22 12:33:11.890774064 (316h29m10.442244525s) [noise=true , address=45.91.116.14:4213 , piece_id=HPFISTE2N3B4RBDPAZAGVZJ5WYLCPHTCYH4WMJ56IWEHR3ZBEMOQ, seg_pos={0 0}, upload=278731]: Uploading piece... Nov 22 12:33:11.890934851 (316h29m10.442405312s) [noise=true , address=45.91.116.6:4000 , piece_id=T4CI3KKNMFPT3ZIODHE7BQ4DVHVH2QTSO6OB7QZPL5SNPXQJE6TA, seg_pos={0 0}, upload=278731]: Uploading piece... Nov 22 12:33:11.891167002 (316h29m10.442637463s) [noise=true , address=77.90.31.17:4787 , piece_id=IYIEVH6CARRYCARKOLKWVBICQDNDRCTGQMWF3VJZ6YBC3GO5OWHQ, seg_pos={0 0}, upload=278731]: Uploading piece... Nov 22 12:33:11.891369828 (316h29m10.442840289s) [noise=true , address=45.91.116.7:4039 , piece_id=HSRY63FXTEPCP2T46V77RVSC24GGRBAYLLU7DTKOLWGVOYHF6YVA, seg_pos={0 0}, upload=278731]: Uploading piece... Nov 22 12:33:11.891556875 (316h29m10.443027336s) [noise=true , address=77.90.31.11:4572 , piece_id=G4XSIPSWOLSEOGGFA2NLOSDCGTLZPI3P5LDNNLB5RGMU5OQGWXGQ, seg_pos={0 0}, upload=278731]: Uploading piece... Nov 22 12:33:11.891896812 (316h29m10.443367273s) [noise=true , address=77.90.31.19:4863 , piece_id=3PM3Q6X3ADKX2RKIIFDMK57LTJNJHNFK4UYPKFK3ZEQLYQ23EP3A, seg_pos={0 0}, upload=278731]: Uploading piece... Nov 22 12:33:11.892019235 (316h29m10.443489696s) [noise=true , address=77.90.31.10:4524 , piece_id=Z2IGDHPDU6JAR5CAFQBGYHO3TQ66K4BZOAMXSC7X3RFWLGHOOMTQ, seg_pos={0 0}, upload=278731]: Uploading piece... Nov 22 12:33:11.892316521 (316h29m10.443786982s) [noise=true , address=77.90.31.13:4644 , piece_id=UPDZGC3CMJXA3AOJX65VLASU37NLFGPIJLCAPN32TEQB5QHTU7EQ, seg_pos={0 0}, upload=278731]: Uploading piece... Nov 22 12:33:11.892682318 (316h29m10.444152779s) [noise=true , address=45.91.116.9:4274 , piece_id=MXWRAXYLBBGLUVTA5GP3JVX2IRM2UI2IC35D7KX3XMFEBXLIDX5A, seg_pos={0 0}, upload=278731]: Uploading piece... Nov 22 12:33:13.353758506 (316h29m11.905228957s) [upload=278731]: Next returned nil segment Nov 22 12:33:13.353772043 (316h29m11.905242504s) [upload=278731]: Waiting for error group managing segments... Nov 22 12:33:13.751703718 (316h29m12.303174169s) [noise=true , address=145.79.227.32:4858 , piece_id=K5L3LEUKZWHS36XJHNNY4TYRXCP4DG5UAKCZXPJJTSG26CKUI4HQ, seg_pos={0 0}, upload=278731]: Done uploading piece. err: <nil> Nov 22 12:33:13.753660293 (316h29m12.305130764s) [noise=true , address=145.79.227.26:4674 , piece_id=UP7VIVRDRL6KERJSX3VLWTKGW3ESGCQ7BLYCBXGBB5UNIDE6OHQQ, seg_pos={0 0}, upload=278731]: Done uploading piece. err: <nil> Nov 22 12:33:13.760046123 (316h29m12.311516574s) [noise=true , address=145.79.227.3:4167 , piece_id=4STIVXGU7SOXWAIPG2PXWBLNDD7EYQ56E2DBN2UZ2AZ3TZSXCTIQ, seg_pos={0 0}, upload=278731]: Done uploading piece. err: <nil> Nov 22 12:33:57.373975014 (316h29m55.925445475s) [noise=true , address=77.90.31.13:4644 , piece_id=UPDZGC3CMJXA3AOJX65VLASU37NLFGPIJLCAPN32TEQB5QHTU7EQ, seg_pos={0 0}, upload=278731]: Done uploading piece. err: <nil> Nov 22 12:34:06.122275078 (316h30m4.673745539s) [noise=true , address=45.91.116.13:4181 , piece_id=MQNJEBQFZMHNDKAH2K3JEA7YBK2UK5BGSNCJF3MASF3CKVA5VD6Q, seg_pos={0 0}, upload=278731]: Done uploading piece. err: <nil> Nov 22 12:34:10.949171331 (316h30m9.500641792s) [noise=true , address=77.90.31.21:4890 , piece_id=DUEFPJUTYMD564Z4III63LZM6L4IL2ZTGM3QDZ6CG7WV5APHBNRA, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:1iE3mDnEhGDsMBtmUXC4EwvSrpyHLoybzVXkC9eoFuYEMHeBrd, address:77.90.31.21:4890): protocol: read tcp 188.166.225.170:55900->77.90.31.21:4890: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 12:34:10.949753650 (316h30m9.501224111s) [noise=true , address=45.91.116.12:4237 , piece_id=OG6XREYEGOYVFFMXCGBERRAIEMI2QMUGSBJBYP7GJQZ7FOAI4BIA, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:1FMjHdNGV42a9QZyMmSPTJWQrA8CdSRvXqrimnt6UGVreCzmgF, address:45.91.116.12:4237): protocol: rpc: dial tcp 45.91.116.12:4237: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 12:34:10.950066235 (316h30m9.501536706s) [noise=true , address=77.90.31.19:4863 , piece_id=3PM3Q6X3ADKX2RKIIFDMK57LTJNJHNFK4UYPKFK3ZEQLYQ23EP3A, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:124x1cbCPJRK8YkgSP6UzHymoLu1E1znnfg5WBiD89kSDLi8JeL, address:77.90.31.19:4863): protocol: write tcp 188.166.225.170:45968->77.90.31.19:4863: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 12:34:10.950246558 (316h30m9.50171702s) [noise=true , address=77.90.31.10:4524 , piece_id=Z2IGDHPDU6JAR5CAFQBGYHO3TQ66K4BZOAMXSC7X3RFWLGHOOMTQ, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:12uN7CznoK4AD8FqFkNUndawdGJ9rmjAe7oNnjJkV844ScY87qe, address:77.90.31.10:4524): protocol: write tcp 188.166.225.170:40266->77.90.31.10:4524: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 12:34:10.950593058 (316h30m9.502063529s) [noise=true , address=77.90.31.14:4690 , piece_id=RJ3KXY3HURT7Q4HZYTOLSAL7Y6VMDGGRIQP6XHE4YB6OAA3PJRAA, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:1WPKeCcN1uEhGjZHAJgCoJMaHxAWouGCdaU1vZwAS7kX22rxcv, address:77.90.31.14:4690): protocol: rpc: dial tcp 77.90.31.14:4690: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 12:34:10.950853535 (316h30m9.502324006s) [noise=true , address=45.91.116.14:4213 , piece_id=HPFISTE2N3B4RBDPAZAGVZJ5WYLCPHTCYH4WMJ56IWEHR3ZBEMOQ, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:12tQRqowH4aDr9h3s1WLs1o7na1ueiafwgUeeHfAB6g7mcE6e1t, address:45.91.116.14:4213): protocol: rpc: dial tcp 45.91.116.14:4213: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 12:34:10.951186438 (316h30m9.502656889s) [noise=true , address=45.91.116.6:4000 , piece_id=T4CI3KKNMFPT3ZIODHE7BQ4DVHVH2QTSO6OB7QZPL5SNPXQJE6TA, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:12CuCqHJYbanbuaEMHvtcuQXhvcdahD8eyT2fKDhqtxdSZ6ES4f, address:45.91.116.6:4000): protocol: rpc: dial tcp 45.91.116.6:4000: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 12:34:10.951409864 (316h30m9.502880325s) [noise=true , address=45.91.116.7:4039 , piece_id=HSRY63FXTEPCP2T46V77RVSC24GGRBAYLLU7DTKOLWGVOYHF6YVA, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:16oq1Yv7JfHrg4gdHfhwuMj4UsWeQQPayKV6Pj1NY2CgJhuPr2, address:45.91.116.7:4039): protocol: write tcp 188.166.225.170:36904->45.91.116.7:4039: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 12:34:10.951660191 (316h30m9.503130652s) [noise=true , address=77.90.31.11:4572 , piece_id=G4XSIPSWOLSEOGGFA2NLOSDCGTLZPI3P5LDNNLB5RGMU5OQGWXGQ, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:1ZAfYZ4wgpZVvatj6wtHona3hTaPcLxAmXC1DT2uKQyo8nLQQ7, address:77.90.31.11:4572): protocol: rpc: dial tcp 77.90.31.11:4572: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 12:34:10.952019855 (316h30m9.503490316s) [noise=true , address=45.91.116.9:4274 , piece_id=MXWRAXYLBBGLUVTA5GP3JVX2IRM2UI2IC35D7KX3XMFEBXLIDX5A, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:12g6jFa38dTz5UdgsQaEuM2MSH1H7cxaysWsVmbHkM2Dd2W2iDb, address:45.91.116.9:4274): protocol: rpc: dial tcp 45.91.116.9:4274: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 12:34:10.952290972 (316h30m9.503761443s) [noise=true , address=77.90.31.15:4720 , piece_id=BDQUF6C5TT3ASE3DIN6SSVJHOZAJK4ZEW33OIYPTGB4LDVRELSUA, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:12VbsxFqMB3vVphSAJqYpbBzua7Goq45E6ovJSaExkpLtfkcc74, address:77.90.31.15:4720): protocol: rpc: dial tcp 77.90.31.15:4720: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 12:34:10.952544654 (316h30m9.504015105s) [noise=true , address=45.91.116.10:4110 , piece_id=H6FYA6K4N4AQYU3MN5RGYNBMYFXXBJDBTUCN37EAT7WBRNTX2YQA, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:123KwusATu1ZKtNNVHZZZBE67ZEnVb8puyfQidBi3fEWREqdf1h, address:45.91.116.10:4110): protocol: rpc: dial tcp 45.91.116.10:4110: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 12:34:10.953548155 (316h30m9.505018617s) [noise=true , address=77.90.31.17:4787 , piece_id=IYIEVH6CARRYCARKOLKWVBICQDNDRCTGQMWF3VJZ6YBC3GO5OWHQ, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:1Lietonb5aVbaZQt6wQ2EoxwJkSEn9oL9Dwyrc9mTyomEKikNt, address:77.90.31.17:4787): protocol: read tcp 188.166.225.170:57908->77.90.31.17:4787: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 12:34:10.953579816 (316h30m9.505050277s) [seg_pos={0 0}, upload=278731]: Done waiting for segment. successful: 5 optimal: 14 errs: <nil> Nov 22 12:34:11.195719123 (316h30m9.747189584s) [upload=278731]: Done uploading segments... Nov 22 12:34:11.195932840 (316h30m9.747403301s) [upload=278731]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 5) Duration: 5.907486262s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [X.............................................X.............] duration: 3.465124676s trace: https://jaeger.dev.storj.io/trace/38596674a32f39c1 FAIL (2025-11-22 13:17:27 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.465233673s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [XX.............................................X............] duration: 3.514549337s trace: https://jaeger.dev.storj.io/trace/38596674a32f39c1 FAIL (2025-11-22 13:16:52 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.51457672s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 5.980435839s trace: https://jaeger.dev.storj.io/trace/6c800b4c3a84b58c FAIL (2025-11-22 13:16:46 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 5) ======LOGS====== N5RGYNBMYFXXBJDBTUCN37EAT7WBRNTX2YQA, seg_pos={0 0}, upload=278731]: Uploading piece... Nov 22 12:33:11.890374814 (316h29m10.441845275s) [noise=true , address=145.79.227.32:4858 , piece_id=K5L3LEUKZWHS36XJHNNY4TYRXCP4DG5UAKCZXPJJTSG26CKUI4HQ, seg_pos={0 0}, upload=278731]: Uploading piece... Nov 22 12:33:11.890535981 (316h29m10.442006442s) [noise=true , address=77.90.31.21:4890 , piece_id=DUEFPJUTYMD564Z4III63LZM6L4IL2ZTGM3QDZ6CG7WV5APHBNRA, seg_pos={0 0}, upload=278731]: Uploading piece... Nov 22 12:33:11.890774064 (316h29m10.442244525s) [noise=true , address=45.91.116.14:4213 , piece_id=HPFISTE2N3B4RBDPAZAGVZJ5WYLCPHTCYH4WMJ56IWEHR3ZBEMOQ, seg_pos={0 0}, upload=278731]: Uploading piece... Nov 22 12:33:11.890934851 (316h29m10.442405312s) [noise=true , address=45.91.116.6:4000 , piece_id=T4CI3KKNMFPT3ZIODHE7BQ4DVHVH2QTSO6OB7QZPL5SNPXQJE6TA, seg_pos={0 0}, upload=278731]: Uploading piece... Nov 22 12:33:11.891167002 (316h29m10.442637463s) [noise=true , address=77.90.31.17:4787 , piece_id=IYIEVH6CARRYCARKOLKWVBICQDNDRCTGQMWF3VJZ6YBC3GO5OWHQ, seg_pos={0 0}, upload=278731]: Uploading piece... Nov 22 12:33:11.891369828 (316h29m10.442840289s) [noise=true , address=45.91.116.7:4039 , piece_id=HSRY63FXTEPCP2T46V77RVSC24GGRBAYLLU7DTKOLWGVOYHF6YVA, seg_pos={0 0}, upload=278731]: Uploading piece... Nov 22 12:33:11.891556875 (316h29m10.443027336s) [noise=true , address=77.90.31.11:4572 , piece_id=G4XSIPSWOLSEOGGFA2NLOSDCGTLZPI3P5LDNNLB5RGMU5OQGWXGQ, seg_pos={0 0}, upload=278731]: Uploading piece... Nov 22 12:33:11.891896812 (316h29m10.443367273s) [noise=true , address=77.90.31.19:4863 , piece_id=3PM3Q6X3ADKX2RKIIFDMK57LTJNJHNFK4UYPKFK3ZEQLYQ23EP3A, seg_pos={0 0}, upload=278731]: Uploading piece... Nov 22 12:33:11.892019235 (316h29m10.443489696s) [noise=true , address=77.90.31.10:4524 , piece_id=Z2IGDHPDU6JAR5CAFQBGYHO3TQ66K4BZOAMXSC7X3RFWLGHOOMTQ, seg_pos={0 0}, upload=278731]: Uploading piece... Nov 22 12:33:11.892316521 (316h29m10.443786982s) [noise=true , address=77.90.31.13:4644 , piece_id=UPDZGC3CMJXA3AOJX65VLASU37NLFGPIJLCAPN32TEQB5QHTU7EQ, seg_pos={0 0}, upload=278731]: Uploading piece... Nov 22 12:33:11.892682318 (316h29m10.444152779s) [noise=true , address=45.91.116.9:4274 , piece_id=MXWRAXYLBBGLUVTA5GP3JVX2IRM2UI2IC35D7KX3XMFEBXLIDX5A, seg_pos={0 0}, upload=278731]: Uploading piece... Nov 22 12:33:13.353758506 (316h29m11.905228957s) [upload=278731]: Next returned nil segment Nov 22 12:33:13.353772043 (316h29m11.905242504s) [upload=278731]: Waiting for error group managing segments... Nov 22 12:33:13.751703718 (316h29m12.303174169s) [noise=true , address=145.79.227.32:4858 , piece_id=K5L3LEUKZWHS36XJHNNY4TYRXCP4DG5UAKCZXPJJTSG26CKUI4HQ, seg_pos={0 0}, upload=278731]: Done uploading piece. err: <nil> Nov 22 12:33:13.753660293 (316h29m12.305130764s) [noise=true , address=145.79.227.26:4674 , piece_id=UP7VIVRDRL6KERJSX3VLWTKGW3ESGCQ7BLYCBXGBB5UNIDE6OHQQ, seg_pos={0 0}, upload=278731]: Done uploading piece. err: <nil> Nov 22 12:33:13.760046123 (316h29m12.311516574s) [noise=true , address=145.79.227.3:4167 , piece_id=4STIVXGU7SOXWAIPG2PXWBLNDD7EYQ56E2DBN2UZ2AZ3TZSXCTIQ, seg_pos={0 0}, upload=278731]: Done uploading piece. err: <nil> Nov 22 12:33:57.373975014 (316h29m55.925445475s) [noise=true , address=77.90.31.13:4644 , piece_id=UPDZGC3CMJXA3AOJX65VLASU37NLFGPIJLCAPN32TEQB5QHTU7EQ, seg_pos={0 0}, upload=278731]: Done uploading piece. err: <nil> Nov 22 12:34:06.122275078 (316h30m4.673745539s) [noise=true , address=45.91.116.13:4181 , piece_id=MQNJEBQFZMHNDKAH2K3JEA7YBK2UK5BGSNCJF3MASF3CKVA5VD6Q, seg_pos={0 0}, upload=278731]: Done uploading piece. err: <nil> Nov 22 12:34:10.949171331 (316h30m9.500641792s) [noise=true , address=77.90.31.21:4890 , piece_id=DUEFPJUTYMD564Z4III63LZM6L4IL2ZTGM3QDZ6CG7WV5APHBNRA, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:1iE3mDnEhGDsMBtmUXC4EwvSrpyHLoybzVXkC9eoFuYEMHeBrd, address:77.90.31.21:4890): protocol: read tcp 188.166.225.170:55900->77.90.31.21:4890: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 12:34:10.949753650 (316h30m9.501224111s) [noise=true , address=45.91.116.12:4237 , piece_id=OG6XREYEGOYVFFMXCGBERRAIEMI2QMUGSBJBYP7GJQZ7FOAI4BIA, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:1FMjHdNGV42a9QZyMmSPTJWQrA8CdSRvXqrimnt6UGVreCzmgF, address:45.91.116.12:4237): protocol: rpc: dial tcp 45.91.116.12:4237: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 12:34:10.950066235 (316h30m9.501536706s) [noise=true , address=77.90.31.19:4863 , piece_id=3PM3Q6X3ADKX2RKIIFDMK57LTJNJHNFK4UYPKFK3ZEQLYQ23EP3A, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:124x1cbCPJRK8YkgSP6UzHymoLu1E1znnfg5WBiD89kSDLi8JeL, address:77.90.31.19:4863): protocol: write tcp 188.166.225.170:45968->77.90.31.19:4863: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 12:34:10.950246558 (316h30m9.50171702s) [noise=true , address=77.90.31.10:4524 , piece_id=Z2IGDHPDU6JAR5CAFQBGYHO3TQ66K4BZOAMXSC7X3RFWLGHOOMTQ, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:12uN7CznoK4AD8FqFkNUndawdGJ9rmjAe7oNnjJkV844ScY87qe, address:77.90.31.10:4524): protocol: write tcp 188.166.225.170:40266->77.90.31.10:4524: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 12:34:10.950593058 (316h30m9.502063529s) [noise=true , address=77.90.31.14:4690 , piece_id=RJ3KXY3HURT7Q4HZYTOLSAL7Y6VMDGGRIQP6XHE4YB6OAA3PJRAA, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:1WPKeCcN1uEhGjZHAJgCoJMaHxAWouGCdaU1vZwAS7kX22rxcv, address:77.90.31.14:4690): protocol: rpc: dial tcp 77.90.31.14:4690: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 12:34:10.950853535 (316h30m9.502324006s) [noise=true , address=45.91.116.14:4213 , piece_id=HPFISTE2N3B4RBDPAZAGVZJ5WYLCPHTCYH4WMJ56IWEHR3ZBEMOQ, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:12tQRqowH4aDr9h3s1WLs1o7na1ueiafwgUeeHfAB6g7mcE6e1t, address:45.91.116.14:4213): protocol: rpc: dial tcp 45.91.116.14:4213: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 12:34:10.951186438 (316h30m9.502656889s) [noise=true , address=45.91.116.6:4000 , piece_id=T4CI3KKNMFPT3ZIODHE7BQ4DVHVH2QTSO6OB7QZPL5SNPXQJE6TA, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:12CuCqHJYbanbuaEMHvtcuQXhvcdahD8eyT2fKDhqtxdSZ6ES4f, address:45.91.116.6:4000): protocol: rpc: dial tcp 45.91.116.6:4000: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 12:34:10.951409864 (316h30m9.502880325s) [noise=true , address=45.91.116.7:4039 , piece_id=HSRY63FXTEPCP2T46V77RVSC24GGRBAYLLU7DTKOLWGVOYHF6YVA, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:16oq1Yv7JfHrg4gdHfhwuMj4UsWeQQPayKV6Pj1NY2CgJhuPr2, address:45.91.116.7:4039): protocol: write tcp 188.166.225.170:36904->45.91.116.7:4039: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 12:34:10.951660191 (316h30m9.503130652s) [noise=true , address=77.90.31.11:4572 , piece_id=G4XSIPSWOLSEOGGFA2NLOSDCGTLZPI3P5LDNNLB5RGMU5OQGWXGQ, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:1ZAfYZ4wgpZVvatj6wtHona3hTaPcLxAmXC1DT2uKQyo8nLQQ7, address:77.90.31.11:4572): protocol: rpc: dial tcp 77.90.31.11:4572: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 12:34:10.952019855 (316h30m9.503490316s) [noise=true , address=45.91.116.9:4274 , piece_id=MXWRAXYLBBGLUVTA5GP3JVX2IRM2UI2IC35D7KX3XMFEBXLIDX5A, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:12g6jFa38dTz5UdgsQaEuM2MSH1H7cxaysWsVmbHkM2Dd2W2iDb, address:45.91.116.9:4274): protocol: rpc: dial tcp 45.91.116.9:4274: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 12:34:10.952290972 (316h30m9.503761443s) [noise=true , address=77.90.31.15:4720 , piece_id=BDQUF6C5TT3ASE3DIN6SSVJHOZAJK4ZEW33OIYPTGB4LDVRELSUA, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:12VbsxFqMB3vVphSAJqYpbBzua7Goq45E6ovJSaExkpLtfkcc74, address:77.90.31.15:4720): protocol: rpc: dial tcp 77.90.31.15:4720: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 12:34:10.952544654 (316h30m9.504015105s) [noise=true , address=45.91.116.10:4110 , piece_id=H6FYA6K4N4AQYU3MN5RGYNBMYFXXBJDBTUCN37EAT7WBRNTX2YQA, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:123KwusATu1ZKtNNVHZZZBE67ZEnVb8puyfQidBi3fEWREqdf1h, address:45.91.116.10:4110): protocol: rpc: dial tcp 45.91.116.10:4110: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 12:34:10.953548155 (316h30m9.505018617s) [noise=true , address=77.90.31.17:4787 , piece_id=IYIEVH6CARRYCARKOLKWVBICQDNDRCTGQMWF3VJZ6YBC3GO5OWHQ, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:1Lietonb5aVbaZQt6wQ2EoxwJkSEn9oL9Dwyrc9mTyomEKikNt, address:77.90.31.17:4787): protocol: read tcp 188.166.225.170:57908->77.90.31.17:4787: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 12:34:10.953579816 (316h30m9.505050277s) [seg_pos={0 0}, upload=278731]: Done waiting for segment. successful: 5 optimal: 14 errs: <nil> Nov 22 12:34:11.195719123 (316h30m9.747189584s) [upload=278731]: Done uploading segments... Nov 22 12:34:11.195932840 (316h30m9.747403301s) [upload=278731]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 5) Duration: 5.980471087s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 5.740338484s trace: https://jaeger.dev.storj.io/trace/6c800b4c3a84b58c FAIL (2025-11-22 13:15:42 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 5) ======LOGS====== N5RGYNBMYFXXBJDBTUCN37EAT7WBRNTX2YQA, seg_pos={0 0}, upload=278731]: Uploading piece... Nov 22 12:33:11.890374814 (316h29m10.441845275s) [noise=true , address=145.79.227.32:4858 , piece_id=K5L3LEUKZWHS36XJHNNY4TYRXCP4DG5UAKCZXPJJTSG26CKUI4HQ, seg_pos={0 0}, upload=278731]: Uploading piece... Nov 22 12:33:11.890535981 (316h29m10.442006442s) [noise=true , address=77.90.31.21:4890 , piece_id=DUEFPJUTYMD564Z4III63LZM6L4IL2ZTGM3QDZ6CG7WV5APHBNRA, seg_pos={0 0}, upload=278731]: Uploading piece... Nov 22 12:33:11.890774064 (316h29m10.442244525s) [noise=true , address=45.91.116.14:4213 , piece_id=HPFISTE2N3B4RBDPAZAGVZJ5WYLCPHTCYH4WMJ56IWEHR3ZBEMOQ, seg_pos={0 0}, upload=278731]: Uploading piece... Nov 22 12:33:11.890934851 (316h29m10.442405312s) [noise=true , address=45.91.116.6:4000 , piece_id=T4CI3KKNMFPT3ZIODHE7BQ4DVHVH2QTSO6OB7QZPL5SNPXQJE6TA, seg_pos={0 0}, upload=278731]: Uploading piece... Nov 22 12:33:11.891167002 (316h29m10.442637463s) [noise=true , address=77.90.31.17:4787 , piece_id=IYIEVH6CARRYCARKOLKWVBICQDNDRCTGQMWF3VJZ6YBC3GO5OWHQ, seg_pos={0 0}, upload=278731]: Uploading piece... Nov 22 12:33:11.891369828 (316h29m10.442840289s) [noise=true , address=45.91.116.7:4039 , piece_id=HSRY63FXTEPCP2T46V77RVSC24GGRBAYLLU7DTKOLWGVOYHF6YVA, seg_pos={0 0}, upload=278731]: Uploading piece... Nov 22 12:33:11.891556875 (316h29m10.443027336s) [noise=true , address=77.90.31.11:4572 , piece_id=G4XSIPSWOLSEOGGFA2NLOSDCGTLZPI3P5LDNNLB5RGMU5OQGWXGQ, seg_pos={0 0}, upload=278731]: Uploading piece... Nov 22 12:33:11.891896812 (316h29m10.443367273s) [noise=true , address=77.90.31.19:4863 , piece_id=3PM3Q6X3ADKX2RKIIFDMK57LTJNJHNFK4UYPKFK3ZEQLYQ23EP3A, seg_pos={0 0}, upload=278731]: Uploading piece... Nov 22 12:33:11.892019235 (316h29m10.443489696s) [noise=true , address=77.90.31.10:4524 , piece_id=Z2IGDHPDU6JAR5CAFQBGYHO3TQ66K4BZOAMXSC7X3RFWLGHOOMTQ, seg_pos={0 0}, upload=278731]: Uploading piece... Nov 22 12:33:11.892316521 (316h29m10.443786982s) [noise=true , address=77.90.31.13:4644 , piece_id=UPDZGC3CMJXA3AOJX65VLASU37NLFGPIJLCAPN32TEQB5QHTU7EQ, seg_pos={0 0}, upload=278731]: Uploading piece... Nov 22 12:33:11.892682318 (316h29m10.444152779s) [noise=true , address=45.91.116.9:4274 , piece_id=MXWRAXYLBBGLUVTA5GP3JVX2IRM2UI2IC35D7KX3XMFEBXLIDX5A, seg_pos={0 0}, upload=278731]: Uploading piece... Nov 22 12:33:13.353758506 (316h29m11.905228957s) [upload=278731]: Next returned nil segment Nov 22 12:33:13.353772043 (316h29m11.905242504s) [upload=278731]: Waiting for error group managing segments... Nov 22 12:33:13.751703718 (316h29m12.303174169s) [noise=true , address=145.79.227.32:4858 , piece_id=K5L3LEUKZWHS36XJHNNY4TYRXCP4DG5UAKCZXPJJTSG26CKUI4HQ, seg_pos={0 0}, upload=278731]: Done uploading piece. err: <nil> Nov 22 12:33:13.753660293 (316h29m12.305130764s) [noise=true , address=145.79.227.26:4674 , piece_id=UP7VIVRDRL6KERJSX3VLWTKGW3ESGCQ7BLYCBXGBB5UNIDE6OHQQ, seg_pos={0 0}, upload=278731]: Done uploading piece. err: <nil> Nov 22 12:33:13.760046123 (316h29m12.311516574s) [noise=true , address=145.79.227.3:4167 , piece_id=4STIVXGU7SOXWAIPG2PXWBLNDD7EYQ56E2DBN2UZ2AZ3TZSXCTIQ, seg_pos={0 0}, upload=278731]: Done uploading piece. err: <nil> Nov 22 12:33:57.373975014 (316h29m55.925445475s) [noise=true , address=77.90.31.13:4644 , piece_id=UPDZGC3CMJXA3AOJX65VLASU37NLFGPIJLCAPN32TEQB5QHTU7EQ, seg_pos={0 0}, upload=278731]: Done uploading piece. err: <nil> Nov 22 12:34:06.122275078 (316h30m4.673745539s) [noise=true , address=45.91.116.13:4181 , piece_id=MQNJEBQFZMHNDKAH2K3JEA7YBK2UK5BGSNCJF3MASF3CKVA5VD6Q, seg_pos={0 0}, upload=278731]: Done uploading piece. err: <nil> Nov 22 12:34:10.949171331 (316h30m9.500641792s) [noise=true , address=77.90.31.21:4890 , piece_id=DUEFPJUTYMD564Z4III63LZM6L4IL2ZTGM3QDZ6CG7WV5APHBNRA, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:1iE3mDnEhGDsMBtmUXC4EwvSrpyHLoybzVXkC9eoFuYEMHeBrd, address:77.90.31.21:4890): protocol: read tcp 188.166.225.170:55900->77.90.31.21:4890: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 12:34:10.949753650 (316h30m9.501224111s) [noise=true , address=45.91.116.12:4237 , piece_id=OG6XREYEGOYVFFMXCGBERRAIEMI2QMUGSBJBYP7GJQZ7FOAI4BIA, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:1FMjHdNGV42a9QZyMmSPTJWQrA8CdSRvXqrimnt6UGVreCzmgF, address:45.91.116.12:4237): protocol: rpc: dial tcp 45.91.116.12:4237: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 12:34:10.950066235 (316h30m9.501536706s) [noise=true , address=77.90.31.19:4863 , piece_id=3PM3Q6X3ADKX2RKIIFDMK57LTJNJHNFK4UYPKFK3ZEQLYQ23EP3A, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:124x1cbCPJRK8YkgSP6UzHymoLu1E1znnfg5WBiD89kSDLi8JeL, address:77.90.31.19:4863): protocol: write tcp 188.166.225.170:45968->77.90.31.19:4863: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 12:34:10.950246558 (316h30m9.50171702s) [noise=true , address=77.90.31.10:4524 , piece_id=Z2IGDHPDU6JAR5CAFQBGYHO3TQ66K4BZOAMXSC7X3RFWLGHOOMTQ, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:12uN7CznoK4AD8FqFkNUndawdGJ9rmjAe7oNnjJkV844ScY87qe, address:77.90.31.10:4524): protocol: write tcp 188.166.225.170:40266->77.90.31.10:4524: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 12:34:10.950593058 (316h30m9.502063529s) [noise=true , address=77.90.31.14:4690 , piece_id=RJ3KXY3HURT7Q4HZYTOLSAL7Y6VMDGGRIQP6XHE4YB6OAA3PJRAA, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:1WPKeCcN1uEhGjZHAJgCoJMaHxAWouGCdaU1vZwAS7kX22rxcv, address:77.90.31.14:4690): protocol: rpc: dial tcp 77.90.31.14:4690: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 12:34:10.950853535 (316h30m9.502324006s) [noise=true , address=45.91.116.14:4213 , piece_id=HPFISTE2N3B4RBDPAZAGVZJ5WYLCPHTCYH4WMJ56IWEHR3ZBEMOQ, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:12tQRqowH4aDr9h3s1WLs1o7na1ueiafwgUeeHfAB6g7mcE6e1t, address:45.91.116.14:4213): protocol: rpc: dial tcp 45.91.116.14:4213: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 12:34:10.951186438 (316h30m9.502656889s) [noise=true , address=45.91.116.6:4000 , piece_id=T4CI3KKNMFPT3ZIODHE7BQ4DVHVH2QTSO6OB7QZPL5SNPXQJE6TA, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:12CuCqHJYbanbuaEMHvtcuQXhvcdahD8eyT2fKDhqtxdSZ6ES4f, address:45.91.116.6:4000): protocol: rpc: dial tcp 45.91.116.6:4000: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 12:34:10.951409864 (316h30m9.502880325s) [noise=true , address=45.91.116.7:4039 , piece_id=HSRY63FXTEPCP2T46V77RVSC24GGRBAYLLU7DTKOLWGVOYHF6YVA, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:16oq1Yv7JfHrg4gdHfhwuMj4UsWeQQPayKV6Pj1NY2CgJhuPr2, address:45.91.116.7:4039): protocol: write tcp 188.166.225.170:36904->45.91.116.7:4039: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 12:34:10.951660191 (316h30m9.503130652s) [noise=true , address=77.90.31.11:4572 , piece_id=G4XSIPSWOLSEOGGFA2NLOSDCGTLZPI3P5LDNNLB5RGMU5OQGWXGQ, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:1ZAfYZ4wgpZVvatj6wtHona3hTaPcLxAmXC1DT2uKQyo8nLQQ7, address:77.90.31.11:4572): protocol: rpc: dial tcp 77.90.31.11:4572: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 12:34:10.952019855 (316h30m9.503490316s) [noise=true , address=45.91.116.9:4274 , piece_id=MXWRAXYLBBGLUVTA5GP3JVX2IRM2UI2IC35D7KX3XMFEBXLIDX5A, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:12g6jFa38dTz5UdgsQaEuM2MSH1H7cxaysWsVmbHkM2Dd2W2iDb, address:45.91.116.9:4274): protocol: rpc: dial tcp 45.91.116.9:4274: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 12:34:10.952290972 (316h30m9.503761443s) [noise=true , address=77.90.31.15:4720 , piece_id=BDQUF6C5TT3ASE3DIN6SSVJHOZAJK4ZEW33OIYPTGB4LDVRELSUA, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:12VbsxFqMB3vVphSAJqYpbBzua7Goq45E6ovJSaExkpLtfkcc74, address:77.90.31.15:4720): protocol: rpc: dial tcp 77.90.31.15:4720: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 12:34:10.952544654 (316h30m9.504015105s) [noise=true , address=45.91.116.10:4110 , piece_id=H6FYA6K4N4AQYU3MN5RGYNBMYFXXBJDBTUCN37EAT7WBRNTX2YQA, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:123KwusATu1ZKtNNVHZZZBE67ZEnVb8puyfQidBi3fEWREqdf1h, address:45.91.116.10:4110): protocol: rpc: dial tcp 45.91.116.10:4110: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 12:34:10.953548155 (316h30m9.505018617s) [noise=true , address=77.90.31.17:4787 , piece_id=IYIEVH6CARRYCARKOLKWVBICQDNDRCTGQMWF3VJZ6YBC3GO5OWHQ, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:1Lietonb5aVbaZQt6wQ2EoxwJkSEn9oL9Dwyrc9mTyomEKikNt, address:77.90.31.17:4787): protocol: read tcp 188.166.225.170:57908->77.90.31.17:4787: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 12:34:10.953579816 (316h30m9.505050277s) [seg_pos={0 0}, upload=278731]: Done waiting for segment. successful: 5 optimal: 14 errs: <nil> Nov 22 12:34:11.195719123 (316h30m9.747189584s) [upload=278731]: Done uploading segments... Nov 22 12:34:11.195932840 (316h30m9.747403301s) [upload=278731]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 5) Duration: 5.740372901s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [.XX.............................................X...........] duration: 4.663403745s trace: https://jaeger.dev.storj.io/trace/38596674a32f39c1 FAIL (2025-11-22 13:15:41 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.663433973s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 15.415471185s trace: https://jaeger.dev.storj.io/trace/92df32505639913 FAIL (2025-11-22 13:15:31 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 15.415504338s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [X.XX.............................................X..........] duration: 4.836406933s trace: https://jaeger.dev.storj.io/trace/38596674a32f39c1 FAIL (2025-11-22 13:15:21 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.836446018s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 5.900449794s trace: https://jaeger.dev.storj.io/trace/6c800b4c3a84b58c FAIL (2025-11-22 13:14:42 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 5) ======LOGS====== N5RGYNBMYFXXBJDBTUCN37EAT7WBRNTX2YQA, seg_pos={0 0}, upload=278731]: Uploading piece... Nov 22 12:33:11.890374814 (316h29m10.441845275s) [noise=true , address=145.79.227.32:4858 , piece_id=K5L3LEUKZWHS36XJHNNY4TYRXCP4DG5UAKCZXPJJTSG26CKUI4HQ, seg_pos={0 0}, upload=278731]: Uploading piece... Nov 22 12:33:11.890535981 (316h29m10.442006442s) [noise=true , address=77.90.31.21:4890 , piece_id=DUEFPJUTYMD564Z4III63LZM6L4IL2ZTGM3QDZ6CG7WV5APHBNRA, seg_pos={0 0}, upload=278731]: Uploading piece... Nov 22 12:33:11.890774064 (316h29m10.442244525s) [noise=true , address=45.91.116.14:4213 , piece_id=HPFISTE2N3B4RBDPAZAGVZJ5WYLCPHTCYH4WMJ56IWEHR3ZBEMOQ, seg_pos={0 0}, upload=278731]: Uploading piece... Nov 22 12:33:11.890934851 (316h29m10.442405312s) [noise=true , address=45.91.116.6:4000 , piece_id=T4CI3KKNMFPT3ZIODHE7BQ4DVHVH2QTSO6OB7QZPL5SNPXQJE6TA, seg_pos={0 0}, upload=278731]: Uploading piece... Nov 22 12:33:11.891167002 (316h29m10.442637463s) [noise=true , address=77.90.31.17:4787 , piece_id=IYIEVH6CARRYCARKOLKWVBICQDNDRCTGQMWF3VJZ6YBC3GO5OWHQ, seg_pos={0 0}, upload=278731]: Uploading piece... Nov 22 12:33:11.891369828 (316h29m10.442840289s) [noise=true , address=45.91.116.7:4039 , piece_id=HSRY63FXTEPCP2T46V77RVSC24GGRBAYLLU7DTKOLWGVOYHF6YVA, seg_pos={0 0}, upload=278731]: Uploading piece... Nov 22 12:33:11.891556875 (316h29m10.443027336s) [noise=true , address=77.90.31.11:4572 , piece_id=G4XSIPSWOLSEOGGFA2NLOSDCGTLZPI3P5LDNNLB5RGMU5OQGWXGQ, seg_pos={0 0}, upload=278731]: Uploading piece... Nov 22 12:33:11.891896812 (316h29m10.443367273s) [noise=true , address=77.90.31.19:4863 , piece_id=3PM3Q6X3ADKX2RKIIFDMK57LTJNJHNFK4UYPKFK3ZEQLYQ23EP3A, seg_pos={0 0}, upload=278731]: Uploading piece... Nov 22 12:33:11.892019235 (316h29m10.443489696s) [noise=true , address=77.90.31.10:4524 , piece_id=Z2IGDHPDU6JAR5CAFQBGYHO3TQ66K4BZOAMXSC7X3RFWLGHOOMTQ, seg_pos={0 0}, upload=278731]: Uploading piece... Nov 22 12:33:11.892316521 (316h29m10.443786982s) [noise=true , address=77.90.31.13:4644 , piece_id=UPDZGC3CMJXA3AOJX65VLASU37NLFGPIJLCAPN32TEQB5QHTU7EQ, seg_pos={0 0}, upload=278731]: Uploading piece... Nov 22 12:33:11.892682318 (316h29m10.444152779s) [noise=true , address=45.91.116.9:4274 , piece_id=MXWRAXYLBBGLUVTA5GP3JVX2IRM2UI2IC35D7KX3XMFEBXLIDX5A, seg_pos={0 0}, upload=278731]: Uploading piece... Nov 22 12:33:13.353758506 (316h29m11.905228957s) [upload=278731]: Next returned nil segment Nov 22 12:33:13.353772043 (316h29m11.905242504s) [upload=278731]: Waiting for error group managing segments... Nov 22 12:33:13.751703718 (316h29m12.303174169s) [noise=true , address=145.79.227.32:4858 , piece_id=K5L3LEUKZWHS36XJHNNY4TYRXCP4DG5UAKCZXPJJTSG26CKUI4HQ, seg_pos={0 0}, upload=278731]: Done uploading piece. err: <nil> Nov 22 12:33:13.753660293 (316h29m12.305130764s) [noise=true , address=145.79.227.26:4674 , piece_id=UP7VIVRDRL6KERJSX3VLWTKGW3ESGCQ7BLYCBXGBB5UNIDE6OHQQ, seg_pos={0 0}, upload=278731]: Done uploading piece. err: <nil> Nov 22 12:33:13.760046123 (316h29m12.311516574s) [noise=true , address=145.79.227.3:4167 , piece_id=4STIVXGU7SOXWAIPG2PXWBLNDD7EYQ56E2DBN2UZ2AZ3TZSXCTIQ, seg_pos={0 0}, upload=278731]: Done uploading piece. err: <nil> Nov 22 12:33:57.373975014 (316h29m55.925445475s) [noise=true , address=77.90.31.13:4644 , piece_id=UPDZGC3CMJXA3AOJX65VLASU37NLFGPIJLCAPN32TEQB5QHTU7EQ, seg_pos={0 0}, upload=278731]: Done uploading piece. err: <nil> Nov 22 12:34:06.122275078 (316h30m4.673745539s) [noise=true , address=45.91.116.13:4181 , piece_id=MQNJEBQFZMHNDKAH2K3JEA7YBK2UK5BGSNCJF3MASF3CKVA5VD6Q, seg_pos={0 0}, upload=278731]: Done uploading piece. err: <nil> Nov 22 12:34:10.949171331 (316h30m9.500641792s) [noise=true , address=77.90.31.21:4890 , piece_id=DUEFPJUTYMD564Z4III63LZM6L4IL2ZTGM3QDZ6CG7WV5APHBNRA, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:1iE3mDnEhGDsMBtmUXC4EwvSrpyHLoybzVXkC9eoFuYEMHeBrd, address:77.90.31.21:4890): protocol: read tcp 188.166.225.170:55900->77.90.31.21:4890: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 12:34:10.949753650 (316h30m9.501224111s) [noise=true , address=45.91.116.12:4237 , piece_id=OG6XREYEGOYVFFMXCGBERRAIEMI2QMUGSBJBYP7GJQZ7FOAI4BIA, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:1FMjHdNGV42a9QZyMmSPTJWQrA8CdSRvXqrimnt6UGVreCzmgF, address:45.91.116.12:4237): protocol: rpc: dial tcp 45.91.116.12:4237: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 12:34:10.950066235 (316h30m9.501536706s) [noise=true , address=77.90.31.19:4863 , piece_id=3PM3Q6X3ADKX2RKIIFDMK57LTJNJHNFK4UYPKFK3ZEQLYQ23EP3A, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:124x1cbCPJRK8YkgSP6UzHymoLu1E1znnfg5WBiD89kSDLi8JeL, address:77.90.31.19:4863): protocol: write tcp 188.166.225.170:45968->77.90.31.19:4863: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 12:34:10.950246558 (316h30m9.50171702s) [noise=true , address=77.90.31.10:4524 , piece_id=Z2IGDHPDU6JAR5CAFQBGYHO3TQ66K4BZOAMXSC7X3RFWLGHOOMTQ, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:12uN7CznoK4AD8FqFkNUndawdGJ9rmjAe7oNnjJkV844ScY87qe, address:77.90.31.10:4524): protocol: write tcp 188.166.225.170:40266->77.90.31.10:4524: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 12:34:10.950593058 (316h30m9.502063529s) [noise=true , address=77.90.31.14:4690 , piece_id=RJ3KXY3HURT7Q4HZYTOLSAL7Y6VMDGGRIQP6XHE4YB6OAA3PJRAA, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:1WPKeCcN1uEhGjZHAJgCoJMaHxAWouGCdaU1vZwAS7kX22rxcv, address:77.90.31.14:4690): protocol: rpc: dial tcp 77.90.31.14:4690: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 12:34:10.950853535 (316h30m9.502324006s) [noise=true , address=45.91.116.14:4213 , piece_id=HPFISTE2N3B4RBDPAZAGVZJ5WYLCPHTCYH4WMJ56IWEHR3ZBEMOQ, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:12tQRqowH4aDr9h3s1WLs1o7na1ueiafwgUeeHfAB6g7mcE6e1t, address:45.91.116.14:4213): protocol: rpc: dial tcp 45.91.116.14:4213: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 12:34:10.951186438 (316h30m9.502656889s) [noise=true , address=45.91.116.6:4000 , piece_id=T4CI3KKNMFPT3ZIODHE7BQ4DVHVH2QTSO6OB7QZPL5SNPXQJE6TA, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:12CuCqHJYbanbuaEMHvtcuQXhvcdahD8eyT2fKDhqtxdSZ6ES4f, address:45.91.116.6:4000): protocol: rpc: dial tcp 45.91.116.6:4000: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 12:34:10.951409864 (316h30m9.502880325s) [noise=true , address=45.91.116.7:4039 , piece_id=HSRY63FXTEPCP2T46V77RVSC24GGRBAYLLU7DTKOLWGVOYHF6YVA, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:16oq1Yv7JfHrg4gdHfhwuMj4UsWeQQPayKV6Pj1NY2CgJhuPr2, address:45.91.116.7:4039): protocol: write tcp 188.166.225.170:36904->45.91.116.7:4039: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 12:34:10.951660191 (316h30m9.503130652s) [noise=true , address=77.90.31.11:4572 , piece_id=G4XSIPSWOLSEOGGFA2NLOSDCGTLZPI3P5LDNNLB5RGMU5OQGWXGQ, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:1ZAfYZ4wgpZVvatj6wtHona3hTaPcLxAmXC1DT2uKQyo8nLQQ7, address:77.90.31.11:4572): protocol: rpc: dial tcp 77.90.31.11:4572: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 12:34:10.952019855 (316h30m9.503490316s) [noise=true , address=45.91.116.9:4274 , piece_id=MXWRAXYLBBGLUVTA5GP3JVX2IRM2UI2IC35D7KX3XMFEBXLIDX5A, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:12g6jFa38dTz5UdgsQaEuM2MSH1H7cxaysWsVmbHkM2Dd2W2iDb, address:45.91.116.9:4274): protocol: rpc: dial tcp 45.91.116.9:4274: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 12:34:10.952290972 (316h30m9.503761443s) [noise=true , address=77.90.31.15:4720 , piece_id=BDQUF6C5TT3ASE3DIN6SSVJHOZAJK4ZEW33OIYPTGB4LDVRELSUA, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:12VbsxFqMB3vVphSAJqYpbBzua7Goq45E6ovJSaExkpLtfkcc74, address:77.90.31.15:4720): protocol: rpc: dial tcp 77.90.31.15:4720: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 12:34:10.952544654 (316h30m9.504015105s) [noise=true , address=45.91.116.10:4110 , piece_id=H6FYA6K4N4AQYU3MN5RGYNBMYFXXBJDBTUCN37EAT7WBRNTX2YQA, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:123KwusATu1ZKtNNVHZZZBE67ZEnVb8puyfQidBi3fEWREqdf1h, address:45.91.116.10:4110): protocol: rpc: dial tcp 45.91.116.10:4110: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 12:34:10.953548155 (316h30m9.505018617s) [noise=true , address=77.90.31.17:4787 , piece_id=IYIEVH6CARRYCARKOLKWVBICQDNDRCTGQMWF3VJZ6YBC3GO5OWHQ, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:1Lietonb5aVbaZQt6wQ2EoxwJkSEn9oL9Dwyrc9mTyomEKikNt, address:77.90.31.17:4787): protocol: read tcp 188.166.225.170:57908->77.90.31.17:4787: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 12:34:10.953579816 (316h30m9.505050277s) [seg_pos={0 0}, upload=278731]: Done waiting for segment. successful: 5 optimal: 14 errs: <nil> Nov 22 12:34:11.195719123 (316h30m9.747189584s) [upload=278731]: Done uploading segments... Nov 22 12:34:11.195932840 (316h30m9.747403301s) [upload=278731]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 5) Duration: 5.900479229s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [.X.XX.............................................X.........] duration: 4.81895442s trace: https://jaeger.dev.storj.io/trace/38596674a32f39c1 FAIL (2025-11-22 13:14:35 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.81898623s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 6.727423203s trace: https://jaeger.dev.storj.io/trace/92df32505639913 FAIL (2025-11-22 13:14:34 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.727454683s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 5.74148101s trace: https://jaeger.dev.storj.io/trace/6c800b4c3a84b58c FAIL (2025-11-22 13:13:55 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 5) ======LOGS====== N5RGYNBMYFXXBJDBTUCN37EAT7WBRNTX2YQA, seg_pos={0 0}, upload=278731]: Uploading piece... Nov 22 12:33:11.890374814 (316h29m10.441845275s) [noise=true , address=145.79.227.32:4858 , piece_id=K5L3LEUKZWHS36XJHNNY4TYRXCP4DG5UAKCZXPJJTSG26CKUI4HQ, seg_pos={0 0}, upload=278731]: Uploading piece... Nov 22 12:33:11.890535981 (316h29m10.442006442s) [noise=true , address=77.90.31.21:4890 , piece_id=DUEFPJUTYMD564Z4III63LZM6L4IL2ZTGM3QDZ6CG7WV5APHBNRA, seg_pos={0 0}, upload=278731]: Uploading piece... Nov 22 12:33:11.890774064 (316h29m10.442244525s) [noise=true , address=45.91.116.14:4213 , piece_id=HPFISTE2N3B4RBDPAZAGVZJ5WYLCPHTCYH4WMJ56IWEHR3ZBEMOQ, seg_pos={0 0}, upload=278731]: Uploading piece... Nov 22 12:33:11.890934851 (316h29m10.442405312s) [noise=true , address=45.91.116.6:4000 , piece_id=T4CI3KKNMFPT3ZIODHE7BQ4DVHVH2QTSO6OB7QZPL5SNPXQJE6TA, seg_pos={0 0}, upload=278731]: Uploading piece... Nov 22 12:33:11.891167002 (316h29m10.442637463s) [noise=true , address=77.90.31.17:4787 , piece_id=IYIEVH6CARRYCARKOLKWVBICQDNDRCTGQMWF3VJZ6YBC3GO5OWHQ, seg_pos={0 0}, upload=278731]: Uploading piece... Nov 22 12:33:11.891369828 (316h29m10.442840289s) [noise=true , address=45.91.116.7:4039 , piece_id=HSRY63FXTEPCP2T46V77RVSC24GGRBAYLLU7DTKOLWGVOYHF6YVA, seg_pos={0 0}, upload=278731]: Uploading piece... Nov 22 12:33:11.891556875 (316h29m10.443027336s) [noise=true , address=77.90.31.11:4572 , piece_id=G4XSIPSWOLSEOGGFA2NLOSDCGTLZPI3P5LDNNLB5RGMU5OQGWXGQ, seg_pos={0 0}, upload=278731]: Uploading piece... Nov 22 12:33:11.891896812 (316h29m10.443367273s) [noise=true , address=77.90.31.19:4863 , piece_id=3PM3Q6X3ADKX2RKIIFDMK57LTJNJHNFK4UYPKFK3ZEQLYQ23EP3A, seg_pos={0 0}, upload=278731]: Uploading piece... Nov 22 12:33:11.892019235 (316h29m10.443489696s) [noise=true , address=77.90.31.10:4524 , piece_id=Z2IGDHPDU6JAR5CAFQBGYHO3TQ66K4BZOAMXSC7X3RFWLGHOOMTQ, seg_pos={0 0}, upload=278731]: Uploading piece... Nov 22 12:33:11.892316521 (316h29m10.443786982s) [noise=true , address=77.90.31.13:4644 , piece_id=UPDZGC3CMJXA3AOJX65VLASU37NLFGPIJLCAPN32TEQB5QHTU7EQ, seg_pos={0 0}, upload=278731]: Uploading piece... Nov 22 12:33:11.892682318 (316h29m10.444152779s) [noise=true , address=45.91.116.9:4274 , piece_id=MXWRAXYLBBGLUVTA5GP3JVX2IRM2UI2IC35D7KX3XMFEBXLIDX5A, seg_pos={0 0}, upload=278731]: Uploading piece... Nov 22 12:33:13.353758506 (316h29m11.905228957s) [upload=278731]: Next returned nil segment Nov 22 12:33:13.353772043 (316h29m11.905242504s) [upload=278731]: Waiting for error group managing segments... Nov 22 12:33:13.751703718 (316h29m12.303174169s) [noise=true , address=145.79.227.32:4858 , piece_id=K5L3LEUKZWHS36XJHNNY4TYRXCP4DG5UAKCZXPJJTSG26CKUI4HQ, seg_pos={0 0}, upload=278731]: Done uploading piece. err: <nil> Nov 22 12:33:13.753660293 (316h29m12.305130764s) [noise=true , address=145.79.227.26:4674 , piece_id=UP7VIVRDRL6KERJSX3VLWTKGW3ESGCQ7BLYCBXGBB5UNIDE6OHQQ, seg_pos={0 0}, upload=278731]: Done uploading piece. err: <nil> Nov 22 12:33:13.760046123 (316h29m12.311516574s) [noise=true , address=145.79.227.3:4167 , piece_id=4STIVXGU7SOXWAIPG2PXWBLNDD7EYQ56E2DBN2UZ2AZ3TZSXCTIQ, seg_pos={0 0}, upload=278731]: Done uploading piece. err: <nil> Nov 22 12:33:57.373975014 (316h29m55.925445475s) [noise=true , address=77.90.31.13:4644 , piece_id=UPDZGC3CMJXA3AOJX65VLASU37NLFGPIJLCAPN32TEQB5QHTU7EQ, seg_pos={0 0}, upload=278731]: Done uploading piece. err: <nil> Nov 22 12:34:06.122275078 (316h30m4.673745539s) [noise=true , address=45.91.116.13:4181 , piece_id=MQNJEBQFZMHNDKAH2K3JEA7YBK2UK5BGSNCJF3MASF3CKVA5VD6Q, seg_pos={0 0}, upload=278731]: Done uploading piece. err: <nil> Nov 22 12:34:10.949171331 (316h30m9.500641792s) [noise=true , address=77.90.31.21:4890 , piece_id=DUEFPJUTYMD564Z4III63LZM6L4IL2ZTGM3QDZ6CG7WV5APHBNRA, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:1iE3mDnEhGDsMBtmUXC4EwvSrpyHLoybzVXkC9eoFuYEMHeBrd, address:77.90.31.21:4890): protocol: read tcp 188.166.225.170:55900->77.90.31.21:4890: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 12:34:10.949753650 (316h30m9.501224111s) [noise=true , address=45.91.116.12:4237 , piece_id=OG6XREYEGOYVFFMXCGBERRAIEMI2QMUGSBJBYP7GJQZ7FOAI4BIA, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:1FMjHdNGV42a9QZyMmSPTJWQrA8CdSRvXqrimnt6UGVreCzmgF, address:45.91.116.12:4237): protocol: rpc: dial tcp 45.91.116.12:4237: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 12:34:10.950066235 (316h30m9.501536706s) [noise=true , address=77.90.31.19:4863 , piece_id=3PM3Q6X3ADKX2RKIIFDMK57LTJNJHNFK4UYPKFK3ZEQLYQ23EP3A, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:124x1cbCPJRK8YkgSP6UzHymoLu1E1znnfg5WBiD89kSDLi8JeL, address:77.90.31.19:4863): protocol: write tcp 188.166.225.170:45968->77.90.31.19:4863: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 12:34:10.950246558 (316h30m9.50171702s) [noise=true , address=77.90.31.10:4524 , piece_id=Z2IGDHPDU6JAR5CAFQBGYHO3TQ66K4BZOAMXSC7X3RFWLGHOOMTQ, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:12uN7CznoK4AD8FqFkNUndawdGJ9rmjAe7oNnjJkV844ScY87qe, address:77.90.31.10:4524): protocol: write tcp 188.166.225.170:40266->77.90.31.10:4524: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 12:34:10.950593058 (316h30m9.502063529s) [noise=true , address=77.90.31.14:4690 , piece_id=RJ3KXY3HURT7Q4HZYTOLSAL7Y6VMDGGRIQP6XHE4YB6OAA3PJRAA, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:1WPKeCcN1uEhGjZHAJgCoJMaHxAWouGCdaU1vZwAS7kX22rxcv, address:77.90.31.14:4690): protocol: rpc: dial tcp 77.90.31.14:4690: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 12:34:10.950853535 (316h30m9.502324006s) [noise=true , address=45.91.116.14:4213 , piece_id=HPFISTE2N3B4RBDPAZAGVZJ5WYLCPHTCYH4WMJ56IWEHR3ZBEMOQ, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:12tQRqowH4aDr9h3s1WLs1o7na1ueiafwgUeeHfAB6g7mcE6e1t, address:45.91.116.14:4213): protocol: rpc: dial tcp 45.91.116.14:4213: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 12:34:10.951186438 (316h30m9.502656889s) [noise=true , address=45.91.116.6:4000 , piece_id=T4CI3KKNMFPT3ZIODHE7BQ4DVHVH2QTSO6OB7QZPL5SNPXQJE6TA, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:12CuCqHJYbanbuaEMHvtcuQXhvcdahD8eyT2fKDhqtxdSZ6ES4f, address:45.91.116.6:4000): protocol: rpc: dial tcp 45.91.116.6:4000: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 12:34:10.951409864 (316h30m9.502880325s) [noise=true , address=45.91.116.7:4039 , piece_id=HSRY63FXTEPCP2T46V77RVSC24GGRBAYLLU7DTKOLWGVOYHF6YVA, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:16oq1Yv7JfHrg4gdHfhwuMj4UsWeQQPayKV6Pj1NY2CgJhuPr2, address:45.91.116.7:4039): protocol: write tcp 188.166.225.170:36904->45.91.116.7:4039: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 12:34:10.951660191 (316h30m9.503130652s) [noise=true , address=77.90.31.11:4572 , piece_id=G4XSIPSWOLSEOGGFA2NLOSDCGTLZPI3P5LDNNLB5RGMU5OQGWXGQ, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:1ZAfYZ4wgpZVvatj6wtHona3hTaPcLxAmXC1DT2uKQyo8nLQQ7, address:77.90.31.11:4572): protocol: rpc: dial tcp 77.90.31.11:4572: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 12:34:10.952019855 (316h30m9.503490316s) [noise=true , address=45.91.116.9:4274 , piece_id=MXWRAXYLBBGLUVTA5GP3JVX2IRM2UI2IC35D7KX3XMFEBXLIDX5A, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:12g6jFa38dTz5UdgsQaEuM2MSH1H7cxaysWsVmbHkM2Dd2W2iDb, address:45.91.116.9:4274): protocol: rpc: dial tcp 45.91.116.9:4274: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 12:34:10.952290972 (316h30m9.503761443s) [noise=true , address=77.90.31.15:4720 , piece_id=BDQUF6C5TT3ASE3DIN6SSVJHOZAJK4ZEW33OIYPTGB4LDVRELSUA, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:12VbsxFqMB3vVphSAJqYpbBzua7Goq45E6ovJSaExkpLtfkcc74, address:77.90.31.15:4720): protocol: rpc: dial tcp 77.90.31.15:4720: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 12:34:10.952544654 (316h30m9.504015105s) [noise=true , address=45.91.116.10:4110 , piece_id=H6FYA6K4N4AQYU3MN5RGYNBMYFXXBJDBTUCN37EAT7WBRNTX2YQA, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:123KwusATu1ZKtNNVHZZZBE67ZEnVb8puyfQidBi3fEWREqdf1h, address:45.91.116.10:4110): protocol: rpc: dial tcp 45.91.116.10:4110: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 12:34:10.953548155 (316h30m9.505018617s) [noise=true , address=77.90.31.17:4787 , piece_id=IYIEVH6CARRYCARKOLKWVBICQDNDRCTGQMWF3VJZ6YBC3GO5OWHQ, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:1Lietonb5aVbaZQt6wQ2EoxwJkSEn9oL9Dwyrc9mTyomEKikNt, address:77.90.31.17:4787): protocol: read tcp 188.166.225.170:57908->77.90.31.17:4787: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 12:34:10.953579816 (316h30m9.505050277s) [seg_pos={0 0}, upload=278731]: Done waiting for segment. successful: 5 optimal: 14 errs: <nil> Nov 22 12:34:11.195719123 (316h30m9.747189584s) [upload=278731]: Done uploading segments... Nov 22 12:34:11.195932840 (316h30m9.747403301s) [upload=278731]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 5) Duration: 5.741510325s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 13.464722092s trace: https://jaeger.dev.storj.io/trace/92df32505639913 FAIL (2025-11-22 13:13:35 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 13.464753751s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 5.60436167s trace: https://jaeger.dev.storj.io/trace/5ae3d2bf80495000 FAIL (2025-11-22 13:13:30 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 22 12:31:07.276308940 (316h27m5.827779401s) [upload=278701]: Starting upload Nov 22 12:31:07.276483523 (316h27m5.827953994s) [upload=278701]: Done starting upload Nov 22 12:31:07.276562243 (316h27m5.828032704s) [upload=278701]: Uploading segments... Nov 22 12:31:07.276950593 (316h27m5.828421054s) [upload=278701]: Got next segment. Inline: false Nov 22 12:31:07.277027639 (316h27m5.8284981s) [upload=278701]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 22 12:31:07.277034181 (316h27m5.828504643s) [upload=278701]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 22 12:31:07.520212027 (316h27m6.071682488s) [seg_pos={0 0}, upload=278701]: Begin upload segment... Nov 22 12:31:07.520269256 (316h27m6.071739717s) [seg_pos={0 0}, upload=278701]: Done begin upload segment. Nov 22 12:31:07.520308992 (316h27m6.071779453s) [noise=true , address=145.79.227.27:4689 , piece_id=76S65KEZ7I2E76RR2JXOZ3ONGXCWSJA5XB7XQVFEEKGGNC34Z4EQ, seg_pos={0 0}, upload=278701]: Uploading piece... Nov 22 12:31:07.520426746 (316h27m6.071897207s) [noise=true , address=145.79.227.21:4474 , piece_id=PCRWYQVCAZCPOILAEDHJGDBFKX4GEJ6JXCF7FUS7OVFRUTDPJI4Q, seg_pos={0 0}, upload=278701]: Uploading piece... Nov 22 12:31:07.521175282 (316h27m6.072645743s) [noise=true , address=145.79.227.17:4972 , piece_id=UZK7LCIOBGGDNJZUVDBOLBJHGOIA4SUT2O3PD67PYQD3J2DGBJEA, seg_pos={0 0}, upload=278701]: Uploading piece... Nov 22 12:31:07.521527573 (316h27m6.072998044s) [noise=true , address=145.79.227.23:4546 , piece_id=73UPPN2SYWNWEVWHCF4BRGKNO5GCRGOV3S5JZPX4MW75EJZRQOWQ, seg_pos={0 0}, upload=278701]: Uploading piece... Nov 22 12:31:07.521694671 (316h27m6.073165132s) [noise=true , address=45.91.116.10:4082 , piece_id=GBDYXAWONOHQGBGM3NCRQYDSPFOAIHRJHBB2W6ZGJLMZUJLV65XA, seg_pos={0 0}, upload=278701]: Uploading piece... Nov 22 12:31:07.522219410 (316h27m6.073689871s) [noise=true , address=145.79.227.20:4437 , piece_id=CZ32MB4AEZZ6YKCNWKRYALVAK7MT3KNXDSZCQTPUV2Q53NX6Y3BQ, seg_pos={0 0}, upload=278701]: Uploading piece... Nov 22 12:31:07.522261240 (316h27m6.073731691s) [noise=true , address=145.79.227.16:4909 , piece_id=ZTL4ER5YJWSDGO5F5KLRUL5WRTA6WKVOJOERIBTRIIVEWXUY5Z2Q, seg_pos={0 0}, upload=278701]: Uploading piece... Nov 22 12:31:07.522719151 (316h27m6.074189612s) [noise=true , address=145.79.227.26:4654 , piece_id=EM4RRGMPPNVR7VTXP3BED43FJFIUDZYVT4S5OUK2TRXZHYNMQ64Q, seg_pos={0 0}, upload=278701]: Uploading piece... Nov 22 12:31:07.523017269 (316h27m6.07448772s) [noise=true , address=77.90.31.13:4627 , piece_id=Q7U5BFXHYQBBJQCU4EQGAR75WKL5QMBJ3LO34UYGO7QHLMDJBKWQ, seg_pos={0 0}, upload=278701]: Uploading piece... Nov 22 12:31:07.523216949 (316h27m6.07468741s) [noise=true , address=145.79.227.25:4618 , piece_id=CV6M3CTPQR2MPQZFU6W34MFQERT6MMN3RRF2N3W35FGI464OSUUA, seg_pos={0 0}, upload=278701]: Uploading piece... Nov 22 12:31:07.523744804 (316h27m6.075215265s) [noise=true , address=145.79.227.6:4258 , piece_id=B5D4TCVJORSAWDAU4DQ5VAV4CCRF5WUI24DUVUORAEKLAWGI7ABQ, seg_pos={0 0}, upload=278701]: Uploading piece... Nov 22 12:31:07.523828153 (316h27m6.075298614s) [noise=true , address=145.79.227.24:4583 , piece_id=MLW4X34EWXKPOCFVI4VIQSL6RMUOJ5HRYHQ4HY35JIFT4ZFJ6F3Q, seg_pos={0 0}, upload=278701]: Uploading piece... Nov 22 12:31:07.524255396 (316h27m6.075725858s) [noise=true , address=45.91.116.7:4009 , piece_id=FXQGF3UQN5CPFBS2HWD7BAXR4QIOF42MMK3VJQ7AK4THYJWOFFLQ, seg_pos={0 0}, upload=278701]: Uploading piece... Nov 22 12:31:07.524662051 (316h27m6.076132512s) [noise=true , address=145.79.227.15:4874 , piece_id=TYAXIF3ZJQQSFGFANYF56DG4O5WBBXK4M7OT67E7ARYNZ663HESA, seg_pos={0 0}, upload=278701]: Uploading piece... Nov 22 12:31:07.524746221 (316h27m6.076216682s) [noise=true , address=145.79.227.28:4726 , piece_id=3LBJV2B4CVXEMHCOPRW7DR5QBKPS6DFMPTHQSVNR4XRTZZVIGBQA, seg_pos={0 0}, upload=278701]: Uploading piece... Nov 22 12:31:07.525227157 (316h27m6.076697608s) [noise=true , address=77.90.31.21:4879 , piece_id=GHTH55OGGXBAZZ5FGPCDYVHDYEETM7DNVTPSTTFFFRZMKZZ3XRXQ, seg_pos={0 0}, upload=278701]: Uploading piece... Nov 22 12:31:09.134610631 (316h27m7.686081093s) [upload=278701]: Next returned nil segment Nov 22 12:31:09.134628024 (316h27m7.686098495s) [upload=278701]: Waiting for error group managing segments... Nov 22 12:31:09.502125778 (316h27m8.053596239s) [noise=true , address=145.79.227.16:4909 , piece_id=ZTL4ER5YJWSDGO5F5KLRUL5WRTA6WKVOJOERIBTRIIVEWXUY5Z2Q, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil> Nov 22 12:31:09.503083742 (316h27m8.054554193s) [noise=true , address=145.79.227.20:4437 , piece_id=CZ32MB4AEZZ6YKCNWKRYALVAK7MT3KNXDSZCQTPUV2Q53NX6Y3BQ, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil> Nov 22 12:31:09.517732716 (316h27m8.069203177s) [noise=true , address=145.79.227.25:4618 , piece_id=CV6M3CTPQR2MPQZFU6W34MFQERT6MMN3RRF2N3W35FGI464OSUUA, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil> Nov 22 12:31:09.518029582 (316h27m8.069500043s) [noise=true , address=145.79.227.28:4726 , piece_id=3LBJV2B4CVXEMHCOPRW7DR5QBKPS6DFMPTHQSVNR4XRTZZVIGBQA, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil> Nov 22 12:31:09.519935051 (316h27m8.071405512s) [noise=true , address=145.79.227.26:4654 , piece_id=EM4RRGMPPNVR7VTXP3BED43FJFIUDZYVT4S5OUK2TRXZHYNMQ64Q, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil> Nov 22 12:31:09.520607661 (316h27m8.072078122s) [noise=true , address=145.79.227.27:4689 , piece_id=76S65KEZ7I2E76RR2JXOZ3ONGXCWSJA5XB7XQVFEEKGGNC34Z4EQ, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil> Nov 22 12:31:09.521187736 (316h27m8.072658197s) [noise=true , address=145.79.227.23:4546 , piece_id=73UPPN2SYWNWEVWHCF4BRGKNO5GCRGOV3S5JZPX4MW75EJZRQOWQ, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil> Nov 22 12:31:09.532974760 (316h27m8.084445221s) [noise=true , address=145.79.227.24:4583 , piece_id=MLW4X34EWXKPOCFVI4VIQSL6RMUOJ5HRYHQ4HY35JIFT4ZFJ6F3Q, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil> Nov 22 12:31:09.533688800 (316h27m8.085159271s) [noise=true , address=145.79.227.6:4258 , piece_id=B5D4TCVJORSAWDAU4DQ5VAV4CCRF5WUI24DUVUORAEKLAWGI7ABQ, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil> Nov 22 12:31:09.534460109 (316h27m8.08593057s) [noise=true , address=145.79.227.15:4874 , piece_id=TYAXIF3ZJQQSFGFANYF56DG4O5WBBXK4M7OT67E7ARYNZ663HESA, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil> Nov 22 12:31:09.542955375 (316h27m8.094425826s) [noise=true , address=145.79.227.17:4972 , piece_id=UZK7LCIOBGGDNJZUVDBOLBJHGOIA4SUT2O3PD67PYQD3J2DGBJEA, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil> Nov 22 12:31:09.563098982 (316h27m8.114569443s) [noise=true , address=145.79.227.21:4474 , piece_id=PCRWYQVCAZCPOILAEDHJGDBFKX4GEJ6JXCF7FUS7OVFRUTDPJI4Q, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil> Nov 22 12:31:36.585475346 (316h27m35.136945807s) [noise=true , address=45.91.116.10:4082 , piece_id=GBDYXAWONOHQGBGM3NCRQYDSPFOAIHRJHBB2W6ZGJLMZUJLV65XA, seg_pos={0 0}, upload=278701]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: read tcp 188.166.225.170:54326->45.91.116.10:4082: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 12:31:36.585499923 (316h27m35.136970384s) [noise=true , address=77.90.31.21:4879 , piece_id=GHTH55OGGXBAZZ5FGPCDYVHDYEETM7DNVTPSTTFFFRZMKZZ3XRXQ, seg_pos={0 0}, upload=278701]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: rpc: dial tcp 77.90.31.21:4879: i/o timeout; context deadline exceeded Nov 22 12:31:36.585749648 (316h27m35.137220119s) [noise=true , address=77.90.31.13:4627 , piece_id=Q7U5BFXHYQBBJQCU4EQGAR75WKL5QMBJ3LO34UYGO7QHLMDJBKWQ, seg_pos={0 0}, upload=278701]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: context deadline exceeded; context deadline exceeded Nov 22 12:31:36.585848086 (316h27m35.137318547s) [noise=true , address=45.91.116.7:4009 , piece_id=FXQGF3UQN5CPFBS2HWD7BAXR4QIOF42MMK3VJQ7AK4THYJWOFFLQ, seg_pos={0 0}, upload=278701]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: context deadline exceeded; context deadline exceeded Nov 22 12:31:36.585900936 (316h27m35.137371397s) [seg_pos={0 0}, upload=278701]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 22 12:31:36.828881025 (316h27m35.380351486s) [upload=278701]: Done uploading segments... Nov 22 12:31:36.828915801 (316h27m35.380386262s) [upload=278701]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.604393672s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [..X.XX.............................................X........] duration: 5.001324916s trace: https://jaeger.dev.storj.io/trace/38596674a32f39c1 FAIL (2025-11-22 13:13:24 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.00158423s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 5 out of 60 is failed (threshold=6) [X..X.XX.............................................X.......] duration: 4.564749529s trace: https://jaeger.dev.storj.io/trace/38596674a32f39c1 FAIL (2025-11-22 13:12:54 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.564777623s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)