Orbiter: EU/US/AP/EDGE | Code | Config | Playbook | Escalation Procedure
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
5 out of 60 is failed (threshold=3) [.X....X....................XX.........X.....................] duration:
5.545069887s
FAIL (2025-08-12 10:06:37 UTC)
orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== Aug 12 09:49:40.239600832 (88h23m34.670680953s) [upload=92081]: Starting upload Aug 12 09:49:40.239656178 (88h23m34.670736309s) [upload=92081]: Done starting upload Aug 12 09:49:40.239727585 (88h23m34.670807716s) [upload=92081]: Uploading segments... Aug 12 09:49:40.239830302 (88h23m34.670910433s) [upload=92081]: Got next segment. Inline: false Aug 12 09:49:40.239871160 (88h23m34.670951291s) [upload=92081]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 12 09:49:40.239875899 (88h23m34.67095603s) [upload=92081]: Flush batch item: *pb.BatchRequestItem_SegmentBegin Aug 12 09:49:40.454591416 (88h23m34.885671547s) [upload=92081]: Next segment err: metaclient: Too Many Requests Aug 12 09:49:40.454655870 (88h23m34.885735991s) [upload=92081]: Done uploading segments... Aug 12 09:49:40.454668995 (88h23m34.885749126s) [upload=92081]: Upload finished. err: metaclient: Too Many Requests
Duration: 5.545090175s
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
5 out of 60 is failed (threshold=3) [..X....X....................XX.........X....................] duration:
5.624245598s
FAIL (2025-08-12 10:05:41 UTC)
orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== Aug 12 09:49:40.239600832 (88h23m34.670680953s) [upload=92081]: Starting upload Aug 12 09:49:40.239656178 (88h23m34.670736309s) [upload=92081]: Done starting upload Aug 12 09:49:40.239727585 (88h23m34.670807716s) [upload=92081]: Uploading segments... Aug 12 09:49:40.239830302 (88h23m34.670910433s) [upload=92081]: Got next segment. Inline: false Aug 12 09:49:40.239871160 (88h23m34.670951291s) [upload=92081]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 12 09:49:40.239875899 (88h23m34.67095603s) [upload=92081]: Flush batch item: *pb.BatchRequestItem_SegmentBegin Aug 12 09:49:40.454591416 (88h23m34.885671547s) [upload=92081]: Next segment err: metaclient: Too Many Requests Aug 12 09:49:40.454655870 (88h23m34.885735991s) [upload=92081]: Done uploading segments... Aug 12 09:49:40.454668995 (88h23m34.885749126s) [upload=92081]: Upload finished. err: metaclient: Too Many Requests
Duration: 5.624272149s
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)antiblip:
3 out of 60 is failed (threshold=3) [......X........X................................X...........] duration:
1.84151351s
FAIL (2025-08-12 10:05:35 UTC)
orbiter warning: operation: ListObjects ======ERROR====== unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/1K ======LOGS====== Aug 12 09:58:16.036259937 (88h32m10.467340068s) [upload=92223]: Starting upload Aug 12 09:58:16.036308489 (88h32m10.46738862s) [upload=92223]: Done starting upload Aug 12 09:58:16.036367583 (88h32m10.467447714s) [upload=92223]: Uploading segments... Aug 12 09:58:16.036403541 (88h32m10.467483672s) [upload=92223]: Got next segment. Inline: true Aug 12 09:58:16.036407308 (88h32m10.467487429s) [upload=92223]: Waiting for error group managing segments... Aug 12 09:58:16.036463356 (88h32m10.467543487s) [upload=92223]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 12 09:58:16.036467453 (88h32m10.467547584s) [upload=92223]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 12 09:58:16.036469998 (88h32m10.467550129s) [upload=92223]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 12 09:58:16.335141187 (88h32m10.766221308s) [upload=92223]: Done uploading segments... Aug 12 09:58:16.335177998 (88h32m10.766258129s) [upload=92223]: Upload finished. err: <nil>
Duration: 1.841540331s
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
5 out of 60 is failed (threshold=3) [...X....X....................XX.........X...................] duration:
5.598458716s
FAIL (2025-08-12 10:05:10 UTC)
orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== Aug 12 09:49:40.239600832 (88h23m34.670680953s) [upload=92081]: Starting upload Aug 12 09:49:40.239656178 (88h23m34.670736309s) [upload=92081]: Done starting upload Aug 12 09:49:40.239727585 (88h23m34.670807716s) [upload=92081]: Uploading segments... Aug 12 09:49:40.239830302 (88h23m34.670910433s) [upload=92081]: Got next segment. Inline: false Aug 12 09:49:40.239871160 (88h23m34.670951291s) [upload=92081]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 12 09:49:40.239875899 (88h23m34.67095603s) [upload=92081]: Flush batch item: *pb.BatchRequestItem_SegmentBegin Aug 12 09:49:40.454591416 (88h23m34.885671547s) [upload=92081]: Next segment err: metaclient: Too Many Requests Aug 12 09:49:40.454655870 (88h23m34.885735991s) [upload=92081]: Done uploading segments... Aug 12 09:49:40.454668995 (88h23m34.885749126s) [upload=92081]: Upload finished. err: metaclient: Too Many Requests
Duration: 5.598485287s
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)antiblip:
3 out of 60 is failed (threshold=3) [.......X........X................................X..........] duration:
1.846934043s
FAIL (2025-08-12 10:04:56 UTC)
orbiter warning: operation: ListObjects ======ERROR====== unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/1K ======LOGS====== Aug 12 09:58:16.036259937 (88h32m10.467340068s) [upload=92223]: Starting upload Aug 12 09:58:16.036308489 (88h32m10.46738862s) [upload=92223]: Done starting upload Aug 12 09:58:16.036367583 (88h32m10.467447714s) [upload=92223]: Uploading segments... Aug 12 09:58:16.036403541 (88h32m10.467483672s) [upload=92223]: Got next segment. Inline: true Aug 12 09:58:16.036407308 (88h32m10.467487429s) [upload=92223]: Waiting for error group managing segments... Aug 12 09:58:16.036463356 (88h32m10.467543487s) [upload=92223]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 12 09:58:16.036467453 (88h32m10.467547584s) [upload=92223]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 12 09:58:16.036469998 (88h32m10.467550129s) [upload=92223]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 12 09:58:16.335141187 (88h32m10.766221308s) [upload=92223]: Done uploading segments... Aug 12 09:58:16.335177998 (88h32m10.766258129s) [upload=92223]: Upload finished. err: <nil>
Duration: 1.846961185s
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
5 out of 60 is failed (threshold=3) [....X....X....................XX.........X..................] duration:
5.541808079s
FAIL (2025-08-12 10:04:21 UTC)
orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== Aug 12 09:49:40.239600832 (88h23m34.670680953s) [upload=92081]: Starting upload Aug 12 09:49:40.239656178 (88h23m34.670736309s) [upload=92081]: Done starting upload Aug 12 09:49:40.239727585 (88h23m34.670807716s) [upload=92081]: Uploading segments... Aug 12 09:49:40.239830302 (88h23m34.670910433s) [upload=92081]: Got next segment. Inline: false Aug 12 09:49:40.239871160 (88h23m34.670951291s) [upload=92081]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 12 09:49:40.239875899 (88h23m34.67095603s) [upload=92081]: Flush batch item: *pb.BatchRequestItem_SegmentBegin Aug 12 09:49:40.454591416 (88h23m34.885671547s) [upload=92081]: Next segment err: metaclient: Too Many Requests Aug 12 09:49:40.454655870 (88h23m34.885735991s) [upload=92081]: Done uploading segments... Aug 12 09:49:40.454668995 (88h23m34.885749126s) [upload=92081]: Upload finished. err: metaclient: Too Many Requests
Duration: 5.541833427s
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)antiblip:
4 out of 60 is failed (threshold=3) [X.......X........X................................X.........] duration:
2.031486105s
FAIL (2025-08-12 10:04:09 UTC)
orbiter warning: operation: ListObjects ======ERROR====== unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/1K ======LOGS====== Aug 12 09:58:16.036259937 (88h32m10.467340068s) [upload=92223]: Starting upload Aug 12 09:58:16.036308489 (88h32m10.46738862s) [upload=92223]: Done starting upload Aug 12 09:58:16.036367583 (88h32m10.467447714s) [upload=92223]: Uploading segments... Aug 12 09:58:16.036403541 (88h32m10.467483672s) [upload=92223]: Got next segment. Inline: true Aug 12 09:58:16.036407308 (88h32m10.467487429s) [upload=92223]: Waiting for error group managing segments... Aug 12 09:58:16.036463356 (88h32m10.467543487s) [upload=92223]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 12 09:58:16.036467453 (88h32m10.467547584s) [upload=92223]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 12 09:58:16.036469998 (88h32m10.467550129s) [upload=92223]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 12 09:58:16.335141187 (88h32m10.766221308s) [upload=92223]: Done uploading segments... Aug 12 09:58:16.335177998 (88h32m10.766258129s) [upload=92223]: Upload finished. err: <nil>
Duration: 2.031507375s
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)antiblip:
4 out of 60 is failed (threshold=3) [.X.......X........X................................X........] duration:
1.872899679s
FAIL (2025-08-12 10:03:51 UTC)
orbiter warning: operation: ListObjects ======ERROR====== unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/1K ======LOGS====== Aug 12 09:58:16.036259937 (88h32m10.467340068s) [upload=92223]: Starting upload Aug 12 09:58:16.036308489 (88h32m10.46738862s) [upload=92223]: Done starting upload Aug 12 09:58:16.036367583 (88h32m10.467447714s) [upload=92223]: Uploading segments... Aug 12 09:58:16.036403541 (88h32m10.467483672s) [upload=92223]: Got next segment. Inline: true Aug 12 09:58:16.036407308 (88h32m10.467487429s) [upload=92223]: Waiting for error group managing segments... Aug 12 09:58:16.036463356 (88h32m10.467543487s) [upload=92223]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 12 09:58:16.036467453 (88h32m10.467547584s) [upload=92223]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 12 09:58:16.036469998 (88h32m10.467550129s) [upload=92223]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 12 09:58:16.335141187 (88h32m10.766221308s) [upload=92223]: Done uploading segments... Aug 12 09:58:16.335177998 (88h32m10.766258129s) [upload=92223]: Upload finished. err: <nil>
Duration: 1.872927372s
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline)antiblip:
1 out of 60 is failed (threshold=3) [X...........................................................] duration:
2.305388646s
FAIL (2025-08-12 10:03:46 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.305408473s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)antiblip:
4 out of 60 is failed (threshold=3) [..X.......X........X................................X.......] duration:
1.836232994s
FAIL (2025-08-12 10:03:30 UTC)
orbiter warning: operation: ListObjects ======ERROR====== unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/1K ======LOGS====== Aug 12 09:58:16.036259937 (88h32m10.467340068s) [upload=92223]: Starting upload Aug 12 09:58:16.036308489 (88h32m10.46738862s) [upload=92223]: Done starting upload Aug 12 09:58:16.036367583 (88h32m10.467447714s) [upload=92223]: Uploading segments... Aug 12 09:58:16.036403541 (88h32m10.467483672s) [upload=92223]: Got next segment. Inline: true Aug 12 09:58:16.036407308 (88h32m10.467487429s) [upload=92223]: Waiting for error group managing segments... Aug 12 09:58:16.036463356 (88h32m10.467543487s) [upload=92223]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 12 09:58:16.036467453 (88h32m10.467547584s) [upload=92223]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 12 09:58:16.036469998 (88h32m10.467550129s) [upload=92223]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 12 09:58:16.335141187 (88h32m10.766221308s) [upload=92223]: Done uploading segments... Aug 12 09:58:16.335177998 (88h32m10.766258129s) [upload=92223]: Upload finished. err: <nil>
Duration: 1.836256729s
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
5 out of 60 is failed (threshold=3) [.....X....X....................XX.........X.................] duration:
5.86229736s
FAIL (2025-08-12 10:03:10 UTC)
orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== Aug 12 09:49:40.239600832 (88h23m34.670680953s) [upload=92081]: Starting upload Aug 12 09:49:40.239656178 (88h23m34.670736309s) [upload=92081]: Done starting upload Aug 12 09:49:40.239727585 (88h23m34.670807716s) [upload=92081]: Uploading segments... Aug 12 09:49:40.239830302 (88h23m34.670910433s) [upload=92081]: Got next segment. Inline: false Aug 12 09:49:40.239871160 (88h23m34.670951291s) [upload=92081]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 12 09:49:40.239875899 (88h23m34.67095603s) [upload=92081]: Flush batch item: *pb.BatchRequestItem_SegmentBegin Aug 12 09:49:40.454591416 (88h23m34.885671547s) [upload=92081]: Next segment err: metaclient: Too Many Requests Aug 12 09:49:40.454655870 (88h23m34.885735991s) [upload=92081]: Done uploading segments... Aug 12 09:49:40.454668995 (88h23m34.885749126s) [upload=92081]: Upload finished. err: metaclient: Too Many Requests
Duration: 5.862324142s
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)antiblip:
4 out of 60 is failed (threshold=3) [...X.......X........X................................X......] duration:
1.952535469s
FAIL (2025-08-12 10:03:04 UTC)
orbiter warning: operation: ListObjects ======ERROR====== unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/1K ======LOGS====== Aug 12 09:58:16.036259937 (88h32m10.467340068s) [upload=92223]: Starting upload Aug 12 09:58:16.036308489 (88h32m10.46738862s) [upload=92223]: Done starting upload Aug 12 09:58:16.036367583 (88h32m10.467447714s) [upload=92223]: Uploading segments... Aug 12 09:58:16.036403541 (88h32m10.467483672s) [upload=92223]: Got next segment. Inline: true Aug 12 09:58:16.036407308 (88h32m10.467487429s) [upload=92223]: Waiting for error group managing segments... Aug 12 09:58:16.036463356 (88h32m10.467543487s) [upload=92223]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 12 09:58:16.036467453 (88h32m10.467547584s) [upload=92223]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 12 09:58:16.036469998 (88h32m10.467550129s) [upload=92223]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 12 09:58:16.335141187 (88h32m10.766221308s) [upload=92223]: Done uploading segments... Aug 12 09:58:16.335177998 (88h32m10.766258129s) [upload=92223]: Upload finished. err: <nil>
Duration: 1.952558683s
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)antiblip:
4 out of 60 is failed (threshold=3) [....X.......X........X................................X.....] duration:
2.225566432s
FAIL (2025-08-12 10:02:35 UTC)
orbiter warning: operation: ListObjects ======ERROR====== unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/1K ======LOGS====== Aug 12 09:58:16.036259937 (88h32m10.467340068s) [upload=92223]: Starting upload Aug 12 09:58:16.036308489 (88h32m10.46738862s) [upload=92223]: Done starting upload Aug 12 09:58:16.036367583 (88h32m10.467447714s) [upload=92223]: Uploading segments... Aug 12 09:58:16.036403541 (88h32m10.467483672s) [upload=92223]: Got next segment. Inline: true Aug 12 09:58:16.036407308 (88h32m10.467487429s) [upload=92223]: Waiting for error group managing segments... Aug 12 09:58:16.036463356 (88h32m10.467543487s) [upload=92223]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 12 09:58:16.036467453 (88h32m10.467547584s) [upload=92223]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 12 09:58:16.036469998 (88h32m10.467550129s) [upload=92223]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 12 09:58:16.335141187 (88h32m10.766221308s) [upload=92223]: Done uploading segments... Aug 12 09:58:16.335177998 (88h32m10.766258129s) [upload=92223]: Upload finished. err: <nil>
Duration: 2.225610176s
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline)antiblip:
1 out of 60 is failed (threshold=3) [.X..........................................................] duration:
2.326281499s
FAIL (2025-08-12 10:02:35 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.32630862s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [X...........................................................] duration:
6.31829371s
FAIL (2025-08-12 10:02:04 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.318315572s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
5 out of 60 is failed (threshold=3) [......X....X....................XX.........X................] duration:
5.81578327s
FAIL (2025-08-12 10:02:01 UTC)
orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== Aug 12 09:49:40.239600832 (88h23m34.670680953s) [upload=92081]: Starting upload Aug 12 09:49:40.239656178 (88h23m34.670736309s) [upload=92081]: Done starting upload Aug 12 09:49:40.239727585 (88h23m34.670807716s) [upload=92081]: Uploading segments... Aug 12 09:49:40.239830302 (88h23m34.670910433s) [upload=92081]: Got next segment. Inline: false Aug 12 09:49:40.239871160 (88h23m34.670951291s) [upload=92081]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 12 09:49:40.239875899 (88h23m34.67095603s) [upload=92081]: Flush batch item: *pb.BatchRequestItem_SegmentBegin Aug 12 09:49:40.454591416 (88h23m34.885671547s) [upload=92081]: Next segment err: metaclient: Too Many Requests Aug 12 09:49:40.454655870 (88h23m34.885735991s) [upload=92081]: Done uploading segments... Aug 12 09:49:40.454668995 (88h23m34.885749126s) [upload=92081]: Upload finished. err: metaclient: Too Many Requests
Duration: 5.815807468s
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline)antiblip:
1 out of 60 is failed (threshold=3) [..X.........................................................] duration:
2.343589167s
FAIL (2025-08-12 10:01:55 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.343612991s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)antiblip:
4 out of 60 is failed (threshold=3) [.....X.......X........X................................X....] duration:
1.841136308s
FAIL (2025-08-12 10:01:27 UTC)
orbiter warning: operation: ListObjects ======ERROR====== unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/1K ======LOGS====== Aug 12 09:58:16.036259937 (88h32m10.467340068s) [upload=92223]: Starting upload Aug 12 09:58:16.036308489 (88h32m10.46738862s) [upload=92223]: Done starting upload Aug 12 09:58:16.036367583 (88h32m10.467447714s) [upload=92223]: Uploading segments... Aug 12 09:58:16.036403541 (88h32m10.467483672s) [upload=92223]: Got next segment. Inline: true Aug 12 09:58:16.036407308 (88h32m10.467487429s) [upload=92223]: Waiting for error group managing segments... Aug 12 09:58:16.036463356 (88h32m10.467543487s) [upload=92223]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 12 09:58:16.036467453 (88h32m10.467547584s) [upload=92223]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 12 09:58:16.036469998 (88h32m10.467550129s) [upload=92223]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 12 09:58:16.335141187 (88h32m10.766221308s) [upload=92223]: Done uploading segments... Aug 12 09:58:16.335177998 (88h32m10.766258129s) [upload=92223]: Upload finished. err: <nil>
Duration: 1.841166644s
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.X..........................................................] duration:
6.144220533s
FAIL (2025-08-12 10:01:19 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.144245489s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
5 out of 60 is failed (threshold=3) [.......X....X....................XX.........X...............] duration:
6.001683024s
FAIL (2025-08-12 10:01:10 UTC)
orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== Aug 12 09:49:40.239600832 (88h23m34.670680953s) [upload=92081]: Starting upload Aug 12 09:49:40.239656178 (88h23m34.670736309s) [upload=92081]: Done starting upload Aug 12 09:49:40.239727585 (88h23m34.670807716s) [upload=92081]: Uploading segments... Aug 12 09:49:40.239830302 (88h23m34.670910433s) [upload=92081]: Got next segment. Inline: false Aug 12 09:49:40.239871160 (88h23m34.670951291s) [upload=92081]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 12 09:49:40.239875899 (88h23m34.67095603s) [upload=92081]: Flush batch item: *pb.BatchRequestItem_SegmentBegin Aug 12 09:49:40.454591416 (88h23m34.885671547s) [upload=92081]: Next segment err: metaclient: Too Many Requests Aug 12 09:49:40.454655870 (88h23m34.885735991s) [upload=92081]: Done uploading segments... Aug 12 09:49:40.454668995 (88h23m34.885749126s) [upload=92081]: Upload finished. err: metaclient: Too Many Requests
Duration: 6.001705769s
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline)antiblip:
1 out of 60 is failed (threshold=3) [...X........................................................] duration:
2.377496131s
FAIL (2025-08-12 10:01:07 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.377510238s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
5 out of 60 is failed (threshold=3) [........X....X....................XX.........X..............] duration:
5.546696644s
FAIL (2025-08-12 10:00:44 UTC)
orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== Aug 12 09:49:40.239600832 (88h23m34.670680953s) [upload=92081]: Starting upload Aug 12 09:49:40.239656178 (88h23m34.670736309s) [upload=92081]: Done starting upload Aug 12 09:49:40.239727585 (88h23m34.670807716s) [upload=92081]: Uploading segments... Aug 12 09:49:40.239830302 (88h23m34.670910433s) [upload=92081]: Got next segment. Inline: false Aug 12 09:49:40.239871160 (88h23m34.670951291s) [upload=92081]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 12 09:49:40.239875899 (88h23m34.67095603s) [upload=92081]: Flush batch item: *pb.BatchRequestItem_SegmentBegin Aug 12 09:49:40.454591416 (88h23m34.885671547s) [upload=92081]: Next segment err: metaclient: Too Many Requests Aug 12 09:49:40.454655870 (88h23m34.885735991s) [upload=92081]: Done uploading segments... Aug 12 09:49:40.454668995 (88h23m34.885749126s) [upload=92081]: Upload finished. err: metaclient: Too Many Requests
Duration: 5.546724167s
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)antiblip:
4 out of 60 is failed (threshold=3) [......X.......X........X................................X...] duration:
1.889575368s
FAIL (2025-08-12 10:00:38 UTC)
orbiter warning: operation: ListObjects ======ERROR====== unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/1K ======LOGS====== Aug 12 09:58:16.036259937 (88h32m10.467340068s) [upload=92223]: Starting upload Aug 12 09:58:16.036308489 (88h32m10.46738862s) [upload=92223]: Done starting upload Aug 12 09:58:16.036367583 (88h32m10.467447714s) [upload=92223]: Uploading segments... Aug 12 09:58:16.036403541 (88h32m10.467483672s) [upload=92223]: Got next segment. Inline: true Aug 12 09:58:16.036407308 (88h32m10.467487429s) [upload=92223]: Waiting for error group managing segments... Aug 12 09:58:16.036463356 (88h32m10.467543487s) [upload=92223]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 12 09:58:16.036467453 (88h32m10.467547584s) [upload=92223]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 12 09:58:16.036469998 (88h32m10.467550129s) [upload=92223]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 12 09:58:16.335141187 (88h32m10.766221308s) [upload=92223]: Done uploading segments... Aug 12 09:58:16.335177998 (88h32m10.766258129s) [upload=92223]: Upload finished. err: <nil>
Duration: 1.889599966s
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..X.........................................................] duration:
5.973234276s
FAIL (2025-08-12 10:00:34 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.973261287s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline)antiblip:
1 out of 60 is failed (threshold=3) [....X.......................................................] duration:
2.287970235s
FAIL (2025-08-12 10:00:16 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.287994161s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...X........................................................] duration:
5.946709236s
FAIL (2025-08-12 10:00:11 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.946760393s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
5 out of 60 is failed (threshold=3) [.........X....X....................XX.........X.............] duration:
5.590715279s
FAIL (2025-08-12 09:59:52 UTC)
orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== Aug 12 09:49:40.239600832 (88h23m34.670680953s) [upload=92081]: Starting upload Aug 12 09:49:40.239656178 (88h23m34.670736309s) [upload=92081]: Done starting upload Aug 12 09:49:40.239727585 (88h23m34.670807716s) [upload=92081]: Uploading segments... Aug 12 09:49:40.239830302 (88h23m34.670910433s) [upload=92081]: Got next segment. Inline: false Aug 12 09:49:40.239871160 (88h23m34.670951291s) [upload=92081]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 12 09:49:40.239875899 (88h23m34.67095603s) [upload=92081]: Flush batch item: *pb.BatchRequestItem_SegmentBegin Aug 12 09:49:40.454591416 (88h23m34.885671547s) [upload=92081]: Next segment err: metaclient: Too Many Requests Aug 12 09:49:40.454655870 (88h23m34.885735991s) [upload=92081]: Done uploading segments... Aug 12 09:49:40.454668995 (88h23m34.885749126s) [upload=92081]: Upload finished. err: metaclient: Too Many Requests
Duration: 5.590744806s
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline)antiblip:
1 out of 60 is failed (threshold=3) [.....X......................................................] duration:
2.306988921s
FAIL (2025-08-12 09:59:46 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.307011575s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)antiblip:
4 out of 60 is failed (threshold=3) [.......X.......X........X................................X..] duration:
1.855721256s
FAIL (2025-08-12 09:59:43 UTC)
orbiter warning: operation: ListObjects ======ERROR====== unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/1K ======LOGS====== Aug 12 09:58:16.036259937 (88h32m10.467340068s) [upload=92223]: Starting upload Aug 12 09:58:16.036308489 (88h32m10.46738862s) [upload=92223]: Done starting upload Aug 12 09:58:16.036367583 (88h32m10.467447714s) [upload=92223]: Uploading segments... Aug 12 09:58:16.036403541 (88h32m10.467483672s) [upload=92223]: Got next segment. Inline: true Aug 12 09:58:16.036407308 (88h32m10.467487429s) [upload=92223]: Waiting for error group managing segments... Aug 12 09:58:16.036463356 (88h32m10.467543487s) [upload=92223]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 12 09:58:16.036467453 (88h32m10.467547584s) [upload=92223]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 12 09:58:16.036469998 (88h32m10.467550129s) [upload=92223]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 12 09:58:16.335141187 (88h32m10.766221308s) [upload=92223]: Done uploading segments... Aug 12 09:58:16.335177998 (88h32m10.766258129s) [upload=92223]: Upload finished. err: <nil>
Duration: 1.855756353s
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [....X.......................................................] duration:
6.290863877s
FAIL (2025-08-12 09:59:07 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.290886902s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
5 out of 60 is failed (threshold=3) [..........X....X....................XX.........X............] duration:
5.646053817s
FAIL (2025-08-12 09:58:54 UTC)
orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== Aug 12 09:49:40.239600832 (88h23m34.670680953s) [upload=92081]: Starting upload Aug 12 09:49:40.239656178 (88h23m34.670736309s) [upload=92081]: Done starting upload Aug 12 09:49:40.239727585 (88h23m34.670807716s) [upload=92081]: Uploading segments... Aug 12 09:49:40.239830302 (88h23m34.670910433s) [upload=92081]: Got next segment. Inline: false Aug 12 09:49:40.239871160 (88h23m34.670951291s) [upload=92081]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 12 09:49:40.239875899 (88h23m34.67095603s) [upload=92081]: Flush batch item: *pb.BatchRequestItem_SegmentBegin Aug 12 09:49:40.454591416 (88h23m34.885671547s) [upload=92081]: Next segment err: metaclient: Too Many Requests Aug 12 09:49:40.454655870 (88h23m34.885735991s) [upload=92081]: Done uploading segments... Aug 12 09:49:40.454668995 (88h23m34.885749126s) [upload=92081]: Upload finished. err: metaclient: Too Many Requests
Duration: 5.646071712s
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)antiblip:
4 out of 60 is failed (threshold=3) [........X.......X........X................................X.] duration:
1.972033387s
FAIL (2025-08-12 09:58:54 UTC)
orbiter warning: operation: ListObjects ======ERROR====== unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/1K ======LOGS====== Aug 12 09:58:16.036259937 (88h32m10.467340068s) [upload=92223]: Starting upload Aug 12 09:58:16.036308489 (88h32m10.46738862s) [upload=92223]: Done starting upload Aug 12 09:58:16.036367583 (88h32m10.467447714s) [upload=92223]: Uploading segments... Aug 12 09:58:16.036403541 (88h32m10.467483672s) [upload=92223]: Got next segment. Inline: true Aug 12 09:58:16.036407308 (88h32m10.467487429s) [upload=92223]: Waiting for error group managing segments... Aug 12 09:58:16.036463356 (88h32m10.467543487s) [upload=92223]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 12 09:58:16.036467453 (88h32m10.467547584s) [upload=92223]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 12 09:58:16.036469998 (88h32m10.467550129s) [upload=92223]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 12 09:58:16.335141187 (88h32m10.766221308s) [upload=92223]: Done uploading segments... Aug 12 09:58:16.335177998 (88h32m10.766258129s) [upload=92223]: Upload finished. err: <nil>
Duration: 1.972060268s
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline)antiblip:
1 out of 60 is failed (threshold=3) [......X.....................................................] duration:
2.215523966s
FAIL (2025-08-12 09:58:41 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.215547031s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.....X......................................................] duration:
6.457217781s
FAIL (2025-08-12 09:58:32 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.457236978s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)antiblip:
4 out of 60 is failed (threshold=3) [.........X.......X........X................................X] duration:
1.392843425s
FAIL (2025-08-12 09:58:16 UTC)
orbiter warning: operation: ListObjects ======ERROR====== unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/1K ======LOGS====== Aug 12 09:58:16.036259937 (88h32m10.467340068s) [upload=92223]: Starting upload Aug 12 09:58:16.036308489 (88h32m10.46738862s) [upload=92223]: Done starting upload Aug 12 09:58:16.036367583 (88h32m10.467447714s) [upload=92223]: Uploading segments... Aug 12 09:58:16.036403541 (88h32m10.467483672s) [upload=92223]: Got next segment. Inline: true Aug 12 09:58:16.036407308 (88h32m10.467487429s) [upload=92223]: Waiting for error group managing segments... Aug 12 09:58:16.036463356 (88h32m10.467543487s) [upload=92223]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 12 09:58:16.036467453 (88h32m10.467547584s) [upload=92223]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 12 09:58:16.036469998 (88h32m10.467550129s) [upload=92223]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 12 09:58:16.335141187 (88h32m10.766221308s) [upload=92223]: Done uploading segments... Aug 12 09:58:16.335177998 (88h32m10.766258129s) [upload=92223]: Upload finished. err: <nil>
Duration: 1.39292412s
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline)antiblip:
1 out of 60 is failed (threshold=3) [.......X....................................................] duration:
2.361430014s
FAIL (2025-08-12 09:58:01 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.361452317s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
5 out of 60 is failed (threshold=3) [...........X....X....................XX.........X...........] duration:
5.578084371s
FAIL (2025-08-12 09:57:44 UTC)
orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== Aug 12 09:49:40.239600832 (88h23m34.670680953s) [upload=92081]: Starting upload Aug 12 09:49:40.239656178 (88h23m34.670736309s) [upload=92081]: Done starting upload Aug 12 09:49:40.239727585 (88h23m34.670807716s) [upload=92081]: Uploading segments... Aug 12 09:49:40.239830302 (88h23m34.670910433s) [upload=92081]: Got next segment. Inline: false Aug 12 09:49:40.239871160 (88h23m34.670951291s) [upload=92081]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 12 09:49:40.239875899 (88h23m34.67095603s) [upload=92081]: Flush batch item: *pb.BatchRequestItem_SegmentBegin Aug 12 09:49:40.454591416 (88h23m34.885671547s) [upload=92081]: Next segment err: metaclient: Too Many Requests Aug 12 09:49:40.454655870 (88h23m34.885735991s) [upload=92081]: Done uploading segments... Aug 12 09:49:40.454668995 (88h23m34.885749126s) [upload=92081]: Upload finished. err: metaclient: Too Many Requests
Duration: 5.578108567s
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline)antiblip:
1 out of 60 is failed (threshold=3) [........X...................................................] duration:
2.382887005s
FAIL (2025-08-12 09:57:26 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.382901612s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
5 out of 60 is failed (threshold=3) [............X....X....................XX.........X..........] duration:
5.607503279s
FAIL (2025-08-12 09:57:23 UTC)
orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== Aug 12 09:49:40.239600832 (88h23m34.670680953s) [upload=92081]: Starting upload Aug 12 09:49:40.239656178 (88h23m34.670736309s) [upload=92081]: Done starting upload Aug 12 09:49:40.239727585 (88h23m34.670807716s) [upload=92081]: Uploading segments... Aug 12 09:49:40.239830302 (88h23m34.670910433s) [upload=92081]: Got next segment. Inline: false Aug 12 09:49:40.239871160 (88h23m34.670951291s) [upload=92081]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 12 09:49:40.239875899 (88h23m34.67095603s) [upload=92081]: Flush batch item: *pb.BatchRequestItem_SegmentBegin Aug 12 09:49:40.454591416 (88h23m34.885671547s) [upload=92081]: Next segment err: metaclient: Too Many Requests Aug 12 09:49:40.454655870 (88h23m34.885735991s) [upload=92081]: Done uploading segments... Aug 12 09:49:40.454668995 (88h23m34.885749126s) [upload=92081]: Upload finished. err: metaclient: Too Many Requests
Duration: 5.607531062s
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [......X.....................................................] duration:
6.036310462s
FAIL (2025-08-12 09:57:20 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.036332534s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)antiblip:
3 out of 60 is failed (threshold=3) [..........X.......X........X................................] duration:
1.834571736s
FAIL (2025-08-12 09:57:06 UTC)
orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/1K):: after c36fafdabf96bb5b9c4464bc922fb40045dd340ad6b7f61aa3a428437d8c7f85 != before c9b346cbcc10dabc7edd608987d023da7f45c3973830ffa574bb2139ca951f3f ======LOGS====== Aug 12 09:32:43.528352715 (88h6m37.959432847s) [upload=91793]: Starting upload Aug 12 09:32:43.528398883 (88h6m37.959479014s) [upload=91793]: Done starting upload Aug 12 09:32:43.528451965 (88h6m37.959532126s) [upload=91793]: Uploading segments... Aug 12 09:32:43.528498204 (88h6m37.959578365s) [upload=91793]: Got next segment. Inline: true Aug 12 09:32:43.528504716 (88h6m37.959584867s) [upload=91793]: Waiting for error group managing segments... Aug 12 09:32:43.528567947 (88h6m37.959648078s) [upload=91793]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 12 09:32:43.528574670 (88h6m37.959654861s) [upload=91793]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 12 09:32:43.528579549 (88h6m37.9596597s) [upload=91793]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 12 09:32:43.987672872 (88h6m38.418753003s) [upload=91793]: Done uploading segments... Aug 12 09:32:43.987690746 (88h6m38.418770878s) [upload=91793]: Upload finished. err: <nil>
Duration: 1.834600591s
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.......X....................................................] duration:
6.212371008s
FAIL (2025-08-12 09:56:55 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.212387379s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline)antiblip:
1 out of 60 is failed (threshold=3) [.........X..................................................] duration:
2.25098387s
FAIL (2025-08-12 09:56:55 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.251034195s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [........X...................................................] duration:
6.037635451s
FAIL (2025-08-12 09:56:28 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.037663946s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
5 out of 60 is failed (threshold=3) [.............X....X....................XX.........X.........] duration:
5.740955333s
FAIL (2025-08-12 09:56:10 UTC)
orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== Aug 12 09:49:40.239600832 (88h23m34.670680953s) [upload=92081]: Starting upload Aug 12 09:49:40.239656178 (88h23m34.670736309s) [upload=92081]: Done starting upload Aug 12 09:49:40.239727585 (88h23m34.670807716s) [upload=92081]: Uploading segments... Aug 12 09:49:40.239830302 (88h23m34.670910433s) [upload=92081]: Got next segment. Inline: false Aug 12 09:49:40.239871160 (88h23m34.670951291s) [upload=92081]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 12 09:49:40.239875899 (88h23m34.67095603s) [upload=92081]: Flush batch item: *pb.BatchRequestItem_SegmentBegin Aug 12 09:49:40.454591416 (88h23m34.885671547s) [upload=92081]: Next segment err: metaclient: Too Many Requests Aug 12 09:49:40.454655870 (88h23m34.885735991s) [upload=92081]: Done uploading segments... Aug 12 09:49:40.454668995 (88h23m34.885749126s) [upload=92081]: Upload finished. err: metaclient: Too Many Requests
Duration: 5.740982174s
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)antiblip:
3 out of 60 is failed (threshold=3) [...........X.......X........X...............................] duration:
1.815267958s
FAIL (2025-08-12 09:56:06 UTC)
orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/1K):: after c36fafdabf96bb5b9c4464bc922fb40045dd340ad6b7f61aa3a428437d8c7f85 != before c9b346cbcc10dabc7edd608987d023da7f45c3973830ffa574bb2139ca951f3f ======LOGS====== Aug 12 09:32:43.528352715 (88h6m37.959432847s) [upload=91793]: Starting upload Aug 12 09:32:43.528398883 (88h6m37.959479014s) [upload=91793]: Done starting upload Aug 12 09:32:43.528451965 (88h6m37.959532126s) [upload=91793]: Uploading segments... Aug 12 09:32:43.528498204 (88h6m37.959578365s) [upload=91793]: Got next segment. Inline: true Aug 12 09:32:43.528504716 (88h6m37.959584867s) [upload=91793]: Waiting for error group managing segments... Aug 12 09:32:43.528567947 (88h6m37.959648078s) [upload=91793]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 12 09:32:43.528574670 (88h6m37.959654861s) [upload=91793]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 12 09:32:43.528579549 (88h6m37.9596597s) [upload=91793]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 12 09:32:43.987672872 (88h6m38.418753003s) [upload=91793]: Done uploading segments... Aug 12 09:32:43.987690746 (88h6m38.418770878s) [upload=91793]: Upload finished. err: <nil>
Duration: 1.815294468s
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline)antiblip:
1 out of 60 is failed (threshold=3) [..........X.................................................] duration:
2.492326588s
FAIL (2025-08-12 09:55:49 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.492352287s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.........X..................................................] duration:
6.029413913s
FAIL (2025-08-12 09:55:44 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.029431126s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
5 out of 60 is failed (threshold=3) [..............X....X....................XX.........X........] duration:
5.554904992s
FAIL (2025-08-12 09:55:25 UTC)
orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== Aug 12 09:49:40.239600832 (88h23m34.670680953s) [upload=92081]: Starting upload Aug 12 09:49:40.239656178 (88h23m34.670736309s) [upload=92081]: Done starting upload Aug 12 09:49:40.239727585 (88h23m34.670807716s) [upload=92081]: Uploading segments... Aug 12 09:49:40.239830302 (88h23m34.670910433s) [upload=92081]: Got next segment. Inline: false Aug 12 09:49:40.239871160 (88h23m34.670951291s) [upload=92081]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 12 09:49:40.239875899 (88h23m34.67095603s) [upload=92081]: Flush batch item: *pb.BatchRequestItem_SegmentBegin Aug 12 09:49:40.454591416 (88h23m34.885671547s) [upload=92081]: Next segment err: metaclient: Too Many Requests Aug 12 09:49:40.454655870 (88h23m34.885735991s) [upload=92081]: Done uploading segments... Aug 12 09:49:40.454668995 (88h23m34.885749126s) [upload=92081]: Upload finished. err: metaclient: Too Many Requests
Duration: 5.554934278s
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..........X.................................................] duration:
6.757794057s
FAIL (2025-08-12 09:55:21 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.75781746s
edge
gateway
us-select-1
|