Orbiter History

checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...............X.......X.X..................................] duration: 6.453553554s FAIL (2025-08-13 14:41:02 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.453575897s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 5 out of 60 is failed (threshold=3) [................X..X.............X.....X.X..................] duration: 2.323326518s FAIL (2025-08-13 14:40:39 UTC) orbiter warning: operation: ListObjects ======ERROR====== unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/1K ======LOGS====== Aug 13 14:25:57.508778246 (116h59m51.939858377s) [upload=121717]: Starting upload Aug 13 14:25:57.508850885 (116h59m51.939931016s) [upload=121717]: Done starting upload Aug 13 14:25:57.508940917 (116h59m51.940021058s) [upload=121717]: Uploading segments... Aug 13 14:25:57.509065897 (116h59m51.940146038s) [upload=121717]: Got next segment. Inline: true Aug 13 14:25:57.509071818 (116h59m51.940151959s) [upload=121717]: Waiting for error group managing segments... Aug 13 14:25:57.509160628 (116h59m51.940240769s) [upload=121717]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 13 14:25:57.509169355 (116h59m51.940249496s) [upload=121717]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 13 14:25:57.509174675 (116h59m51.940254806s) [upload=121717]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 13 14:25:58.324969756 (116h59m52.756049887s) [upload=121717]: Done uploading segments... Aug 13 14:25:58.324989163 (116h59m52.756069294s) [upload=121717]: Upload finished. err: <nil> Duration:* 2.32335379s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 5 out of 60 is failed (threshold=3) [..X.....X......X.....................X.......X..............] duration: 6.627087219s FAIL (2025-08-13 14:40:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== Aug 13 14:31:01.866494186 (117h4m56.297574318s) [upload=121799]: Starting upload Aug 13 14:31:01.866533341 (117h4m56.297613472s) [upload=121799]: Done starting upload Aug 13 14:31:01.866740558 (117h4m56.297820699s) [upload=121799]: Uploading segments... Aug 13 14:31:01.867428624 (117h4m56.298508765s) [upload=121799]: Got next segment. Inline: false Aug 13 14:31:01.867473409 (117h4m56.29855354s) [upload=121799]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 13 14:31:01.867478218 (117h4m56.298558349s) [upload=121799]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 13 14:31:02.174794903 (117h4m56.605875034s) [upload=121799]: Next segment err: metaclient: Too Many Requests Aug 13 14:31:02.174815282 (117h4m56.605895413s) [upload=121799]: Done uploading segments... Aug 13 14:31:02.174842604 (117h4m56.605922735s) [upload=121799]: Upload finished. err: metaclient: Too Many Requests Duration: 6.627112888s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [................X.......X.X.................................] duration: 5.935452836s FAIL (2025-08-13 14:39:48 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.935471061s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 5 out of 60 is failed (threshold=3) [.................X..X.............X.....X.X.................] duration: 1.939201897s FAIL (2025-08-13 14:39:39 UTC) orbiter warning: operation: ListObjects ======ERROR====== unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/1K ======LOGS====== Aug 13 14:25:57.508778246 (116h59m51.939858377s) [upload=121717]: Starting upload Aug 13 14:25:57.508850885 (116h59m51.939931016s) [upload=121717]: Done starting upload Aug 13 14:25:57.508940917 (116h59m51.940021058s) [upload=121717]: Uploading segments... Aug 13 14:25:57.509065897 (116h59m51.940146038s) [upload=121717]: Got next segment. Inline: true Aug 13 14:25:57.509071818 (116h59m51.940151959s) [upload=121717]: Waiting for error group managing segments... Aug 13 14:25:57.509160628 (116h59m51.940240769s) [upload=121717]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 13 14:25:57.509169355 (116h59m51.940249496s) [upload=121717]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 13 14:25:57.509174675 (116h59m51.940254806s) [upload=121717]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 13 14:25:58.324969756 (116h59m52.756049887s) [upload=121717]: Done uploading segments... Aug 13 14:25:58.324989163 (116h59m52.756069294s) [upload=121717]: Upload finished. err: <nil> Duration:* 1.939225961s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 5 out of 60 is failed (threshold=3) [...X.....X......X.....................X.......X.............] duration: 7.434235724s FAIL (2025-08-13 14:39:21 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== Aug 13 14:31:01.866494186 (117h4m56.297574318s) [upload=121799]: Starting upload Aug 13 14:31:01.866533341 (117h4m56.297613472s) [upload=121799]: Done starting upload Aug 13 14:31:01.866740558 (117h4m56.297820699s) [upload=121799]: Uploading segments... Aug 13 14:31:01.867428624 (117h4m56.298508765s) [upload=121799]: Got next segment. Inline: false Aug 13 14:31:01.867473409 (117h4m56.29855354s) [upload=121799]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 13 14:31:01.867478218 (117h4m56.298558349s) [upload=121799]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 13 14:31:02.174794903 (117h4m56.605875034s) [upload=121799]: Next segment err: metaclient: Too Many Requests Aug 13 14:31:02.174815282 (117h4m56.605895413s) [upload=121799]: Done uploading segments... Aug 13 14:31:02.174842604 (117h4m56.605922735s) [upload=121799]: Upload finished. err: metaclient: Too Many Requests Duration: 7.434263435s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.................X.......X.X................................] duration: 6.750607161s FAIL (2025-08-13 14:39:14 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.750624353s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 5 out of 60 is failed (threshold=3) [..................X..X.............X.....X.X................] duration: 1.93016814s FAIL (2025-08-13 14:38:51 UTC) orbiter warning: operation: ListObjects ======ERROR====== unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/1K ======LOGS====== Aug 13 14:25:57.508778246 (116h59m51.939858377s) [upload=121717]: Starting upload Aug 13 14:25:57.508850885 (116h59m51.939931016s) [upload=121717]: Done starting upload Aug 13 14:25:57.508940917 (116h59m51.940021058s) [upload=121717]: Uploading segments... Aug 13 14:25:57.509065897 (116h59m51.940146038s) [upload=121717]: Got next segment. Inline: true Aug 13 14:25:57.509071818 (116h59m51.940151959s) [upload=121717]: Waiting for error group managing segments... Aug 13 14:25:57.509160628 (116h59m51.940240769s) [upload=121717]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 13 14:25:57.509169355 (116h59m51.940249496s) [upload=121717]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 13 14:25:57.509174675 (116h59m51.940254806s) [upload=121717]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 13 14:25:58.324969756 (116h59m52.756049887s) [upload=121717]: Done uploading segments... Aug 13 14:25:58.324989163 (116h59m52.756069294s) [upload=121717]: Upload finished. err: <nil> Duration:* 1.93019398s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 5 out of 60 is failed (threshold=3) [....X.....X......X.....................X.......X............] duration: 7.375181119s FAIL (2025-08-13 14:38:44 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== Aug 13 14:31:01.866494186 (117h4m56.297574318s) [upload=121799]: Starting upload Aug 13 14:31:01.866533341 (117h4m56.297613472s) [upload=121799]: Done starting upload Aug 13 14:31:01.866740558 (117h4m56.297820699s) [upload=121799]: Uploading segments... Aug 13 14:31:01.867428624 (117h4m56.298508765s) [upload=121799]: Got next segment. Inline: false Aug 13 14:31:01.867473409 (117h4m56.29855354s) [upload=121799]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 13 14:31:01.867478218 (117h4m56.298558349s) [upload=121799]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 13 14:31:02.174794903 (117h4m56.605875034s) [upload=121799]: Next segment err: metaclient: Too Many Requests Aug 13 14:31:02.174815282 (117h4m56.605895413s) [upload=121799]: Done uploading segments... Aug 13 14:31:02.174842604 (117h4m56.605922735s) [upload=121799]: Upload finished. err: metaclient: Too Many Requests Duration: 7.375205866s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 5 out of 60 is failed (threshold=3) [.....X.....X......X.....................X.......X...........] duration: 7.222372034s FAIL (2025-08-13 14:38:21 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== Aug 13 14:31:01.866494186 (117h4m56.297574318s) [upload=121799]: Starting upload Aug 13 14:31:01.866533341 (117h4m56.297613472s) [upload=121799]: Done starting upload Aug 13 14:31:01.866740558 (117h4m56.297820699s) [upload=121799]: Uploading segments... Aug 13 14:31:01.867428624 (117h4m56.298508765s) [upload=121799]: Got next segment. Inline: false Aug 13 14:31:01.867473409 (117h4m56.29855354s) [upload=121799]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 13 14:31:01.867478218 (117h4m56.298558349s) [upload=121799]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 13 14:31:02.174794903 (117h4m56.605875034s) [upload=121799]: Next segment err: metaclient: Too Many Requests Aug 13 14:31:02.174815282 (117h4m56.605895413s) [upload=121799]: Done uploading segments... Aug 13 14:31:02.174842604 (117h4m56.605922735s) [upload=121799]: Upload finished. err: metaclient: Too Many Requests Duration: 7.22239653s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..................X.......X.X...............................] duration: 12.176369495s FAIL (2025-08-13 14:38:18 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 12.176393071s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 5 out of 60 is failed (threshold=3) [...................X..X.............X.....X.X...............] duration: 2.118077814s FAIL (2025-08-13 14:37:44 UTC) orbiter warning: operation: ListObjects ======ERROR====== unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/1K ======LOGS====== Aug 13 14:25:57.508778246 (116h59m51.939858377s) [upload=121717]: Starting upload Aug 13 14:25:57.508850885 (116h59m51.939931016s) [upload=121717]: Done starting upload Aug 13 14:25:57.508940917 (116h59m51.940021058s) [upload=121717]: Uploading segments... Aug 13 14:25:57.509065897 (116h59m51.940146038s) [upload=121717]: Got next segment. Inline: true Aug 13 14:25:57.509071818 (116h59m51.940151959s) [upload=121717]: Waiting for error group managing segments... Aug 13 14:25:57.509160628 (116h59m51.940240769s) [upload=121717]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 13 14:25:57.509169355 (116h59m51.940249496s) [upload=121717]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 13 14:25:57.509174675 (116h59m51.940254806s) [upload=121717]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 13 14:25:58.324969756 (116h59m52.756049887s) [upload=121717]: Done uploading segments... Aug 13 14:25:58.324989163 (116h59m52.756069294s) [upload=121717]: Upload finished. err: <nil> Duration:* 2.118102342s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 5 out of 60 is failed (threshold=3) [......X.....X......X.....................X.......X..........] duration: 6.173043022s FAIL (2025-08-13 14:37:14 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== Aug 13 14:31:01.866494186 (117h4m56.297574318s) [upload=121799]: Starting upload Aug 13 14:31:01.866533341 (117h4m56.297613472s) [upload=121799]: Done starting upload Aug 13 14:31:01.866740558 (117h4m56.297820699s) [upload=121799]: Uploading segments... Aug 13 14:31:01.867428624 (117h4m56.298508765s) [upload=121799]: Got next segment. Inline: false Aug 13 14:31:01.867473409 (117h4m56.29855354s) [upload=121799]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 13 14:31:01.867478218 (117h4m56.298558349s) [upload=121799]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 13 14:31:02.174794903 (117h4m56.605875034s) [upload=121799]: Next segment err: metaclient: Too Many Requests Aug 13 14:31:02.174815282 (117h4m56.605895413s) [upload=121799]: Done uploading segments... Aug 13 14:31:02.174842604 (117h4m56.605922735s) [upload=121799]: Upload finished. err: metaclient: Too Many Requests Duration: 6.173064934s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...................X.......X.X..............................] duration: 7.012889821s FAIL (2025-08-13 14:37:09 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 7.012940718s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 5 out of 60 is failed (threshold=3) [.......X.....X......X.....................X.......X.........] duration: 6.366377347s FAIL (2025-08-13 14:36:44 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== Aug 13 14:31:01.866494186 (117h4m56.297574318s) [upload=121799]: Starting upload Aug 13 14:31:01.866533341 (117h4m56.297613472s) [upload=121799]: Done starting upload Aug 13 14:31:01.866740558 (117h4m56.297820699s) [upload=121799]: Uploading segments... Aug 13 14:31:01.867428624 (117h4m56.298508765s) [upload=121799]: Got next segment. Inline: false Aug 13 14:31:01.867473409 (117h4m56.29855354s) [upload=121799]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 13 14:31:01.867478218 (117h4m56.298558349s) [upload=121799]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 13 14:31:02.174794903 (117h4m56.605875034s) [upload=121799]: Next segment err: metaclient: Too Many Requests Aug 13 14:31:02.174815282 (117h4m56.605895413s) [upload=121799]: Done uploading segments... Aug 13 14:31:02.174842604 (117h4m56.605922735s) [upload=121799]: Upload finished. err: metaclient: Too Many Requests Duration: 6.366405201s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 5 out of 60 is failed (threshold=3) [....................X..X.............X.....X.X..............] duration: 1.923705918s FAIL (2025-08-13 14:36:29 UTC) orbiter warning: operation: ListObjects ======ERROR====== unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/1K ======LOGS====== Aug 13 14:25:57.508778246 (116h59m51.939858377s) [upload=121717]: Starting upload Aug 13 14:25:57.508850885 (116h59m51.939931016s) [upload=121717]: Done starting upload Aug 13 14:25:57.508940917 (116h59m51.940021058s) [upload=121717]: Uploading segments... Aug 13 14:25:57.509065897 (116h59m51.940146038s) [upload=121717]: Got next segment. Inline: true Aug 13 14:25:57.509071818 (116h59m51.940151959s) [upload=121717]: Waiting for error group managing segments... Aug 13 14:25:57.509160628 (116h59m51.940240769s) [upload=121717]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 13 14:25:57.509169355 (116h59m51.940249496s) [upload=121717]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 13 14:25:57.509174675 (116h59m51.940254806s) [upload=121717]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 13 14:25:58.324969756 (116h59m52.756049887s) [upload=121717]: Done uploading segments... Aug 13 14:25:58.324989163 (116h59m52.756069294s) [upload=121717]: Upload finished. err: <nil> Duration:* 1.923735775s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 5 out of 60 is failed (threshold=3) [........X.....X......X.....................X.......X........] duration: 6.184621134s FAIL (2025-08-13 14:36:27 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== Aug 13 14:31:01.866494186 (117h4m56.297574318s) [upload=121799]: Starting upload Aug 13 14:31:01.866533341 (117h4m56.297613472s) [upload=121799]: Done starting upload Aug 13 14:31:01.866740558 (117h4m56.297820699s) [upload=121799]: Uploading segments... Aug 13 14:31:01.867428624 (117h4m56.298508765s) [upload=121799]: Got next segment. Inline: false Aug 13 14:31:01.867473409 (117h4m56.29855354s) [upload=121799]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 13 14:31:01.867478218 (117h4m56.298558349s) [upload=121799]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 13 14:31:02.174794903 (117h4m56.605875034s) [upload=121799]: Next segment err: metaclient: Too Many Requests Aug 13 14:31:02.174815282 (117h4m56.605895413s) [upload=121799]: Done uploading segments... Aug 13 14:31:02.174842604 (117h4m56.605922735s) [upload=121799]: Upload finished. err: metaclient: Too Many Requests Duration: 6.184644328s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [....................X.......X.X.............................] duration: 9.583541736s FAIL (2025-08-13 14:36:15 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 9.583562587s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 5 out of 60 is failed (threshold=3) [.........X.....X......X.....................X.......X.......] duration: 7.444340564s FAIL (2025-08-13 14:36:12 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== Aug 13 14:31:01.866494186 (117h4m56.297574318s) [upload=121799]: Starting upload Aug 13 14:31:01.866533341 (117h4m56.297613472s) [upload=121799]: Done starting upload Aug 13 14:31:01.866740558 (117h4m56.297820699s) [upload=121799]: Uploading segments... Aug 13 14:31:01.867428624 (117h4m56.298508765s) [upload=121799]: Got next segment. Inline: false Aug 13 14:31:01.867473409 (117h4m56.29855354s) [upload=121799]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 13 14:31:01.867478218 (117h4m56.298558349s) [upload=121799]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 13 14:31:02.174794903 (117h4m56.605875034s) [upload=121799]: Next segment err: metaclient: Too Many Requests Aug 13 14:31:02.174815282 (117h4m56.605895413s) [upload=121799]: Done uploading segments... Aug 13 14:31:02.174842604 (117h4m56.605922735s) [upload=121799]: Upload finished. err: metaclient: Too Many Requests Duration: 7.444365592s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.....................X.......X.X............................] duration: 9.896782536s FAIL (2025-08-13 14:35:58 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 9.896804888s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 5 out of 60 is failed (threshold=3) [.....................X..X.............X.....X.X.............] duration: 2.096824385s FAIL (2025-08-13 14:35:20 UTC) orbiter warning: operation: ListObjects ======ERROR====== unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/1K ======LOGS====== Aug 13 14:25:57.508778246 (116h59m51.939858377s) [upload=121717]: Starting upload Aug 13 14:25:57.508850885 (116h59m51.939931016s) [upload=121717]: Done starting upload Aug 13 14:25:57.508940917 (116h59m51.940021058s) [upload=121717]: Uploading segments... Aug 13 14:25:57.509065897 (116h59m51.940146038s) [upload=121717]: Got next segment. Inline: true Aug 13 14:25:57.509071818 (116h59m51.940151959s) [upload=121717]: Waiting for error group managing segments... Aug 13 14:25:57.509160628 (116h59m51.940240769s) [upload=121717]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 13 14:25:57.509169355 (116h59m51.940249496s) [upload=121717]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 13 14:25:57.509174675 (116h59m51.940254806s) [upload=121717]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 13 14:25:58.324969756 (116h59m52.756049887s) [upload=121717]: Done uploading segments... Aug 13 14:25:58.324989163 (116h59m52.756069294s) [upload=121717]: Upload finished. err: <nil> Duration:* 2.096848843s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 5 out of 60 is failed (threshold=3) [..........X.....X......X.....................X.......X......] duration: 6.483163993s FAIL (2025-08-13 14:35:06 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== Aug 13 14:31:01.866494186 (117h4m56.297574318s) [upload=121799]: Starting upload Aug 13 14:31:01.866533341 (117h4m56.297613472s) [upload=121799]: Done starting upload Aug 13 14:31:01.866740558 (117h4m56.297820699s) [upload=121799]: Uploading segments... Aug 13 14:31:01.867428624 (117h4m56.298508765s) [upload=121799]: Got next segment. Inline: false Aug 13 14:31:01.867473409 (117h4m56.29855354s) [upload=121799]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 13 14:31:01.867478218 (117h4m56.298558349s) [upload=121799]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 13 14:31:02.174794903 (117h4m56.605875034s) [upload=121799]: Next segment err: metaclient: Too Many Requests Aug 13 14:31:02.174815282 (117h4m56.605895413s) [upload=121799]: Done uploading segments... Aug 13 14:31:02.174842604 (117h4m56.605922735s) [upload=121799]: Upload finished. err: metaclient: Too Many Requests Duration: 6.483192719s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [......................X.......X.X...........................] duration: 9.092406573s FAIL (2025-08-13 14:35:05 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 9.092433284s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 5 out of 60 is failed (threshold=3) [...........X.....X......X.....................X.......X.....] duration: 7.58146269s FAIL (2025-08-13 14:34:30 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== Aug 13 14:31:01.866494186 (117h4m56.297574318s) [upload=121799]: Starting upload Aug 13 14:31:01.866533341 (117h4m56.297613472s) [upload=121799]: Done starting upload Aug 13 14:31:01.866740558 (117h4m56.297820699s) [upload=121799]: Uploading segments... Aug 13 14:31:01.867428624 (117h4m56.298508765s) [upload=121799]: Got next segment. Inline: false Aug 13 14:31:01.867473409 (117h4m56.29855354s) [upload=121799]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 13 14:31:01.867478218 (117h4m56.298558349s) [upload=121799]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 13 14:31:02.174794903 (117h4m56.605875034s) [upload=121799]: Next segment err: metaclient: Too Many Requests Aug 13 14:31:02.174815282 (117h4m56.605895413s) [upload=121799]: Done uploading segments... Aug 13 14:31:02.174842604 (117h4m56.605922735s) [upload=121799]: Upload finished. err: metaclient: Too Many Requests Duration: 7.581505211s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.......................X.......X.X..........................] duration: 5.972857892s FAIL (2025-08-13 14:34:25 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.972874024s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 5 out of 60 is failed (threshold=3) [............X.....X......X.....................X.......X....] duration: 7.197655487s FAIL (2025-08-13 14:34:10 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== Aug 13 14:31:01.866494186 (117h4m56.297574318s) [upload=121799]: Starting upload Aug 13 14:31:01.866533341 (117h4m56.297613472s) [upload=121799]: Done starting upload Aug 13 14:31:01.866740558 (117h4m56.297820699s) [upload=121799]: Uploading segments... Aug 13 14:31:01.867428624 (117h4m56.298508765s) [upload=121799]: Got next segment. Inline: false Aug 13 14:31:01.867473409 (117h4m56.29855354s) [upload=121799]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 13 14:31:01.867478218 (117h4m56.298558349s) [upload=121799]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 13 14:31:02.174794903 (117h4m56.605875034s) [upload=121799]: Next segment err: metaclient: Too Many Requests Aug 13 14:31:02.174815282 (117h4m56.605895413s) [upload=121799]: Done uploading segments... Aug 13 14:31:02.174842604 (117h4m56.605922735s) [upload=121799]: Upload finished. err: metaclient: Too Many Requests Duration: 7.197679473s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 5 out of 60 is failed (threshold=3) [......................X..X.............X.....X.X............] duration: 1.97268205s FAIL (2025-08-13 14:34:09 UTC) orbiter warning: operation: ListObjects ======ERROR====== unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/1K ======LOGS====== Aug 13 14:25:57.508778246 (116h59m51.939858377s) [upload=121717]: Starting upload Aug 13 14:25:57.508850885 (116h59m51.939931016s) [upload=121717]: Done starting upload Aug 13 14:25:57.508940917 (116h59m51.940021058s) [upload=121717]: Uploading segments... Aug 13 14:25:57.509065897 (116h59m51.940146038s) [upload=121717]: Got next segment. Inline: true Aug 13 14:25:57.509071818 (116h59m51.940151959s) [upload=121717]: Waiting for error group managing segments... Aug 13 14:25:57.509160628 (116h59m51.940240769s) [upload=121717]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 13 14:25:57.509169355 (116h59m51.940249496s) [upload=121717]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 13 14:25:57.509174675 (116h59m51.940254806s) [upload=121717]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 13 14:25:58.324969756 (116h59m52.756049887s) [upload=121717]: Done uploading segments... Aug 13 14:25:58.324989163 (116h59m52.756069294s) [upload=121717]: Upload finished. err: <nil> Duration:* 1.972710274s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [........................X.......X.X.........................] duration: 7.992144681s FAIL (2025-08-13 14:34:06 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 7.992518576s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 5 out of 60 is failed (threshold=3) [.......................X..X.............X.....X.X...........] duration: 1.942183994s FAIL (2025-08-13 14:33:38 UTC) orbiter warning: operation: ListObjects ======ERROR====== unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/1K ======LOGS====== Aug 13 14:25:57.508778246 (116h59m51.939858377s) [upload=121717]: Starting upload Aug 13 14:25:57.508850885 (116h59m51.939931016s) [upload=121717]: Done starting upload Aug 13 14:25:57.508940917 (116h59m51.940021058s) [upload=121717]: Uploading segments... Aug 13 14:25:57.509065897 (116h59m51.940146038s) [upload=121717]: Got next segment. Inline: true Aug 13 14:25:57.509071818 (116h59m51.940151959s) [upload=121717]: Waiting for error group managing segments... Aug 13 14:25:57.509160628 (116h59m51.940240769s) [upload=121717]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 13 14:25:57.509169355 (116h59m51.940249496s) [upload=121717]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 13 14:25:57.509174675 (116h59m51.940254806s) [upload=121717]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 13 14:25:58.324969756 (116h59m52.756049887s) [upload=121717]: Done uploading segments... Aug 13 14:25:58.324989163 (116h59m52.756069294s) [upload=121717]: Upload finished. err: <nil> Duration:* 1.942205945s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 5 out of 60 is failed (threshold=3) [.............X.....X......X.....................X.......X...] duration: 6.11909311s FAIL (2025-08-13 14:33:21 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== Aug 13 14:31:01.866494186 (117h4m56.297574318s) [upload=121799]: Starting upload Aug 13 14:31:01.866533341 (117h4m56.297613472s) [upload=121799]: Done starting upload Aug 13 14:31:01.866740558 (117h4m56.297820699s) [upload=121799]: Uploading segments... Aug 13 14:31:01.867428624 (117h4m56.298508765s) [upload=121799]: Got next segment. Inline: false Aug 13 14:31:01.867473409 (117h4m56.29855354s) [upload=121799]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 13 14:31:01.867478218 (117h4m56.298558349s) [upload=121799]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 13 14:31:02.174794903 (117h4m56.605875034s) [upload=121799]: Next segment err: metaclient: Too Many Requests Aug 13 14:31:02.174815282 (117h4m56.605895413s) [upload=121799]: Done uploading segments... Aug 13 14:31:02.174842604 (117h4m56.605922735s) [upload=121799]: Upload finished. err: metaclient: Too Many Requests Duration: 6.119122535s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 5 out of 60 is failed (threshold=3) [........................X..X.............X.....X.X..........] duration: 1.91936254s FAIL (2025-08-13 14:33:05 UTC) orbiter warning: operation: ListObjects ======ERROR====== unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/1K ======LOGS====== Aug 13 14:25:57.508778246 (116h59m51.939858377s) [upload=121717]: Starting upload Aug 13 14:25:57.508850885 (116h59m51.939931016s) [upload=121717]: Done starting upload Aug 13 14:25:57.508940917 (116h59m51.940021058s) [upload=121717]: Uploading segments... Aug 13 14:25:57.509065897 (116h59m51.940146038s) [upload=121717]: Got next segment. Inline: true Aug 13 14:25:57.509071818 (116h59m51.940151959s) [upload=121717]: Waiting for error group managing segments... Aug 13 14:25:57.509160628 (116h59m51.940240769s) [upload=121717]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 13 14:25:57.509169355 (116h59m51.940249496s) [upload=121717]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 13 14:25:57.509174675 (116h59m51.940254806s) [upload=121717]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 13 14:25:58.324969756 (116h59m52.756049887s) [upload=121717]: Done uploading segments... Aug 13 14:25:58.324989163 (116h59m52.756069294s) [upload=121717]: Upload finished. err: <nil> Duration:* 1.919390585s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 5 out of 60 is failed (threshold=3) [..............X.....X......X.....................X.......X..] duration: 6.434883777s FAIL (2025-08-13 14:33:04 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== Aug 13 14:31:01.866494186 (117h4m56.297574318s) [upload=121799]: Starting upload Aug 13 14:31:01.866533341 (117h4m56.297613472s) [upload=121799]: Done starting upload Aug 13 14:31:01.866740558 (117h4m56.297820699s) [upload=121799]: Uploading segments... Aug 13 14:31:01.867428624 (117h4m56.298508765s) [upload=121799]: Got next segment. Inline: false Aug 13 14:31:01.867473409 (117h4m56.29855354s) [upload=121799]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 13 14:31:01.867478218 (117h4m56.298558349s) [upload=121799]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 13 14:31:02.174794903 (117h4m56.605875034s) [upload=121799]: Next segment err: metaclient: Too Many Requests Aug 13 14:31:02.174815282 (117h4m56.605895413s) [upload=121799]: Done uploading segments... Aug 13 14:31:02.174842604 (117h4m56.605922735s) [upload=121799]: Upload finished. err: metaclient: Too Many Requests Duration: 6.434907933s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.........................X.......X.X........................] duration: 6.05434352s FAIL (2025-08-13 14:32:52 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.054359982s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 5 out of 60 is failed (threshold=3) [.........................X..X.............X.....X.X.........] duration: 1.970449038s FAIL (2025-08-13 14:32:13 UTC) orbiter warning: operation: ListObjects ======ERROR====== unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/1K ======LOGS====== Aug 13 14:25:57.508778246 (116h59m51.939858377s) [upload=121717]: Starting upload Aug 13 14:25:57.508850885 (116h59m51.939931016s) [upload=121717]: Done starting upload Aug 13 14:25:57.508940917 (116h59m51.940021058s) [upload=121717]: Uploading segments... Aug 13 14:25:57.509065897 (116h59m51.940146038s) [upload=121717]: Got next segment. Inline: true Aug 13 14:25:57.509071818 (116h59m51.940151959s) [upload=121717]: Waiting for error group managing segments... Aug 13 14:25:57.509160628 (116h59m51.940240769s) [upload=121717]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 13 14:25:57.509169355 (116h59m51.940249496s) [upload=121717]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 13 14:25:57.509174675 (116h59m51.940254806s) [upload=121717]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 13 14:25:58.324969756 (116h59m52.756049887s) [upload=121717]: Done uploading segments... Aug 13 14:25:58.324989163 (116h59m52.756069294s) [upload=121717]: Upload finished. err: <nil> Duration:* 1.970470357s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..........................X.......X.X.......................] duration: 6.182736254s FAIL (2025-08-13 14:32:03 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.182757545s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 5 out of 60 is failed (threshold=3) [...............X.....X......X.....................X.......X.] duration: 7.256769325s FAIL (2025-08-13 14:31:55 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== Aug 13 14:31:01.866494186 (117h4m56.297574318s) [upload=121799]: Starting upload Aug 13 14:31:01.866533341 (117h4m56.297613472s) [upload=121799]: Done starting upload Aug 13 14:31:01.866740558 (117h4m56.297820699s) [upload=121799]: Uploading segments... Aug 13 14:31:01.867428624 (117h4m56.298508765s) [upload=121799]: Got next segment. Inline: false Aug 13 14:31:01.867473409 (117h4m56.29855354s) [upload=121799]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 13 14:31:01.867478218 (117h4m56.298558349s) [upload=121799]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 13 14:31:02.174794903 (117h4m56.605875034s) [upload=121799]: Next segment err: metaclient: Too Many Requests Aug 13 14:31:02.174815282 (117h4m56.605895413s) [upload=121799]: Done uploading segments... Aug 13 14:31:02.174842604 (117h4m56.605922735s) [upload=121799]: Upload finished. err: metaclient: Too Many Requests Duration: 7.25680355s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 5 out of 60 is failed (threshold=3) [..........................X..X.............X.....X.X........] duration: 1.952296934s FAIL (2025-08-13 14:31:44 UTC) orbiter warning: operation: ListObjects ======ERROR====== unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/1K ======LOGS====== Aug 13 14:25:57.508778246 (116h59m51.939858377s) [upload=121717]: Starting upload Aug 13 14:25:57.508850885 (116h59m51.939931016s) [upload=121717]: Done starting upload Aug 13 14:25:57.508940917 (116h59m51.940021058s) [upload=121717]: Uploading segments... Aug 13 14:25:57.509065897 (116h59m51.940146038s) [upload=121717]: Got next segment. Inline: true Aug 13 14:25:57.509071818 (116h59m51.940151959s) [upload=121717]: Waiting for error group managing segments... Aug 13 14:25:57.509160628 (116h59m51.940240769s) [upload=121717]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 13 14:25:57.509169355 (116h59m51.940249496s) [upload=121717]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 13 14:25:57.509174675 (116h59m51.940254806s) [upload=121717]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 13 14:25:58.324969756 (116h59m52.756049887s) [upload=121717]: Done uploading segments... Aug 13 14:25:58.324989163 (116h59m52.756069294s) [upload=121717]: Upload finished. err: <nil> Duration:* 1.95232081s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...........................X.......X.X......................] duration: 7.21772301s FAIL (2025-08-13 14:31:43 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 7.217750271s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [............................X.......X.X.....................] duration: 6.338157729s FAIL (2025-08-13 14:31:19 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.338183258s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 5 out of 60 is failed (threshold=3) [................X.....X......X.....................X.......X] duration: 998.189345ms FAIL (2025-08-13 14:31:02 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== Aug 13 14:31:01.866494186 (117h4m56.297574318s) [upload=121799]: Starting upload Aug 13 14:31:01.866533341 (117h4m56.297613472s) [upload=121799]: Done starting upload Aug 13 14:31:01.866740558 (117h4m56.297820699s) [upload=121799]: Uploading segments... Aug 13 14:31:01.867428624 (117h4m56.298508765s) [upload=121799]: Got next segment. Inline: false Aug 13 14:31:01.867473409 (117h4m56.29855354s) [upload=121799]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 13 14:31:01.867478218 (117h4m56.298558349s) [upload=121799]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 13 14:31:02.174794903 (117h4m56.605875034s) [upload=121799]: Next segment err: metaclient: Too Many Requests Aug 13 14:31:02.174815282 (117h4m56.605895413s) [upload=121799]: Done uploading segments... Aug 13 14:31:02.174842604 (117h4m56.605922735s) [upload=121799]: Upload finished. err: metaclient: Too Many Requests Duration: 998.208993ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 4 out of 60 is failed (threshold=3) [.................X.....X......X.....................X.......] duration: 6.394861469s FAIL (2025-08-13 14:30:42 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/4M") ======LOGS====== Aug 13 14:25:34.670450049 (116h59m29.10153018s) [upload=121710]: Starting upload Aug 13 14:25:34.670500074 (116h59m29.101580205s) [upload=121710]: Done starting upload Aug 13 14:25:34.670554288 (116h59m29.101634409s) [upload=121710]: Uploading segments... Aug 13 14:25:34.670715417 (116h59m29.101795548s) [upload=121710]: Got next segment. Inline: false Aug 13 14:25:34.670762726 (116h59m29.101842857s) [upload=121710]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 13 14:25:34.670769440 (116h59m29.101849631s) [upload=121710]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 13 14:25:35.141817037 (116h59m29.572897168s) [seg_pos={0 0}, upload=121710]: Begin upload segment... Aug 13 14:25:35.141863436 (116h59m29.572943567s) [seg_pos={0 0}, upload=121710]: Done begin upload segment. Aug 13 14:25:35.141887462 (116h59m29.572967593s) [noise=true , address=104.36.82.38:4719 , piece_id=PYZMWL3OZIULLOGDVKRF2RSLGLDGGWTCFWHQ66CPKPPEP6JGCVXQ, seg_pos={0 0}, upload=121710]: Uploading piece... Aug 13 14:25:35.141902260 (116h59m29.572982391s) [noise=true , address=104.36.82.59:4328 , piece_id=EVWPCNQWESLH26NS62FKFXKOIW64GUDZFCABBK6H4FSYP35OA5OA, seg_pos={0 0}, upload=121710]: Uploading piece... Aug 13 14:25:35.141945994 (116h59m29.573026125s) [noise=true , address=104.36.82.56:4235 , piece_id=HS4RRPCLDB5ZMD2EJZZCTSCWVY3TWZC6QDVS4TRXRHO3EGZGR7UQ, seg_pos={0 0}, upload=121710]: Uploading piece... Aug 13 14:25:35.142200952 (116h59m29.573281083s) [noise=true , address=104.36.82.60:4371 , piece_id=RTCVTBKO26FY7ZUNFYYJ2IOQ5XJCGG774ZP4WFLOVK5D47ULGXOA, seg_pos={0 0}, upload=121710]: Uploading piece... Aug 13 14:25:35.142305862 (116h59m29.573385993s) [noise=true , address=104.36.82.44:4874 , piece_id=C2ATKCPCO7P4KSHSMRRNGYCMH4IRDKUHL5ERDJKIJ4OECUCLQCFQ, seg_pos={0 0}, upload=121710]: Uploading piece... Aug 13 14:25:35.142439999 (116h59m29.57352013s) [noise=true , address=104.36.82.19:4408 , piece_id=LT43HOEP3H4NTUKHSRUXPSRO4XBEJ6NOHIAPCBPZKVXPIW7SLLQQ, seg_pos={0 0}, upload=121710]: Uploading piece... Aug 13 14:25:35.142605826 (116h59m29.573685957s) [noise=true , address=77.90.31.15:4702 , piece_id=SOXOEHUIJGYJKQ64YWBZMGAZMQJHR5RQMP3VFS5SGDSFM6J52ZMQ, seg_pos={0 0}, upload=121710]: Uploading piece... Aug 13 14:25:35.142682213 (116h59m29.573762344s) [noise=true , address=77.90.31.19:4846 , piece_id=3UYXBMMHS5MPUNXI6PY65K3AGHIIIVSGXJDEUVXLTVXCZYWTL7JA, seg_pos={0 0}, upload=121710]: Uploading piece... Aug 13 14:25:35.142910119 (116h59m29.57399024s) [noise=true , address=104.36.82.37:4678 , piece_id=S5AV5WE4DOUZIRLT3GMKQLPLHARLF24HTVG3QNJ2GWZPYIC55PGQ, seg_pos={0 0}, upload=121710]: Uploading piece... Aug 13 14:25:35.142917733 (116h59m29.573997864s) [noise=true , address=66.23.193.175:4564 , piece_id=S4CHPP736M3D5GOURHJRONGXSYNTREBY3S243VJ6E5ZM75TDTLPA, seg_pos={0 0}, upload=121710]: Uploading piece... Aug 13 14:25:35.143203881 (116h59m29.574284012s) [noise=true , address=64.49.13.140:4511 , piece_id=VTJ5NDREGII5G4IOP44JNYRJ5IFHHRP7S7OWTRFOGL4FCKBAUDYA, seg_pos={0 0}, upload=121710]: Uploading piece... Aug 13 14:25:35.143239018 (116h59m29.574319149s) [noise=true , address=104.36.82.34:4560 , piece_id=4ZZFZKJRUXPMKOE2RDH6ZWFP3TVUOPSRQS7FAB2YANQ3MKA7JLWQ, seg_pos={0 0}, upload=121710]: Uploading piece... Aug 13 14:25:35.143481342 (116h59m29.574561473s) [noise=true , address=104.36.82.39:4727 , piece_id=UYFASTBHHBMDAY7ZUYHNSNSDDZMPPZ4AIR6X6HC6ERBXLXF4AOUQ, seg_pos={0 0}, upload=121710]: Uploading piece... Aug 13 14:25:35.143486071 (116h59m29.574566192s) [noise=true , address=104.36.82.23:4474 , piece_id=MDCBWJSKMP57ZE2V5WIKCG7QWIMRAOOIIFAAJQ5YXVWBEG2EG45Q, seg_pos={0 0}, upload=121710]: Uploading piece... Aug 13 14:25:35.143749975 (116h59m29.574830106s) [noise=true , address=77.90.31.11:4580 , piece_id=CBVCJWDPVPPQW2JIAV25UUFT7EVH62WJOP3FK4M7GUVJAO6P2QGA, seg_pos={0 0}, upload=121710]: Uploading piece... Aug 13 14:25:35.144014762 (116h59m29.575094893s) [noise=true , address=104.36.82.62:4412 , piece_id=CTYSBOF4JVZB5JTYNQDMP6Q5XK5KVETQH5SHHSR65K6HF3TZM2WQ, seg_pos={0 0}, upload=121710]: Uploading piece... Aug 13 14:25:35.146502863 (116h59m29.577582994s) [noise=true , address=79.127.201.215:4358 , piece_id=5LNV5OH2YBCRCO7GJOK6Q3VSYC47ZJ4VTZQO3XVN7BLIRFRB4J5A, seg_pos={0 0}, upload=121710]: Uploading piece... Aug 13 14:25:35.146602894 (116h59m29.577683025s) [noise=true , address=104.36.82.36:4642 , piece_id=TILD5YD3ZU4AOT5227X6PIEIKYXF52D2AZSNDWLOENLTQ4ZMVSLA, seg_pos={0 0}, upload=121710]: Uploading piece... Aug 13 14:25:36.312798371 (116h59m30.743878492s) [upload=121710]: Next returned nil segment Aug 13 14:25:36.312814072 (116h59m30.743894203s) [upload=121710]: Waiting for error group managing segments... Aug 13 14:25:36.608833520 (116h59m31.039913651s) [noise=true , address=66.23.193.175:4564 , piece_id=S4CHPP736M3D5GOURHJRONGXSYNTREBY3S243VJ6E5ZM75TDTLPA, seg_pos={0 0}, upload=121710]: Done uploading piece. err: <nil> Aug 13 14:25:37.109920204 (116h59m31.541000325s) [noise=true , address=79.127.201.215:4358 , piece_id=5LNV5OH2YBCRCO7GJOK6Q3VSYC47ZJ4VTZQO3XVN7BLIRFRB4J5A, seg_pos={0 0}, upload=121710]: Done uploading piece. err: <nil> Aug 13 14:25:37.110478522 (116h59m31.541558653s) [noise=true , address=104.36.82.36:4642 , piece_id=TILD5YD3ZU4AOT5227X6PIEIKYXF52D2AZSNDWLOENLTQ4ZMVSLA, seg_pos={0 0}, upload=121710]: Done uploading piece. err: <nil> Aug 13 14:25:37.127900888 (116h59m31.558981009s) [noise=true , address=77.90.31.11:4580 , piece_id=CBVCJWDPVPPQW2JIAV25UUFT7EVH62WJOP3FK4M7GUVJAO6P2QGA, seg_pos={0 0}, upload=121710]: Done uploading piece. err: <nil> Aug 13 14:25:37.166070970 (116h59m31.597151101s) [noise=true , address=104.36.82.37:4678 , piece_id=S5AV5WE4DOUZIRLT3GMKQLPLHARLF24HTVG3QNJ2GWZPYIC55PGQ, seg_pos={0 0}, upload=121710]: Done uploading piece. err: <nil> Aug 13 14:25:37.167794408 (116h59m31.598874539s) [noise=true , address=104.36.82.19:4408 , piece_id=LT43HOEP3H4NTUKHSRUXPSRO4XBEJ6NOHIAPCBPZKVXPIW7SLLQQ, seg_pos={0 0}, upload=121710]: Done uploading piece. err: <nil> Aug 13 14:25:37.167973721 (116h59m31.599053852s) [noise=true , address=104.36.82.59:4328 , piece_id=EVWPCNQWESLH26NS62FKFXKOIW64GUDZFCABBK6H4FSYP35OA5OA, seg_pos={0 0}, upload=121710]: Done uploading piece. err: <nil> Aug 13 14:25:37.195711503 (116h59m31.626791634s) [noise=true , address=104.36.82.34:4560 , piece_id=4ZZFZKJRUXPMKOE2RDH6ZWFP3TVUOPSRQS7FAB2YANQ3MKA7JLWQ, seg_pos={0 0}, upload=121710]: Done uploading piece. err: <nil> Aug 13 14:25:37.209376008 (116h59m31.640456129s) [noise=true , address=104.36.82.39:4727 , piece_id=UYFASTBHHBMDAY7ZUYHNSNSDDZMPPZ4AIR6X6HC6ERBXLXF4AOUQ, seg_pos={0 0}, upload=121710]: Done uploading piece. err: <nil> Aug 13 14:25:37.252511451 (116h59m31.683591582s) [noise=true , address=77.90.31.19:4846 , piece_id=3UYXBMMHS5MPUNXI6PY65K3AGHIIIVSGXJDEUVXLTVXCZYWTL7JA, seg_pos={0 0}, upload=121710]: Done uploading piece. err: <nil> Aug 13 14:25:37.256659508 (116h59m31.687739639s) [noise=true , address=104.36.82.38:4719 , piece_id=PYZMWL3OZIULLOGDVKRF2RSLGLDGGWTCFWHQ66CPKPPEP6JGCVXQ, seg_pos={0 0}, upload=121710]: Done uploading piece. err: <nil> Aug 13 14:25:37.294487646 (116h59m31.725567777s) [noise=true , address=77.90.31.15:4702 , piece_id=SOXOEHUIJGYJKQ64YWBZMGAZMQJHR5RQMP3VFS5SGDSFM6J52ZMQ, seg_pos={0 0}, upload=121710]: Done uploading piece. err: <nil> Aug 13 14:25:37.310182075 (116h59m31.741262206s) [noise=true , address=104.36.82.62:4412 , piece_id=CTYSBOF4JVZB5JTYNQDMP6Q5XK5KVETQH5SHHSR65K6HF3TZM2WQ, seg_pos={0 0}, upload=121710]: Done uploading piece. err: <nil> Aug 13 14:25:37.514693066 (116h59m31.945773197s) [noise=true , address=104.36.82.23:4474 , piece_id=MDCBWJSKMP57ZE2V5WIKCG7QWIMRAOOIIFAAJQ5YXVWBEG2EG45Q, seg_pos={0 0}, upload=121710]: Done uploading piece. err: <nil> Aug 13 14:25:37.514716501 (116h59m31.945796632s) [seg_pos={0 0}, upload=121710]: Segment reached optimal threshold of 14 pieces. Aug 13 14:25:37.514831381 (116h59m31.945911512s) [noise=true , address=64.49.13.140:4511 , piece_id=VTJ5NDREGII5G4IOP44JNYRJ5IFHHRP7S7OWTRFOGL4FCKBAUDYA, seg_pos={0 0}, upload=121710]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1F79nPRFzy9hjCq5KYvLy6bZERw9cEvczCN2o66Cf6ccKEpkAk): protocol: expected piece hash; context canceled; EOF Aug 13 14:25:37.514874733 (116h59m31.945954864s) [noise=true , address=104.36.82.56:4235 , piece_id=HS4RRPCLDB5ZMD2EJZZCTSCWVY3TWZC6QDVS4TRXRHO3EGZGR7UQ, seg_pos={0 0}, upload=121710]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1a1sW1G1WPL5WQ2XxprWKQMsJkcb5ETSkk3ULp1TjRaZWptaT): protocol: expected piece hash; context canceled; EOF Aug 13 14:25:37.515019360 (116h59m31.946099492s) [noise=true , address=104.36.82.44:4874 , piece_id=C2ATKCPCO7P4KSHSMRRNGYCMH4IRDKUHL5ERDJKIJ4OECUCLQCFQ, seg_pos={0 0}, upload=121710]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YApHMARaC7Wxeqopt72gpnyJS1xYBwhV7wbU9NJdR6W7hukyq): protocol: write tcp 188.166.225.170:50256->104.36.82.44:4874: use of closed network connection; write tcp 188.166.225.170:50256->104.36.82.44:4874: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50256->104.36.82.44:4874: use of closed network connection Aug 13 14:25:37.515053225 (116h59m31.946133356s) [noise=true , address=104.36.82.60:4371 , piece_id=RTCVTBKO26FY7ZUNFYYJ2IOQ5XJCGG774ZP4WFLOVK5D47ULGXOA, seg_pos={0 0}, upload=121710]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KJP5DHmfnSFQF3Q74GzMUvcAz9zh16td4pnsqHAwToiM4D1vd): protocol: write tcp 188.166.225.170:54100->104.36.82.60:4371: use of closed network connection; write tcp 188.166.225.170:54100->104.36.82.60:4371: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54100->104.36.82.60:4371: use of closed network connection Aug 13 14:25:37.515074867 (116h59m31.946154998s) [seg_pos={0 0}, upload=121710]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 13 14:25:37.515134190 (116h59m31.946214321s) [upload=121710]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 13 14:25:37.515138007 (116h59m31.946218138s) [upload=121710]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 13 14:25:38.573880469 (116h59m33.0049606s) [upload=121710]: Done uploading segments... Aug 13 14:25:38.573909565 (116h59m33.004989686s) [upload=121710]: Upload finished. err: <nil> Duration: 6.394889082s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 5 out of 60 is failed (threshold=3) [...........................X..X.............X.....X.X.......] duration: 1.958595916s FAIL (2025-08-13 14:30:41 UTC) orbiter warning: operation: ListObjects ======ERROR====== unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/1K ======LOGS====== Aug 13 14:25:57.508778246 (116h59m51.939858377s) [upload=121717]: Starting upload Aug 13 14:25:57.508850885 (116h59m51.939931016s) [upload=121717]: Done starting upload Aug 13 14:25:57.508940917 (116h59m51.940021058s) [upload=121717]: Uploading segments... Aug 13 14:25:57.509065897 (116h59m51.940146038s) [upload=121717]: Got next segment. Inline: true Aug 13 14:25:57.509071818 (116h59m51.940151959s) [upload=121717]: Waiting for error group managing segments... Aug 13 14:25:57.509160628 (116h59m51.940240769s) [upload=121717]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 13 14:25:57.509169355 (116h59m51.940249496s) [upload=121717]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 13 14:25:57.509174675 (116h59m51.940254806s) [upload=121717]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 13 14:25:58.324969756 (116h59m52.756049887s) [upload=121717]: Done uploading segments... Aug 13 14:25:58.324989163 (116h59m52.756069294s) [upload=121717]: Upload finished. err: <nil> Duration:* 1.958653636s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 5 out of 60 is failed (threshold=3) [............................X..X.............X.....X.X......] duration: 2.123789821s FAIL (2025-08-13 14:30:25 UTC) orbiter warning: operation: ListObjects ======ERROR====== unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/1K ======LOGS====== Aug 13 14:25:57.508778246 (116h59m51.939858377s) [upload=121717]: Starting upload Aug 13 14:25:57.508850885 (116h59m51.939931016s) [upload=121717]: Done starting upload Aug 13 14:25:57.508940917 (116h59m51.940021058s) [upload=121717]: Uploading segments... Aug 13 14:25:57.509065897 (116h59m51.940146038s) [upload=121717]: Got next segment. Inline: true Aug 13 14:25:57.509071818 (116h59m51.940151959s) [upload=121717]: Waiting for error group managing segments... Aug 13 14:25:57.509160628 (116h59m51.940240769s) [upload=121717]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 13 14:25:57.509169355 (116h59m51.940249496s) [upload=121717]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 13 14:25:57.509174675 (116h59m51.940254806s) [upload=121717]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 13 14:25:58.324969756 (116h59m52.756049887s) [upload=121717]: Done uploading segments... Aug 13 14:25:58.324989163 (116h59m52.756069294s) [upload=121717]: Upload finished. err: <nil> Duration:* 2.123820519s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.............................X.......X.X....................] duration: 7.870990261s FAIL (2025-08-13 14:30:22 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 7.871015761s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 4 out of 60 is failed (threshold=3) [..................X.....X......X.....................X......] duration: 7.418479991s FAIL (2025-08-13 14:29:55 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/4M") ======LOGS====== Aug 13 14:25:34.670450049 (116h59m29.10153018s) [upload=121710]: Starting upload Aug 13 14:25:34.670500074 (116h59m29.101580205s) [upload=121710]: Done starting upload Aug 13 14:25:34.670554288 (116h59m29.101634409s) [upload=121710]: Uploading segments... Aug 13 14:25:34.670715417 (116h59m29.101795548s) [upload=121710]: Got next segment. Inline: false Aug 13 14:25:34.670762726 (116h59m29.101842857s) [upload=121710]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 13 14:25:34.670769440 (116h59m29.101849631s) [upload=121710]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 13 14:25:35.141817037 (116h59m29.572897168s) [seg_pos={0 0}, upload=121710]: Begin upload segment... Aug 13 14:25:35.141863436 (116h59m29.572943567s) [seg_pos={0 0}, upload=121710]: Done begin upload segment. Aug 13 14:25:35.141887462 (116h59m29.572967593s) [noise=true , address=104.36.82.38:4719 , piece_id=PYZMWL3OZIULLOGDVKRF2RSLGLDGGWTCFWHQ66CPKPPEP6JGCVXQ, seg_pos={0 0}, upload=121710]: Uploading piece... Aug 13 14:25:35.141902260 (116h59m29.572982391s) [noise=true , address=104.36.82.59:4328 , piece_id=EVWPCNQWESLH26NS62FKFXKOIW64GUDZFCABBK6H4FSYP35OA5OA, seg_pos={0 0}, upload=121710]: Uploading piece... Aug 13 14:25:35.141945994 (116h59m29.573026125s) [noise=true , address=104.36.82.56:4235 , piece_id=HS4RRPCLDB5ZMD2EJZZCTSCWVY3TWZC6QDVS4TRXRHO3EGZGR7UQ, seg_pos={0 0}, upload=121710]: Uploading piece... Aug 13 14:25:35.142200952 (116h59m29.573281083s) [noise=true , address=104.36.82.60:4371 , piece_id=RTCVTBKO26FY7ZUNFYYJ2IOQ5XJCGG774ZP4WFLOVK5D47ULGXOA, seg_pos={0 0}, upload=121710]: Uploading piece... Aug 13 14:25:35.142305862 (116h59m29.573385993s) [noise=true , address=104.36.82.44:4874 , piece_id=C2ATKCPCO7P4KSHSMRRNGYCMH4IRDKUHL5ERDJKIJ4OECUCLQCFQ, seg_pos={0 0}, upload=121710]: Uploading piece... Aug 13 14:25:35.142439999 (116h59m29.57352013s) [noise=true , address=104.36.82.19:4408 , piece_id=LT43HOEP3H4NTUKHSRUXPSRO4XBEJ6NOHIAPCBPZKVXPIW7SLLQQ, seg_pos={0 0}, upload=121710]: Uploading piece... Aug 13 14:25:35.142605826 (116h59m29.573685957s) [noise=true , address=77.90.31.15:4702 , piece_id=SOXOEHUIJGYJKQ64YWBZMGAZMQJHR5RQMP3VFS5SGDSFM6J52ZMQ, seg_pos={0 0}, upload=121710]: Uploading piece... Aug 13 14:25:35.142682213 (116h59m29.573762344s) [noise=true , address=77.90.31.19:4846 , piece_id=3UYXBMMHS5MPUNXI6PY65K3AGHIIIVSGXJDEUVXLTVXCZYWTL7JA, seg_pos={0 0}, upload=121710]: Uploading piece... Aug 13 14:25:35.142910119 (116h59m29.57399024s) [noise=true , address=104.36.82.37:4678 , piece_id=S5AV5WE4DOUZIRLT3GMKQLPLHARLF24HTVG3QNJ2GWZPYIC55PGQ, seg_pos={0 0}, upload=121710]: Uploading piece... Aug 13 14:25:35.142917733 (116h59m29.573997864s) [noise=true , address=66.23.193.175:4564 , piece_id=S4CHPP736M3D5GOURHJRONGXSYNTREBY3S243VJ6E5ZM75TDTLPA, seg_pos={0 0}, upload=121710]: Uploading piece... Aug 13 14:25:35.143203881 (116h59m29.574284012s) [noise=true , address=64.49.13.140:4511 , piece_id=VTJ5NDREGII5G4IOP44JNYRJ5IFHHRP7S7OWTRFOGL4FCKBAUDYA, seg_pos={0 0}, upload=121710]: Uploading piece... Aug 13 14:25:35.143239018 (116h59m29.574319149s) [noise=true , address=104.36.82.34:4560 , piece_id=4ZZFZKJRUXPMKOE2RDH6ZWFP3TVUOPSRQS7FAB2YANQ3MKA7JLWQ, seg_pos={0 0}, upload=121710]: Uploading piece... Aug 13 14:25:35.143481342 (116h59m29.574561473s) [noise=true , address=104.36.82.39:4727 , piece_id=UYFASTBHHBMDAY7ZUYHNSNSDDZMPPZ4AIR6X6HC6ERBXLXF4AOUQ, seg_pos={0 0}, upload=121710]: Uploading piece... Aug 13 14:25:35.143486071 (116h59m29.574566192s) [noise=true , address=104.36.82.23:4474 , piece_id=MDCBWJSKMP57ZE2V5WIKCG7QWIMRAOOIIFAAJQ5YXVWBEG2EG45Q, seg_pos={0 0}, upload=121710]: Uploading piece... Aug 13 14:25:35.143749975 (116h59m29.574830106s) [noise=true , address=77.90.31.11:4580 , piece_id=CBVCJWDPVPPQW2JIAV25UUFT7EVH62WJOP3FK4M7GUVJAO6P2QGA, seg_pos={0 0}, upload=121710]: Uploading piece... Aug 13 14:25:35.144014762 (116h59m29.575094893s) [noise=true , address=104.36.82.62:4412 , piece_id=CTYSBOF4JVZB5JTYNQDMP6Q5XK5KVETQH5SHHSR65K6HF3TZM2WQ, seg_pos={0 0}, upload=121710]: Uploading piece... Aug 13 14:25:35.146502863 (116h59m29.577582994s) [noise=true , address=79.127.201.215:4358 , piece_id=5LNV5OH2YBCRCO7GJOK6Q3VSYC47ZJ4VTZQO3XVN7BLIRFRB4J5A, seg_pos={0 0}, upload=121710]: Uploading piece... Aug 13 14:25:35.146602894 (116h59m29.577683025s) [noise=true , address=104.36.82.36:4642 , piece_id=TILD5YD3ZU4AOT5227X6PIEIKYXF52D2AZSNDWLOENLTQ4ZMVSLA, seg_pos={0 0}, upload=121710]: Uploading piece... Aug 13 14:25:36.312798371 (116h59m30.743878492s) [upload=121710]: Next returned nil segment Aug 13 14:25:36.312814072 (116h59m30.743894203s) [upload=121710]: Waiting for error group managing segments... Aug 13 14:25:36.608833520 (116h59m31.039913651s) [noise=true , address=66.23.193.175:4564 , piece_id=S4CHPP736M3D5GOURHJRONGXSYNTREBY3S243VJ6E5ZM75TDTLPA, seg_pos={0 0}, upload=121710]: Done uploading piece. err: <nil> Aug 13 14:25:37.109920204 (116h59m31.541000325s) [noise=true , address=79.127.201.215:4358 , piece_id=5LNV5OH2YBCRCO7GJOK6Q3VSYC47ZJ4VTZQO3XVN7BLIRFRB4J5A, seg_pos={0 0}, upload=121710]: Done uploading piece. err: <nil> Aug 13 14:25:37.110478522 (116h59m31.541558653s) [noise=true , address=104.36.82.36:4642 , piece_id=TILD5YD3ZU4AOT5227X6PIEIKYXF52D2AZSNDWLOENLTQ4ZMVSLA, seg_pos={0 0}, upload=121710]: Done uploading piece. err: <nil> Aug 13 14:25:37.127900888 (116h59m31.558981009s) [noise=true , address=77.90.31.11:4580 , piece_id=CBVCJWDPVPPQW2JIAV25UUFT7EVH62WJOP3FK4M7GUVJAO6P2QGA, seg_pos={0 0}, upload=121710]: Done uploading piece. err: <nil> Aug 13 14:25:37.166070970 (116h59m31.597151101s) [noise=true , address=104.36.82.37:4678 , piece_id=S5AV5WE4DOUZIRLT3GMKQLPLHARLF24HTVG3QNJ2GWZPYIC55PGQ, seg_pos={0 0}, upload=121710]: Done uploading piece. err: <nil> Aug 13 14:25:37.167794408 (116h59m31.598874539s) [noise=true , address=104.36.82.19:4408 , piece_id=LT43HOEP3H4NTUKHSRUXPSRO4XBEJ6NOHIAPCBPZKVXPIW7SLLQQ, seg_pos={0 0}, upload=121710]: Done uploading piece. err: <nil> Aug 13 14:25:37.167973721 (116h59m31.599053852s) [noise=true , address=104.36.82.59:4328 , piece_id=EVWPCNQWESLH26NS62FKFXKOIW64GUDZFCABBK6H4FSYP35OA5OA, seg_pos={0 0}, upload=121710]: Done uploading piece. err: <nil> Aug 13 14:25:37.195711503 (116h59m31.626791634s) [noise=true , address=104.36.82.34:4560 , piece_id=4ZZFZKJRUXPMKOE2RDH6ZWFP3TVUOPSRQS7FAB2YANQ3MKA7JLWQ, seg_pos={0 0}, upload=121710]: Done uploading piece. err: <nil> Aug 13 14:25:37.209376008 (116h59m31.640456129s) [noise=true , address=104.36.82.39:4727 , piece_id=UYFASTBHHBMDAY7ZUYHNSNSDDZMPPZ4AIR6X6HC6ERBXLXF4AOUQ, seg_pos={0 0}, upload=121710]: Done uploading piece. err: <nil> Aug 13 14:25:37.252511451 (116h59m31.683591582s) [noise=true , address=77.90.31.19:4846 , piece_id=3UYXBMMHS5MPUNXI6PY65K3AGHIIIVSGXJDEUVXLTVXCZYWTL7JA, seg_pos={0 0}, upload=121710]: Done uploading piece. err: <nil> Aug 13 14:25:37.256659508 (116h59m31.687739639s) [noise=true , address=104.36.82.38:4719 , piece_id=PYZMWL3OZIULLOGDVKRF2RSLGLDGGWTCFWHQ66CPKPPEP6JGCVXQ, seg_pos={0 0}, upload=121710]: Done uploading piece. err: <nil> Aug 13 14:25:37.294487646 (116h59m31.725567777s) [noise=true , address=77.90.31.15:4702 , piece_id=SOXOEHUIJGYJKQ64YWBZMGAZMQJHR5RQMP3VFS5SGDSFM6J52ZMQ, seg_pos={0 0}, upload=121710]: Done uploading piece. err: <nil> Aug 13 14:25:37.310182075 (116h59m31.741262206s) [noise=true , address=104.36.82.62:4412 , piece_id=CTYSBOF4JVZB5JTYNQDMP6Q5XK5KVETQH5SHHSR65K6HF3TZM2WQ, seg_pos={0 0}, upload=121710]: Done uploading piece. err: <nil> Aug 13 14:25:37.514693066 (116h59m31.945773197s) [noise=true , address=104.36.82.23:4474 , piece_id=MDCBWJSKMP57ZE2V5WIKCG7QWIMRAOOIIFAAJQ5YXVWBEG2EG45Q, seg_pos={0 0}, upload=121710]: Done uploading piece. err: <nil> Aug 13 14:25:37.514716501 (116h59m31.945796632s) [seg_pos={0 0}, upload=121710]: Segment reached optimal threshold of 14 pieces. Aug 13 14:25:37.514831381 (116h59m31.945911512s) [noise=true , address=64.49.13.140:4511 , piece_id=VTJ5NDREGII5G4IOP44JNYRJ5IFHHRP7S7OWTRFOGL4FCKBAUDYA, seg_pos={0 0}, upload=121710]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1F79nPRFzy9hjCq5KYvLy6bZERw9cEvczCN2o66Cf6ccKEpkAk): protocol: expected piece hash; context canceled; EOF Aug 13 14:25:37.514874733 (116h59m31.945954864s) [noise=true , address=104.36.82.56:4235 , piece_id=HS4RRPCLDB5ZMD2EJZZCTSCWVY3TWZC6QDVS4TRXRHO3EGZGR7UQ, seg_pos={0 0}, upload=121710]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1a1sW1G1WPL5WQ2XxprWKQMsJkcb5ETSkk3ULp1TjRaZWptaT): protocol: expected piece hash; context canceled; EOF Aug 13 14:25:37.515019360 (116h59m31.946099492s) [noise=true , address=104.36.82.44:4874 , piece_id=C2ATKCPCO7P4KSHSMRRNGYCMH4IRDKUHL5ERDJKIJ4OECUCLQCFQ, seg_pos={0 0}, upload=121710]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YApHMARaC7Wxeqopt72gpnyJS1xYBwhV7wbU9NJdR6W7hukyq): protocol: write tcp 188.166.225.170:50256->104.36.82.44:4874: use of closed network connection; write tcp 188.166.225.170:50256->104.36.82.44:4874: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50256->104.36.82.44:4874: use of closed network connection Aug 13 14:25:37.515053225 (116h59m31.946133356s) [noise=true , address=104.36.82.60:4371 , piece_id=RTCVTBKO26FY7ZUNFYYJ2IOQ5XJCGG774ZP4WFLOVK5D47ULGXOA, seg_pos={0 0}, upload=121710]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KJP5DHmfnSFQF3Q74GzMUvcAz9zh16td4pnsqHAwToiM4D1vd): protocol: write tcp 188.166.225.170:54100->104.36.82.60:4371: use of closed network connection; write tcp 188.166.225.170:54100->104.36.82.60:4371: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54100->104.36.82.60:4371: use of closed network connection Aug 13 14:25:37.515074867 (116h59m31.946154998s) [seg_pos={0 0}, upload=121710]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 13 14:25:37.515134190 (116h59m31.946214321s) [upload=121710]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 13 14:25:37.515138007 (116h59m31.946218138s) [upload=121710]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 13 14:25:38.573880469 (116h59m33.0049606s) [upload=121710]: Done uploading segments... Aug 13 14:25:38.573909565 (116h59m33.004989686s) [upload=121710]: Upload finished. err: <nil> Duration: 7.418506753s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 5 out of 60 is failed (threshold=3) [.............................X..X.............X.....X.X.....] duration: 1.935881319s FAIL (2025-08-13 14:29:42 UTC) orbiter warning: operation: ListObjects ======ERROR====== unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/1K ======LOGS====== Aug 13 14:25:57.508778246 (116h59m51.939858377s) [upload=121717]: Starting upload Aug 13 14:25:57.508850885 (116h59m51.939931016s) [upload=121717]: Done starting upload Aug 13 14:25:57.508940917 (116h59m51.940021058s) [upload=121717]: Uploading segments... Aug 13 14:25:57.509065897 (116h59m51.940146038s) [upload=121717]: Got next segment. Inline: true Aug 13 14:25:57.509071818 (116h59m51.940151959s) [upload=121717]: Waiting for error group managing segments... Aug 13 14:25:57.509160628 (116h59m51.940240769s) [upload=121717]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 13 14:25:57.509169355 (116h59m51.940249496s) [upload=121717]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 13 14:25:57.509174675 (116h59m51.940254806s) [upload=121717]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 13 14:25:58.324969756 (116h59m52.756049887s) [upload=121717]: Done uploading segments... Aug 13 14:25:58.324989163 (116h59m52.756069294s) [upload=121717]: Upload finished. err: <nil> Duration:* 1.93590842s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..............................X.......X.X...................] duration: 10.180850864s FAIL (2025-08-13 14:29:28 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 10.180872195s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 5 out of 60 is failed (threshold=3) [X..................X.....X......X.....................X.....] duration: 6.748920737s FAIL (2025-08-13 14:29:21 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/4M") ======LOGS====== Aug 13 14:25:34.670450049 (116h59m29.10153018s) [upload=121710]: Starting upload Aug 13 14:25:34.670500074 (116h59m29.101580205s) [upload=121710]: Done starting upload Aug 13 14:25:34.670554288 (116h59m29.101634409s) [upload=121710]: Uploading segments... Aug 13 14:25:34.670715417 (116h59m29.101795548s) [upload=121710]: Got next segment. Inline: false Aug 13 14:25:34.670762726 (116h59m29.101842857s) [upload=121710]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 13 14:25:34.670769440 (116h59m29.101849631s) [upload=121710]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 13 14:25:35.141817037 (116h59m29.572897168s) [seg_pos={0 0}, upload=121710]: Begin upload segment... Aug 13 14:25:35.141863436 (116h59m29.572943567s) [seg_pos={0 0}, upload=121710]: Done begin upload segment. Aug 13 14:25:35.141887462 (116h59m29.572967593s) [noise=true , address=104.36.82.38:4719 , piece_id=PYZMWL3OZIULLOGDVKRF2RSLGLDGGWTCFWHQ66CPKPPEP6JGCVXQ, seg_pos={0 0}, upload=121710]: Uploading piece... Aug 13 14:25:35.141902260 (116h59m29.572982391s) [noise=true , address=104.36.82.59:4328 , piece_id=EVWPCNQWESLH26NS62FKFXKOIW64GUDZFCABBK6H4FSYP35OA5OA, seg_pos={0 0}, upload=121710]: Uploading piece... Aug 13 14:25:35.141945994 (116h59m29.573026125s) [noise=true , address=104.36.82.56:4235 , piece_id=HS4RRPCLDB5ZMD2EJZZCTSCWVY3TWZC6QDVS4TRXRHO3EGZGR7UQ, seg_pos={0 0}, upload=121710]: Uploading piece... Aug 13 14:25:35.142200952 (116h59m29.573281083s) [noise=true , address=104.36.82.60:4371 , piece_id=RTCVTBKO26FY7ZUNFYYJ2IOQ5XJCGG774ZP4WFLOVK5D47ULGXOA, seg_pos={0 0}, upload=121710]: Uploading piece... Aug 13 14:25:35.142305862 (116h59m29.573385993s) [noise=true , address=104.36.82.44:4874 , piece_id=C2ATKCPCO7P4KSHSMRRNGYCMH4IRDKUHL5ERDJKIJ4OECUCLQCFQ, seg_pos={0 0}, upload=121710]: Uploading piece... Aug 13 14:25:35.142439999 (116h59m29.57352013s) [noise=true , address=104.36.82.19:4408 , piece_id=LT43HOEP3H4NTUKHSRUXPSRO4XBEJ6NOHIAPCBPZKVXPIW7SLLQQ, seg_pos={0 0}, upload=121710]: Uploading piece... Aug 13 14:25:35.142605826 (116h59m29.573685957s) [noise=true , address=77.90.31.15:4702 , piece_id=SOXOEHUIJGYJKQ64YWBZMGAZMQJHR5RQMP3VFS5SGDSFM6J52ZMQ, seg_pos={0 0}, upload=121710]: Uploading piece... Aug 13 14:25:35.142682213 (116h59m29.573762344s) [noise=true , address=77.90.31.19:4846 , piece_id=3UYXBMMHS5MPUNXI6PY65K3AGHIIIVSGXJDEUVXLTVXCZYWTL7JA, seg_pos={0 0}, upload=121710]: Uploading piece... Aug 13 14:25:35.142910119 (116h59m29.57399024s) [noise=true , address=104.36.82.37:4678 , piece_id=S5AV5WE4DOUZIRLT3GMKQLPLHARLF24HTVG3QNJ2GWZPYIC55PGQ, seg_pos={0 0}, upload=121710]: Uploading piece... Aug 13 14:25:35.142917733 (116h59m29.573997864s) [noise=true , address=66.23.193.175:4564 , piece_id=S4CHPP736M3D5GOURHJRONGXSYNTREBY3S243VJ6E5ZM75TDTLPA, seg_pos={0 0}, upload=121710]: Uploading piece... Aug 13 14:25:35.143203881 (116h59m29.574284012s) [noise=true , address=64.49.13.140:4511 , piece_id=VTJ5NDREGII5G4IOP44JNYRJ5IFHHRP7S7OWTRFOGL4FCKBAUDYA, seg_pos={0 0}, upload=121710]: Uploading piece... Aug 13 14:25:35.143239018 (116h59m29.574319149s) [noise=true , address=104.36.82.34:4560 , piece_id=4ZZFZKJRUXPMKOE2RDH6ZWFP3TVUOPSRQS7FAB2YANQ3MKA7JLWQ, seg_pos={0 0}, upload=121710]: Uploading piece... Aug 13 14:25:35.143481342 (116h59m29.574561473s) [noise=true , address=104.36.82.39:4727 , piece_id=UYFASTBHHBMDAY7ZUYHNSNSDDZMPPZ4AIR6X6HC6ERBXLXF4AOUQ, seg_pos={0 0}, upload=121710]: Uploading piece... Aug 13 14:25:35.143486071 (116h59m29.574566192s) [noise=true , address=104.36.82.23:4474 , piece_id=MDCBWJSKMP57ZE2V5WIKCG7QWIMRAOOIIFAAJQ5YXVWBEG2EG45Q, seg_pos={0 0}, upload=121710]: Uploading piece... Aug 13 14:25:35.143749975 (116h59m29.574830106s) [noise=true , address=77.90.31.11:4580 , piece_id=CBVCJWDPVPPQW2JIAV25UUFT7EVH62WJOP3FK4M7GUVJAO6P2QGA, seg_pos={0 0}, upload=121710]: Uploading piece... Aug 13 14:25:35.144014762 (116h59m29.575094893s) [noise=true , address=104.36.82.62:4412 , piece_id=CTYSBOF4JVZB5JTYNQDMP6Q5XK5KVETQH5SHHSR65K6HF3TZM2WQ, seg_pos={0 0}, upload=121710]: Uploading piece... Aug 13 14:25:35.146502863 (116h59m29.577582994s) [noise=true , address=79.127.201.215:4358 , piece_id=5LNV5OH2YBCRCO7GJOK6Q3VSYC47ZJ4VTZQO3XVN7BLIRFRB4J5A, seg_pos={0 0}, upload=121710]: Uploading piece... Aug 13 14:25:35.146602894 (116h59m29.577683025s) [noise=true , address=104.36.82.36:4642 , piece_id=TILD5YD3ZU4AOT5227X6PIEIKYXF52D2AZSNDWLOENLTQ4ZMVSLA, seg_pos={0 0}, upload=121710]: Uploading piece... Aug 13 14:25:36.312798371 (116h59m30.743878492s) [upload=121710]: Next returned nil segment Aug 13 14:25:36.312814072 (116h59m30.743894203s) [upload=121710]: Waiting for error group managing segments... Aug 13 14:25:36.608833520 (116h59m31.039913651s) [noise=true , address=66.23.193.175:4564 , piece_id=S4CHPP736M3D5GOURHJRONGXSYNTREBY3S243VJ6E5ZM75TDTLPA, seg_pos={0 0}, upload=121710]: Done uploading piece. err: <nil> Aug 13 14:25:37.109920204 (116h59m31.541000325s) [noise=true , address=79.127.201.215:4358 , piece_id=5LNV5OH2YBCRCO7GJOK6Q3VSYC47ZJ4VTZQO3XVN7BLIRFRB4J5A, seg_pos={0 0}, upload=121710]: Done uploading piece. err: <nil> Aug 13 14:25:37.110478522 (116h59m31.541558653s) [noise=true , address=104.36.82.36:4642 , piece_id=TILD5YD3ZU4AOT5227X6PIEIKYXF52D2AZSNDWLOENLTQ4ZMVSLA, seg_pos={0 0}, upload=121710]: Done uploading piece. err: <nil> Aug 13 14:25:37.127900888 (116h59m31.558981009s) [noise=true , address=77.90.31.11:4580 , piece_id=CBVCJWDPVPPQW2JIAV25UUFT7EVH62WJOP3FK4M7GUVJAO6P2QGA, seg_pos={0 0}, upload=121710]: Done uploading piece. err: <nil> Aug 13 14:25:37.166070970 (116h59m31.597151101s) [noise=true , address=104.36.82.37:4678 , piece_id=S5AV5WE4DOUZIRLT3GMKQLPLHARLF24HTVG3QNJ2GWZPYIC55PGQ, seg_pos={0 0}, upload=121710]: Done uploading piece. err: <nil> Aug 13 14:25:37.167794408 (116h59m31.598874539s) [noise=true , address=104.36.82.19:4408 , piece_id=LT43HOEP3H4NTUKHSRUXPSRO4XBEJ6NOHIAPCBPZKVXPIW7SLLQQ, seg_pos={0 0}, upload=121710]: Done uploading piece. err: <nil> Aug 13 14:25:37.167973721 (116h59m31.599053852s) [noise=true , address=104.36.82.59:4328 , piece_id=EVWPCNQWESLH26NS62FKFXKOIW64GUDZFCABBK6H4FSYP35OA5OA, seg_pos={0 0}, upload=121710]: Done uploading piece. err: <nil> Aug 13 14:25:37.195711503 (116h59m31.626791634s) [noise=true , address=104.36.82.34:4560 , piece_id=4ZZFZKJRUXPMKOE2RDH6ZWFP3TVUOPSRQS7FAB2YANQ3MKA7JLWQ, seg_pos={0 0}, upload=121710]: Done uploading piece. err: <nil> Aug 13 14:25:37.209376008 (116h59m31.640456129s) [noise=true , address=104.36.82.39:4727 , piece_id=UYFASTBHHBMDAY7ZUYHNSNSDDZMPPZ4AIR6X6HC6ERBXLXF4AOUQ, seg_pos={0 0}, upload=121710]: Done uploading piece. err: <nil> Aug 13 14:25:37.252511451 (116h59m31.683591582s) [noise=true , address=77.90.31.19:4846 , piece_id=3UYXBMMHS5MPUNXI6PY65K3AGHIIIVSGXJDEUVXLTVXCZYWTL7JA, seg_pos={0 0}, upload=121710]: Done uploading piece. err: <nil> Aug 13 14:25:37.256659508 (116h59m31.687739639s) [noise=true , address=104.36.82.38:4719 , piece_id=PYZMWL3OZIULLOGDVKRF2RSLGLDGGWTCFWHQ66CPKPPEP6JGCVXQ, seg_pos={0 0}, upload=121710]: Done uploading piece. err: <nil> Aug 13 14:25:37.294487646 (116h59m31.725567777s) [noise=true , address=77.90.31.15:4702 , piece_id=SOXOEHUIJGYJKQ64YWBZMGAZMQJHR5RQMP3VFS5SGDSFM6J52ZMQ, seg_pos={0 0}, upload=121710]: Done uploading piece. err: <nil> Aug 13 14:25:37.310182075 (116h59m31.741262206s) [noise=true , address=104.36.82.62:4412 , piece_id=CTYSBOF4JVZB5JTYNQDMP6Q5XK5KVETQH5SHHSR65K6HF3TZM2WQ, seg_pos={0 0}, upload=121710]: Done uploading piece. err: <nil> Aug 13 14:25:37.514693066 (116h59m31.945773197s) [noise=true , address=104.36.82.23:4474 , piece_id=MDCBWJSKMP57ZE2V5WIKCG7QWIMRAOOIIFAAJQ5YXVWBEG2EG45Q, seg_pos={0 0}, upload=121710]: Done uploading piece. err: <nil> Aug 13 14:25:37.514716501 (116h59m31.945796632s) [seg_pos={0 0}, upload=121710]: Segment reached optimal threshold of 14 pieces. Aug 13 14:25:37.514831381 (116h59m31.945911512s) [noise=true , address=64.49.13.140:4511 , piece_id=VTJ5NDREGII5G4IOP44JNYRJ5IFHHRP7S7OWTRFOGL4FCKBAUDYA, seg_pos={0 0}, upload=121710]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1F79nPRFzy9hjCq5KYvLy6bZERw9cEvczCN2o66Cf6ccKEpkAk): protocol: expected piece hash; context canceled; EOF Aug 13 14:25:37.514874733 (116h59m31.945954864s) [noise=true , address=104.36.82.56:4235 , piece_id=HS4RRPCLDB5ZMD2EJZZCTSCWVY3TWZC6QDVS4TRXRHO3EGZGR7UQ, seg_pos={0 0}, upload=121710]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1a1sW1G1WPL5WQ2XxprWKQMsJkcb5ETSkk3ULp1TjRaZWptaT): protocol: expected piece hash; context canceled; EOF Aug 13 14:25:37.515019360 (116h59m31.946099492s) [noise=true , address=104.36.82.44:4874 , piece_id=C2ATKCPCO7P4KSHSMRRNGYCMH4IRDKUHL5ERDJKIJ4OECUCLQCFQ, seg_pos={0 0}, upload=121710]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YApHMARaC7Wxeqopt72gpnyJS1xYBwhV7wbU9NJdR6W7hukyq): protocol: write tcp 188.166.225.170:50256->104.36.82.44:4874: use of closed network connection; write tcp 188.166.225.170:50256->104.36.82.44:4874: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50256->104.36.82.44:4874: use of closed network connection Aug 13 14:25:37.515053225 (116h59m31.946133356s) [noise=true , address=104.36.82.60:4371 , piece_id=RTCVTBKO26FY7ZUNFYYJ2IOQ5XJCGG774ZP4WFLOVK5D47ULGXOA, seg_pos={0 0}, upload=121710]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KJP5DHmfnSFQF3Q74GzMUvcAz9zh16td4pnsqHAwToiM4D1vd): protocol: write tcp 188.166.225.170:54100->104.36.82.60:4371: use of closed network connection; write tcp 188.166.225.170:54100->104.36.82.60:4371: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54100->104.36.82.60:4371: use of closed network connection Aug 13 14:25:37.515074867 (116h59m31.946154998s) [seg_pos={0 0}, upload=121710]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 13 14:25:37.515134190 (116h59m31.946214321s) [upload=121710]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 13 14:25:37.515138007 (116h59m31.946218138s) [upload=121710]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 13 14:25:38.573880469 (116h59m33.0049606s) [upload=121710]: Done uploading segments... Aug 13 14:25:38.573909565 (116h59m33.004989686s) [upload=121710]: Upload finished. err: <nil> Duration: 6.748945576s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 5 out of 60 is failed (threshold=3) [.X..................X.....X......X.....................X....] duration: 6.393914388s FAIL (2025-08-13 14:28:59 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/4M") ======LOGS====== Aug 13 14:25:34.670450049 (116h59m29.10153018s) [upload=121710]: Starting upload Aug 13 14:25:34.670500074 (116h59m29.101580205s) [upload=121710]: Done starting upload Aug 13 14:25:34.670554288 (116h59m29.101634409s) [upload=121710]: Uploading segments... Aug 13 14:25:34.670715417 (116h59m29.101795548s) [upload=121710]: Got next segment. Inline: false Aug 13 14:25:34.670762726 (116h59m29.101842857s) [upload=121710]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 13 14:25:34.670769440 (116h59m29.101849631s) [upload=121710]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 13 14:25:35.141817037 (116h59m29.572897168s) [seg_pos={0 0}, upload=121710]: Begin upload segment... Aug 13 14:25:35.141863436 (116h59m29.572943567s) [seg_pos={0 0}, upload=121710]: Done begin upload segment. Aug 13 14:25:35.141887462 (116h59m29.572967593s) [noise=true , address=104.36.82.38:4719 , piece_id=PYZMWL3OZIULLOGDVKRF2RSLGLDGGWTCFWHQ66CPKPPEP6JGCVXQ, seg_pos={0 0}, upload=121710]: Uploading piece... Aug 13 14:25:35.141902260 (116h59m29.572982391s) [noise=true , address=104.36.82.59:4328 , piece_id=EVWPCNQWESLH26NS62FKFXKOIW64GUDZFCABBK6H4FSYP35OA5OA, seg_pos={0 0}, upload=121710]: Uploading piece... Aug 13 14:25:35.141945994 (116h59m29.573026125s) [noise=true , address=104.36.82.56:4235 , piece_id=HS4RRPCLDB5ZMD2EJZZCTSCWVY3TWZC6QDVS4TRXRHO3EGZGR7UQ, seg_pos={0 0}, upload=121710]: Uploading piece... Aug 13 14:25:35.142200952 (116h59m29.573281083s) [noise=true , address=104.36.82.60:4371 , piece_id=RTCVTBKO26FY7ZUNFYYJ2IOQ5XJCGG774ZP4WFLOVK5D47ULGXOA, seg_pos={0 0}, upload=121710]: Uploading piece... Aug 13 14:25:35.142305862 (116h59m29.573385993s) [noise=true , address=104.36.82.44:4874 , piece_id=C2ATKCPCO7P4KSHSMRRNGYCMH4IRDKUHL5ERDJKIJ4OECUCLQCFQ, seg_pos={0 0}, upload=121710]: Uploading piece... Aug 13 14:25:35.142439999 (116h59m29.57352013s) [noise=true , address=104.36.82.19:4408 , piece_id=LT43HOEP3H4NTUKHSRUXPSRO4XBEJ6NOHIAPCBPZKVXPIW7SLLQQ, seg_pos={0 0}, upload=121710]: Uploading piece... Aug 13 14:25:35.142605826 (116h59m29.573685957s) [noise=true , address=77.90.31.15:4702 , piece_id=SOXOEHUIJGYJKQ64YWBZMGAZMQJHR5RQMP3VFS5SGDSFM6J52ZMQ, seg_pos={0 0}, upload=121710]: Uploading piece... Aug 13 14:25:35.142682213 (116h59m29.573762344s) [noise=true , address=77.90.31.19:4846 , piece_id=3UYXBMMHS5MPUNXI6PY65K3AGHIIIVSGXJDEUVXLTVXCZYWTL7JA, seg_pos={0 0}, upload=121710]: Uploading piece... Aug 13 14:25:35.142910119 (116h59m29.57399024s) [noise=true , address=104.36.82.37:4678 , piece_id=S5AV5WE4DOUZIRLT3GMKQLPLHARLF24HTVG3QNJ2GWZPYIC55PGQ, seg_pos={0 0}, upload=121710]: Uploading piece... Aug 13 14:25:35.142917733 (116h59m29.573997864s) [noise=true , address=66.23.193.175:4564 , piece_id=S4CHPP736M3D5GOURHJRONGXSYNTREBY3S243VJ6E5ZM75TDTLPA, seg_pos={0 0}, upload=121710]: Uploading piece... Aug 13 14:25:35.143203881 (116h59m29.574284012s) [noise=true , address=64.49.13.140:4511 , piece_id=VTJ5NDREGII5G4IOP44JNYRJ5IFHHRP7S7OWTRFOGL4FCKBAUDYA, seg_pos={0 0}, upload=121710]: Uploading piece... Aug 13 14:25:35.143239018 (116h59m29.574319149s) [noise=true , address=104.36.82.34:4560 , piece_id=4ZZFZKJRUXPMKOE2RDH6ZWFP3TVUOPSRQS7FAB2YANQ3MKA7JLWQ, seg_pos={0 0}, upload=121710]: Uploading piece... Aug 13 14:25:35.143481342 (116h59m29.574561473s) [noise=true , address=104.36.82.39:4727 , piece_id=UYFASTBHHBMDAY7ZUYHNSNSDDZMPPZ4AIR6X6HC6ERBXLXF4AOUQ, seg_pos={0 0}, upload=121710]: Uploading piece... Aug 13 14:25:35.143486071 (116h59m29.574566192s) [noise=true , address=104.36.82.23:4474 , piece_id=MDCBWJSKMP57ZE2V5WIKCG7QWIMRAOOIIFAAJQ5YXVWBEG2EG45Q, seg_pos={0 0}, upload=121710]: Uploading piece... Aug 13 14:25:35.143749975 (116h59m29.574830106s) [noise=true , address=77.90.31.11:4580 , piece_id=CBVCJWDPVPPQW2JIAV25UUFT7EVH62WJOP3FK4M7GUVJAO6P2QGA, seg_pos={0 0}, upload=121710]: Uploading piece... Aug 13 14:25:35.144014762 (116h59m29.575094893s) [noise=true , address=104.36.82.62:4412 , piece_id=CTYSBOF4JVZB5JTYNQDMP6Q5XK5KVETQH5SHHSR65K6HF3TZM2WQ, seg_pos={0 0}, upload=121710]: Uploading piece... Aug 13 14:25:35.146502863 (116h59m29.577582994s) [noise=true , address=79.127.201.215:4358 , piece_id=5LNV5OH2YBCRCO7GJOK6Q3VSYC47ZJ4VTZQO3XVN7BLIRFRB4J5A, seg_pos={0 0}, upload=121710]: Uploading piece... Aug 13 14:25:35.146602894 (116h59m29.577683025s) [noise=true , address=104.36.82.36:4642 , piece_id=TILD5YD3ZU4AOT5227X6PIEIKYXF52D2AZSNDWLOENLTQ4ZMVSLA, seg_pos={0 0}, upload=121710]: Uploading piece... Aug 13 14:25:36.312798371 (116h59m30.743878492s) [upload=121710]: Next returned nil segment Aug 13 14:25:36.312814072 (116h59m30.743894203s) [upload=121710]: Waiting for error group managing segments... Aug 13 14:25:36.608833520 (116h59m31.039913651s) [noise=true , address=66.23.193.175:4564 , piece_id=S4CHPP736M3D5GOURHJRONGXSYNTREBY3S243VJ6E5ZM75TDTLPA, seg_pos={0 0}, upload=121710]: Done uploading piece. err: <nil> Aug 13 14:25:37.109920204 (116h59m31.541000325s) [noise=true , address=79.127.201.215:4358 , piece_id=5LNV5OH2YBCRCO7GJOK6Q3VSYC47ZJ4VTZQO3XVN7BLIRFRB4J5A, seg_pos={0 0}, upload=121710]: Done uploading piece. err: <nil> Aug 13 14:25:37.110478522 (116h59m31.541558653s) [noise=true , address=104.36.82.36:4642 , piece_id=TILD5YD3ZU4AOT5227X6PIEIKYXF52D2AZSNDWLOENLTQ4ZMVSLA, seg_pos={0 0}, upload=121710]: Done uploading piece. err: <nil> Aug 13 14:25:37.127900888 (116h59m31.558981009s) [noise=true , address=77.90.31.11:4580 , piece_id=CBVCJWDPVPPQW2JIAV25UUFT7EVH62WJOP3FK4M7GUVJAO6P2QGA, seg_pos={0 0}, upload=121710]: Done uploading piece. err: <nil> Aug 13 14:25:37.166070970 (116h59m31.597151101s) [noise=true , address=104.36.82.37:4678 , piece_id=S5AV5WE4DOUZIRLT3GMKQLPLHARLF24HTVG3QNJ2GWZPYIC55PGQ, seg_pos={0 0}, upload=121710]: Done uploading piece. err: <nil> Aug 13 14:25:37.167794408 (116h59m31.598874539s) [noise=true , address=104.36.82.19:4408 , piece_id=LT43HOEP3H4NTUKHSRUXPSRO4XBEJ6NOHIAPCBPZKVXPIW7SLLQQ, seg_pos={0 0}, upload=121710]: Done uploading piece. err: <nil> Aug 13 14:25:37.167973721 (116h59m31.599053852s) [noise=true , address=104.36.82.59:4328 , piece_id=EVWPCNQWESLH26NS62FKFXKOIW64GUDZFCABBK6H4FSYP35OA5OA, seg_pos={0 0}, upload=121710]: Done uploading piece. err: <nil> Aug 13 14:25:37.195711503 (116h59m31.626791634s) [noise=true , address=104.36.82.34:4560 , piece_id=4ZZFZKJRUXPMKOE2RDH6ZWFP3TVUOPSRQS7FAB2YANQ3MKA7JLWQ, seg_pos={0 0}, upload=121710]: Done uploading piece. err: <nil> Aug 13 14:25:37.209376008 (116h59m31.640456129s) [noise=true , address=104.36.82.39:4727 , piece_id=UYFASTBHHBMDAY7ZUYHNSNSDDZMPPZ4AIR6X6HC6ERBXLXF4AOUQ, seg_pos={0 0}, upload=121710]: Done uploading piece. err: <nil> Aug 13 14:25:37.252511451 (116h59m31.683591582s) [noise=true , address=77.90.31.19:4846 , piece_id=3UYXBMMHS5MPUNXI6PY65K3AGHIIIVSGXJDEUVXLTVXCZYWTL7JA, seg_pos={0 0}, upload=121710]: Done uploading piece. err: <nil> Aug 13 14:25:37.256659508 (116h59m31.687739639s) [noise=true , address=104.36.82.38:4719 , piece_id=PYZMWL3OZIULLOGDVKRF2RSLGLDGGWTCFWHQ66CPKPPEP6JGCVXQ, seg_pos={0 0}, upload=121710]: Done uploading piece. err: <nil> Aug 13 14:25:37.294487646 (116h59m31.725567777s) [noise=true , address=77.90.31.15:4702 , piece_id=SOXOEHUIJGYJKQ64YWBZMGAZMQJHR5RQMP3VFS5SGDSFM6J52ZMQ, seg_pos={0 0}, upload=121710]: Done uploading piece. err: <nil> Aug 13 14:25:37.310182075 (116h59m31.741262206s) [noise=true , address=104.36.82.62:4412 , piece_id=CTYSBOF4JVZB5JTYNQDMP6Q5XK5KVETQH5SHHSR65K6HF3TZM2WQ, seg_pos={0 0}, upload=121710]: Done uploading piece. err: <nil> Aug 13 14:25:37.514693066 (116h59m31.945773197s) [noise=true , address=104.36.82.23:4474 , piece_id=MDCBWJSKMP57ZE2V5WIKCG7QWIMRAOOIIFAAJQ5YXVWBEG2EG45Q, seg_pos={0 0}, upload=121710]: Done uploading piece. err: <nil> Aug 13 14:25:37.514716501 (116h59m31.945796632s) [seg_pos={0 0}, upload=121710]: Segment reached optimal threshold of 14 pieces. Aug 13 14:25:37.514831381 (116h59m31.945911512s) [noise=true , address=64.49.13.140:4511 , piece_id=VTJ5NDREGII5G4IOP44JNYRJ5IFHHRP7S7OWTRFOGL4FCKBAUDYA, seg_pos={0 0}, upload=121710]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1F79nPRFzy9hjCq5KYvLy6bZERw9cEvczCN2o66Cf6ccKEpkAk): protocol: expected piece hash; context canceled; EOF Aug 13 14:25:37.514874733 (116h59m31.945954864s) [noise=true , address=104.36.82.56:4235 , piece_id=HS4RRPCLDB5ZMD2EJZZCTSCWVY3TWZC6QDVS4TRXRHO3EGZGR7UQ, seg_pos={0 0}, upload=121710]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1a1sW1G1WPL5WQ2XxprWKQMsJkcb5ETSkk3ULp1TjRaZWptaT): protocol: expected piece hash; context canceled; EOF Aug 13 14:25:37.515019360 (116h59m31.946099492s) [noise=true , address=104.36.82.44:4874 , piece_id=C2ATKCPCO7P4KSHSMRRNGYCMH4IRDKUHL5ERDJKIJ4OECUCLQCFQ, seg_pos={0 0}, upload=121710]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YApHMARaC7Wxeqopt72gpnyJS1xYBwhV7wbU9NJdR6W7hukyq): protocol: write tcp 188.166.225.170:50256->104.36.82.44:4874: use of closed network connection; write tcp 188.166.225.170:50256->104.36.82.44:4874: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50256->104.36.82.44:4874: use of closed network connection Aug 13 14:25:37.515053225 (116h59m31.946133356s) [noise=true , address=104.36.82.60:4371 , piece_id=RTCVTBKO26FY7ZUNFYYJ2IOQ5XJCGG774ZP4WFLOVK5D47ULGXOA, seg_pos={0 0}, upload=121710]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KJP5DHmfnSFQF3Q74GzMUvcAz9zh16td4pnsqHAwToiM4D1vd): protocol: write tcp 188.166.225.170:54100->104.36.82.60:4371: use of closed network connection; write tcp 188.166.225.170:54100->104.36.82.60:4371: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54100->104.36.82.60:4371: use of closed network connection Aug 13 14:25:37.515074867 (116h59m31.946154998s) [seg_pos={0 0}, upload=121710]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 13 14:25:37.515134190 (116h59m31.946214321s) [upload=121710]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 13 14:25:37.515138007 (116h59m31.946218138s) [upload=121710]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 13 14:25:38.573880469 (116h59m33.0049606s) [upload=121710]: Done uploading segments... Aug 13 14:25:38.573909565 (116h59m33.004989686s) [upload=121710]: Upload finished. err: <nil> Duration: 6.393938284s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 5 out of 60 is failed (threshold=3) [..............................X..X.............X.....X.X....] duration: 1.937276475s FAIL (2025-08-13 14:28:36 UTC) orbiter warning: operation: ListObjects ======ERROR====== unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/1K ======LOGS====== Aug 13 14:25:57.508778246 (116h59m51.939858377s) [upload=121717]: Starting upload Aug 13 14:25:57.508850885 (116h59m51.939931016s) [upload=121717]: Done starting upload Aug 13 14:25:57.508940917 (116h59m51.940021058s) [upload=121717]: Uploading segments... Aug 13 14:25:57.509065897 (116h59m51.940146038s) [upload=121717]: Got next segment. Inline: true Aug 13 14:25:57.509071818 (116h59m51.940151959s) [upload=121717]: Waiting for error group managing segments... Aug 13 14:25:57.509160628 (116h59m51.940240769s) [upload=121717]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 13 14:25:57.509169355 (116h59m51.940249496s) [upload=121717]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 13 14:25:57.509174675 (116h59m51.940254806s) [upload=121717]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 13 14:25:58.324969756 (116h59m52.756049887s) [upload=121717]: Done uploading segments... Aug 13 14:25:58.324989163 (116h59m52.756069294s) [upload=121717]: Upload finished. err: <nil> Duration:* 1.937305331s core uplink us-select-1 us1-placement-20 metainfo-projects

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)