Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 8 out of 60 is failed (threshold=3) [..........................XXXXXX.X.X........................] duration: 6.801755254s FAIL (2025-08-07 23:40:14 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.801780843s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 3 out of 60 is failed (threshold=3) [...............................X.....X.X....................] duration: 2.503871414s FAIL (2025-08-07 23:40:13 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.503906451s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [....................X......X................................] duration: 1.919068482s FAIL (2025-08-07 23:40:09 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== 30.892541890 (13h31m30.481886877s) [upload=14032]: Upload finished. err: <nil> Aug 7 23:11:48.390915893 (13h31m47.98026088s) [upload=14038]: Starting upload Aug 7 23:11:48.390965198 (13h31m47.980310185s) [upload=14038]: Done starting upload Aug 7 23:11:48.391014242 (13h31m47.980359229s) [upload=14038]: Uploading segments... Aug 7 23:11:48.391054007 (13h31m47.980399004s) [upload=14038]: Got next segment. Inline: true Aug 7 23:11:48.391057494 (13h31m47.980402471s) [upload=14038]: Waiting for error group managing segments... Aug 7 23:11:48.391089254 (13h31m47.980434231s) [upload=14038]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:11:48.391093213 (13h31m47.98043819s) [upload=14038]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:11:48.391095457 (13h31m47.980440444s) [upload=14038]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:11:48.701950724 (13h31m48.291295711s) [upload=14038]: Done uploading segments... Aug 7 23:11:48.701969449 (13h31m48.291314436s) [upload=14038]: Upload finished. err: <nil> Aug 7 23:12:06.430245154 (13h32m6.019590141s) [upload=14048]: Starting upload Aug 7 23:12:06.430293076 (13h32m6.019638063s) [upload=14048]: Done starting upload Aug 7 23:12:06.430361757 (13h32m6.019706745s) [upload=14048]: Uploading segments... Aug 7 23:12:06.430459223 (13h32m6.01980421s) [upload=14048]: Got next segment. Inline: true Aug 7 23:12:06.430464543 (13h32m6.01980953s) [upload=14048]: Waiting for error group managing segments... Aug 7 23:12:06.430527555 (13h32m6.019872542s) [upload=14048]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:12:06.430533796 (13h32m6.019878783s) [upload=14048]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:12:06.430536692 (13h32m6.019881679s) [upload=14048]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:12:06.729952062 (13h32m6.319297049s) [upload=14048]: Done uploading segments... Aug 7 23:12:06.729968844 (13h32m6.319313832s) [upload=14048]: Upload finished. err: <nil> Aug 7 23:12:37.551701338 (13h32m37.141046325s) [upload=14055]: Starting upload Aug 7 23:12:37.551746985 (13h32m37.141091972s) [upload=14055]: Done starting upload Aug 7 23:12:37.551807301 (13h32m37.141152288s) [upload=14055]: Uploading segments... Aug 7 23:12:37.551914826 (13h32m37.141259833s) [upload=14055]: Got next segment. Inline: true Aug 7 23:12:37.551921779 (13h32m37.141266766s) [upload=14055]: Waiting for error group managing segments... Aug 7 23:12:37.551977145 (13h32m37.141322122s) [upload=14055]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:12:37.551984059 (13h32m37.141329086s) [upload=14055]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:12:37.551988918 (13h32m37.141333936s) [upload=14055]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:12:37.767068918 (13h32m37.356413905s) [upload=14055]: Done uploading segments... Aug 7 23:12:37.767104365 (13h32m37.356449342s) [upload=14055]: Upload finished. err: metaclient: Too Many Requests Aug 7 23:13:47.522489627 (13h33m47.111834614s) [upload=14072]: Starting upload Aug 7 23:13:47.522559681 (13h33m47.111904668s) [upload=14072]: Done starting upload Aug 7 23:13:47.522629674 (13h33m47.111974661s) [upload=14072]: Uploading segments... Aug 7 23:13:47.522726771 (13h33m47.112071778s) [upload=14072]: Got next segment. Inline: true Aug 7 23:13:47.522735617 (13h33m47.112080614s) [upload=14072]: Waiting for error group managing segments... Aug 7 23:13:47.522790582 (13h33m47.112135569s) [upload=14072]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:13:47.522796493 (13h33m47.112141481s) [upload=14072]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:13:47.522799320 (13h33m47.112144307s) [upload=14072]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:13:47.796144342 (13h33m47.385489329s) [upload=14072]: Done uploading segments... Aug 7 23:13:47.796162656 (13h33m47.385507643s) [upload=14072]: Upload finished. err: <nil> Aug 7 23:14:22.239921861 (13h34m21.829266848s) [upload=14081]: Starting upload Aug 7 23:14:22.239973791 (13h34m21.829318778s) [upload=14081]: Done starting upload Aug 7 23:14:22.240087399 (13h34m21.829432396s) [upload=14081]: Uploading segments... Aug 7 23:14:22.240187159 (13h34m21.829532156s) [upload=14081]: Got next segment. Inline: true Aug 7 23:14:22.240194182 (13h34m21.829539179s) [upload=14081]: Waiting for error group managing segments... Aug 7 23:14:22.240255971 (13h34m21.829600958s) [upload=14081]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:14:22.240261311 (13h34m21.829606288s) [upload=14081]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:14:22.240263816 (13h34m21.829608803s) [upload=14081]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:14:22.537482123 (13h34m22.12682711s) [upload=14081]: Done uploading segments... Aug 7 23:14:22.537500677 (13h34m22.126845664s) [upload=14081]: Upload finished. err: <nil> Aug 7 23:14:56.968922831 (13h34m56.558267818s) [upload=14090]: Starting upload Aug 7 23:14:56.968974069 (13h34m56.558319056s) [upload=14090]: Done starting upload Aug 7 23:14:56.969028383 (13h34m56.55837337s) [upload=14090]: Uploading segments... Aug 7 23:14:56.969059602 (13h34m56.55840459s) [upload=14090]: Got next segment. Inline: true Aug 7 23:14:56.969063500 (13h34m56.558408487s) [upload=14090]: Waiting for error group managing segments... Aug 7 23:14:56.969102906 (13h34m56.558447893s) [upload=14090]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:14:56.969107034 (13h34m56.558452021s) [upload=14090]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:14:56.969109899 (13h34m56.558454886s) [upload=14090]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:14:57.268863065 (13h34m56.858208052s) [upload=14090]: Done uploading segments... Aug 7 23:14:57.268882201 (13h34m56.858227188s) [upload=14090]: Upload finished. err: <nil> Aug 7 23:16:09.138990771 (13h36m8.728335758s) [upload=14107]: Starting upload Aug 7 23:16:09.139048231 (13h36m8.728393208s) [upload=14107]: Done starting upload Aug 7 23:16:09.139094229 (13h36m8.728439226s) [upload=14107]: Uploading segments... Aug 7 23:16:09.139131069 (13h36m8.728476056s) [upload=14107]: Got next segment. Inline: true Aug 7 23:16:09.139135007 (13h36m8.728479994s) [upload=14107]: Waiting for error group managing segments... Aug 7 23:16:09.139167859 (13h36m8.728512846s) [upload=14107]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:16:09.139171937 (13h36m8.728516924s) [upload=14107]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:16:09.139199730 (13h36m8.728544707s) [upload=14107]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:16:09.409287150 (13h36m8.998632137s) [upload=14107]: Done uploading segments... Aug 7 23:16:09.409303952 (13h36m8.998648939s) [upload=14107]: Upload finished. err: <nil> Aug 7 23:16:30.665596455 (13h36m30.254941442s) [upload=14111]: Starting upload Aug 7 23:16:30.665642382 (13h36m30.254987369s) [upload=14111]: Done starting upload Aug 7 23:16:30.665687749 (13h36m30.255032726s) [upload=14111]: Uploading segments... Aug 7 23:16:30.665725801 (13h36m30.255070788s) [upload=14111]: Got next segment. Inline: true Aug 7 23:16:30.665729409 (13h36m30.255074386s) [upload=14111]: Waiting for error group managing segments... Aug 7 23:16:30.665763554 (13h36m30.255108541s) [upload=14111]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:16:30.665768984 (13h36m30.255113971s) [upload=14111]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:16:30.665771289 (13h36m30.255116276s) [upload=14111]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:16:30.945395745 (13h36m30.534740732s) [upload=14111]: Done uploading segments... Aug 7 23:16:30.945418538 (13h36m30.534763526s) [upload=14111]: Upload finished. err: <nil> Aug 7 23:17:25.677744155 (13h37m25.267089143s) [upload=14125]: Starting upload Aug 7 23:17:25.677793930 (13h37m25.267138917s) [upload=14125]: Done starting upload Aug 7 23:17:25.677862562 (13h37m25.267207539s) [upload=14125]: Uploading segments... Aug 7 23:17:25.677896967 (13h37m25.267241954s) [upload=14125]: Got next segment. Inline: true Aug 7 23:17:25.677900704 (13h37m25.267245681s) [upload=14125]: Waiting for error group managing segments... Aug 7 23:17:25.677933046 (13h37m25.267278033s) [upload=14125]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:17:25.677936884 (13h37m25.267281871s) [upload=14125]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:17:25.677939309 (13h37m25.267284296s) [upload=14125]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:17:26.502642065 (13h37m26.091987052s) [upload=14125]: Done uploading segments... Aug 7 23:17:26.502660942 (13h37m26.092005919s) [upload=14125]: Upload finished. err: <nil> Aug 7 23:17:53.025114325 (13h37m52.614459313s) [upload=14131]: Starting upload Aug 7 23:17:53.025169641 (13h37m52.614514618s) [upload=14131]: Done starting upload Aug 7 23:17:53.025219857 (13h37m52.614564844s) [upload=14131]: Uploading segments... Aug 7 23:17:53.025267939 (13h37m52.614612936s) [upload=14131]: Got next segment. Inline: true Aug 7 23:17:53.025271916 (13h37m52.614616903s) [upload=14131]: Waiting for error group managing segments... Aug 7 23:17:53.025302495 (13h37m52.614647482s) [upload=14131]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:17:53.025306382 (13h37m52.614651369s) [upload=14131]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:17:53.025308497 (13h37m52.614653484s) [upload=14131]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:17:53.302835519 (13h37m52.892180506s) [upload=14131]: Done uploading segments... Aug 7 23:17:53.302854656 (13h37m52.892199633s) [upload=14131]: Upload finished. err: metaclient: Too Many Requests Duration: 1.919095222s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 647.744942ms target: https://us1.storj.io/login FAIL (2025-08-07 23:39:55 UTC) orbiter warning: Get "https://us1.storj.io/login": dial tcp 34.150.199.48:443: connect: connection refused Duration: 647.772676ms core satellite us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 8 out of 60 is failed (threshold=3) [...........................XXXXXX.X.X.......................] duration: 5.618181279s FAIL (2025-08-07 23:39:53 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.618199003s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [.....................X......X...............................] duration: 1.867022769s FAIL (2025-08-07 23:39:52 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== 30.892541890 (13h31m30.481886877s) [upload=14032]: Upload finished. err: <nil> Aug 7 23:11:48.390915893 (13h31m47.98026088s) [upload=14038]: Starting upload Aug 7 23:11:48.390965198 (13h31m47.980310185s) [upload=14038]: Done starting upload Aug 7 23:11:48.391014242 (13h31m47.980359229s) [upload=14038]: Uploading segments... Aug 7 23:11:48.391054007 (13h31m47.980399004s) [upload=14038]: Got next segment. Inline: true Aug 7 23:11:48.391057494 (13h31m47.980402471s) [upload=14038]: Waiting for error group managing segments... Aug 7 23:11:48.391089254 (13h31m47.980434231s) [upload=14038]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:11:48.391093213 (13h31m47.98043819s) [upload=14038]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:11:48.391095457 (13h31m47.980440444s) [upload=14038]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:11:48.701950724 (13h31m48.291295711s) [upload=14038]: Done uploading segments... Aug 7 23:11:48.701969449 (13h31m48.291314436s) [upload=14038]: Upload finished. err: <nil> Aug 7 23:12:06.430245154 (13h32m6.019590141s) [upload=14048]: Starting upload Aug 7 23:12:06.430293076 (13h32m6.019638063s) [upload=14048]: Done starting upload Aug 7 23:12:06.430361757 (13h32m6.019706745s) [upload=14048]: Uploading segments... Aug 7 23:12:06.430459223 (13h32m6.01980421s) [upload=14048]: Got next segment. Inline: true Aug 7 23:12:06.430464543 (13h32m6.01980953s) [upload=14048]: Waiting for error group managing segments... Aug 7 23:12:06.430527555 (13h32m6.019872542s) [upload=14048]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:12:06.430533796 (13h32m6.019878783s) [upload=14048]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:12:06.430536692 (13h32m6.019881679s) [upload=14048]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:12:06.729952062 (13h32m6.319297049s) [upload=14048]: Done uploading segments... Aug 7 23:12:06.729968844 (13h32m6.319313832s) [upload=14048]: Upload finished. err: <nil> Aug 7 23:12:37.551701338 (13h32m37.141046325s) [upload=14055]: Starting upload Aug 7 23:12:37.551746985 (13h32m37.141091972s) [upload=14055]: Done starting upload Aug 7 23:12:37.551807301 (13h32m37.141152288s) [upload=14055]: Uploading segments... Aug 7 23:12:37.551914826 (13h32m37.141259833s) [upload=14055]: Got next segment. Inline: true Aug 7 23:12:37.551921779 (13h32m37.141266766s) [upload=14055]: Waiting for error group managing segments... Aug 7 23:12:37.551977145 (13h32m37.141322122s) [upload=14055]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:12:37.551984059 (13h32m37.141329086s) [upload=14055]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:12:37.551988918 (13h32m37.141333936s) [upload=14055]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:12:37.767068918 (13h32m37.356413905s) [upload=14055]: Done uploading segments... Aug 7 23:12:37.767104365 (13h32m37.356449342s) [upload=14055]: Upload finished. err: metaclient: Too Many Requests Aug 7 23:13:47.522489627 (13h33m47.111834614s) [upload=14072]: Starting upload Aug 7 23:13:47.522559681 (13h33m47.111904668s) [upload=14072]: Done starting upload Aug 7 23:13:47.522629674 (13h33m47.111974661s) [upload=14072]: Uploading segments... Aug 7 23:13:47.522726771 (13h33m47.112071778s) [upload=14072]: Got next segment. Inline: true Aug 7 23:13:47.522735617 (13h33m47.112080614s) [upload=14072]: Waiting for error group managing segments... Aug 7 23:13:47.522790582 (13h33m47.112135569s) [upload=14072]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:13:47.522796493 (13h33m47.112141481s) [upload=14072]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:13:47.522799320 (13h33m47.112144307s) [upload=14072]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:13:47.796144342 (13h33m47.385489329s) [upload=14072]: Done uploading segments... Aug 7 23:13:47.796162656 (13h33m47.385507643s) [upload=14072]: Upload finished. err: <nil> Aug 7 23:14:22.239921861 (13h34m21.829266848s) [upload=14081]: Starting upload Aug 7 23:14:22.239973791 (13h34m21.829318778s) [upload=14081]: Done starting upload Aug 7 23:14:22.240087399 (13h34m21.829432396s) [upload=14081]: Uploading segments... Aug 7 23:14:22.240187159 (13h34m21.829532156s) [upload=14081]: Got next segment. Inline: true Aug 7 23:14:22.240194182 (13h34m21.829539179s) [upload=14081]: Waiting for error group managing segments... Aug 7 23:14:22.240255971 (13h34m21.829600958s) [upload=14081]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:14:22.240261311 (13h34m21.829606288s) [upload=14081]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:14:22.240263816 (13h34m21.829608803s) [upload=14081]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:14:22.537482123 (13h34m22.12682711s) [upload=14081]: Done uploading segments... Aug 7 23:14:22.537500677 (13h34m22.126845664s) [upload=14081]: Upload finished. err: <nil> Aug 7 23:14:56.968922831 (13h34m56.558267818s) [upload=14090]: Starting upload Aug 7 23:14:56.968974069 (13h34m56.558319056s) [upload=14090]: Done starting upload Aug 7 23:14:56.969028383 (13h34m56.55837337s) [upload=14090]: Uploading segments... Aug 7 23:14:56.969059602 (13h34m56.55840459s) [upload=14090]: Got next segment. Inline: true Aug 7 23:14:56.969063500 (13h34m56.558408487s) [upload=14090]: Waiting for error group managing segments... Aug 7 23:14:56.969102906 (13h34m56.558447893s) [upload=14090]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:14:56.969107034 (13h34m56.558452021s) [upload=14090]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:14:56.969109899 (13h34m56.558454886s) [upload=14090]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:14:57.268863065 (13h34m56.858208052s) [upload=14090]: Done uploading segments... Aug 7 23:14:57.268882201 (13h34m56.858227188s) [upload=14090]: Upload finished. err: <nil> Aug 7 23:16:09.138990771 (13h36m8.728335758s) [upload=14107]: Starting upload Aug 7 23:16:09.139048231 (13h36m8.728393208s) [upload=14107]: Done starting upload Aug 7 23:16:09.139094229 (13h36m8.728439226s) [upload=14107]: Uploading segments... Aug 7 23:16:09.139131069 (13h36m8.728476056s) [upload=14107]: Got next segment. Inline: true Aug 7 23:16:09.139135007 (13h36m8.728479994s) [upload=14107]: Waiting for error group managing segments... Aug 7 23:16:09.139167859 (13h36m8.728512846s) [upload=14107]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:16:09.139171937 (13h36m8.728516924s) [upload=14107]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:16:09.139199730 (13h36m8.728544707s) [upload=14107]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:16:09.409287150 (13h36m8.998632137s) [upload=14107]: Done uploading segments... Aug 7 23:16:09.409303952 (13h36m8.998648939s) [upload=14107]: Upload finished. err: <nil> Aug 7 23:16:30.665596455 (13h36m30.254941442s) [upload=14111]: Starting upload Aug 7 23:16:30.665642382 (13h36m30.254987369s) [upload=14111]: Done starting upload Aug 7 23:16:30.665687749 (13h36m30.255032726s) [upload=14111]: Uploading segments... Aug 7 23:16:30.665725801 (13h36m30.255070788s) [upload=14111]: Got next segment. Inline: true Aug 7 23:16:30.665729409 (13h36m30.255074386s) [upload=14111]: Waiting for error group managing segments... Aug 7 23:16:30.665763554 (13h36m30.255108541s) [upload=14111]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:16:30.665768984 (13h36m30.255113971s) [upload=14111]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:16:30.665771289 (13h36m30.255116276s) [upload=14111]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:16:30.945395745 (13h36m30.534740732s) [upload=14111]: Done uploading segments... Aug 7 23:16:30.945418538 (13h36m30.534763526s) [upload=14111]: Upload finished. err: <nil> Aug 7 23:17:25.677744155 (13h37m25.267089143s) [upload=14125]: Starting upload Aug 7 23:17:25.677793930 (13h37m25.267138917s) [upload=14125]: Done starting upload Aug 7 23:17:25.677862562 (13h37m25.267207539s) [upload=14125]: Uploading segments... Aug 7 23:17:25.677896967 (13h37m25.267241954s) [upload=14125]: Got next segment. Inline: true Aug 7 23:17:25.677900704 (13h37m25.267245681s) [upload=14125]: Waiting for error group managing segments... Aug 7 23:17:25.677933046 (13h37m25.267278033s) [upload=14125]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:17:25.677936884 (13h37m25.267281871s) [upload=14125]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:17:25.677939309 (13h37m25.267284296s) [upload=14125]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:17:26.502642065 (13h37m26.091987052s) [upload=14125]: Done uploading segments... Aug 7 23:17:26.502660942 (13h37m26.092005919s) [upload=14125]: Upload finished. err: <nil> Aug 7 23:17:53.025114325 (13h37m52.614459313s) [upload=14131]: Starting upload Aug 7 23:17:53.025169641 (13h37m52.614514618s) [upload=14131]: Done starting upload Aug 7 23:17:53.025219857 (13h37m52.614564844s) [upload=14131]: Uploading segments... Aug 7 23:17:53.025267939 (13h37m52.614612936s) [upload=14131]: Got next segment. Inline: true Aug 7 23:17:53.025271916 (13h37m52.614616903s) [upload=14131]: Waiting for error group managing segments... Aug 7 23:17:53.025302495 (13h37m52.614647482s) [upload=14131]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:17:53.025306382 (13h37m52.614651369s) [upload=14131]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:17:53.025308497 (13h37m52.614653484s) [upload=14131]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:17:53.302835519 (13h37m52.892180506s) [upload=14131]: Done uploading segments... Aug 7 23:17:53.302854656 (13h37m52.892199633s) [upload=14131]: Upload finished. err: metaclient: Too Many Requests Duration: 1.867043989s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 3 out of 60 is failed (threshold=3) [....X....................X.....................X............] duration: 6.347913184s FAIL (2025-08-07 23:39:51 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/4M") ======LOGS====== 916s) [upload=14346]: Starting upload Aug 7 23:30:30.723725030 (13h50m30.313070017s) [upload=14346]: Done starting upload Aug 7 23:30:30.723796056 (13h50m30.313141043s) [upload=14346]: Uploading segments... Aug 7 23:30:30.723935803 (13h50m30.31328079s) [upload=14346]: Got next segment. Inline: false Aug 7 23:30:30.723960240 (13h50m30.313305227s) [upload=14346]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:30:30.723964218 (13h50m30.313309205s) [upload=14346]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:30:31.181251948 (13h50m30.770596926s) [seg_pos={0 0}, upload=14346]: Begin upload segment... Aug 7 23:30:31.181299649 (13h50m30.770644636s) [seg_pos={0 0}, upload=14346]: Done begin upload segment. Aug 7 23:30:31.181318165 (13h50m30.770663152s) [noise=true , address=104.36.82.52:4086 , piece_id=JLEUROLLGI7X6MUOVFMEWIFCQNTYJXSZW2JMHGEAI66XREHK5LLQ, seg_pos={0 0}, upload=14346]: Uploading piece... Aug 7 23:30:31.181329696 (13h50m30.770674693s) [noise=true , address=104.36.82.54:4172 , piece_id=2QZVEPRDWBX4KY62QMLTPKNZ7QA2K3YDWJETONKWYOGLMPUVSVWA, seg_pos={0 0}, upload=14346]: Uploading piece... Aug 7 23:30:31.181716817 (13h50m30.771061804s) [noise=true , address=77.90.31.17:4797 , piece_id=P6M6EN2C7M6FVKVWV4LE6JEHSZTB24YEACVG7TMSNBQFLD6NLKDQ, seg_pos={0 0}, upload=14346]: Uploading piece... Aug 7 23:30:31.181721437 (13h50m30.771066424s) [noise=true , address=104.36.82.46:4489 , piece_id=KYHATGRQXZ6XXQFUSTYUYBI26BTC7VSN5N6OM766B3QOSWN747IA, seg_pos={0 0}, upload=14346]: Uploading piece... Aug 7 23:30:31.182063421 (13h50m30.771408408s) [noise=true , address=104.36.82.22:4534 , piece_id=5BROJMGAIDCZNPDEQH32QVVYD2JP4U7IN5DCDUP7LBMU2E6Z36CA, seg_pos={0 0}, upload=14346]: Uploading piece... Aug 7 23:30:31.182318108 (13h50m30.771663085s) [noise=true , address=104.36.82.58:4267 , piece_id=QVRI7PRWV4JSPRWQIJD6M7UJAWDBWN2VIVGOYUJ7NGO77SHZ4UJQ, seg_pos={0 0}, upload=14346]: Uploading piece... Aug 7 23:30:31.182483654 (13h50m30.771828641s) [noise=true , address=64.49.13.138:4489 , piece_id=GC6CUUPTPHVDNMD2ZDX5EUJSEUNL6I6OM3MYQNM25F2G7X64YQLQ, seg_pos={0 0}, upload=14346]: Uploading piece... Aug 7 23:30:31.182639132 (13h50m30.771984119s) [noise=true , address=45.91.116.11:4149 , piece_id=JGSWIYMTFANVCD326G2TRN24X26XYQQSJT6HFWBQAHJLQ2SLBA4A, seg_pos={0 0}, upload=14346]: Uploading piece... Aug 7 23:30:31.182934928 (13h50m30.772279905s) [noise=true , address=104.36.82.62:4428 , piece_id=P7EOR2HLYCVOBQBDUZ3OYNVO4RV7WRXLHVREWKIKE7SHYBV3VRVQ, seg_pos={0 0}, upload=14346]: Uploading piece... Aug 7 23:30:31.183037023 (13h50m30.77238201s) [noise=true , address=64.49.13.155:4917 , piece_id=GLY2EUU2PUD7JB4V7PAKNNFK6ZDWGN2DKS5YHTXUNAJXWGRYRTJA, seg_pos={0 0}, upload=14346]: Uploading piece... Aug 7 23:30:31.183158877 (13h50m30.772503864s) [noise=true , address=104.36.82.47:4534 , piece_id=PJCWOCQ7UKEHHSERR6C5RQC4Y3TBZRHZF6KPMZIMTAFS6E4525EA, seg_pos={0 0}, upload=14346]: Uploading piece... Aug 7 23:30:31.183353329 (13h50m30.772698316s) [noise=true , address=104.36.82.44:4883 , piece_id=7GULYV2REOYLHXZXRSPIQM46DRT4YKT7FXC3YCHAP7W7332RRTRA, seg_pos={0 0}, upload=14346]: Uploading piece... Aug 7 23:30:31.183386993 (13h50m30.77273198s) [noise=true , address=104.36.82.52:4097 , piece_id=REDOEPTMV6TQIM5HWQ2DQBPENXJFHVZV5LGM244DRAMGSV6HSNRQ, seg_pos={0 0}, upload=14346]: Uploading piece... Aug 7 23:30:31.183591444 (13h50m30.772936431s) [noise=true , address=104.36.82.60:4353 , piece_id=3TWE6IJ3OCY5CMMRHZ5DG2WBOHLWF3MEESPOGUMU3YA5NSGW3AKA, seg_pos={0 0}, upload=14346]: Uploading piece... Aug 7 23:30:31.183849538 (13h50m30.773194525s) [noise=true , address=64.49.13.27:4960 , piece_id=RCNXI4GI75JF5TWSSGMTNPWOTU4C3TADGATSBJQKJBRLVELNSGUA, seg_pos={0 0}, upload=14346]: Uploading piece... Aug 7 23:30:31.184184719 (13h50m30.773529706s) [noise=true , address=104.36.82.61:4392 , piece_id=OSMRSZL5LDLYLPM3KN5YVIMBU44HS72BHZOAB7W5SS5Z4DY4F4JA, seg_pos={0 0}, upload=14346]: Uploading piece... Aug 7 23:30:31.184422474 (13h50m30.773767461s) [noise=true , address=104.36.82.8:4995 , piece_id=NS2X5RBAQQWBIMO2U4XYCYA37GVF6OBAO3GZSTEPJZOXRMPMS62A, seg_pos={0 0}, upload=14346]: Uploading piece... Aug 7 23:30:31.184480636 (13h50m30.773825623s) [noise=true , address=104.36.82.45:4910 , piece_id=QEYOF4IB42OVZTWGW3YD7ZUYX2KZ6KKOUXFUHQQYOW4QY3JUGTDQ, seg_pos={0 0}, upload=14346]: Uploading piece... Aug 7 23:30:32.295332666 (13h50m31.884677653s) [upload=14346]: Next returned nil segment Aug 7 23:30:32.295347043 (13h50m31.88469204s) [upload=14346]: Waiting for error group managing segments... Aug 7 23:30:32.576560505 (13h50m32.165905492s) [noise=true , address=64.49.13.155:4917 , piece_id=GLY2EUU2PUD7JB4V7PAKNNFK6ZDWGN2DKS5YHTXUNAJXWGRYRTJA, seg_pos={0 0}, upload=14346]: Done uploading piece. err: <nil> Aug 7 23:30:32.584528981 (13h50m32.173873958s) [noise=true , address=64.49.13.138:4489 , piece_id=GC6CUUPTPHVDNMD2ZDX5EUJSEUNL6I6OM3MYQNM25F2G7X64YQLQ, seg_pos={0 0}, upload=14346]: Done uploading piece. err: <nil> Aug 7 23:30:32.647385961 (13h50m32.236730948s) [noise=true , address=64.49.13.27:4960 , piece_id=RCNXI4GI75JF5TWSSGMTNPWOTU4C3TADGATSBJQKJBRLVELNSGUA, seg_pos={0 0}, upload=14346]: Done uploading piece. err: <nil> Aug 7 23:30:33.094757008 (13h50m32.684101995s) [noise=true , address=104.36.82.52:4097 , piece_id=REDOEPTMV6TQIM5HWQ2DQBPENXJFHVZV5LGM244DRAMGSV6HSNRQ, seg_pos={0 0}, upload=14346]: Done uploading piece. err: <nil> Aug 7 23:30:33.167659426 (13h50m32.757004403s) [noise=true , address=104.36.82.45:4910 , piece_id=QEYOF4IB42OVZTWGW3YD7ZUYX2KZ6KKOUXFUHQQYOW4QY3JUGTDQ, seg_pos={0 0}, upload=14346]: Done uploading piece. err: <nil> Aug 7 23:30:33.198379320 (13h50m32.787724307s) [noise=true , address=104.36.82.54:4172 , piece_id=2QZVEPRDWBX4KY62QMLTPKNZ7QA2K3YDWJETONKWYOGLMPUVSVWA, seg_pos={0 0}, upload=14346]: Done uploading piece. err: <nil> Aug 7 23:30:33.235504739 (13h50m32.824849726s) [noise=true , address=104.36.82.58:4267 , piece_id=QVRI7PRWV4JSPRWQIJD6M7UJAWDBWN2VIVGOYUJ7NGO77SHZ4UJQ, seg_pos={0 0}, upload=14346]: Done uploading piece. err: <nil> Aug 7 23:30:33.264110248 (13h50m32.853455225s) [noise=true , address=104.36.82.52:4086 , piece_id=JLEUROLLGI7X6MUOVFMEWIFCQNTYJXSZW2JMHGEAI66XREHK5LLQ, seg_pos={0 0}, upload=14346]: Done uploading piece. err: <nil> Aug 7 23:30:33.327068230 (13h50m32.916413217s) [noise=true , address=104.36.82.8:4995 , piece_id=NS2X5RBAQQWBIMO2U4XYCYA37GVF6OBAO3GZSTEPJZOXRMPMS62A, seg_pos={0 0}, upload=14346]: Done uploading piece. err: <nil> Aug 7 23:30:33.366778193 (13h50m32.956123181s) [noise=true , address=104.36.82.60:4353 , piece_id=3TWE6IJ3OCY5CMMRHZ5DG2WBOHLWF3MEESPOGUMU3YA5NSGW3AKA, seg_pos={0 0}, upload=14346]: Done uploading piece. err: <nil> Aug 7 23:30:33.542767028 (13h50m33.132112016s) [noise=true , address=104.36.82.46:4489 , piece_id=KYHATGRQXZ6XXQFUSTYUYBI26BTC7VSN5N6OM766B3QOSWN747IA, seg_pos={0 0}, upload=14346]: Done uploading piece. err: <nil> Aug 7 23:30:33.575265094 (13h50m33.164610081s) [noise=true , address=104.36.82.22:4534 , piece_id=5BROJMGAIDCZNPDEQH32QVVYD2JP4U7IN5DCDUP7LBMU2E6Z36CA, seg_pos={0 0}, upload=14346]: Done uploading piece. err: <nil> Aug 7 23:30:33.672325632 (13h50m33.261670619s) [noise=true , address=77.90.31.17:4797 , piece_id=P6M6EN2C7M6FVKVWV4LE6JEHSZTB24YEACVG7TMSNBQFLD6NLKDQ, seg_pos={0 0}, upload=14346]: Done uploading piece. err: <nil> Aug 7 23:30:33.750650465 (13h50m33.339995452s) [noise=true , address=45.91.116.11:4149 , piece_id=JGSWIYMTFANVCD326G2TRN24X26XYQQSJT6HFWBQAHJLQ2SLBA4A, seg_pos={0 0}, upload=14346]: Done uploading piece. err: <nil> Aug 7 23:30:33.750672889 (13h50m33.340017876s) [seg_pos={0 0}, upload=14346]: Segment reached optimal threshold of 14 pieces. Aug 7 23:30:33.750785674 (13h50m33.340130661s) [noise=true , address=104.36.82.47:4534 , piece_id=PJCWOCQ7UKEHHSERR6C5RQC4Y3TBZRHZF6KPMZIMTAFS6E4525EA, seg_pos={0 0}, upload=14346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VDrNGcCnaKEmG5NN4nex1JoZAgdCEfX5CdsKbxytfLfKzPbr): protocol: expected piece hash; context canceled; EOF Aug 7 23:30:33.750885715 (13h50m33.340230702s) [noise=true , address=104.36.82.61:4392 , piece_id=OSMRSZL5LDLYLPM3KN5YVIMBU44HS72BHZOAB7W5SS5Z4DY4F4JA, seg_pos={0 0}, upload=14346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14FTsLbtv1vp43aRdDZZBY2Gf6epidM1A8HSyQKVokXGFFZGZk): protocol: expected piece hash; context canceled; EOF Aug 7 23:30:33.750979806 (13h50m33.340324793s) [noise=true , address=104.36.82.62:4428 , piece_id=P7EOR2HLYCVOBQBDUZ3OYNVO4RV7WRXLHVREWKIKE7SHYBV3VRVQ, seg_pos={0 0}, upload=14346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Ui5Ye48cKNGFPakMKsjdBi13Xo7bzrmXPNEmRBLKwpMtVAA3R): protocol: rpc: dial tcp 104.36.82.62:4428: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 23:30:33.751073955 (13h50m33.340418942s) [noise=true , address=104.36.82.44:4883 , piece_id=7GULYV2REOYLHXZXRSPIQM46DRT4YKT7FXC3YCHAP7W7332RRTRA, seg_pos={0 0}, upload=14346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cx1avL4umDm2pzJS1rNZqeDZjZMWnMtNsyNTdMdkxhpa1ByUN): protocol: context canceled; context canceled Aug 7 23:30:33.751119673 (13h50m33.34046466s) [seg_pos={0 0}, upload=14346]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 23:30:33.751224474 (13h50m33.340569471s) [upload=14346]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 23:30:33.751230776 (13h50m33.340575763s) [upload=14346]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:30:34.056540019 (13h50m33.645885006s) [upload=14346]: Done uploading segments... Aug 7 23:30:34.056559826 (13h50m33.645904823s) [upload=14346]: Upload finished. err: <nil> Duration: 6.347939404s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [..............................XX.X..........................] duration: 6.614381325s FAIL (2025-08-07 23:39:44 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== PC5WABH5C4FB5BLC55F4FZQAU4IAHMEA, seg_pos={0 0}, upload=14126]: Uploading piece... Aug 7 23:17:27.821935980 (13h37m27.411280967s) [noise=true , address=104.36.82.8:4983 , piece_id=AVNQRSGZB5XHG3VFYFHRLBLKQ7B4YWXGIPWDK32DYJSMFXUFTSNA, seg_pos={0 0}, upload=14126]: Uploading piece... Aug 7 23:17:27.822230193 (13h37m27.41157518s) [noise=true , address=104.36.82.16:4310 , piece_id=OAD3ECB3B5RXQB362MCLIWSQC4PKS3QUHWVMIMK4XPGH63JKO3LQ, seg_pos={0 0}, upload=14126]: Uploading piece... Aug 7 23:17:27.822283705 (13h37m27.411628692s) [noise=true , address=104.36.82.59:4323 , piece_id=WNEN4TRJFQ64H74WZOQGGP3V5F7Y3A7OIMORF7M7WEYOR252JSRA, seg_pos={0 0}, upload=14126]: Uploading piece... Aug 7 23:17:27.822684663 (13h37m27.41202965s) [noise=true , address=104.36.82.39:4747 , piece_id=7OQQP63SGNWWQEYR75RRJRHZSL52NRYH3BCS65XI4SWA24BKERUQ, seg_pos={0 0}, upload=14126]: Uploading piece... Aug 7 23:17:27.823034542 (13h37m27.412379529s) [noise=true , address=104.36.82.59:4322 , piece_id=D4THBNGRCWHLJC6HETJRODVC7ZRMOWMIAHGM6GYVU7LFIJNBTOZQ, seg_pos={0 0}, upload=14126]: Uploading piece... Aug 7 23:17:27.823366667 (13h37m27.412711654s) [noise=true , address=104.36.82.60:4343 , piece_id=UHOQFZPYXHO56TUWMTSQE5XVWFGO66ATAA3EU7TQ5KX2MIN2ZJQQ, seg_pos={0 0}, upload=14126]: Uploading piece... Aug 7 23:17:27.823697740 (13h37m27.413042727s) [noise=true , address=79.127.201.215:4355 , piece_id=KUJDHE66HM6IAFRND7KRYDII2FB7AA3S5K5S244JPQZUD4BYXFWA, seg_pos={0 0}, upload=14126]: Uploading piece... Aug 7 23:17:27.824090602 (13h37m27.413435589s) [noise=true , address=104.36.82.10:4093 , piece_id=VQUPLYS4X67PIYNTDH76KJ3DTBIWIH6UM3TQJ6VEREZIUJIH7A2Q, seg_pos={0 0}, upload=14126]: Uploading piece... Aug 7 23:17:27.824398111 (13h37m27.413743098s) [noise=true , address=64.49.13.170:4718 , piece_id=VXSII2V5QDMEJ5G5PJWNHXSJ6HF5NLLTGEDN5CHKKRWYHIAKHMVA, seg_pos={0 0}, upload=14126]: Uploading piece... Aug 7 23:17:27.824731088 (13h37m27.414076065s) [noise=true , address=104.36.82.43:4847 , piece_id=5Y56Z5NDMWSATFWNBYDWKBZB7SUSJ3ABAFMOY6PJ3B24SPPGJH3A, seg_pos={0 0}, upload=14126]: Uploading piece... Aug 7 23:17:27.828362365 (13h37m27.417707362s) [noise=true , address=104.36.82.61:4404 , piece_id=KGEALBUHVOZEFP7V7LPYXQG5APDPXBEWG6UG34QGOCYOHPFHEZNQ, seg_pos={0 0}, upload=14126]: Uploading piece... Aug 7 23:17:27.831994514 (13h37m27.421339501s) [noise=true , address=104.36.82.18:4370 , piece_id=HHYIBQ2JWTCPEOVRZQLKEWT7BFUZDXXZP6FYED3MPEWTVODBQUQA, seg_pos={0 0}, upload=14126]: Uploading piece... Aug 7 23:17:27.835602707 (13h37m27.424947694s) [noise=true , address=45.91.116.6:4988 , piece_id=QF7HXJOV5JYZ73FJMUUGT2JJUR3W2PY3C7CE6EQQCYGM72VDSTXQ, seg_pos={0 0}, upload=14126]: Uploading piece... Aug 7 23:17:27.837475180 (13h37m27.426820177s) [noise=true , address=104.36.82.34:4547 , piece_id=2XYJPP25OROUVGLPDMGFVMDOD4AYOFE6BRQ4OFMVEO2D73XVPNAQ, seg_pos={0 0}, upload=14126]: Uploading piece... Aug 7 23:17:27.841864818 (13h37m27.431209825s) [noise=true , address=104.36.82.62:4439 , piece_id=F4FXK4T3IKUXUGL37K3DEDNWWAUZZFEJVT7WBYDVVEZSXO6QG4OQ, seg_pos={0 0}, upload=14126]: Uploading piece... Aug 7 23:17:29.010273953 (13h37m28.59961894s) [upload=14126]: Next returned nil segment Aug 7 23:17:29.010293008 (13h37m28.599637996s) [upload=14126]: Waiting for error group managing segments... Aug 7 23:17:29.309514867 (13h37m28.898859844s) [noise=true , address=64.49.13.170:4718 , piece_id=VXSII2V5QDMEJ5G5PJWNHXSJ6HF5NLLTGEDN5CHKKRWYHIAKHMVA, seg_pos={0 0}, upload=14126]: Done uploading piece. err: <nil> Aug 7 23:17:29.779221569 (13h37m29.368566556s) [noise=true , address=79.127.201.215:4355 , piece_id=KUJDHE66HM6IAFRND7KRYDII2FB7AA3S5K5S244JPQZUD4BYXFWA, seg_pos={0 0}, upload=14126]: Done uploading piece. err: <nil> Aug 7 23:17:29.827123595 (13h37m29.416468582s) [noise=true , address=104.36.82.34:4547 , piece_id=2XYJPP25OROUVGLPDMGFVMDOD4AYOFE6BRQ4OFMVEO2D73XVPNAQ, seg_pos={0 0}, upload=14126]: Done uploading piece. err: <nil> Aug 7 23:17:29.905373281 (13h37m29.494718258s) [noise=true , address=104.36.82.39:4747 , piece_id=7OQQP63SGNWWQEYR75RRJRHZSL52NRYH3BCS65XI4SWA24BKERUQ, seg_pos={0 0}, upload=14126]: Done uploading piece. err: <nil> Aug 7 23:17:29.927717560 (13h37m29.517062537s) [noise=true , address=104.36.82.43:4847 , piece_id=5Y56Z5NDMWSATFWNBYDWKBZB7SUSJ3ABAFMOY6PJ3B24SPPGJH3A, seg_pos={0 0}, upload=14126]: Done uploading piece. err: <nil> Aug 7 23:17:30.139039260 (13h37m29.728384247s) [noise=true , address=104.36.82.46:4511 , piece_id=4A43V3IOJTGJNWLKNYB7PC5WABH5C4FB5BLC55F4FZQAU4IAHMEA, seg_pos={0 0}, upload=14126]: Done uploading piece. err: <nil> Aug 7 23:17:30.142609049 (13h37m29.731954036s) [noise=true , address=104.36.82.10:4093 , piece_id=VQUPLYS4X67PIYNTDH76KJ3DTBIWIH6UM3TQJ6VEREZIUJIH7A2Q, seg_pos={0 0}, upload=14126]: Done uploading piece. err: <nil> Aug 7 23:17:30.265344466 (13h37m29.854689463s) [noise=true , address=104.36.82.8:4983 , piece_id=AVNQRSGZB5XHG3VFYFHRLBLKQ7B4YWXGIPWDK32DYJSMFXUFTSNA, seg_pos={0 0}, upload=14126]: Done uploading piece. err: <nil> Aug 7 23:17:30.341283132 (13h37m29.930628119s) [noise=true , address=45.91.116.6:4988 , piece_id=QF7HXJOV5JYZ73FJMUUGT2JJUR3W2PY3C7CE6EQQCYGM72VDSTXQ, seg_pos={0 0}, upload=14126]: Done uploading piece. err: <nil> Aug 7 23:17:30.343588482 (13h37m29.932933459s) [noise=true , address=104.36.82.61:4404 , piece_id=KGEALBUHVOZEFP7V7LPYXQG5APDPXBEWG6UG34QGOCYOHPFHEZNQ, seg_pos={0 0}, upload=14126]: Done uploading piece. err: <nil> Aug 7 23:17:30.538382992 (13h37m30.127727969s) [noise=true , address=104.36.82.37:4666 , piece_id=WXBTFS2SS7DMVYBL5BHW7TBXH7FJODGB5JJDT6WP4LOO3LU4GNDA, seg_pos={0 0}, upload=14126]: Done uploading piece. err: <nil> Aug 7 23:17:30.631974070 (13h37m30.221319047s) [noise=true , address=104.36.82.9:4028 , piece_id=HSPPTMR3OOL5DWG25IF3LXYTSOUASUM3BFX7RKHZHCUTBP4B3GQA, seg_pos={0 0}, upload=14126]: Done uploading piece. err: <nil> Aug 7 23:17:31.962954475 (13h37m31.552299462s) [noise=true , address=104.36.82.59:4322 , piece_id=D4THBNGRCWHLJC6HETJRODVC7ZRMOWMIAHGM6GYVU7LFIJNBTOZQ, seg_pos={0 0}, upload=14126]: Done uploading piece. err: <nil> Aug 7 23:17:32.006950608 (13h37m31.596295635s) [noise=true , address=104.36.82.59:4323 , piece_id=WNEN4TRJFQ64H74WZOQGGP3V5F7Y3A7OIMORF7M7WEYOR252JSRA, seg_pos={0 0}, upload=14126]: Done uploading piece. err: <nil> Aug 7 23:17:32.006984062 (13h37m31.596329049s) [seg_pos={0 0}, upload=14126]: Segment reached optimal threshold of 14 pieces. Aug 7 23:17:32.007159788 (13h37m31.596504775s) [noise=true , address=104.36.82.18:4370 , piece_id=HHYIBQ2JWTCPEOVRZQLKEWT7BFUZDXXZP6FYED3MPEWTVODBQUQA, seg_pos={0 0}, upload=14126]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SXZN7WJvcHys9BRd4Lq8NAZ3MSgHf9ZVv7SgaiJNRosdmFVpm): protocol: expected piece hash; context canceled; EOF Aug 7 23:17:32.007279266 (13h37m31.596624253s) [noise=true , address=104.36.82.60:4343 , piece_id=UHOQFZPYXHO56TUWMTSQE5XVWFGO66ATAA3EU7TQ5KX2MIN2ZJQQ, seg_pos={0 0}, upload=14126]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16EsEMRW53sjKVag2WpSDGLhELyE2bSXyW4aAurCvDFKRNkXuf): protocol: write tcp 188.166.225.170:53332->104.36.82.60:4343: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 23:17:32.007291060 (13h37m31.596636047s) [noise=true , address=104.36.82.16:4310 , piece_id=OAD3ECB3B5RXQB362MCLIWSQC4PKS3QUHWVMIMK4XPGH63JKO3LQ, seg_pos={0 0}, upload=14126]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1g14evoaeUnhJThf5QKbQT1Gd8WW9MHvaYQhQ1GRgwPfDLoC5R): protocol: write tcp 188.166.225.170:46238->104.36.82.16:4310: use of closed network connection; write tcp 188.166.225.170:46238->104.36.82.16:4310: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46238->104.36.82.16:4310: use of closed network connection Aug 7 23:17:32.007409626 (13h37m31.596754613s) [noise=true , address=104.36.82.62:4439 , piece_id=F4FXK4T3IKUXUGL37K3DEDNWWAUZZFEJVT7WBYDVVEZSXO6QG4OQ, seg_pos={0 0}, upload=14126]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HRW3L15ka62qA7mWmEJaJ2ydTA5f8sVY1iL3Fn1R7ubisPF3W): protocol: read tcp 188.166.225.170:33134->104.36.82.62:4439: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 23:17:32.007441958 (13h37m31.596786945s) [seg_pos={0 0}, upload=14126]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 23:17:32.007533564 (13h37m31.596878551s) [upload=14126]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 23:17:32.007541158 (13h37m31.596886145s) [upload=14126]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:17:39.583862369 (13h37m39.173207346s) [upload=14126]: Done uploading segments... Aug 7 23:17:39.583880514 (13h37m39.173225501s) [upload=14126]: Upload finished. err: <nil> Aug 7 23:17:57.117137796 (13h37m56.706482783s) [upload=14133]: Starting upload Aug 7 23:17:57.117195166 (13h37m56.706540153s) [upload=14133]: Done starting upload Aug 7 23:17:57.117244931 (13h37m56.706589918s) [upload=14133]: Uploading segments... Aug 7 23:17:57.117378968 (13h37m56.706723955s) [upload=14133]: Got next segment. Inline: false Aug 7 23:17:57.117404277 (13h37m56.706749264s) [upload=14133]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:17:57.117408454 (13h37m56.706753441s) [upload=14133]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:18:56.470635719 (13h38m56.059980716s) [upload=14133]: Next segment err: context deadline exceeded Aug 7 23:18:56.470717675 (13h38m56.060062703s) [upload=14133]: Done uploading segments... Aug 7 23:18:56.470739287 (13h38m56.060084304s) [upload=14133]: Upload finished. err: context deadline exceeded Duration: 6.614400341s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 8 out of 60 is failed (threshold=3) [............................XXXXXX.X.X......................] duration: 5.622942211s FAIL (2025-08-07 23:39:38 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.622966839s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [.....................X..X.XXXX.X............................] duration: 6.240699925s FAIL (2025-08-07 23:39:33 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== 8]: Upload finished. err: context deadline exceeded Aug 7 23:18:32.336054087 (13h38m31.925399094s) [upload=14140]: Starting upload Aug 7 23:18:32.336180319 (13h38m31.925525306s) [upload=14140]: Done starting upload Aug 7 23:18:32.336261133 (13h38m31.92560616s) [upload=14140]: Uploading segments... Aug 7 23:18:32.336590001 (13h38m31.925935029s) [upload=14140]: Got next segment. Inline: false Aug 7 23:18:32.336650628 (13h38m31.925995655s) [upload=14140]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:18:32.336658754 (13h38m31.926003761s) [upload=14140]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:18:32.571732430 (13h38m32.161077417s) [seg_pos={0 0}, upload=14140]: Begin upload segment... Aug 7 23:18:32.571774531 (13h38m32.161119528s) [seg_pos={0 0}, upload=14140]: Done begin upload segment. Aug 7 23:18:32.571805400 (13h38m32.161150417s) [noise=true , address=77.90.31.12:4589 , piece_id=ZMKTDC4M5JHWVARXQDQAKRFSXM3OQQZ2IGQASJCK2OKYX3IQXHVA, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.571844173 (13h38m32.161189161s) [noise=true , address=104.36.82.9:4007 , piece_id=VZLD7RU4SK5MAQE22D7OBVT6XOQAN75UNDMT7JNPCRZNRKFX36XQ, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.572218380 (13h38m32.161563367s) [noise=true , address=104.36.82.10:4077 , piece_id=WYGEC4FYUPAR45MZIWLZQ2ASF5SUPECS4YI3UTRRHEVAKUEYCGYA, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.572289677 (13h38m32.161634654s) [noise=true , address=77.90.31.17:4770 , piece_id=ISA43NF4AKPK2SGYDPKT6ZIAD5WJ27A6GT6JZA2GHQUCAQBRRAFA, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.572494809 (13h38m32.161839796s) [noise=true , address=104.36.82.7:4911 , piece_id=FAMSQZJMALQIEAG5BVR5OV5WHEHN432Y4DZD3DNPC5CKP7X7GMQA, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.572588257 (13h38m32.161933244s) [noise=true , address=77.90.31.15:4697 , piece_id=6RYZJ22TCTJFRBMQV7M545VHB475CHDV2DBKCGO3MJP65VEMJ5FQ, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.572772450 (13h38m32.162117427s) [noise=true , address=45.91.116.12:4225 , piece_id=22GAFTM5SITGTVJEELA3BJP7W2QZW7BVYJGJMKJOLE2RF2GZ5CGQ, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.572865959 (13h38m32.162210946s) [noise=true , address=45.91.116.12:4227 , piece_id=T3JVU7VMSYBTYXBVGRD7LSNIG3PWF3QDVDNDQ24JQOSV7ZFIODNA, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.573029050 (13h38m32.162374037s) [noise=true , address=77.90.31.19:4842 , piece_id=3ZRHNMBPQUJH5SZZ75UJWLYIYZ5YI7CC5OMIJJFM7WVKZWJX6R3A, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.573227882 (13h38m32.162572869s) [noise=true , address=104.36.82.4:4804 , piece_id=LTSBCPH74YSWH7IHN6MH4NWTBJ5FQ7534VERPCY5NIDV24KYPXMQ, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.573293797 (13h38m32.162638774s) [noise=true , address=104.36.82.15:4259 , piece_id=GFCTNTXSY4WVSTPAD3AZIYCHE4O5EN24WN25VUNCRNWCSUWCYBMA, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.573504962 (13h38m32.162849949s) [noise=true , address=77.90.31.16:4734 , piece_id=OJPJZ5RF7QY4KNFBZ3MAVTA63HMC4JRG3HQLBN7SVROVKFQZY7BQ, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.573527936 (13h38m32.162872923s) [noise=true , address=45.91.116.7:4009 , piece_id=VKYMGNJEGFLB5M3L3WLD37C5ZILQGLRJROSQ4ZEIBN46CQSY5JZQ, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.574078970 (13h38m32.163423947s) [noise=true , address=45.91.116.11:4119 , piece_id=UXO65SB4OZ57BCKOWGE2VDV7KCSKCGE7I7K4RTGRYJ7VU5GKFPHQ, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.574216453 (13h38m32.16356143s) [noise=true , address=104.36.82.12:4149 , piece_id=2K5BCSGU4UN43HF7VHG3MBNMQEGSIZPPHTDHDFI3XPY5DDG6NKOA, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.576753347 (13h38m32.166098334s) [noise=true , address=104.36.82.8:4971 , piece_id=DNQVVV7QT2A2NFMIRURJ5IZRFPYZVX2NADRTX2P2YORA2CKMWY6Q, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:34.327895253 (13h38m33.91724024s) [upload=14140]: Next returned nil segment Aug 7 23:18:34.327912295 (13h38m33.917257282s) [upload=14140]: Waiting for error group managing segments... Aug 7 23:18:34.716828436 (13h38m34.306173413s) [noise=true , address=104.36.82.7:4911 , piece_id=FAMSQZJMALQIEAG5BVR5OV5WHEHN432Y4DZD3DNPC5CKP7X7GMQA, seg_pos={0 0}, upload=14140]: Done uploading piece. err: <nil> Aug 7 23:18:34.729509878 (13h38m34.318854865s) [noise=true , address=104.36.82.12:4149 , piece_id=2K5BCSGU4UN43HF7VHG3MBNMQEGSIZPPHTDHDFI3XPY5DDG6NKOA, seg_pos={0 0}, upload=14140]: Done uploading piece. err: <nil> Aug 7 23:18:34.738858565 (13h38m34.328203592s) [noise=true , address=104.36.82.4:4804 , piece_id=LTSBCPH74YSWH7IHN6MH4NWTBJ5FQ7534VERPCY5NIDV24KYPXMQ, seg_pos={0 0}, upload=14140]: Done uploading piece. err: <nil> Aug 7 23:18:34.783948702 (13h38m34.373293679s) [noise=true , address=77.90.31.19:4842 , piece_id=3ZRHNMBPQUJH5SZZ75UJWLYIYZ5YI7CC5OMIJJFM7WVKZWJX6R3A, seg_pos={0 0}, upload=14140]: Done uploading piece. err: <nil> Aug 7 23:18:34.824404112 (13h38m34.413749099s) [noise=true , address=104.36.82.15:4259 , piece_id=GFCTNTXSY4WVSTPAD3AZIYCHE4O5EN24WN25VUNCRNWCSUWCYBMA, seg_pos={0 0}, upload=14140]: Done uploading piece. err: <nil> Aug 7 23:18:34.863114634 (13h38m34.452459611s) [noise=true , address=104.36.82.9:4007 , piece_id=VZLD7RU4SK5MAQE22D7OBVT6XOQAN75UNDMT7JNPCRZNRKFX36XQ, seg_pos={0 0}, upload=14140]: Done uploading piece. err: <nil> Aug 7 23:18:34.889984711 (13h38m34.479329698s) [noise=true , address=77.90.31.16:4734 , piece_id=OJPJZ5RF7QY4KNFBZ3MAVTA63HMC4JRG3HQLBN7SVROVKFQZY7BQ, seg_pos={0 0}, upload=14140]: Done uploading piece. err: <nil> Aug 7 23:18:34.982222673 (13h38m34.57156766s) [noise=true , address=104.36.82.10:4077 , piece_id=WYGEC4FYUPAR45MZIWLZQ2ASF5SUPECS4YI3UTRRHEVAKUEYCGYA, seg_pos={0 0}, upload=14140]: Done uploading piece. err: <nil> Aug 7 23:18:34.999676977 (13h38m34.589021964s) [noise=true , address=77.90.31.17:4770 , piece_id=ISA43NF4AKPK2SGYDPKT6ZIAD5WJ27A6GT6JZA2GHQUCAQBRRAFA, seg_pos={0 0}, upload=14140]: Done uploading piece. err: <nil> Aug 7 23:18:35.002083712 (13h38m34.591428699s) [noise=true , address=45.91.116.12:4225 , piece_id=22GAFTM5SITGTVJEELA3BJP7W2QZW7BVYJGJMKJOLE2RF2GZ5CGQ, seg_pos={0 0}, upload=14140]: Done uploading piece. err: <nil> Aug 7 23:18:35.055266482 (13h38m34.644611469s) [noise=true , address=45.91.116.12:4227 , piece_id=T3JVU7VMSYBTYXBVGRD7LSNIG3PWF3QDVDNDQ24JQOSV7ZFIODNA, seg_pos={0 0}, upload=14140]: Done uploading piece. err: <nil> Aug 7 23:18:35.081593190 (13h38m34.670938177s) [noise=true , address=45.91.116.11:4119 , piece_id=UXO65SB4OZ57BCKOWGE2VDV7KCSKCGE7I7K4RTGRYJ7VU5GKFPHQ, seg_pos={0 0}, upload=14140]: Done uploading piece. err: <nil> Aug 7 23:18:35.123608726 (13h38m34.712953703s) [noise=true , address=77.90.31.12:4589 , piece_id=ZMKTDC4M5JHWVARXQDQAKRFSXM3OQQZ2IGQASJCK2OKYX3IQXHVA, seg_pos={0 0}, upload=14140]: Done uploading piece. err: <nil> Aug 7 23:18:35.123631779 (13h38m34.712976766s) [seg_pos={0 0}, upload=14140]: Segment reached optimal threshold of 13 pieces. Aug 7 23:18:35.123785944 (13h38m34.713130931s) [noise=true , address=45.91.116.7:4009 , piece_id=VKYMGNJEGFLB5M3L3WLD37C5ZILQGLRJROSQ4ZEIBN46CQSY5JZQ, seg_pos={0 0}, upload=14140]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF Aug 7 23:18:35.123945530 (13h38m34.713290517s) [noise=true , address=77.90.31.15:4697 , piece_id=6RYZJ22TCTJFRBMQV7M545VHB475CHDV2DBKCGO3MJP65VEMJ5FQ, seg_pos={0 0}, upload=14140]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: expected piece hash; context canceled; EOF Aug 7 23:18:35.123972431 (13h38m34.713317418s) [noise=true , address=104.36.82.8:4971 , piece_id=DNQVVV7QT2A2NFMIRURJ5IZRFPYZVX2NADRTX2P2YORA2CKMWY6Q, seg_pos={0 0}, upload=14140]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo): protocol: read tcp 188.166.225.170:44698->104.36.82.8:4971: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 23:18:35.123986628 (13h38m34.713331615s) [seg_pos={0 0}, upload=14140]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 7 23:18:35.124083804 (13h38m34.713428791s) [upload=14140]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 23:18:35.124087611 (13h38m34.713432599s) [upload=14140]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:18:35.414558916 (13h38m35.003903903s) [upload=14140]: Done uploading segments... Aug 7 23:18:35.414577682 (13h38m35.00392267s) [upload=14140]: Upload finished. err: <nil> Aug 7 23:18:58.905554387 (13h38m58.494899394s) [upload=14146]: Starting upload Aug 7 23:18:58.905620613 (13h38m58.4949656s) [upload=14146]: Done starting upload Aug 7 23:18:58.905683373 (13h38m58.49502836s) [upload=14146]: Uploading segments... Aug 7 23:18:58.905783355 (13h38m58.495128342s) [upload=14146]: Got next segment. Inline: false Aug 7 23:18:58.905832459 (13h38m58.495177446s) [upload=14146]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:18:58.905836627 (13h38m58.495181614s) [upload=14146]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:19:28.246792926 (13h39m27.836137913s) [upload=14146]: Next segment err: context deadline exceeded Aug 7 23:19:28.246867379 (13h39m27.836212366s) [upload=14146]: Done uploading segments... Aug 7 23:19:28.246884292 (13h39m27.836229269s) [upload=14146]: Upload finished. err: context deadline exceeded Duration: 6.240732668s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 5 out of 60 is failed (threshold=3) [.........................XXXXX..............................] duration: 1.884005439s FAIL (2025-08-07 23:39:27 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== d. err: <nil> Aug 7 23:07:52.874987115 (13h27m52.464332102s) [upload=13971]: Starting upload Aug 7 23:07:52.875039626 (13h27m52.464384603s) [upload=13971]: Done starting upload Aug 7 23:07:52.875078921 (13h27m52.464423908s) [upload=13971]: Uploading segments... Aug 7 23:07:52.875111583 (13h27m52.46445657s) [upload=13971]: Got next segment. Inline: true Aug 7 23:07:52.875115000 (13h27m52.464459987s) [upload=13971]: Waiting for error group managing segments... Aug 7 23:07:52.875144226 (13h27m52.464489203s) [upload=13971]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:07:52.875147783 (13h27m52.46449277s) [upload=13971]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:07:52.875150268 (13h27m52.464495255s) [upload=13971]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:07:53.152523417 (13h27m52.741868404s) [upload=13971]: Done uploading segments... Aug 7 23:07:53.152567841 (13h27m52.741912828s) [upload=13971]: Upload finished. err: <nil> Aug 7 23:08:13.160315277 (13h28m12.749660264s) [upload=13978]: Starting upload Aug 7 23:08:13.160372306 (13h28m12.749717293s) [upload=13978]: Done starting upload Aug 7 23:08:13.160419306 (13h28m12.749764293s) [upload=13978]: Uploading segments... Aug 7 23:08:13.160446879 (13h28m12.749791866s) [upload=13978]: Got next segment. Inline: true Aug 7 23:08:13.160450095 (13h28m12.749795082s) [upload=13978]: Waiting for error group managing segments... Aug 7 23:08:13.160481565 (13h28m12.749826542s) [upload=13978]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:08:13.160485202 (13h28m12.749830189s) [upload=13978]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:08:13.160487416 (13h28m12.749832403s) [upload=13978]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:08:13.673421309 (13h28m13.262766296s) [upload=13978]: Done uploading segments... Aug 7 23:08:13.673439774 (13h28m13.262784761s) [upload=13978]: Upload finished. err: <nil> Aug 7 23:09:08.585854499 (13h29m8.175199486s) [upload=13993]: Starting upload Aug 7 23:09:08.585909965 (13h29m8.175254952s) [upload=13993]: Done starting upload Aug 7 23:09:08.585953217 (13h29m8.175298204s) [upload=13993]: Uploading segments... Aug 7 23:09:08.585980830 (13h29m8.175325817s) [upload=13993]: Got next segment. Inline: true Aug 7 23:09:08.585984206 (13h29m8.175329183s) [upload=13993]: Waiting for error group managing segments... Aug 7 23:09:08.586014684 (13h29m8.175359671s) [upload=13993]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:09:08.586018372 (13h29m8.175363359s) [upload=13993]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:09:08.586020767 (13h29m8.175365754s) [upload=13993]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:09:08.854829559 (13h29m8.444174546s) [upload=13993]: Done uploading segments... Aug 7 23:09:08.854846763 (13h29m8.44419175s) [upload=13993]: Upload finished. err: <nil> Aug 7 23:10:03.068961384 (13h30m2.658306371s) [upload=14012]: Starting upload Aug 7 23:10:03.069021258 (13h30m2.658366245s) [upload=14012]: Done starting upload Aug 7 23:10:03.069094389 (13h30m2.658439376s) [upload=14012]: Uploading segments... Aug 7 23:10:03.069128504 (13h30m2.658473491s) [upload=14012]: Got next segment. Inline: true Aug 7 23:10:03.069132782 (13h30m2.658477779s) [upload=14012]: Waiting for error group managing segments... Aug 7 23:10:03.069170935 (13h30m2.658515932s) [upload=14012]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:10:03.069175905 (13h30m2.658520892s) [upload=14012]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:10:03.069178650 (13h30m2.658523647s) [upload=14012]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:10:03.364714047 (13h30m2.954059064s) [upload=14012]: Done uploading segments... Aug 7 23:10:03.364761748 (13h30m2.954106745s) [upload=14012]: Upload finished. err: <nil> Aug 7 23:10:44.905335214 (13h30m44.494680201s) [upload=14022]: Starting upload Aug 7 23:10:44.905402192 (13h30m44.494747179s) [upload=14022]: Done starting upload Aug 7 23:10:44.905449983 (13h30m44.49479496s) [upload=14022]: Uploading segments... Aug 7 23:10:44.905479650 (13h30m44.494824628s) [upload=14022]: Got next segment. Inline: true Aug 7 23:10:44.905482787 (13h30m44.494827774s) [upload=14022]: Waiting for error group managing segments... Aug 7 23:10:44.905516160 (13h30m44.494861147s) [upload=14022]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:10:44.905519717 (13h30m44.494864694s) [upload=14022]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:10:44.905522222 (13h30m44.494867209s) [upload=14022]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:10:45.196197762 (13h30m44.785542749s) [upload=14022]: Done uploading segments... Aug 7 23:10:45.196218401 (13h30m44.785563389s) [upload=14022]: Upload finished. err: <nil> Aug 7 23:11:56.158319777 (13h31m55.747664764s) [upload=14041]: Starting upload Aug 7 23:11:56.158381214 (13h31m55.747726201s) [upload=14041]: Done starting upload Aug 7 23:11:56.158454435 (13h31m55.747799422s) [upload=14041]: Uploading segments... Aug 7 23:11:56.158485985 (13h31m55.747830972s) [upload=14041]: Got next segment. Inline: true Aug 7 23:11:56.158489752 (13h31m55.747834739s) [upload=14041]: Waiting for error group managing segments... Aug 7 23:11:56.158533225 (13h31m55.747878212s) [upload=14041]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:11:56.158536933 (13h31m55.74788192s) [upload=14041]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:11:56.158539077 (13h31m55.747884064s) [upload=14041]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:12:55.512693945 (13h32m55.102038922s) [upload=14041]: Done uploading segments... Aug 7 23:12:55.512714414 (13h32m55.102059401s) [upload=14041]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:13:04.863194286 (13h33m4.452539303s) [upload=14061]: Starting upload Aug 7 23:13:04.863346336 (13h33m4.452691313s) [upload=14061]: Done starting upload Aug 7 23:13:04.863414577 (13h33m4.452759614s) [upload=14061]: Uploading segments... Aug 7 23:13:04.863490322 (13h33m4.452835309s) [upload=14061]: Got next segment. Inline: true Aug 7 23:13:04.863497876 (13h33m4.452842863s) [upload=14061]: Waiting for error group managing segments... Aug 7 23:13:04.863564063 (13h33m4.45290905s) [upload=14061]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:13:04.863569523 (13h33m4.45291451s) [upload=14061]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:13:04.863571978 (13h33m4.452916965s) [upload=14061]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:14:04.218769699 (13h34m3.808114676s) [upload=14061]: Done uploading segments... Aug 7 23:14:04.218789687 (13h34m3.808134664s) [upload=14061]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:14:05.097995389 (13h34m4.687340376s) [upload=14076]: Starting upload Aug 7 23:14:05.098050264 (13h34m4.687395251s) [upload=14076]: Done starting upload Aug 7 23:14:05.098107544 (13h34m4.687452521s) [upload=14076]: Uploading segments... Aug 7 23:14:05.098195913 (13h34m4.6875409s) [upload=14076]: Got next segment. Inline: true Aug 7 23:14:05.098203497 (13h34m4.687548484s) [upload=14076]: Waiting for error group managing segments... Aug 7 23:14:05.098273151 (13h34m4.687618138s) [upload=14076]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:14:05.098281256 (13h34m4.687626253s) [upload=14076]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:14:05.098285083 (13h34m4.68763007s) [upload=14076]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:15:04.449929685 (13h35m4.039274672s) [upload=14076]: Done uploading segments... Aug 7 23:15:04.449952750 (13h35m4.039297737s) [upload=14076]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:15:05.510260535 (13h35m5.099605522s) [upload=14092]: Starting upload Aug 7 23:15:05.510334587 (13h35m5.099679574s) [upload=14092]: Done starting upload Aug 7 23:15:05.510383931 (13h35m5.099728918s) [upload=14092]: Uploading segments... Aug 7 23:15:05.510421683 (13h35m5.09976667s) [upload=14092]: Got next segment. Inline: true Aug 7 23:15:05.510425931 (13h35m5.099770918s) [upload=14092]: Waiting for error group managing segments... Aug 7 23:15:05.510461179 (13h35m5.099806166s) [upload=14092]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:15:05.510464996 (13h35m5.099809984s) [upload=14092]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:15:05.510467561 (13h35m5.099812548s) [upload=14092]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:16:04.861169851 (13h36m4.450514858s) [upload=14092]: Done uploading segments... Aug 7 23:16:04.861192945 (13h36m4.450537942s) [upload=14092]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:16:05.761665335 (13h36m5.351010322s) [upload=14104]: Starting upload Aug 7 23:16:05.761743936 (13h36m5.351088913s) [upload=14104]: Done starting upload Aug 7 23:16:05.761859217 (13h36m5.351204194s) [upload=14104]: Uploading segments... Aug 7 23:16:05.761910234 (13h36m5.351255221s) [upload=14104]: Got next segment. Inline: true Aug 7 23:16:05.761914483 (13h36m5.35125947s) [upload=14104]: Waiting for error group managing segments... Aug 7 23:16:05.761947837 (13h36m5.351292814s) [upload=14104]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:16:05.761951704 (13h36m5.351296691s) [upload=14104]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:16:05.761955882 (13h36m5.351300859s) [upload=14104]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:17:05.106651963 (13h37m4.69599695s) [upload=14104]: Done uploading segments... Aug 7 23:17:05.106703883 (13h37m4.69604887s) [upload=14104]: Upload finished. err: metaclient: context deadline exceeded Duration: 1.88402677s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 4 out of 60 is failed (threshold=3) [...........................XXXX.............................] duration: 1.951974582s FAIL (2025-08-07 23:39:26 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== .175884539 (13h29m53.765229526s) [upload=14007]: Starting upload Aug 7 23:09:54.175937640 (13h29m53.765282618s) [upload=14007]: Done starting upload Aug 7 23:09:54.175979842 (13h29m53.765324829s) [upload=14007]: Uploading segments... Aug 7 23:09:54.176041389 (13h29m53.765386376s) [upload=14007]: Got next segment. Inline: true Aug 7 23:09:54.176045487 (13h29m53.765390474s) [upload=14007]: Waiting for error group managing segments... Aug 7 23:09:54.176077538 (13h29m53.765422525s) [upload=14007]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:09:54.176082318 (13h29m53.765427305s) [upload=14007]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:09:54.176084522 (13h29m53.765429509s) [upload=14007]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:09:54.453731274 (13h29m54.043076261s) [upload=14007]: Done uploading segments... Aug 7 23:09:54.453750932 (13h29m54.043095909s) [upload=14007]: Upload finished. err: <nil> Aug 7 23:10:39.330325211 (13h30m38.919670198s) [upload=14021]: Starting upload Aug 7 23:10:39.330417358 (13h30m38.919762345s) [upload=14021]: Done starting upload Aug 7 23:10:39.330504384 (13h30m38.919849371s) [upload=14021]: Uploading segments... Aug 7 23:10:39.330543368 (13h30m38.919888355s) [upload=14021]: Got next segment. Inline: true Aug 7 23:10:39.330546675 (13h30m38.919891662s) [upload=14021]: Waiting for error group managing segments... Aug 7 23:10:39.330581371 (13h30m38.919926349s) [upload=14021]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:10:39.330585309 (13h30m38.919930286s) [upload=14021]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:10:39.330587663 (13h30m38.91993265s) [upload=14021]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:10:39.610133078 (13h30m39.199478065s) [upload=14021]: Done uploading segments... Aug 7 23:10:39.610150061 (13h30m39.199495048s) [upload=14021]: Upload finished. err: <nil> Aug 7 23:11:32.695175239 (13h31m32.284520226s) [upload=14034]: Starting upload Aug 7 23:11:32.695285580 (13h31m32.284630567s) [upload=14034]: Done starting upload Aug 7 23:11:32.695336297 (13h31m32.284681284s) [upload=14034]: Uploading segments... Aug 7 23:11:32.695365753 (13h31m32.28471074s) [upload=14034]: Got next segment. Inline: true Aug 7 23:11:32.695369610 (13h31m32.284714598s) [upload=14034]: Waiting for error group managing segments... Aug 7 23:11:32.695401581 (13h31m32.284746569s) [upload=14034]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:11:32.695405168 (13h31m32.284750155s) [upload=14034]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:11:32.695407543 (13h31m32.28475253s) [upload=14034]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:11:32.954750627 (13h31m32.544095614s) [upload=14034]: Done uploading segments... Aug 7 23:11:32.954767650 (13h31m32.544112627s) [upload=14034]: Upload finished. err: <nil> Aug 7 23:12:37.502051138 (13h32m37.091396125s) [upload=14054]: Starting upload Aug 7 23:12:37.502104370 (13h32m37.091449357s) [upload=14054]: Done starting upload Aug 7 23:12:37.502150869 (13h32m37.091495856s) [upload=14054]: Uploading segments... Aug 7 23:12:37.502182519 (13h32m37.091527506s) [upload=14054]: Got next segment. Inline: true Aug 7 23:12:37.502185986 (13h32m37.091530963s) [upload=14054]: Waiting for error group managing segments... Aug 7 23:12:37.502216213 (13h32m37.0915612s) [upload=14054]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:12:37.502219780 (13h32m37.091564767s) [upload=14054]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:12:37.502222425 (13h32m37.091567412s) [upload=14054]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:12:37.772497714 (13h32m37.361842701s) [upload=14054]: Done uploading segments... Aug 7 23:12:37.772511151 (13h32m37.361856138s) [upload=14054]: Upload finished. err: <nil> Aug 7 23:13:44.834965069 (13h33m44.424310046s) [upload=14070]: Starting upload Aug 7 23:13:44.835030293 (13h33m44.42437528s) [upload=14070]: Done starting upload Aug 7 23:13:44.835093484 (13h33m44.424438471s) [upload=14070]: Uploading segments... Aug 7 23:13:44.835133321 (13h33m44.424478308s) [upload=14070]: Got next segment. Inline: true Aug 7 23:13:44.835136998 (13h33m44.424481985s) [upload=14070]: Waiting for error group managing segments... Aug 7 23:13:44.835214376 (13h33m44.424559363s) [upload=14070]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:13:44.835220728 (13h33m44.424565715s) [upload=14070]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:13:44.835222972 (13h33m44.424567959s) [upload=14070]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:13:45.220889346 (13h33m44.810234343s) [upload=14070]: Done uploading segments... Aug 7 23:13:45.220937208 (13h33m44.810282205s) [upload=14070]: Upload finished. err: <nil> Aug 7 23:14:39.930436267 (13h34m39.519781254s) [upload=14085]: Starting upload Aug 7 23:14:39.930523634 (13h34m39.519868621s) [upload=14085]: Done starting upload Aug 7 23:14:39.930574131 (13h34m39.519919118s) [upload=14085]: Uploading segments... Aug 7 23:14:39.930614779 (13h34m39.519959766s) [upload=14085]: Got next segment. Inline: true Aug 7 23:14:39.930618847 (13h34m39.519963824s) [upload=14085]: Waiting for error group managing segments... Aug 7 23:14:39.930656519 (13h34m39.520001506s) [upload=14085]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:14:39.930660767 (13h34m39.520005754s) [upload=14085]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:14:39.930663112 (13h34m39.520008099s) [upload=14085]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:14:40.207143612 (13h34m39.796488599s) [upload=14085]: Done uploading segments... Aug 7 23:14:40.207161857 (13h34m39.796506844s) [upload=14085]: Upload finished. err: <nil> Aug 7 23:14:59.171620401 (13h34m58.760965388s) [upload=14091]: Starting upload Aug 7 23:14:59.171676218 (13h34m58.761021205s) [upload=14091]: Done starting upload Aug 7 23:14:59.171718468 (13h34m58.761063455s) [upload=14091]: Uploading segments... Aug 7 23:14:59.171763204 (13h34m58.761108191s) [upload=14091]: Got next segment. Inline: true Aug 7 23:14:59.171776990 (13h34m58.761121977s) [upload=14091]: Waiting for error group managing segments... Aug 7 23:14:59.171816787 (13h34m58.761161774s) [upload=14091]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:14:59.171820764 (13h34m58.761165741s) [upload=14091]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:14:59.171823429 (13h34m58.761168416s) [upload=14091]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:15:28.519847650 (13h35m28.109192637s) [upload=14091]: Done uploading segments... Aug 7 23:15:28.519870794 (13h35m28.109215781s) [upload=14091]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:15:34.319815009 (13h35m33.909159996s) [upload=14100]: Starting upload Aug 7 23:15:34.319867980 (13h35m33.909212967s) [upload=14100]: Done starting upload Aug 7 23:15:34.319916702 (13h35m33.909261679s) [upload=14100]: Uploading segments... Aug 7 23:15:34.319945739 (13h35m33.909290726s) [upload=14100]: Got next segment. Inline: true Aug 7 23:15:34.319949416 (13h35m33.909294403s) [upload=14100]: Waiting for error group managing segments... Aug 7 23:15:34.319988731 (13h35m33.909333718s) [upload=14100]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:15:34.319992698 (13h35m33.909337685s) [upload=14100]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:15:34.319995163 (13h35m33.90934015s) [upload=14100]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:16:03.672763058 (13h36m3.262108045s) [upload=14100]: Done uploading segments... Aug 7 23:16:03.672786503 (13h36m3.26213149s) [upload=14100]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:16:43.252280188 (13h36m42.841625175s) [upload=14114]: Starting upload Aug 7 23:16:43.252335193 (13h36m42.84168018s) [upload=14114]: Done starting upload Aug 7 23:16:43.252381041 (13h36m42.841726018s) [upload=14114]: Uploading segments... Aug 7 23:16:43.252412271 (13h36m42.841757258s) [upload=14114]: Got next segment. Inline: true Aug 7 23:16:43.252416208 (13h36m42.841761195s) [upload=14114]: Waiting for error group managing segments... Aug 7 23:16:43.252453641 (13h36m42.841798628s) [upload=14114]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:16:43.252457388 (13h36m42.841802375s) [upload=14114]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:16:43.252459622 (13h36m42.841804609s) [upload=14114]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:17:12.607365819 (13h37m12.196710806s) [upload=14114]: Done uploading segments... Aug 7 23:17:12.607385377 (13h37m12.196730364s) [upload=14114]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:17:56.238720000 (13h37m55.828064987s) [upload=14132]: Starting upload Aug 7 23:17:56.238785927 (13h37m55.828130914s) [upload=14132]: Done starting upload Aug 7 23:17:56.238851903 (13h37m55.82819689s) [upload=14132]: Uploading segments... Aug 7 23:17:56.238885677 (13h37m55.828230664s) [upload=14132]: Got next segment. Inline: true Aug 7 23:17:56.238888933 (13h37m55.82823392s) [upload=14132]: Waiting for error group managing segments... Aug 7 23:17:56.238932207 (13h37m55.828277194s) [upload=14132]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:17:56.238935984 (13h37m55.828280971s) [upload=14132]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:17:56.238938278 (13h37m55.828283265s) [upload=14132]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:18:25.593341271 (13h38m25.182686258s) [upload=14132]: Done uploading segments... Aug 7 23:18:25.593355268 (13h38m25.182700245s) [upload=14132]: Upload finished. err: metaclient: context deadline exceeded Duration: 1.951995591s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 4 out of 60 is failed (threshold=3) [.........................X...X..X.X.........................] duration: 1.957046237s FAIL (2025-08-07 23:39:25 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 1.957071304s core uplink us1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [.........................XXXX.X..XX.........................] duration: 6.069483364s FAIL (2025-08-07 23:39:23 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.069506588s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 3 out of 60 is failed (threshold=3) [................................X.....X.X...................] duration: 2.224034284s FAIL (2025-08-07 23:39:21 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.224068539s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 8 out of 60 is failed (threshold=3) [.............................XXXXXX.X.X.....................] duration: 5.787804887s FAIL (2025-08-07 23:39:17 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.787850183s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 5.673033862s FAIL (2025-08-07 23:39:14 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.673061496s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [......................X..X.XXXX.X...........................] duration: 6.848612743s FAIL (2025-08-07 23:39:10 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== 8]: Upload finished. err: context deadline exceeded Aug 7 23:18:32.336054087 (13h38m31.925399094s) [upload=14140]: Starting upload Aug 7 23:18:32.336180319 (13h38m31.925525306s) [upload=14140]: Done starting upload Aug 7 23:18:32.336261133 (13h38m31.92560616s) [upload=14140]: Uploading segments... Aug 7 23:18:32.336590001 (13h38m31.925935029s) [upload=14140]: Got next segment. Inline: false Aug 7 23:18:32.336650628 (13h38m31.925995655s) [upload=14140]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:18:32.336658754 (13h38m31.926003761s) [upload=14140]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:18:32.571732430 (13h38m32.161077417s) [seg_pos={0 0}, upload=14140]: Begin upload segment... Aug 7 23:18:32.571774531 (13h38m32.161119528s) [seg_pos={0 0}, upload=14140]: Done begin upload segment. Aug 7 23:18:32.571805400 (13h38m32.161150417s) [noise=true , address=77.90.31.12:4589 , piece_id=ZMKTDC4M5JHWVARXQDQAKRFSXM3OQQZ2IGQASJCK2OKYX3IQXHVA, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.571844173 (13h38m32.161189161s) [noise=true , address=104.36.82.9:4007 , piece_id=VZLD7RU4SK5MAQE22D7OBVT6XOQAN75UNDMT7JNPCRZNRKFX36XQ, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.572218380 (13h38m32.161563367s) [noise=true , address=104.36.82.10:4077 , piece_id=WYGEC4FYUPAR45MZIWLZQ2ASF5SUPECS4YI3UTRRHEVAKUEYCGYA, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.572289677 (13h38m32.161634654s) [noise=true , address=77.90.31.17:4770 , piece_id=ISA43NF4AKPK2SGYDPKT6ZIAD5WJ27A6GT6JZA2GHQUCAQBRRAFA, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.572494809 (13h38m32.161839796s) [noise=true , address=104.36.82.7:4911 , piece_id=FAMSQZJMALQIEAG5BVR5OV5WHEHN432Y4DZD3DNPC5CKP7X7GMQA, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.572588257 (13h38m32.161933244s) [noise=true , address=77.90.31.15:4697 , piece_id=6RYZJ22TCTJFRBMQV7M545VHB475CHDV2DBKCGO3MJP65VEMJ5FQ, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.572772450 (13h38m32.162117427s) [noise=true , address=45.91.116.12:4225 , piece_id=22GAFTM5SITGTVJEELA3BJP7W2QZW7BVYJGJMKJOLE2RF2GZ5CGQ, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.572865959 (13h38m32.162210946s) [noise=true , address=45.91.116.12:4227 , piece_id=T3JVU7VMSYBTYXBVGRD7LSNIG3PWF3QDVDNDQ24JQOSV7ZFIODNA, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.573029050 (13h38m32.162374037s) [noise=true , address=77.90.31.19:4842 , piece_id=3ZRHNMBPQUJH5SZZ75UJWLYIYZ5YI7CC5OMIJJFM7WVKZWJX6R3A, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.573227882 (13h38m32.162572869s) [noise=true , address=104.36.82.4:4804 , piece_id=LTSBCPH74YSWH7IHN6MH4NWTBJ5FQ7534VERPCY5NIDV24KYPXMQ, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.573293797 (13h38m32.162638774s) [noise=true , address=104.36.82.15:4259 , piece_id=GFCTNTXSY4WVSTPAD3AZIYCHE4O5EN24WN25VUNCRNWCSUWCYBMA, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.573504962 (13h38m32.162849949s) [noise=true , address=77.90.31.16:4734 , piece_id=OJPJZ5RF7QY4KNFBZ3MAVTA63HMC4JRG3HQLBN7SVROVKFQZY7BQ, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.573527936 (13h38m32.162872923s) [noise=true , address=45.91.116.7:4009 , piece_id=VKYMGNJEGFLB5M3L3WLD37C5ZILQGLRJROSQ4ZEIBN46CQSY5JZQ, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.574078970 (13h38m32.163423947s) [noise=true , address=45.91.116.11:4119 , piece_id=UXO65SB4OZ57BCKOWGE2VDV7KCSKCGE7I7K4RTGRYJ7VU5GKFPHQ, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.574216453 (13h38m32.16356143s) [noise=true , address=104.36.82.12:4149 , piece_id=2K5BCSGU4UN43HF7VHG3MBNMQEGSIZPPHTDHDFI3XPY5DDG6NKOA, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.576753347 (13h38m32.166098334s) [noise=true , address=104.36.82.8:4971 , piece_id=DNQVVV7QT2A2NFMIRURJ5IZRFPYZVX2NADRTX2P2YORA2CKMWY6Q, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:34.327895253 (13h38m33.91724024s) [upload=14140]: Next returned nil segment Aug 7 23:18:34.327912295 (13h38m33.917257282s) [upload=14140]: Waiting for error group managing segments... Aug 7 23:18:34.716828436 (13h38m34.306173413s) [noise=true , address=104.36.82.7:4911 , piece_id=FAMSQZJMALQIEAG5BVR5OV5WHEHN432Y4DZD3DNPC5CKP7X7GMQA, seg_pos={0 0}, upload=14140]: Done uploading piece. err: <nil> Aug 7 23:18:34.729509878 (13h38m34.318854865s) [noise=true , address=104.36.82.12:4149 , piece_id=2K5BCSGU4UN43HF7VHG3MBNMQEGSIZPPHTDHDFI3XPY5DDG6NKOA, seg_pos={0 0}, upload=14140]: Done uploading piece. err: <nil> Aug 7 23:18:34.738858565 (13h38m34.328203592s) [noise=true , address=104.36.82.4:4804 , piece_id=LTSBCPH74YSWH7IHN6MH4NWTBJ5FQ7534VERPCY5NIDV24KYPXMQ, seg_pos={0 0}, upload=14140]: Done uploading piece. err: <nil> Aug 7 23:18:34.783948702 (13h38m34.373293679s) [noise=true , address=77.90.31.19:4842 , piece_id=3ZRHNMBPQUJH5SZZ75UJWLYIYZ5YI7CC5OMIJJFM7WVKZWJX6R3A, seg_pos={0 0}, upload=14140]: Done uploading piece. err: <nil> Aug 7 23:18:34.824404112 (13h38m34.413749099s) [noise=true , address=104.36.82.15:4259 , piece_id=GFCTNTXSY4WVSTPAD3AZIYCHE4O5EN24WN25VUNCRNWCSUWCYBMA, seg_pos={0 0}, upload=14140]: Done uploading piece. err: <nil> Aug 7 23:18:34.863114634 (13h38m34.452459611s) [noise=true , address=104.36.82.9:4007 , piece_id=VZLD7RU4SK5MAQE22D7OBVT6XOQAN75UNDMT7JNPCRZNRKFX36XQ, seg_pos={0 0}, upload=14140]: Done uploading piece. err: <nil> Aug 7 23:18:34.889984711 (13h38m34.479329698s) [noise=true , address=77.90.31.16:4734 , piece_id=OJPJZ5RF7QY4KNFBZ3MAVTA63HMC4JRG3HQLBN7SVROVKFQZY7BQ, seg_pos={0 0}, upload=14140]: Done uploading piece. err: <nil> Aug 7 23:18:34.982222673 (13h38m34.57156766s) [noise=true , address=104.36.82.10:4077 , piece_id=WYGEC4FYUPAR45MZIWLZQ2ASF5SUPECS4YI3UTRRHEVAKUEYCGYA, seg_pos={0 0}, upload=14140]: Done uploading piece. err: <nil> Aug 7 23:18:34.999676977 (13h38m34.589021964s) [noise=true , address=77.90.31.17:4770 , piece_id=ISA43NF4AKPK2SGYDPKT6ZIAD5WJ27A6GT6JZA2GHQUCAQBRRAFA, seg_pos={0 0}, upload=14140]: Done uploading piece. err: <nil> Aug 7 23:18:35.002083712 (13h38m34.591428699s) [noise=true , address=45.91.116.12:4225 , piece_id=22GAFTM5SITGTVJEELA3BJP7W2QZW7BVYJGJMKJOLE2RF2GZ5CGQ, seg_pos={0 0}, upload=14140]: Done uploading piece. err: <nil> Aug 7 23:18:35.055266482 (13h38m34.644611469s) [noise=true , address=45.91.116.12:4227 , piece_id=T3JVU7VMSYBTYXBVGRD7LSNIG3PWF3QDVDNDQ24JQOSV7ZFIODNA, seg_pos={0 0}, upload=14140]: Done uploading piece. err: <nil> Aug 7 23:18:35.081593190 (13h38m34.670938177s) [noise=true , address=45.91.116.11:4119 , piece_id=UXO65SB4OZ57BCKOWGE2VDV7KCSKCGE7I7K4RTGRYJ7VU5GKFPHQ, seg_pos={0 0}, upload=14140]: Done uploading piece. err: <nil> Aug 7 23:18:35.123608726 (13h38m34.712953703s) [noise=true , address=77.90.31.12:4589 , piece_id=ZMKTDC4M5JHWVARXQDQAKRFSXM3OQQZ2IGQASJCK2OKYX3IQXHVA, seg_pos={0 0}, upload=14140]: Done uploading piece. err: <nil> Aug 7 23:18:35.123631779 (13h38m34.712976766s) [seg_pos={0 0}, upload=14140]: Segment reached optimal threshold of 13 pieces. Aug 7 23:18:35.123785944 (13h38m34.713130931s) [noise=true , address=45.91.116.7:4009 , piece_id=VKYMGNJEGFLB5M3L3WLD37C5ZILQGLRJROSQ4ZEIBN46CQSY5JZQ, seg_pos={0 0}, upload=14140]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF Aug 7 23:18:35.123945530 (13h38m34.713290517s) [noise=true , address=77.90.31.15:4697 , piece_id=6RYZJ22TCTJFRBMQV7M545VHB475CHDV2DBKCGO3MJP65VEMJ5FQ, seg_pos={0 0}, upload=14140]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: expected piece hash; context canceled; EOF Aug 7 23:18:35.123972431 (13h38m34.713317418s) [noise=true , address=104.36.82.8:4971 , piece_id=DNQVVV7QT2A2NFMIRURJ5IZRFPYZVX2NADRTX2P2YORA2CKMWY6Q, seg_pos={0 0}, upload=14140]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo): protocol: read tcp 188.166.225.170:44698->104.36.82.8:4971: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 23:18:35.123986628 (13h38m34.713331615s) [seg_pos={0 0}, upload=14140]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 7 23:18:35.124083804 (13h38m34.713428791s) [upload=14140]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 23:18:35.124087611 (13h38m34.713432599s) [upload=14140]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:18:35.414558916 (13h38m35.003903903s) [upload=14140]: Done uploading segments... Aug 7 23:18:35.414577682 (13h38m35.00392267s) [upload=14140]: Upload finished. err: <nil> Aug 7 23:18:58.905554387 (13h38m58.494899394s) [upload=14146]: Starting upload Aug 7 23:18:58.905620613 (13h38m58.4949656s) [upload=14146]: Done starting upload Aug 7 23:18:58.905683373 (13h38m58.49502836s) [upload=14146]: Uploading segments... Aug 7 23:18:58.905783355 (13h38m58.495128342s) [upload=14146]: Got next segment. Inline: false Aug 7 23:18:58.905832459 (13h38m58.495177446s) [upload=14146]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:18:58.905836627 (13h38m58.495181614s) [upload=14146]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:19:28.246792926 (13h39m27.836137913s) [upload=14146]: Next segment err: context deadline exceeded Aug 7 23:19:28.246867379 (13h39m27.836212366s) [upload=14146]: Done uploading segments... Aug 7 23:19:28.246884292 (13h39m27.836229269s) [upload=14146]: Upload finished. err: context deadline exceeded Duration: 6.848636097s core uplink us-select-1
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 651.09821ms target: https://us1.storj.io/login FAIL (2025-08-07 23:39:09 UTC) orbiter warning: Get "https://us1.storj.io/login": dial tcp 34.150.199.48:443: connect: connection refused Duration: 651.124098ms core satellite us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 3 out of 60 is failed (threshold=3) [.....X....................X.....................X...........] duration: 6.534141889s FAIL (2025-08-07 23:39:06 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/4M") ======LOGS====== 916s) [upload=14346]: Starting upload Aug 7 23:30:30.723725030 (13h50m30.313070017s) [upload=14346]: Done starting upload Aug 7 23:30:30.723796056 (13h50m30.313141043s) [upload=14346]: Uploading segments... Aug 7 23:30:30.723935803 (13h50m30.31328079s) [upload=14346]: Got next segment. Inline: false Aug 7 23:30:30.723960240 (13h50m30.313305227s) [upload=14346]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:30:30.723964218 (13h50m30.313309205s) [upload=14346]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:30:31.181251948 (13h50m30.770596926s) [seg_pos={0 0}, upload=14346]: Begin upload segment... Aug 7 23:30:31.181299649 (13h50m30.770644636s) [seg_pos={0 0}, upload=14346]: Done begin upload segment. Aug 7 23:30:31.181318165 (13h50m30.770663152s) [noise=true , address=104.36.82.52:4086 , piece_id=JLEUROLLGI7X6MUOVFMEWIFCQNTYJXSZW2JMHGEAI66XREHK5LLQ, seg_pos={0 0}, upload=14346]: Uploading piece... Aug 7 23:30:31.181329696 (13h50m30.770674693s) [noise=true , address=104.36.82.54:4172 , piece_id=2QZVEPRDWBX4KY62QMLTPKNZ7QA2K3YDWJETONKWYOGLMPUVSVWA, seg_pos={0 0}, upload=14346]: Uploading piece... Aug 7 23:30:31.181716817 (13h50m30.771061804s) [noise=true , address=77.90.31.17:4797 , piece_id=P6M6EN2C7M6FVKVWV4LE6JEHSZTB24YEACVG7TMSNBQFLD6NLKDQ, seg_pos={0 0}, upload=14346]: Uploading piece... Aug 7 23:30:31.181721437 (13h50m30.771066424s) [noise=true , address=104.36.82.46:4489 , piece_id=KYHATGRQXZ6XXQFUSTYUYBI26BTC7VSN5N6OM766B3QOSWN747IA, seg_pos={0 0}, upload=14346]: Uploading piece... Aug 7 23:30:31.182063421 (13h50m30.771408408s) [noise=true , address=104.36.82.22:4534 , piece_id=5BROJMGAIDCZNPDEQH32QVVYD2JP4U7IN5DCDUP7LBMU2E6Z36CA, seg_pos={0 0}, upload=14346]: Uploading piece... Aug 7 23:30:31.182318108 (13h50m30.771663085s) [noise=true , address=104.36.82.58:4267 , piece_id=QVRI7PRWV4JSPRWQIJD6M7UJAWDBWN2VIVGOYUJ7NGO77SHZ4UJQ, seg_pos={0 0}, upload=14346]: Uploading piece... Aug 7 23:30:31.182483654 (13h50m30.771828641s) [noise=true , address=64.49.13.138:4489 , piece_id=GC6CUUPTPHVDNMD2ZDX5EUJSEUNL6I6OM3MYQNM25F2G7X64YQLQ, seg_pos={0 0}, upload=14346]: Uploading piece... Aug 7 23:30:31.182639132 (13h50m30.771984119s) [noise=true , address=45.91.116.11:4149 , piece_id=JGSWIYMTFANVCD326G2TRN24X26XYQQSJT6HFWBQAHJLQ2SLBA4A, seg_pos={0 0}, upload=14346]: Uploading piece... Aug 7 23:30:31.182934928 (13h50m30.772279905s) [noise=true , address=104.36.82.62:4428 , piece_id=P7EOR2HLYCVOBQBDUZ3OYNVO4RV7WRXLHVREWKIKE7SHYBV3VRVQ, seg_pos={0 0}, upload=14346]: Uploading piece... Aug 7 23:30:31.183037023 (13h50m30.77238201s) [noise=true , address=64.49.13.155:4917 , piece_id=GLY2EUU2PUD7JB4V7PAKNNFK6ZDWGN2DKS5YHTXUNAJXWGRYRTJA, seg_pos={0 0}, upload=14346]: Uploading piece... Aug 7 23:30:31.183158877 (13h50m30.772503864s) [noise=true , address=104.36.82.47:4534 , piece_id=PJCWOCQ7UKEHHSERR6C5RQC4Y3TBZRHZF6KPMZIMTAFS6E4525EA, seg_pos={0 0}, upload=14346]: Uploading piece... Aug 7 23:30:31.183353329 (13h50m30.772698316s) [noise=true , address=104.36.82.44:4883 , piece_id=7GULYV2REOYLHXZXRSPIQM46DRT4YKT7FXC3YCHAP7W7332RRTRA, seg_pos={0 0}, upload=14346]: Uploading piece... Aug 7 23:30:31.183386993 (13h50m30.77273198s) [noise=true , address=104.36.82.52:4097 , piece_id=REDOEPTMV6TQIM5HWQ2DQBPENXJFHVZV5LGM244DRAMGSV6HSNRQ, seg_pos={0 0}, upload=14346]: Uploading piece... Aug 7 23:30:31.183591444 (13h50m30.772936431s) [noise=true , address=104.36.82.60:4353 , piece_id=3TWE6IJ3OCY5CMMRHZ5DG2WBOHLWF3MEESPOGUMU3YA5NSGW3AKA, seg_pos={0 0}, upload=14346]: Uploading piece... Aug 7 23:30:31.183849538 (13h50m30.773194525s) [noise=true , address=64.49.13.27:4960 , piece_id=RCNXI4GI75JF5TWSSGMTNPWOTU4C3TADGATSBJQKJBRLVELNSGUA, seg_pos={0 0}, upload=14346]: Uploading piece... Aug 7 23:30:31.184184719 (13h50m30.773529706s) [noise=true , address=104.36.82.61:4392 , piece_id=OSMRSZL5LDLYLPM3KN5YVIMBU44HS72BHZOAB7W5SS5Z4DY4F4JA, seg_pos={0 0}, upload=14346]: Uploading piece... Aug 7 23:30:31.184422474 (13h50m30.773767461s) [noise=true , address=104.36.82.8:4995 , piece_id=NS2X5RBAQQWBIMO2U4XYCYA37GVF6OBAO3GZSTEPJZOXRMPMS62A, seg_pos={0 0}, upload=14346]: Uploading piece... Aug 7 23:30:31.184480636 (13h50m30.773825623s) [noise=true , address=104.36.82.45:4910 , piece_id=QEYOF4IB42OVZTWGW3YD7ZUYX2KZ6KKOUXFUHQQYOW4QY3JUGTDQ, seg_pos={0 0}, upload=14346]: Uploading piece... Aug 7 23:30:32.295332666 (13h50m31.884677653s) [upload=14346]: Next returned nil segment Aug 7 23:30:32.295347043 (13h50m31.88469204s) [upload=14346]: Waiting for error group managing segments... Aug 7 23:30:32.576560505 (13h50m32.165905492s) [noise=true , address=64.49.13.155:4917 , piece_id=GLY2EUU2PUD7JB4V7PAKNNFK6ZDWGN2DKS5YHTXUNAJXWGRYRTJA, seg_pos={0 0}, upload=14346]: Done uploading piece. err: <nil> Aug 7 23:30:32.584528981 (13h50m32.173873958s) [noise=true , address=64.49.13.138:4489 , piece_id=GC6CUUPTPHVDNMD2ZDX5EUJSEUNL6I6OM3MYQNM25F2G7X64YQLQ, seg_pos={0 0}, upload=14346]: Done uploading piece. err: <nil> Aug 7 23:30:32.647385961 (13h50m32.236730948s) [noise=true , address=64.49.13.27:4960 , piece_id=RCNXI4GI75JF5TWSSGMTNPWOTU4C3TADGATSBJQKJBRLVELNSGUA, seg_pos={0 0}, upload=14346]: Done uploading piece. err: <nil> Aug 7 23:30:33.094757008 (13h50m32.684101995s) [noise=true , address=104.36.82.52:4097 , piece_id=REDOEPTMV6TQIM5HWQ2DQBPENXJFHVZV5LGM244DRAMGSV6HSNRQ, seg_pos={0 0}, upload=14346]: Done uploading piece. err: <nil> Aug 7 23:30:33.167659426 (13h50m32.757004403s) [noise=true , address=104.36.82.45:4910 , piece_id=QEYOF4IB42OVZTWGW3YD7ZUYX2KZ6KKOUXFUHQQYOW4QY3JUGTDQ, seg_pos={0 0}, upload=14346]: Done uploading piece. err: <nil> Aug 7 23:30:33.198379320 (13h50m32.787724307s) [noise=true , address=104.36.82.54:4172 , piece_id=2QZVEPRDWBX4KY62QMLTPKNZ7QA2K3YDWJETONKWYOGLMPUVSVWA, seg_pos={0 0}, upload=14346]: Done uploading piece. err: <nil> Aug 7 23:30:33.235504739 (13h50m32.824849726s) [noise=true , address=104.36.82.58:4267 , piece_id=QVRI7PRWV4JSPRWQIJD6M7UJAWDBWN2VIVGOYUJ7NGO77SHZ4UJQ, seg_pos={0 0}, upload=14346]: Done uploading piece. err: <nil> Aug 7 23:30:33.264110248 (13h50m32.853455225s) [noise=true , address=104.36.82.52:4086 , piece_id=JLEUROLLGI7X6MUOVFMEWIFCQNTYJXSZW2JMHGEAI66XREHK5LLQ, seg_pos={0 0}, upload=14346]: Done uploading piece. err: <nil> Aug 7 23:30:33.327068230 (13h50m32.916413217s) [noise=true , address=104.36.82.8:4995 , piece_id=NS2X5RBAQQWBIMO2U4XYCYA37GVF6OBAO3GZSTEPJZOXRMPMS62A, seg_pos={0 0}, upload=14346]: Done uploading piece. err: <nil> Aug 7 23:30:33.366778193 (13h50m32.956123181s) [noise=true , address=104.36.82.60:4353 , piece_id=3TWE6IJ3OCY5CMMRHZ5DG2WBOHLWF3MEESPOGUMU3YA5NSGW3AKA, seg_pos={0 0}, upload=14346]: Done uploading piece. err: <nil> Aug 7 23:30:33.542767028 (13h50m33.132112016s) [noise=true , address=104.36.82.46:4489 , piece_id=KYHATGRQXZ6XXQFUSTYUYBI26BTC7VSN5N6OM766B3QOSWN747IA, seg_pos={0 0}, upload=14346]: Done uploading piece. err: <nil> Aug 7 23:30:33.575265094 (13h50m33.164610081s) [noise=true , address=104.36.82.22:4534 , piece_id=5BROJMGAIDCZNPDEQH32QVVYD2JP4U7IN5DCDUP7LBMU2E6Z36CA, seg_pos={0 0}, upload=14346]: Done uploading piece. err: <nil> Aug 7 23:30:33.672325632 (13h50m33.261670619s) [noise=true , address=77.90.31.17:4797 , piece_id=P6M6EN2C7M6FVKVWV4LE6JEHSZTB24YEACVG7TMSNBQFLD6NLKDQ, seg_pos={0 0}, upload=14346]: Done uploading piece. err: <nil> Aug 7 23:30:33.750650465 (13h50m33.339995452s) [noise=true , address=45.91.116.11:4149 , piece_id=JGSWIYMTFANVCD326G2TRN24X26XYQQSJT6HFWBQAHJLQ2SLBA4A, seg_pos={0 0}, upload=14346]: Done uploading piece. err: <nil> Aug 7 23:30:33.750672889 (13h50m33.340017876s) [seg_pos={0 0}, upload=14346]: Segment reached optimal threshold of 14 pieces. Aug 7 23:30:33.750785674 (13h50m33.340130661s) [noise=true , address=104.36.82.47:4534 , piece_id=PJCWOCQ7UKEHHSERR6C5RQC4Y3TBZRHZF6KPMZIMTAFS6E4525EA, seg_pos={0 0}, upload=14346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VDrNGcCnaKEmG5NN4nex1JoZAgdCEfX5CdsKbxytfLfKzPbr): protocol: expected piece hash; context canceled; EOF Aug 7 23:30:33.750885715 (13h50m33.340230702s) [noise=true , address=104.36.82.61:4392 , piece_id=OSMRSZL5LDLYLPM3KN5YVIMBU44HS72BHZOAB7W5SS5Z4DY4F4JA, seg_pos={0 0}, upload=14346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14FTsLbtv1vp43aRdDZZBY2Gf6epidM1A8HSyQKVokXGFFZGZk): protocol: expected piece hash; context canceled; EOF Aug 7 23:30:33.750979806 (13h50m33.340324793s) [noise=true , address=104.36.82.62:4428 , piece_id=P7EOR2HLYCVOBQBDUZ3OYNVO4RV7WRXLHVREWKIKE7SHYBV3VRVQ, seg_pos={0 0}, upload=14346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Ui5Ye48cKNGFPakMKsjdBi13Xo7bzrmXPNEmRBLKwpMtVAA3R): protocol: rpc: dial tcp 104.36.82.62:4428: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 23:30:33.751073955 (13h50m33.340418942s) [noise=true , address=104.36.82.44:4883 , piece_id=7GULYV2REOYLHXZXRSPIQM46DRT4YKT7FXC3YCHAP7W7332RRTRA, seg_pos={0 0}, upload=14346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cx1avL4umDm2pzJS1rNZqeDZjZMWnMtNsyNTdMdkxhpa1ByUN): protocol: context canceled; context canceled Aug 7 23:30:33.751119673 (13h50m33.34046466s) [seg_pos={0 0}, upload=14346]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 23:30:33.751224474 (13h50m33.340569471s) [upload=14346]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 23:30:33.751230776 (13h50m33.340575763s) [upload=14346]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:30:34.056540019 (13h50m33.645885006s) [upload=14346]: Done uploading segments... Aug 7 23:30:34.056559826 (13h50m33.645904823s) [upload=14346]: Upload finished. err: <nil> Duration: 6.534168149s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [......................X......X..............................] duration: 1.830664967s FAIL (2025-08-07 23:38:57 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== 30.892541890 (13h31m30.481886877s) [upload=14032]: Upload finished. err: <nil> Aug 7 23:11:48.390915893 (13h31m47.98026088s) [upload=14038]: Starting upload Aug 7 23:11:48.390965198 (13h31m47.980310185s) [upload=14038]: Done starting upload Aug 7 23:11:48.391014242 (13h31m47.980359229s) [upload=14038]: Uploading segments... Aug 7 23:11:48.391054007 (13h31m47.980399004s) [upload=14038]: Got next segment. Inline: true Aug 7 23:11:48.391057494 (13h31m47.980402471s) [upload=14038]: Waiting for error group managing segments... Aug 7 23:11:48.391089254 (13h31m47.980434231s) [upload=14038]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:11:48.391093213 (13h31m47.98043819s) [upload=14038]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:11:48.391095457 (13h31m47.980440444s) [upload=14038]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:11:48.701950724 (13h31m48.291295711s) [upload=14038]: Done uploading segments... Aug 7 23:11:48.701969449 (13h31m48.291314436s) [upload=14038]: Upload finished. err: <nil> Aug 7 23:12:06.430245154 (13h32m6.019590141s) [upload=14048]: Starting upload Aug 7 23:12:06.430293076 (13h32m6.019638063s) [upload=14048]: Done starting upload Aug 7 23:12:06.430361757 (13h32m6.019706745s) [upload=14048]: Uploading segments... Aug 7 23:12:06.430459223 (13h32m6.01980421s) [upload=14048]: Got next segment. Inline: true Aug 7 23:12:06.430464543 (13h32m6.01980953s) [upload=14048]: Waiting for error group managing segments... Aug 7 23:12:06.430527555 (13h32m6.019872542s) [upload=14048]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:12:06.430533796 (13h32m6.019878783s) [upload=14048]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:12:06.430536692 (13h32m6.019881679s) [upload=14048]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:12:06.729952062 (13h32m6.319297049s) [upload=14048]: Done uploading segments... Aug 7 23:12:06.729968844 (13h32m6.319313832s) [upload=14048]: Upload finished. err: <nil> Aug 7 23:12:37.551701338 (13h32m37.141046325s) [upload=14055]: Starting upload Aug 7 23:12:37.551746985 (13h32m37.141091972s) [upload=14055]: Done starting upload Aug 7 23:12:37.551807301 (13h32m37.141152288s) [upload=14055]: Uploading segments... Aug 7 23:12:37.551914826 (13h32m37.141259833s) [upload=14055]: Got next segment. Inline: true Aug 7 23:12:37.551921779 (13h32m37.141266766s) [upload=14055]: Waiting for error group managing segments... Aug 7 23:12:37.551977145 (13h32m37.141322122s) [upload=14055]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:12:37.551984059 (13h32m37.141329086s) [upload=14055]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:12:37.551988918 (13h32m37.141333936s) [upload=14055]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:12:37.767068918 (13h32m37.356413905s) [upload=14055]: Done uploading segments... Aug 7 23:12:37.767104365 (13h32m37.356449342s) [upload=14055]: Upload finished. err: metaclient: Too Many Requests Aug 7 23:13:47.522489627 (13h33m47.111834614s) [upload=14072]: Starting upload Aug 7 23:13:47.522559681 (13h33m47.111904668s) [upload=14072]: Done starting upload Aug 7 23:13:47.522629674 (13h33m47.111974661s) [upload=14072]: Uploading segments... Aug 7 23:13:47.522726771 (13h33m47.112071778s) [upload=14072]: Got next segment. Inline: true Aug 7 23:13:47.522735617 (13h33m47.112080614s) [upload=14072]: Waiting for error group managing segments... Aug 7 23:13:47.522790582 (13h33m47.112135569s) [upload=14072]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:13:47.522796493 (13h33m47.112141481s) [upload=14072]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:13:47.522799320 (13h33m47.112144307s) [upload=14072]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:13:47.796144342 (13h33m47.385489329s) [upload=14072]: Done uploading segments... Aug 7 23:13:47.796162656 (13h33m47.385507643s) [upload=14072]: Upload finished. err: <nil> Aug 7 23:14:22.239921861 (13h34m21.829266848s) [upload=14081]: Starting upload Aug 7 23:14:22.239973791 (13h34m21.829318778s) [upload=14081]: Done starting upload Aug 7 23:14:22.240087399 (13h34m21.829432396s) [upload=14081]: Uploading segments... Aug 7 23:14:22.240187159 (13h34m21.829532156s) [upload=14081]: Got next segment. Inline: true Aug 7 23:14:22.240194182 (13h34m21.829539179s) [upload=14081]: Waiting for error group managing segments... Aug 7 23:14:22.240255971 (13h34m21.829600958s) [upload=14081]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:14:22.240261311 (13h34m21.829606288s) [upload=14081]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:14:22.240263816 (13h34m21.829608803s) [upload=14081]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:14:22.537482123 (13h34m22.12682711s) [upload=14081]: Done uploading segments... Aug 7 23:14:22.537500677 (13h34m22.126845664s) [upload=14081]: Upload finished. err: <nil> Aug 7 23:14:56.968922831 (13h34m56.558267818s) [upload=14090]: Starting upload Aug 7 23:14:56.968974069 (13h34m56.558319056s) [upload=14090]: Done starting upload Aug 7 23:14:56.969028383 (13h34m56.55837337s) [upload=14090]: Uploading segments... Aug 7 23:14:56.969059602 (13h34m56.55840459s) [upload=14090]: Got next segment. Inline: true Aug 7 23:14:56.969063500 (13h34m56.558408487s) [upload=14090]: Waiting for error group managing segments... Aug 7 23:14:56.969102906 (13h34m56.558447893s) [upload=14090]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:14:56.969107034 (13h34m56.558452021s) [upload=14090]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:14:56.969109899 (13h34m56.558454886s) [upload=14090]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:14:57.268863065 (13h34m56.858208052s) [upload=14090]: Done uploading segments... Aug 7 23:14:57.268882201 (13h34m56.858227188s) [upload=14090]: Upload finished. err: <nil> Aug 7 23:16:09.138990771 (13h36m8.728335758s) [upload=14107]: Starting upload Aug 7 23:16:09.139048231 (13h36m8.728393208s) [upload=14107]: Done starting upload Aug 7 23:16:09.139094229 (13h36m8.728439226s) [upload=14107]: Uploading segments... Aug 7 23:16:09.139131069 (13h36m8.728476056s) [upload=14107]: Got next segment. Inline: true Aug 7 23:16:09.139135007 (13h36m8.728479994s) [upload=14107]: Waiting for error group managing segments... Aug 7 23:16:09.139167859 (13h36m8.728512846s) [upload=14107]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:16:09.139171937 (13h36m8.728516924s) [upload=14107]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:16:09.139199730 (13h36m8.728544707s) [upload=14107]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:16:09.409287150 (13h36m8.998632137s) [upload=14107]: Done uploading segments... Aug 7 23:16:09.409303952 (13h36m8.998648939s) [upload=14107]: Upload finished. err: <nil> Aug 7 23:16:30.665596455 (13h36m30.254941442s) [upload=14111]: Starting upload Aug 7 23:16:30.665642382 (13h36m30.254987369s) [upload=14111]: Done starting upload Aug 7 23:16:30.665687749 (13h36m30.255032726s) [upload=14111]: Uploading segments... Aug 7 23:16:30.665725801 (13h36m30.255070788s) [upload=14111]: Got next segment. Inline: true Aug 7 23:16:30.665729409 (13h36m30.255074386s) [upload=14111]: Waiting for error group managing segments... Aug 7 23:16:30.665763554 (13h36m30.255108541s) [upload=14111]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:16:30.665768984 (13h36m30.255113971s) [upload=14111]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:16:30.665771289 (13h36m30.255116276s) [upload=14111]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:16:30.945395745 (13h36m30.534740732s) [upload=14111]: Done uploading segments... Aug 7 23:16:30.945418538 (13h36m30.534763526s) [upload=14111]: Upload finished. err: <nil> Aug 7 23:17:25.677744155 (13h37m25.267089143s) [upload=14125]: Starting upload Aug 7 23:17:25.677793930 (13h37m25.267138917s) [upload=14125]: Done starting upload Aug 7 23:17:25.677862562 (13h37m25.267207539s) [upload=14125]: Uploading segments... Aug 7 23:17:25.677896967 (13h37m25.267241954s) [upload=14125]: Got next segment. Inline: true Aug 7 23:17:25.677900704 (13h37m25.267245681s) [upload=14125]: Waiting for error group managing segments... Aug 7 23:17:25.677933046 (13h37m25.267278033s) [upload=14125]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:17:25.677936884 (13h37m25.267281871s) [upload=14125]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:17:25.677939309 (13h37m25.267284296s) [upload=14125]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:17:26.502642065 (13h37m26.091987052s) [upload=14125]: Done uploading segments... Aug 7 23:17:26.502660942 (13h37m26.092005919s) [upload=14125]: Upload finished. err: <nil> Aug 7 23:17:53.025114325 (13h37m52.614459313s) [upload=14131]: Starting upload Aug 7 23:17:53.025169641 (13h37m52.614514618s) [upload=14131]: Done starting upload Aug 7 23:17:53.025219857 (13h37m52.614564844s) [upload=14131]: Uploading segments... Aug 7 23:17:53.025267939 (13h37m52.614612936s) [upload=14131]: Got next segment. Inline: true Aug 7 23:17:53.025271916 (13h37m52.614616903s) [upload=14131]: Waiting for error group managing segments... Aug 7 23:17:53.025302495 (13h37m52.614647482s) [upload=14131]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:17:53.025306382 (13h37m52.614651369s) [upload=14131]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:17:53.025308497 (13h37m52.614653484s) [upload=14131]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:17:53.302835519 (13h37m52.892180506s) [upload=14131]: Done uploading segments... Aug 7 23:17:53.302854656 (13h37m52.892199633s) [upload=14131]: Upload finished. err: metaclient: Too Many Requests Duration: 1.830718009s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [...............................XX.X.........................] duration: 8.759569492s FAIL (2025-08-07 23:38:55 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== PC5WABH5C4FB5BLC55F4FZQAU4IAHMEA, seg_pos={0 0}, upload=14126]: Uploading piece... Aug 7 23:17:27.821935980 (13h37m27.411280967s) [noise=true , address=104.36.82.8:4983 , piece_id=AVNQRSGZB5XHG3VFYFHRLBLKQ7B4YWXGIPWDK32DYJSMFXUFTSNA, seg_pos={0 0}, upload=14126]: Uploading piece... Aug 7 23:17:27.822230193 (13h37m27.41157518s) [noise=true , address=104.36.82.16:4310 , piece_id=OAD3ECB3B5RXQB362MCLIWSQC4PKS3QUHWVMIMK4XPGH63JKO3LQ, seg_pos={0 0}, upload=14126]: Uploading piece... Aug 7 23:17:27.822283705 (13h37m27.411628692s) [noise=true , address=104.36.82.59:4323 , piece_id=WNEN4TRJFQ64H74WZOQGGP3V5F7Y3A7OIMORF7M7WEYOR252JSRA, seg_pos={0 0}, upload=14126]: Uploading piece... Aug 7 23:17:27.822684663 (13h37m27.41202965s) [noise=true , address=104.36.82.39:4747 , piece_id=7OQQP63SGNWWQEYR75RRJRHZSL52NRYH3BCS65XI4SWA24BKERUQ, seg_pos={0 0}, upload=14126]: Uploading piece... Aug 7 23:17:27.823034542 (13h37m27.412379529s) [noise=true , address=104.36.82.59:4322 , piece_id=D4THBNGRCWHLJC6HETJRODVC7ZRMOWMIAHGM6GYVU7LFIJNBTOZQ, seg_pos={0 0}, upload=14126]: Uploading piece... Aug 7 23:17:27.823366667 (13h37m27.412711654s) [noise=true , address=104.36.82.60:4343 , piece_id=UHOQFZPYXHO56TUWMTSQE5XVWFGO66ATAA3EU7TQ5KX2MIN2ZJQQ, seg_pos={0 0}, upload=14126]: Uploading piece... Aug 7 23:17:27.823697740 (13h37m27.413042727s) [noise=true , address=79.127.201.215:4355 , piece_id=KUJDHE66HM6IAFRND7KRYDII2FB7AA3S5K5S244JPQZUD4BYXFWA, seg_pos={0 0}, upload=14126]: Uploading piece... Aug 7 23:17:27.824090602 (13h37m27.413435589s) [noise=true , address=104.36.82.10:4093 , piece_id=VQUPLYS4X67PIYNTDH76KJ3DTBIWIH6UM3TQJ6VEREZIUJIH7A2Q, seg_pos={0 0}, upload=14126]: Uploading piece... Aug 7 23:17:27.824398111 (13h37m27.413743098s) [noise=true , address=64.49.13.170:4718 , piece_id=VXSII2V5QDMEJ5G5PJWNHXSJ6HF5NLLTGEDN5CHKKRWYHIAKHMVA, seg_pos={0 0}, upload=14126]: Uploading piece... Aug 7 23:17:27.824731088 (13h37m27.414076065s) [noise=true , address=104.36.82.43:4847 , piece_id=5Y56Z5NDMWSATFWNBYDWKBZB7SUSJ3ABAFMOY6PJ3B24SPPGJH3A, seg_pos={0 0}, upload=14126]: Uploading piece... Aug 7 23:17:27.828362365 (13h37m27.417707362s) [noise=true , address=104.36.82.61:4404 , piece_id=KGEALBUHVOZEFP7V7LPYXQG5APDPXBEWG6UG34QGOCYOHPFHEZNQ, seg_pos={0 0}, upload=14126]: Uploading piece... Aug 7 23:17:27.831994514 (13h37m27.421339501s) [noise=true , address=104.36.82.18:4370 , piece_id=HHYIBQ2JWTCPEOVRZQLKEWT7BFUZDXXZP6FYED3MPEWTVODBQUQA, seg_pos={0 0}, upload=14126]: Uploading piece... Aug 7 23:17:27.835602707 (13h37m27.424947694s) [noise=true , address=45.91.116.6:4988 , piece_id=QF7HXJOV5JYZ73FJMUUGT2JJUR3W2PY3C7CE6EQQCYGM72VDSTXQ, seg_pos={0 0}, upload=14126]: Uploading piece... Aug 7 23:17:27.837475180 (13h37m27.426820177s) [noise=true , address=104.36.82.34:4547 , piece_id=2XYJPP25OROUVGLPDMGFVMDOD4AYOFE6BRQ4OFMVEO2D73XVPNAQ, seg_pos={0 0}, upload=14126]: Uploading piece... Aug 7 23:17:27.841864818 (13h37m27.431209825s) [noise=true , address=104.36.82.62:4439 , piece_id=F4FXK4T3IKUXUGL37K3DEDNWWAUZZFEJVT7WBYDVVEZSXO6QG4OQ, seg_pos={0 0}, upload=14126]: Uploading piece... Aug 7 23:17:29.010273953 (13h37m28.59961894s) [upload=14126]: Next returned nil segment Aug 7 23:17:29.010293008 (13h37m28.599637996s) [upload=14126]: Waiting for error group managing segments... Aug 7 23:17:29.309514867 (13h37m28.898859844s) [noise=true , address=64.49.13.170:4718 , piece_id=VXSII2V5QDMEJ5G5PJWNHXSJ6HF5NLLTGEDN5CHKKRWYHIAKHMVA, seg_pos={0 0}, upload=14126]: Done uploading piece. err: <nil> Aug 7 23:17:29.779221569 (13h37m29.368566556s) [noise=true , address=79.127.201.215:4355 , piece_id=KUJDHE66HM6IAFRND7KRYDII2FB7AA3S5K5S244JPQZUD4BYXFWA, seg_pos={0 0}, upload=14126]: Done uploading piece. err: <nil> Aug 7 23:17:29.827123595 (13h37m29.416468582s) [noise=true , address=104.36.82.34:4547 , piece_id=2XYJPP25OROUVGLPDMGFVMDOD4AYOFE6BRQ4OFMVEO2D73XVPNAQ, seg_pos={0 0}, upload=14126]: Done uploading piece. err: <nil> Aug 7 23:17:29.905373281 (13h37m29.494718258s) [noise=true , address=104.36.82.39:4747 , piece_id=7OQQP63SGNWWQEYR75RRJRHZSL52NRYH3BCS65XI4SWA24BKERUQ, seg_pos={0 0}, upload=14126]: Done uploading piece. err: <nil> Aug 7 23:17:29.927717560 (13h37m29.517062537s) [noise=true , address=104.36.82.43:4847 , piece_id=5Y56Z5NDMWSATFWNBYDWKBZB7SUSJ3ABAFMOY6PJ3B24SPPGJH3A, seg_pos={0 0}, upload=14126]: Done uploading piece. err: <nil> Aug 7 23:17:30.139039260 (13h37m29.728384247s) [noise=true , address=104.36.82.46:4511 , piece_id=4A43V3IOJTGJNWLKNYB7PC5WABH5C4FB5BLC55F4FZQAU4IAHMEA, seg_pos={0 0}, upload=14126]: Done uploading piece. err: <nil> Aug 7 23:17:30.142609049 (13h37m29.731954036s) [noise=true , address=104.36.82.10:4093 , piece_id=VQUPLYS4X67PIYNTDH76KJ3DTBIWIH6UM3TQJ6VEREZIUJIH7A2Q, seg_pos={0 0}, upload=14126]: Done uploading piece. err: <nil> Aug 7 23:17:30.265344466 (13h37m29.854689463s) [noise=true , address=104.36.82.8:4983 , piece_id=AVNQRSGZB5XHG3VFYFHRLBLKQ7B4YWXGIPWDK32DYJSMFXUFTSNA, seg_pos={0 0}, upload=14126]: Done uploading piece. err: <nil> Aug 7 23:17:30.341283132 (13h37m29.930628119s) [noise=true , address=45.91.116.6:4988 , piece_id=QF7HXJOV5JYZ73FJMUUGT2JJUR3W2PY3C7CE6EQQCYGM72VDSTXQ, seg_pos={0 0}, upload=14126]: Done uploading piece. err: <nil> Aug 7 23:17:30.343588482 (13h37m29.932933459s) [noise=true , address=104.36.82.61:4404 , piece_id=KGEALBUHVOZEFP7V7LPYXQG5APDPXBEWG6UG34QGOCYOHPFHEZNQ, seg_pos={0 0}, upload=14126]: Done uploading piece. err: <nil> Aug 7 23:17:30.538382992 (13h37m30.127727969s) [noise=true , address=104.36.82.37:4666 , piece_id=WXBTFS2SS7DMVYBL5BHW7TBXH7FJODGB5JJDT6WP4LOO3LU4GNDA, seg_pos={0 0}, upload=14126]: Done uploading piece. err: <nil> Aug 7 23:17:30.631974070 (13h37m30.221319047s) [noise=true , address=104.36.82.9:4028 , piece_id=HSPPTMR3OOL5DWG25IF3LXYTSOUASUM3BFX7RKHZHCUTBP4B3GQA, seg_pos={0 0}, upload=14126]: Done uploading piece. err: <nil> Aug 7 23:17:31.962954475 (13h37m31.552299462s) [noise=true , address=104.36.82.59:4322 , piece_id=D4THBNGRCWHLJC6HETJRODVC7ZRMOWMIAHGM6GYVU7LFIJNBTOZQ, seg_pos={0 0}, upload=14126]: Done uploading piece. err: <nil> Aug 7 23:17:32.006950608 (13h37m31.596295635s) [noise=true , address=104.36.82.59:4323 , piece_id=WNEN4TRJFQ64H74WZOQGGP3V5F7Y3A7OIMORF7M7WEYOR252JSRA, seg_pos={0 0}, upload=14126]: Done uploading piece. err: <nil> Aug 7 23:17:32.006984062 (13h37m31.596329049s) [seg_pos={0 0}, upload=14126]: Segment reached optimal threshold of 14 pieces. Aug 7 23:17:32.007159788 (13h37m31.596504775s) [noise=true , address=104.36.82.18:4370 , piece_id=HHYIBQ2JWTCPEOVRZQLKEWT7BFUZDXXZP6FYED3MPEWTVODBQUQA, seg_pos={0 0}, upload=14126]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SXZN7WJvcHys9BRd4Lq8NAZ3MSgHf9ZVv7SgaiJNRosdmFVpm): protocol: expected piece hash; context canceled; EOF Aug 7 23:17:32.007279266 (13h37m31.596624253s) [noise=true , address=104.36.82.60:4343 , piece_id=UHOQFZPYXHO56TUWMTSQE5XVWFGO66ATAA3EU7TQ5KX2MIN2ZJQQ, seg_pos={0 0}, upload=14126]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16EsEMRW53sjKVag2WpSDGLhELyE2bSXyW4aAurCvDFKRNkXuf): protocol: write tcp 188.166.225.170:53332->104.36.82.60:4343: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 23:17:32.007291060 (13h37m31.596636047s) [noise=true , address=104.36.82.16:4310 , piece_id=OAD3ECB3B5RXQB362MCLIWSQC4PKS3QUHWVMIMK4XPGH63JKO3LQ, seg_pos={0 0}, upload=14126]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1g14evoaeUnhJThf5QKbQT1Gd8WW9MHvaYQhQ1GRgwPfDLoC5R): protocol: write tcp 188.166.225.170:46238->104.36.82.16:4310: use of closed network connection; write tcp 188.166.225.170:46238->104.36.82.16:4310: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46238->104.36.82.16:4310: use of closed network connection Aug 7 23:17:32.007409626 (13h37m31.596754613s) [noise=true , address=104.36.82.62:4439 , piece_id=F4FXK4T3IKUXUGL37K3DEDNWWAUZZFEJVT7WBYDVVEZSXO6QG4OQ, seg_pos={0 0}, upload=14126]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HRW3L15ka62qA7mWmEJaJ2ydTA5f8sVY1iL3Fn1R7ubisPF3W): protocol: read tcp 188.166.225.170:33134->104.36.82.62:4439: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 23:17:32.007441958 (13h37m31.596786945s) [seg_pos={0 0}, upload=14126]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 23:17:32.007533564 (13h37m31.596878551s) [upload=14126]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 23:17:32.007541158 (13h37m31.596886145s) [upload=14126]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:17:39.583862369 (13h37m39.173207346s) [upload=14126]: Done uploading segments... Aug 7 23:17:39.583880514 (13h37m39.173225501s) [upload=14126]: Upload finished. err: <nil> Aug 7 23:17:57.117137796 (13h37m56.706482783s) [upload=14133]: Starting upload Aug 7 23:17:57.117195166 (13h37m56.706540153s) [upload=14133]: Done starting upload Aug 7 23:17:57.117244931 (13h37m56.706589918s) [upload=14133]: Uploading segments... Aug 7 23:17:57.117378968 (13h37m56.706723955s) [upload=14133]: Got next segment. Inline: false Aug 7 23:17:57.117404277 (13h37m56.706749264s) [upload=14133]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:17:57.117408454 (13h37m56.706753441s) [upload=14133]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:18:56.470635719 (13h38m56.059980716s) [upload=14133]: Next segment err: context deadline exceeded Aug 7 23:18:56.470717675 (13h38m56.060062703s) [upload=14133]: Done uploading segments... Aug 7 23:18:56.470739287 (13h38m56.060084304s) [upload=14133]: Upload finished. err: context deadline exceeded Duration: 8.759602224s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 5.765618s FAIL (2025-08-07 23:38:45 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.765641805s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 3 out of 60 is failed (threshold=3) [......X....................X.....................X..........] duration: 6.065511435s FAIL (2025-08-07 23:38:44 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/4M") ======LOGS====== 916s) [upload=14346]: Starting upload Aug 7 23:30:30.723725030 (13h50m30.313070017s) [upload=14346]: Done starting upload Aug 7 23:30:30.723796056 (13h50m30.313141043s) [upload=14346]: Uploading segments... Aug 7 23:30:30.723935803 (13h50m30.31328079s) [upload=14346]: Got next segment. Inline: false Aug 7 23:30:30.723960240 (13h50m30.313305227s) [upload=14346]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:30:30.723964218 (13h50m30.313309205s) [upload=14346]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:30:31.181251948 (13h50m30.770596926s) [seg_pos={0 0}, upload=14346]: Begin upload segment... Aug 7 23:30:31.181299649 (13h50m30.770644636s) [seg_pos={0 0}, upload=14346]: Done begin upload segment. Aug 7 23:30:31.181318165 (13h50m30.770663152s) [noise=true , address=104.36.82.52:4086 , piece_id=JLEUROLLGI7X6MUOVFMEWIFCQNTYJXSZW2JMHGEAI66XREHK5LLQ, seg_pos={0 0}, upload=14346]: Uploading piece... Aug 7 23:30:31.181329696 (13h50m30.770674693s) [noise=true , address=104.36.82.54:4172 , piece_id=2QZVEPRDWBX4KY62QMLTPKNZ7QA2K3YDWJETONKWYOGLMPUVSVWA, seg_pos={0 0}, upload=14346]: Uploading piece... Aug 7 23:30:31.181716817 (13h50m30.771061804s) [noise=true , address=77.90.31.17:4797 , piece_id=P6M6EN2C7M6FVKVWV4LE6JEHSZTB24YEACVG7TMSNBQFLD6NLKDQ, seg_pos={0 0}, upload=14346]: Uploading piece... Aug 7 23:30:31.181721437 (13h50m30.771066424s) [noise=true , address=104.36.82.46:4489 , piece_id=KYHATGRQXZ6XXQFUSTYUYBI26BTC7VSN5N6OM766B3QOSWN747IA, seg_pos={0 0}, upload=14346]: Uploading piece... Aug 7 23:30:31.182063421 (13h50m30.771408408s) [noise=true , address=104.36.82.22:4534 , piece_id=5BROJMGAIDCZNPDEQH32QVVYD2JP4U7IN5DCDUP7LBMU2E6Z36CA, seg_pos={0 0}, upload=14346]: Uploading piece... Aug 7 23:30:31.182318108 (13h50m30.771663085s) [noise=true , address=104.36.82.58:4267 , piece_id=QVRI7PRWV4JSPRWQIJD6M7UJAWDBWN2VIVGOYUJ7NGO77SHZ4UJQ, seg_pos={0 0}, upload=14346]: Uploading piece... Aug 7 23:30:31.182483654 (13h50m30.771828641s) [noise=true , address=64.49.13.138:4489 , piece_id=GC6CUUPTPHVDNMD2ZDX5EUJSEUNL6I6OM3MYQNM25F2G7X64YQLQ, seg_pos={0 0}, upload=14346]: Uploading piece... Aug 7 23:30:31.182639132 (13h50m30.771984119s) [noise=true , address=45.91.116.11:4149 , piece_id=JGSWIYMTFANVCD326G2TRN24X26XYQQSJT6HFWBQAHJLQ2SLBA4A, seg_pos={0 0}, upload=14346]: Uploading piece... Aug 7 23:30:31.182934928 (13h50m30.772279905s) [noise=true , address=104.36.82.62:4428 , piece_id=P7EOR2HLYCVOBQBDUZ3OYNVO4RV7WRXLHVREWKIKE7SHYBV3VRVQ, seg_pos={0 0}, upload=14346]: Uploading piece... Aug 7 23:30:31.183037023 (13h50m30.77238201s) [noise=true , address=64.49.13.155:4917 , piece_id=GLY2EUU2PUD7JB4V7PAKNNFK6ZDWGN2DKS5YHTXUNAJXWGRYRTJA, seg_pos={0 0}, upload=14346]: Uploading piece... Aug 7 23:30:31.183158877 (13h50m30.772503864s) [noise=true , address=104.36.82.47:4534 , piece_id=PJCWOCQ7UKEHHSERR6C5RQC4Y3TBZRHZF6KPMZIMTAFS6E4525EA, seg_pos={0 0}, upload=14346]: Uploading piece... Aug 7 23:30:31.183353329 (13h50m30.772698316s) [noise=true , address=104.36.82.44:4883 , piece_id=7GULYV2REOYLHXZXRSPIQM46DRT4YKT7FXC3YCHAP7W7332RRTRA, seg_pos={0 0}, upload=14346]: Uploading piece... Aug 7 23:30:31.183386993 (13h50m30.77273198s) [noise=true , address=104.36.82.52:4097 , piece_id=REDOEPTMV6TQIM5HWQ2DQBPENXJFHVZV5LGM244DRAMGSV6HSNRQ, seg_pos={0 0}, upload=14346]: Uploading piece... Aug 7 23:30:31.183591444 (13h50m30.772936431s) [noise=true , address=104.36.82.60:4353 , piece_id=3TWE6IJ3OCY5CMMRHZ5DG2WBOHLWF3MEESPOGUMU3YA5NSGW3AKA, seg_pos={0 0}, upload=14346]: Uploading piece... Aug 7 23:30:31.183849538 (13h50m30.773194525s) [noise=true , address=64.49.13.27:4960 , piece_id=RCNXI4GI75JF5TWSSGMTNPWOTU4C3TADGATSBJQKJBRLVELNSGUA, seg_pos={0 0}, upload=14346]: Uploading piece... Aug 7 23:30:31.184184719 (13h50m30.773529706s) [noise=true , address=104.36.82.61:4392 , piece_id=OSMRSZL5LDLYLPM3KN5YVIMBU44HS72BHZOAB7W5SS5Z4DY4F4JA, seg_pos={0 0}, upload=14346]: Uploading piece... Aug 7 23:30:31.184422474 (13h50m30.773767461s) [noise=true , address=104.36.82.8:4995 , piece_id=NS2X5RBAQQWBIMO2U4XYCYA37GVF6OBAO3GZSTEPJZOXRMPMS62A, seg_pos={0 0}, upload=14346]: Uploading piece... Aug 7 23:30:31.184480636 (13h50m30.773825623s) [noise=true , address=104.36.82.45:4910 , piece_id=QEYOF4IB42OVZTWGW3YD7ZUYX2KZ6KKOUXFUHQQYOW4QY3JUGTDQ, seg_pos={0 0}, upload=14346]: Uploading piece... Aug 7 23:30:32.295332666 (13h50m31.884677653s) [upload=14346]: Next returned nil segment Aug 7 23:30:32.295347043 (13h50m31.88469204s) [upload=14346]: Waiting for error group managing segments... Aug 7 23:30:32.576560505 (13h50m32.165905492s) [noise=true , address=64.49.13.155:4917 , piece_id=GLY2EUU2PUD7JB4V7PAKNNFK6ZDWGN2DKS5YHTXUNAJXWGRYRTJA, seg_pos={0 0}, upload=14346]: Done uploading piece. err: <nil> Aug 7 23:30:32.584528981 (13h50m32.173873958s) [noise=true , address=64.49.13.138:4489 , piece_id=GC6CUUPTPHVDNMD2ZDX5EUJSEUNL6I6OM3MYQNM25F2G7X64YQLQ, seg_pos={0 0}, upload=14346]: Done uploading piece. err: <nil> Aug 7 23:30:32.647385961 (13h50m32.236730948s) [noise=true , address=64.49.13.27:4960 , piece_id=RCNXI4GI75JF5TWSSGMTNPWOTU4C3TADGATSBJQKJBRLVELNSGUA, seg_pos={0 0}, upload=14346]: Done uploading piece. err: <nil> Aug 7 23:30:33.094757008 (13h50m32.684101995s) [noise=true , address=104.36.82.52:4097 , piece_id=REDOEPTMV6TQIM5HWQ2DQBPENXJFHVZV5LGM244DRAMGSV6HSNRQ, seg_pos={0 0}, upload=14346]: Done uploading piece. err: <nil> Aug 7 23:30:33.167659426 (13h50m32.757004403s) [noise=true , address=104.36.82.45:4910 , piece_id=QEYOF4IB42OVZTWGW3YD7ZUYX2KZ6KKOUXFUHQQYOW4QY3JUGTDQ, seg_pos={0 0}, upload=14346]: Done uploading piece. err: <nil> Aug 7 23:30:33.198379320 (13h50m32.787724307s) [noise=true , address=104.36.82.54:4172 , piece_id=2QZVEPRDWBX4KY62QMLTPKNZ7QA2K3YDWJETONKWYOGLMPUVSVWA, seg_pos={0 0}, upload=14346]: Done uploading piece. err: <nil> Aug 7 23:30:33.235504739 (13h50m32.824849726s) [noise=true , address=104.36.82.58:4267 , piece_id=QVRI7PRWV4JSPRWQIJD6M7UJAWDBWN2VIVGOYUJ7NGO77SHZ4UJQ, seg_pos={0 0}, upload=14346]: Done uploading piece. err: <nil> Aug 7 23:30:33.264110248 (13h50m32.853455225s) [noise=true , address=104.36.82.52:4086 , piece_id=JLEUROLLGI7X6MUOVFMEWIFCQNTYJXSZW2JMHGEAI66XREHK5LLQ, seg_pos={0 0}, upload=14346]: Done uploading piece. err: <nil> Aug 7 23:30:33.327068230 (13h50m32.916413217s) [noise=true , address=104.36.82.8:4995 , piece_id=NS2X5RBAQQWBIMO2U4XYCYA37GVF6OBAO3GZSTEPJZOXRMPMS62A, seg_pos={0 0}, upload=14346]: Done uploading piece. err: <nil> Aug 7 23:30:33.366778193 (13h50m32.956123181s) [noise=true , address=104.36.82.60:4353 , piece_id=3TWE6IJ3OCY5CMMRHZ5DG2WBOHLWF3MEESPOGUMU3YA5NSGW3AKA, seg_pos={0 0}, upload=14346]: Done uploading piece. err: <nil> Aug 7 23:30:33.542767028 (13h50m33.132112016s) [noise=true , address=104.36.82.46:4489 , piece_id=KYHATGRQXZ6XXQFUSTYUYBI26BTC7VSN5N6OM766B3QOSWN747IA, seg_pos={0 0}, upload=14346]: Done uploading piece. err: <nil> Aug 7 23:30:33.575265094 (13h50m33.164610081s) [noise=true , address=104.36.82.22:4534 , piece_id=5BROJMGAIDCZNPDEQH32QVVYD2JP4U7IN5DCDUP7LBMU2E6Z36CA, seg_pos={0 0}, upload=14346]: Done uploading piece. err: <nil> Aug 7 23:30:33.672325632 (13h50m33.261670619s) [noise=true , address=77.90.31.17:4797 , piece_id=P6M6EN2C7M6FVKVWV4LE6JEHSZTB24YEACVG7TMSNBQFLD6NLKDQ, seg_pos={0 0}, upload=14346]: Done uploading piece. err: <nil> Aug 7 23:30:33.750650465 (13h50m33.339995452s) [noise=true , address=45.91.116.11:4149 , piece_id=JGSWIYMTFANVCD326G2TRN24X26XYQQSJT6HFWBQAHJLQ2SLBA4A, seg_pos={0 0}, upload=14346]: Done uploading piece. err: <nil> Aug 7 23:30:33.750672889 (13h50m33.340017876s) [seg_pos={0 0}, upload=14346]: Segment reached optimal threshold of 14 pieces. Aug 7 23:30:33.750785674 (13h50m33.340130661s) [noise=true , address=104.36.82.47:4534 , piece_id=PJCWOCQ7UKEHHSERR6C5RQC4Y3TBZRHZF6KPMZIMTAFS6E4525EA, seg_pos={0 0}, upload=14346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VDrNGcCnaKEmG5NN4nex1JoZAgdCEfX5CdsKbxytfLfKzPbr): protocol: expected piece hash; context canceled; EOF Aug 7 23:30:33.750885715 (13h50m33.340230702s) [noise=true , address=104.36.82.61:4392 , piece_id=OSMRSZL5LDLYLPM3KN5YVIMBU44HS72BHZOAB7W5SS5Z4DY4F4JA, seg_pos={0 0}, upload=14346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14FTsLbtv1vp43aRdDZZBY2Gf6epidM1A8HSyQKVokXGFFZGZk): protocol: expected piece hash; context canceled; EOF Aug 7 23:30:33.750979806 (13h50m33.340324793s) [noise=true , address=104.36.82.62:4428 , piece_id=P7EOR2HLYCVOBQBDUZ3OYNVO4RV7WRXLHVREWKIKE7SHYBV3VRVQ, seg_pos={0 0}, upload=14346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Ui5Ye48cKNGFPakMKsjdBi13Xo7bzrmXPNEmRBLKwpMtVAA3R): protocol: rpc: dial tcp 104.36.82.62:4428: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 23:30:33.751073955 (13h50m33.340418942s) [noise=true , address=104.36.82.44:4883 , piece_id=7GULYV2REOYLHXZXRSPIQM46DRT4YKT7FXC3YCHAP7W7332RRTRA, seg_pos={0 0}, upload=14346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cx1avL4umDm2pzJS1rNZqeDZjZMWnMtNsyNTdMdkxhpa1ByUN): protocol: context canceled; context canceled Aug 7 23:30:33.751119673 (13h50m33.34046466s) [seg_pos={0 0}, upload=14346]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 23:30:33.751224474 (13h50m33.340569471s) [upload=14346]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 23:30:33.751230776 (13h50m33.340575763s) [upload=14346]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:30:34.056540019 (13h50m33.645885006s) [upload=14346]: Done uploading segments... Aug 7 23:30:34.056559826 (13h50m33.645904823s) [upload=14346]: Upload finished. err: <nil> Duration: 6.065554016s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [.......................X..X.XXXX.X..........................] duration: 6.57025591s FAIL (2025-08-07 23:38:42 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== 8]: Upload finished. err: context deadline exceeded Aug 7 23:18:32.336054087 (13h38m31.925399094s) [upload=14140]: Starting upload Aug 7 23:18:32.336180319 (13h38m31.925525306s) [upload=14140]: Done starting upload Aug 7 23:18:32.336261133 (13h38m31.92560616s) [upload=14140]: Uploading segments... Aug 7 23:18:32.336590001 (13h38m31.925935029s) [upload=14140]: Got next segment. Inline: false Aug 7 23:18:32.336650628 (13h38m31.925995655s) [upload=14140]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:18:32.336658754 (13h38m31.926003761s) [upload=14140]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:18:32.571732430 (13h38m32.161077417s) [seg_pos={0 0}, upload=14140]: Begin upload segment... Aug 7 23:18:32.571774531 (13h38m32.161119528s) [seg_pos={0 0}, upload=14140]: Done begin upload segment. Aug 7 23:18:32.571805400 (13h38m32.161150417s) [noise=true , address=77.90.31.12:4589 , piece_id=ZMKTDC4M5JHWVARXQDQAKRFSXM3OQQZ2IGQASJCK2OKYX3IQXHVA, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.571844173 (13h38m32.161189161s) [noise=true , address=104.36.82.9:4007 , piece_id=VZLD7RU4SK5MAQE22D7OBVT6XOQAN75UNDMT7JNPCRZNRKFX36XQ, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.572218380 (13h38m32.161563367s) [noise=true , address=104.36.82.10:4077 , piece_id=WYGEC4FYUPAR45MZIWLZQ2ASF5SUPECS4YI3UTRRHEVAKUEYCGYA, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.572289677 (13h38m32.161634654s) [noise=true , address=77.90.31.17:4770 , piece_id=ISA43NF4AKPK2SGYDPKT6ZIAD5WJ27A6GT6JZA2GHQUCAQBRRAFA, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.572494809 (13h38m32.161839796s) [noise=true , address=104.36.82.7:4911 , piece_id=FAMSQZJMALQIEAG5BVR5OV5WHEHN432Y4DZD3DNPC5CKP7X7GMQA, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.572588257 (13h38m32.161933244s) [noise=true , address=77.90.31.15:4697 , piece_id=6RYZJ22TCTJFRBMQV7M545VHB475CHDV2DBKCGO3MJP65VEMJ5FQ, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.572772450 (13h38m32.162117427s) [noise=true , address=45.91.116.12:4225 , piece_id=22GAFTM5SITGTVJEELA3BJP7W2QZW7BVYJGJMKJOLE2RF2GZ5CGQ, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.572865959 (13h38m32.162210946s) [noise=true , address=45.91.116.12:4227 , piece_id=T3JVU7VMSYBTYXBVGRD7LSNIG3PWF3QDVDNDQ24JQOSV7ZFIODNA, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.573029050 (13h38m32.162374037s) [noise=true , address=77.90.31.19:4842 , piece_id=3ZRHNMBPQUJH5SZZ75UJWLYIYZ5YI7CC5OMIJJFM7WVKZWJX6R3A, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.573227882 (13h38m32.162572869s) [noise=true , address=104.36.82.4:4804 , piece_id=LTSBCPH74YSWH7IHN6MH4NWTBJ5FQ7534VERPCY5NIDV24KYPXMQ, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.573293797 (13h38m32.162638774s) [noise=true , address=104.36.82.15:4259 , piece_id=GFCTNTXSY4WVSTPAD3AZIYCHE4O5EN24WN25VUNCRNWCSUWCYBMA, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.573504962 (13h38m32.162849949s) [noise=true , address=77.90.31.16:4734 , piece_id=OJPJZ5RF7QY4KNFBZ3MAVTA63HMC4JRG3HQLBN7SVROVKFQZY7BQ, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.573527936 (13h38m32.162872923s) [noise=true , address=45.91.116.7:4009 , piece_id=VKYMGNJEGFLB5M3L3WLD37C5ZILQGLRJROSQ4ZEIBN46CQSY5JZQ, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.574078970 (13h38m32.163423947s) [noise=true , address=45.91.116.11:4119 , piece_id=UXO65SB4OZ57BCKOWGE2VDV7KCSKCGE7I7K4RTGRYJ7VU5GKFPHQ, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.574216453 (13h38m32.16356143s) [noise=true , address=104.36.82.12:4149 , piece_id=2K5BCSGU4UN43HF7VHG3MBNMQEGSIZPPHTDHDFI3XPY5DDG6NKOA, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.576753347 (13h38m32.166098334s) [noise=true , address=104.36.82.8:4971 , piece_id=DNQVVV7QT2A2NFMIRURJ5IZRFPYZVX2NADRTX2P2YORA2CKMWY6Q, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:34.327895253 (13h38m33.91724024s) [upload=14140]: Next returned nil segment Aug 7 23:18:34.327912295 (13h38m33.917257282s) [upload=14140]: Waiting for error group managing segments... Aug 7 23:18:34.716828436 (13h38m34.306173413s) [noise=true , address=104.36.82.7:4911 , piece_id=FAMSQZJMALQIEAG5BVR5OV5WHEHN432Y4DZD3DNPC5CKP7X7GMQA, seg_pos={0 0}, upload=14140]: Done uploading piece. err: <nil> Aug 7 23:18:34.729509878 (13h38m34.318854865s) [noise=true , address=104.36.82.12:4149 , piece_id=2K5BCSGU4UN43HF7VHG3MBNMQEGSIZPPHTDHDFI3XPY5DDG6NKOA, seg_pos={0 0}, upload=14140]: Done uploading piece. err: <nil> Aug 7 23:18:34.738858565 (13h38m34.328203592s) [noise=true , address=104.36.82.4:4804 , piece_id=LTSBCPH74YSWH7IHN6MH4NWTBJ5FQ7534VERPCY5NIDV24KYPXMQ, seg_pos={0 0}, upload=14140]: Done uploading piece. err: <nil> Aug 7 23:18:34.783948702 (13h38m34.373293679s) [noise=true , address=77.90.31.19:4842 , piece_id=3ZRHNMBPQUJH5SZZ75UJWLYIYZ5YI7CC5OMIJJFM7WVKZWJX6R3A, seg_pos={0 0}, upload=14140]: Done uploading piece. err: <nil> Aug 7 23:18:34.824404112 (13h38m34.413749099s) [noise=true , address=104.36.82.15:4259 , piece_id=GFCTNTXSY4WVSTPAD3AZIYCHE4O5EN24WN25VUNCRNWCSUWCYBMA, seg_pos={0 0}, upload=14140]: Done uploading piece. err: <nil> Aug 7 23:18:34.863114634 (13h38m34.452459611s) [noise=true , address=104.36.82.9:4007 , piece_id=VZLD7RU4SK5MAQE22D7OBVT6XOQAN75UNDMT7JNPCRZNRKFX36XQ, seg_pos={0 0}, upload=14140]: Done uploading piece. err: <nil> Aug 7 23:18:34.889984711 (13h38m34.479329698s) [noise=true , address=77.90.31.16:4734 , piece_id=OJPJZ5RF7QY4KNFBZ3MAVTA63HMC4JRG3HQLBN7SVROVKFQZY7BQ, seg_pos={0 0}, upload=14140]: Done uploading piece. err: <nil> Aug 7 23:18:34.982222673 (13h38m34.57156766s) [noise=true , address=104.36.82.10:4077 , piece_id=WYGEC4FYUPAR45MZIWLZQ2ASF5SUPECS4YI3UTRRHEVAKUEYCGYA, seg_pos={0 0}, upload=14140]: Done uploading piece. err: <nil> Aug 7 23:18:34.999676977 (13h38m34.589021964s) [noise=true , address=77.90.31.17:4770 , piece_id=ISA43NF4AKPK2SGYDPKT6ZIAD5WJ27A6GT6JZA2GHQUCAQBRRAFA, seg_pos={0 0}, upload=14140]: Done uploading piece. err: <nil> Aug 7 23:18:35.002083712 (13h38m34.591428699s) [noise=true , address=45.91.116.12:4225 , piece_id=22GAFTM5SITGTVJEELA3BJP7W2QZW7BVYJGJMKJOLE2RF2GZ5CGQ, seg_pos={0 0}, upload=14140]: Done uploading piece. err: <nil> Aug 7 23:18:35.055266482 (13h38m34.644611469s) [noise=true , address=45.91.116.12:4227 , piece_id=T3JVU7VMSYBTYXBVGRD7LSNIG3PWF3QDVDNDQ24JQOSV7ZFIODNA, seg_pos={0 0}, upload=14140]: Done uploading piece. err: <nil> Aug 7 23:18:35.081593190 (13h38m34.670938177s) [noise=true , address=45.91.116.11:4119 , piece_id=UXO65SB4OZ57BCKOWGE2VDV7KCSKCGE7I7K4RTGRYJ7VU5GKFPHQ, seg_pos={0 0}, upload=14140]: Done uploading piece. err: <nil> Aug 7 23:18:35.123608726 (13h38m34.712953703s) [noise=true , address=77.90.31.12:4589 , piece_id=ZMKTDC4M5JHWVARXQDQAKRFSXM3OQQZ2IGQASJCK2OKYX3IQXHVA, seg_pos={0 0}, upload=14140]: Done uploading piece. err: <nil> Aug 7 23:18:35.123631779 (13h38m34.712976766s) [seg_pos={0 0}, upload=14140]: Segment reached optimal threshold of 13 pieces. Aug 7 23:18:35.123785944 (13h38m34.713130931s) [noise=true , address=45.91.116.7:4009 , piece_id=VKYMGNJEGFLB5M3L3WLD37C5ZILQGLRJROSQ4ZEIBN46CQSY5JZQ, seg_pos={0 0}, upload=14140]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF Aug 7 23:18:35.123945530 (13h38m34.713290517s) [noise=true , address=77.90.31.15:4697 , piece_id=6RYZJ22TCTJFRBMQV7M545VHB475CHDV2DBKCGO3MJP65VEMJ5FQ, seg_pos={0 0}, upload=14140]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: expected piece hash; context canceled; EOF Aug 7 23:18:35.123972431 (13h38m34.713317418s) [noise=true , address=104.36.82.8:4971 , piece_id=DNQVVV7QT2A2NFMIRURJ5IZRFPYZVX2NADRTX2P2YORA2CKMWY6Q, seg_pos={0 0}, upload=14140]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo): protocol: read tcp 188.166.225.170:44698->104.36.82.8:4971: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 23:18:35.123986628 (13h38m34.713331615s) [seg_pos={0 0}, upload=14140]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 7 23:18:35.124083804 (13h38m34.713428791s) [upload=14140]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 23:18:35.124087611 (13h38m34.713432599s) [upload=14140]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:18:35.414558916 (13h38m35.003903903s) [upload=14140]: Done uploading segments... Aug 7 23:18:35.414577682 (13h38m35.00392267s) [upload=14140]: Upload finished. err: <nil> Aug 7 23:18:58.905554387 (13h38m58.494899394s) [upload=14146]: Starting upload Aug 7 23:18:58.905620613 (13h38m58.4949656s) [upload=14146]: Done starting upload Aug 7 23:18:58.905683373 (13h38m58.49502836s) [upload=14146]: Uploading segments... Aug 7 23:18:58.905783355 (13h38m58.495128342s) [upload=14146]: Got next segment. Inline: false Aug 7 23:18:58.905832459 (13h38m58.495177446s) [upload=14146]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:18:58.905836627 (13h38m58.495181614s) [upload=14146]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:19:28.246792926 (13h39m27.836137913s) [upload=14146]: Next segment err: context deadline exceeded Aug 7 23:19:28.246867379 (13h39m27.836212366s) [upload=14146]: Done uploading segments... Aug 7 23:19:28.246884292 (13h39m27.836229269s) [upload=14146]: Upload finished. err: context deadline exceeded Duration: 6.570279275s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 8 out of 60 is failed (threshold=3) [..............................XXXXXX.X.X....................] duration: 5.491449864s FAIL (2025-08-07 23:38:36 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.491470815s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [.......................X......X.............................] duration: 1.918240921s FAIL (2025-08-07 23:38:34 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== 30.892541890 (13h31m30.481886877s) [upload=14032]: Upload finished. err: <nil> Aug 7 23:11:48.390915893 (13h31m47.98026088s) [upload=14038]: Starting upload Aug 7 23:11:48.390965198 (13h31m47.980310185s) [upload=14038]: Done starting upload Aug 7 23:11:48.391014242 (13h31m47.980359229s) [upload=14038]: Uploading segments... Aug 7 23:11:48.391054007 (13h31m47.980399004s) [upload=14038]: Got next segment. Inline: true Aug 7 23:11:48.391057494 (13h31m47.980402471s) [upload=14038]: Waiting for error group managing segments... Aug 7 23:11:48.391089254 (13h31m47.980434231s) [upload=14038]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:11:48.391093213 (13h31m47.98043819s) [upload=14038]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:11:48.391095457 (13h31m47.980440444s) [upload=14038]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:11:48.701950724 (13h31m48.291295711s) [upload=14038]: Done uploading segments... Aug 7 23:11:48.701969449 (13h31m48.291314436s) [upload=14038]: Upload finished. err: <nil> Aug 7 23:12:06.430245154 (13h32m6.019590141s) [upload=14048]: Starting upload Aug 7 23:12:06.430293076 (13h32m6.019638063s) [upload=14048]: Done starting upload Aug 7 23:12:06.430361757 (13h32m6.019706745s) [upload=14048]: Uploading segments... Aug 7 23:12:06.430459223 (13h32m6.01980421s) [upload=14048]: Got next segment. Inline: true Aug 7 23:12:06.430464543 (13h32m6.01980953s) [upload=14048]: Waiting for error group managing segments... Aug 7 23:12:06.430527555 (13h32m6.019872542s) [upload=14048]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:12:06.430533796 (13h32m6.019878783s) [upload=14048]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:12:06.430536692 (13h32m6.019881679s) [upload=14048]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:12:06.729952062 (13h32m6.319297049s) [upload=14048]: Done uploading segments... Aug 7 23:12:06.729968844 (13h32m6.319313832s) [upload=14048]: Upload finished. err: <nil> Aug 7 23:12:37.551701338 (13h32m37.141046325s) [upload=14055]: Starting upload Aug 7 23:12:37.551746985 (13h32m37.141091972s) [upload=14055]: Done starting upload Aug 7 23:12:37.551807301 (13h32m37.141152288s) [upload=14055]: Uploading segments... Aug 7 23:12:37.551914826 (13h32m37.141259833s) [upload=14055]: Got next segment. Inline: true Aug 7 23:12:37.551921779 (13h32m37.141266766s) [upload=14055]: Waiting for error group managing segments... Aug 7 23:12:37.551977145 (13h32m37.141322122s) [upload=14055]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:12:37.551984059 (13h32m37.141329086s) [upload=14055]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:12:37.551988918 (13h32m37.141333936s) [upload=14055]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:12:37.767068918 (13h32m37.356413905s) [upload=14055]: Done uploading segments... Aug 7 23:12:37.767104365 (13h32m37.356449342s) [upload=14055]: Upload finished. err: metaclient: Too Many Requests Aug 7 23:13:47.522489627 (13h33m47.111834614s) [upload=14072]: Starting upload Aug 7 23:13:47.522559681 (13h33m47.111904668s) [upload=14072]: Done starting upload Aug 7 23:13:47.522629674 (13h33m47.111974661s) [upload=14072]: Uploading segments... Aug 7 23:13:47.522726771 (13h33m47.112071778s) [upload=14072]: Got next segment. Inline: true Aug 7 23:13:47.522735617 (13h33m47.112080614s) [upload=14072]: Waiting for error group managing segments... Aug 7 23:13:47.522790582 (13h33m47.112135569s) [upload=14072]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:13:47.522796493 (13h33m47.112141481s) [upload=14072]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:13:47.522799320 (13h33m47.112144307s) [upload=14072]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:13:47.796144342 (13h33m47.385489329s) [upload=14072]: Done uploading segments... Aug 7 23:13:47.796162656 (13h33m47.385507643s) [upload=14072]: Upload finished. err: <nil> Aug 7 23:14:22.239921861 (13h34m21.829266848s) [upload=14081]: Starting upload Aug 7 23:14:22.239973791 (13h34m21.829318778s) [upload=14081]: Done starting upload Aug 7 23:14:22.240087399 (13h34m21.829432396s) [upload=14081]: Uploading segments... Aug 7 23:14:22.240187159 (13h34m21.829532156s) [upload=14081]: Got next segment. Inline: true Aug 7 23:14:22.240194182 (13h34m21.829539179s) [upload=14081]: Waiting for error group managing segments... Aug 7 23:14:22.240255971 (13h34m21.829600958s) [upload=14081]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:14:22.240261311 (13h34m21.829606288s) [upload=14081]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:14:22.240263816 (13h34m21.829608803s) [upload=14081]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:14:22.537482123 (13h34m22.12682711s) [upload=14081]: Done uploading segments... Aug 7 23:14:22.537500677 (13h34m22.126845664s) [upload=14081]: Upload finished. err: <nil> Aug 7 23:14:56.968922831 (13h34m56.558267818s) [upload=14090]: Starting upload Aug 7 23:14:56.968974069 (13h34m56.558319056s) [upload=14090]: Done starting upload Aug 7 23:14:56.969028383 (13h34m56.55837337s) [upload=14090]: Uploading segments... Aug 7 23:14:56.969059602 (13h34m56.55840459s) [upload=14090]: Got next segment. Inline: true Aug 7 23:14:56.969063500 (13h34m56.558408487s) [upload=14090]: Waiting for error group managing segments... Aug 7 23:14:56.969102906 (13h34m56.558447893s) [upload=14090]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:14:56.969107034 (13h34m56.558452021s) [upload=14090]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:14:56.969109899 (13h34m56.558454886s) [upload=14090]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:14:57.268863065 (13h34m56.858208052s) [upload=14090]: Done uploading segments... Aug 7 23:14:57.268882201 (13h34m56.858227188s) [upload=14090]: Upload finished. err: <nil> Aug 7 23:16:09.138990771 (13h36m8.728335758s) [upload=14107]: Starting upload Aug 7 23:16:09.139048231 (13h36m8.728393208s) [upload=14107]: Done starting upload Aug 7 23:16:09.139094229 (13h36m8.728439226s) [upload=14107]: Uploading segments... Aug 7 23:16:09.139131069 (13h36m8.728476056s) [upload=14107]: Got next segment. Inline: true Aug 7 23:16:09.139135007 (13h36m8.728479994s) [upload=14107]: Waiting for error group managing segments... Aug 7 23:16:09.139167859 (13h36m8.728512846s) [upload=14107]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:16:09.139171937 (13h36m8.728516924s) [upload=14107]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:16:09.139199730 (13h36m8.728544707s) [upload=14107]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:16:09.409287150 (13h36m8.998632137s) [upload=14107]: Done uploading segments... Aug 7 23:16:09.409303952 (13h36m8.998648939s) [upload=14107]: Upload finished. err: <nil> Aug 7 23:16:30.665596455 (13h36m30.254941442s) [upload=14111]: Starting upload Aug 7 23:16:30.665642382 (13h36m30.254987369s) [upload=14111]: Done starting upload Aug 7 23:16:30.665687749 (13h36m30.255032726s) [upload=14111]: Uploading segments... Aug 7 23:16:30.665725801 (13h36m30.255070788s) [upload=14111]: Got next segment. Inline: true Aug 7 23:16:30.665729409 (13h36m30.255074386s) [upload=14111]: Waiting for error group managing segments... Aug 7 23:16:30.665763554 (13h36m30.255108541s) [upload=14111]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:16:30.665768984 (13h36m30.255113971s) [upload=14111]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:16:30.665771289 (13h36m30.255116276s) [upload=14111]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:16:30.945395745 (13h36m30.534740732s) [upload=14111]: Done uploading segments... Aug 7 23:16:30.945418538 (13h36m30.534763526s) [upload=14111]: Upload finished. err: <nil> Aug 7 23:17:25.677744155 (13h37m25.267089143s) [upload=14125]: Starting upload Aug 7 23:17:25.677793930 (13h37m25.267138917s) [upload=14125]: Done starting upload Aug 7 23:17:25.677862562 (13h37m25.267207539s) [upload=14125]: Uploading segments... Aug 7 23:17:25.677896967 (13h37m25.267241954s) [upload=14125]: Got next segment. Inline: true Aug 7 23:17:25.677900704 (13h37m25.267245681s) [upload=14125]: Waiting for error group managing segments... Aug 7 23:17:25.677933046 (13h37m25.267278033s) [upload=14125]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:17:25.677936884 (13h37m25.267281871s) [upload=14125]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:17:25.677939309 (13h37m25.267284296s) [upload=14125]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:17:26.502642065 (13h37m26.091987052s) [upload=14125]: Done uploading segments... Aug 7 23:17:26.502660942 (13h37m26.092005919s) [upload=14125]: Upload finished. err: <nil> Aug 7 23:17:53.025114325 (13h37m52.614459313s) [upload=14131]: Starting upload Aug 7 23:17:53.025169641 (13h37m52.614514618s) [upload=14131]: Done starting upload Aug 7 23:17:53.025219857 (13h37m52.614564844s) [upload=14131]: Uploading segments... Aug 7 23:17:53.025267939 (13h37m52.614612936s) [upload=14131]: Got next segment. Inline: true Aug 7 23:17:53.025271916 (13h37m52.614616903s) [upload=14131]: Waiting for error group managing segments... Aug 7 23:17:53.025302495 (13h37m52.614647482s) [upload=14131]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:17:53.025306382 (13h37m52.614651369s) [upload=14131]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:17:53.025308497 (13h37m52.614653484s) [upload=14131]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:17:53.302835519 (13h37m52.892180506s) [upload=14131]: Done uploading segments... Aug 7 23:17:53.302854656 (13h37m52.892199633s) [upload=14131]: Upload finished. err: metaclient: Too Many Requests Duration: 1.918267169s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [................................XX.X........................] duration: 8.046542523s FAIL (2025-08-07 23:38:27 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== PC5WABH5C4FB5BLC55F4FZQAU4IAHMEA, seg_pos={0 0}, upload=14126]: Uploading piece... Aug 7 23:17:27.821935980 (13h37m27.411280967s) [noise=true , address=104.36.82.8:4983 , piece_id=AVNQRSGZB5XHG3VFYFHRLBLKQ7B4YWXGIPWDK32DYJSMFXUFTSNA, seg_pos={0 0}, upload=14126]: Uploading piece... Aug 7 23:17:27.822230193 (13h37m27.41157518s) [noise=true , address=104.36.82.16:4310 , piece_id=OAD3ECB3B5RXQB362MCLIWSQC4PKS3QUHWVMIMK4XPGH63JKO3LQ, seg_pos={0 0}, upload=14126]: Uploading piece... Aug 7 23:17:27.822283705 (13h37m27.411628692s) [noise=true , address=104.36.82.59:4323 , piece_id=WNEN4TRJFQ64H74WZOQGGP3V5F7Y3A7OIMORF7M7WEYOR252JSRA, seg_pos={0 0}, upload=14126]: Uploading piece... Aug 7 23:17:27.822684663 (13h37m27.41202965s) [noise=true , address=104.36.82.39:4747 , piece_id=7OQQP63SGNWWQEYR75RRJRHZSL52NRYH3BCS65XI4SWA24BKERUQ, seg_pos={0 0}, upload=14126]: Uploading piece... Aug 7 23:17:27.823034542 (13h37m27.412379529s) [noise=true , address=104.36.82.59:4322 , piece_id=D4THBNGRCWHLJC6HETJRODVC7ZRMOWMIAHGM6GYVU7LFIJNBTOZQ, seg_pos={0 0}, upload=14126]: Uploading piece... Aug 7 23:17:27.823366667 (13h37m27.412711654s) [noise=true , address=104.36.82.60:4343 , piece_id=UHOQFZPYXHO56TUWMTSQE5XVWFGO66ATAA3EU7TQ5KX2MIN2ZJQQ, seg_pos={0 0}, upload=14126]: Uploading piece... Aug 7 23:17:27.823697740 (13h37m27.413042727s) [noise=true , address=79.127.201.215:4355 , piece_id=KUJDHE66HM6IAFRND7KRYDII2FB7AA3S5K5S244JPQZUD4BYXFWA, seg_pos={0 0}, upload=14126]: Uploading piece... Aug 7 23:17:27.824090602 (13h37m27.413435589s) [noise=true , address=104.36.82.10:4093 , piece_id=VQUPLYS4X67PIYNTDH76KJ3DTBIWIH6UM3TQJ6VEREZIUJIH7A2Q, seg_pos={0 0}, upload=14126]: Uploading piece... Aug 7 23:17:27.824398111 (13h37m27.413743098s) [noise=true , address=64.49.13.170:4718 , piece_id=VXSII2V5QDMEJ5G5PJWNHXSJ6HF5NLLTGEDN5CHKKRWYHIAKHMVA, seg_pos={0 0}, upload=14126]: Uploading piece... Aug 7 23:17:27.824731088 (13h37m27.414076065s) [noise=true , address=104.36.82.43:4847 , piece_id=5Y56Z5NDMWSATFWNBYDWKBZB7SUSJ3ABAFMOY6PJ3B24SPPGJH3A, seg_pos={0 0}, upload=14126]: Uploading piece... Aug 7 23:17:27.828362365 (13h37m27.417707362s) [noise=true , address=104.36.82.61:4404 , piece_id=KGEALBUHVOZEFP7V7LPYXQG5APDPXBEWG6UG34QGOCYOHPFHEZNQ, seg_pos={0 0}, upload=14126]: Uploading piece... Aug 7 23:17:27.831994514 (13h37m27.421339501s) [noise=true , address=104.36.82.18:4370 , piece_id=HHYIBQ2JWTCPEOVRZQLKEWT7BFUZDXXZP6FYED3MPEWTVODBQUQA, seg_pos={0 0}, upload=14126]: Uploading piece... Aug 7 23:17:27.835602707 (13h37m27.424947694s) [noise=true , address=45.91.116.6:4988 , piece_id=QF7HXJOV5JYZ73FJMUUGT2JJUR3W2PY3C7CE6EQQCYGM72VDSTXQ, seg_pos={0 0}, upload=14126]: Uploading piece... Aug 7 23:17:27.837475180 (13h37m27.426820177s) [noise=true , address=104.36.82.34:4547 , piece_id=2XYJPP25OROUVGLPDMGFVMDOD4AYOFE6BRQ4OFMVEO2D73XVPNAQ, seg_pos={0 0}, upload=14126]: Uploading piece... Aug 7 23:17:27.841864818 (13h37m27.431209825s) [noise=true , address=104.36.82.62:4439 , piece_id=F4FXK4T3IKUXUGL37K3DEDNWWAUZZFEJVT7WBYDVVEZSXO6QG4OQ, seg_pos={0 0}, upload=14126]: Uploading piece... Aug 7 23:17:29.010273953 (13h37m28.59961894s) [upload=14126]: Next returned nil segment Aug 7 23:17:29.010293008 (13h37m28.599637996s) [upload=14126]: Waiting for error group managing segments... Aug 7 23:17:29.309514867 (13h37m28.898859844s) [noise=true , address=64.49.13.170:4718 , piece_id=VXSII2V5QDMEJ5G5PJWNHXSJ6HF5NLLTGEDN5CHKKRWYHIAKHMVA, seg_pos={0 0}, upload=14126]: Done uploading piece. err: <nil> Aug 7 23:17:29.779221569 (13h37m29.368566556s) [noise=true , address=79.127.201.215:4355 , piece_id=KUJDHE66HM6IAFRND7KRYDII2FB7AA3S5K5S244JPQZUD4BYXFWA, seg_pos={0 0}, upload=14126]: Done uploading piece. err: <nil> Aug 7 23:17:29.827123595 (13h37m29.416468582s) [noise=true , address=104.36.82.34:4547 , piece_id=2XYJPP25OROUVGLPDMGFVMDOD4AYOFE6BRQ4OFMVEO2D73XVPNAQ, seg_pos={0 0}, upload=14126]: Done uploading piece. err: <nil> Aug 7 23:17:29.905373281 (13h37m29.494718258s) [noise=true , address=104.36.82.39:4747 , piece_id=7OQQP63SGNWWQEYR75RRJRHZSL52NRYH3BCS65XI4SWA24BKERUQ, seg_pos={0 0}, upload=14126]: Done uploading piece. err: <nil> Aug 7 23:17:29.927717560 (13h37m29.517062537s) [noise=true , address=104.36.82.43:4847 , piece_id=5Y56Z5NDMWSATFWNBYDWKBZB7SUSJ3ABAFMOY6PJ3B24SPPGJH3A, seg_pos={0 0}, upload=14126]: Done uploading piece. err: <nil> Aug 7 23:17:30.139039260 (13h37m29.728384247s) [noise=true , address=104.36.82.46:4511 , piece_id=4A43V3IOJTGJNWLKNYB7PC5WABH5C4FB5BLC55F4FZQAU4IAHMEA, seg_pos={0 0}, upload=14126]: Done uploading piece. err: <nil> Aug 7 23:17:30.142609049 (13h37m29.731954036s) [noise=true , address=104.36.82.10:4093 , piece_id=VQUPLYS4X67PIYNTDH76KJ3DTBIWIH6UM3TQJ6VEREZIUJIH7A2Q, seg_pos={0 0}, upload=14126]: Done uploading piece. err: <nil> Aug 7 23:17:30.265344466 (13h37m29.854689463s) [noise=true , address=104.36.82.8:4983 , piece_id=AVNQRSGZB5XHG3VFYFHRLBLKQ7B4YWXGIPWDK32DYJSMFXUFTSNA, seg_pos={0 0}, upload=14126]: Done uploading piece. err: <nil> Aug 7 23:17:30.341283132 (13h37m29.930628119s) [noise=true , address=45.91.116.6:4988 , piece_id=QF7HXJOV5JYZ73FJMUUGT2JJUR3W2PY3C7CE6EQQCYGM72VDSTXQ, seg_pos={0 0}, upload=14126]: Done uploading piece. err: <nil> Aug 7 23:17:30.343588482 (13h37m29.932933459s) [noise=true , address=104.36.82.61:4404 , piece_id=KGEALBUHVOZEFP7V7LPYXQG5APDPXBEWG6UG34QGOCYOHPFHEZNQ, seg_pos={0 0}, upload=14126]: Done uploading piece. err: <nil> Aug 7 23:17:30.538382992 (13h37m30.127727969s) [noise=true , address=104.36.82.37:4666 , piece_id=WXBTFS2SS7DMVYBL5BHW7TBXH7FJODGB5JJDT6WP4LOO3LU4GNDA, seg_pos={0 0}, upload=14126]: Done uploading piece. err: <nil> Aug 7 23:17:30.631974070 (13h37m30.221319047s) [noise=true , address=104.36.82.9:4028 , piece_id=HSPPTMR3OOL5DWG25IF3LXYTSOUASUM3BFX7RKHZHCUTBP4B3GQA, seg_pos={0 0}, upload=14126]: Done uploading piece. err: <nil> Aug 7 23:17:31.962954475 (13h37m31.552299462s) [noise=true , address=104.36.82.59:4322 , piece_id=D4THBNGRCWHLJC6HETJRODVC7ZRMOWMIAHGM6GYVU7LFIJNBTOZQ, seg_pos={0 0}, upload=14126]: Done uploading piece. err: <nil> Aug 7 23:17:32.006950608 (13h37m31.596295635s) [noise=true , address=104.36.82.59:4323 , piece_id=WNEN4TRJFQ64H74WZOQGGP3V5F7Y3A7OIMORF7M7WEYOR252JSRA, seg_pos={0 0}, upload=14126]: Done uploading piece. err: <nil> Aug 7 23:17:32.006984062 (13h37m31.596329049s) [seg_pos={0 0}, upload=14126]: Segment reached optimal threshold of 14 pieces. Aug 7 23:17:32.007159788 (13h37m31.596504775s) [noise=true , address=104.36.82.18:4370 , piece_id=HHYIBQ2JWTCPEOVRZQLKEWT7BFUZDXXZP6FYED3MPEWTVODBQUQA, seg_pos={0 0}, upload=14126]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SXZN7WJvcHys9BRd4Lq8NAZ3MSgHf9ZVv7SgaiJNRosdmFVpm): protocol: expected piece hash; context canceled; EOF Aug 7 23:17:32.007279266 (13h37m31.596624253s) [noise=true , address=104.36.82.60:4343 , piece_id=UHOQFZPYXHO56TUWMTSQE5XVWFGO66ATAA3EU7TQ5KX2MIN2ZJQQ, seg_pos={0 0}, upload=14126]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16EsEMRW53sjKVag2WpSDGLhELyE2bSXyW4aAurCvDFKRNkXuf): protocol: write tcp 188.166.225.170:53332->104.36.82.60:4343: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 23:17:32.007291060 (13h37m31.596636047s) [noise=true , address=104.36.82.16:4310 , piece_id=OAD3ECB3B5RXQB362MCLIWSQC4PKS3QUHWVMIMK4XPGH63JKO3LQ, seg_pos={0 0}, upload=14126]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1g14evoaeUnhJThf5QKbQT1Gd8WW9MHvaYQhQ1GRgwPfDLoC5R): protocol: write tcp 188.166.225.170:46238->104.36.82.16:4310: use of closed network connection; write tcp 188.166.225.170:46238->104.36.82.16:4310: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46238->104.36.82.16:4310: use of closed network connection Aug 7 23:17:32.007409626 (13h37m31.596754613s) [noise=true , address=104.36.82.62:4439 , piece_id=F4FXK4T3IKUXUGL37K3DEDNWWAUZZFEJVT7WBYDVVEZSXO6QG4OQ, seg_pos={0 0}, upload=14126]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HRW3L15ka62qA7mWmEJaJ2ydTA5f8sVY1iL3Fn1R7ubisPF3W): protocol: read tcp 188.166.225.170:33134->104.36.82.62:4439: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 23:17:32.007441958 (13h37m31.596786945s) [seg_pos={0 0}, upload=14126]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 23:17:32.007533564 (13h37m31.596878551s) [upload=14126]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 23:17:32.007541158 (13h37m31.596886145s) [upload=14126]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:17:39.583862369 (13h37m39.173207346s) [upload=14126]: Done uploading segments... Aug 7 23:17:39.583880514 (13h37m39.173225501s) [upload=14126]: Upload finished. err: <nil> Aug 7 23:17:57.117137796 (13h37m56.706482783s) [upload=14133]: Starting upload Aug 7 23:17:57.117195166 (13h37m56.706540153s) [upload=14133]: Done starting upload Aug 7 23:17:57.117244931 (13h37m56.706589918s) [upload=14133]: Uploading segments... Aug 7 23:17:57.117378968 (13h37m56.706723955s) [upload=14133]: Got next segment. Inline: false Aug 7 23:17:57.117404277 (13h37m56.706749264s) [upload=14133]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:17:57.117408454 (13h37m56.706753441s) [upload=14133]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:18:56.470635719 (13h38m56.059980716s) [upload=14133]: Next segment err: context deadline exceeded Aug 7 23:18:56.470717675 (13h38m56.060062703s) [upload=14133]: Done uploading segments... Aug 7 23:18:56.470739287 (13h38m56.060084304s) [upload=14133]: Upload finished. err: context deadline exceeded Duration: 8.046570507s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 4 out of 60 is failed (threshold=3) [..........................X...X..X.X........................] duration: 1.873538112s FAIL (2025-08-07 23:38:27 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 1.873564945s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 4 out of 60 is failed (threshold=3) [............................XXXX............................] duration: 1.888870829s FAIL (2025-08-07 23:38:25 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== .175884539 (13h29m53.765229526s) [upload=14007]: Starting upload Aug 7 23:09:54.175937640 (13h29m53.765282618s) [upload=14007]: Done starting upload Aug 7 23:09:54.175979842 (13h29m53.765324829s) [upload=14007]: Uploading segments... Aug 7 23:09:54.176041389 (13h29m53.765386376s) [upload=14007]: Got next segment. Inline: true Aug 7 23:09:54.176045487 (13h29m53.765390474s) [upload=14007]: Waiting for error group managing segments... Aug 7 23:09:54.176077538 (13h29m53.765422525s) [upload=14007]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:09:54.176082318 (13h29m53.765427305s) [upload=14007]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:09:54.176084522 (13h29m53.765429509s) [upload=14007]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:09:54.453731274 (13h29m54.043076261s) [upload=14007]: Done uploading segments... Aug 7 23:09:54.453750932 (13h29m54.043095909s) [upload=14007]: Upload finished. err: <nil> Aug 7 23:10:39.330325211 (13h30m38.919670198s) [upload=14021]: Starting upload Aug 7 23:10:39.330417358 (13h30m38.919762345s) [upload=14021]: Done starting upload Aug 7 23:10:39.330504384 (13h30m38.919849371s) [upload=14021]: Uploading segments... Aug 7 23:10:39.330543368 (13h30m38.919888355s) [upload=14021]: Got next segment. Inline: true Aug 7 23:10:39.330546675 (13h30m38.919891662s) [upload=14021]: Waiting for error group managing segments... Aug 7 23:10:39.330581371 (13h30m38.919926349s) [upload=14021]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:10:39.330585309 (13h30m38.919930286s) [upload=14021]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:10:39.330587663 (13h30m38.91993265s) [upload=14021]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:10:39.610133078 (13h30m39.199478065s) [upload=14021]: Done uploading segments... Aug 7 23:10:39.610150061 (13h30m39.199495048s) [upload=14021]: Upload finished. err: <nil> Aug 7 23:11:32.695175239 (13h31m32.284520226s) [upload=14034]: Starting upload Aug 7 23:11:32.695285580 (13h31m32.284630567s) [upload=14034]: Done starting upload Aug 7 23:11:32.695336297 (13h31m32.284681284s) [upload=14034]: Uploading segments... Aug 7 23:11:32.695365753 (13h31m32.28471074s) [upload=14034]: Got next segment. Inline: true Aug 7 23:11:32.695369610 (13h31m32.284714598s) [upload=14034]: Waiting for error group managing segments... Aug 7 23:11:32.695401581 (13h31m32.284746569s) [upload=14034]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:11:32.695405168 (13h31m32.284750155s) [upload=14034]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:11:32.695407543 (13h31m32.28475253s) [upload=14034]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:11:32.954750627 (13h31m32.544095614s) [upload=14034]: Done uploading segments... Aug 7 23:11:32.954767650 (13h31m32.544112627s) [upload=14034]: Upload finished. err: <nil> Aug 7 23:12:37.502051138 (13h32m37.091396125s) [upload=14054]: Starting upload Aug 7 23:12:37.502104370 (13h32m37.091449357s) [upload=14054]: Done starting upload Aug 7 23:12:37.502150869 (13h32m37.091495856s) [upload=14054]: Uploading segments... Aug 7 23:12:37.502182519 (13h32m37.091527506s) [upload=14054]: Got next segment. Inline: true Aug 7 23:12:37.502185986 (13h32m37.091530963s) [upload=14054]: Waiting for error group managing segments... Aug 7 23:12:37.502216213 (13h32m37.0915612s) [upload=14054]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:12:37.502219780 (13h32m37.091564767s) [upload=14054]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:12:37.502222425 (13h32m37.091567412s) [upload=14054]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:12:37.772497714 (13h32m37.361842701s) [upload=14054]: Done uploading segments... Aug 7 23:12:37.772511151 (13h32m37.361856138s) [upload=14054]: Upload finished. err: <nil> Aug 7 23:13:44.834965069 (13h33m44.424310046s) [upload=14070]: Starting upload Aug 7 23:13:44.835030293 (13h33m44.42437528s) [upload=14070]: Done starting upload Aug 7 23:13:44.835093484 (13h33m44.424438471s) [upload=14070]: Uploading segments... Aug 7 23:13:44.835133321 (13h33m44.424478308s) [upload=14070]: Got next segment. Inline: true Aug 7 23:13:44.835136998 (13h33m44.424481985s) [upload=14070]: Waiting for error group managing segments... Aug 7 23:13:44.835214376 (13h33m44.424559363s) [upload=14070]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:13:44.835220728 (13h33m44.424565715s) [upload=14070]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:13:44.835222972 (13h33m44.424567959s) [upload=14070]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:13:45.220889346 (13h33m44.810234343s) [upload=14070]: Done uploading segments... Aug 7 23:13:45.220937208 (13h33m44.810282205s) [upload=14070]: Upload finished. err: <nil> Aug 7 23:14:39.930436267 (13h34m39.519781254s) [upload=14085]: Starting upload Aug 7 23:14:39.930523634 (13h34m39.519868621s) [upload=14085]: Done starting upload Aug 7 23:14:39.930574131 (13h34m39.519919118s) [upload=14085]: Uploading segments... Aug 7 23:14:39.930614779 (13h34m39.519959766s) [upload=14085]: Got next segment. Inline: true Aug 7 23:14:39.930618847 (13h34m39.519963824s) [upload=14085]: Waiting for error group managing segments... Aug 7 23:14:39.930656519 (13h34m39.520001506s) [upload=14085]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:14:39.930660767 (13h34m39.520005754s) [upload=14085]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:14:39.930663112 (13h34m39.520008099s) [upload=14085]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:14:40.207143612 (13h34m39.796488599s) [upload=14085]: Done uploading segments... Aug 7 23:14:40.207161857 (13h34m39.796506844s) [upload=14085]: Upload finished. err: <nil> Aug 7 23:14:59.171620401 (13h34m58.760965388s) [upload=14091]: Starting upload Aug 7 23:14:59.171676218 (13h34m58.761021205s) [upload=14091]: Done starting upload Aug 7 23:14:59.171718468 (13h34m58.761063455s) [upload=14091]: Uploading segments... Aug 7 23:14:59.171763204 (13h34m58.761108191s) [upload=14091]: Got next segment. Inline: true Aug 7 23:14:59.171776990 (13h34m58.761121977s) [upload=14091]: Waiting for error group managing segments... Aug 7 23:14:59.171816787 (13h34m58.761161774s) [upload=14091]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:14:59.171820764 (13h34m58.761165741s) [upload=14091]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:14:59.171823429 (13h34m58.761168416s) [upload=14091]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:15:28.519847650 (13h35m28.109192637s) [upload=14091]: Done uploading segments... Aug 7 23:15:28.519870794 (13h35m28.109215781s) [upload=14091]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:15:34.319815009 (13h35m33.909159996s) [upload=14100]: Starting upload Aug 7 23:15:34.319867980 (13h35m33.909212967s) [upload=14100]: Done starting upload Aug 7 23:15:34.319916702 (13h35m33.909261679s) [upload=14100]: Uploading segments... Aug 7 23:15:34.319945739 (13h35m33.909290726s) [upload=14100]: Got next segment. Inline: true Aug 7 23:15:34.319949416 (13h35m33.909294403s) [upload=14100]: Waiting for error group managing segments... Aug 7 23:15:34.319988731 (13h35m33.909333718s) [upload=14100]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:15:34.319992698 (13h35m33.909337685s) [upload=14100]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:15:34.319995163 (13h35m33.90934015s) [upload=14100]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:16:03.672763058 (13h36m3.262108045s) [upload=14100]: Done uploading segments... Aug 7 23:16:03.672786503 (13h36m3.26213149s) [upload=14100]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:16:43.252280188 (13h36m42.841625175s) [upload=14114]: Starting upload Aug 7 23:16:43.252335193 (13h36m42.84168018s) [upload=14114]: Done starting upload Aug 7 23:16:43.252381041 (13h36m42.841726018s) [upload=14114]: Uploading segments... Aug 7 23:16:43.252412271 (13h36m42.841757258s) [upload=14114]: Got next segment. Inline: true Aug 7 23:16:43.252416208 (13h36m42.841761195s) [upload=14114]: Waiting for error group managing segments... Aug 7 23:16:43.252453641 (13h36m42.841798628s) [upload=14114]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:16:43.252457388 (13h36m42.841802375s) [upload=14114]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:16:43.252459622 (13h36m42.841804609s) [upload=14114]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:17:12.607365819 (13h37m12.196710806s) [upload=14114]: Done uploading segments... Aug 7 23:17:12.607385377 (13h37m12.196730364s) [upload=14114]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:17:56.238720000 (13h37m55.828064987s) [upload=14132]: Starting upload Aug 7 23:17:56.238785927 (13h37m55.828130914s) [upload=14132]: Done starting upload Aug 7 23:17:56.238851903 (13h37m55.82819689s) [upload=14132]: Uploading segments... Aug 7 23:17:56.238885677 (13h37m55.828230664s) [upload=14132]: Got next segment. Inline: true Aug 7 23:17:56.238888933 (13h37m55.82823392s) [upload=14132]: Waiting for error group managing segments... Aug 7 23:17:56.238932207 (13h37m55.828277194s) [upload=14132]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:17:56.238935984 (13h37m55.828280971s) [upload=14132]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:17:56.238938278 (13h37m55.828283265s) [upload=14132]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:18:25.593341271 (13h38m25.182686258s) [upload=14132]: Done uploading segments... Aug 7 23:18:25.593355268 (13h38m25.182700245s) [upload=14132]: Upload finished. err: metaclient: context deadline exceeded Duration: 1.88889675s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 3 out of 60 is failed (threshold=3) [.................................X.....X.X..................] duration: 2.250716238s FAIL (2025-08-07 23:38:23 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.250735765s edge gateway us-select-1
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 649.202122ms target: https://us1.storj.io/login FAIL (2025-08-07 23:38:21 UTC) orbiter warning: Get "https://us1.storj.io/login": dial tcp 34.150.199.48:443: connect: connection refused Duration: 649.22703ms core satellite us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 5 out of 60 is failed (threshold=3) [..........................XXXXX.............................] duration: 1.873247189s FAIL (2025-08-07 23:38:16 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== d. err: <nil> Aug 7 23:07:52.874987115 (13h27m52.464332102s) [upload=13971]: Starting upload Aug 7 23:07:52.875039626 (13h27m52.464384603s) [upload=13971]: Done starting upload Aug 7 23:07:52.875078921 (13h27m52.464423908s) [upload=13971]: Uploading segments... Aug 7 23:07:52.875111583 (13h27m52.46445657s) [upload=13971]: Got next segment. Inline: true Aug 7 23:07:52.875115000 (13h27m52.464459987s) [upload=13971]: Waiting for error group managing segments... Aug 7 23:07:52.875144226 (13h27m52.464489203s) [upload=13971]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:07:52.875147783 (13h27m52.46449277s) [upload=13971]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:07:52.875150268 (13h27m52.464495255s) [upload=13971]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:07:53.152523417 (13h27m52.741868404s) [upload=13971]: Done uploading segments... Aug 7 23:07:53.152567841 (13h27m52.741912828s) [upload=13971]: Upload finished. err: <nil> Aug 7 23:08:13.160315277 (13h28m12.749660264s) [upload=13978]: Starting upload Aug 7 23:08:13.160372306 (13h28m12.749717293s) [upload=13978]: Done starting upload Aug 7 23:08:13.160419306 (13h28m12.749764293s) [upload=13978]: Uploading segments... Aug 7 23:08:13.160446879 (13h28m12.749791866s) [upload=13978]: Got next segment. Inline: true Aug 7 23:08:13.160450095 (13h28m12.749795082s) [upload=13978]: Waiting for error group managing segments... Aug 7 23:08:13.160481565 (13h28m12.749826542s) [upload=13978]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:08:13.160485202 (13h28m12.749830189s) [upload=13978]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:08:13.160487416 (13h28m12.749832403s) [upload=13978]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:08:13.673421309 (13h28m13.262766296s) [upload=13978]: Done uploading segments... Aug 7 23:08:13.673439774 (13h28m13.262784761s) [upload=13978]: Upload finished. err: <nil> Aug 7 23:09:08.585854499 (13h29m8.175199486s) [upload=13993]: Starting upload Aug 7 23:09:08.585909965 (13h29m8.175254952s) [upload=13993]: Done starting upload Aug 7 23:09:08.585953217 (13h29m8.175298204s) [upload=13993]: Uploading segments... Aug 7 23:09:08.585980830 (13h29m8.175325817s) [upload=13993]: Got next segment. Inline: true Aug 7 23:09:08.585984206 (13h29m8.175329183s) [upload=13993]: Waiting for error group managing segments... Aug 7 23:09:08.586014684 (13h29m8.175359671s) [upload=13993]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:09:08.586018372 (13h29m8.175363359s) [upload=13993]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:09:08.586020767 (13h29m8.175365754s) [upload=13993]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:09:08.854829559 (13h29m8.444174546s) [upload=13993]: Done uploading segments... Aug 7 23:09:08.854846763 (13h29m8.44419175s) [upload=13993]: Upload finished. err: <nil> Aug 7 23:10:03.068961384 (13h30m2.658306371s) [upload=14012]: Starting upload Aug 7 23:10:03.069021258 (13h30m2.658366245s) [upload=14012]: Done starting upload Aug 7 23:10:03.069094389 (13h30m2.658439376s) [upload=14012]: Uploading segments... Aug 7 23:10:03.069128504 (13h30m2.658473491s) [upload=14012]: Got next segment. Inline: true Aug 7 23:10:03.069132782 (13h30m2.658477779s) [upload=14012]: Waiting for error group managing segments... Aug 7 23:10:03.069170935 (13h30m2.658515932s) [upload=14012]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:10:03.069175905 (13h30m2.658520892s) [upload=14012]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:10:03.069178650 (13h30m2.658523647s) [upload=14012]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:10:03.364714047 (13h30m2.954059064s) [upload=14012]: Done uploading segments... Aug 7 23:10:03.364761748 (13h30m2.954106745s) [upload=14012]: Upload finished. err: <nil> Aug 7 23:10:44.905335214 (13h30m44.494680201s) [upload=14022]: Starting upload Aug 7 23:10:44.905402192 (13h30m44.494747179s) [upload=14022]: Done starting upload Aug 7 23:10:44.905449983 (13h30m44.49479496s) [upload=14022]: Uploading segments... Aug 7 23:10:44.905479650 (13h30m44.494824628s) [upload=14022]: Got next segment. Inline: true Aug 7 23:10:44.905482787 (13h30m44.494827774s) [upload=14022]: Waiting for error group managing segments... Aug 7 23:10:44.905516160 (13h30m44.494861147s) [upload=14022]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:10:44.905519717 (13h30m44.494864694s) [upload=14022]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:10:44.905522222 (13h30m44.494867209s) [upload=14022]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:10:45.196197762 (13h30m44.785542749s) [upload=14022]: Done uploading segments... Aug 7 23:10:45.196218401 (13h30m44.785563389s) [upload=14022]: Upload finished. err: <nil> Aug 7 23:11:56.158319777 (13h31m55.747664764s) [upload=14041]: Starting upload Aug 7 23:11:56.158381214 (13h31m55.747726201s) [upload=14041]: Done starting upload Aug 7 23:11:56.158454435 (13h31m55.747799422s) [upload=14041]: Uploading segments... Aug 7 23:11:56.158485985 (13h31m55.747830972s) [upload=14041]: Got next segment. Inline: true Aug 7 23:11:56.158489752 (13h31m55.747834739s) [upload=14041]: Waiting for error group managing segments... Aug 7 23:11:56.158533225 (13h31m55.747878212s) [upload=14041]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:11:56.158536933 (13h31m55.74788192s) [upload=14041]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:11:56.158539077 (13h31m55.747884064s) [upload=14041]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:12:55.512693945 (13h32m55.102038922s) [upload=14041]: Done uploading segments... Aug 7 23:12:55.512714414 (13h32m55.102059401s) [upload=14041]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:13:04.863194286 (13h33m4.452539303s) [upload=14061]: Starting upload Aug 7 23:13:04.863346336 (13h33m4.452691313s) [upload=14061]: Done starting upload Aug 7 23:13:04.863414577 (13h33m4.452759614s) [upload=14061]: Uploading segments... Aug 7 23:13:04.863490322 (13h33m4.452835309s) [upload=14061]: Got next segment. Inline: true Aug 7 23:13:04.863497876 (13h33m4.452842863s) [upload=14061]: Waiting for error group managing segments... Aug 7 23:13:04.863564063 (13h33m4.45290905s) [upload=14061]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:13:04.863569523 (13h33m4.45291451s) [upload=14061]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:13:04.863571978 (13h33m4.452916965s) [upload=14061]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:14:04.218769699 (13h34m3.808114676s) [upload=14061]: Done uploading segments... Aug 7 23:14:04.218789687 (13h34m3.808134664s) [upload=14061]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:14:05.097995389 (13h34m4.687340376s) [upload=14076]: Starting upload Aug 7 23:14:05.098050264 (13h34m4.687395251s) [upload=14076]: Done starting upload Aug 7 23:14:05.098107544 (13h34m4.687452521s) [upload=14076]: Uploading segments... Aug 7 23:14:05.098195913 (13h34m4.6875409s) [upload=14076]: Got next segment. Inline: true Aug 7 23:14:05.098203497 (13h34m4.687548484s) [upload=14076]: Waiting for error group managing segments... Aug 7 23:14:05.098273151 (13h34m4.687618138s) [upload=14076]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:14:05.098281256 (13h34m4.687626253s) [upload=14076]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:14:05.098285083 (13h34m4.68763007s) [upload=14076]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:15:04.449929685 (13h35m4.039274672s) [upload=14076]: Done uploading segments... Aug 7 23:15:04.449952750 (13h35m4.039297737s) [upload=14076]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:15:05.510260535 (13h35m5.099605522s) [upload=14092]: Starting upload Aug 7 23:15:05.510334587 (13h35m5.099679574s) [upload=14092]: Done starting upload Aug 7 23:15:05.510383931 (13h35m5.099728918s) [upload=14092]: Uploading segments... Aug 7 23:15:05.510421683 (13h35m5.09976667s) [upload=14092]: Got next segment. Inline: true Aug 7 23:15:05.510425931 (13h35m5.099770918s) [upload=14092]: Waiting for error group managing segments... Aug 7 23:15:05.510461179 (13h35m5.099806166s) [upload=14092]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:15:05.510464996 (13h35m5.099809984s) [upload=14092]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:15:05.510467561 (13h35m5.099812548s) [upload=14092]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:16:04.861169851 (13h36m4.450514858s) [upload=14092]: Done uploading segments... Aug 7 23:16:04.861192945 (13h36m4.450537942s) [upload=14092]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:16:05.761665335 (13h36m5.351010322s) [upload=14104]: Starting upload Aug 7 23:16:05.761743936 (13h36m5.351088913s) [upload=14104]: Done starting upload Aug 7 23:16:05.761859217 (13h36m5.351204194s) [upload=14104]: Uploading segments... Aug 7 23:16:05.761910234 (13h36m5.351255221s) [upload=14104]: Got next segment. Inline: true Aug 7 23:16:05.761914483 (13h36m5.35125947s) [upload=14104]: Waiting for error group managing segments... Aug 7 23:16:05.761947837 (13h36m5.351292814s) [upload=14104]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:16:05.761951704 (13h36m5.351296691s) [upload=14104]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:16:05.761955882 (13h36m5.351300859s) [upload=14104]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:17:05.106651963 (13h37m4.69599695s) [upload=14104]: Done uploading segments... Aug 7 23:17:05.106703883 (13h37m4.69604887s) [upload=14104]: Upload finished. err: metaclient: context deadline exceeded Duration: 1.873271907s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [..........................XXXX.X..XX........................] duration: 5.980285246s FAIL (2025-08-07 23:38:12 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.980305122s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [........................X..X.XXXX.X.........................] duration: 6.476450772s FAIL (2025-08-07 23:38:10 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== 8]: Upload finished. err: context deadline exceeded Aug 7 23:18:32.336054087 (13h38m31.925399094s) [upload=14140]: Starting upload Aug 7 23:18:32.336180319 (13h38m31.925525306s) [upload=14140]: Done starting upload Aug 7 23:18:32.336261133 (13h38m31.92560616s) [upload=14140]: Uploading segments... Aug 7 23:18:32.336590001 (13h38m31.925935029s) [upload=14140]: Got next segment. Inline: false Aug 7 23:18:32.336650628 (13h38m31.925995655s) [upload=14140]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:18:32.336658754 (13h38m31.926003761s) [upload=14140]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:18:32.571732430 (13h38m32.161077417s) [seg_pos={0 0}, upload=14140]: Begin upload segment... Aug 7 23:18:32.571774531 (13h38m32.161119528s) [seg_pos={0 0}, upload=14140]: Done begin upload segment. Aug 7 23:18:32.571805400 (13h38m32.161150417s) [noise=true , address=77.90.31.12:4589 , piece_id=ZMKTDC4M5JHWVARXQDQAKRFSXM3OQQZ2IGQASJCK2OKYX3IQXHVA, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.571844173 (13h38m32.161189161s) [noise=true , address=104.36.82.9:4007 , piece_id=VZLD7RU4SK5MAQE22D7OBVT6XOQAN75UNDMT7JNPCRZNRKFX36XQ, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.572218380 (13h38m32.161563367s) [noise=true , address=104.36.82.10:4077 , piece_id=WYGEC4FYUPAR45MZIWLZQ2ASF5SUPECS4YI3UTRRHEVAKUEYCGYA, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.572289677 (13h38m32.161634654s) [noise=true , address=77.90.31.17:4770 , piece_id=ISA43NF4AKPK2SGYDPKT6ZIAD5WJ27A6GT6JZA2GHQUCAQBRRAFA, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.572494809 (13h38m32.161839796s) [noise=true , address=104.36.82.7:4911 , piece_id=FAMSQZJMALQIEAG5BVR5OV5WHEHN432Y4DZD3DNPC5CKP7X7GMQA, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.572588257 (13h38m32.161933244s) [noise=true , address=77.90.31.15:4697 , piece_id=6RYZJ22TCTJFRBMQV7M545VHB475CHDV2DBKCGO3MJP65VEMJ5FQ, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.572772450 (13h38m32.162117427s) [noise=true , address=45.91.116.12:4225 , piece_id=22GAFTM5SITGTVJEELA3BJP7W2QZW7BVYJGJMKJOLE2RF2GZ5CGQ, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.572865959 (13h38m32.162210946s) [noise=true , address=45.91.116.12:4227 , piece_id=T3JVU7VMSYBTYXBVGRD7LSNIG3PWF3QDVDNDQ24JQOSV7ZFIODNA, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.573029050 (13h38m32.162374037s) [noise=true , address=77.90.31.19:4842 , piece_id=3ZRHNMBPQUJH5SZZ75UJWLYIYZ5YI7CC5OMIJJFM7WVKZWJX6R3A, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.573227882 (13h38m32.162572869s) [noise=true , address=104.36.82.4:4804 , piece_id=LTSBCPH74YSWH7IHN6MH4NWTBJ5FQ7534VERPCY5NIDV24KYPXMQ, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.573293797 (13h38m32.162638774s) [noise=true , address=104.36.82.15:4259 , piece_id=GFCTNTXSY4WVSTPAD3AZIYCHE4O5EN24WN25VUNCRNWCSUWCYBMA, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.573504962 (13h38m32.162849949s) [noise=true , address=77.90.31.16:4734 , piece_id=OJPJZ5RF7QY4KNFBZ3MAVTA63HMC4JRG3HQLBN7SVROVKFQZY7BQ, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.573527936 (13h38m32.162872923s) [noise=true , address=45.91.116.7:4009 , piece_id=VKYMGNJEGFLB5M3L3WLD37C5ZILQGLRJROSQ4ZEIBN46CQSY5JZQ, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.574078970 (13h38m32.163423947s) [noise=true , address=45.91.116.11:4119 , piece_id=UXO65SB4OZ57BCKOWGE2VDV7KCSKCGE7I7K4RTGRYJ7VU5GKFPHQ, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.574216453 (13h38m32.16356143s) [noise=true , address=104.36.82.12:4149 , piece_id=2K5BCSGU4UN43HF7VHG3MBNMQEGSIZPPHTDHDFI3XPY5DDG6NKOA, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.576753347 (13h38m32.166098334s) [noise=true , address=104.36.82.8:4971 , piece_id=DNQVVV7QT2A2NFMIRURJ5IZRFPYZVX2NADRTX2P2YORA2CKMWY6Q, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:34.327895253 (13h38m33.91724024s) [upload=14140]: Next returned nil segment Aug 7 23:18:34.327912295 (13h38m33.917257282s) [upload=14140]: Waiting for error group managing segments... Aug 7 23:18:34.716828436 (13h38m34.306173413s) [noise=true , address=104.36.82.7:4911 , piece_id=FAMSQZJMALQIEAG5BVR5OV5WHEHN432Y4DZD3DNPC5CKP7X7GMQA, seg_pos={0 0}, upload=14140]: Done uploading piece. err: <nil> Aug 7 23:18:34.729509878 (13h38m34.318854865s) [noise=true , address=104.36.82.12:4149 , piece_id=2K5BCSGU4UN43HF7VHG3MBNMQEGSIZPPHTDHDFI3XPY5DDG6NKOA, seg_pos={0 0}, upload=14140]: Done uploading piece. err: <nil> Aug 7 23:18:34.738858565 (13h38m34.328203592s) [noise=true , address=104.36.82.4:4804 , piece_id=LTSBCPH74YSWH7IHN6MH4NWTBJ5FQ7534VERPCY5NIDV24KYPXMQ, seg_pos={0 0}, upload=14140]: Done uploading piece. err: <nil> Aug 7 23:18:34.783948702 (13h38m34.373293679s) [noise=true , address=77.90.31.19:4842 , piece_id=3ZRHNMBPQUJH5SZZ75UJWLYIYZ5YI7CC5OMIJJFM7WVKZWJX6R3A, seg_pos={0 0}, upload=14140]: Done uploading piece. err: <nil> Aug 7 23:18:34.824404112 (13h38m34.413749099s) [noise=true , address=104.36.82.15:4259 , piece_id=GFCTNTXSY4WVSTPAD3AZIYCHE4O5EN24WN25VUNCRNWCSUWCYBMA, seg_pos={0 0}, upload=14140]: Done uploading piece. err: <nil> Aug 7 23:18:34.863114634 (13h38m34.452459611s) [noise=true , address=104.36.82.9:4007 , piece_id=VZLD7RU4SK5MAQE22D7OBVT6XOQAN75UNDMT7JNPCRZNRKFX36XQ, seg_pos={0 0}, upload=14140]: Done uploading piece. err: <nil> Aug 7 23:18:34.889984711 (13h38m34.479329698s) [noise=true , address=77.90.31.16:4734 , piece_id=OJPJZ5RF7QY4KNFBZ3MAVTA63HMC4JRG3HQLBN7SVROVKFQZY7BQ, seg_pos={0 0}, upload=14140]: Done uploading piece. err: <nil> Aug 7 23:18:34.982222673 (13h38m34.57156766s) [noise=true , address=104.36.82.10:4077 , piece_id=WYGEC4FYUPAR45MZIWLZQ2ASF5SUPECS4YI3UTRRHEVAKUEYCGYA, seg_pos={0 0}, upload=14140]: Done uploading piece. err: <nil> Aug 7 23:18:34.999676977 (13h38m34.589021964s) [noise=true , address=77.90.31.17:4770 , piece_id=ISA43NF4AKPK2SGYDPKT6ZIAD5WJ27A6GT6JZA2GHQUCAQBRRAFA, seg_pos={0 0}, upload=14140]: Done uploading piece. err: <nil> Aug 7 23:18:35.002083712 (13h38m34.591428699s) [noise=true , address=45.91.116.12:4225 , piece_id=22GAFTM5SITGTVJEELA3BJP7W2QZW7BVYJGJMKJOLE2RF2GZ5CGQ, seg_pos={0 0}, upload=14140]: Done uploading piece. err: <nil> Aug 7 23:18:35.055266482 (13h38m34.644611469s) [noise=true , address=45.91.116.12:4227 , piece_id=T3JVU7VMSYBTYXBVGRD7LSNIG3PWF3QDVDNDQ24JQOSV7ZFIODNA, seg_pos={0 0}, upload=14140]: Done uploading piece. err: <nil> Aug 7 23:18:35.081593190 (13h38m34.670938177s) [noise=true , address=45.91.116.11:4119 , piece_id=UXO65SB4OZ57BCKOWGE2VDV7KCSKCGE7I7K4RTGRYJ7VU5GKFPHQ, seg_pos={0 0}, upload=14140]: Done uploading piece. err: <nil> Aug 7 23:18:35.123608726 (13h38m34.712953703s) [noise=true , address=77.90.31.12:4589 , piece_id=ZMKTDC4M5JHWVARXQDQAKRFSXM3OQQZ2IGQASJCK2OKYX3IQXHVA, seg_pos={0 0}, upload=14140]: Done uploading piece. err: <nil> Aug 7 23:18:35.123631779 (13h38m34.712976766s) [seg_pos={0 0}, upload=14140]: Segment reached optimal threshold of 13 pieces. Aug 7 23:18:35.123785944 (13h38m34.713130931s) [noise=true , address=45.91.116.7:4009 , piece_id=VKYMGNJEGFLB5M3L3WLD37C5ZILQGLRJROSQ4ZEIBN46CQSY5JZQ, seg_pos={0 0}, upload=14140]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF Aug 7 23:18:35.123945530 (13h38m34.713290517s) [noise=true , address=77.90.31.15:4697 , piece_id=6RYZJ22TCTJFRBMQV7M545VHB475CHDV2DBKCGO3MJP65VEMJ5FQ, seg_pos={0 0}, upload=14140]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: expected piece hash; context canceled; EOF Aug 7 23:18:35.123972431 (13h38m34.713317418s) [noise=true , address=104.36.82.8:4971 , piece_id=DNQVVV7QT2A2NFMIRURJ5IZRFPYZVX2NADRTX2P2YORA2CKMWY6Q, seg_pos={0 0}, upload=14140]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo): protocol: read tcp 188.166.225.170:44698->104.36.82.8:4971: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 23:18:35.123986628 (13h38m34.713331615s) [seg_pos={0 0}, upload=14140]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 7 23:18:35.124083804 (13h38m34.713428791s) [upload=14140]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 23:18:35.124087611 (13h38m34.713432599s) [upload=14140]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:18:35.414558916 (13h38m35.003903903s) [upload=14140]: Done uploading segments... Aug 7 23:18:35.414577682 (13h38m35.00392267s) [upload=14140]: Upload finished. err: <nil> Aug 7 23:18:58.905554387 (13h38m58.494899394s) [upload=14146]: Starting upload Aug 7 23:18:58.905620613 (13h38m58.4949656s) [upload=14146]: Done starting upload Aug 7 23:18:58.905683373 (13h38m58.49502836s) [upload=14146]: Uploading segments... Aug 7 23:18:58.905783355 (13h38m58.495128342s) [upload=14146]: Got next segment. Inline: false Aug 7 23:18:58.905832459 (13h38m58.495177446s) [upload=14146]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:18:58.905836627 (13h38m58.495181614s) [upload=14146]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:19:28.246792926 (13h39m27.836137913s) [upload=14146]: Next segment err: context deadline exceeded Aug 7 23:19:28.246867379 (13h39m27.836212366s) [upload=14146]: Done uploading segments... Aug 7 23:19:28.246884292 (13h39m27.836229269s) [upload=14146]: Upload finished. err: context deadline exceeded Duration: 6.476481702s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 4 out of 60 is failed (threshold=3) [.............................XXXX...........................] duration: 2.131774136s FAIL (2025-08-07 23:38:06 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== .175884539 (13h29m53.765229526s) [upload=14007]: Starting upload Aug 7 23:09:54.175937640 (13h29m53.765282618s) [upload=14007]: Done starting upload Aug 7 23:09:54.175979842 (13h29m53.765324829s) [upload=14007]: Uploading segments... Aug 7 23:09:54.176041389 (13h29m53.765386376s) [upload=14007]: Got next segment. Inline: true Aug 7 23:09:54.176045487 (13h29m53.765390474s) [upload=14007]: Waiting for error group managing segments... Aug 7 23:09:54.176077538 (13h29m53.765422525s) [upload=14007]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:09:54.176082318 (13h29m53.765427305s) [upload=14007]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:09:54.176084522 (13h29m53.765429509s) [upload=14007]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:09:54.453731274 (13h29m54.043076261s) [upload=14007]: Done uploading segments... Aug 7 23:09:54.453750932 (13h29m54.043095909s) [upload=14007]: Upload finished. err: <nil> Aug 7 23:10:39.330325211 (13h30m38.919670198s) [upload=14021]: Starting upload Aug 7 23:10:39.330417358 (13h30m38.919762345s) [upload=14021]: Done starting upload Aug 7 23:10:39.330504384 (13h30m38.919849371s) [upload=14021]: Uploading segments... Aug 7 23:10:39.330543368 (13h30m38.919888355s) [upload=14021]: Got next segment. Inline: true Aug 7 23:10:39.330546675 (13h30m38.919891662s) [upload=14021]: Waiting for error group managing segments... Aug 7 23:10:39.330581371 (13h30m38.919926349s) [upload=14021]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:10:39.330585309 (13h30m38.919930286s) [upload=14021]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:10:39.330587663 (13h30m38.91993265s) [upload=14021]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:10:39.610133078 (13h30m39.199478065s) [upload=14021]: Done uploading segments... Aug 7 23:10:39.610150061 (13h30m39.199495048s) [upload=14021]: Upload finished. err: <nil> Aug 7 23:11:32.695175239 (13h31m32.284520226s) [upload=14034]: Starting upload Aug 7 23:11:32.695285580 (13h31m32.284630567s) [upload=14034]: Done starting upload Aug 7 23:11:32.695336297 (13h31m32.284681284s) [upload=14034]: Uploading segments... Aug 7 23:11:32.695365753 (13h31m32.28471074s) [upload=14034]: Got next segment. Inline: true Aug 7 23:11:32.695369610 (13h31m32.284714598s) [upload=14034]: Waiting for error group managing segments... Aug 7 23:11:32.695401581 (13h31m32.284746569s) [upload=14034]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:11:32.695405168 (13h31m32.284750155s) [upload=14034]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:11:32.695407543 (13h31m32.28475253s) [upload=14034]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:11:32.954750627 (13h31m32.544095614s) [upload=14034]: Done uploading segments... Aug 7 23:11:32.954767650 (13h31m32.544112627s) [upload=14034]: Upload finished. err: <nil> Aug 7 23:12:37.502051138 (13h32m37.091396125s) [upload=14054]: Starting upload Aug 7 23:12:37.502104370 (13h32m37.091449357s) [upload=14054]: Done starting upload Aug 7 23:12:37.502150869 (13h32m37.091495856s) [upload=14054]: Uploading segments... Aug 7 23:12:37.502182519 (13h32m37.091527506s) [upload=14054]: Got next segment. Inline: true Aug 7 23:12:37.502185986 (13h32m37.091530963s) [upload=14054]: Waiting for error group managing segments... Aug 7 23:12:37.502216213 (13h32m37.0915612s) [upload=14054]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:12:37.502219780 (13h32m37.091564767s) [upload=14054]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:12:37.502222425 (13h32m37.091567412s) [upload=14054]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:12:37.772497714 (13h32m37.361842701s) [upload=14054]: Done uploading segments... Aug 7 23:12:37.772511151 (13h32m37.361856138s) [upload=14054]: Upload finished. err: <nil> Aug 7 23:13:44.834965069 (13h33m44.424310046s) [upload=14070]: Starting upload Aug 7 23:13:44.835030293 (13h33m44.42437528s) [upload=14070]: Done starting upload Aug 7 23:13:44.835093484 (13h33m44.424438471s) [upload=14070]: Uploading segments... Aug 7 23:13:44.835133321 (13h33m44.424478308s) [upload=14070]: Got next segment. Inline: true Aug 7 23:13:44.835136998 (13h33m44.424481985s) [upload=14070]: Waiting for error group managing segments... Aug 7 23:13:44.835214376 (13h33m44.424559363s) [upload=14070]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:13:44.835220728 (13h33m44.424565715s) [upload=14070]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:13:44.835222972 (13h33m44.424567959s) [upload=14070]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:13:45.220889346 (13h33m44.810234343s) [upload=14070]: Done uploading segments... Aug 7 23:13:45.220937208 (13h33m44.810282205s) [upload=14070]: Upload finished. err: <nil> Aug 7 23:14:39.930436267 (13h34m39.519781254s) [upload=14085]: Starting upload Aug 7 23:14:39.930523634 (13h34m39.519868621s) [upload=14085]: Done starting upload Aug 7 23:14:39.930574131 (13h34m39.519919118s) [upload=14085]: Uploading segments... Aug 7 23:14:39.930614779 (13h34m39.519959766s) [upload=14085]: Got next segment. Inline: true Aug 7 23:14:39.930618847 (13h34m39.519963824s) [upload=14085]: Waiting for error group managing segments... Aug 7 23:14:39.930656519 (13h34m39.520001506s) [upload=14085]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:14:39.930660767 (13h34m39.520005754s) [upload=14085]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:14:39.930663112 (13h34m39.520008099s) [upload=14085]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:14:40.207143612 (13h34m39.796488599s) [upload=14085]: Done uploading segments... Aug 7 23:14:40.207161857 (13h34m39.796506844s) [upload=14085]: Upload finished. err: <nil> Aug 7 23:14:59.171620401 (13h34m58.760965388s) [upload=14091]: Starting upload Aug 7 23:14:59.171676218 (13h34m58.761021205s) [upload=14091]: Done starting upload Aug 7 23:14:59.171718468 (13h34m58.761063455s) [upload=14091]: Uploading segments... Aug 7 23:14:59.171763204 (13h34m58.761108191s) [upload=14091]: Got next segment. Inline: true Aug 7 23:14:59.171776990 (13h34m58.761121977s) [upload=14091]: Waiting for error group managing segments... Aug 7 23:14:59.171816787 (13h34m58.761161774s) [upload=14091]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:14:59.171820764 (13h34m58.761165741s) [upload=14091]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:14:59.171823429 (13h34m58.761168416s) [upload=14091]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:15:28.519847650 (13h35m28.109192637s) [upload=14091]: Done uploading segments... Aug 7 23:15:28.519870794 (13h35m28.109215781s) [upload=14091]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:15:34.319815009 (13h35m33.909159996s) [upload=14100]: Starting upload Aug 7 23:15:34.319867980 (13h35m33.909212967s) [upload=14100]: Done starting upload Aug 7 23:15:34.319916702 (13h35m33.909261679s) [upload=14100]: Uploading segments... Aug 7 23:15:34.319945739 (13h35m33.909290726s) [upload=14100]: Got next segment. Inline: true Aug 7 23:15:34.319949416 (13h35m33.909294403s) [upload=14100]: Waiting for error group managing segments... Aug 7 23:15:34.319988731 (13h35m33.909333718s) [upload=14100]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:15:34.319992698 (13h35m33.909337685s) [upload=14100]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:15:34.319995163 (13h35m33.90934015s) [upload=14100]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:16:03.672763058 (13h36m3.262108045s) [upload=14100]: Done uploading segments... Aug 7 23:16:03.672786503 (13h36m3.26213149s) [upload=14100]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:16:43.252280188 (13h36m42.841625175s) [upload=14114]: Starting upload Aug 7 23:16:43.252335193 (13h36m42.84168018s) [upload=14114]: Done starting upload Aug 7 23:16:43.252381041 (13h36m42.841726018s) [upload=14114]: Uploading segments... Aug 7 23:16:43.252412271 (13h36m42.841757258s) [upload=14114]: Got next segment. Inline: true Aug 7 23:16:43.252416208 (13h36m42.841761195s) [upload=14114]: Waiting for error group managing segments... Aug 7 23:16:43.252453641 (13h36m42.841798628s) [upload=14114]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:16:43.252457388 (13h36m42.841802375s) [upload=14114]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:16:43.252459622 (13h36m42.841804609s) [upload=14114]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:17:12.607365819 (13h37m12.196710806s) [upload=14114]: Done uploading segments... Aug 7 23:17:12.607385377 (13h37m12.196730364s) [upload=14114]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:17:56.238720000 (13h37m55.828064987s) [upload=14132]: Starting upload Aug 7 23:17:56.238785927 (13h37m55.828130914s) [upload=14132]: Done starting upload Aug 7 23:17:56.238851903 (13h37m55.82819689s) [upload=14132]: Uploading segments... Aug 7 23:17:56.238885677 (13h37m55.828230664s) [upload=14132]: Got next segment. Inline: true Aug 7 23:17:56.238888933 (13h37m55.82823392s) [upload=14132]: Waiting for error group managing segments... Aug 7 23:17:56.238932207 (13h37m55.828277194s) [upload=14132]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:17:56.238935984 (13h37m55.828280971s) [upload=14132]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:17:56.238938278 (13h37m55.828283265s) [upload=14132]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:18:25.593341271 (13h38m25.182686258s) [upload=14132]: Done uploading segments... Aug 7 23:18:25.593355268 (13h38m25.182700245s) [upload=14132]: Upload finished. err: metaclient: context deadline exceeded Duration: 2.131832005s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 3 out of 60 is failed (threshold=3) [.......X....................X.....................X.........] duration: 6.147412045s FAIL (2025-08-07 23:37:54 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/4M") ======LOGS====== 916s) [upload=14346]: Starting upload Aug 7 23:30:30.723725030 (13h50m30.313070017s) [upload=14346]: Done starting upload Aug 7 23:30:30.723796056 (13h50m30.313141043s) [upload=14346]: Uploading segments... Aug 7 23:30:30.723935803 (13h50m30.31328079s) [upload=14346]: Got next segment. Inline: false Aug 7 23:30:30.723960240 (13h50m30.313305227s) [upload=14346]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:30:30.723964218 (13h50m30.313309205s) [upload=14346]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:30:31.181251948 (13h50m30.770596926s) [seg_pos={0 0}, upload=14346]: Begin upload segment... Aug 7 23:30:31.181299649 (13h50m30.770644636s) [seg_pos={0 0}, upload=14346]: Done begin upload segment. Aug 7 23:30:31.181318165 (13h50m30.770663152s) [noise=true , address=104.36.82.52:4086 , piece_id=JLEUROLLGI7X6MUOVFMEWIFCQNTYJXSZW2JMHGEAI66XREHK5LLQ, seg_pos={0 0}, upload=14346]: Uploading piece... Aug 7 23:30:31.181329696 (13h50m30.770674693s) [noise=true , address=104.36.82.54:4172 , piece_id=2QZVEPRDWBX4KY62QMLTPKNZ7QA2K3YDWJETONKWYOGLMPUVSVWA, seg_pos={0 0}, upload=14346]: Uploading piece... Aug 7 23:30:31.181716817 (13h50m30.771061804s) [noise=true , address=77.90.31.17:4797 , piece_id=P6M6EN2C7M6FVKVWV4LE6JEHSZTB24YEACVG7TMSNBQFLD6NLKDQ, seg_pos={0 0}, upload=14346]: Uploading piece... Aug 7 23:30:31.181721437 (13h50m30.771066424s) [noise=true , address=104.36.82.46:4489 , piece_id=KYHATGRQXZ6XXQFUSTYUYBI26BTC7VSN5N6OM766B3QOSWN747IA, seg_pos={0 0}, upload=14346]: Uploading piece... Aug 7 23:30:31.182063421 (13h50m30.771408408s) [noise=true , address=104.36.82.22:4534 , piece_id=5BROJMGAIDCZNPDEQH32QVVYD2JP4U7IN5DCDUP7LBMU2E6Z36CA, seg_pos={0 0}, upload=14346]: Uploading piece... Aug 7 23:30:31.182318108 (13h50m30.771663085s) [noise=true , address=104.36.82.58:4267 , piece_id=QVRI7PRWV4JSPRWQIJD6M7UJAWDBWN2VIVGOYUJ7NGO77SHZ4UJQ, seg_pos={0 0}, upload=14346]: Uploading piece... Aug 7 23:30:31.182483654 (13h50m30.771828641s) [noise=true , address=64.49.13.138:4489 , piece_id=GC6CUUPTPHVDNMD2ZDX5EUJSEUNL6I6OM3MYQNM25F2G7X64YQLQ, seg_pos={0 0}, upload=14346]: Uploading piece... Aug 7 23:30:31.182639132 (13h50m30.771984119s) [noise=true , address=45.91.116.11:4149 , piece_id=JGSWIYMTFANVCD326G2TRN24X26XYQQSJT6HFWBQAHJLQ2SLBA4A, seg_pos={0 0}, upload=14346]: Uploading piece... Aug 7 23:30:31.182934928 (13h50m30.772279905s) [noise=true , address=104.36.82.62:4428 , piece_id=P7EOR2HLYCVOBQBDUZ3OYNVO4RV7WRXLHVREWKIKE7SHYBV3VRVQ, seg_pos={0 0}, upload=14346]: Uploading piece... Aug 7 23:30:31.183037023 (13h50m30.77238201s) [noise=true , address=64.49.13.155:4917 , piece_id=GLY2EUU2PUD7JB4V7PAKNNFK6ZDWGN2DKS5YHTXUNAJXWGRYRTJA, seg_pos={0 0}, upload=14346]: Uploading piece... Aug 7 23:30:31.183158877 (13h50m30.772503864s) [noise=true , address=104.36.82.47:4534 , piece_id=PJCWOCQ7UKEHHSERR6C5RQC4Y3TBZRHZF6KPMZIMTAFS6E4525EA, seg_pos={0 0}, upload=14346]: Uploading piece... Aug 7 23:30:31.183353329 (13h50m30.772698316s) [noise=true , address=104.36.82.44:4883 , piece_id=7GULYV2REOYLHXZXRSPIQM46DRT4YKT7FXC3YCHAP7W7332RRTRA, seg_pos={0 0}, upload=14346]: Uploading piece... Aug 7 23:30:31.183386993 (13h50m30.77273198s) [noise=true , address=104.36.82.52:4097 , piece_id=REDOEPTMV6TQIM5HWQ2DQBPENXJFHVZV5LGM244DRAMGSV6HSNRQ, seg_pos={0 0}, upload=14346]: Uploading piece... Aug 7 23:30:31.183591444 (13h50m30.772936431s) [noise=true , address=104.36.82.60:4353 , piece_id=3TWE6IJ3OCY5CMMRHZ5DG2WBOHLWF3MEESPOGUMU3YA5NSGW3AKA, seg_pos={0 0}, upload=14346]: Uploading piece... Aug 7 23:30:31.183849538 (13h50m30.773194525s) [noise=true , address=64.49.13.27:4960 , piece_id=RCNXI4GI75JF5TWSSGMTNPWOTU4C3TADGATSBJQKJBRLVELNSGUA, seg_pos={0 0}, upload=14346]: Uploading piece... Aug 7 23:30:31.184184719 (13h50m30.773529706s) [noise=true , address=104.36.82.61:4392 , piece_id=OSMRSZL5LDLYLPM3KN5YVIMBU44HS72BHZOAB7W5SS5Z4DY4F4JA, seg_pos={0 0}, upload=14346]: Uploading piece... Aug 7 23:30:31.184422474 (13h50m30.773767461s) [noise=true , address=104.36.82.8:4995 , piece_id=NS2X5RBAQQWBIMO2U4XYCYA37GVF6OBAO3GZSTEPJZOXRMPMS62A, seg_pos={0 0}, upload=14346]: Uploading piece... Aug 7 23:30:31.184480636 (13h50m30.773825623s) [noise=true , address=104.36.82.45:4910 , piece_id=QEYOF4IB42OVZTWGW3YD7ZUYX2KZ6KKOUXFUHQQYOW4QY3JUGTDQ, seg_pos={0 0}, upload=14346]: Uploading piece... Aug 7 23:30:32.295332666 (13h50m31.884677653s) [upload=14346]: Next returned nil segment Aug 7 23:30:32.295347043 (13h50m31.88469204s) [upload=14346]: Waiting for error group managing segments... Aug 7 23:30:32.576560505 (13h50m32.165905492s) [noise=true , address=64.49.13.155:4917 , piece_id=GLY2EUU2PUD7JB4V7PAKNNFK6ZDWGN2DKS5YHTXUNAJXWGRYRTJA, seg_pos={0 0}, upload=14346]: Done uploading piece. err: <nil> Aug 7 23:30:32.584528981 (13h50m32.173873958s) [noise=true , address=64.49.13.138:4489 , piece_id=GC6CUUPTPHVDNMD2ZDX5EUJSEUNL6I6OM3MYQNM25F2G7X64YQLQ, seg_pos={0 0}, upload=14346]: Done uploading piece. err: <nil> Aug 7 23:30:32.647385961 (13h50m32.236730948s) [noise=true , address=64.49.13.27:4960 , piece_id=RCNXI4GI75JF5TWSSGMTNPWOTU4C3TADGATSBJQKJBRLVELNSGUA, seg_pos={0 0}, upload=14346]: Done uploading piece. err: <nil> Aug 7 23:30:33.094757008 (13h50m32.684101995s) [noise=true , address=104.36.82.52:4097 , piece_id=REDOEPTMV6TQIM5HWQ2DQBPENXJFHVZV5LGM244DRAMGSV6HSNRQ, seg_pos={0 0}, upload=14346]: Done uploading piece. err: <nil> Aug 7 23:30:33.167659426 (13h50m32.757004403s) [noise=true , address=104.36.82.45:4910 , piece_id=QEYOF4IB42OVZTWGW3YD7ZUYX2KZ6KKOUXFUHQQYOW4QY3JUGTDQ, seg_pos={0 0}, upload=14346]: Done uploading piece. err: <nil> Aug 7 23:30:33.198379320 (13h50m32.787724307s) [noise=true , address=104.36.82.54:4172 , piece_id=2QZVEPRDWBX4KY62QMLTPKNZ7QA2K3YDWJETONKWYOGLMPUVSVWA, seg_pos={0 0}, upload=14346]: Done uploading piece. err: <nil> Aug 7 23:30:33.235504739 (13h50m32.824849726s) [noise=true , address=104.36.82.58:4267 , piece_id=QVRI7PRWV4JSPRWQIJD6M7UJAWDBWN2VIVGOYUJ7NGO77SHZ4UJQ, seg_pos={0 0}, upload=14346]: Done uploading piece. err: <nil> Aug 7 23:30:33.264110248 (13h50m32.853455225s) [noise=true , address=104.36.82.52:4086 , piece_id=JLEUROLLGI7X6MUOVFMEWIFCQNTYJXSZW2JMHGEAI66XREHK5LLQ, seg_pos={0 0}, upload=14346]: Done uploading piece. err: <nil> Aug 7 23:30:33.327068230 (13h50m32.916413217s) [noise=true , address=104.36.82.8:4995 , piece_id=NS2X5RBAQQWBIMO2U4XYCYA37GVF6OBAO3GZSTEPJZOXRMPMS62A, seg_pos={0 0}, upload=14346]: Done uploading piece. err: <nil> Aug 7 23:30:33.366778193 (13h50m32.956123181s) [noise=true , address=104.36.82.60:4353 , piece_id=3TWE6IJ3OCY5CMMRHZ5DG2WBOHLWF3MEESPOGUMU3YA5NSGW3AKA, seg_pos={0 0}, upload=14346]: Done uploading piece. err: <nil> Aug 7 23:30:33.542767028 (13h50m33.132112016s) [noise=true , address=104.36.82.46:4489 , piece_id=KYHATGRQXZ6XXQFUSTYUYBI26BTC7VSN5N6OM766B3QOSWN747IA, seg_pos={0 0}, upload=14346]: Done uploading piece. err: <nil> Aug 7 23:30:33.575265094 (13h50m33.164610081s) [noise=true , address=104.36.82.22:4534 , piece_id=5BROJMGAIDCZNPDEQH32QVVYD2JP4U7IN5DCDUP7LBMU2E6Z36CA, seg_pos={0 0}, upload=14346]: Done uploading piece. err: <nil> Aug 7 23:30:33.672325632 (13h50m33.261670619s) [noise=true , address=77.90.31.17:4797 , piece_id=P6M6EN2C7M6FVKVWV4LE6JEHSZTB24YEACVG7TMSNBQFLD6NLKDQ, seg_pos={0 0}, upload=14346]: Done uploading piece. err: <nil> Aug 7 23:30:33.750650465 (13h50m33.339995452s) [noise=true , address=45.91.116.11:4149 , piece_id=JGSWIYMTFANVCD326G2TRN24X26XYQQSJT6HFWBQAHJLQ2SLBA4A, seg_pos={0 0}, upload=14346]: Done uploading piece. err: <nil> Aug 7 23:30:33.750672889 (13h50m33.340017876s) [seg_pos={0 0}, upload=14346]: Segment reached optimal threshold of 14 pieces. Aug 7 23:30:33.750785674 (13h50m33.340130661s) [noise=true , address=104.36.82.47:4534 , piece_id=PJCWOCQ7UKEHHSERR6C5RQC4Y3TBZRHZF6KPMZIMTAFS6E4525EA, seg_pos={0 0}, upload=14346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VDrNGcCnaKEmG5NN4nex1JoZAgdCEfX5CdsKbxytfLfKzPbr): protocol: expected piece hash; context canceled; EOF Aug 7 23:30:33.750885715 (13h50m33.340230702s) [noise=true , address=104.36.82.61:4392 , piece_id=OSMRSZL5LDLYLPM3KN5YVIMBU44HS72BHZOAB7W5SS5Z4DY4F4JA, seg_pos={0 0}, upload=14346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14FTsLbtv1vp43aRdDZZBY2Gf6epidM1A8HSyQKVokXGFFZGZk): protocol: expected piece hash; context canceled; EOF Aug 7 23:30:33.750979806 (13h50m33.340324793s) [noise=true , address=104.36.82.62:4428 , piece_id=P7EOR2HLYCVOBQBDUZ3OYNVO4RV7WRXLHVREWKIKE7SHYBV3VRVQ, seg_pos={0 0}, upload=14346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Ui5Ye48cKNGFPakMKsjdBi13Xo7bzrmXPNEmRBLKwpMtVAA3R): protocol: rpc: dial tcp 104.36.82.62:4428: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 23:30:33.751073955 (13h50m33.340418942s) [noise=true , address=104.36.82.44:4883 , piece_id=7GULYV2REOYLHXZXRSPIQM46DRT4YKT7FXC3YCHAP7W7332RRTRA, seg_pos={0 0}, upload=14346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cx1avL4umDm2pzJS1rNZqeDZjZMWnMtNsyNTdMdkxhpa1ByUN): protocol: context canceled; context canceled Aug 7 23:30:33.751119673 (13h50m33.34046466s) [seg_pos={0 0}, upload=14346]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 23:30:33.751224474 (13h50m33.340569471s) [upload=14346]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 23:30:33.751230776 (13h50m33.340575763s) [upload=14346]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:30:34.056540019 (13h50m33.645885006s) [upload=14346]: Done uploading segments... Aug 7 23:30:34.056559826 (13h50m33.645904823s) [upload=14346]: Upload finished. err: <nil> Duration: 6.14743599s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 4 out of 60 is failed (threshold=3) [...........................X...X..X.X.......................] duration: 1.902710097s FAIL (2025-08-07 23:37:52 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 1.902735556s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 8 out of 60 is failed (threshold=3) [...............................XXXXXX.X.X...................] duration: 5.53848728s FAIL (2025-08-07 23:37:48 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.538504294s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 651.349739ms target: https://us1.storj.io/login FAIL (2025-08-07 23:37:47 UTC) orbiter warning: Get "https://us1.storj.io/login": dial tcp 34.150.199.48:443: connect: connection refused Duration: 651.375409ms core satellite us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [.................................XX.X.......................] duration: 7.59958261s FAIL (2025-08-07 23:37:44 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== PC5WABH5C4FB5BLC55F4FZQAU4IAHMEA, seg_pos={0 0}, upload=14126]: Uploading piece... Aug 7 23:17:27.821935980 (13h37m27.411280967s) [noise=true , address=104.36.82.8:4983 , piece_id=AVNQRSGZB5XHG3VFYFHRLBLKQ7B4YWXGIPWDK32DYJSMFXUFTSNA, seg_pos={0 0}, upload=14126]: Uploading piece... Aug 7 23:17:27.822230193 (13h37m27.41157518s) [noise=true , address=104.36.82.16:4310 , piece_id=OAD3ECB3B5RXQB362MCLIWSQC4PKS3QUHWVMIMK4XPGH63JKO3LQ, seg_pos={0 0}, upload=14126]: Uploading piece... Aug 7 23:17:27.822283705 (13h37m27.411628692s) [noise=true , address=104.36.82.59:4323 , piece_id=WNEN4TRJFQ64H74WZOQGGP3V5F7Y3A7OIMORF7M7WEYOR252JSRA, seg_pos={0 0}, upload=14126]: Uploading piece... Aug 7 23:17:27.822684663 (13h37m27.41202965s) [noise=true , address=104.36.82.39:4747 , piece_id=7OQQP63SGNWWQEYR75RRJRHZSL52NRYH3BCS65XI4SWA24BKERUQ, seg_pos={0 0}, upload=14126]: Uploading piece... Aug 7 23:17:27.823034542 (13h37m27.412379529s) [noise=true , address=104.36.82.59:4322 , piece_id=D4THBNGRCWHLJC6HETJRODVC7ZRMOWMIAHGM6GYVU7LFIJNBTOZQ, seg_pos={0 0}, upload=14126]: Uploading piece... Aug 7 23:17:27.823366667 (13h37m27.412711654s) [noise=true , address=104.36.82.60:4343 , piece_id=UHOQFZPYXHO56TUWMTSQE5XVWFGO66ATAA3EU7TQ5KX2MIN2ZJQQ, seg_pos={0 0}, upload=14126]: Uploading piece... Aug 7 23:17:27.823697740 (13h37m27.413042727s) [noise=true , address=79.127.201.215:4355 , piece_id=KUJDHE66HM6IAFRND7KRYDII2FB7AA3S5K5S244JPQZUD4BYXFWA, seg_pos={0 0}, upload=14126]: Uploading piece... Aug 7 23:17:27.824090602 (13h37m27.413435589s) [noise=true , address=104.36.82.10:4093 , piece_id=VQUPLYS4X67PIYNTDH76KJ3DTBIWIH6UM3TQJ6VEREZIUJIH7A2Q, seg_pos={0 0}, upload=14126]: Uploading piece... Aug 7 23:17:27.824398111 (13h37m27.413743098s) [noise=true , address=64.49.13.170:4718 , piece_id=VXSII2V5QDMEJ5G5PJWNHXSJ6HF5NLLTGEDN5CHKKRWYHIAKHMVA, seg_pos={0 0}, upload=14126]: Uploading piece... Aug 7 23:17:27.824731088 (13h37m27.414076065s) [noise=true , address=104.36.82.43:4847 , piece_id=5Y56Z5NDMWSATFWNBYDWKBZB7SUSJ3ABAFMOY6PJ3B24SPPGJH3A, seg_pos={0 0}, upload=14126]: Uploading piece... Aug 7 23:17:27.828362365 (13h37m27.417707362s) [noise=true , address=104.36.82.61:4404 , piece_id=KGEALBUHVOZEFP7V7LPYXQG5APDPXBEWG6UG34QGOCYOHPFHEZNQ, seg_pos={0 0}, upload=14126]: Uploading piece... Aug 7 23:17:27.831994514 (13h37m27.421339501s) [noise=true , address=104.36.82.18:4370 , piece_id=HHYIBQ2JWTCPEOVRZQLKEWT7BFUZDXXZP6FYED3MPEWTVODBQUQA, seg_pos={0 0}, upload=14126]: Uploading piece... Aug 7 23:17:27.835602707 (13h37m27.424947694s) [noise=true , address=45.91.116.6:4988 , piece_id=QF7HXJOV5JYZ73FJMUUGT2JJUR3W2PY3C7CE6EQQCYGM72VDSTXQ, seg_pos={0 0}, upload=14126]: Uploading piece... Aug 7 23:17:27.837475180 (13h37m27.426820177s) [noise=true , address=104.36.82.34:4547 , piece_id=2XYJPP25OROUVGLPDMGFVMDOD4AYOFE6BRQ4OFMVEO2D73XVPNAQ, seg_pos={0 0}, upload=14126]: Uploading piece... Aug 7 23:17:27.841864818 (13h37m27.431209825s) [noise=true , address=104.36.82.62:4439 , piece_id=F4FXK4T3IKUXUGL37K3DEDNWWAUZZFEJVT7WBYDVVEZSXO6QG4OQ, seg_pos={0 0}, upload=14126]: Uploading piece... Aug 7 23:17:29.010273953 (13h37m28.59961894s) [upload=14126]: Next returned nil segment Aug 7 23:17:29.010293008 (13h37m28.599637996s) [upload=14126]: Waiting for error group managing segments... Aug 7 23:17:29.309514867 (13h37m28.898859844s) [noise=true , address=64.49.13.170:4718 , piece_id=VXSII2V5QDMEJ5G5PJWNHXSJ6HF5NLLTGEDN5CHKKRWYHIAKHMVA, seg_pos={0 0}, upload=14126]: Done uploading piece. err: <nil> Aug 7 23:17:29.779221569 (13h37m29.368566556s) [noise=true , address=79.127.201.215:4355 , piece_id=KUJDHE66HM6IAFRND7KRYDII2FB7AA3S5K5S244JPQZUD4BYXFWA, seg_pos={0 0}, upload=14126]: Done uploading piece. err: <nil> Aug 7 23:17:29.827123595 (13h37m29.416468582s) [noise=true , address=104.36.82.34:4547 , piece_id=2XYJPP25OROUVGLPDMGFVMDOD4AYOFE6BRQ4OFMVEO2D73XVPNAQ, seg_pos={0 0}, upload=14126]: Done uploading piece. err: <nil> Aug 7 23:17:29.905373281 (13h37m29.494718258s) [noise=true , address=104.36.82.39:4747 , piece_id=7OQQP63SGNWWQEYR75RRJRHZSL52NRYH3BCS65XI4SWA24BKERUQ, seg_pos={0 0}, upload=14126]: Done uploading piece. err: <nil> Aug 7 23:17:29.927717560 (13h37m29.517062537s) [noise=true , address=104.36.82.43:4847 , piece_id=5Y56Z5NDMWSATFWNBYDWKBZB7SUSJ3ABAFMOY6PJ3B24SPPGJH3A, seg_pos={0 0}, upload=14126]: Done uploading piece. err: <nil> Aug 7 23:17:30.139039260 (13h37m29.728384247s) [noise=true , address=104.36.82.46:4511 , piece_id=4A43V3IOJTGJNWLKNYB7PC5WABH5C4FB5BLC55F4FZQAU4IAHMEA, seg_pos={0 0}, upload=14126]: Done uploading piece. err: <nil> Aug 7 23:17:30.142609049 (13h37m29.731954036s) [noise=true , address=104.36.82.10:4093 , piece_id=VQUPLYS4X67PIYNTDH76KJ3DTBIWIH6UM3TQJ6VEREZIUJIH7A2Q, seg_pos={0 0}, upload=14126]: Done uploading piece. err: <nil> Aug 7 23:17:30.265344466 (13h37m29.854689463s) [noise=true , address=104.36.82.8:4983 , piece_id=AVNQRSGZB5XHG3VFYFHRLBLKQ7B4YWXGIPWDK32DYJSMFXUFTSNA, seg_pos={0 0}, upload=14126]: Done uploading piece. err: <nil> Aug 7 23:17:30.341283132 (13h37m29.930628119s) [noise=true , address=45.91.116.6:4988 , piece_id=QF7HXJOV5JYZ73FJMUUGT2JJUR3W2PY3C7CE6EQQCYGM72VDSTXQ, seg_pos={0 0}, upload=14126]: Done uploading piece. err: <nil> Aug 7 23:17:30.343588482 (13h37m29.932933459s) [noise=true , address=104.36.82.61:4404 , piece_id=KGEALBUHVOZEFP7V7LPYXQG5APDPXBEWG6UG34QGOCYOHPFHEZNQ, seg_pos={0 0}, upload=14126]: Done uploading piece. err: <nil> Aug 7 23:17:30.538382992 (13h37m30.127727969s) [noise=true , address=104.36.82.37:4666 , piece_id=WXBTFS2SS7DMVYBL5BHW7TBXH7FJODGB5JJDT6WP4LOO3LU4GNDA, seg_pos={0 0}, upload=14126]: Done uploading piece. err: <nil> Aug 7 23:17:30.631974070 (13h37m30.221319047s) [noise=true , address=104.36.82.9:4028 , piece_id=HSPPTMR3OOL5DWG25IF3LXYTSOUASUM3BFX7RKHZHCUTBP4B3GQA, seg_pos={0 0}, upload=14126]: Done uploading piece. err: <nil> Aug 7 23:17:31.962954475 (13h37m31.552299462s) [noise=true , address=104.36.82.59:4322 , piece_id=D4THBNGRCWHLJC6HETJRODVC7ZRMOWMIAHGM6GYVU7LFIJNBTOZQ, seg_pos={0 0}, upload=14126]: Done uploading piece. err: <nil> Aug 7 23:17:32.006950608 (13h37m31.596295635s) [noise=true , address=104.36.82.59:4323 , piece_id=WNEN4TRJFQ64H74WZOQGGP3V5F7Y3A7OIMORF7M7WEYOR252JSRA, seg_pos={0 0}, upload=14126]: Done uploading piece. err: <nil> Aug 7 23:17:32.006984062 (13h37m31.596329049s) [seg_pos={0 0}, upload=14126]: Segment reached optimal threshold of 14 pieces. Aug 7 23:17:32.007159788 (13h37m31.596504775s) [noise=true , address=104.36.82.18:4370 , piece_id=HHYIBQ2JWTCPEOVRZQLKEWT7BFUZDXXZP6FYED3MPEWTVODBQUQA, seg_pos={0 0}, upload=14126]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SXZN7WJvcHys9BRd4Lq8NAZ3MSgHf9ZVv7SgaiJNRosdmFVpm): protocol: expected piece hash; context canceled; EOF Aug 7 23:17:32.007279266 (13h37m31.596624253s) [noise=true , address=104.36.82.60:4343 , piece_id=UHOQFZPYXHO56TUWMTSQE5XVWFGO66ATAA3EU7TQ5KX2MIN2ZJQQ, seg_pos={0 0}, upload=14126]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16EsEMRW53sjKVag2WpSDGLhELyE2bSXyW4aAurCvDFKRNkXuf): protocol: write tcp 188.166.225.170:53332->104.36.82.60:4343: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 23:17:32.007291060 (13h37m31.596636047s) [noise=true , address=104.36.82.16:4310 , piece_id=OAD3ECB3B5RXQB362MCLIWSQC4PKS3QUHWVMIMK4XPGH63JKO3LQ, seg_pos={0 0}, upload=14126]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1g14evoaeUnhJThf5QKbQT1Gd8WW9MHvaYQhQ1GRgwPfDLoC5R): protocol: write tcp 188.166.225.170:46238->104.36.82.16:4310: use of closed network connection; write tcp 188.166.225.170:46238->104.36.82.16:4310: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46238->104.36.82.16:4310: use of closed network connection Aug 7 23:17:32.007409626 (13h37m31.596754613s) [noise=true , address=104.36.82.62:4439 , piece_id=F4FXK4T3IKUXUGL37K3DEDNWWAUZZFEJVT7WBYDVVEZSXO6QG4OQ, seg_pos={0 0}, upload=14126]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HRW3L15ka62qA7mWmEJaJ2ydTA5f8sVY1iL3Fn1R7ubisPF3W): protocol: read tcp 188.166.225.170:33134->104.36.82.62:4439: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 23:17:32.007441958 (13h37m31.596786945s) [seg_pos={0 0}, upload=14126]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 23:17:32.007533564 (13h37m31.596878551s) [upload=14126]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 23:17:32.007541158 (13h37m31.596886145s) [upload=14126]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:17:39.583862369 (13h37m39.173207346s) [upload=14126]: Done uploading segments... Aug 7 23:17:39.583880514 (13h37m39.173225501s) [upload=14126]: Upload finished. err: <nil> Aug 7 23:17:57.117137796 (13h37m56.706482783s) [upload=14133]: Starting upload Aug 7 23:17:57.117195166 (13h37m56.706540153s) [upload=14133]: Done starting upload Aug 7 23:17:57.117244931 (13h37m56.706589918s) [upload=14133]: Uploading segments... Aug 7 23:17:57.117378968 (13h37m56.706723955s) [upload=14133]: Got next segment. Inline: false Aug 7 23:17:57.117404277 (13h37m56.706749264s) [upload=14133]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:17:57.117408454 (13h37m56.706753441s) [upload=14133]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:18:56.470635719 (13h38m56.059980716s) [upload=14133]: Next segment err: context deadline exceeded Aug 7 23:18:56.470717675 (13h38m56.060062703s) [upload=14133]: Done uploading segments... Aug 7 23:18:56.470739287 (13h38m56.060084304s) [upload=14133]: Upload finished. err: context deadline exceeded Duration: 7.599608629s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 5.85585774s FAIL (2025-08-07 23:37:40 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.855884322s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [........................X......X............................] duration: 1.874826438s FAIL (2025-08-07 23:37:28 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== 30.892541890 (13h31m30.481886877s) [upload=14032]: Upload finished. err: <nil> Aug 7 23:11:48.390915893 (13h31m47.98026088s) [upload=14038]: Starting upload Aug 7 23:11:48.390965198 (13h31m47.980310185s) [upload=14038]: Done starting upload Aug 7 23:11:48.391014242 (13h31m47.980359229s) [upload=14038]: Uploading segments... Aug 7 23:11:48.391054007 (13h31m47.980399004s) [upload=14038]: Got next segment. Inline: true Aug 7 23:11:48.391057494 (13h31m47.980402471s) [upload=14038]: Waiting for error group managing segments... Aug 7 23:11:48.391089254 (13h31m47.980434231s) [upload=14038]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:11:48.391093213 (13h31m47.98043819s) [upload=14038]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:11:48.391095457 (13h31m47.980440444s) [upload=14038]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:11:48.701950724 (13h31m48.291295711s) [upload=14038]: Done uploading segments... Aug 7 23:11:48.701969449 (13h31m48.291314436s) [upload=14038]: Upload finished. err: <nil> Aug 7 23:12:06.430245154 (13h32m6.019590141s) [upload=14048]: Starting upload Aug 7 23:12:06.430293076 (13h32m6.019638063s) [upload=14048]: Done starting upload Aug 7 23:12:06.430361757 (13h32m6.019706745s) [upload=14048]: Uploading segments... Aug 7 23:12:06.430459223 (13h32m6.01980421s) [upload=14048]: Got next segment. Inline: true Aug 7 23:12:06.430464543 (13h32m6.01980953s) [upload=14048]: Waiting for error group managing segments... Aug 7 23:12:06.430527555 (13h32m6.019872542s) [upload=14048]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:12:06.430533796 (13h32m6.019878783s) [upload=14048]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:12:06.430536692 (13h32m6.019881679s) [upload=14048]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:12:06.729952062 (13h32m6.319297049s) [upload=14048]: Done uploading segments... Aug 7 23:12:06.729968844 (13h32m6.319313832s) [upload=14048]: Upload finished. err: <nil> Aug 7 23:12:37.551701338 (13h32m37.141046325s) [upload=14055]: Starting upload Aug 7 23:12:37.551746985 (13h32m37.141091972s) [upload=14055]: Done starting upload Aug 7 23:12:37.551807301 (13h32m37.141152288s) [upload=14055]: Uploading segments... Aug 7 23:12:37.551914826 (13h32m37.141259833s) [upload=14055]: Got next segment. Inline: true Aug 7 23:12:37.551921779 (13h32m37.141266766s) [upload=14055]: Waiting for error group managing segments... Aug 7 23:12:37.551977145 (13h32m37.141322122s) [upload=14055]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:12:37.551984059 (13h32m37.141329086s) [upload=14055]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:12:37.551988918 (13h32m37.141333936s) [upload=14055]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:12:37.767068918 (13h32m37.356413905s) [upload=14055]: Done uploading segments... Aug 7 23:12:37.767104365 (13h32m37.356449342s) [upload=14055]: Upload finished. err: metaclient: Too Many Requests Aug 7 23:13:47.522489627 (13h33m47.111834614s) [upload=14072]: Starting upload Aug 7 23:13:47.522559681 (13h33m47.111904668s) [upload=14072]: Done starting upload Aug 7 23:13:47.522629674 (13h33m47.111974661s) [upload=14072]: Uploading segments... Aug 7 23:13:47.522726771 (13h33m47.112071778s) [upload=14072]: Got next segment. Inline: true Aug 7 23:13:47.522735617 (13h33m47.112080614s) [upload=14072]: Waiting for error group managing segments... Aug 7 23:13:47.522790582 (13h33m47.112135569s) [upload=14072]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:13:47.522796493 (13h33m47.112141481s) [upload=14072]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:13:47.522799320 (13h33m47.112144307s) [upload=14072]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:13:47.796144342 (13h33m47.385489329s) [upload=14072]: Done uploading segments... Aug 7 23:13:47.796162656 (13h33m47.385507643s) [upload=14072]: Upload finished. err: <nil> Aug 7 23:14:22.239921861 (13h34m21.829266848s) [upload=14081]: Starting upload Aug 7 23:14:22.239973791 (13h34m21.829318778s) [upload=14081]: Done starting upload Aug 7 23:14:22.240087399 (13h34m21.829432396s) [upload=14081]: Uploading segments... Aug 7 23:14:22.240187159 (13h34m21.829532156s) [upload=14081]: Got next segment. Inline: true Aug 7 23:14:22.240194182 (13h34m21.829539179s) [upload=14081]: Waiting for error group managing segments... Aug 7 23:14:22.240255971 (13h34m21.829600958s) [upload=14081]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:14:22.240261311 (13h34m21.829606288s) [upload=14081]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:14:22.240263816 (13h34m21.829608803s) [upload=14081]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:14:22.537482123 (13h34m22.12682711s) [upload=14081]: Done uploading segments... Aug 7 23:14:22.537500677 (13h34m22.126845664s) [upload=14081]: Upload finished. err: <nil> Aug 7 23:14:56.968922831 (13h34m56.558267818s) [upload=14090]: Starting upload Aug 7 23:14:56.968974069 (13h34m56.558319056s) [upload=14090]: Done starting upload Aug 7 23:14:56.969028383 (13h34m56.55837337s) [upload=14090]: Uploading segments... Aug 7 23:14:56.969059602 (13h34m56.55840459s) [upload=14090]: Got next segment. Inline: true Aug 7 23:14:56.969063500 (13h34m56.558408487s) [upload=14090]: Waiting for error group managing segments... Aug 7 23:14:56.969102906 (13h34m56.558447893s) [upload=14090]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:14:56.969107034 (13h34m56.558452021s) [upload=14090]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:14:56.969109899 (13h34m56.558454886s) [upload=14090]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:14:57.268863065 (13h34m56.858208052s) [upload=14090]: Done uploading segments... Aug 7 23:14:57.268882201 (13h34m56.858227188s) [upload=14090]: Upload finished. err: <nil> Aug 7 23:16:09.138990771 (13h36m8.728335758s) [upload=14107]: Starting upload Aug 7 23:16:09.139048231 (13h36m8.728393208s) [upload=14107]: Done starting upload Aug 7 23:16:09.139094229 (13h36m8.728439226s) [upload=14107]: Uploading segments... Aug 7 23:16:09.139131069 (13h36m8.728476056s) [upload=14107]: Got next segment. Inline: true Aug 7 23:16:09.139135007 (13h36m8.728479994s) [upload=14107]: Waiting for error group managing segments... Aug 7 23:16:09.139167859 (13h36m8.728512846s) [upload=14107]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:16:09.139171937 (13h36m8.728516924s) [upload=14107]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:16:09.139199730 (13h36m8.728544707s) [upload=14107]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:16:09.409287150 (13h36m8.998632137s) [upload=14107]: Done uploading segments... Aug 7 23:16:09.409303952 (13h36m8.998648939s) [upload=14107]: Upload finished. err: <nil> Aug 7 23:16:30.665596455 (13h36m30.254941442s) [upload=14111]: Starting upload Aug 7 23:16:30.665642382 (13h36m30.254987369s) [upload=14111]: Done starting upload Aug 7 23:16:30.665687749 (13h36m30.255032726s) [upload=14111]: Uploading segments... Aug 7 23:16:30.665725801 (13h36m30.255070788s) [upload=14111]: Got next segment. Inline: true Aug 7 23:16:30.665729409 (13h36m30.255074386s) [upload=14111]: Waiting for error group managing segments... Aug 7 23:16:30.665763554 (13h36m30.255108541s) [upload=14111]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:16:30.665768984 (13h36m30.255113971s) [upload=14111]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:16:30.665771289 (13h36m30.255116276s) [upload=14111]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:16:30.945395745 (13h36m30.534740732s) [upload=14111]: Done uploading segments... Aug 7 23:16:30.945418538 (13h36m30.534763526s) [upload=14111]: Upload finished. err: <nil> Aug 7 23:17:25.677744155 (13h37m25.267089143s) [upload=14125]: Starting upload Aug 7 23:17:25.677793930 (13h37m25.267138917s) [upload=14125]: Done starting upload Aug 7 23:17:25.677862562 (13h37m25.267207539s) [upload=14125]: Uploading segments... Aug 7 23:17:25.677896967 (13h37m25.267241954s) [upload=14125]: Got next segment. Inline: true Aug 7 23:17:25.677900704 (13h37m25.267245681s) [upload=14125]: Waiting for error group managing segments... Aug 7 23:17:25.677933046 (13h37m25.267278033s) [upload=14125]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:17:25.677936884 (13h37m25.267281871s) [upload=14125]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:17:25.677939309 (13h37m25.267284296s) [upload=14125]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:17:26.502642065 (13h37m26.091987052s) [upload=14125]: Done uploading segments... Aug 7 23:17:26.502660942 (13h37m26.092005919s) [upload=14125]: Upload finished. err: <nil> Aug 7 23:17:53.025114325 (13h37m52.614459313s) [upload=14131]: Starting upload Aug 7 23:17:53.025169641 (13h37m52.614514618s) [upload=14131]: Done starting upload Aug 7 23:17:53.025219857 (13h37m52.614564844s) [upload=14131]: Uploading segments... Aug 7 23:17:53.025267939 (13h37m52.614612936s) [upload=14131]: Got next segment. Inline: true Aug 7 23:17:53.025271916 (13h37m52.614616903s) [upload=14131]: Waiting for error group managing segments... Aug 7 23:17:53.025302495 (13h37m52.614647482s) [upload=14131]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:17:53.025306382 (13h37m52.614651369s) [upload=14131]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:17:53.025308497 (13h37m52.614653484s) [upload=14131]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:17:53.302835519 (13h37m52.892180506s) [upload=14131]: Done uploading segments... Aug 7 23:17:53.302854656 (13h37m52.892199633s) [upload=14131]: Upload finished. err: metaclient: Too Many Requests Duration: 1.874858932s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [...........................XXXX.X..XX.......................] duration: 6.499854144s FAIL (2025-08-07 23:37:27 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.49987747s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 3 out of 60 is failed (threshold=3) [..................................X.....X.X.................] duration: 2.386994498s FAIL (2025-08-07 23:37:23 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.387016199s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 5 out of 60 is failed (threshold=3) [...........................XXXXX............................] duration: 1.906173543s FAIL (2025-08-07 23:37:22 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== d. err: <nil> Aug 7 23:07:52.874987115 (13h27m52.464332102s) [upload=13971]: Starting upload Aug 7 23:07:52.875039626 (13h27m52.464384603s) [upload=13971]: Done starting upload Aug 7 23:07:52.875078921 (13h27m52.464423908s) [upload=13971]: Uploading segments... Aug 7 23:07:52.875111583 (13h27m52.46445657s) [upload=13971]: Got next segment. Inline: true Aug 7 23:07:52.875115000 (13h27m52.464459987s) [upload=13971]: Waiting for error group managing segments... Aug 7 23:07:52.875144226 (13h27m52.464489203s) [upload=13971]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:07:52.875147783 (13h27m52.46449277s) [upload=13971]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:07:52.875150268 (13h27m52.464495255s) [upload=13971]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:07:53.152523417 (13h27m52.741868404s) [upload=13971]: Done uploading segments... Aug 7 23:07:53.152567841 (13h27m52.741912828s) [upload=13971]: Upload finished. err: <nil> Aug 7 23:08:13.160315277 (13h28m12.749660264s) [upload=13978]: Starting upload Aug 7 23:08:13.160372306 (13h28m12.749717293s) [upload=13978]: Done starting upload Aug 7 23:08:13.160419306 (13h28m12.749764293s) [upload=13978]: Uploading segments... Aug 7 23:08:13.160446879 (13h28m12.749791866s) [upload=13978]: Got next segment. Inline: true Aug 7 23:08:13.160450095 (13h28m12.749795082s) [upload=13978]: Waiting for error group managing segments... Aug 7 23:08:13.160481565 (13h28m12.749826542s) [upload=13978]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:08:13.160485202 (13h28m12.749830189s) [upload=13978]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:08:13.160487416 (13h28m12.749832403s) [upload=13978]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:08:13.673421309 (13h28m13.262766296s) [upload=13978]: Done uploading segments... Aug 7 23:08:13.673439774 (13h28m13.262784761s) [upload=13978]: Upload finished. err: <nil> Aug 7 23:09:08.585854499 (13h29m8.175199486s) [upload=13993]: Starting upload Aug 7 23:09:08.585909965 (13h29m8.175254952s) [upload=13993]: Done starting upload Aug 7 23:09:08.585953217 (13h29m8.175298204s) [upload=13993]: Uploading segments... Aug 7 23:09:08.585980830 (13h29m8.175325817s) [upload=13993]: Got next segment. Inline: true Aug 7 23:09:08.585984206 (13h29m8.175329183s) [upload=13993]: Waiting for error group managing segments... Aug 7 23:09:08.586014684 (13h29m8.175359671s) [upload=13993]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:09:08.586018372 (13h29m8.175363359s) [upload=13993]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:09:08.586020767 (13h29m8.175365754s) [upload=13993]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:09:08.854829559 (13h29m8.444174546s) [upload=13993]: Done uploading segments... Aug 7 23:09:08.854846763 (13h29m8.44419175s) [upload=13993]: Upload finished. err: <nil> Aug 7 23:10:03.068961384 (13h30m2.658306371s) [upload=14012]: Starting upload Aug 7 23:10:03.069021258 (13h30m2.658366245s) [upload=14012]: Done starting upload Aug 7 23:10:03.069094389 (13h30m2.658439376s) [upload=14012]: Uploading segments... Aug 7 23:10:03.069128504 (13h30m2.658473491s) [upload=14012]: Got next segment. Inline: true Aug 7 23:10:03.069132782 (13h30m2.658477779s) [upload=14012]: Waiting for error group managing segments... Aug 7 23:10:03.069170935 (13h30m2.658515932s) [upload=14012]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:10:03.069175905 (13h30m2.658520892s) [upload=14012]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:10:03.069178650 (13h30m2.658523647s) [upload=14012]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:10:03.364714047 (13h30m2.954059064s) [upload=14012]: Done uploading segments... Aug 7 23:10:03.364761748 (13h30m2.954106745s) [upload=14012]: Upload finished. err: <nil> Aug 7 23:10:44.905335214 (13h30m44.494680201s) [upload=14022]: Starting upload Aug 7 23:10:44.905402192 (13h30m44.494747179s) [upload=14022]: Done starting upload Aug 7 23:10:44.905449983 (13h30m44.49479496s) [upload=14022]: Uploading segments... Aug 7 23:10:44.905479650 (13h30m44.494824628s) [upload=14022]: Got next segment. Inline: true Aug 7 23:10:44.905482787 (13h30m44.494827774s) [upload=14022]: Waiting for error group managing segments... Aug 7 23:10:44.905516160 (13h30m44.494861147s) [upload=14022]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:10:44.905519717 (13h30m44.494864694s) [upload=14022]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:10:44.905522222 (13h30m44.494867209s) [upload=14022]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:10:45.196197762 (13h30m44.785542749s) [upload=14022]: Done uploading segments... Aug 7 23:10:45.196218401 (13h30m44.785563389s) [upload=14022]: Upload finished. err: <nil> Aug 7 23:11:56.158319777 (13h31m55.747664764s) [upload=14041]: Starting upload Aug 7 23:11:56.158381214 (13h31m55.747726201s) [upload=14041]: Done starting upload Aug 7 23:11:56.158454435 (13h31m55.747799422s) [upload=14041]: Uploading segments... Aug 7 23:11:56.158485985 (13h31m55.747830972s) [upload=14041]: Got next segment. Inline: true Aug 7 23:11:56.158489752 (13h31m55.747834739s) [upload=14041]: Waiting for error group managing segments... Aug 7 23:11:56.158533225 (13h31m55.747878212s) [upload=14041]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:11:56.158536933 (13h31m55.74788192s) [upload=14041]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:11:56.158539077 (13h31m55.747884064s) [upload=14041]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:12:55.512693945 (13h32m55.102038922s) [upload=14041]: Done uploading segments... Aug 7 23:12:55.512714414 (13h32m55.102059401s) [upload=14041]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:13:04.863194286 (13h33m4.452539303s) [upload=14061]: Starting upload Aug 7 23:13:04.863346336 (13h33m4.452691313s) [upload=14061]: Done starting upload Aug 7 23:13:04.863414577 (13h33m4.452759614s) [upload=14061]: Uploading segments... Aug 7 23:13:04.863490322 (13h33m4.452835309s) [upload=14061]: Got next segment. Inline: true Aug 7 23:13:04.863497876 (13h33m4.452842863s) [upload=14061]: Waiting for error group managing segments... Aug 7 23:13:04.863564063 (13h33m4.45290905s) [upload=14061]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:13:04.863569523 (13h33m4.45291451s) [upload=14061]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:13:04.863571978 (13h33m4.452916965s) [upload=14061]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:14:04.218769699 (13h34m3.808114676s) [upload=14061]: Done uploading segments... Aug 7 23:14:04.218789687 (13h34m3.808134664s) [upload=14061]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:14:05.097995389 (13h34m4.687340376s) [upload=14076]: Starting upload Aug 7 23:14:05.098050264 (13h34m4.687395251s) [upload=14076]: Done starting upload Aug 7 23:14:05.098107544 (13h34m4.687452521s) [upload=14076]: Uploading segments... Aug 7 23:14:05.098195913 (13h34m4.6875409s) [upload=14076]: Got next segment. Inline: true Aug 7 23:14:05.098203497 (13h34m4.687548484s) [upload=14076]: Waiting for error group managing segments... Aug 7 23:14:05.098273151 (13h34m4.687618138s) [upload=14076]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:14:05.098281256 (13h34m4.687626253s) [upload=14076]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:14:05.098285083 (13h34m4.68763007s) [upload=14076]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:15:04.449929685 (13h35m4.039274672s) [upload=14076]: Done uploading segments... Aug 7 23:15:04.449952750 (13h35m4.039297737s) [upload=14076]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:15:05.510260535 (13h35m5.099605522s) [upload=14092]: Starting upload Aug 7 23:15:05.510334587 (13h35m5.099679574s) [upload=14092]: Done starting upload Aug 7 23:15:05.510383931 (13h35m5.099728918s) [upload=14092]: Uploading segments... Aug 7 23:15:05.510421683 (13h35m5.09976667s) [upload=14092]: Got next segment. Inline: true Aug 7 23:15:05.510425931 (13h35m5.099770918s) [upload=14092]: Waiting for error group managing segments... Aug 7 23:15:05.510461179 (13h35m5.099806166s) [upload=14092]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:15:05.510464996 (13h35m5.099809984s) [upload=14092]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:15:05.510467561 (13h35m5.099812548s) [upload=14092]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:16:04.861169851 (13h36m4.450514858s) [upload=14092]: Done uploading segments... Aug 7 23:16:04.861192945 (13h36m4.450537942s) [upload=14092]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:16:05.761665335 (13h36m5.351010322s) [upload=14104]: Starting upload Aug 7 23:16:05.761743936 (13h36m5.351088913s) [upload=14104]: Done starting upload Aug 7 23:16:05.761859217 (13h36m5.351204194s) [upload=14104]: Uploading segments... Aug 7 23:16:05.761910234 (13h36m5.351255221s) [upload=14104]: Got next segment. Inline: true Aug 7 23:16:05.761914483 (13h36m5.35125947s) [upload=14104]: Waiting for error group managing segments... Aug 7 23:16:05.761947837 (13h36m5.351292814s) [upload=14104]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:16:05.761951704 (13h36m5.351296691s) [upload=14104]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:16:05.761955882 (13h36m5.351300859s) [upload=14104]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:17:05.106651963 (13h37m4.69599695s) [upload=14104]: Done uploading segments... Aug 7 23:17:05.106703883 (13h37m4.69604887s) [upload=14104]: Upload finished. err: metaclient: context deadline exceeded Duration: 1.906203932s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [.........................X..X.XXXX.X........................] duration: 6.516328033s FAIL (2025-08-07 23:37:12 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== 8]: Upload finished. err: context deadline exceeded Aug 7 23:18:32.336054087 (13h38m31.925399094s) [upload=14140]: Starting upload Aug 7 23:18:32.336180319 (13h38m31.925525306s) [upload=14140]: Done starting upload Aug 7 23:18:32.336261133 (13h38m31.92560616s) [upload=14140]: Uploading segments... Aug 7 23:18:32.336590001 (13h38m31.925935029s) [upload=14140]: Got next segment. Inline: false Aug 7 23:18:32.336650628 (13h38m31.925995655s) [upload=14140]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:18:32.336658754 (13h38m31.926003761s) [upload=14140]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:18:32.571732430 (13h38m32.161077417s) [seg_pos={0 0}, upload=14140]: Begin upload segment... Aug 7 23:18:32.571774531 (13h38m32.161119528s) [seg_pos={0 0}, upload=14140]: Done begin upload segment. Aug 7 23:18:32.571805400 (13h38m32.161150417s) [noise=true , address=77.90.31.12:4589 , piece_id=ZMKTDC4M5JHWVARXQDQAKRFSXM3OQQZ2IGQASJCK2OKYX3IQXHVA, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.571844173 (13h38m32.161189161s) [noise=true , address=104.36.82.9:4007 , piece_id=VZLD7RU4SK5MAQE22D7OBVT6XOQAN75UNDMT7JNPCRZNRKFX36XQ, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.572218380 (13h38m32.161563367s) [noise=true , address=104.36.82.10:4077 , piece_id=WYGEC4FYUPAR45MZIWLZQ2ASF5SUPECS4YI3UTRRHEVAKUEYCGYA, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.572289677 (13h38m32.161634654s) [noise=true , address=77.90.31.17:4770 , piece_id=ISA43NF4AKPK2SGYDPKT6ZIAD5WJ27A6GT6JZA2GHQUCAQBRRAFA, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.572494809 (13h38m32.161839796s) [noise=true , address=104.36.82.7:4911 , piece_id=FAMSQZJMALQIEAG5BVR5OV5WHEHN432Y4DZD3DNPC5CKP7X7GMQA, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.572588257 (13h38m32.161933244s) [noise=true , address=77.90.31.15:4697 , piece_id=6RYZJ22TCTJFRBMQV7M545VHB475CHDV2DBKCGO3MJP65VEMJ5FQ, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.572772450 (13h38m32.162117427s) [noise=true , address=45.91.116.12:4225 , piece_id=22GAFTM5SITGTVJEELA3BJP7W2QZW7BVYJGJMKJOLE2RF2GZ5CGQ, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.572865959 (13h38m32.162210946s) [noise=true , address=45.91.116.12:4227 , piece_id=T3JVU7VMSYBTYXBVGRD7LSNIG3PWF3QDVDNDQ24JQOSV7ZFIODNA, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.573029050 (13h38m32.162374037s) [noise=true , address=77.90.31.19:4842 , piece_id=3ZRHNMBPQUJH5SZZ75UJWLYIYZ5YI7CC5OMIJJFM7WVKZWJX6R3A, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.573227882 (13h38m32.162572869s) [noise=true , address=104.36.82.4:4804 , piece_id=LTSBCPH74YSWH7IHN6MH4NWTBJ5FQ7534VERPCY5NIDV24KYPXMQ, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.573293797 (13h38m32.162638774s) [noise=true , address=104.36.82.15:4259 , piece_id=GFCTNTXSY4WVSTPAD3AZIYCHE4O5EN24WN25VUNCRNWCSUWCYBMA, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.573504962 (13h38m32.162849949s) [noise=true , address=77.90.31.16:4734 , piece_id=OJPJZ5RF7QY4KNFBZ3MAVTA63HMC4JRG3HQLBN7SVROVKFQZY7BQ, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.573527936 (13h38m32.162872923s) [noise=true , address=45.91.116.7:4009 , piece_id=VKYMGNJEGFLB5M3L3WLD37C5ZILQGLRJROSQ4ZEIBN46CQSY5JZQ, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.574078970 (13h38m32.163423947s) [noise=true , address=45.91.116.11:4119 , piece_id=UXO65SB4OZ57BCKOWGE2VDV7KCSKCGE7I7K4RTGRYJ7VU5GKFPHQ, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.574216453 (13h38m32.16356143s) [noise=true , address=104.36.82.12:4149 , piece_id=2K5BCSGU4UN43HF7VHG3MBNMQEGSIZPPHTDHDFI3XPY5DDG6NKOA, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:32.576753347 (13h38m32.166098334s) [noise=true , address=104.36.82.8:4971 , piece_id=DNQVVV7QT2A2NFMIRURJ5IZRFPYZVX2NADRTX2P2YORA2CKMWY6Q, seg_pos={0 0}, upload=14140]: Uploading piece... Aug 7 23:18:34.327895253 (13h38m33.91724024s) [upload=14140]: Next returned nil segment Aug 7 23:18:34.327912295 (13h38m33.917257282s) [upload=14140]: Waiting for error group managing segments... Aug 7 23:18:34.716828436 (13h38m34.306173413s) [noise=true , address=104.36.82.7:4911 , piece_id=FAMSQZJMALQIEAG5BVR5OV5WHEHN432Y4DZD3DNPC5CKP7X7GMQA, seg_pos={0 0}, upload=14140]: Done uploading piece. err: <nil> Aug 7 23:18:34.729509878 (13h38m34.318854865s) [noise=true , address=104.36.82.12:4149 , piece_id=2K5BCSGU4UN43HF7VHG3MBNMQEGSIZPPHTDHDFI3XPY5DDG6NKOA, seg_pos={0 0}, upload=14140]: Done uploading piece. err: <nil> Aug 7 23:18:34.738858565 (13h38m34.328203592s) [noise=true , address=104.36.82.4:4804 , piece_id=LTSBCPH74YSWH7IHN6MH4NWTBJ5FQ7534VERPCY5NIDV24KYPXMQ, seg_pos={0 0}, upload=14140]: Done uploading piece. err: <nil> Aug 7 23:18:34.783948702 (13h38m34.373293679s) [noise=true , address=77.90.31.19:4842 , piece_id=3ZRHNMBPQUJH5SZZ75UJWLYIYZ5YI7CC5OMIJJFM7WVKZWJX6R3A, seg_pos={0 0}, upload=14140]: Done uploading piece. err: <nil> Aug 7 23:18:34.824404112 (13h38m34.413749099s) [noise=true , address=104.36.82.15:4259 , piece_id=GFCTNTXSY4WVSTPAD3AZIYCHE4O5EN24WN25VUNCRNWCSUWCYBMA, seg_pos={0 0}, upload=14140]: Done uploading piece. err: <nil> Aug 7 23:18:34.863114634 (13h38m34.452459611s) [noise=true , address=104.36.82.9:4007 , piece_id=VZLD7RU4SK5MAQE22D7OBVT6XOQAN75UNDMT7JNPCRZNRKFX36XQ, seg_pos={0 0}, upload=14140]: Done uploading piece. err: <nil> Aug 7 23:18:34.889984711 (13h38m34.479329698s) [noise=true , address=77.90.31.16:4734 , piece_id=OJPJZ5RF7QY4KNFBZ3MAVTA63HMC4JRG3HQLBN7SVROVKFQZY7BQ, seg_pos={0 0}, upload=14140]: Done uploading piece. err: <nil> Aug 7 23:18:34.982222673 (13h38m34.57156766s) [noise=true , address=104.36.82.10:4077 , piece_id=WYGEC4FYUPAR45MZIWLZQ2ASF5SUPECS4YI3UTRRHEVAKUEYCGYA, seg_pos={0 0}, upload=14140]: Done uploading piece. err: <nil> Aug 7 23:18:34.999676977 (13h38m34.589021964s) [noise=true , address=77.90.31.17:4770 , piece_id=ISA43NF4AKPK2SGYDPKT6ZIAD5WJ27A6GT6JZA2GHQUCAQBRRAFA, seg_pos={0 0}, upload=14140]: Done uploading piece. err: <nil> Aug 7 23:18:35.002083712 (13h38m34.591428699s) [noise=true , address=45.91.116.12:4225 , piece_id=22GAFTM5SITGTVJEELA3BJP7W2QZW7BVYJGJMKJOLE2RF2GZ5CGQ, seg_pos={0 0}, upload=14140]: Done uploading piece. err: <nil> Aug 7 23:18:35.055266482 (13h38m34.644611469s) [noise=true , address=45.91.116.12:4227 , piece_id=T3JVU7VMSYBTYXBVGRD7LSNIG3PWF3QDVDNDQ24JQOSV7ZFIODNA, seg_pos={0 0}, upload=14140]: Done uploading piece. err: <nil> Aug 7 23:18:35.081593190 (13h38m34.670938177s) [noise=true , address=45.91.116.11:4119 , piece_id=UXO65SB4OZ57BCKOWGE2VDV7KCSKCGE7I7K4RTGRYJ7VU5GKFPHQ, seg_pos={0 0}, upload=14140]: Done uploading piece. err: <nil> Aug 7 23:18:35.123608726 (13h38m34.712953703s) [noise=true , address=77.90.31.12:4589 , piece_id=ZMKTDC4M5JHWVARXQDQAKRFSXM3OQQZ2IGQASJCK2OKYX3IQXHVA, seg_pos={0 0}, upload=14140]: Done uploading piece. err: <nil> Aug 7 23:18:35.123631779 (13h38m34.712976766s) [seg_pos={0 0}, upload=14140]: Segment reached optimal threshold of 13 pieces. Aug 7 23:18:35.123785944 (13h38m34.713130931s) [noise=true , address=45.91.116.7:4009 , piece_id=VKYMGNJEGFLB5M3L3WLD37C5ZILQGLRJROSQ4ZEIBN46CQSY5JZQ, seg_pos={0 0}, upload=14140]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF Aug 7 23:18:35.123945530 (13h38m34.713290517s) [noise=true , address=77.90.31.15:4697 , piece_id=6RYZJ22TCTJFRBMQV7M545VHB475CHDV2DBKCGO3MJP65VEMJ5FQ, seg_pos={0 0}, upload=14140]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: expected piece hash; context canceled; EOF Aug 7 23:18:35.123972431 (13h38m34.713317418s) [noise=true , address=104.36.82.8:4971 , piece_id=DNQVVV7QT2A2NFMIRURJ5IZRFPYZVX2NADRTX2P2YORA2CKMWY6Q, seg_pos={0 0}, upload=14140]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo): protocol: read tcp 188.166.225.170:44698->104.36.82.8:4971: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 23:18:35.123986628 (13h38m34.713331615s) [seg_pos={0 0}, upload=14140]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 7 23:18:35.124083804 (13h38m34.713428791s) [upload=14140]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 23:18:35.124087611 (13h38m34.713432599s) [upload=14140]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:18:35.414558916 (13h38m35.003903903s) [upload=14140]: Done uploading segments... Aug 7 23:18:35.414577682 (13h38m35.00392267s) [upload=14140]: Upload finished. err: <nil> Aug 7 23:18:58.905554387 (13h38m58.494899394s) [upload=14146]: Starting upload Aug 7 23:18:58.905620613 (13h38m58.4949656s) [upload=14146]: Done starting upload Aug 7 23:18:58.905683373 (13h38m58.49502836s) [upload=14146]: Uploading segments... Aug 7 23:18:58.905783355 (13h38m58.495128342s) [upload=14146]: Got next segment. Inline: false Aug 7 23:18:58.905832459 (13h38m58.495177446s) [upload=14146]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:18:58.905836627 (13h38m58.495181614s) [upload=14146]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:19:28.246792926 (13h39m27.836137913s) [upload=14146]: Next segment err: context deadline exceeded Aug 7 23:19:28.246867379 (13h39m27.836212366s) [upload=14146]: Done uploading segments... Aug 7 23:19:28.246884292 (13h39m27.836229269s) [upload=14146]: Upload finished. err: context deadline exceeded Duration: 6.516354092s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 3 out of 60 is failed (threshold=3) [........X....................X.....................X........] duration: 5.946077144s FAIL (2025-08-07 23:37:12 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/4M") ======LOGS====== 916s) [upload=14346]: Starting upload Aug 7 23:30:30.723725030 (13h50m30.313070017s) [upload=14346]: Done starting upload Aug 7 23:30:30.723796056 (13h50m30.313141043s) [upload=14346]: Uploading segments... Aug 7 23:30:30.723935803 (13h50m30.31328079s) [upload=14346]: Got next segment. Inline: false Aug 7 23:30:30.723960240 (13h50m30.313305227s) [upload=14346]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:30:30.723964218 (13h50m30.313309205s) [upload=14346]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:30:31.181251948 (13h50m30.770596926s) [seg_pos={0 0}, upload=14346]: Begin upload segment... Aug 7 23:30:31.181299649 (13h50m30.770644636s) [seg_pos={0 0}, upload=14346]: Done begin upload segment. Aug 7 23:30:31.181318165 (13h50m30.770663152s) [noise=true , address=104.36.82.52:4086 , piece_id=JLEUROLLGI7X6MUOVFMEWIFCQNTYJXSZW2JMHGEAI66XREHK5LLQ, seg_pos={0 0}, upload=14346]: Uploading piece... Aug 7 23:30:31.181329696 (13h50m30.770674693s) [noise=true , address=104.36.82.54:4172 , piece_id=2QZVEPRDWBX4KY62QMLTPKNZ7QA2K3YDWJETONKWYOGLMPUVSVWA, seg_pos={0 0}, upload=14346]: Uploading piece... Aug 7 23:30:31.181716817 (13h50m30.771061804s) [noise=true , address=77.90.31.17:4797 , piece_id=P6M6EN2C7M6FVKVWV4LE6JEHSZTB24YEACVG7TMSNBQFLD6NLKDQ, seg_pos={0 0}, upload=14346]: Uploading piece... Aug 7 23:30:31.181721437 (13h50m30.771066424s) [noise=true , address=104.36.82.46:4489 , piece_id=KYHATGRQXZ6XXQFUSTYUYBI26BTC7VSN5N6OM766B3QOSWN747IA, seg_pos={0 0}, upload=14346]: Uploading piece... Aug 7 23:30:31.182063421 (13h50m30.771408408s) [noise=true , address=104.36.82.22:4534 , piece_id=5BROJMGAIDCZNPDEQH32QVVYD2JP4U7IN5DCDUP7LBMU2E6Z36CA, seg_pos={0 0}, upload=14346]: Uploading piece... Aug 7 23:30:31.182318108 (13h50m30.771663085s) [noise=true , address=104.36.82.58:4267 , piece_id=QVRI7PRWV4JSPRWQIJD6M7UJAWDBWN2VIVGOYUJ7NGO77SHZ4UJQ, seg_pos={0 0}, upload=14346]: Uploading piece... Aug 7 23:30:31.182483654 (13h50m30.771828641s) [noise=true , address=64.49.13.138:4489 , piece_id=GC6CUUPTPHVDNMD2ZDX5EUJSEUNL6I6OM3MYQNM25F2G7X64YQLQ, seg_pos={0 0}, upload=14346]: Uploading piece... Aug 7 23:30:31.182639132 (13h50m30.771984119s) [noise=true , address=45.91.116.11:4149 , piece_id=JGSWIYMTFANVCD326G2TRN24X26XYQQSJT6HFWBQAHJLQ2SLBA4A, seg_pos={0 0}, upload=14346]: Uploading piece... Aug 7 23:30:31.182934928 (13h50m30.772279905s) [noise=true , address=104.36.82.62:4428 , piece_id=P7EOR2HLYCVOBQBDUZ3OYNVO4RV7WRXLHVREWKIKE7SHYBV3VRVQ, seg_pos={0 0}, upload=14346]: Uploading piece... Aug 7 23:30:31.183037023 (13h50m30.77238201s) [noise=true , address=64.49.13.155:4917 , piece_id=GLY2EUU2PUD7JB4V7PAKNNFK6ZDWGN2DKS5YHTXUNAJXWGRYRTJA, seg_pos={0 0}, upload=14346]: Uploading piece... Aug 7 23:30:31.183158877 (13h50m30.772503864s) [noise=true , address=104.36.82.47:4534 , piece_id=PJCWOCQ7UKEHHSERR6C5RQC4Y3TBZRHZF6KPMZIMTAFS6E4525EA, seg_pos={0 0}, upload=14346]: Uploading piece... Aug 7 23:30:31.183353329 (13h50m30.772698316s) [noise=true , address=104.36.82.44:4883 , piece_id=7GULYV2REOYLHXZXRSPIQM46DRT4YKT7FXC3YCHAP7W7332RRTRA, seg_pos={0 0}, upload=14346]: Uploading piece... Aug 7 23:30:31.183386993 (13h50m30.77273198s) [noise=true , address=104.36.82.52:4097 , piece_id=REDOEPTMV6TQIM5HWQ2DQBPENXJFHVZV5LGM244DRAMGSV6HSNRQ, seg_pos={0 0}, upload=14346]: Uploading piece... Aug 7 23:30:31.183591444 (13h50m30.772936431s) [noise=true , address=104.36.82.60:4353 , piece_id=3TWE6IJ3OCY5CMMRHZ5DG2WBOHLWF3MEESPOGUMU3YA5NSGW3AKA, seg_pos={0 0}, upload=14346]: Uploading piece... Aug 7 23:30:31.183849538 (13h50m30.773194525s) [noise=true , address=64.49.13.27:4960 , piece_id=RCNXI4GI75JF5TWSSGMTNPWOTU4C3TADGATSBJQKJBRLVELNSGUA, seg_pos={0 0}, upload=14346]: Uploading piece... Aug 7 23:30:31.184184719 (13h50m30.773529706s) [noise=true , address=104.36.82.61:4392 , piece_id=OSMRSZL5LDLYLPM3KN5YVIMBU44HS72BHZOAB7W5SS5Z4DY4F4JA, seg_pos={0 0}, upload=14346]: Uploading piece... Aug 7 23:30:31.184422474 (13h50m30.773767461s) [noise=true , address=104.36.82.8:4995 , piece_id=NS2X5RBAQQWBIMO2U4XYCYA37GVF6OBAO3GZSTEPJZOXRMPMS62A, seg_pos={0 0}, upload=14346]: Uploading piece... Aug 7 23:30:31.184480636 (13h50m30.773825623s) [noise=true , address=104.36.82.45:4910 , piece_id=QEYOF4IB42OVZTWGW3YD7ZUYX2KZ6KKOUXFUHQQYOW4QY3JUGTDQ, seg_pos={0 0}, upload=14346]: Uploading piece... Aug 7 23:30:32.295332666 (13h50m31.884677653s) [upload=14346]: Next returned nil segment Aug 7 23:30:32.295347043 (13h50m31.88469204s) [upload=14346]: Waiting for error group managing segments... Aug 7 23:30:32.576560505 (13h50m32.165905492s) [noise=true , address=64.49.13.155:4917 , piece_id=GLY2EUU2PUD7JB4V7PAKNNFK6ZDWGN2DKS5YHTXUNAJXWGRYRTJA, seg_pos={0 0}, upload=14346]: Done uploading piece. err: <nil> Aug 7 23:30:32.584528981 (13h50m32.173873958s) [noise=true , address=64.49.13.138:4489 , piece_id=GC6CUUPTPHVDNMD2ZDX5EUJSEUNL6I6OM3MYQNM25F2G7X64YQLQ, seg_pos={0 0}, upload=14346]: Done uploading piece. err: <nil> Aug 7 23:30:32.647385961 (13h50m32.236730948s) [noise=true , address=64.49.13.27:4960 , piece_id=RCNXI4GI75JF5TWSSGMTNPWOTU4C3TADGATSBJQKJBRLVELNSGUA, seg_pos={0 0}, upload=14346]: Done uploading piece. err: <nil> Aug 7 23:30:33.094757008 (13h50m32.684101995s) [noise=true , address=104.36.82.52:4097 , piece_id=REDOEPTMV6TQIM5HWQ2DQBPENXJFHVZV5LGM244DRAMGSV6HSNRQ, seg_pos={0 0}, upload=14346]: Done uploading piece. err: <nil> Aug 7 23:30:33.167659426 (13h50m32.757004403s) [noise=true , address=104.36.82.45:4910 , piece_id=QEYOF4IB42OVZTWGW3YD7ZUYX2KZ6KKOUXFUHQQYOW4QY3JUGTDQ, seg_pos={0 0}, upload=14346]: Done uploading piece. err: <nil> Aug 7 23:30:33.198379320 (13h50m32.787724307s) [noise=true , address=104.36.82.54:4172 , piece_id=2QZVEPRDWBX4KY62QMLTPKNZ7QA2K3YDWJETONKWYOGLMPUVSVWA, seg_pos={0 0}, upload=14346]: Done uploading piece. err: <nil> Aug 7 23:30:33.235504739 (13h50m32.824849726s) [noise=true , address=104.36.82.58:4267 , piece_id=QVRI7PRWV4JSPRWQIJD6M7UJAWDBWN2VIVGOYUJ7NGO77SHZ4UJQ, seg_pos={0 0}, upload=14346]: Done uploading piece. err: <nil> Aug 7 23:30:33.264110248 (13h50m32.853455225s) [noise=true , address=104.36.82.52:4086 , piece_id=JLEUROLLGI7X6MUOVFMEWIFCQNTYJXSZW2JMHGEAI66XREHK5LLQ, seg_pos={0 0}, upload=14346]: Done uploading piece. err: <nil> Aug 7 23:30:33.327068230 (13h50m32.916413217s) [noise=true , address=104.36.82.8:4995 , piece_id=NS2X5RBAQQWBIMO2U4XYCYA37GVF6OBAO3GZSTEPJZOXRMPMS62A, seg_pos={0 0}, upload=14346]: Done uploading piece. err: <nil> Aug 7 23:30:33.366778193 (13h50m32.956123181s) [noise=true , address=104.36.82.60:4353 , piece_id=3TWE6IJ3OCY5CMMRHZ5DG2WBOHLWF3MEESPOGUMU3YA5NSGW3AKA, seg_pos={0 0}, upload=14346]: Done uploading piece. err: <nil> Aug 7 23:30:33.542767028 (13h50m33.132112016s) [noise=true , address=104.36.82.46:4489 , piece_id=KYHATGRQXZ6XXQFUSTYUYBI26BTC7VSN5N6OM766B3QOSWN747IA, seg_pos={0 0}, upload=14346]: Done uploading piece. err: <nil> Aug 7 23:30:33.575265094 (13h50m33.164610081s) [noise=true , address=104.36.82.22:4534 , piece_id=5BROJMGAIDCZNPDEQH32QVVYD2JP4U7IN5DCDUP7LBMU2E6Z36CA, seg_pos={0 0}, upload=14346]: Done uploading piece. err: <nil> Aug 7 23:30:33.672325632 (13h50m33.261670619s) [noise=true , address=77.90.31.17:4797 , piece_id=P6M6EN2C7M6FVKVWV4LE6JEHSZTB24YEACVG7TMSNBQFLD6NLKDQ, seg_pos={0 0}, upload=14346]: Done uploading piece. err: <nil> Aug 7 23:30:33.750650465 (13h50m33.339995452s) [noise=true , address=45.91.116.11:4149 , piece_id=JGSWIYMTFANVCD326G2TRN24X26XYQQSJT6HFWBQAHJLQ2SLBA4A, seg_pos={0 0}, upload=14346]: Done uploading piece. err: <nil> Aug 7 23:30:33.750672889 (13h50m33.340017876s) [seg_pos={0 0}, upload=14346]: Segment reached optimal threshold of 14 pieces. Aug 7 23:30:33.750785674 (13h50m33.340130661s) [noise=true , address=104.36.82.47:4534 , piece_id=PJCWOCQ7UKEHHSERR6C5RQC4Y3TBZRHZF6KPMZIMTAFS6E4525EA, seg_pos={0 0}, upload=14346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VDrNGcCnaKEmG5NN4nex1JoZAgdCEfX5CdsKbxytfLfKzPbr): protocol: expected piece hash; context canceled; EOF Aug 7 23:30:33.750885715 (13h50m33.340230702s) [noise=true , address=104.36.82.61:4392 , piece_id=OSMRSZL5LDLYLPM3KN5YVIMBU44HS72BHZOAB7W5SS5Z4DY4F4JA, seg_pos={0 0}, upload=14346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14FTsLbtv1vp43aRdDZZBY2Gf6epidM1A8HSyQKVokXGFFZGZk): protocol: expected piece hash; context canceled; EOF Aug 7 23:30:33.750979806 (13h50m33.340324793s) [noise=true , address=104.36.82.62:4428 , piece_id=P7EOR2HLYCVOBQBDUZ3OYNVO4RV7WRXLHVREWKIKE7SHYBV3VRVQ, seg_pos={0 0}, upload=14346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Ui5Ye48cKNGFPakMKsjdBi13Xo7bzrmXPNEmRBLKwpMtVAA3R): protocol: rpc: dial tcp 104.36.82.62:4428: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 23:30:33.751073955 (13h50m33.340418942s) [noise=true , address=104.36.82.44:4883 , piece_id=7GULYV2REOYLHXZXRSPIQM46DRT4YKT7FXC3YCHAP7W7332RRTRA, seg_pos={0 0}, upload=14346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cx1avL4umDm2pzJS1rNZqeDZjZMWnMtNsyNTdMdkxhpa1ByUN): protocol: context canceled; context canceled Aug 7 23:30:33.751119673 (13h50m33.34046466s) [seg_pos={0 0}, upload=14346]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 23:30:33.751224474 (13h50m33.340569471s) [upload=14346]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 23:30:33.751230776 (13h50m33.340575763s) [upload=14346]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:30:34.056540019 (13h50m33.645885006s) [upload=14346]: Done uploading segments... Aug 7 23:30:34.056559826 (13h50m33.645904823s) [upload=14346]: Upload finished. err: <nil> Duration: 5.946183057s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 8 out of 60 is failed (threshold=3) [................................XXXXXX.X.X..................] duration: 5.493532412s FAIL (2025-08-07 23:37:11 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.493555374s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 4 out of 60 is failed (threshold=3) [..............................XXXX..........................] duration: 1.860568147s FAIL (2025-08-07 23:37:01 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== .175884539 (13h29m53.765229526s) [upload=14007]: Starting upload Aug 7 23:09:54.175937640 (13h29m53.765282618s) [upload=14007]: Done starting upload Aug 7 23:09:54.175979842 (13h29m53.765324829s) [upload=14007]: Uploading segments... Aug 7 23:09:54.176041389 (13h29m53.765386376s) [upload=14007]: Got next segment. Inline: true Aug 7 23:09:54.176045487 (13h29m53.765390474s) [upload=14007]: Waiting for error group managing segments... Aug 7 23:09:54.176077538 (13h29m53.765422525s) [upload=14007]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:09:54.176082318 (13h29m53.765427305s) [upload=14007]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:09:54.176084522 (13h29m53.765429509s) [upload=14007]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:09:54.453731274 (13h29m54.043076261s) [upload=14007]: Done uploading segments... Aug 7 23:09:54.453750932 (13h29m54.043095909s) [upload=14007]: Upload finished. err: <nil> Aug 7 23:10:39.330325211 (13h30m38.919670198s) [upload=14021]: Starting upload Aug 7 23:10:39.330417358 (13h30m38.919762345s) [upload=14021]: Done starting upload Aug 7 23:10:39.330504384 (13h30m38.919849371s) [upload=14021]: Uploading segments... Aug 7 23:10:39.330543368 (13h30m38.919888355s) [upload=14021]: Got next segment. Inline: true Aug 7 23:10:39.330546675 (13h30m38.919891662s) [upload=14021]: Waiting for error group managing segments... Aug 7 23:10:39.330581371 (13h30m38.919926349s) [upload=14021]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:10:39.330585309 (13h30m38.919930286s) [upload=14021]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:10:39.330587663 (13h30m38.91993265s) [upload=14021]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:10:39.610133078 (13h30m39.199478065s) [upload=14021]: Done uploading segments... Aug 7 23:10:39.610150061 (13h30m39.199495048s) [upload=14021]: Upload finished. err: <nil> Aug 7 23:11:32.695175239 (13h31m32.284520226s) [upload=14034]: Starting upload Aug 7 23:11:32.695285580 (13h31m32.284630567s) [upload=14034]: Done starting upload Aug 7 23:11:32.695336297 (13h31m32.284681284s) [upload=14034]: Uploading segments... Aug 7 23:11:32.695365753 (13h31m32.28471074s) [upload=14034]: Got next segment. Inline: true Aug 7 23:11:32.695369610 (13h31m32.284714598s) [upload=14034]: Waiting for error group managing segments... Aug 7 23:11:32.695401581 (13h31m32.284746569s) [upload=14034]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:11:32.695405168 (13h31m32.284750155s) [upload=14034]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:11:32.695407543 (13h31m32.28475253s) [upload=14034]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:11:32.954750627 (13h31m32.544095614s) [upload=14034]: Done uploading segments... Aug 7 23:11:32.954767650 (13h31m32.544112627s) [upload=14034]: Upload finished. err: <nil> Aug 7 23:12:37.502051138 (13h32m37.091396125s) [upload=14054]: Starting upload Aug 7 23:12:37.502104370 (13h32m37.091449357s) [upload=14054]: Done starting upload Aug 7 23:12:37.502150869 (13h32m37.091495856s) [upload=14054]: Uploading segments... Aug 7 23:12:37.502182519 (13h32m37.091527506s) [upload=14054]: Got next segment. Inline: true Aug 7 23:12:37.502185986 (13h32m37.091530963s) [upload=14054]: Waiting for error group managing segments... Aug 7 23:12:37.502216213 (13h32m37.0915612s) [upload=14054]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:12:37.502219780 (13h32m37.091564767s) [upload=14054]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:12:37.502222425 (13h32m37.091567412s) [upload=14054]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:12:37.772497714 (13h32m37.361842701s) [upload=14054]: Done uploading segments... Aug 7 23:12:37.772511151 (13h32m37.361856138s) [upload=14054]: Upload finished. err: <nil> Aug 7 23:13:44.834965069 (13h33m44.424310046s) [upload=14070]: Starting upload Aug 7 23:13:44.835030293 (13h33m44.42437528s) [upload=14070]: Done starting upload Aug 7 23:13:44.835093484 (13h33m44.424438471s) [upload=14070]: Uploading segments... Aug 7 23:13:44.835133321 (13h33m44.424478308s) [upload=14070]: Got next segment. Inline: true Aug 7 23:13:44.835136998 (13h33m44.424481985s) [upload=14070]: Waiting for error group managing segments... Aug 7 23:13:44.835214376 (13h33m44.424559363s) [upload=14070]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:13:44.835220728 (13h33m44.424565715s) [upload=14070]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:13:44.835222972 (13h33m44.424567959s) [upload=14070]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:13:45.220889346 (13h33m44.810234343s) [upload=14070]: Done uploading segments... Aug 7 23:13:45.220937208 (13h33m44.810282205s) [upload=14070]: Upload finished. err: <nil> Aug 7 23:14:39.930436267 (13h34m39.519781254s) [upload=14085]: Starting upload Aug 7 23:14:39.930523634 (13h34m39.519868621s) [upload=14085]: Done starting upload Aug 7 23:14:39.930574131 (13h34m39.519919118s) [upload=14085]: Uploading segments... Aug 7 23:14:39.930614779 (13h34m39.519959766s) [upload=14085]: Got next segment. Inline: true Aug 7 23:14:39.930618847 (13h34m39.519963824s) [upload=14085]: Waiting for error group managing segments... Aug 7 23:14:39.930656519 (13h34m39.520001506s) [upload=14085]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:14:39.930660767 (13h34m39.520005754s) [upload=14085]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:14:39.930663112 (13h34m39.520008099s) [upload=14085]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:14:40.207143612 (13h34m39.796488599s) [upload=14085]: Done uploading segments... Aug 7 23:14:40.207161857 (13h34m39.796506844s) [upload=14085]: Upload finished. err: <nil> Aug 7 23:14:59.171620401 (13h34m58.760965388s) [upload=14091]: Starting upload Aug 7 23:14:59.171676218 (13h34m58.761021205s) [upload=14091]: Done starting upload Aug 7 23:14:59.171718468 (13h34m58.761063455s) [upload=14091]: Uploading segments... Aug 7 23:14:59.171763204 (13h34m58.761108191s) [upload=14091]: Got next segment. Inline: true Aug 7 23:14:59.171776990 (13h34m58.761121977s) [upload=14091]: Waiting for error group managing segments... Aug 7 23:14:59.171816787 (13h34m58.761161774s) [upload=14091]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:14:59.171820764 (13h34m58.761165741s) [upload=14091]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:14:59.171823429 (13h34m58.761168416s) [upload=14091]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:15:28.519847650 (13h35m28.109192637s) [upload=14091]: Done uploading segments... Aug 7 23:15:28.519870794 (13h35m28.109215781s) [upload=14091]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:15:34.319815009 (13h35m33.909159996s) [upload=14100]: Starting upload Aug 7 23:15:34.319867980 (13h35m33.909212967s) [upload=14100]: Done starting upload Aug 7 23:15:34.319916702 (13h35m33.909261679s) [upload=14100]: Uploading segments... Aug 7 23:15:34.319945739 (13h35m33.909290726s) [upload=14100]: Got next segment. Inline: true Aug 7 23:15:34.319949416 (13h35m33.909294403s) [upload=14100]: Waiting for error group managing segments... Aug 7 23:15:34.319988731 (13h35m33.909333718s) [upload=14100]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:15:34.319992698 (13h35m33.909337685s) [upload=14100]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:15:34.319995163 (13h35m33.90934015s) [upload=14100]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:16:03.672763058 (13h36m3.262108045s) [upload=14100]: Done uploading segments... Aug 7 23:16:03.672786503 (13h36m3.26213149s) [upload=14100]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:16:43.252280188 (13h36m42.841625175s) [upload=14114]: Starting upload Aug 7 23:16:43.252335193 (13h36m42.84168018s) [upload=14114]: Done starting upload Aug 7 23:16:43.252381041 (13h36m42.841726018s) [upload=14114]: Uploading segments... Aug 7 23:16:43.252412271 (13h36m42.841757258s) [upload=14114]: Got next segment. Inline: true Aug 7 23:16:43.252416208 (13h36m42.841761195s) [upload=14114]: Waiting for error group managing segments... Aug 7 23:16:43.252453641 (13h36m42.841798628s) [upload=14114]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:16:43.252457388 (13h36m42.841802375s) [upload=14114]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:16:43.252459622 (13h36m42.841804609s) [upload=14114]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:17:12.607365819 (13h37m12.196710806s) [upload=14114]: Done uploading segments... Aug 7 23:17:12.607385377 (13h37m12.196730364s) [upload=14114]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:17:56.238720000 (13h37m55.828064987s) [upload=14132]: Starting upload Aug 7 23:17:56.238785927 (13h37m55.828130914s) [upload=14132]: Done starting upload Aug 7 23:17:56.238851903 (13h37m55.82819689s) [upload=14132]: Uploading segments... Aug 7 23:17:56.238885677 (13h37m55.828230664s) [upload=14132]: Got next segment. Inline: true Aug 7 23:17:56.238888933 (13h37m55.82823392s) [upload=14132]: Waiting for error group managing segments... Aug 7 23:17:56.238932207 (13h37m55.828277194s) [upload=14132]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:17:56.238935984 (13h37m55.828280971s) [upload=14132]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:17:56.238938278 (13h37m55.828283265s) [upload=14132]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:18:25.593341271 (13h38m25.182686258s) [upload=14132]: Done uploading segments... Aug 7 23:18:25.593355268 (13h38m25.182700245s) [upload=14132]: Upload finished. err: metaclient: context deadline exceeded Duration: 1.860601853s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)