Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 3 out of 60 is failed (threshold=3) [...X..........X.................X...........................] duration: 1.822250891s FAIL (2025-08-07 22:13:26 UTC) orbiter warning: operation: ListObjects ======ERROR====== unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/1K ======LOGS====== 8m27.919686751s) [upload=12612]: Done uploading segments... Aug 7 21:48:28.330358527 (12h8m27.919703514s) [upload=12612]: Upload finished. err: <nil> Aug 7 21:48:54.943937458 (12h8m54.533282445s) [upload=12618]: Starting upload Aug 7 21:48:54.944002622 (12h8m54.533347609s) [upload=12618]: Done starting upload Aug 7 21:48:54.944106100 (12h8m54.533451087s) [upload=12618]: Uploading segments... Aug 7 21:48:54.944196242 (12h8m54.53354132s) [upload=12618]: Got next segment. Inline: true Aug 7 21:48:54.944206543 (12h8m54.53355158s) [upload=12618]: Waiting for error group managing segments... Aug 7 21:48:54.944254484 (12h8m54.533599461s) [upload=12618]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:48:54.944259223 (12h8m54.5336042s) [upload=12618]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:48:54.944262139 (12h8m54.533607126s) [upload=12618]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:48:55.330753187 (12h8m54.920098174s) [upload=12618]: Done uploading segments... Aug 7 21:48:55.330774959 (12h8m54.920119936s) [upload=12618]: Upload finished. err: <nil> Aug 7 21:49:36.650951942 (12h9m36.240296929s) [upload=12631]: Starting upload Aug 7 21:49:36.650993252 (12h9m36.240338239s) [upload=12631]: Done starting upload Aug 7 21:49:36.651049029 (12h9m36.240394016s) [upload=12631]: Uploading segments... Aug 7 21:49:36.651075309 (12h9m36.240420286s) [upload=12631]: Got next segment. Inline: true Aug 7 21:49:36.651078775 (12h9m36.240423763s) [upload=12631]: Waiting for error group managing segments... Aug 7 21:49:36.651115776 (12h9m36.240460773s) [upload=12631]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:49:36.651120254 (12h9m36.240465251s) [upload=12631]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:49:36.651124473 (12h9m36.24046947s) [upload=12631]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:49:36.920683881 (12h9m36.510028868s) [upload=12631]: Done uploading segments... Aug 7 21:49:36.920707246 (12h9m36.510052233s) [upload=12631]: Upload finished. err: <nil> Aug 7 21:50:03.363429937 (12h10m2.952774924s) [upload=12636]: Starting upload Aug 7 21:50:03.363493408 (12h10m2.952838405s) [upload=12636]: Done starting upload Aug 7 21:50:03.363582518 (12h10m2.952927515s) [upload=12636]: Uploading segments... Aug 7 21:50:03.363692078 (12h10m2.953037085s) [upload=12636]: Got next segment. Inline: true Aug 7 21:50:03.363699752 (12h10m2.953044739s) [upload=12636]: Waiting for error group managing segments... Aug 7 21:50:03.363764116 (12h10m2.953109103s) [upload=12636]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:50:03.363773323 (12h10m2.95311831s) [upload=12636]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:50:03.363777271 (12h10m2.953122258s) [upload=12636]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:50:03.636022787 (12h10m3.225367774s) [upload=12636]: Done uploading segments... Aug 7 21:50:03.636042204 (12h10m3.225387202s) [upload=12636]: Upload finished. err: <nil> Aug 7 21:50:20.944550175 (12h10m20.533895152s) [upload=12642]: Starting upload Aug 7 21:50:20.944594911 (12h10m20.533939898s) [upload=12642]: Done starting upload Aug 7 21:50:20.944644405 (12h10m20.533989392s) [upload=12642]: Uploading segments... Aug 7 21:50:20.944674473 (12h10m20.534019461s) [upload=12642]: Got next segment. Inline: true Aug 7 21:50:20.944677820 (12h10m20.534022807s) [upload=12642]: Waiting for error group managing segments... Aug 7 21:50:20.944709450 (12h10m20.534054437s) [upload=12642]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:50:20.944715131 (12h10m20.534060118s) [upload=12642]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:50:20.944717505 (12h10m20.534062493s) [upload=12642]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:50:21.212077476 (12h10m20.801422473s) [upload=12642]: Done uploading segments... Aug 7 21:50:21.212101602 (12h10m20.801446599s) [upload=12642]: Upload finished. err: <nil> Aug 7 21:51:22.649621444 (12h11m22.238966431s) [upload=12660]: Starting upload Aug 7 21:51:22.649696837 (12h11m22.239041824s) [upload=12660]: Done starting upload Aug 7 21:51:22.649777742 (12h11m22.239122729s) [upload=12660]: Uploading segments... Aug 7 21:51:22.649867044 (12h11m22.239212041s) [upload=12660]: Got next segment. Inline: true Aug 7 21:51:22.649872053 (12h11m22.23921704s) [upload=12660]: Waiting for error group managing segments... Aug 7 21:51:22.649940053 (12h11m22.23928504s) [upload=12660]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:51:22.649946785 (12h11m22.239291772s) [upload=12660]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:51:22.649949932 (12h11m22.239294919s) [upload=12660]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:51:22.914527578 (12h11m22.503872565s) [upload=12660]: Done uploading segments... Aug 7 21:51:22.914549130 (12h11m22.503894117s) [upload=12660]: Upload finished. err: <nil> Aug 7 21:51:38.628060070 (12h11m38.217405057s) [upload=12665]: Starting upload Aug 7 21:51:38.628100548 (12h11m38.217445535s) [upload=12665]: Done starting upload Aug 7 21:51:38.628155373 (12h11m38.21750036s) [upload=12665]: Uploading segments... Aug 7 21:51:38.628187814 (12h11m38.217532802s) [upload=12665]: Got next segment. Inline: true Aug 7 21:51:38.628191381 (12h11m38.217536368s) [upload=12665]: Waiting for error group managing segments... Aug 7 21:51:38.628222691 (12h11m38.217567678s) [upload=12665]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:51:38.628226268 (12h11m38.217571255s) [upload=12665]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:51:38.628228662 (12h11m38.217573649s) [upload=12665]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:51:38.940125222 (12h11m38.529470269s) [upload=12665]: Done uploading segments... Aug 7 21:51:38.940147124 (12h11m38.529492191s) [upload=12665]: Upload finished. err: <nil> Aug 7 21:52:16.262879794 (12h12m15.852224781s) [upload=12676]: Starting upload Aug 7 21:52:16.262939168 (12h12m15.852284155s) [upload=12676]: Done starting upload Aug 7 21:52:16.263047876 (12h12m15.852392873s) [upload=12676]: Uploading segments... Aug 7 21:52:16.263139601 (12h12m15.852484598s) [upload=12676]: Got next segment. Inline: true Aug 7 21:52:16.263144440 (12h12m15.852489417s) [upload=12676]: Waiting for error group managing segments... Aug 7 21:52:16.263191931 (12h12m15.852536918s) [upload=12676]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:52:16.263197472 (12h12m15.852542449s) [upload=12676]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:52:16.263200056 (12h12m15.852545043s) [upload=12676]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:52:16.570506954 (12h12m16.159851931s) [upload=12676]: Done uploading segments... Aug 7 21:52:16.570524978 (12h12m16.159869965s) [upload=12676]: Upload finished. err: <nil> Aug 7 21:53:00.574165558 (12h13m0.163510545s) [upload=12691]: Starting upload Aug 7 21:53:00.574212286 (12h13m0.163557273s) [upload=12691]: Done starting upload Aug 7 21:53:00.574248476 (12h13m0.163593463s) [upload=12691]: Uploading segments... Aug 7 21:53:00.574327307 (12h13m0.163672294s) [upload=12691]: Got next segment. Inline: true Aug 7 21:53:00.574332868 (12h13m0.163677855s) [upload=12691]: Waiting for error group managing segments... Aug 7 21:53:00.574373505 (12h13m0.163718492s) [upload=12691]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:53:00.574390598 (12h13m0.163735585s) [upload=12691]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:53:00.574393373 (12h13m0.16373835s) [upload=12691]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:53:00.840335180 (12h13m0.429680167s) [upload=12691]: Done uploading segments... Aug 7 21:53:00.840351500 (12h13m0.429696487s) [upload=12691]: Upload finished. err: <nil> Aug 7 21:53:46.136340579 (12h13m45.725685566s) [upload=12705]: Starting upload Aug 7 21:53:46.136382029 (12h13m45.725727016s) [upload=12705]: Done starting upload Aug 7 21:53:46.136495205 (12h13m45.725840192s) [upload=12705]: Uploading segments... Aug 7 21:53:46.136529601 (12h13m45.725874588s) [upload=12705]: Got next segment. Inline: true Aug 7 21:53:46.136533228 (12h13m45.725878215s) [upload=12705]: Waiting for error group managing segments... Aug 7 21:53:46.136581701 (12h13m45.725926688s) [upload=12705]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:53:46.136585949 (12h13m45.725930936s) [upload=12705]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:53:46.136588393 (12h13m45.72593338s) [upload=12705]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:53:46.427021148 (12h13m46.016366155s) [upload=12705]: Done uploading segments... Aug 7 21:53:46.427045194 (12h13m46.016390181s) [upload=12705]: Upload finished. err: <nil> Aug 7 21:54:41.670591536 (12h14m41.259936523s) [upload=12721]: Starting upload Aug 7 21:54:41.670632916 (12h14m41.259977903s) [upload=12721]: Done starting upload Aug 7 21:54:41.670677612 (12h14m41.260022599s) [upload=12721]: Uploading segments... Aug 7 21:54:41.670708571 (12h14m41.260053558s) [upload=12721]: Got next segment. Inline: true Aug 7 21:54:41.670713510 (12h14m41.260058497s) [upload=12721]: Waiting for error group managing segments... Aug 7 21:54:41.670746032 (12h14m41.260091019s) [upload=12721]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:54:41.670749979 (12h14m41.260094966s) [upload=12721]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:54:41.670752424 (12h14m41.260097411s) [upload=12721]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:54:41.940759391 (12h14m41.530104368s) [upload=12721]: Done uploading segments... Aug 7 21:54:41.940776734 (12h14m41.530121721s) [upload=12721]: Upload finished. err: <nil> Duration: 1.822270578s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 21 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXX.X...X...................................XX] duration: 30.001101883s FAIL (2025-08-07 22:13:22 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== h25m35.922756853s) [upload=12901]: Upload finished. err: <nil> Aug 7 22:06:23.632385139 (12h26m23.221730126s) [upload=12916]: Starting upload Aug 7 22:06:23.632456987 (12h26m23.221801974s) [upload=12916]: Done starting upload Aug 7 22:06:23.632524355 (12h26m23.221869342s) [upload=12916]: Uploading segments... Aug 7 22:06:23.632558691 (12h26m23.221903679s) [upload=12916]: Got next segment. Inline: true Aug 7 22:06:23.632562028 (12h26m23.221907015s) [upload=12916]: Waiting for error group managing segments... Aug 7 22:06:23.632596073 (12h26m23.22194105s) [upload=12916]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:06:23.632600020 (12h26m23.221945007s) [upload=12916]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:06:23.632602304 (12h26m23.221947281s) [upload=12916]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:06:23.903184888 (12h26m23.492529875s) [upload=12916]: Done uploading segments... Aug 7 22:06:23.903202192 (12h26m23.492547179s) [upload=12916]: Upload finished. err: <nil> Aug 7 22:07:18.062211999 (12h27m17.651556986s) [upload=12928]: Starting upload Aug 7 22:07:18.062271193 (12h27m17.651616181s) [upload=12928]: Done starting upload Aug 7 22:07:18.062330397 (12h27m17.651675374s) [upload=12928]: Uploading segments... Aug 7 22:07:18.062360224 (12h27m17.651705211s) [upload=12928]: Got next segment. Inline: true Aug 7 22:07:18.062363690 (12h27m17.651708677s) [upload=12928]: Waiting for error group managing segments... Aug 7 22:07:18.062394940 (12h27m17.651739917s) [upload=12928]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:07:18.062398466 (12h27m17.651743453s) [upload=12928]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:07:18.062400671 (12h27m17.651745648s) [upload=12928]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:07:18.332201201 (12h27m17.921546188s) [upload=12928]: Done uploading segments... Aug 7 22:07:18.332221129 (12h27m17.921566116s) [upload=12928]: Upload finished. err: <nil> Aug 7 22:08:06.628735015 (12h28m6.218080002s) [upload=12949]: Starting upload Aug 7 22:08:06.628795140 (12h28m6.218140127s) [upload=12949]: Done starting upload Aug 7 22:08:06.628857911 (12h28m6.218202898s) [upload=12949]: Uploading segments... Aug 7 22:08:06.628887237 (12h28m6.218232224s) [upload=12949]: Got next segment. Inline: true Aug 7 22:08:06.628890713 (12h28m6.2182357s) [upload=12949]: Waiting for error group managing segments... Aug 7 22:08:06.628923866 (12h28m6.218268853s) [upload=12949]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:08:06.628935168 (12h28m6.218280195s) [upload=12949]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:08:06.628938295 (12h28m6.218283272s) [upload=12949]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:08:06.909068236 (12h28m6.498413233s) [upload=12949]: Done uploading segments... Aug 7 22:08:06.909091591 (12h28m6.498436598s) [upload=12949]: Upload finished. err: <nil> Aug 7 22:08:44.103027985 (12h28m43.692372972s) [upload=12960]: Starting upload Aug 7 22:08:44.103093421 (12h28m43.692438408s) [upload=12960]: Done starting upload Aug 7 22:08:44.103142554 (12h28m43.692487541s) [upload=12960]: Uploading segments... Aug 7 22:08:44.103198422 (12h28m43.692543419s) [upload=12960]: Got next segment. Inline: true Aug 7 22:08:44.103202429 (12h28m43.692547416s) [upload=12960]: Waiting for error group managing segments... Aug 7 22:08:44.103235592 (12h28m43.69258058s) [upload=12960]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:08:44.103239219 (12h28m43.692584206s) [upload=12960]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:08:44.103241373 (12h28m43.69258636s) [upload=12960]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:08:44.393696841 (12h28m43.983041818s) [upload=12960]: Done uploading segments... Aug 7 22:08:44.393715166 (12h28m43.983060143s) [upload=12960]: Upload finished. err: <nil> Aug 7 22:09:14.345236113 (12h29m13.9345811s) [upload=12972]: Starting upload Aug 7 22:09:14.345291719 (12h29m13.934636706s) [upload=12972]: Done starting upload Aug 7 22:09:14.345358137 (12h29m13.934703114s) [upload=12972]: Uploading segments... Aug 7 22:09:14.345388645 (12h29m13.934733632s) [upload=12972]: Got next segment. Inline: true Aug 7 22:09:14.345392141 (12h29m13.934737118s) [upload=12972]: Waiting for error group managing segments... Aug 7 22:09:14.345426327 (12h29m13.934771314s) [upload=12972]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:09:14.345430324 (12h29m13.934775301s) [upload=12972]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:09:14.345432839 (12h29m13.934777826s) [upload=12972]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:09:14.834006850 (12h29m14.423351857s) [upload=12972]: Done uploading segments... Aug 7 22:09:14.834026347 (12h29m14.423371334s) [upload=12972]: Upload finished. err: <nil> Aug 7 22:09:53.131289328 (12h29m52.720634315s) [upload=12982]: Starting upload Aug 7 22:09:53.131364442 (12h29m52.720709489s) [upload=12982]: Done starting upload Aug 7 22:09:53.131436210 (12h29m52.720781197s) [upload=12982]: Uploading segments... Aug 7 22:09:53.131465205 (12h29m52.720810192s) [upload=12982]: Got next segment. Inline: true Aug 7 22:09:53.131468902 (12h29m52.720813889s) [upload=12982]: Waiting for error group managing segments... Aug 7 22:09:53.131498809 (12h29m52.720843796s) [upload=12982]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:09:53.131503017 (12h29m52.720848004s) [upload=12982]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:09:53.131505141 (12h29m52.720850118s) [upload=12982]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:09:53.419040148 (12h29m53.008385125s) [upload=12982]: Done uploading segments... Aug 7 22:09:53.419058042 (12h29m53.008403029s) [upload=12982]: Upload finished. err: <nil> Aug 7 22:10:12.329552115 (12h30m11.918897102s) [upload=12989]: Starting upload Aug 7 22:10:12.330067942 (12h30m11.919412929s) [upload=12989]: Done starting upload Aug 7 22:10:12.330139077 (12h30m11.919484054s) [upload=12989]: Uploading segments... Aug 7 22:10:12.330186228 (12h30m11.919531255s) [upload=12989]: Got next segment. Inline: true Aug 7 22:10:12.330192710 (12h30m11.919537737s) [upload=12989]: Waiting for error group managing segments... Aug 7 22:10:12.330240762 (12h30m11.919585749s) [upload=12989]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:10:12.330248126 (12h30m11.919593113s) [upload=12989]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:10:12.330252083 (12h30m11.91959709s) [upload=12989]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:10:12.591107606 (12h30m12.180452593s) [upload=12989]: Done uploading segments... Aug 7 22:10:12.591124920 (12h30m12.180469907s) [upload=12989]: Upload finished. err: <nil> Aug 7 22:10:57.486329607 (12h30m57.075674594s) [upload=13003]: Starting upload Aug 7 22:10:57.486414950 (12h30m57.075759927s) [upload=13003]: Done starting upload Aug 7 22:10:57.486491847 (12h30m57.075836834s) [upload=13003]: Uploading segments... Aug 7 22:10:57.486573694 (12h30m57.075918681s) [upload=13003]: Got next segment. Inline: true Aug 7 22:10:57.486578242 (12h30m57.07592322s) [upload=13003]: Waiting for error group managing segments... Aug 7 22:10:57.486645681 (12h30m57.075990658s) [upload=13003]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:10:57.486650361 (12h30m57.075995348s) [upload=13003]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:10:57.486652795 (12h30m57.075997782s) [upload=13003]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:10:57.751495816 (12h30m57.340840803s) [upload=13003]: Done uploading segments... Aug 7 22:10:57.751515013 (12h30m57.34086s) [upload=13003]: Upload finished. err: <nil> Aug 7 22:11:44.715949867 (12h31m44.305294854s) [upload=13015]: Starting upload Aug 7 22:11:44.716016213 (12h31m44.3053612s) [upload=13015]: Done starting upload Aug 7 22:11:44.716066600 (12h31m44.305411587s) [upload=13015]: Uploading segments... Aug 7 22:11:44.716104322 (12h31m44.305449309s) [upload=13015]: Got next segment. Inline: true Aug 7 22:11:44.716107879 (12h31m44.305452866s) [upload=13015]: Waiting for error group managing segments... Aug 7 22:11:44.716139289 (12h31m44.305484276s) [upload=13015]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:11:44.716142826 (12h31m44.305487803s) [upload=13015]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:11:44.716146493 (12h31m44.30549147s) [upload=13015]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:12:14.067179909 (12h32m13.656524896s) [upload=13015]: Done uploading segments... Aug 7 22:12:14.067200478 (12h32m13.656545465s) [upload=13015]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:12:53.398168829 (12h32m52.987513816s) [upload=13035]: Starting upload Aug 7 22:12:53.398221350 (12h32m52.987566337s) [upload=13035]: Done starting upload Aug 7 22:12:53.398263170 (12h32m52.987608147s) [upload=13035]: Uploading segments... Aug 7 22:12:53.398291284 (12h32m52.987636261s) [upload=13035]: Got next segment. Inline: true Aug 7 22:12:53.398295111 (12h32m52.987640088s) [upload=13035]: Waiting for error group managing segments... Aug 7 22:12:53.398338094 (12h32m52.987683081s) [upload=13035]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:12:53.398344366 (12h32m52.987689363s) [upload=13035]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:12:53.398348754 (12h32m52.987693771s) [upload=13035]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:13:22.757670742 (12h33m22.347015729s) [upload=13035]: Done uploading segments... Aug 7 22:13:22.757697282 (12h33m22.34704227s) [upload=13035]: Upload finished. err: metaclient: context deadline exceeded Duration: 30.001122822s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 25 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXX.X.X.X................................XX] duration: 30.00124303s FAIL (2025-08-07 22:13:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== 81s) [seg_pos={0 0}, upload=13000]: Done begin upload segment. Aug 7 22:10:53.209625824 (12h30m52.798970811s) [noise=true , address=77.90.31.12:4590 , piece_id=QCC2XA4NPGVTUZXRRNXV22QVU77S6QB6R4QEQIYUAFWVON3J35RQ, seg_pos={0 0}, upload=13000]: Uploading piece... Aug 7 22:10:53.209635153 (12h30m52.79898014s) [noise=true , address=104.36.82.4:4804 , piece_id=BI6ZPR56BKHTA2QGM3BNUBFJM6PAD4K27P5TYHRGLG4NBD56LC4A, seg_pos={0 0}, upload=13000]: Uploading piece... Aug 7 22:10:53.209952860 (12h30m52.799297837s) [noise=true , address=45.91.116.4:4939 , piece_id=FQ4Z6Y4KIOGJLPCZFVPCUJSYOMGDOTZNAIEKGNWKASCLKOWLCTUQ, seg_pos={0 0}, upload=13000]: Uploading piece... Aug 7 22:10:53.209970634 (12h30m52.799315621s) [noise=true , address=77.90.31.10:4520 , piece_id=KSMIVSJ4QSOZAXZN7EVTTC4DGFR2TL4JLJQW2NPYYGWAJDKSZ2ZA, seg_pos={0 0}, upload=13000]: Uploading piece... Aug 7 22:10:53.210195384 (12h30m52.799540371s) [noise=true , address=104.36.82.4:4802 , piece_id=B4Q65J747GHE2ZRZDM7RLWC3AQC7TLK5D53REAUA5DMHT3TGA44Q, seg_pos={0 0}, upload=13000]: Uploading piece... Aug 7 22:10:53.210227876 (12h30m52.799572873s) [noise=true , address=77.90.31.18:4805 , piece_id=VQ3CGKCPEKYPCAIOHQ2FZSINJBHBCCQUZD2AD4AIHX72SZN7SEKA, seg_pos={0 0}, upload=13000]: Uploading piece... Aug 7 22:10:53.210467886 (12h30m52.799812863s) [noise=true , address=45.91.116.7:4009 , piece_id=47X2WBQ4XYRNAZDBVU572QSRHXUNSYRCOUALM4SF4WRHCDW6QTCA, seg_pos={0 0}, upload=13000]: Uploading piece... Aug 7 22:10:53.210477654 (12h30m52.799822641s) [noise=true , address=77.90.31.13:4627 , piece_id=3IQP6QL756UL7CNL357KLNUTHQCSEG5337GUT2PD2AS623PTIYCA, seg_pos={0 0}, upload=13000]: Uploading piece... Aug 7 22:10:53.210669050 (12h30m52.800014027s) [noise=true , address=77.90.31.17:4770 , piece_id=BG2ZL7XC52NEJC5FRS4V75MEPJH34B27IPLTJNQLSX4N3OZGVHXA, seg_pos={0 0}, upload=13000]: Uploading piece... Aug 7 22:10:53.210729946 (12h30m52.800074934s) [noise=true , address=45.91.116.5:4298 , piece_id=XJIIUMO7PI6XTBLXE7IRPS2YYI53MRGPEW7HS5CWFXUSIXBT5MBQ, seg_pos={0 0}, upload=13000]: Uploading piece... Aug 7 22:10:53.210967301 (12h30m52.800312288s) [noise=true , address=45.91.116.14:4189 , piece_id=WTRHRDN2HJN3MQI6334ZHVKLMCT7XH7S42MKYCEK5RY2JHDBESOQ, seg_pos={0 0}, upload=13000]: Uploading piece... Aug 7 22:10:53.211206058 (12h30m52.800551045s) [noise=true , address=104.36.82.12:4150 , piece_id=XG7E3FHDR2VCPREGTWFFJ52JLGUPRUKQJ75KFYEUXPI4OUNTB3ZA, seg_pos={0 0}, upload=13000]: Uploading piece... Aug 7 22:10:53.211450766 (12h30m52.800795753s) [noise=true , address=104.36.82.7:4909 , piece_id=6ZPKL2IEZN54MOXLAC7P4GURLWMJ7MUBNCBXPUGF6TD6257G7TAA, seg_pos={0 0}, upload=13000]: Uploading piece... Aug 7 22:10:53.211637383 (12h30m52.80098237s) [noise=true , address=104.36.82.6:4873 , piece_id=FIZQQ56RDZJUYC2JM24K5UMWTQI64SLS56HH54F4EUIXCWH46FOQ, seg_pos={0 0}, upload=13000]: Uploading piece... Aug 7 22:10:53.211831044 (12h30m52.801176031s) [noise=true , address=45.91.116.13:4154 , piece_id=TPFKH6V3LAM6B2O7L744BZPLT64I2BZVYJXFPEPLIGMVEQ7246YQ, seg_pos={0 0}, upload=13000]: Uploading piece... Aug 7 22:10:53.212051075 (12h30m52.801396062s) [noise=true , address=77.90.31.11:4555 , piece_id=A6WWFITSRKE4GVTQODOYU36ORK7OVE3MVVR7B4FPSSP64DATOQ6A, seg_pos={0 0}, upload=13000]: Uploading piece... Aug 7 22:10:54.951706811 (12h30m54.541051848s) [upload=13000]: Next returned nil segment Aug 7 22:10:54.951730396 (12h30m54.541075413s) [upload=13000]: Waiting for error group managing segments... Aug 7 22:10:55.341563384 (12h30m54.930908361s) [noise=true , address=77.90.31.10:4520 , piece_id=KSMIVSJ4QSOZAXZN7EVTTC4DGFR2TL4JLJQW2NPYYGWAJDKSZ2ZA, seg_pos={0 0}, upload=13000]: Done uploading piece. err: <nil> Aug 7 22:10:55.391397870 (12h30m54.980742857s) [noise=true , address=45.91.116.14:4189 , piece_id=WTRHRDN2HJN3MQI6334ZHVKLMCT7XH7S42MKYCEK5RY2JHDBESOQ, seg_pos={0 0}, upload=13000]: Done uploading piece. err: <nil> Aug 7 22:10:55.429722625 (12h30m55.019067602s) [noise=true , address=77.90.31.12:4590 , piece_id=QCC2XA4NPGVTUZXRRNXV22QVU77S6QB6R4QEQIYUAFWVON3J35RQ, seg_pos={0 0}, upload=13000]: Done uploading piece. err: <nil> Aug 7 22:10:55.430446209 (12h30m55.019791196s) [noise=true , address=104.36.82.6:4873 , piece_id=FIZQQ56RDZJUYC2JM24K5UMWTQI64SLS56HH54F4EUIXCWH46FOQ, seg_pos={0 0}, upload=13000]: Done uploading piece. err: <nil> Aug 7 22:10:55.438319623 (12h30m55.0276646s) [noise=true , address=45.91.116.5:4298 , piece_id=XJIIUMO7PI6XTBLXE7IRPS2YYI53MRGPEW7HS5CWFXUSIXBT5MBQ, seg_pos={0 0}, upload=13000]: Done uploading piece. err: <nil> Aug 7 22:10:55.440502008 (12h30m55.029846995s) [noise=true , address=45.91.116.13:4154 , piece_id=TPFKH6V3LAM6B2O7L744BZPLT64I2BZVYJXFPEPLIGMVEQ7246YQ, seg_pos={0 0}, upload=13000]: Done uploading piece. err: <nil> Aug 7 22:10:55.449485475 (12h30m55.038830462s) [noise=true , address=104.36.82.7:4909 , piece_id=6ZPKL2IEZN54MOXLAC7P4GURLWMJ7MUBNCBXPUGF6TD6257G7TAA, seg_pos={0 0}, upload=13000]: Done uploading piece. err: <nil> Aug 7 22:10:55.449773497 (12h30m55.039118484s) [noise=true , address=77.90.31.13:4627 , piece_id=3IQP6QL756UL7CNL357KLNUTHQCSEG5337GUT2PD2AS623PTIYCA, seg_pos={0 0}, upload=13000]: Done uploading piece. err: <nil> Aug 7 22:10:55.486432234 (12h30m55.075777221s) [noise=true , address=77.90.31.11:4555 , piece_id=A6WWFITSRKE4GVTQODOYU36ORK7OVE3MVVR7B4FPSSP64DATOQ6A, seg_pos={0 0}, upload=13000]: Done uploading piece. err: <nil> Aug 7 22:10:55.522908903 (12h30m55.11225389s) [noise=true , address=77.90.31.18:4805 , piece_id=VQ3CGKCPEKYPCAIOHQ2FZSINJBHBCCQUZD2AD4AIHX72SZN7SEKA, seg_pos={0 0}, upload=13000]: Done uploading piece. err: <nil> Aug 7 22:10:55.546118308 (12h30m55.135463295s) [noise=true , address=104.36.82.4:4802 , piece_id=B4Q65J747GHE2ZRZDM7RLWC3AQC7TLK5D53REAUA5DMHT3TGA44Q, seg_pos={0 0}, upload=13000]: Done uploading piece. err: <nil> Aug 7 22:10:55.568801126 (12h30m55.158146113s) [noise=true , address=104.36.82.4:4804 , piece_id=BI6ZPR56BKHTA2QGM3BNUBFJM6PAD4K27P5TYHRGLG4NBD56LC4A, seg_pos={0 0}, upload=13000]: Done uploading piece. err: <nil> Aug 7 22:10:55.587208293 (12h30m55.176553341s) [noise=true , address=77.90.31.17:4770 , piece_id=BG2ZL7XC52NEJC5FRS4V75MEPJH34B27IPLTJNQLSX4N3OZGVHXA, seg_pos={0 0}, upload=13000]: Done uploading piece. err: <nil> Aug 7 22:10:55.587245686 (12h30m55.176590713s) [seg_pos={0 0}, upload=13000]: Segment reached optimal threshold of 13 pieces. Aug 7 22:10:55.587371516 (12h30m55.176716533s) [noise=true , address=45.91.116.4:4939 , piece_id=FQ4Z6Y4KIOGJLPCZFVPCUJSYOMGDOTZNAIEKGNWKASCLKOWLCTUQ, seg_pos={0 0}, upload=13000]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: expected piece hash; context canceled; EOF Aug 7 22:10:55.587442001 (12h30m55.176786988s) [noise=true , address=45.91.116.7:4009 , piece_id=47X2WBQ4XYRNAZDBVU572QSRHXUNSYRCOUALM4SF4WRHCDW6QTCA, seg_pos={0 0}, upload=13000]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF Aug 7 22:10:55.587644678 (12h30m55.176989726s) [noise=true , address=104.36.82.12:4150 , piece_id=XG7E3FHDR2VCPREGTWFFJ52JLGUPRUKQJ75KFYEUXPI4OUNTB3ZA, seg_pos={0 0}, upload=13000]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK): protocol: context canceled; context canceled Aug 7 22:10:55.587681550 (12h30m55.177026557s) [seg_pos={0 0}, upload=13000]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 7 22:10:55.587777072 (12h30m55.177122109s) [upload=13000]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 22:10:55.587787912 (12h30m55.177132929s) [upload=13000]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:10:55.887211865 (12h30m55.476556883s) [upload=13000]: Done uploading segments... Aug 7 22:10:55.887228857 (12h30m55.476573844s) [upload=13000]: Upload finished. err: <nil> Aug 7 22:11:40.872949328 (12h31m40.462294315s) [upload=13014]: Starting upload Aug 7 22:11:40.873003161 (12h31m40.462348148s) [upload=13014]: Done starting upload Aug 7 22:11:40.873072924 (12h31m40.462417911s) [upload=13014]: Uploading segments... Aug 7 22:11:40.873431210 (12h31m40.462776287s) [upload=13014]: Got next segment. Inline: false Aug 7 22:11:40.873471537 (12h31m40.462816525s) [upload=13014]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:11:40.873480314 (12h31m40.462825331s) [upload=13014]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:12:10.229450117 (12h32m9.818795124s) [upload=13014]: Next segment err: context deadline exceeded Aug 7 22:12:10.229589654 (12h32m9.818934631s) [upload=13014]: Done uploading segments... Aug 7 22:12:10.229603972 (12h32m9.818948949s) [upload=13014]: Upload finished. err: context deadline exceeded Aug 7 22:12:48.993762773 (12h32m48.58310776s) [upload=13033]: Starting upload Aug 7 22:12:48.993844269 (12h32m48.583189246s) [upload=13033]: Done starting upload Aug 7 22:12:48.993904404 (12h32m48.583249391s) [upload=13033]: Uploading segments... Aug 7 22:12:48.994022439 (12h32m48.583367427s) [upload=13033]: Got next segment. Inline: false Aug 7 22:12:48.994048309 (12h32m48.583393296s) [upload=13033]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:12:48.994052647 (12h32m48.583397634s) [upload=13033]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:13:18.347381136 (12h33m17.936726133s) [upload=13033]: Next segment err: context deadline exceeded Aug 7 22:13:18.347514681 (12h33m17.936859668s) [upload=13033]: Done uploading segments... Aug 7 22:13:18.347529670 (12h33m17.936874657s) [upload=13033]: Upload finished. err: context deadline exceeded Duration: 30.001262007s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXX.......X................................XX] duration: 30.000333249s FAIL (2025-08-07 22:13:18 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.000345773s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 26 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXX.XX..X...............................XX] duration: 1m0.000927128s FAIL (2025-08-07 22:13:10 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 2h25m20.758607216s) [upload=12897]: Upload finished. err: <nil> Aug 7 22:05:56.583599672 (12h25m56.172944659s) [upload=12908]: Starting upload Aug 7 22:05:56.583668895 (12h25m56.173013882s) [upload=12908]: Done starting upload Aug 7 22:05:56.583716636 (12h25m56.173061623s) [upload=12908]: Uploading segments... Aug 7 22:05:56.583747204 (12h25m56.173092191s) [upload=12908]: Got next segment. Inline: true Aug 7 22:05:56.583750742 (12h25m56.173095729s) [upload=12908]: Waiting for error group managing segments... Aug 7 22:05:56.583785789 (12h25m56.173130776s) [upload=12908]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:05:56.583789445 (12h25m56.173134432s) [upload=12908]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:05:56.583795787 (12h25m56.173140774s) [upload=12908]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:05:56.916665113 (12h25m56.5060101s) [upload=12908]: Done uploading segments... Aug 7 22:05:56.916682988 (12h25m56.506027975s) [upload=12908]: Upload finished. err: <nil> Aug 7 22:06:48.168861264 (12h26m47.758206241s) [upload=12922]: Starting upload Aug 7 22:06:48.168923543 (12h26m47.75826853s) [upload=12922]: Done starting upload Aug 7 22:06:48.168989780 (12h26m47.758334767s) [upload=12922]: Uploading segments... Aug 7 22:06:48.169093067 (12h26m47.758438044s) [upload=12922]: Got next segment. Inline: true Aug 7 22:06:48.169098358 (12h26m47.758443345s) [upload=12922]: Waiting for error group managing segments... Aug 7 22:06:48.169143945 (12h26m47.758488932s) [upload=12922]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:06:48.169149034 (12h26m47.758494021s) [upload=12922]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:06:48.169151550 (12h26m47.758496537s) [upload=12922]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:06:48.443503957 (12h26m48.032848934s) [upload=12922]: Done uploading segments... Aug 7 22:06:48.443520579 (12h26m48.032865556s) [upload=12922]: Upload finished. err: <nil> Aug 7 22:07:24.544076557 (12h27m24.133421534s) [upload=12933]: Starting upload Aug 7 22:07:24.544154457 (12h27m24.133499444s) [upload=12933]: Done starting upload Aug 7 22:07:24.544212037 (12h27m24.133557024s) [upload=12933]: Uploading segments... Aug 7 22:07:24.544273624 (12h27m24.133618611s) [upload=12933]: Got next segment. Inline: true Aug 7 22:07:24.544278614 (12h27m24.133623601s) [upload=12933]: Waiting for error group managing segments... Aug 7 22:07:24.544329832 (12h27m24.133674819s) [upload=12933]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:07:24.544334911 (12h27m24.133679898s) [upload=12933]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:07:24.544337847 (12h27m24.133682834s) [upload=12933]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:07:24.872527048 (12h27m24.461872036s) [upload=12933]: Done uploading segments... Aug 7 22:07:24.872546185 (12h27m24.461891172s) [upload=12933]: Upload finished. err: <nil> Aug 7 22:07:42.678048362 (12h27m42.267393349s) [upload=12938]: Starting upload Aug 7 22:07:42.678122394 (12h27m42.267467381s) [upload=12938]: Done starting upload Aug 7 22:07:42.678184523 (12h27m42.26752951s) [upload=12938]: Uploading segments... Aug 7 22:07:42.678220613 (12h27m42.2675656s) [upload=12938]: Got next segment. Inline: true Aug 7 22:07:42.678225311 (12h27m42.267570299s) [upload=12938]: Waiting for error group managing segments... Aug 7 22:07:42.678261721 (12h27m42.267606708s) [upload=12938]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:07:42.678265758 (12h27m42.267610735s) [upload=12938]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:07:42.678268203 (12h27m42.26761319s) [upload=12938]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:07:42.941204890 (12h27m42.530549877s) [upload=12938]: Done uploading segments... Aug 7 22:07:42.941220821 (12h27m42.530565808s) [upload=12938]: Upload finished. err: <nil> Aug 7 22:08:12.564234560 (12h28m12.153579557s) [upload=12951]: Starting upload Aug 7 22:08:12.564392563 (12h28m12.15373757s) [upload=12951]: Done starting upload Aug 7 22:08:12.564478317 (12h28m12.153823314s) [upload=12951]: Uploading segments... Aug 7 22:08:12.564532671 (12h28m12.153877658s) [upload=12951]: Got next segment. Inline: true Aug 7 22:08:12.564537029 (12h28m12.153882006s) [upload=12951]: Waiting for error group managing segments... Aug 7 22:08:12.564572748 (12h28m12.153917735s) [upload=12951]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:08:12.564577227 (12h28m12.153922204s) [upload=12951]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:08:12.564579511 (12h28m12.153924498s) [upload=12951]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:08:12.853863397 (12h28m12.443208384s) [upload=12951]: Done uploading segments... Aug 7 22:08:12.853882363 (12h28m12.44322735s) [upload=12951]: Upload finished. err: <nil> Aug 7 22:08:48.226332068 (12h28m47.815677055s) [upload=12963]: Starting upload Aug 7 22:08:48.226393765 (12h28m47.815738743s) [upload=12963]: Done starting upload Aug 7 22:08:48.226441998 (12h28m47.815786975s) [upload=12963]: Uploading segments... Aug 7 22:08:48.226471044 (12h28m47.815816031s) [upload=12963]: Got next segment. Inline: true Aug 7 22:08:48.226474330 (12h28m47.815819317s) [upload=12963]: Waiting for error group managing segments... Aug 7 22:08:48.226505239 (12h28m47.815850216s) [upload=12963]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:08:48.226508785 (12h28m47.815853772s) [upload=12963]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:08:48.226511250 (12h28m47.815856237s) [upload=12963]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:08:48.493962093 (12h28m48.08330708s) [upload=12963]: Done uploading segments... Aug 7 22:08:48.493980028 (12h28m48.083325015s) [upload=12963]: Upload finished. err: <nil> Aug 7 22:09:16.735671384 (12h29m16.325016371s) [upload=12973]: Starting upload Aug 7 22:09:16.735728593 (12h29m16.32507358s) [upload=12973]: Done starting upload Aug 7 22:09:16.735770444 (12h29m16.325115431s) [upload=12973]: Uploading segments... Aug 7 22:09:16.735799409 (12h29m16.325144396s) [upload=12973]: Got next segment. Inline: true Aug 7 22:09:16.735802535 (12h29m16.325147522s) [upload=12973]: Waiting for error group managing segments... Aug 7 22:09:16.735831881 (12h29m16.325176868s) [upload=12973]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:09:16.735835217 (12h29m16.325180204s) [upload=12973]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:09:16.735837421 (12h29m16.325182408s) [upload=12973]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:09:17.021230773 (12h29m16.61057576s) [upload=12973]: Done uploading segments... Aug 7 22:09:17.021247365 (12h29m16.610592352s) [upload=12973]: Upload finished. err: <nil> Aug 7 22:10:03.229004755 (12h30m2.818349752s) [upload=12987]: Starting upload Aug 7 22:10:03.229086592 (12h30m2.818431589s) [upload=12987]: Done starting upload Aug 7 22:10:03.229165603 (12h30m2.8185106s) [upload=12987]: Uploading segments... Aug 7 22:10:03.229268600 (12h30m2.818613618s) [upload=12987]: Got next segment. Inline: true Aug 7 22:10:03.229296584 (12h30m2.818641581s) [upload=12987]: Waiting for error group managing segments... Aug 7 22:10:03.229348544 (12h30m2.818693531s) [upload=12987]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:10:03.229354835 (12h30m2.818699822s) [upload=12987]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:10:03.229357520 (12h30m2.818702517s) [upload=12987]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:10:03.732013236 (12h30m3.321358233s) [upload=12987]: Done uploading segments... Aug 7 22:10:03.732033424 (12h30m3.321378411s) [upload=12987]: Upload finished. err: <nil> Aug 7 22:11:11.152680180 (12h31m10.742025167s) [upload=13005]: Starting upload Aug 7 22:11:11.152732820 (12h31m10.742077807s) [upload=13005]: Done starting upload Aug 7 22:11:11.152799828 (12h31m10.742144815s) [upload=13005]: Uploading segments... Aug 7 22:11:11.152844534 (12h31m10.742189531s) [upload=13005]: Got next segment. Inline: true Aug 7 22:11:11.152848501 (12h31m10.742193489s) [upload=13005]: Waiting for error group managing segments... Aug 7 22:11:11.152885622 (12h31m10.742230609s) [upload=13005]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:11:11.152889760 (12h31m10.742234747s) [upload=13005]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:11:11.152893338 (12h31m10.742238325s) [upload=13005]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:12:10.507162881 (12h32m10.096507878s) [upload=13005]: Done uploading segments... Aug 7 22:12:10.507172128 (12h32m10.096517115s) [upload=13005]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:12:11.394499496 (12h32m10.983844483s) [upload=13024]: Starting upload Aug 7 22:12:11.394565451 (12h32m10.983910429s) [upload=13024]: Done starting upload Aug 7 22:12:11.394614746 (12h32m10.983959733s) [upload=13024]: Uploading segments... Aug 7 22:12:11.394660623 (12h32m10.98400561s) [upload=13024]: Got next segment. Inline: true Aug 7 22:12:11.394666585 (12h32m10.984011572s) [upload=13024]: Waiting for error group managing segments... Aug 7 22:12:11.394716149 (12h32m10.984061136s) [upload=13024]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:12:11.394722612 (12h32m10.9840676s) [upload=13024]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:12:11.394727622 (12h32m10.984072609s) [upload=13024]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:13:10.756105924 (12h33m10.345450911s) [upload=13024]: Done uploading segments... Aug 7 22:13:10.756126203 (12h33m10.34547118s) [upload=13024]: Upload finished. err: metaclient: context deadline exceeded Duration: 1m0.000939792s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 27 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXX...X..X............................XX] duration: 1m0.000756958s FAIL (2025-08-07 22:13:08 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.000770254s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 3 out of 60 is failed (threshold=3) [....X..........X.................X..........................] duration: 1.918256493s FAIL (2025-08-07 22:13:05 UTC) orbiter warning: operation: ListObjects ======ERROR====== unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/1K ======LOGS====== 8m27.919686751s) [upload=12612]: Done uploading segments... Aug 7 21:48:28.330358527 (12h8m27.919703514s) [upload=12612]: Upload finished. err: <nil> Aug 7 21:48:54.943937458 (12h8m54.533282445s) [upload=12618]: Starting upload Aug 7 21:48:54.944002622 (12h8m54.533347609s) [upload=12618]: Done starting upload Aug 7 21:48:54.944106100 (12h8m54.533451087s) [upload=12618]: Uploading segments... Aug 7 21:48:54.944196242 (12h8m54.53354132s) [upload=12618]: Got next segment. Inline: true Aug 7 21:48:54.944206543 (12h8m54.53355158s) [upload=12618]: Waiting for error group managing segments... Aug 7 21:48:54.944254484 (12h8m54.533599461s) [upload=12618]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:48:54.944259223 (12h8m54.5336042s) [upload=12618]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:48:54.944262139 (12h8m54.533607126s) [upload=12618]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:48:55.330753187 (12h8m54.920098174s) [upload=12618]: Done uploading segments... Aug 7 21:48:55.330774959 (12h8m54.920119936s) [upload=12618]: Upload finished. err: <nil> Aug 7 21:49:36.650951942 (12h9m36.240296929s) [upload=12631]: Starting upload Aug 7 21:49:36.650993252 (12h9m36.240338239s) [upload=12631]: Done starting upload Aug 7 21:49:36.651049029 (12h9m36.240394016s) [upload=12631]: Uploading segments... Aug 7 21:49:36.651075309 (12h9m36.240420286s) [upload=12631]: Got next segment. Inline: true Aug 7 21:49:36.651078775 (12h9m36.240423763s) [upload=12631]: Waiting for error group managing segments... Aug 7 21:49:36.651115776 (12h9m36.240460773s) [upload=12631]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:49:36.651120254 (12h9m36.240465251s) [upload=12631]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:49:36.651124473 (12h9m36.24046947s) [upload=12631]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:49:36.920683881 (12h9m36.510028868s) [upload=12631]: Done uploading segments... Aug 7 21:49:36.920707246 (12h9m36.510052233s) [upload=12631]: Upload finished. err: <nil> Aug 7 21:50:03.363429937 (12h10m2.952774924s) [upload=12636]: Starting upload Aug 7 21:50:03.363493408 (12h10m2.952838405s) [upload=12636]: Done starting upload Aug 7 21:50:03.363582518 (12h10m2.952927515s) [upload=12636]: Uploading segments... Aug 7 21:50:03.363692078 (12h10m2.953037085s) [upload=12636]: Got next segment. Inline: true Aug 7 21:50:03.363699752 (12h10m2.953044739s) [upload=12636]: Waiting for error group managing segments... Aug 7 21:50:03.363764116 (12h10m2.953109103s) [upload=12636]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:50:03.363773323 (12h10m2.95311831s) [upload=12636]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:50:03.363777271 (12h10m2.953122258s) [upload=12636]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:50:03.636022787 (12h10m3.225367774s) [upload=12636]: Done uploading segments... Aug 7 21:50:03.636042204 (12h10m3.225387202s) [upload=12636]: Upload finished. err: <nil> Aug 7 21:50:20.944550175 (12h10m20.533895152s) [upload=12642]: Starting upload Aug 7 21:50:20.944594911 (12h10m20.533939898s) [upload=12642]: Done starting upload Aug 7 21:50:20.944644405 (12h10m20.533989392s) [upload=12642]: Uploading segments... Aug 7 21:50:20.944674473 (12h10m20.534019461s) [upload=12642]: Got next segment. Inline: true Aug 7 21:50:20.944677820 (12h10m20.534022807s) [upload=12642]: Waiting for error group managing segments... Aug 7 21:50:20.944709450 (12h10m20.534054437s) [upload=12642]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:50:20.944715131 (12h10m20.534060118s) [upload=12642]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:50:20.944717505 (12h10m20.534062493s) [upload=12642]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:50:21.212077476 (12h10m20.801422473s) [upload=12642]: Done uploading segments... Aug 7 21:50:21.212101602 (12h10m20.801446599s) [upload=12642]: Upload finished. err: <nil> Aug 7 21:51:22.649621444 (12h11m22.238966431s) [upload=12660]: Starting upload Aug 7 21:51:22.649696837 (12h11m22.239041824s) [upload=12660]: Done starting upload Aug 7 21:51:22.649777742 (12h11m22.239122729s) [upload=12660]: Uploading segments... Aug 7 21:51:22.649867044 (12h11m22.239212041s) [upload=12660]: Got next segment. Inline: true Aug 7 21:51:22.649872053 (12h11m22.23921704s) [upload=12660]: Waiting for error group managing segments... Aug 7 21:51:22.649940053 (12h11m22.23928504s) [upload=12660]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:51:22.649946785 (12h11m22.239291772s) [upload=12660]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:51:22.649949932 (12h11m22.239294919s) [upload=12660]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:51:22.914527578 (12h11m22.503872565s) [upload=12660]: Done uploading segments... Aug 7 21:51:22.914549130 (12h11m22.503894117s) [upload=12660]: Upload finished. err: <nil> Aug 7 21:51:38.628060070 (12h11m38.217405057s) [upload=12665]: Starting upload Aug 7 21:51:38.628100548 (12h11m38.217445535s) [upload=12665]: Done starting upload Aug 7 21:51:38.628155373 (12h11m38.21750036s) [upload=12665]: Uploading segments... Aug 7 21:51:38.628187814 (12h11m38.217532802s) [upload=12665]: Got next segment. Inline: true Aug 7 21:51:38.628191381 (12h11m38.217536368s) [upload=12665]: Waiting for error group managing segments... Aug 7 21:51:38.628222691 (12h11m38.217567678s) [upload=12665]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:51:38.628226268 (12h11m38.217571255s) [upload=12665]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:51:38.628228662 (12h11m38.217573649s) [upload=12665]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:51:38.940125222 (12h11m38.529470269s) [upload=12665]: Done uploading segments... Aug 7 21:51:38.940147124 (12h11m38.529492191s) [upload=12665]: Upload finished. err: <nil> Aug 7 21:52:16.262879794 (12h12m15.852224781s) [upload=12676]: Starting upload Aug 7 21:52:16.262939168 (12h12m15.852284155s) [upload=12676]: Done starting upload Aug 7 21:52:16.263047876 (12h12m15.852392873s) [upload=12676]: Uploading segments... Aug 7 21:52:16.263139601 (12h12m15.852484598s) [upload=12676]: Got next segment. Inline: true Aug 7 21:52:16.263144440 (12h12m15.852489417s) [upload=12676]: Waiting for error group managing segments... Aug 7 21:52:16.263191931 (12h12m15.852536918s) [upload=12676]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:52:16.263197472 (12h12m15.852542449s) [upload=12676]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:52:16.263200056 (12h12m15.852545043s) [upload=12676]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:52:16.570506954 (12h12m16.159851931s) [upload=12676]: Done uploading segments... Aug 7 21:52:16.570524978 (12h12m16.159869965s) [upload=12676]: Upload finished. err: <nil> Aug 7 21:53:00.574165558 (12h13m0.163510545s) [upload=12691]: Starting upload Aug 7 21:53:00.574212286 (12h13m0.163557273s) [upload=12691]: Done starting upload Aug 7 21:53:00.574248476 (12h13m0.163593463s) [upload=12691]: Uploading segments... Aug 7 21:53:00.574327307 (12h13m0.163672294s) [upload=12691]: Got next segment. Inline: true Aug 7 21:53:00.574332868 (12h13m0.163677855s) [upload=12691]: Waiting for error group managing segments... Aug 7 21:53:00.574373505 (12h13m0.163718492s) [upload=12691]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:53:00.574390598 (12h13m0.163735585s) [upload=12691]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:53:00.574393373 (12h13m0.16373835s) [upload=12691]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:53:00.840335180 (12h13m0.429680167s) [upload=12691]: Done uploading segments... Aug 7 21:53:00.840351500 (12h13m0.429696487s) [upload=12691]: Upload finished. err: <nil> Aug 7 21:53:46.136340579 (12h13m45.725685566s) [upload=12705]: Starting upload Aug 7 21:53:46.136382029 (12h13m45.725727016s) [upload=12705]: Done starting upload Aug 7 21:53:46.136495205 (12h13m45.725840192s) [upload=12705]: Uploading segments... Aug 7 21:53:46.136529601 (12h13m45.725874588s) [upload=12705]: Got next segment. Inline: true Aug 7 21:53:46.136533228 (12h13m45.725878215s) [upload=12705]: Waiting for error group managing segments... Aug 7 21:53:46.136581701 (12h13m45.725926688s) [upload=12705]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:53:46.136585949 (12h13m45.725930936s) [upload=12705]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:53:46.136588393 (12h13m45.72593338s) [upload=12705]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:53:46.427021148 (12h13m46.016366155s) [upload=12705]: Done uploading segments... Aug 7 21:53:46.427045194 (12h13m46.016390181s) [upload=12705]: Upload finished. err: <nil> Aug 7 21:54:41.670591536 (12h14m41.259936523s) [upload=12721]: Starting upload Aug 7 21:54:41.670632916 (12h14m41.259977903s) [upload=12721]: Done starting upload Aug 7 21:54:41.670677612 (12h14m41.260022599s) [upload=12721]: Uploading segments... Aug 7 21:54:41.670708571 (12h14m41.260053558s) [upload=12721]: Got next segment. Inline: true Aug 7 21:54:41.670713510 (12h14m41.260058497s) [upload=12721]: Waiting for error group managing segments... Aug 7 21:54:41.670746032 (12h14m41.260091019s) [upload=12721]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:54:41.670749979 (12h14m41.260094966s) [upload=12721]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:54:41.670752424 (12h14m41.260097411s) [upload=12721]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:54:41.940759391 (12h14m41.530104368s) [upload=12721]: Done uploading segments... Aug 7 21:54:41.940776734 (12h14m41.530121721s) [upload=12721]: Upload finished. err: <nil> Duration: 1.918287273s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 23 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXX.....X................................X] duration: 1m0.000967941s FAIL (2025-08-07 22:13:02 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== s) [noise=true , address=104.36.82.59:4325 , piece_id=FO3UZRA3MKCQ3DDRVDVXFR32LEGJNVVW4HKDIEGKRTBSV53R7E4A, seg_pos={0 0}, upload=13001]: Uploading piece... Aug 7 22:10:54.751934857 (12h30m54.341279854s) [noise=true , address=104.36.82.45:4911 , piece_id=SMAUQHJZL6NGGFEECWS7U2MK53ZHBTFTVZQBNZEUI3GJ7S4DDJ5Q, seg_pos={0 0}, upload=13001]: Uploading piece... Aug 7 22:10:54.752288584 (12h30m54.341633571s) [noise=true , address=104.36.82.44:4887 , piece_id=L75OUMWJPYPFMXTL6GUUZYNE6WRUT7FEMCP3NJ53WBUMZ4LV7SZA, seg_pos={0 0}, upload=13001]: Uploading piece... Aug 7 22:10:54.752508705 (12h30m54.341853693s) [noise=true , address=104.36.82.36:4635 , piece_id=IEDOBRNLEVVODT6MBHMTAP4XXZA63XXH4TPWYHHBLDSAD5PXRYBA, seg_pos={0 0}, upload=13001]: Uploading piece... Aug 7 22:10:54.752670415 (12h30m54.342015402s) [noise=true , address=64.49.13.170:4724 , piece_id=FKWKVJ3OHOPVGP24JDDOATON5JCMJ6NDYHHZMSYKD4QSPGLC3IQQ, seg_pos={0 0}, upload=13001]: Uploading piece... Aug 7 22:10:54.752978735 (12h30m54.342323732s) [noise=true , address=193.149.164.143:4702 , piece_id=JUWPBAFHOXTG3ROQ7FM56PXBHDU3IB54I7RRDYN533NKNSNKQMRA, seg_pos={0 0}, upload=13001]: Uploading piece... Aug 7 22:10:54.753049730 (12h30m54.342394717s) [noise=true , address=104.36.82.62:4442 , piece_id=I2R3NEH23OBIAME6WTWRVGWOHDQ3772NE432ZENZUDTHHZKZVJ3A, seg_pos={0 0}, upload=13001]: Uploading piece... Aug 7 22:10:54.753344033 (12h30m54.34268903s) [noise=true , address=64.49.13.27:4968 , piece_id=75OO5N6T4S76ADG3JGE6VUEELEBQ3SJI53D2KZS5UQTYBN5JUP5Q, seg_pos={0 0}, upload=13001]: Uploading piece... Aug 7 22:10:54.753427573 (12h30m54.34277257s) [noise=true , address=104.36.82.60:4340 , piece_id=7WEKMGBEPFZPAAE3J2BHWZYTW6LPSNQQHK7SN4243QXNJGPYAS7A, seg_pos={0 0}, upload=13001]: Uploading piece... Aug 7 22:10:54.753681599 (12h30m54.343026586s) [noise=true , address=64.49.13.140:4511 , piece_id=N3RDCDJCVYSZEFI5WGBZGCFBDYYCM3PK4UFM3GACPP7A3ZTWFYBA, seg_pos={0 0}, upload=13001]: Uploading piece... Aug 7 22:10:54.754003535 (12h30m54.343348522s) [noise=true , address=104.36.82.54:4187 , piece_id=5TXRHEQUUJI5OOXJQ4MLO44LQC565ZYA4UJWZ3IQQJAWU7ONTCWQ, seg_pos={0 0}, upload=13001]: Uploading piece... Aug 7 22:10:54.754495266 (12h30m54.343840253s) [noise=true , address=104.36.82.46:4487 , piece_id=567IYX3KLIBE2LXMAGYO52V6MPQ2STZD6NAP5WKKCFI77JC32PXA, seg_pos={0 0}, upload=13001]: Uploading piece... Aug 7 22:10:54.754945627 (12h30m54.344290614s) [noise=true , address=104.36.82.47:4524 , piece_id=NWPFOUA5GSBJ4RG3IXFTTLAECGV6FNKOCGYXVF7ZASXFRHKTWD4A, seg_pos={0 0}, upload=13001]: Uploading piece... Aug 7 22:10:54.755644855 (12h30m54.344989842s) [noise=true , address=77.90.31.15:4704 , piece_id=LSQD2P74HKF4X76B2M257RQNRPU3AVMGC635CWMGKSJMEQ62LAWA, seg_pos={0 0}, upload=13001]: Uploading piece... Aug 7 22:10:54.756057805 (12h30m54.345402792s) [noise=true , address=104.36.82.55:4205 , piece_id=DLG5GJWV7OJ2AB6UMU7Y5ZH5W6BNZGUPT3D375YA4QEHKYLPJXHQ, seg_pos={0 0}, upload=13001]: Uploading piece... Aug 7 22:10:55.874691120 (12h30m55.464036107s) [upload=13001]: Next returned nil segment Aug 7 22:10:55.874712882 (12h30m55.464057869s) [upload=13001]: Waiting for error group managing segments... Aug 7 22:10:56.161312978 (12h30m55.750657966s) [noise=true , address=64.49.13.140:4511 , piece_id=N3RDCDJCVYSZEFI5WGBZGCFBDYYCM3PK4UFM3GACPP7A3ZTWFYBA, seg_pos={0 0}, upload=13001]: Done uploading piece. err: <nil> Aug 7 22:10:56.162308593 (12h30m55.751653581s) [noise=true , address=64.49.13.170:4724 , piece_id=FKWKVJ3OHOPVGP24JDDOATON5JCMJ6NDYHHZMSYKD4QSPGLC3IQQ, seg_pos={0 0}, upload=13001]: Done uploading piece. err: <nil> Aug 7 22:10:56.208707537 (12h30m55.798052524s) [noise=true , address=64.49.13.27:4968 , piece_id=75OO5N6T4S76ADG3JGE6VUEELEBQ3SJI53D2KZS5UQTYBN5JUP5Q, seg_pos={0 0}, upload=13001]: Done uploading piece. err: <nil> Aug 7 22:10:56.289643397 (12h30m55.878988374s) [noise=true , address=193.149.164.143:4702 , piece_id=JUWPBAFHOXTG3ROQ7FM56PXBHDU3IB54I7RRDYN533NKNSNKQMRA, seg_pos={0 0}, upload=13001]: Done uploading piece. err: <nil> Aug 7 22:10:56.717621837 (12h30m56.306966824s) [noise=true , address=104.36.82.55:4205 , piece_id=DLG5GJWV7OJ2AB6UMU7Y5ZH5W6BNZGUPT3D375YA4QEHKYLPJXHQ, seg_pos={0 0}, upload=13001]: Done uploading piece. err: <nil> Aug 7 22:10:56.762634227 (12h30m56.351979214s) [noise=true , address=104.36.82.45:4911 , piece_id=SMAUQHJZL6NGGFEECWS7U2MK53ZHBTFTVZQBNZEUI3GJ7S4DDJ5Q, seg_pos={0 0}, upload=13001]: Done uploading piece. err: <nil> Aug 7 22:10:56.787442963 (12h30m56.37678795s) [noise=true , address=104.36.82.46:4487 , piece_id=567IYX3KLIBE2LXMAGYO52V6MPQ2STZD6NAP5WKKCFI77JC32PXA, seg_pos={0 0}, upload=13001]: Done uploading piece. err: <nil> Aug 7 22:10:56.789302119 (12h30m56.378647106s) [noise=true , address=104.36.82.15:4262 , piece_id=C67QLLAIHDKJQGIWHTB7OBB3XANVOHBTKKKX72VS24VWJPOQ4QHA, seg_pos={0 0}, upload=13001]: Done uploading piece. err: <nil> Aug 7 22:10:56.798785704 (12h30m56.388130691s) [noise=true , address=104.36.82.62:4442 , piece_id=I2R3NEH23OBIAME6WTWRVGWOHDQ3772NE432ZENZUDTHHZKZVJ3A, seg_pos={0 0}, upload=13001]: Done uploading piece. err: <nil> Aug 7 22:10:56.808594450 (12h30m56.397939437s) [noise=true , address=77.90.31.15:4704 , piece_id=LSQD2P74HKF4X76B2M257RQNRPU3AVMGC635CWMGKSJMEQ62LAWA, seg_pos={0 0}, upload=13001]: Done uploading piece. err: <nil> Aug 7 22:10:56.836765378 (12h30m56.426110365s) [noise=true , address=104.36.82.36:4635 , piece_id=IEDOBRNLEVVODT6MBHMTAP4XXZA63XXH4TPWYHHBLDSAD5PXRYBA, seg_pos={0 0}, upload=13001]: Done uploading piece. err: <nil> Aug 7 22:10:56.868605666 (12h30m56.457950653s) [noise=true , address=104.36.82.44:4887 , piece_id=L75OUMWJPYPFMXTL6GUUZYNE6WRUT7FEMCP3NJ53WBUMZ4LV7SZA, seg_pos={0 0}, upload=13001]: Done uploading piece. err: <nil> Aug 7 22:10:56.954035535 (12h30m56.543380522s) [noise=true , address=104.36.82.47:4524 , piece_id=NWPFOUA5GSBJ4RG3IXFTTLAECGV6FNKOCGYXVF7ZASXFRHKTWD4A, seg_pos={0 0}, upload=13001]: Done uploading piece. err: <nil> Aug 7 22:10:57.022668687 (12h30m56.612013664s) [noise=true , address=45.91.116.11:4132 , piece_id=KY43L4P6QWVUQSWCCFYJVXF3Q4TRDBGN6NTJZI6KHROJA673QQTQ, seg_pos={0 0}, upload=13001]: Done uploading piece. err: <nil> Aug 7 22:10:57.022695458 (12h30m56.612040445s) [seg_pos={0 0}, upload=13001]: Segment reached optimal threshold of 14 pieces. Aug 7 22:10:57.022801632 (12h30m56.612146619s) [noise=true , address=104.36.82.60:4340 , piece_id=7WEKMGBEPFZPAAE3J2BHWZYTW6LPSNQQHK7SN4243QXNJGPYAS7A, seg_pos={0 0}, upload=13001]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XeA4A8wbmv7wgPjRQnNbP9xgDQz4MzJH7PasaYsTrsMzTJRhC): protocol: expected piece hash; context canceled; EOF Aug 7 22:10:57.022983821 (12h30m56.612328808s) [noise=true , address=104.36.82.37:4658 , piece_id=UNE7SCJLP6M74UGJAZE6D4QBOEUNO5Q3P3YSS2GOALMHW24BXMTQ, seg_pos={0 0}, upload=13001]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gcSHRechDzTDwivRoAwNFfNj5Zjqvi6oJwVqzbCQinwgWtpPr): protocol: write tcp 188.166.225.170:49650->104.36.82.37:4658: use of closed network connection; write tcp 188.166.225.170:49650->104.36.82.37:4658: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49650->104.36.82.37:4658: use of closed network connection Aug 7 22:10:57.023020220 (12h30m56.612365207s) [noise=true , address=104.36.82.59:4325 , piece_id=FO3UZRA3MKCQ3DDRVDVXFR32LEGJNVVW4HKDIEGKRTBSV53R7E4A, seg_pos={0 0}, upload=13001]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1onAnFbXC1LbUAD31oHGp75v9DpnWmx8LTe6Wazf7XuR3rKnJw): protocol: write tcp 188.166.225.170:53516->104.36.82.59:4325: use of closed network connection; write tcp 188.166.225.170:53516->104.36.82.59:4325: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53516->104.36.82.59:4325: use of closed network connection Aug 7 22:10:57.023052371 (12h30m56.612397358s) [noise=true , address=104.36.82.54:4187 , piece_id=5TXRHEQUUJI5OOXJQ4MLO44LQC565ZYA4UJWZ3IQQJAWU7ONTCWQ, seg_pos={0 0}, upload=13001]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12J7goNH48bnMJyaCAPgUznrX1n6bNFNZXGqpuftNLVLxQxprsF): protocol: read tcp 188.166.225.170:36578->104.36.82.54:4187: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 22:10:57.023094472 (12h30m56.612439459s) [seg_pos={0 0}, upload=13001]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 22:10:57.023157663 (12h30m56.61250265s) [upload=13001]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 22:10:57.023170819 (12h30m56.612515806s) [upload=13001]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:10:57.312858834 (12h30m56.902203821s) [upload=13001]: Done uploading segments... Aug 7 22:10:57.312875305 (12h30m56.902220292s) [upload=13001]: Upload finished. err: <nil> Aug 7 22:12:03.463576538 (12h32m3.052921515s) [upload=13021]: Starting upload Aug 7 22:12:03.463644107 (12h32m3.052989094s) [upload=13021]: Done starting upload Aug 7 22:12:03.463720814 (12h32m3.053065801s) [upload=13021]: Uploading segments... Aug 7 22:12:03.463803302 (12h32m3.053148289s) [upload=13021]: Got next segment. Inline: false Aug 7 22:12:03.463829302 (12h32m3.053174279s) [upload=13021]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:12:03.463833710 (12h32m3.053178697s) [upload=13021]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:13:02.821328005 (12h33m2.410672992s) [upload=13021]: Next segment err: context deadline exceeded Aug 7 22:13:02.821359495 (12h33m2.410704482s) [upload=13021]: Done uploading segments... Aug 7 22:13:02.821394342 (12h33m2.410739329s) [upload=13021]: Upload finished. err: context deadline exceeded Duration: 1m0.000988851s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway) antiblip: 3 out of 60 is failed (threshold=3) [X.X...X.....................................................] duration: 910.594844ms remote: "tcp:45.38.24.2:443" target: https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410 FAIL (2025-08-07 22:12:59 UTC) orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": context deadline exceeded Duration: 910.616716ms edge gateway vhost us-select-1
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway) antiblip: 7 out of 60 is failed (threshold=3) [....XX.XX..X.XX.............................................] duration: 265.730297ms remote: "tcp:207.211.208.129:443" target: https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222 FAIL (2025-08-07 22:12:55 UTC) orbiter warning: Get "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222": context deadline exceeded Duration: 265.755536ms edge gateway vhost bgp us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 26 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXX..XX..X..............................XX] duration: 30.00140392s FAIL (2025-08-07 22:12:50 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 30.001428316s core uplink us1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXX.......X................................X] duration: 30.001290162s FAIL (2025-08-07 22:12:47 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.001308908s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 25 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXX...X................................XXX] duration: 30.00087294s FAIL (2025-08-07 22:12:40 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 30.000887118s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 4 out of 60 is failed (threshold=3) [...........X..X....................X....................X...] duration: 6.14759863s FAIL (2025-08-07 22:12:39 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/4M):: after 34e70bda2fe1d7206098dbfb59fc4576ab613f2b752f601da4a5b93473be374e != before ef9ba24509b0003982578e9daec1e6051f8d6ac0bef4f9769b005a8c713045cf ======LOGS====== ding segments... Aug 7 22:10:42.792858699 (12h30m42.382203686s) [upload=12997]: Got next segment. Inline: false Aug 7 22:10:42.792883667 (12h30m42.382228644s) [upload=12997]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:10:42.792887915 (12h30m42.382232902s) [upload=12997]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:10:43.239239407 (12h30m42.828584384s) [seg_pos={0 0}, upload=12997]: Begin upload segment... Aug 7 22:10:43.239300844 (12h30m42.828645831s) [seg_pos={0 0}, upload=12997]: Done begin upload segment. Aug 7 22:10:43.239329008 (12h30m42.828673995s) [noise=true , address=104.36.82.61:4394 , piece_id=BZ4OH34HU6COIEBD542MEZ5YGPXPLFMQKIO2EOV23UBTDTFK5XAA, seg_pos={0 0}, upload=12997]: Uploading piece... Aug 7 22:10:43.239541695 (12h30m42.828886692s) [noise=true , address=104.36.82.37:4687 , piece_id=I7ZESSKAS4ZUYX3CUI4ZRZWY2PLRATRNR6FGVGVAXQD6JT3DRUIA, seg_pos={0 0}, upload=12997]: Uploading piece... Aug 7 22:10:43.239923255 (12h30m42.829268242s) [noise=true , address=104.36.82.10:4087 , piece_id=PDZWS5ZZVKC5BCXJIY43NBQVXM7NCLKJZNSVTD3G2K6F2YEY4RVA, seg_pos={0 0}, upload=12997]: Uploading piece... Aug 7 22:10:43.240056509 (12h30m42.829401506s) [noise=true , address=45.91.116.6:4995 , piece_id=GYQIOSTEVRSY6VKX3GQCUC5YZY5S3ZV7R6X2LBG3QRTOOQD53NOA, seg_pos={0 0}, upload=12997]: Uploading piece... Aug 7 22:10:43.240324512 (12h30m42.829669499s) [noise=true , address=104.36.82.38:4705 , piece_id=M557M36RJRMMND4F33G7BOBYVMESQZZN7ZFESJ5CLHEKXWBCL5SA, seg_pos={0 0}, upload=12997]: Uploading piece... Aug 7 22:10:43.240397562 (12h30m42.829742549s) [noise=true , address=79.127.201.215:4351 , piece_id=367MVHSAIIGC6UZQY4EMH25DJCYYTAIUMPFF3PY426RZJHIO44DA, seg_pos={0 0}, upload=12997]: Uploading piece... Aug 7 22:10:43.240579942 (12h30m42.829924929s) [noise=true , address=104.36.82.47:4524 , piece_id=JDRSOKIHNV2RD3COKNJQHRYG2OXOPUBQOG3AHCTYOKKYC3LZISNQ, seg_pos={0 0}, upload=12997]: Uploading piece... Aug 7 22:10:43.240814029 (12h30m42.830159016s) [noise=true , address=64.49.13.169:4710 , piece_id=KOUJTCIUNCXFZSJWG6ZKR4KV42WV3ABECSARKOQ5YB5W4LAJTGAA, seg_pos={0 0}, upload=12997]: Uploading piece... Aug 7 22:10:43.240831132 (12h30m42.830176119s) [noise=true , address=104.36.82.59:4331 , piece_id=54OHTXRTR2J4S4HECSB7DFPHSC7T4UGBBC4X7WI72IVCMESI5BQQ, seg_pos={0 0}, upload=12997]: Uploading piece... Aug 7 22:10:43.241273939 (12h30m42.830618926s) [noise=true , address=104.36.82.44:4881 , piece_id=YWXSTJSJ46ERQLZ6A2E36CX3R2FI4XVVEZGUMDYJX4ZG3M3R5ACQ, seg_pos={0 0}, upload=12997]: Uploading piece... Aug 7 22:10:43.241589863 (12h30m42.83093485s) [noise=true , address=104.36.82.2:4750 , piece_id=24FHN36X2VIWLTYUDHSJNLSOHHEZ56K6DBIGSMI2ZCA26CPT7L7Q, seg_pos={0 0}, upload=12997]: Uploading piece... Aug 7 22:10:43.241938992 (12h30m42.831283979s) [noise=true , address=104.36.82.3:4791 , piece_id=DJ46BVH37DMTY3YD6VG2AEPNNTPYF7EH4IA3Y633UH3QO6XUTKPQ, seg_pos={0 0}, upload=12997]: Uploading piece... Aug 7 22:10:43.242014867 (12h30m42.831359854s) [noise=true , address=104.36.82.58:4270 , piece_id=HLYO6DZ75CNGFXJ6DA5SKQNVPD5COMDHFAETYUZTBOSKMPT4BUEQ, seg_pos={0 0}, upload=12997]: Uploading piece... Aug 7 22:10:43.242250207 (12h30m42.831595194s) [noise=true , address=104.36.82.44:4885 , piece_id=FJ726HA3PDRWAYDFGUEK2Y7D3B77ERCFPJSGWNKKPLGWXC3WK7GA, seg_pos={0 0}, upload=12997]: Uploading piece... Aug 7 22:10:43.242484636 (12h30m42.831829623s) [noise=true , address=79.127.205.246:4254 , piece_id=V3AT6KY72VDSIMMOZUC4JECVAQSPAHMAPC5CXDVCO46Z6RTPRUVA, seg_pos={0 0}, upload=12997]: Uploading piece... Aug 7 22:10:43.242691451 (12h30m42.832036438s) [noise=true , address=104.36.82.8:4995 , piece_id=BI77BF2ZGP3ZMB2ABMBUCKQCEF3Z6QHRC7IT6Q2ATIF6VON32WWQ, seg_pos={0 0}, upload=12997]: Uploading piece... Aug 7 22:10:43.242894410 (12h30m42.832239397s) [noise=true , address=104.36.82.43:4859 , piece_id=P2S2T5IIUTG4D6EQILFQ7OR5E5JOZHAYO563X7ED5UBN7THKW4QA, seg_pos={0 0}, upload=12997]: Uploading piece... Aug 7 22:10:43.243127415 (12h30m42.832472402s) [noise=true , address=104.36.82.46:4509 , piece_id=WQ3UWQPCMAZ7EDHTSURUQAKL6KWHUA3GSJI6U2G5ZUABRGCTFEXQ, seg_pos={0 0}, upload=12997]: Uploading piece... Aug 7 22:10:44.351806366 (12h30m43.941151353s) [upload=12997]: Next returned nil segment Aug 7 22:10:44.351819280 (12h30m43.941164257s) [upload=12997]: Waiting for error group managing segments... Aug 7 22:10:44.651888156 (12h30m44.241233144s) [noise=true , address=79.127.205.246:4254 , piece_id=V3AT6KY72VDSIMMOZUC4JECVAQSPAHMAPC5CXDVCO46Z6RTPRUVA, seg_pos={0 0}, upload=12997]: Done uploading piece. err: <nil> Aug 7 22:10:44.674573650 (12h30m44.263918637s) [noise=true , address=64.49.13.169:4710 , piece_id=KOUJTCIUNCXFZSJWG6ZKR4KV42WV3ABECSARKOQ5YB5W4LAJTGAA, seg_pos={0 0}, upload=12997]: Done uploading piece. err: <nil> Aug 7 22:10:45.186610734 (12h30m44.775955721s) [noise=true , address=104.36.82.38:4705 , piece_id=M557M36RJRMMND4F33G7BOBYVMESQZZN7ZFESJ5CLHEKXWBCL5SA, seg_pos={0 0}, upload=12997]: Done uploading piece. err: <nil> Aug 7 22:10:45.204653265 (12h30m44.793998252s) [noise=true , address=104.36.82.58:4270 , piece_id=HLYO6DZ75CNGFXJ6DA5SKQNVPD5COMDHFAETYUZTBOSKMPT4BUEQ, seg_pos={0 0}, upload=12997]: Done uploading piece. err: <nil> Aug 7 22:10:45.218003397 (12h30m44.807348374s) [noise=true , address=79.127.201.215:4351 , piece_id=367MVHSAIIGC6UZQY4EMH25DJCYYTAIUMPFF3PY426RZJHIO44DA, seg_pos={0 0}, upload=12997]: Done uploading piece. err: <nil> Aug 7 22:10:45.276599656 (12h30m44.865944643s) [noise=true , address=104.36.82.46:4509 , piece_id=WQ3UWQPCMAZ7EDHTSURUQAKL6KWHUA3GSJI6U2G5ZUABRGCTFEXQ, seg_pos={0 0}, upload=12997]: Done uploading piece. err: <nil> Aug 7 22:10:45.316254365 (12h30m44.905599352s) [noise=true , address=104.36.82.2:4750 , piece_id=24FHN36X2VIWLTYUDHSJNLSOHHEZ56K6DBIGSMI2ZCA26CPT7L7Q, seg_pos={0 0}, upload=12997]: Done uploading piece. err: <nil> Aug 7 22:10:45.349441480 (12h30m44.938786467s) [noise=true , address=104.36.82.3:4791 , piece_id=DJ46BVH37DMTY3YD6VG2AEPNNTPYF7EH4IA3Y633UH3QO6XUTKPQ, seg_pos={0 0}, upload=12997]: Done uploading piece. err: <nil> Aug 7 22:10:45.352466728 (12h30m44.941811715s) [noise=true , address=104.36.82.44:4881 , piece_id=YWXSTJSJ46ERQLZ6A2E36CX3R2FI4XVVEZGUMDYJX4ZG3M3R5ACQ, seg_pos={0 0}, upload=12997]: Done uploading piece. err: <nil> Aug 7 22:10:45.418760245 (12h30m45.008105232s) [noise=true , address=104.36.82.37:4687 , piece_id=I7ZESSKAS4ZUYX3CUI4ZRZWY2PLRATRNR6FGVGVAXQD6JT3DRUIA, seg_pos={0 0}, upload=12997]: Done uploading piece. err: <nil> Aug 7 22:10:45.540632266 (12h30m45.129977253s) [noise=true , address=104.36.82.10:4087 , piece_id=PDZWS5ZZVKC5BCXJIY43NBQVXM7NCLKJZNSVTD3G2K6F2YEY4RVA, seg_pos={0 0}, upload=12997]: Done uploading piece. err: <nil> Aug 7 22:10:45.567016576 (12h30m45.156361563s) [noise=true , address=104.36.82.59:4331 , piece_id=54OHTXRTR2J4S4HECSB7DFPHSC7T4UGBBC4X7WI72IVCMESI5BQQ, seg_pos={0 0}, upload=12997]: Done uploading piece. err: <nil> Aug 7 22:10:45.597615539 (12h30m45.186960526s) [noise=true , address=104.36.82.44:4885 , piece_id=FJ726HA3PDRWAYDFGUEK2Y7D3B77ERCFPJSGWNKKPLGWXC3WK7GA, seg_pos={0 0}, upload=12997]: Done uploading piece. err: <nil> Aug 7 22:10:45.676238976 (12h30m45.265584013s) [noise=true , address=45.91.116.6:4995 , piece_id=GYQIOSTEVRSY6VKX3GQCUC5YZY5S3ZV7R6X2LBG3QRTOOQD53NOA, seg_pos={0 0}, upload=12997]: Done uploading piece. err: <nil> Aug 7 22:10:45.676277709 (12h30m45.265622696s) [seg_pos={0 0}, upload=12997]: Segment reached optimal threshold of 14 pieces. Aug 7 22:10:45.676633781 (12h30m45.265978758s) [noise=true , address=104.36.82.8:4995 , piece_id=BI77BF2ZGP3ZMB2ABMBUCKQCEF3Z6QHRC7IT6Q2ATIF6VON32WWQ, seg_pos={0 0}, upload=12997]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tvoTD1LoSj3q2rLiDckfT56dXNKKwdVu1ZAsBZExNFVTom9tH): protocol: expected piece hash; context canceled; EOF Aug 7 22:10:45.676841539 (12h30m45.266186526s) [noise=true , address=104.36.82.43:4859 , piece_id=P2S2T5IIUTG4D6EQILFQ7OR5E5JOZHAYO563X7ED5UBN7THKW4QA, seg_pos={0 0}, upload=12997]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:126SqCtRGPziEeopRjA4nNkjkCZWvUwkt5XaMgcqZKJwoxuYWY7): protocol: rpc: dial tcp 104.36.82.43:4859: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 22:10:45.676869993 (12h30m45.26621498s) [noise=true , address=104.36.82.47:4524 , piece_id=JDRSOKIHNV2RD3COKNJQHRYG2OXOPUBQOG3AHCTYOKKYC3LZISNQ, seg_pos={0 0}, upload=12997]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xfbCzjSwFDRrR1gPggeN8tNCLbRRWDVzCJ6mqhZsbVSe8d4s4): protocol: read tcp 188.166.225.170:58322->104.36.82.47:4524: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 22:10:45.676908757 (12h30m45.266253744s) [noise=true , address=104.36.82.61:4394 , piece_id=BZ4OH34HU6COIEBD542MEZ5YGPXPLFMQKIO2EOV23UBTDTFK5XAA, seg_pos={0 0}, upload=12997]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1dSpTqCxeAcL9PVYbP7BhmuKE7papFPc9swB1Fy83WcFDgQbcF): protocol: rpc: dial tcp 104.36.82.61:4394: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 22:10:45.676927984 (12h30m45.266272961s) [seg_pos={0 0}, upload=12997]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 22:10:45.677012455 (12h30m45.266357443s) [upload=12997]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 22:10:45.677016543 (12h30m45.26636152s) [upload=12997]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:10:45.998268658 (12h30m45.587613635s) [upload=12997]: Done uploading segments... Aug 7 22:10:45.998285902 (12h30m45.587630889s) [upload=12997]: Upload finished. err: <nil> Duration: 6.147625631s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway) antiblip: 7 out of 60 is failed (threshold=3) [.....XX.XX..X.XX............................................] duration: 231.023445ms remote: "tcp:207.211.208.129:443" target: https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222 FAIL (2025-08-07 22:12:37 UTC) orbiter warning: Get "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222": context deadline exceeded Duration: 231.051569ms edge gateway vhost bgp us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 3 out of 60 is failed (threshold=3) [.....X..........X.................X.........................] duration: 1.842652344s FAIL (2025-08-07 22:12:27 UTC) orbiter warning: operation: ListObjects ======ERROR====== unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/1K ======LOGS====== 8m27.919686751s) [upload=12612]: Done uploading segments... Aug 7 21:48:28.330358527 (12h8m27.919703514s) [upload=12612]: Upload finished. err: <nil> Aug 7 21:48:54.943937458 (12h8m54.533282445s) [upload=12618]: Starting upload Aug 7 21:48:54.944002622 (12h8m54.533347609s) [upload=12618]: Done starting upload Aug 7 21:48:54.944106100 (12h8m54.533451087s) [upload=12618]: Uploading segments... Aug 7 21:48:54.944196242 (12h8m54.53354132s) [upload=12618]: Got next segment. Inline: true Aug 7 21:48:54.944206543 (12h8m54.53355158s) [upload=12618]: Waiting for error group managing segments... Aug 7 21:48:54.944254484 (12h8m54.533599461s) [upload=12618]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:48:54.944259223 (12h8m54.5336042s) [upload=12618]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:48:54.944262139 (12h8m54.533607126s) [upload=12618]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:48:55.330753187 (12h8m54.920098174s) [upload=12618]: Done uploading segments... Aug 7 21:48:55.330774959 (12h8m54.920119936s) [upload=12618]: Upload finished. err: <nil> Aug 7 21:49:36.650951942 (12h9m36.240296929s) [upload=12631]: Starting upload Aug 7 21:49:36.650993252 (12h9m36.240338239s) [upload=12631]: Done starting upload Aug 7 21:49:36.651049029 (12h9m36.240394016s) [upload=12631]: Uploading segments... Aug 7 21:49:36.651075309 (12h9m36.240420286s) [upload=12631]: Got next segment. Inline: true Aug 7 21:49:36.651078775 (12h9m36.240423763s) [upload=12631]: Waiting for error group managing segments... Aug 7 21:49:36.651115776 (12h9m36.240460773s) [upload=12631]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:49:36.651120254 (12h9m36.240465251s) [upload=12631]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:49:36.651124473 (12h9m36.24046947s) [upload=12631]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:49:36.920683881 (12h9m36.510028868s) [upload=12631]: Done uploading segments... Aug 7 21:49:36.920707246 (12h9m36.510052233s) [upload=12631]: Upload finished. err: <nil> Aug 7 21:50:03.363429937 (12h10m2.952774924s) [upload=12636]: Starting upload Aug 7 21:50:03.363493408 (12h10m2.952838405s) [upload=12636]: Done starting upload Aug 7 21:50:03.363582518 (12h10m2.952927515s) [upload=12636]: Uploading segments... Aug 7 21:50:03.363692078 (12h10m2.953037085s) [upload=12636]: Got next segment. Inline: true Aug 7 21:50:03.363699752 (12h10m2.953044739s) [upload=12636]: Waiting for error group managing segments... Aug 7 21:50:03.363764116 (12h10m2.953109103s) [upload=12636]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:50:03.363773323 (12h10m2.95311831s) [upload=12636]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:50:03.363777271 (12h10m2.953122258s) [upload=12636]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:50:03.636022787 (12h10m3.225367774s) [upload=12636]: Done uploading segments... Aug 7 21:50:03.636042204 (12h10m3.225387202s) [upload=12636]: Upload finished. err: <nil> Aug 7 21:50:20.944550175 (12h10m20.533895152s) [upload=12642]: Starting upload Aug 7 21:50:20.944594911 (12h10m20.533939898s) [upload=12642]: Done starting upload Aug 7 21:50:20.944644405 (12h10m20.533989392s) [upload=12642]: Uploading segments... Aug 7 21:50:20.944674473 (12h10m20.534019461s) [upload=12642]: Got next segment. Inline: true Aug 7 21:50:20.944677820 (12h10m20.534022807s) [upload=12642]: Waiting for error group managing segments... Aug 7 21:50:20.944709450 (12h10m20.534054437s) [upload=12642]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:50:20.944715131 (12h10m20.534060118s) [upload=12642]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:50:20.944717505 (12h10m20.534062493s) [upload=12642]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:50:21.212077476 (12h10m20.801422473s) [upload=12642]: Done uploading segments... Aug 7 21:50:21.212101602 (12h10m20.801446599s) [upload=12642]: Upload finished. err: <nil> Aug 7 21:51:22.649621444 (12h11m22.238966431s) [upload=12660]: Starting upload Aug 7 21:51:22.649696837 (12h11m22.239041824s) [upload=12660]: Done starting upload Aug 7 21:51:22.649777742 (12h11m22.239122729s) [upload=12660]: Uploading segments... Aug 7 21:51:22.649867044 (12h11m22.239212041s) [upload=12660]: Got next segment. Inline: true Aug 7 21:51:22.649872053 (12h11m22.23921704s) [upload=12660]: Waiting for error group managing segments... Aug 7 21:51:22.649940053 (12h11m22.23928504s) [upload=12660]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:51:22.649946785 (12h11m22.239291772s) [upload=12660]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:51:22.649949932 (12h11m22.239294919s) [upload=12660]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:51:22.914527578 (12h11m22.503872565s) [upload=12660]: Done uploading segments... Aug 7 21:51:22.914549130 (12h11m22.503894117s) [upload=12660]: Upload finished. err: <nil> Aug 7 21:51:38.628060070 (12h11m38.217405057s) [upload=12665]: Starting upload Aug 7 21:51:38.628100548 (12h11m38.217445535s) [upload=12665]: Done starting upload Aug 7 21:51:38.628155373 (12h11m38.21750036s) [upload=12665]: Uploading segments... Aug 7 21:51:38.628187814 (12h11m38.217532802s) [upload=12665]: Got next segment. Inline: true Aug 7 21:51:38.628191381 (12h11m38.217536368s) [upload=12665]: Waiting for error group managing segments... Aug 7 21:51:38.628222691 (12h11m38.217567678s) [upload=12665]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:51:38.628226268 (12h11m38.217571255s) [upload=12665]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:51:38.628228662 (12h11m38.217573649s) [upload=12665]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:51:38.940125222 (12h11m38.529470269s) [upload=12665]: Done uploading segments... Aug 7 21:51:38.940147124 (12h11m38.529492191s) [upload=12665]: Upload finished. err: <nil> Aug 7 21:52:16.262879794 (12h12m15.852224781s) [upload=12676]: Starting upload Aug 7 21:52:16.262939168 (12h12m15.852284155s) [upload=12676]: Done starting upload Aug 7 21:52:16.263047876 (12h12m15.852392873s) [upload=12676]: Uploading segments... Aug 7 21:52:16.263139601 (12h12m15.852484598s) [upload=12676]: Got next segment. Inline: true Aug 7 21:52:16.263144440 (12h12m15.852489417s) [upload=12676]: Waiting for error group managing segments... Aug 7 21:52:16.263191931 (12h12m15.852536918s) [upload=12676]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:52:16.263197472 (12h12m15.852542449s) [upload=12676]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:52:16.263200056 (12h12m15.852545043s) [upload=12676]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:52:16.570506954 (12h12m16.159851931s) [upload=12676]: Done uploading segments... Aug 7 21:52:16.570524978 (12h12m16.159869965s) [upload=12676]: Upload finished. err: <nil> Aug 7 21:53:00.574165558 (12h13m0.163510545s) [upload=12691]: Starting upload Aug 7 21:53:00.574212286 (12h13m0.163557273s) [upload=12691]: Done starting upload Aug 7 21:53:00.574248476 (12h13m0.163593463s) [upload=12691]: Uploading segments... Aug 7 21:53:00.574327307 (12h13m0.163672294s) [upload=12691]: Got next segment. Inline: true Aug 7 21:53:00.574332868 (12h13m0.163677855s) [upload=12691]: Waiting for error group managing segments... Aug 7 21:53:00.574373505 (12h13m0.163718492s) [upload=12691]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:53:00.574390598 (12h13m0.163735585s) [upload=12691]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:53:00.574393373 (12h13m0.16373835s) [upload=12691]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:53:00.840335180 (12h13m0.429680167s) [upload=12691]: Done uploading segments... Aug 7 21:53:00.840351500 (12h13m0.429696487s) [upload=12691]: Upload finished. err: <nil> Aug 7 21:53:46.136340579 (12h13m45.725685566s) [upload=12705]: Starting upload Aug 7 21:53:46.136382029 (12h13m45.725727016s) [upload=12705]: Done starting upload Aug 7 21:53:46.136495205 (12h13m45.725840192s) [upload=12705]: Uploading segments... Aug 7 21:53:46.136529601 (12h13m45.725874588s) [upload=12705]: Got next segment. Inline: true Aug 7 21:53:46.136533228 (12h13m45.725878215s) [upload=12705]: Waiting for error group managing segments... Aug 7 21:53:46.136581701 (12h13m45.725926688s) [upload=12705]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:53:46.136585949 (12h13m45.725930936s) [upload=12705]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:53:46.136588393 (12h13m45.72593338s) [upload=12705]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:53:46.427021148 (12h13m46.016366155s) [upload=12705]: Done uploading segments... Aug 7 21:53:46.427045194 (12h13m46.016390181s) [upload=12705]: Upload finished. err: <nil> Aug 7 21:54:41.670591536 (12h14m41.259936523s) [upload=12721]: Starting upload Aug 7 21:54:41.670632916 (12h14m41.259977903s) [upload=12721]: Done starting upload Aug 7 21:54:41.670677612 (12h14m41.260022599s) [upload=12721]: Uploading segments... Aug 7 21:54:41.670708571 (12h14m41.260053558s) [upload=12721]: Got next segment. Inline: true Aug 7 21:54:41.670713510 (12h14m41.260058497s) [upload=12721]: Waiting for error group managing segments... Aug 7 21:54:41.670746032 (12h14m41.260091019s) [upload=12721]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:54:41.670749979 (12h14m41.260094966s) [upload=12721]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:54:41.670752424 (12h14m41.260097411s) [upload=12721]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:54:41.940759391 (12h14m41.530104368s) [upload=12721]: Done uploading segments... Aug 7 21:54:41.940776734 (12h14m41.530121721s) [upload=12721]: Upload finished. err: <nil> Duration: 1.84267107s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 19 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXX.X........................................X] duration: 30.001578839s FAIL (2025-08-07 22:12:18 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.001605972s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 4 out of 60 is failed (threshold=3) [............X..X....................X....................X..] duration: 8.020526082s FAIL (2025-08-07 22:12:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/4M):: after 34e70bda2fe1d7206098dbfb59fc4576ab613f2b752f601da4a5b93473be374e != before ef9ba24509b0003982578e9daec1e6051f8d6ac0bef4f9769b005a8c713045cf ======LOGS====== ding segments... Aug 7 22:10:42.792858699 (12h30m42.382203686s) [upload=12997]: Got next segment. Inline: false Aug 7 22:10:42.792883667 (12h30m42.382228644s) [upload=12997]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:10:42.792887915 (12h30m42.382232902s) [upload=12997]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:10:43.239239407 (12h30m42.828584384s) [seg_pos={0 0}, upload=12997]: Begin upload segment... Aug 7 22:10:43.239300844 (12h30m42.828645831s) [seg_pos={0 0}, upload=12997]: Done begin upload segment. Aug 7 22:10:43.239329008 (12h30m42.828673995s) [noise=true , address=104.36.82.61:4394 , piece_id=BZ4OH34HU6COIEBD542MEZ5YGPXPLFMQKIO2EOV23UBTDTFK5XAA, seg_pos={0 0}, upload=12997]: Uploading piece... Aug 7 22:10:43.239541695 (12h30m42.828886692s) [noise=true , address=104.36.82.37:4687 , piece_id=I7ZESSKAS4ZUYX3CUI4ZRZWY2PLRATRNR6FGVGVAXQD6JT3DRUIA, seg_pos={0 0}, upload=12997]: Uploading piece... Aug 7 22:10:43.239923255 (12h30m42.829268242s) [noise=true , address=104.36.82.10:4087 , piece_id=PDZWS5ZZVKC5BCXJIY43NBQVXM7NCLKJZNSVTD3G2K6F2YEY4RVA, seg_pos={0 0}, upload=12997]: Uploading piece... Aug 7 22:10:43.240056509 (12h30m42.829401506s) [noise=true , address=45.91.116.6:4995 , piece_id=GYQIOSTEVRSY6VKX3GQCUC5YZY5S3ZV7R6X2LBG3QRTOOQD53NOA, seg_pos={0 0}, upload=12997]: Uploading piece... Aug 7 22:10:43.240324512 (12h30m42.829669499s) [noise=true , address=104.36.82.38:4705 , piece_id=M557M36RJRMMND4F33G7BOBYVMESQZZN7ZFESJ5CLHEKXWBCL5SA, seg_pos={0 0}, upload=12997]: Uploading piece... Aug 7 22:10:43.240397562 (12h30m42.829742549s) [noise=true , address=79.127.201.215:4351 , piece_id=367MVHSAIIGC6UZQY4EMH25DJCYYTAIUMPFF3PY426RZJHIO44DA, seg_pos={0 0}, upload=12997]: Uploading piece... Aug 7 22:10:43.240579942 (12h30m42.829924929s) [noise=true , address=104.36.82.47:4524 , piece_id=JDRSOKIHNV2RD3COKNJQHRYG2OXOPUBQOG3AHCTYOKKYC3LZISNQ, seg_pos={0 0}, upload=12997]: Uploading piece... Aug 7 22:10:43.240814029 (12h30m42.830159016s) [noise=true , address=64.49.13.169:4710 , piece_id=KOUJTCIUNCXFZSJWG6ZKR4KV42WV3ABECSARKOQ5YB5W4LAJTGAA, seg_pos={0 0}, upload=12997]: Uploading piece... Aug 7 22:10:43.240831132 (12h30m42.830176119s) [noise=true , address=104.36.82.59:4331 , piece_id=54OHTXRTR2J4S4HECSB7DFPHSC7T4UGBBC4X7WI72IVCMESI5BQQ, seg_pos={0 0}, upload=12997]: Uploading piece... Aug 7 22:10:43.241273939 (12h30m42.830618926s) [noise=true , address=104.36.82.44:4881 , piece_id=YWXSTJSJ46ERQLZ6A2E36CX3R2FI4XVVEZGUMDYJX4ZG3M3R5ACQ, seg_pos={0 0}, upload=12997]: Uploading piece... Aug 7 22:10:43.241589863 (12h30m42.83093485s) [noise=true , address=104.36.82.2:4750 , piece_id=24FHN36X2VIWLTYUDHSJNLSOHHEZ56K6DBIGSMI2ZCA26CPT7L7Q, seg_pos={0 0}, upload=12997]: Uploading piece... Aug 7 22:10:43.241938992 (12h30m42.831283979s) [noise=true , address=104.36.82.3:4791 , piece_id=DJ46BVH37DMTY3YD6VG2AEPNNTPYF7EH4IA3Y633UH3QO6XUTKPQ, seg_pos={0 0}, upload=12997]: Uploading piece... Aug 7 22:10:43.242014867 (12h30m42.831359854s) [noise=true , address=104.36.82.58:4270 , piece_id=HLYO6DZ75CNGFXJ6DA5SKQNVPD5COMDHFAETYUZTBOSKMPT4BUEQ, seg_pos={0 0}, upload=12997]: Uploading piece... Aug 7 22:10:43.242250207 (12h30m42.831595194s) [noise=true , address=104.36.82.44:4885 , piece_id=FJ726HA3PDRWAYDFGUEK2Y7D3B77ERCFPJSGWNKKPLGWXC3WK7GA, seg_pos={0 0}, upload=12997]: Uploading piece... Aug 7 22:10:43.242484636 (12h30m42.831829623s) [noise=true , address=79.127.205.246:4254 , piece_id=V3AT6KY72VDSIMMOZUC4JECVAQSPAHMAPC5CXDVCO46Z6RTPRUVA, seg_pos={0 0}, upload=12997]: Uploading piece... Aug 7 22:10:43.242691451 (12h30m42.832036438s) [noise=true , address=104.36.82.8:4995 , piece_id=BI77BF2ZGP3ZMB2ABMBUCKQCEF3Z6QHRC7IT6Q2ATIF6VON32WWQ, seg_pos={0 0}, upload=12997]: Uploading piece... Aug 7 22:10:43.242894410 (12h30m42.832239397s) [noise=true , address=104.36.82.43:4859 , piece_id=P2S2T5IIUTG4D6EQILFQ7OR5E5JOZHAYO563X7ED5UBN7THKW4QA, seg_pos={0 0}, upload=12997]: Uploading piece... Aug 7 22:10:43.243127415 (12h30m42.832472402s) [noise=true , address=104.36.82.46:4509 , piece_id=WQ3UWQPCMAZ7EDHTSURUQAKL6KWHUA3GSJI6U2G5ZUABRGCTFEXQ, seg_pos={0 0}, upload=12997]: Uploading piece... Aug 7 22:10:44.351806366 (12h30m43.941151353s) [upload=12997]: Next returned nil segment Aug 7 22:10:44.351819280 (12h30m43.941164257s) [upload=12997]: Waiting for error group managing segments... Aug 7 22:10:44.651888156 (12h30m44.241233144s) [noise=true , address=79.127.205.246:4254 , piece_id=V3AT6KY72VDSIMMOZUC4JECVAQSPAHMAPC5CXDVCO46Z6RTPRUVA, seg_pos={0 0}, upload=12997]: Done uploading piece. err: <nil> Aug 7 22:10:44.674573650 (12h30m44.263918637s) [noise=true , address=64.49.13.169:4710 , piece_id=KOUJTCIUNCXFZSJWG6ZKR4KV42WV3ABECSARKOQ5YB5W4LAJTGAA, seg_pos={0 0}, upload=12997]: Done uploading piece. err: <nil> Aug 7 22:10:45.186610734 (12h30m44.775955721s) [noise=true , address=104.36.82.38:4705 , piece_id=M557M36RJRMMND4F33G7BOBYVMESQZZN7ZFESJ5CLHEKXWBCL5SA, seg_pos={0 0}, upload=12997]: Done uploading piece. err: <nil> Aug 7 22:10:45.204653265 (12h30m44.793998252s) [noise=true , address=104.36.82.58:4270 , piece_id=HLYO6DZ75CNGFXJ6DA5SKQNVPD5COMDHFAETYUZTBOSKMPT4BUEQ, seg_pos={0 0}, upload=12997]: Done uploading piece. err: <nil> Aug 7 22:10:45.218003397 (12h30m44.807348374s) [noise=true , address=79.127.201.215:4351 , piece_id=367MVHSAIIGC6UZQY4EMH25DJCYYTAIUMPFF3PY426RZJHIO44DA, seg_pos={0 0}, upload=12997]: Done uploading piece. err: <nil> Aug 7 22:10:45.276599656 (12h30m44.865944643s) [noise=true , address=104.36.82.46:4509 , piece_id=WQ3UWQPCMAZ7EDHTSURUQAKL6KWHUA3GSJI6U2G5ZUABRGCTFEXQ, seg_pos={0 0}, upload=12997]: Done uploading piece. err: <nil> Aug 7 22:10:45.316254365 (12h30m44.905599352s) [noise=true , address=104.36.82.2:4750 , piece_id=24FHN36X2VIWLTYUDHSJNLSOHHEZ56K6DBIGSMI2ZCA26CPT7L7Q, seg_pos={0 0}, upload=12997]: Done uploading piece. err: <nil> Aug 7 22:10:45.349441480 (12h30m44.938786467s) [noise=true , address=104.36.82.3:4791 , piece_id=DJ46BVH37DMTY3YD6VG2AEPNNTPYF7EH4IA3Y633UH3QO6XUTKPQ, seg_pos={0 0}, upload=12997]: Done uploading piece. err: <nil> Aug 7 22:10:45.352466728 (12h30m44.941811715s) [noise=true , address=104.36.82.44:4881 , piece_id=YWXSTJSJ46ERQLZ6A2E36CX3R2FI4XVVEZGUMDYJX4ZG3M3R5ACQ, seg_pos={0 0}, upload=12997]: Done uploading piece. err: <nil> Aug 7 22:10:45.418760245 (12h30m45.008105232s) [noise=true , address=104.36.82.37:4687 , piece_id=I7ZESSKAS4ZUYX3CUI4ZRZWY2PLRATRNR6FGVGVAXQD6JT3DRUIA, seg_pos={0 0}, upload=12997]: Done uploading piece. err: <nil> Aug 7 22:10:45.540632266 (12h30m45.129977253s) [noise=true , address=104.36.82.10:4087 , piece_id=PDZWS5ZZVKC5BCXJIY43NBQVXM7NCLKJZNSVTD3G2K6F2YEY4RVA, seg_pos={0 0}, upload=12997]: Done uploading piece. err: <nil> Aug 7 22:10:45.567016576 (12h30m45.156361563s) [noise=true , address=104.36.82.59:4331 , piece_id=54OHTXRTR2J4S4HECSB7DFPHSC7T4UGBBC4X7WI72IVCMESI5BQQ, seg_pos={0 0}, upload=12997]: Done uploading piece. err: <nil> Aug 7 22:10:45.597615539 (12h30m45.186960526s) [noise=true , address=104.36.82.44:4885 , piece_id=FJ726HA3PDRWAYDFGUEK2Y7D3B77ERCFPJSGWNKKPLGWXC3WK7GA, seg_pos={0 0}, upload=12997]: Done uploading piece. err: <nil> Aug 7 22:10:45.676238976 (12h30m45.265584013s) [noise=true , address=45.91.116.6:4995 , piece_id=GYQIOSTEVRSY6VKX3GQCUC5YZY5S3ZV7R6X2LBG3QRTOOQD53NOA, seg_pos={0 0}, upload=12997]: Done uploading piece. err: <nil> Aug 7 22:10:45.676277709 (12h30m45.265622696s) [seg_pos={0 0}, upload=12997]: Segment reached optimal threshold of 14 pieces. Aug 7 22:10:45.676633781 (12h30m45.265978758s) [noise=true , address=104.36.82.8:4995 , piece_id=BI77BF2ZGP3ZMB2ABMBUCKQCEF3Z6QHRC7IT6Q2ATIF6VON32WWQ, seg_pos={0 0}, upload=12997]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tvoTD1LoSj3q2rLiDckfT56dXNKKwdVu1ZAsBZExNFVTom9tH): protocol: expected piece hash; context canceled; EOF Aug 7 22:10:45.676841539 (12h30m45.266186526s) [noise=true , address=104.36.82.43:4859 , piece_id=P2S2T5IIUTG4D6EQILFQ7OR5E5JOZHAYO563X7ED5UBN7THKW4QA, seg_pos={0 0}, upload=12997]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:126SqCtRGPziEeopRjA4nNkjkCZWvUwkt5XaMgcqZKJwoxuYWY7): protocol: rpc: dial tcp 104.36.82.43:4859: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 22:10:45.676869993 (12h30m45.26621498s) [noise=true , address=104.36.82.47:4524 , piece_id=JDRSOKIHNV2RD3COKNJQHRYG2OXOPUBQOG3AHCTYOKKYC3LZISNQ, seg_pos={0 0}, upload=12997]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xfbCzjSwFDRrR1gPggeN8tNCLbRRWDVzCJ6mqhZsbVSe8d4s4): protocol: read tcp 188.166.225.170:58322->104.36.82.47:4524: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 22:10:45.676908757 (12h30m45.266253744s) [noise=true , address=104.36.82.61:4394 , piece_id=BZ4OH34HU6COIEBD542MEZ5YGPXPLFMQKIO2EOV23UBTDTFK5XAA, seg_pos={0 0}, upload=12997]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1dSpTqCxeAcL9PVYbP7BhmuKE7papFPc9swB1Fy83WcFDgQbcF): protocol: rpc: dial tcp 104.36.82.61:4394: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 22:10:45.676927984 (12h30m45.266272961s) [seg_pos={0 0}, upload=12997]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 22:10:45.677012455 (12h30m45.266357443s) [upload=12997]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 22:10:45.677016543 (12h30m45.26636152s) [upload=12997]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:10:45.998268658 (12h30m45.587613635s) [upload=12997]: Done uploading segments... Aug 7 22:10:45.998285902 (12h30m45.587630889s) [upload=12997]: Upload finished. err: <nil> Duration: 8.020550699s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 21 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXX.X...X...................................X] duration: 30.000701872s FAIL (2025-08-07 22:12:14 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== . Aug 7 22:04:32.232433629 (12h24m31.821778616s) [upload=12883]: Upload finished. err: <nil> Aug 7 22:05:36.062977324 (12h25m35.652322311s) [upload=12901]: Starting upload Aug 7 22:05:36.063061485 (12h25m35.652406472s) [upload=12901]: Done starting upload Aug 7 22:05:36.063107994 (12h25m35.652452981s) [upload=12901]: Uploading segments... Aug 7 22:05:36.063139605 (12h25m35.652484582s) [upload=12901]: Got next segment. Inline: true Aug 7 22:05:36.063143231 (12h25m35.652488208s) [upload=12901]: Waiting for error group managing segments... Aug 7 22:05:36.063175854 (12h25m35.652520841s) [upload=12901]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:05:36.063179671 (12h25m35.652524648s) [upload=12901]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:05:36.063182005 (12h25m35.652526992s) [upload=12901]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:05:36.333388461 (12h25m35.922733478s) [upload=12901]: Done uploading segments... Aug 7 22:05:36.333411826 (12h25m35.922756853s) [upload=12901]: Upload finished. err: <nil> Aug 7 22:06:23.632385139 (12h26m23.221730126s) [upload=12916]: Starting upload Aug 7 22:06:23.632456987 (12h26m23.221801974s) [upload=12916]: Done starting upload Aug 7 22:06:23.632524355 (12h26m23.221869342s) [upload=12916]: Uploading segments... Aug 7 22:06:23.632558691 (12h26m23.221903679s) [upload=12916]: Got next segment. Inline: true Aug 7 22:06:23.632562028 (12h26m23.221907015s) [upload=12916]: Waiting for error group managing segments... Aug 7 22:06:23.632596073 (12h26m23.22194105s) [upload=12916]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:06:23.632600020 (12h26m23.221945007s) [upload=12916]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:06:23.632602304 (12h26m23.221947281s) [upload=12916]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:06:23.903184888 (12h26m23.492529875s) [upload=12916]: Done uploading segments... Aug 7 22:06:23.903202192 (12h26m23.492547179s) [upload=12916]: Upload finished. err: <nil> Aug 7 22:07:18.062211999 (12h27m17.651556986s) [upload=12928]: Starting upload Aug 7 22:07:18.062271193 (12h27m17.651616181s) [upload=12928]: Done starting upload Aug 7 22:07:18.062330397 (12h27m17.651675374s) [upload=12928]: Uploading segments... Aug 7 22:07:18.062360224 (12h27m17.651705211s) [upload=12928]: Got next segment. Inline: true Aug 7 22:07:18.062363690 (12h27m17.651708677s) [upload=12928]: Waiting for error group managing segments... Aug 7 22:07:18.062394940 (12h27m17.651739917s) [upload=12928]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:07:18.062398466 (12h27m17.651743453s) [upload=12928]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:07:18.062400671 (12h27m17.651745648s) [upload=12928]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:07:18.332201201 (12h27m17.921546188s) [upload=12928]: Done uploading segments... Aug 7 22:07:18.332221129 (12h27m17.921566116s) [upload=12928]: Upload finished. err: <nil> Aug 7 22:08:06.628735015 (12h28m6.218080002s) [upload=12949]: Starting upload Aug 7 22:08:06.628795140 (12h28m6.218140127s) [upload=12949]: Done starting upload Aug 7 22:08:06.628857911 (12h28m6.218202898s) [upload=12949]: Uploading segments... Aug 7 22:08:06.628887237 (12h28m6.218232224s) [upload=12949]: Got next segment. Inline: true Aug 7 22:08:06.628890713 (12h28m6.2182357s) [upload=12949]: Waiting for error group managing segments... Aug 7 22:08:06.628923866 (12h28m6.218268853s) [upload=12949]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:08:06.628935168 (12h28m6.218280195s) [upload=12949]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:08:06.628938295 (12h28m6.218283272s) [upload=12949]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:08:06.909068236 (12h28m6.498413233s) [upload=12949]: Done uploading segments... Aug 7 22:08:06.909091591 (12h28m6.498436598s) [upload=12949]: Upload finished. err: <nil> Aug 7 22:08:44.103027985 (12h28m43.692372972s) [upload=12960]: Starting upload Aug 7 22:08:44.103093421 (12h28m43.692438408s) [upload=12960]: Done starting upload Aug 7 22:08:44.103142554 (12h28m43.692487541s) [upload=12960]: Uploading segments... Aug 7 22:08:44.103198422 (12h28m43.692543419s) [upload=12960]: Got next segment. Inline: true Aug 7 22:08:44.103202429 (12h28m43.692547416s) [upload=12960]: Waiting for error group managing segments... Aug 7 22:08:44.103235592 (12h28m43.69258058s) [upload=12960]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:08:44.103239219 (12h28m43.692584206s) [upload=12960]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:08:44.103241373 (12h28m43.69258636s) [upload=12960]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:08:44.393696841 (12h28m43.983041818s) [upload=12960]: Done uploading segments... Aug 7 22:08:44.393715166 (12h28m43.983060143s) [upload=12960]: Upload finished. err: <nil> Aug 7 22:09:14.345236113 (12h29m13.9345811s) [upload=12972]: Starting upload Aug 7 22:09:14.345291719 (12h29m13.934636706s) [upload=12972]: Done starting upload Aug 7 22:09:14.345358137 (12h29m13.934703114s) [upload=12972]: Uploading segments... Aug 7 22:09:14.345388645 (12h29m13.934733632s) [upload=12972]: Got next segment. Inline: true Aug 7 22:09:14.345392141 (12h29m13.934737118s) [upload=12972]: Waiting for error group managing segments... Aug 7 22:09:14.345426327 (12h29m13.934771314s) [upload=12972]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:09:14.345430324 (12h29m13.934775301s) [upload=12972]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:09:14.345432839 (12h29m13.934777826s) [upload=12972]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:09:14.834006850 (12h29m14.423351857s) [upload=12972]: Done uploading segments... Aug 7 22:09:14.834026347 (12h29m14.423371334s) [upload=12972]: Upload finished. err: <nil> Aug 7 22:09:53.131289328 (12h29m52.720634315s) [upload=12982]: Starting upload Aug 7 22:09:53.131364442 (12h29m52.720709489s) [upload=12982]: Done starting upload Aug 7 22:09:53.131436210 (12h29m52.720781197s) [upload=12982]: Uploading segments... Aug 7 22:09:53.131465205 (12h29m52.720810192s) [upload=12982]: Got next segment. Inline: true Aug 7 22:09:53.131468902 (12h29m52.720813889s) [upload=12982]: Waiting for error group managing segments... Aug 7 22:09:53.131498809 (12h29m52.720843796s) [upload=12982]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:09:53.131503017 (12h29m52.720848004s) [upload=12982]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:09:53.131505141 (12h29m52.720850118s) [upload=12982]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:09:53.419040148 (12h29m53.008385125s) [upload=12982]: Done uploading segments... Aug 7 22:09:53.419058042 (12h29m53.008403029s) [upload=12982]: Upload finished. err: <nil> Aug 7 22:10:12.329552115 (12h30m11.918897102s) [upload=12989]: Starting upload Aug 7 22:10:12.330067942 (12h30m11.919412929s) [upload=12989]: Done starting upload Aug 7 22:10:12.330139077 (12h30m11.919484054s) [upload=12989]: Uploading segments... Aug 7 22:10:12.330186228 (12h30m11.919531255s) [upload=12989]: Got next segment. Inline: true Aug 7 22:10:12.330192710 (12h30m11.919537737s) [upload=12989]: Waiting for error group managing segments... Aug 7 22:10:12.330240762 (12h30m11.919585749s) [upload=12989]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:10:12.330248126 (12h30m11.919593113s) [upload=12989]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:10:12.330252083 (12h30m11.91959709s) [upload=12989]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:10:12.591107606 (12h30m12.180452593s) [upload=12989]: Done uploading segments... Aug 7 22:10:12.591124920 (12h30m12.180469907s) [upload=12989]: Upload finished. err: <nil> Aug 7 22:10:57.486329607 (12h30m57.075674594s) [upload=13003]: Starting upload Aug 7 22:10:57.486414950 (12h30m57.075759927s) [upload=13003]: Done starting upload Aug 7 22:10:57.486491847 (12h30m57.075836834s) [upload=13003]: Uploading segments... Aug 7 22:10:57.486573694 (12h30m57.075918681s) [upload=13003]: Got next segment. Inline: true Aug 7 22:10:57.486578242 (12h30m57.07592322s) [upload=13003]: Waiting for error group managing segments... Aug 7 22:10:57.486645681 (12h30m57.075990658s) [upload=13003]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:10:57.486650361 (12h30m57.075995348s) [upload=13003]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:10:57.486652795 (12h30m57.075997782s) [upload=13003]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:10:57.751495816 (12h30m57.340840803s) [upload=13003]: Done uploading segments... Aug 7 22:10:57.751515013 (12h30m57.34086s) [upload=13003]: Upload finished. err: <nil> Aug 7 22:11:44.715949867 (12h31m44.305294854s) [upload=13015]: Starting upload Aug 7 22:11:44.716016213 (12h31m44.3053612s) [upload=13015]: Done starting upload Aug 7 22:11:44.716066600 (12h31m44.305411587s) [upload=13015]: Uploading segments... Aug 7 22:11:44.716104322 (12h31m44.305449309s) [upload=13015]: Got next segment. Inline: true Aug 7 22:11:44.716107879 (12h31m44.305452866s) [upload=13015]: Waiting for error group managing segments... Aug 7 22:11:44.716139289 (12h31m44.305484276s) [upload=13015]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:11:44.716142826 (12h31m44.305487803s) [upload=13015]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:11:44.716146493 (12h31m44.30549147s) [upload=13015]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:12:14.067179909 (12h32m13.656524896s) [upload=13015]: Done uploading segments... Aug 7 22:12:14.067200478 (12h32m13.656545465s) [upload=13015]: Upload finished. err: metaclient: context deadline exceeded Duration: 30.00072174s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXX.......X................................] duration: 23.137692413s FAIL (2025-08-07 22:12:14 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 23.137710207s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 25 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXX.X.X.X................................X] duration: 30.00056776s FAIL (2025-08-07 22:12:10 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== 7768206 (12h29m48.577113193s) [upload=12980]: Done uploading segments... Aug 7 22:09:48.987775841 (12h29m48.577120828s) [upload=12980]: Upload finished. err: <nil> Aug 7 22:10:52.973210152 (12h30m52.562555139s) [upload=13000]: Starting upload Aug 7 22:10:52.973271730 (12h30m52.562616707s) [upload=13000]: Done starting upload Aug 7 22:10:52.973394736 (12h30m52.562739723s) [upload=13000]: Uploading segments... Aug 7 22:10:52.973614526 (12h30m52.562959513s) [upload=13000]: Got next segment. Inline: false Aug 7 22:10:52.973660134 (12h30m52.563005121s) [upload=13000]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:10:52.973664692 (12h30m52.563009679s) [upload=13000]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:10:53.209567723 (12h30m52.79891271s) [seg_pos={0 0}, upload=13000]: Begin upload segment... Aug 7 22:10:53.209604394 (12h30m52.798949381s) [seg_pos={0 0}, upload=13000]: Done begin upload segment. Aug 7 22:10:53.209625824 (12h30m52.798970811s) [noise=true , address=77.90.31.12:4590 , piece_id=QCC2XA4NPGVTUZXRRNXV22QVU77S6QB6R4QEQIYUAFWVON3J35RQ, seg_pos={0 0}, upload=13000]: Uploading piece... Aug 7 22:10:53.209635153 (12h30m52.79898014s) [noise=true , address=104.36.82.4:4804 , piece_id=BI6ZPR56BKHTA2QGM3BNUBFJM6PAD4K27P5TYHRGLG4NBD56LC4A, seg_pos={0 0}, upload=13000]: Uploading piece... Aug 7 22:10:53.209952860 (12h30m52.799297837s) [noise=true , address=45.91.116.4:4939 , piece_id=FQ4Z6Y4KIOGJLPCZFVPCUJSYOMGDOTZNAIEKGNWKASCLKOWLCTUQ, seg_pos={0 0}, upload=13000]: Uploading piece... Aug 7 22:10:53.209970634 (12h30m52.799315621s) [noise=true , address=77.90.31.10:4520 , piece_id=KSMIVSJ4QSOZAXZN7EVTTC4DGFR2TL4JLJQW2NPYYGWAJDKSZ2ZA, seg_pos={0 0}, upload=13000]: Uploading piece... Aug 7 22:10:53.210195384 (12h30m52.799540371s) [noise=true , address=104.36.82.4:4802 , piece_id=B4Q65J747GHE2ZRZDM7RLWC3AQC7TLK5D53REAUA5DMHT3TGA44Q, seg_pos={0 0}, upload=13000]: Uploading piece... Aug 7 22:10:53.210227876 (12h30m52.799572873s) [noise=true , address=77.90.31.18:4805 , piece_id=VQ3CGKCPEKYPCAIOHQ2FZSINJBHBCCQUZD2AD4AIHX72SZN7SEKA, seg_pos={0 0}, upload=13000]: Uploading piece... Aug 7 22:10:53.210467886 (12h30m52.799812863s) [noise=true , address=45.91.116.7:4009 , piece_id=47X2WBQ4XYRNAZDBVU572QSRHXUNSYRCOUALM4SF4WRHCDW6QTCA, seg_pos={0 0}, upload=13000]: Uploading piece... Aug 7 22:10:53.210477654 (12h30m52.799822641s) [noise=true , address=77.90.31.13:4627 , piece_id=3IQP6QL756UL7CNL357KLNUTHQCSEG5337GUT2PD2AS623PTIYCA, seg_pos={0 0}, upload=13000]: Uploading piece... Aug 7 22:10:53.210669050 (12h30m52.800014027s) [noise=true , address=77.90.31.17:4770 , piece_id=BG2ZL7XC52NEJC5FRS4V75MEPJH34B27IPLTJNQLSX4N3OZGVHXA, seg_pos={0 0}, upload=13000]: Uploading piece... Aug 7 22:10:53.210729946 (12h30m52.800074934s) [noise=true , address=45.91.116.5:4298 , piece_id=XJIIUMO7PI6XTBLXE7IRPS2YYI53MRGPEW7HS5CWFXUSIXBT5MBQ, seg_pos={0 0}, upload=13000]: Uploading piece... Aug 7 22:10:53.210967301 (12h30m52.800312288s) [noise=true , address=45.91.116.14:4189 , piece_id=WTRHRDN2HJN3MQI6334ZHVKLMCT7XH7S42MKYCEK5RY2JHDBESOQ, seg_pos={0 0}, upload=13000]: Uploading piece... Aug 7 22:10:53.211206058 (12h30m52.800551045s) [noise=true , address=104.36.82.12:4150 , piece_id=XG7E3FHDR2VCPREGTWFFJ52JLGUPRUKQJ75KFYEUXPI4OUNTB3ZA, seg_pos={0 0}, upload=13000]: Uploading piece... Aug 7 22:10:53.211450766 (12h30m52.800795753s) [noise=true , address=104.36.82.7:4909 , piece_id=6ZPKL2IEZN54MOXLAC7P4GURLWMJ7MUBNCBXPUGF6TD6257G7TAA, seg_pos={0 0}, upload=13000]: Uploading piece... Aug 7 22:10:53.211637383 (12h30m52.80098237s) [noise=true , address=104.36.82.6:4873 , piece_id=FIZQQ56RDZJUYC2JM24K5UMWTQI64SLS56HH54F4EUIXCWH46FOQ, seg_pos={0 0}, upload=13000]: Uploading piece... Aug 7 22:10:53.211831044 (12h30m52.801176031s) [noise=true , address=45.91.116.13:4154 , piece_id=TPFKH6V3LAM6B2O7L744BZPLT64I2BZVYJXFPEPLIGMVEQ7246YQ, seg_pos={0 0}, upload=13000]: Uploading piece... Aug 7 22:10:53.212051075 (12h30m52.801396062s) [noise=true , address=77.90.31.11:4555 , piece_id=A6WWFITSRKE4GVTQODOYU36ORK7OVE3MVVR7B4FPSSP64DATOQ6A, seg_pos={0 0}, upload=13000]: Uploading piece... Aug 7 22:10:54.951706811 (12h30m54.541051848s) [upload=13000]: Next returned nil segment Aug 7 22:10:54.951730396 (12h30m54.541075413s) [upload=13000]: Waiting for error group managing segments... Aug 7 22:10:55.341563384 (12h30m54.930908361s) [noise=true , address=77.90.31.10:4520 , piece_id=KSMIVSJ4QSOZAXZN7EVTTC4DGFR2TL4JLJQW2NPYYGWAJDKSZ2ZA, seg_pos={0 0}, upload=13000]: Done uploading piece. err: <nil> Aug 7 22:10:55.391397870 (12h30m54.980742857s) [noise=true , address=45.91.116.14:4189 , piece_id=WTRHRDN2HJN3MQI6334ZHVKLMCT7XH7S42MKYCEK5RY2JHDBESOQ, seg_pos={0 0}, upload=13000]: Done uploading piece. err: <nil> Aug 7 22:10:55.429722625 (12h30m55.019067602s) [noise=true , address=77.90.31.12:4590 , piece_id=QCC2XA4NPGVTUZXRRNXV22QVU77S6QB6R4QEQIYUAFWVON3J35RQ, seg_pos={0 0}, upload=13000]: Done uploading piece. err: <nil> Aug 7 22:10:55.430446209 (12h30m55.019791196s) [noise=true , address=104.36.82.6:4873 , piece_id=FIZQQ56RDZJUYC2JM24K5UMWTQI64SLS56HH54F4EUIXCWH46FOQ, seg_pos={0 0}, upload=13000]: Done uploading piece. err: <nil> Aug 7 22:10:55.438319623 (12h30m55.0276646s) [noise=true , address=45.91.116.5:4298 , piece_id=XJIIUMO7PI6XTBLXE7IRPS2YYI53MRGPEW7HS5CWFXUSIXBT5MBQ, seg_pos={0 0}, upload=13000]: Done uploading piece. err: <nil> Aug 7 22:10:55.440502008 (12h30m55.029846995s) [noise=true , address=45.91.116.13:4154 , piece_id=TPFKH6V3LAM6B2O7L744BZPLT64I2BZVYJXFPEPLIGMVEQ7246YQ, seg_pos={0 0}, upload=13000]: Done uploading piece. err: <nil> Aug 7 22:10:55.449485475 (12h30m55.038830462s) [noise=true , address=104.36.82.7:4909 , piece_id=6ZPKL2IEZN54MOXLAC7P4GURLWMJ7MUBNCBXPUGF6TD6257G7TAA, seg_pos={0 0}, upload=13000]: Done uploading piece. err: <nil> Aug 7 22:10:55.449773497 (12h30m55.039118484s) [noise=true , address=77.90.31.13:4627 , piece_id=3IQP6QL756UL7CNL357KLNUTHQCSEG5337GUT2PD2AS623PTIYCA, seg_pos={0 0}, upload=13000]: Done uploading piece. err: <nil> Aug 7 22:10:55.486432234 (12h30m55.075777221s) [noise=true , address=77.90.31.11:4555 , piece_id=A6WWFITSRKE4GVTQODOYU36ORK7OVE3MVVR7B4FPSSP64DATOQ6A, seg_pos={0 0}, upload=13000]: Done uploading piece. err: <nil> Aug 7 22:10:55.522908903 (12h30m55.11225389s) [noise=true , address=77.90.31.18:4805 , piece_id=VQ3CGKCPEKYPCAIOHQ2FZSINJBHBCCQUZD2AD4AIHX72SZN7SEKA, seg_pos={0 0}, upload=13000]: Done uploading piece. err: <nil> Aug 7 22:10:55.546118308 (12h30m55.135463295s) [noise=true , address=104.36.82.4:4802 , piece_id=B4Q65J747GHE2ZRZDM7RLWC3AQC7TLK5D53REAUA5DMHT3TGA44Q, seg_pos={0 0}, upload=13000]: Done uploading piece. err: <nil> Aug 7 22:10:55.568801126 (12h30m55.158146113s) [noise=true , address=104.36.82.4:4804 , piece_id=BI6ZPR56BKHTA2QGM3BNUBFJM6PAD4K27P5TYHRGLG4NBD56LC4A, seg_pos={0 0}, upload=13000]: Done uploading piece. err: <nil> Aug 7 22:10:55.587208293 (12h30m55.176553341s) [noise=true , address=77.90.31.17:4770 , piece_id=BG2ZL7XC52NEJC5FRS4V75MEPJH34B27IPLTJNQLSX4N3OZGVHXA, seg_pos={0 0}, upload=13000]: Done uploading piece. err: <nil> Aug 7 22:10:55.587245686 (12h30m55.176590713s) [seg_pos={0 0}, upload=13000]: Segment reached optimal threshold of 13 pieces. Aug 7 22:10:55.587371516 (12h30m55.176716533s) [noise=true , address=45.91.116.4:4939 , piece_id=FQ4Z6Y4KIOGJLPCZFVPCUJSYOMGDOTZNAIEKGNWKASCLKOWLCTUQ, seg_pos={0 0}, upload=13000]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: expected piece hash; context canceled; EOF Aug 7 22:10:55.587442001 (12h30m55.176786988s) [noise=true , address=45.91.116.7:4009 , piece_id=47X2WBQ4XYRNAZDBVU572QSRHXUNSYRCOUALM4SF4WRHCDW6QTCA, seg_pos={0 0}, upload=13000]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF Aug 7 22:10:55.587644678 (12h30m55.176989726s) [noise=true , address=104.36.82.12:4150 , piece_id=XG7E3FHDR2VCPREGTWFFJ52JLGUPRUKQJ75KFYEUXPI4OUNTB3ZA, seg_pos={0 0}, upload=13000]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK): protocol: context canceled; context canceled Aug 7 22:10:55.587681550 (12h30m55.177026557s) [seg_pos={0 0}, upload=13000]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 7 22:10:55.587777072 (12h30m55.177122109s) [upload=13000]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 22:10:55.587787912 (12h30m55.177132929s) [upload=13000]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:10:55.887211865 (12h30m55.476556883s) [upload=13000]: Done uploading segments... Aug 7 22:10:55.887228857 (12h30m55.476573844s) [upload=13000]: Upload finished. err: <nil> Aug 7 22:11:40.872949328 (12h31m40.462294315s) [upload=13014]: Starting upload Aug 7 22:11:40.873003161 (12h31m40.462348148s) [upload=13014]: Done starting upload Aug 7 22:11:40.873072924 (12h31m40.462417911s) [upload=13014]: Uploading segments... Aug 7 22:11:40.873431210 (12h31m40.462776287s) [upload=13014]: Got next segment. Inline: false Aug 7 22:11:40.873471537 (12h31m40.462816525s) [upload=13014]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:11:40.873480314 (12h31m40.462825331s) [upload=13014]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:12:10.229450117 (12h32m9.818795124s) [upload=13014]: Next segment err: context deadline exceeded Aug 7 22:12:10.229589654 (12h32m9.818934631s) [upload=13014]: Done uploading segments... Aug 7 22:12:10.229603972 (12h32m9.818948949s) [upload=13014]: Upload finished. err: context deadline exceeded Duration: 30.000589711s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 26 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXX.XX..X...............................X] duration: 1m0.00041941s FAIL (2025-08-07 22:12:10 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== . Aug 7 22:04:30.683987657 (12h24m30.273332634s) [upload=12882]: Upload finished. err: <nil> Aug 7 22:05:20.868010826 (12h25m20.457355803s) [upload=12897]: Starting upload Aug 7 22:05:20.868072895 (12h25m20.457417882s) [upload=12897]: Done starting upload Aug 7 22:05:20.868114604 (12h25m20.457459591s) [upload=12897]: Uploading segments... Aug 7 22:05:20.868144592 (12h25m20.45748958s) [upload=12897]: Got next segment. Inline: true Aug 7 22:05:20.868148259 (12h25m20.457493246s) [upload=12897]: Waiting for error group managing segments... Aug 7 22:05:20.868177906 (12h25m20.457522883s) [upload=12897]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:05:20.868181673 (12h25m20.45752665s) [upload=12897]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:05:20.868184248 (12h25m20.457529235s) [upload=12897]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:05:21.169244085 (12h25m20.758589072s) [upload=12897]: Done uploading segments... Aug 7 22:05:21.169262229 (12h25m20.758607216s) [upload=12897]: Upload finished. err: <nil> Aug 7 22:05:56.583599672 (12h25m56.172944659s) [upload=12908]: Starting upload Aug 7 22:05:56.583668895 (12h25m56.173013882s) [upload=12908]: Done starting upload Aug 7 22:05:56.583716636 (12h25m56.173061623s) [upload=12908]: Uploading segments... Aug 7 22:05:56.583747204 (12h25m56.173092191s) [upload=12908]: Got next segment. Inline: true Aug 7 22:05:56.583750742 (12h25m56.173095729s) [upload=12908]: Waiting for error group managing segments... Aug 7 22:05:56.583785789 (12h25m56.173130776s) [upload=12908]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:05:56.583789445 (12h25m56.173134432s) [upload=12908]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:05:56.583795787 (12h25m56.173140774s) [upload=12908]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:05:56.916665113 (12h25m56.5060101s) [upload=12908]: Done uploading segments... Aug 7 22:05:56.916682988 (12h25m56.506027975s) [upload=12908]: Upload finished. err: <nil> Aug 7 22:06:48.168861264 (12h26m47.758206241s) [upload=12922]: Starting upload Aug 7 22:06:48.168923543 (12h26m47.75826853s) [upload=12922]: Done starting upload Aug 7 22:06:48.168989780 (12h26m47.758334767s) [upload=12922]: Uploading segments... Aug 7 22:06:48.169093067 (12h26m47.758438044s) [upload=12922]: Got next segment. Inline: true Aug 7 22:06:48.169098358 (12h26m47.758443345s) [upload=12922]: Waiting for error group managing segments... Aug 7 22:06:48.169143945 (12h26m47.758488932s) [upload=12922]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:06:48.169149034 (12h26m47.758494021s) [upload=12922]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:06:48.169151550 (12h26m47.758496537s) [upload=12922]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:06:48.443503957 (12h26m48.032848934s) [upload=12922]: Done uploading segments... Aug 7 22:06:48.443520579 (12h26m48.032865556s) [upload=12922]: Upload finished. err: <nil> Aug 7 22:07:24.544076557 (12h27m24.133421534s) [upload=12933]: Starting upload Aug 7 22:07:24.544154457 (12h27m24.133499444s) [upload=12933]: Done starting upload Aug 7 22:07:24.544212037 (12h27m24.133557024s) [upload=12933]: Uploading segments... Aug 7 22:07:24.544273624 (12h27m24.133618611s) [upload=12933]: Got next segment. Inline: true Aug 7 22:07:24.544278614 (12h27m24.133623601s) [upload=12933]: Waiting for error group managing segments... Aug 7 22:07:24.544329832 (12h27m24.133674819s) [upload=12933]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:07:24.544334911 (12h27m24.133679898s) [upload=12933]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:07:24.544337847 (12h27m24.133682834s) [upload=12933]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:07:24.872527048 (12h27m24.461872036s) [upload=12933]: Done uploading segments... Aug 7 22:07:24.872546185 (12h27m24.461891172s) [upload=12933]: Upload finished. err: <nil> Aug 7 22:07:42.678048362 (12h27m42.267393349s) [upload=12938]: Starting upload Aug 7 22:07:42.678122394 (12h27m42.267467381s) [upload=12938]: Done starting upload Aug 7 22:07:42.678184523 (12h27m42.26752951s) [upload=12938]: Uploading segments... Aug 7 22:07:42.678220613 (12h27m42.2675656s) [upload=12938]: Got next segment. Inline: true Aug 7 22:07:42.678225311 (12h27m42.267570299s) [upload=12938]: Waiting for error group managing segments... Aug 7 22:07:42.678261721 (12h27m42.267606708s) [upload=12938]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:07:42.678265758 (12h27m42.267610735s) [upload=12938]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:07:42.678268203 (12h27m42.26761319s) [upload=12938]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:07:42.941204890 (12h27m42.530549877s) [upload=12938]: Done uploading segments... Aug 7 22:07:42.941220821 (12h27m42.530565808s) [upload=12938]: Upload finished. err: <nil> Aug 7 22:08:12.564234560 (12h28m12.153579557s) [upload=12951]: Starting upload Aug 7 22:08:12.564392563 (12h28m12.15373757s) [upload=12951]: Done starting upload Aug 7 22:08:12.564478317 (12h28m12.153823314s) [upload=12951]: Uploading segments... Aug 7 22:08:12.564532671 (12h28m12.153877658s) [upload=12951]: Got next segment. Inline: true Aug 7 22:08:12.564537029 (12h28m12.153882006s) [upload=12951]: Waiting for error group managing segments... Aug 7 22:08:12.564572748 (12h28m12.153917735s) [upload=12951]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:08:12.564577227 (12h28m12.153922204s) [upload=12951]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:08:12.564579511 (12h28m12.153924498s) [upload=12951]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:08:12.853863397 (12h28m12.443208384s) [upload=12951]: Done uploading segments... Aug 7 22:08:12.853882363 (12h28m12.44322735s) [upload=12951]: Upload finished. err: <nil> Aug 7 22:08:48.226332068 (12h28m47.815677055s) [upload=12963]: Starting upload Aug 7 22:08:48.226393765 (12h28m47.815738743s) [upload=12963]: Done starting upload Aug 7 22:08:48.226441998 (12h28m47.815786975s) [upload=12963]: Uploading segments... Aug 7 22:08:48.226471044 (12h28m47.815816031s) [upload=12963]: Got next segment. Inline: true Aug 7 22:08:48.226474330 (12h28m47.815819317s) [upload=12963]: Waiting for error group managing segments... Aug 7 22:08:48.226505239 (12h28m47.815850216s) [upload=12963]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:08:48.226508785 (12h28m47.815853772s) [upload=12963]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:08:48.226511250 (12h28m47.815856237s) [upload=12963]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:08:48.493962093 (12h28m48.08330708s) [upload=12963]: Done uploading segments... Aug 7 22:08:48.493980028 (12h28m48.083325015s) [upload=12963]: Upload finished. err: <nil> Aug 7 22:09:16.735671384 (12h29m16.325016371s) [upload=12973]: Starting upload Aug 7 22:09:16.735728593 (12h29m16.32507358s) [upload=12973]: Done starting upload Aug 7 22:09:16.735770444 (12h29m16.325115431s) [upload=12973]: Uploading segments... Aug 7 22:09:16.735799409 (12h29m16.325144396s) [upload=12973]: Got next segment. Inline: true Aug 7 22:09:16.735802535 (12h29m16.325147522s) [upload=12973]: Waiting for error group managing segments... Aug 7 22:09:16.735831881 (12h29m16.325176868s) [upload=12973]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:09:16.735835217 (12h29m16.325180204s) [upload=12973]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:09:16.735837421 (12h29m16.325182408s) [upload=12973]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:09:17.021230773 (12h29m16.61057576s) [upload=12973]: Done uploading segments... Aug 7 22:09:17.021247365 (12h29m16.610592352s) [upload=12973]: Upload finished. err: <nil> Aug 7 22:10:03.229004755 (12h30m2.818349752s) [upload=12987]: Starting upload Aug 7 22:10:03.229086592 (12h30m2.818431589s) [upload=12987]: Done starting upload Aug 7 22:10:03.229165603 (12h30m2.8185106s) [upload=12987]: Uploading segments... Aug 7 22:10:03.229268600 (12h30m2.818613618s) [upload=12987]: Got next segment. Inline: true Aug 7 22:10:03.229296584 (12h30m2.818641581s) [upload=12987]: Waiting for error group managing segments... Aug 7 22:10:03.229348544 (12h30m2.818693531s) [upload=12987]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:10:03.229354835 (12h30m2.818699822s) [upload=12987]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:10:03.229357520 (12h30m2.818702517s) [upload=12987]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:10:03.732013236 (12h30m3.321358233s) [upload=12987]: Done uploading segments... Aug 7 22:10:03.732033424 (12h30m3.321378411s) [upload=12987]: Upload finished. err: <nil> Aug 7 22:11:11.152680180 (12h31m10.742025167s) [upload=13005]: Starting upload Aug 7 22:11:11.152732820 (12h31m10.742077807s) [upload=13005]: Done starting upload Aug 7 22:11:11.152799828 (12h31m10.742144815s) [upload=13005]: Uploading segments... Aug 7 22:11:11.152844534 (12h31m10.742189531s) [upload=13005]: Got next segment. Inline: true Aug 7 22:11:11.152848501 (12h31m10.742193489s) [upload=13005]: Waiting for error group managing segments... Aug 7 22:11:11.152885622 (12h31m10.742230609s) [upload=13005]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:11:11.152889760 (12h31m10.742234747s) [upload=13005]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:11:11.152893338 (12h31m10.742238325s) [upload=13005]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:12:10.507162881 (12h32m10.096507878s) [upload=13005]: Done uploading segments... Aug 7 22:12:10.507172128 (12h32m10.096517115s) [upload=13005]: Upload finished. err: metaclient: context deadline exceeded Duration: 1m0.000452102s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 25 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXX...X................................XX] duration: 30.000634187s FAIL (2025-08-07 22:12:10 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 30.000653844s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 27 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXX...X..X............................X] duration: 1m0.001361786s FAIL (2025-08-07 22:12:08 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.001385211s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 3 out of 60 is failed (threshold=3) [......X..........X.................X........................] duration: 1.869818046s FAIL (2025-08-07 22:12:01 UTC) orbiter warning: operation: ListObjects ======ERROR====== unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/1K ======LOGS====== 8m27.919686751s) [upload=12612]: Done uploading segments... Aug 7 21:48:28.330358527 (12h8m27.919703514s) [upload=12612]: Upload finished. err: <nil> Aug 7 21:48:54.943937458 (12h8m54.533282445s) [upload=12618]: Starting upload Aug 7 21:48:54.944002622 (12h8m54.533347609s) [upload=12618]: Done starting upload Aug 7 21:48:54.944106100 (12h8m54.533451087s) [upload=12618]: Uploading segments... Aug 7 21:48:54.944196242 (12h8m54.53354132s) [upload=12618]: Got next segment. Inline: true Aug 7 21:48:54.944206543 (12h8m54.53355158s) [upload=12618]: Waiting for error group managing segments... Aug 7 21:48:54.944254484 (12h8m54.533599461s) [upload=12618]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:48:54.944259223 (12h8m54.5336042s) [upload=12618]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:48:54.944262139 (12h8m54.533607126s) [upload=12618]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:48:55.330753187 (12h8m54.920098174s) [upload=12618]: Done uploading segments... Aug 7 21:48:55.330774959 (12h8m54.920119936s) [upload=12618]: Upload finished. err: <nil> Aug 7 21:49:36.650951942 (12h9m36.240296929s) [upload=12631]: Starting upload Aug 7 21:49:36.650993252 (12h9m36.240338239s) [upload=12631]: Done starting upload Aug 7 21:49:36.651049029 (12h9m36.240394016s) [upload=12631]: Uploading segments... Aug 7 21:49:36.651075309 (12h9m36.240420286s) [upload=12631]: Got next segment. Inline: true Aug 7 21:49:36.651078775 (12h9m36.240423763s) [upload=12631]: Waiting for error group managing segments... Aug 7 21:49:36.651115776 (12h9m36.240460773s) [upload=12631]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:49:36.651120254 (12h9m36.240465251s) [upload=12631]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:49:36.651124473 (12h9m36.24046947s) [upload=12631]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:49:36.920683881 (12h9m36.510028868s) [upload=12631]: Done uploading segments... Aug 7 21:49:36.920707246 (12h9m36.510052233s) [upload=12631]: Upload finished. err: <nil> Aug 7 21:50:03.363429937 (12h10m2.952774924s) [upload=12636]: Starting upload Aug 7 21:50:03.363493408 (12h10m2.952838405s) [upload=12636]: Done starting upload Aug 7 21:50:03.363582518 (12h10m2.952927515s) [upload=12636]: Uploading segments... Aug 7 21:50:03.363692078 (12h10m2.953037085s) [upload=12636]: Got next segment. Inline: true Aug 7 21:50:03.363699752 (12h10m2.953044739s) [upload=12636]: Waiting for error group managing segments... Aug 7 21:50:03.363764116 (12h10m2.953109103s) [upload=12636]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:50:03.363773323 (12h10m2.95311831s) [upload=12636]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:50:03.363777271 (12h10m2.953122258s) [upload=12636]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:50:03.636022787 (12h10m3.225367774s) [upload=12636]: Done uploading segments... Aug 7 21:50:03.636042204 (12h10m3.225387202s) [upload=12636]: Upload finished. err: <nil> Aug 7 21:50:20.944550175 (12h10m20.533895152s) [upload=12642]: Starting upload Aug 7 21:50:20.944594911 (12h10m20.533939898s) [upload=12642]: Done starting upload Aug 7 21:50:20.944644405 (12h10m20.533989392s) [upload=12642]: Uploading segments... Aug 7 21:50:20.944674473 (12h10m20.534019461s) [upload=12642]: Got next segment. Inline: true Aug 7 21:50:20.944677820 (12h10m20.534022807s) [upload=12642]: Waiting for error group managing segments... Aug 7 21:50:20.944709450 (12h10m20.534054437s) [upload=12642]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:50:20.944715131 (12h10m20.534060118s) [upload=12642]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:50:20.944717505 (12h10m20.534062493s) [upload=12642]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:50:21.212077476 (12h10m20.801422473s) [upload=12642]: Done uploading segments... Aug 7 21:50:21.212101602 (12h10m20.801446599s) [upload=12642]: Upload finished. err: <nil> Aug 7 21:51:22.649621444 (12h11m22.238966431s) [upload=12660]: Starting upload Aug 7 21:51:22.649696837 (12h11m22.239041824s) [upload=12660]: Done starting upload Aug 7 21:51:22.649777742 (12h11m22.239122729s) [upload=12660]: Uploading segments... Aug 7 21:51:22.649867044 (12h11m22.239212041s) [upload=12660]: Got next segment. Inline: true Aug 7 21:51:22.649872053 (12h11m22.23921704s) [upload=12660]: Waiting for error group managing segments... Aug 7 21:51:22.649940053 (12h11m22.23928504s) [upload=12660]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:51:22.649946785 (12h11m22.239291772s) [upload=12660]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:51:22.649949932 (12h11m22.239294919s) [upload=12660]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:51:22.914527578 (12h11m22.503872565s) [upload=12660]: Done uploading segments... Aug 7 21:51:22.914549130 (12h11m22.503894117s) [upload=12660]: Upload finished. err: <nil> Aug 7 21:51:38.628060070 (12h11m38.217405057s) [upload=12665]: Starting upload Aug 7 21:51:38.628100548 (12h11m38.217445535s) [upload=12665]: Done starting upload Aug 7 21:51:38.628155373 (12h11m38.21750036s) [upload=12665]: Uploading segments... Aug 7 21:51:38.628187814 (12h11m38.217532802s) [upload=12665]: Got next segment. Inline: true Aug 7 21:51:38.628191381 (12h11m38.217536368s) [upload=12665]: Waiting for error group managing segments... Aug 7 21:51:38.628222691 (12h11m38.217567678s) [upload=12665]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:51:38.628226268 (12h11m38.217571255s) [upload=12665]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:51:38.628228662 (12h11m38.217573649s) [upload=12665]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:51:38.940125222 (12h11m38.529470269s) [upload=12665]: Done uploading segments... Aug 7 21:51:38.940147124 (12h11m38.529492191s) [upload=12665]: Upload finished. err: <nil> Aug 7 21:52:16.262879794 (12h12m15.852224781s) [upload=12676]: Starting upload Aug 7 21:52:16.262939168 (12h12m15.852284155s) [upload=12676]: Done starting upload Aug 7 21:52:16.263047876 (12h12m15.852392873s) [upload=12676]: Uploading segments... Aug 7 21:52:16.263139601 (12h12m15.852484598s) [upload=12676]: Got next segment. Inline: true Aug 7 21:52:16.263144440 (12h12m15.852489417s) [upload=12676]: Waiting for error group managing segments... Aug 7 21:52:16.263191931 (12h12m15.852536918s) [upload=12676]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:52:16.263197472 (12h12m15.852542449s) [upload=12676]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:52:16.263200056 (12h12m15.852545043s) [upload=12676]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:52:16.570506954 (12h12m16.159851931s) [upload=12676]: Done uploading segments... Aug 7 21:52:16.570524978 (12h12m16.159869965s) [upload=12676]: Upload finished. err: <nil> Aug 7 21:53:00.574165558 (12h13m0.163510545s) [upload=12691]: Starting upload Aug 7 21:53:00.574212286 (12h13m0.163557273s) [upload=12691]: Done starting upload Aug 7 21:53:00.574248476 (12h13m0.163593463s) [upload=12691]: Uploading segments... Aug 7 21:53:00.574327307 (12h13m0.163672294s) [upload=12691]: Got next segment. Inline: true Aug 7 21:53:00.574332868 (12h13m0.163677855s) [upload=12691]: Waiting for error group managing segments... Aug 7 21:53:00.574373505 (12h13m0.163718492s) [upload=12691]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:53:00.574390598 (12h13m0.163735585s) [upload=12691]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:53:00.574393373 (12h13m0.16373835s) [upload=12691]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:53:00.840335180 (12h13m0.429680167s) [upload=12691]: Done uploading segments... Aug 7 21:53:00.840351500 (12h13m0.429696487s) [upload=12691]: Upload finished. err: <nil> Aug 7 21:53:46.136340579 (12h13m45.725685566s) [upload=12705]: Starting upload Aug 7 21:53:46.136382029 (12h13m45.725727016s) [upload=12705]: Done starting upload Aug 7 21:53:46.136495205 (12h13m45.725840192s) [upload=12705]: Uploading segments... Aug 7 21:53:46.136529601 (12h13m45.725874588s) [upload=12705]: Got next segment. Inline: true Aug 7 21:53:46.136533228 (12h13m45.725878215s) [upload=12705]: Waiting for error group managing segments... Aug 7 21:53:46.136581701 (12h13m45.725926688s) [upload=12705]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:53:46.136585949 (12h13m45.725930936s) [upload=12705]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:53:46.136588393 (12h13m45.72593338s) [upload=12705]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:53:46.427021148 (12h13m46.016366155s) [upload=12705]: Done uploading segments... Aug 7 21:53:46.427045194 (12h13m46.016390181s) [upload=12705]: Upload finished. err: <nil> Aug 7 21:54:41.670591536 (12h14m41.259936523s) [upload=12721]: Starting upload Aug 7 21:54:41.670632916 (12h14m41.259977903s) [upload=12721]: Done starting upload Aug 7 21:54:41.670677612 (12h14m41.260022599s) [upload=12721]: Uploading segments... Aug 7 21:54:41.670708571 (12h14m41.260053558s) [upload=12721]: Got next segment. Inline: true Aug 7 21:54:41.670713510 (12h14m41.260058497s) [upload=12721]: Waiting for error group managing segments... Aug 7 21:54:41.670746032 (12h14m41.260091019s) [upload=12721]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:54:41.670749979 (12h14m41.260094966s) [upload=12721]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:54:41.670752424 (12h14m41.260097411s) [upload=12721]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:54:41.940759391 (12h14m41.530104368s) [upload=12721]: Done uploading segments... Aug 7 21:54:41.940776734 (12h14m41.530121721s) [upload=12721]: Upload finished. err: <nil> Duration: 1.86984588s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway) antiblip: 3 out of 60 is failed (threshold=3) [.X.X...X....................................................] duration: 773.211537ms remote: "tcp:45.38.24.2:443" target: https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410 FAIL (2025-08-07 22:11:59 UTC) orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": context deadline exceeded Duration: 773.238508ms edge gateway vhost us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 26 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXX..XX..X..............................X] duration: 36.251435182s FAIL (2025-08-07 22:11:55 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 36.251454829s core uplink us1
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway) antiblip: 7 out of 60 is failed (threshold=3) [......XX.XX..X.XX...........................................] duration: 235.031645ms remote: "tcp:207.211.208.129:443" target: https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222 FAIL (2025-08-07 22:11:41 UTC) orbiter warning: Get "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222": context deadline exceeded Duration: 235.058786ms edge gateway vhost bgp us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 4 out of 60 is failed (threshold=3) [.............X..X....................X....................X.] duration: 6.778041484s FAIL (2025-08-07 22:11:21 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/4M):: after 34e70bda2fe1d7206098dbfb59fc4576ab613f2b752f601da4a5b93473be374e != before ef9ba24509b0003982578e9daec1e6051f8d6ac0bef4f9769b005a8c713045cf ======LOGS====== ding segments... Aug 7 22:10:42.792858699 (12h30m42.382203686s) [upload=12997]: Got next segment. Inline: false Aug 7 22:10:42.792883667 (12h30m42.382228644s) [upload=12997]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:10:42.792887915 (12h30m42.382232902s) [upload=12997]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:10:43.239239407 (12h30m42.828584384s) [seg_pos={0 0}, upload=12997]: Begin upload segment... Aug 7 22:10:43.239300844 (12h30m42.828645831s) [seg_pos={0 0}, upload=12997]: Done begin upload segment. Aug 7 22:10:43.239329008 (12h30m42.828673995s) [noise=true , address=104.36.82.61:4394 , piece_id=BZ4OH34HU6COIEBD542MEZ5YGPXPLFMQKIO2EOV23UBTDTFK5XAA, seg_pos={0 0}, upload=12997]: Uploading piece... Aug 7 22:10:43.239541695 (12h30m42.828886692s) [noise=true , address=104.36.82.37:4687 , piece_id=I7ZESSKAS4ZUYX3CUI4ZRZWY2PLRATRNR6FGVGVAXQD6JT3DRUIA, seg_pos={0 0}, upload=12997]: Uploading piece... Aug 7 22:10:43.239923255 (12h30m42.829268242s) [noise=true , address=104.36.82.10:4087 , piece_id=PDZWS5ZZVKC5BCXJIY43NBQVXM7NCLKJZNSVTD3G2K6F2YEY4RVA, seg_pos={0 0}, upload=12997]: Uploading piece... Aug 7 22:10:43.240056509 (12h30m42.829401506s) [noise=true , address=45.91.116.6:4995 , piece_id=GYQIOSTEVRSY6VKX3GQCUC5YZY5S3ZV7R6X2LBG3QRTOOQD53NOA, seg_pos={0 0}, upload=12997]: Uploading piece... Aug 7 22:10:43.240324512 (12h30m42.829669499s) [noise=true , address=104.36.82.38:4705 , piece_id=M557M36RJRMMND4F33G7BOBYVMESQZZN7ZFESJ5CLHEKXWBCL5SA, seg_pos={0 0}, upload=12997]: Uploading piece... Aug 7 22:10:43.240397562 (12h30m42.829742549s) [noise=true , address=79.127.201.215:4351 , piece_id=367MVHSAIIGC6UZQY4EMH25DJCYYTAIUMPFF3PY426RZJHIO44DA, seg_pos={0 0}, upload=12997]: Uploading piece... Aug 7 22:10:43.240579942 (12h30m42.829924929s) [noise=true , address=104.36.82.47:4524 , piece_id=JDRSOKIHNV2RD3COKNJQHRYG2OXOPUBQOG3AHCTYOKKYC3LZISNQ, seg_pos={0 0}, upload=12997]: Uploading piece... Aug 7 22:10:43.240814029 (12h30m42.830159016s) [noise=true , address=64.49.13.169:4710 , piece_id=KOUJTCIUNCXFZSJWG6ZKR4KV42WV3ABECSARKOQ5YB5W4LAJTGAA, seg_pos={0 0}, upload=12997]: Uploading piece... Aug 7 22:10:43.240831132 (12h30m42.830176119s) [noise=true , address=104.36.82.59:4331 , piece_id=54OHTXRTR2J4S4HECSB7DFPHSC7T4UGBBC4X7WI72IVCMESI5BQQ, seg_pos={0 0}, upload=12997]: Uploading piece... Aug 7 22:10:43.241273939 (12h30m42.830618926s) [noise=true , address=104.36.82.44:4881 , piece_id=YWXSTJSJ46ERQLZ6A2E36CX3R2FI4XVVEZGUMDYJX4ZG3M3R5ACQ, seg_pos={0 0}, upload=12997]: Uploading piece... Aug 7 22:10:43.241589863 (12h30m42.83093485s) [noise=true , address=104.36.82.2:4750 , piece_id=24FHN36X2VIWLTYUDHSJNLSOHHEZ56K6DBIGSMI2ZCA26CPT7L7Q, seg_pos={0 0}, upload=12997]: Uploading piece... Aug 7 22:10:43.241938992 (12h30m42.831283979s) [noise=true , address=104.36.82.3:4791 , piece_id=DJ46BVH37DMTY3YD6VG2AEPNNTPYF7EH4IA3Y633UH3QO6XUTKPQ, seg_pos={0 0}, upload=12997]: Uploading piece... Aug 7 22:10:43.242014867 (12h30m42.831359854s) [noise=true , address=104.36.82.58:4270 , piece_id=HLYO6DZ75CNGFXJ6DA5SKQNVPD5COMDHFAETYUZTBOSKMPT4BUEQ, seg_pos={0 0}, upload=12997]: Uploading piece... Aug 7 22:10:43.242250207 (12h30m42.831595194s) [noise=true , address=104.36.82.44:4885 , piece_id=FJ726HA3PDRWAYDFGUEK2Y7D3B77ERCFPJSGWNKKPLGWXC3WK7GA, seg_pos={0 0}, upload=12997]: Uploading piece... Aug 7 22:10:43.242484636 (12h30m42.831829623s) [noise=true , address=79.127.205.246:4254 , piece_id=V3AT6KY72VDSIMMOZUC4JECVAQSPAHMAPC5CXDVCO46Z6RTPRUVA, seg_pos={0 0}, upload=12997]: Uploading piece... Aug 7 22:10:43.242691451 (12h30m42.832036438s) [noise=true , address=104.36.82.8:4995 , piece_id=BI77BF2ZGP3ZMB2ABMBUCKQCEF3Z6QHRC7IT6Q2ATIF6VON32WWQ, seg_pos={0 0}, upload=12997]: Uploading piece... Aug 7 22:10:43.242894410 (12h30m42.832239397s) [noise=true , address=104.36.82.43:4859 , piece_id=P2S2T5IIUTG4D6EQILFQ7OR5E5JOZHAYO563X7ED5UBN7THKW4QA, seg_pos={0 0}, upload=12997]: Uploading piece... Aug 7 22:10:43.243127415 (12h30m42.832472402s) [noise=true , address=104.36.82.46:4509 , piece_id=WQ3UWQPCMAZ7EDHTSURUQAKL6KWHUA3GSJI6U2G5ZUABRGCTFEXQ, seg_pos={0 0}, upload=12997]: Uploading piece... Aug 7 22:10:44.351806366 (12h30m43.941151353s) [upload=12997]: Next returned nil segment Aug 7 22:10:44.351819280 (12h30m43.941164257s) [upload=12997]: Waiting for error group managing segments... Aug 7 22:10:44.651888156 (12h30m44.241233144s) [noise=true , address=79.127.205.246:4254 , piece_id=V3AT6KY72VDSIMMOZUC4JECVAQSPAHMAPC5CXDVCO46Z6RTPRUVA, seg_pos={0 0}, upload=12997]: Done uploading piece. err: <nil> Aug 7 22:10:44.674573650 (12h30m44.263918637s) [noise=true , address=64.49.13.169:4710 , piece_id=KOUJTCIUNCXFZSJWG6ZKR4KV42WV3ABECSARKOQ5YB5W4LAJTGAA, seg_pos={0 0}, upload=12997]: Done uploading piece. err: <nil> Aug 7 22:10:45.186610734 (12h30m44.775955721s) [noise=true , address=104.36.82.38:4705 , piece_id=M557M36RJRMMND4F33G7BOBYVMESQZZN7ZFESJ5CLHEKXWBCL5SA, seg_pos={0 0}, upload=12997]: Done uploading piece. err: <nil> Aug 7 22:10:45.204653265 (12h30m44.793998252s) [noise=true , address=104.36.82.58:4270 , piece_id=HLYO6DZ75CNGFXJ6DA5SKQNVPD5COMDHFAETYUZTBOSKMPT4BUEQ, seg_pos={0 0}, upload=12997]: Done uploading piece. err: <nil> Aug 7 22:10:45.218003397 (12h30m44.807348374s) [noise=true , address=79.127.201.215:4351 , piece_id=367MVHSAIIGC6UZQY4EMH25DJCYYTAIUMPFF3PY426RZJHIO44DA, seg_pos={0 0}, upload=12997]: Done uploading piece. err: <nil> Aug 7 22:10:45.276599656 (12h30m44.865944643s) [noise=true , address=104.36.82.46:4509 , piece_id=WQ3UWQPCMAZ7EDHTSURUQAKL6KWHUA3GSJI6U2G5ZUABRGCTFEXQ, seg_pos={0 0}, upload=12997]: Done uploading piece. err: <nil> Aug 7 22:10:45.316254365 (12h30m44.905599352s) [noise=true , address=104.36.82.2:4750 , piece_id=24FHN36X2VIWLTYUDHSJNLSOHHEZ56K6DBIGSMI2ZCA26CPT7L7Q, seg_pos={0 0}, upload=12997]: Done uploading piece. err: <nil> Aug 7 22:10:45.349441480 (12h30m44.938786467s) [noise=true , address=104.36.82.3:4791 , piece_id=DJ46BVH37DMTY3YD6VG2AEPNNTPYF7EH4IA3Y633UH3QO6XUTKPQ, seg_pos={0 0}, upload=12997]: Done uploading piece. err: <nil> Aug 7 22:10:45.352466728 (12h30m44.941811715s) [noise=true , address=104.36.82.44:4881 , piece_id=YWXSTJSJ46ERQLZ6A2E36CX3R2FI4XVVEZGUMDYJX4ZG3M3R5ACQ, seg_pos={0 0}, upload=12997]: Done uploading piece. err: <nil> Aug 7 22:10:45.418760245 (12h30m45.008105232s) [noise=true , address=104.36.82.37:4687 , piece_id=I7ZESSKAS4ZUYX3CUI4ZRZWY2PLRATRNR6FGVGVAXQD6JT3DRUIA, seg_pos={0 0}, upload=12997]: Done uploading piece. err: <nil> Aug 7 22:10:45.540632266 (12h30m45.129977253s) [noise=true , address=104.36.82.10:4087 , piece_id=PDZWS5ZZVKC5BCXJIY43NBQVXM7NCLKJZNSVTD3G2K6F2YEY4RVA, seg_pos={0 0}, upload=12997]: Done uploading piece. err: <nil> Aug 7 22:10:45.567016576 (12h30m45.156361563s) [noise=true , address=104.36.82.59:4331 , piece_id=54OHTXRTR2J4S4HECSB7DFPHSC7T4UGBBC4X7WI72IVCMESI5BQQ, seg_pos={0 0}, upload=12997]: Done uploading piece. err: <nil> Aug 7 22:10:45.597615539 (12h30m45.186960526s) [noise=true , address=104.36.82.44:4885 , piece_id=FJ726HA3PDRWAYDFGUEK2Y7D3B77ERCFPJSGWNKKPLGWXC3WK7GA, seg_pos={0 0}, upload=12997]: Done uploading piece. err: <nil> Aug 7 22:10:45.676238976 (12h30m45.265584013s) [noise=true , address=45.91.116.6:4995 , piece_id=GYQIOSTEVRSY6VKX3GQCUC5YZY5S3ZV7R6X2LBG3QRTOOQD53NOA, seg_pos={0 0}, upload=12997]: Done uploading piece. err: <nil> Aug 7 22:10:45.676277709 (12h30m45.265622696s) [seg_pos={0 0}, upload=12997]: Segment reached optimal threshold of 14 pieces. Aug 7 22:10:45.676633781 (12h30m45.265978758s) [noise=true , address=104.36.82.8:4995 , piece_id=BI77BF2ZGP3ZMB2ABMBUCKQCEF3Z6QHRC7IT6Q2ATIF6VON32WWQ, seg_pos={0 0}, upload=12997]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tvoTD1LoSj3q2rLiDckfT56dXNKKwdVu1ZAsBZExNFVTom9tH): protocol: expected piece hash; context canceled; EOF Aug 7 22:10:45.676841539 (12h30m45.266186526s) [noise=true , address=104.36.82.43:4859 , piece_id=P2S2T5IIUTG4D6EQILFQ7OR5E5JOZHAYO563X7ED5UBN7THKW4QA, seg_pos={0 0}, upload=12997]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:126SqCtRGPziEeopRjA4nNkjkCZWvUwkt5XaMgcqZKJwoxuYWY7): protocol: rpc: dial tcp 104.36.82.43:4859: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 22:10:45.676869993 (12h30m45.26621498s) [noise=true , address=104.36.82.47:4524 , piece_id=JDRSOKIHNV2RD3COKNJQHRYG2OXOPUBQOG3AHCTYOKKYC3LZISNQ, seg_pos={0 0}, upload=12997]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xfbCzjSwFDRrR1gPggeN8tNCLbRRWDVzCJ6mqhZsbVSe8d4s4): protocol: read tcp 188.166.225.170:58322->104.36.82.47:4524: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 22:10:45.676908757 (12h30m45.266253744s) [noise=true , address=104.36.82.61:4394 , piece_id=BZ4OH34HU6COIEBD542MEZ5YGPXPLFMQKIO2EOV23UBTDTFK5XAA, seg_pos={0 0}, upload=12997]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1dSpTqCxeAcL9PVYbP7BhmuKE7papFPc9swB1Fy83WcFDgQbcF): protocol: rpc: dial tcp 104.36.82.61:4394: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 22:10:45.676927984 (12h30m45.266272961s) [seg_pos={0 0}, upload=12997]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 22:10:45.677012455 (12h30m45.266357443s) [upload=12997]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 22:10:45.677016543 (12h30m45.26636152s) [upload=12997]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:10:45.998268658 (12h30m45.587613635s) [upload=12997]: Done uploading segments... Aug 7 22:10:45.998285902 (12h30m45.587630889s) [upload=12997]: Upload finished. err: <nil> Duration: 6.778068466s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 25 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXX...X................................X] duration: 30.000478862s FAIL (2025-08-07 22:11:06 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 30.00050399s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 23 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXX.....X................................] duration: 6.334941245s FAIL (2025-08-07 22:10:59 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== t segment. Inline: false Aug 7 21:44:25.061589945 (12h4m24.650934922s) [upload=12552]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:44:25.061594724 (12h4m24.650939711s) [upload=12552]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:44:50.602885060 (12h4m50.192230047s) [seg_pos={0 0}, upload=12552]: Begin upload segment... Aug 7 21:44:50.602935446 (12h4m50.192280423s) [seg_pos={0 0}, upload=12552]: Done begin upload segment. Aug 7 21:44:50.603112565 (12h4m50.192457552s) [noise=true , address=79.127.205.233:4624 , piece_id=WEAYHGF7VE6W5ML3QAE3AITMEJGQFH4JQZ7RQKR7O4NKTPR6RZRA, seg_pos={0 0}, upload=12552]: Uploading piece... Aug 7 21:44:50.603695610 (12h4m50.193040597s) [noise=true , address=104.36.82.22:4529 , piece_id=ZTTBM5H6MHVMQ3OFVTVISN6WJ7WB3XBWI7YZKUXCFNQ6NO3D4CGA, seg_pos={0 0}, upload=12552]: Uploading piece... Aug 7 21:44:50.604300768 (12h4m50.193645755s) [noise=true , address=45.91.116.6:4002 , piece_id=AE5EL7NWD6FEQ7NXX2JRHB4MVZQDXRGQU3EER2X7COISFAX6P3XQ, seg_pos={0 0}, upload=12552]: Uploading piece... Aug 7 21:44:50.604588519 (12h4m50.193933496s) [noise=true , address=104.36.82.56:4249 , piece_id=RD4B7IDIJV25L7EJNSYCZGW5QKJEM435E2N3UFUD44Q4S3DCBOXQ, seg_pos={0 0}, upload=12552]: Uploading piece... Aug 7 21:44:50.605153860 (12h4m50.194498837s) [noise=true , address=77.90.31.11:4562 , piece_id=XC6F2WLSXKQB7E7MBRNZBJKUSUFL6XNA5TVX5M3ZJI3ZZMK7UBLA, seg_pos={0 0}, upload=12552]: Uploading piece... Aug 7 21:44:50.605455377 (12h4m50.194800364s) [noise=true , address=104.36.82.23:4466 , piece_id=5J5X2LSP6GCX2PNRIDZYFJPISWSX3YSRID5M2WU6Y5Z6LWRHLC4Q, seg_pos={0 0}, upload=12552]: Uploading piece... Aug 7 21:44:50.605743810 (12h4m50.195088797s) [noise=true , address=104.36.82.62:4420 , piece_id=UOO36PYVE5WFRRZLGYLJ6I6SV4EO7W2ZH2TIXJ2OIZZMY3LTHR2A, seg_pos={0 0}, upload=12552]: Uploading piece... Aug 7 21:44:50.606547457 (12h4m50.195892444s) [noise=true , address=104.36.82.5:4842 , piece_id=E5DSJXN63RFX7EQ4ZTBMUESFW77EB3HQNA5KCQGRVRAA5XO2GN3Q, seg_pos={0 0}, upload=12552]: Uploading piece... Aug 7 21:44:50.606982099 (12h4m50.196327086s) [noise=true , address=104.36.82.59:4301 , piece_id=VVR5CQWCPNPZUFEFCPB6ZL2KASWS5XO65FVDI474RKRZSJFDUQTA, seg_pos={0 0}, upload=12552]: Uploading piece... Aug 7 21:44:50.607295849 (12h4m50.196640836s) [noise=true , address=45.91.116.14:4198 , piece_id=2HRU3JZDQSNGD5NJZDLZSRG2WIO2Y2QBSOAY6KUEGUKDBWM53RSA, seg_pos={0 0}, upload=12552]: Uploading piece... Aug 7 21:44:50.607695593 (12h4m50.19704058s) [noise=true , address=104.36.82.6:4881 , piece_id=VUC33D7TLVFICDLA3334DEKBJE5E5P4HQTJU3VDQNMVYWVESSJWA, seg_pos={0 0}, upload=12552]: Uploading piece... Aug 7 21:44:50.608364103 (12h4m50.19770909s) [noise=true , address=104.36.82.2:4757 , piece_id=H6KNYP5HASPWQIHADTMEZDC5AE5HITCAY73XGNWVEXJOPDEZ3SHA, seg_pos={0 0}, upload=12552]: Uploading piece... Aug 7 21:44:50.608764288 (12h4m50.198109265s) [noise=true , address=104.36.82.46:4493 , piece_id=PS4S4OZ6HSITEJFPJDIBWQ5RGWTDRKMX3NCEWXCLC2PSP5Q2KPZQ, seg_pos={0 0}, upload=12552]: Uploading piece... Aug 7 21:44:50.609065284 (12h4m50.198410271s) [noise=true , address=104.36.82.47:4518 , piece_id=KJ3PJCT3EDH3WZBT7AWWKI6ABGVVAMKZTVJY7Q76FULC72TUWA6Q, seg_pos={0 0}, upload=12552]: Uploading piece... Aug 7 21:44:50.609312777 (12h4m50.198657755s) [noise=true , address=104.36.82.60:4349 , piece_id=GTTF32YMJWBY2GZEFZPIBZXZWNI7MBLUYCTQ3SRFG735PLWXTXBQ, seg_pos={0 0}, upload=12552]: Uploading piece... Aug 7 21:44:50.609589888 (12h4m50.198934875s) [noise=true , address=104.36.82.52:4116 , piece_id=7QA2Q5ACN43TQUOKWA34EZFSSQBE3OWNJTYODHHQ5GCVOA4QZOJA, seg_pos={0 0}, upload=12552]: Uploading piece... Aug 7 21:44:50.610295969 (12h4m50.199641056s) [noise=true , address=104.36.82.37:4678 , piece_id=VJO5JO3SJEXBQZC6PJ34DRKWEQKOPI33ZF4IHNNOOTSD7IBRR4PQ, seg_pos={0 0}, upload=12552]: Uploading piece... Aug 7 21:44:50.610811305 (12h4m50.200156302s) [noise=true , address=64.49.13.169:4714 , piece_id=A6O3H7QLE6KQTHWUKOEROVHPEHVLG22KNF2KCV3HK2KCLXP7SYVA, seg_pos={0 0}, upload=12552]: Uploading piece... Aug 7 21:44:51.715421583 (12h4m51.30476657s) [upload=12552]: Next returned nil segment Aug 7 21:44:51.715438125 (12h4m51.304783112s) [upload=12552]: Waiting for error group managing segments... Aug 7 21:44:51.998131281 (12h4m51.587476268s) [noise=true , address=64.49.13.169:4714 , piece_id=A6O3H7QLE6KQTHWUKOEROVHPEHVLG22KNF2KCV3HK2KCLXP7SYVA, seg_pos={0 0}, upload=12552]: Done uploading piece. err: <nil> Aug 7 21:44:51.999236335 (12h4m51.588581312s) [noise=true , address=79.127.205.233:4624 , piece_id=WEAYHGF7VE6W5ML3QAE3AITMEJGQFH4JQZ7RQKR7O4NKTPR6RZRA, seg_pos={0 0}, upload=12552]: Done uploading piece. err: <nil> Aug 7 21:44:52.638299991 (12h4m52.227644978s) [noise=true , address=45.91.116.6:4002 , piece_id=AE5EL7NWD6FEQ7NXX2JRHB4MVZQDXRGQU3EER2X7COISFAX6P3XQ, seg_pos={0 0}, upload=12552]: Done uploading piece. err: <nil> Aug 7 21:44:52.639927565 (12h4m52.229272542s) [noise=true , address=104.36.82.6:4881 , piece_id=VUC33D7TLVFICDLA3334DEKBJE5E5P4HQTJU3VDQNMVYWVESSJWA, seg_pos={0 0}, upload=12552]: Done uploading piece. err: <nil> Aug 7 21:44:52.643220585 (12h4m52.232565572s) [noise=true , address=104.36.82.2:4757 , piece_id=H6KNYP5HASPWQIHADTMEZDC5AE5HITCAY73XGNWVEXJOPDEZ3SHA, seg_pos={0 0}, upload=12552]: Done uploading piece. err: <nil> Aug 7 21:44:52.711872926 (12h4m52.301217913s) [noise=true , address=104.36.82.56:4249 , piece_id=RD4B7IDIJV25L7EJNSYCZGW5QKJEM435E2N3UFUD44Q4S3DCBOXQ, seg_pos={0 0}, upload=12552]: Done uploading piece. err: <nil> Aug 7 21:44:52.763601896 (12h4m52.352946883s) [noise=true , address=104.36.82.22:4529 , piece_id=ZTTBM5H6MHVMQ3OFVTVISN6WJ7WB3XBWI7YZKUXCFNQ6NO3D4CGA, seg_pos={0 0}, upload=12552]: Done uploading piece. err: <nil> Aug 7 21:44:52.768238797 (12h4m52.357583785s) [noise=true , address=104.36.82.60:4349 , piece_id=GTTF32YMJWBY2GZEFZPIBZXZWNI7MBLUYCTQ3SRFG735PLWXTXBQ, seg_pos={0 0}, upload=12552]: Done uploading piece. err: <nil> Aug 7 21:44:52.798543408 (12h4m52.387888395s) [noise=true , address=45.91.116.14:4198 , piece_id=2HRU3JZDQSNGD5NJZDLZSRG2WIO2Y2QBSOAY6KUEGUKDBWM53RSA, seg_pos={0 0}, upload=12552]: Done uploading piece. err: <nil> Aug 7 21:44:52.861192704 (12h4m52.450537691s) [noise=true , address=77.90.31.11:4562 , piece_id=XC6F2WLSXKQB7E7MBRNZBJKUSUFL6XNA5TVX5M3ZJI3ZZMK7UBLA, seg_pos={0 0}, upload=12552]: Done uploading piece. err: <nil> Aug 7 21:44:53.070218164 (12h4m52.659563151s) [noise=true , address=104.36.82.59:4301 , piece_id=VVR5CQWCPNPZUFEFCPB6ZL2KASWS5XO65FVDI474RKRZSJFDUQTA, seg_pos={0 0}, upload=12552]: Done uploading piece. err: <nil> Aug 7 21:44:53.558930346 (12h4m53.148275333s) [noise=true , address=104.36.82.46:4493 , piece_id=PS4S4OZ6HSITEJFPJDIBWQ5RGWTDRKMX3NCEWXCLC2PSP5Q2KPZQ, seg_pos={0 0}, upload=12552]: Done uploading piece. err: <nil> Aug 7 21:44:55.132342777 (12h4m54.721687765s) [noise=true , address=104.36.82.37:4678 , piece_id=VJO5JO3SJEXBQZC6PJ34DRKWEQKOPI33ZF4IHNNOOTSD7IBRR4PQ, seg_pos={0 0}, upload=12552]: Done uploading piece. err: <nil> Aug 7 21:44:55.265236558 (12h4m54.854581545s) [noise=true , address=104.36.82.52:4116 , piece_id=7QA2Q5ACN43TQUOKWA34EZFSSQBE3OWNJTYODHHQ5GCVOA4QZOJA, seg_pos={0 0}, upload=12552]: Done uploading piece. err: <nil> Aug 7 21:44:55.265254161 (12h4m54.854599138s) [seg_pos={0 0}, upload=12552]: Segment reached optimal threshold of 14 pieces. Aug 7 21:44:55.265482308 (12h4m54.854827295s) [noise=true , address=104.36.82.5:4842 , piece_id=E5DSJXN63RFX7EQ4ZTBMUESFW77EB3HQNA5KCQGRVRAA5XO2GN3Q, seg_pos={0 0}, upload=12552]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12q4VdT5BcjUb16PBYHK6zrnW8dqENKrcGx9zdLtNefDVB1MUSn): protocol: expected piece hash; write tcp 188.166.225.170:57020->104.36.82.5:4842: use of closed network connection; write tcp 188.166.225.170:57020->104.36.82.5:4842: use of closed network connection; EOF Aug 7 21:44:55.265551430 (12h4m54.854896417s) [noise=true , address=104.36.82.47:4518 , piece_id=KJ3PJCT3EDH3WZBT7AWWKI6ABGVVAMKZTVJY7Q76FULC72TUWA6Q, seg_pos={0 0}, upload=12552]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17TZ9JLdCivFF46guhSTW6hcoUmTz5uo5wvLkDqC4BGmpm9eLb): protocol: expected piece hash; context canceled; EOF Aug 7 21:44:55.265766391 (12h4m54.855111378s) [noise=true , address=104.36.82.23:4466 , piece_id=5J5X2LSP6GCX2PNRIDZYFJPISWSX3YSRID5M2WU6Y5Z6LWRHLC4Q, seg_pos={0 0}, upload=12552]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rkNnk9RUhKBJ3EsakqMRb3wYwrhRLhB4ZYbrdfKPDk8erRJpP): protocol: write tcp 188.166.225.170:49724->104.36.82.23:4466: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 21:44:55.265824172 (12h4m54.855169159s) [noise=true , address=104.36.82.62:4420 , piece_id=UOO36PYVE5WFRRZLGYLJ6I6SV4EO7W2ZH2TIXJ2OIZZMY3LTHR2A, seg_pos={0 0}, upload=12552]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SWVQwL5PxDLaSXADwTrAYS2hVcNBuYAA6ReWuRM3UZa57QyYi): protocol: read tcp 188.166.225.170:33092->104.36.82.62:4420: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 21:44:55.265833430 (12h4m54.855178417s) [seg_pos={0 0}, upload=12552]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 21:44:55.265888796 (12h4m54.855233783s) [upload=12552]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 21:44:55.265892303 (12h4m54.85523728s) [upload=12552]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:48:46.088275738 (12h8m45.677620725s) [upload=12552]: Done uploading segments... Aug 7 21:48:46.088301307 (12h8m45.677646284s) [upload=12552]: Upload finished. err: metaclient: context deadline exceeded Duration: 6.334967474s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 3 out of 60 is failed (threshold=3) [.......X..........X.................X.......................] duration: 1.856965538s FAIL (2025-08-07 22:10:59 UTC) orbiter warning: operation: ListObjects ======ERROR====== unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/1K ======LOGS====== 8m27.919686751s) [upload=12612]: Done uploading segments... Aug 7 21:48:28.330358527 (12h8m27.919703514s) [upload=12612]: Upload finished. err: <nil> Aug 7 21:48:54.943937458 (12h8m54.533282445s) [upload=12618]: Starting upload Aug 7 21:48:54.944002622 (12h8m54.533347609s) [upload=12618]: Done starting upload Aug 7 21:48:54.944106100 (12h8m54.533451087s) [upload=12618]: Uploading segments... Aug 7 21:48:54.944196242 (12h8m54.53354132s) [upload=12618]: Got next segment. Inline: true Aug 7 21:48:54.944206543 (12h8m54.53355158s) [upload=12618]: Waiting for error group managing segments... Aug 7 21:48:54.944254484 (12h8m54.533599461s) [upload=12618]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:48:54.944259223 (12h8m54.5336042s) [upload=12618]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:48:54.944262139 (12h8m54.533607126s) [upload=12618]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:48:55.330753187 (12h8m54.920098174s) [upload=12618]: Done uploading segments... Aug 7 21:48:55.330774959 (12h8m54.920119936s) [upload=12618]: Upload finished. err: <nil> Aug 7 21:49:36.650951942 (12h9m36.240296929s) [upload=12631]: Starting upload Aug 7 21:49:36.650993252 (12h9m36.240338239s) [upload=12631]: Done starting upload Aug 7 21:49:36.651049029 (12h9m36.240394016s) [upload=12631]: Uploading segments... Aug 7 21:49:36.651075309 (12h9m36.240420286s) [upload=12631]: Got next segment. Inline: true Aug 7 21:49:36.651078775 (12h9m36.240423763s) [upload=12631]: Waiting for error group managing segments... Aug 7 21:49:36.651115776 (12h9m36.240460773s) [upload=12631]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:49:36.651120254 (12h9m36.240465251s) [upload=12631]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:49:36.651124473 (12h9m36.24046947s) [upload=12631]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:49:36.920683881 (12h9m36.510028868s) [upload=12631]: Done uploading segments... Aug 7 21:49:36.920707246 (12h9m36.510052233s) [upload=12631]: Upload finished. err: <nil> Aug 7 21:50:03.363429937 (12h10m2.952774924s) [upload=12636]: Starting upload Aug 7 21:50:03.363493408 (12h10m2.952838405s) [upload=12636]: Done starting upload Aug 7 21:50:03.363582518 (12h10m2.952927515s) [upload=12636]: Uploading segments... Aug 7 21:50:03.363692078 (12h10m2.953037085s) [upload=12636]: Got next segment. Inline: true Aug 7 21:50:03.363699752 (12h10m2.953044739s) [upload=12636]: Waiting for error group managing segments... Aug 7 21:50:03.363764116 (12h10m2.953109103s) [upload=12636]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:50:03.363773323 (12h10m2.95311831s) [upload=12636]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:50:03.363777271 (12h10m2.953122258s) [upload=12636]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:50:03.636022787 (12h10m3.225367774s) [upload=12636]: Done uploading segments... Aug 7 21:50:03.636042204 (12h10m3.225387202s) [upload=12636]: Upload finished. err: <nil> Aug 7 21:50:20.944550175 (12h10m20.533895152s) [upload=12642]: Starting upload Aug 7 21:50:20.944594911 (12h10m20.533939898s) [upload=12642]: Done starting upload Aug 7 21:50:20.944644405 (12h10m20.533989392s) [upload=12642]: Uploading segments... Aug 7 21:50:20.944674473 (12h10m20.534019461s) [upload=12642]: Got next segment. Inline: true Aug 7 21:50:20.944677820 (12h10m20.534022807s) [upload=12642]: Waiting for error group managing segments... Aug 7 21:50:20.944709450 (12h10m20.534054437s) [upload=12642]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:50:20.944715131 (12h10m20.534060118s) [upload=12642]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:50:20.944717505 (12h10m20.534062493s) [upload=12642]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:50:21.212077476 (12h10m20.801422473s) [upload=12642]: Done uploading segments... Aug 7 21:50:21.212101602 (12h10m20.801446599s) [upload=12642]: Upload finished. err: <nil> Aug 7 21:51:22.649621444 (12h11m22.238966431s) [upload=12660]: Starting upload Aug 7 21:51:22.649696837 (12h11m22.239041824s) [upload=12660]: Done starting upload Aug 7 21:51:22.649777742 (12h11m22.239122729s) [upload=12660]: Uploading segments... Aug 7 21:51:22.649867044 (12h11m22.239212041s) [upload=12660]: Got next segment. Inline: true Aug 7 21:51:22.649872053 (12h11m22.23921704s) [upload=12660]: Waiting for error group managing segments... Aug 7 21:51:22.649940053 (12h11m22.23928504s) [upload=12660]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:51:22.649946785 (12h11m22.239291772s) [upload=12660]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:51:22.649949932 (12h11m22.239294919s) [upload=12660]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:51:22.914527578 (12h11m22.503872565s) [upload=12660]: Done uploading segments... Aug 7 21:51:22.914549130 (12h11m22.503894117s) [upload=12660]: Upload finished. err: <nil> Aug 7 21:51:38.628060070 (12h11m38.217405057s) [upload=12665]: Starting upload Aug 7 21:51:38.628100548 (12h11m38.217445535s) [upload=12665]: Done starting upload Aug 7 21:51:38.628155373 (12h11m38.21750036s) [upload=12665]: Uploading segments... Aug 7 21:51:38.628187814 (12h11m38.217532802s) [upload=12665]: Got next segment. Inline: true Aug 7 21:51:38.628191381 (12h11m38.217536368s) [upload=12665]: Waiting for error group managing segments... Aug 7 21:51:38.628222691 (12h11m38.217567678s) [upload=12665]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:51:38.628226268 (12h11m38.217571255s) [upload=12665]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:51:38.628228662 (12h11m38.217573649s) [upload=12665]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:51:38.940125222 (12h11m38.529470269s) [upload=12665]: Done uploading segments... Aug 7 21:51:38.940147124 (12h11m38.529492191s) [upload=12665]: Upload finished. err: <nil> Aug 7 21:52:16.262879794 (12h12m15.852224781s) [upload=12676]: Starting upload Aug 7 21:52:16.262939168 (12h12m15.852284155s) [upload=12676]: Done starting upload Aug 7 21:52:16.263047876 (12h12m15.852392873s) [upload=12676]: Uploading segments... Aug 7 21:52:16.263139601 (12h12m15.852484598s) [upload=12676]: Got next segment. Inline: true Aug 7 21:52:16.263144440 (12h12m15.852489417s) [upload=12676]: Waiting for error group managing segments... Aug 7 21:52:16.263191931 (12h12m15.852536918s) [upload=12676]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:52:16.263197472 (12h12m15.852542449s) [upload=12676]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:52:16.263200056 (12h12m15.852545043s) [upload=12676]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:52:16.570506954 (12h12m16.159851931s) [upload=12676]: Done uploading segments... Aug 7 21:52:16.570524978 (12h12m16.159869965s) [upload=12676]: Upload finished. err: <nil> Aug 7 21:53:00.574165558 (12h13m0.163510545s) [upload=12691]: Starting upload Aug 7 21:53:00.574212286 (12h13m0.163557273s) [upload=12691]: Done starting upload Aug 7 21:53:00.574248476 (12h13m0.163593463s) [upload=12691]: Uploading segments... Aug 7 21:53:00.574327307 (12h13m0.163672294s) [upload=12691]: Got next segment. Inline: true Aug 7 21:53:00.574332868 (12h13m0.163677855s) [upload=12691]: Waiting for error group managing segments... Aug 7 21:53:00.574373505 (12h13m0.163718492s) [upload=12691]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:53:00.574390598 (12h13m0.163735585s) [upload=12691]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:53:00.574393373 (12h13m0.16373835s) [upload=12691]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:53:00.840335180 (12h13m0.429680167s) [upload=12691]: Done uploading segments... Aug 7 21:53:00.840351500 (12h13m0.429696487s) [upload=12691]: Upload finished. err: <nil> Aug 7 21:53:46.136340579 (12h13m45.725685566s) [upload=12705]: Starting upload Aug 7 21:53:46.136382029 (12h13m45.725727016s) [upload=12705]: Done starting upload Aug 7 21:53:46.136495205 (12h13m45.725840192s) [upload=12705]: Uploading segments... Aug 7 21:53:46.136529601 (12h13m45.725874588s) [upload=12705]: Got next segment. Inline: true Aug 7 21:53:46.136533228 (12h13m45.725878215s) [upload=12705]: Waiting for error group managing segments... Aug 7 21:53:46.136581701 (12h13m45.725926688s) [upload=12705]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:53:46.136585949 (12h13m45.725930936s) [upload=12705]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:53:46.136588393 (12h13m45.72593338s) [upload=12705]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:53:46.427021148 (12h13m46.016366155s) [upload=12705]: Done uploading segments... Aug 7 21:53:46.427045194 (12h13m46.016390181s) [upload=12705]: Upload finished. err: <nil> Aug 7 21:54:41.670591536 (12h14m41.259936523s) [upload=12721]: Starting upload Aug 7 21:54:41.670632916 (12h14m41.259977903s) [upload=12721]: Done starting upload Aug 7 21:54:41.670677612 (12h14m41.260022599s) [upload=12721]: Uploading segments... Aug 7 21:54:41.670708571 (12h14m41.260053558s) [upload=12721]: Got next segment. Inline: true Aug 7 21:54:41.670713510 (12h14m41.260058497s) [upload=12721]: Waiting for error group managing segments... Aug 7 21:54:41.670746032 (12h14m41.260091019s) [upload=12721]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:54:41.670749979 (12h14m41.260094966s) [upload=12721]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:54:41.670752424 (12h14m41.260097411s) [upload=12721]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:54:41.940759391 (12h14m41.530104368s) [upload=12721]: Done uploading segments... Aug 7 21:54:41.940776734 (12h14m41.530121721s) [upload=12721]: Upload finished. err: <nil> Duration: 1.857005294s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 25 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXX.X.X.X................................] duration: 6.023659344s FAIL (2025-08-07 22:10:58 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== context canceled; ecclient: upload cut due to slow connection (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR): protocol: write tcp 188.166.225.170:46242->104.36.82.11:4115: use of closed network connection; write tcp 188.166.225.170:46242->104.36.82.11:4115: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46242->104.36.82.11:4115: use of closed network connection Aug 7 21:44:51.028127350 (12h4m50.617472327s) [seg_pos={0 0}, upload=12555]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 7 21:44:51.028226048 (12h4m50.617571035s) [upload=12555]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 21:44:51.028237751 (12h4m50.617582738s) [upload=12555]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:44:51.322864709 (12h4m50.912209697s) [upload=12555]: Done uploading segments... Aug 7 21:44:51.322882523 (12h4m50.9122275s) [upload=12555]: Upload finished. err: <nil> Aug 7 21:45:58.815791269 (12h5m58.405136256s) [upload=12571]: Starting upload Aug 7 21:45:58.815846996 (12h5m58.405191983s) [upload=12571]: Done starting upload Aug 7 21:45:58.815953540 (12h5m58.405298527s) [upload=12571]: Uploading segments... Aug 7 21:45:58.816065333 (12h5m58.405410321s) [upload=12571]: Got next segment. Inline: false Aug 7 21:45:58.816100661 (12h5m58.405445638s) [upload=12571]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:45:58.816106212 (12h5m58.405451199s) [upload=12571]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:45:59.064845605 (12h5m58.654190592s) [seg_pos={0 0}, upload=12571]: Begin upload segment... Aug 7 21:45:59.064885521 (12h5m58.654230508s) [seg_pos={0 0}, upload=12571]: Done begin upload segment. Aug 7 21:45:59.064910399 (12h5m58.654255376s) [noise=true , address=77.90.31.18:4807 , piece_id=63E5KXGYEJHGHYE4SKTNTMED5R4HSWIST4CR4EV6ZGON3YHV73AA, seg_pos={0 0}, upload=12571]: Uploading piece... Aug 7 21:45:59.064952560 (12h5m58.654297547s) [noise=true , address=45.91.116.12:4227 , piece_id=MDQGJ65RT4QEUFX6OFXPEAOFSTG2VXBRC23DXVVD2IV6V52UFF4Q, seg_pos={0 0}, upload=12571]: Uploading piece... Aug 7 21:45:59.065245920 (12h5m58.654590907s) [noise=true , address=45.91.116.8:4048 , piece_id=2LZRN76O3QS6FODVI52DIIMK7Z77QQBGIXYUAKZCSFI7WMGRTDUA, seg_pos={0 0}, upload=12571]: Uploading piece... Aug 7 21:45:59.065360070 (12h5m58.654705057s) [noise=true , address=45.91.116.9:4262 , piece_id=D4ABUUAX4MHYLFPR6LN2REDC6VRT2W7D4VZHHYM7GZ6HWOWMG2LA, seg_pos={0 0}, upload=12571]: Uploading piece... Aug 7 21:45:59.065517019 (12h5m58.654861996s) [noise=true , address=77.90.31.19:4843 , piece_id=S73TOGLK5HDCBE4BINZ72A3URDR6TUD43IVIFDY5V37EVCN6222A, seg_pos={0 0}, upload=12571]: Uploading piece... Aug 7 21:45:59.065615889 (12h5m58.654960876s) [noise=true , address=104.36.82.6:4875 , piece_id=W43EX62262JONZAGLUREOG3EGMICRBIVX5O445J3OQDQ2O475J7A, seg_pos={0 0}, upload=12571]: Uploading piece... Aug 7 21:45:59.065778009 (12h5m58.655122986s) [noise=true , address=104.36.82.11:4115 , piece_id=AH7EPSWFJFX7XDYUQOFCNXEHKQ5AMQX22NY6OFBMMHATCOVH4YJA, seg_pos={0 0}, upload=12571]: Uploading piece... Aug 7 21:45:59.065884293 (12h5m58.65522928s) [noise=true , address=45.91.116.8:4047 , piece_id=TEMT2SH64IMIPQ3GDE5XCF2R2XLY33HUKRFF6KUWEYH7CDVYD4YQ, seg_pos={0 0}, upload=12571]: Uploading piece... Aug 7 21:45:59.066024761 (12h5m58.655369748s) [noise=true , address=77.90.31.11:4554 , piece_id=LMSB62TMAJONFULH4DAOTPWQMGLUFRZYKVG4HNJ4AUBP7EBP5FFQ, seg_pos={0 0}, upload=12571]: Uploading piece... Aug 7 21:45:59.066243109 (12h5m58.655588096s) [noise=true , address=45.91.116.6:4973 , piece_id=7MZSPAEXC3Q7W7U3QMZUQRUI3OHPQI3XRAJW4OGU7UZ73FEI4LIQ, seg_pos={0 0}, upload=12571]: Uploading piece... Aug 7 21:45:59.066452850 (12h5m58.655797837s) [noise=true , address=104.36.82.6:4874 , piece_id=T3W6RCAW4YPEU42CESOSUSFSNPDBFXR74WZUYQ2AHJNBZV776MVQ, seg_pos={0 0}, upload=12571]: Uploading piece... Aug 7 21:45:59.066669685 (12h5m58.656014672s) [noise=true , address=45.91.116.6:4975 , piece_id=CI7SM56L5KPNQEV52PLIETEBQVH5CFXTKVQOQIZ73YOPEGYC3KOA, seg_pos={0 0}, upload=12571]: Uploading piece... Aug 7 21:45:59.066955041 (12h5m58.656300028s) [noise=true , address=104.36.82.7:4909 , piece_id=EKLBHF6P5BXX5VLSLL4QEOJXQXWM7MCOY67W5OON2UHRNMKOQFAA, seg_pos={0 0}, upload=12571]: Uploading piece... Aug 7 21:45:59.067149393 (12h5m58.65649438s) [noise=true , address=45.91.116.13:4154 , piece_id=LDSWQ62RRGHTT544S6463WLDKW3WFBWIFQGQDHLYWJJD2ESMYJNQ, seg_pos={0 0}, upload=12571]: Uploading piece... Aug 7 21:45:59.067334066 (12h5m58.656679053s) [noise=true , address=104.36.82.16:4294 , piece_id=KDCYWDEXSTKCHR23CAOKKTYUMRQS5Y4LH6KIJPQFF737WA5TST7Q, seg_pos={0 0}, upload=12571]: Uploading piece... Aug 7 21:45:59.067527215 (12h5m58.656872202s) [noise=true , address=77.90.31.15:4698 , piece_id=6L3IAVPA4PQGSMKOGSKWYG4NLM6TSA2YSIAM6S6MYSDUW26CP5CA, seg_pos={0 0}, upload=12571]: Uploading piece... Aug 7 21:46:00.855806043 (12h6m0.44515103s) [upload=12571]: Next returned nil segment Aug 7 21:46:00.855821061 (12h6m0.445166048s) [upload=12571]: Waiting for error group managing segments... Aug 7 21:46:01.180369497 (12h6m0.769714484s) [noise=true , address=77.90.31.15:4698 , piece_id=6L3IAVPA4PQGSMKOGSKWYG4NLM6TSA2YSIAM6S6MYSDUW26CP5CA, seg_pos={0 0}, upload=12571]: Done uploading piece. err: <nil> Aug 7 21:46:01.200492297 (12h6m0.789837284s) [noise=true , address=45.91.116.6:4975 , piece_id=CI7SM56L5KPNQEV52PLIETEBQVH5CFXTKVQOQIZ73YOPEGYC3KOA, seg_pos={0 0}, upload=12571]: Done uploading piece. err: <nil> Aug 7 21:46:01.210262691 (12h6m0.799607678s) [noise=true , address=104.36.82.16:4294 , piece_id=KDCYWDEXSTKCHR23CAOKKTYUMRQS5Y4LH6KIJPQFF737WA5TST7Q, seg_pos={0 0}, upload=12571]: Done uploading piece. err: <nil> Aug 7 21:46:01.220625478 (12h6m0.809970465s) [noise=true , address=104.36.82.7:4909 , piece_id=EKLBHF6P5BXX5VLSLL4QEOJXQXWM7MCOY67W5OON2UHRNMKOQFAA, seg_pos={0 0}, upload=12571]: Done uploading piece. err: <nil> Aug 7 21:46:01.304447281 (12h6m0.893792308s) [noise=true , address=104.36.82.11:4115 , piece_id=AH7EPSWFJFX7XDYUQOFCNXEHKQ5AMQX22NY6OFBMMHATCOVH4YJA, seg_pos={0 0}, upload=12571]: Done uploading piece. err: <nil> Aug 7 21:46:01.374462398 (12h6m0.963807385s) [noise=true , address=45.91.116.8:4047 , piece_id=TEMT2SH64IMIPQ3GDE5XCF2R2XLY33HUKRFF6KUWEYH7CDVYD4YQ, seg_pos={0 0}, upload=12571]: Done uploading piece. err: <nil> Aug 7 21:46:01.378351098 (12h6m0.967696085s) [noise=true , address=104.36.82.6:4875 , piece_id=W43EX62262JONZAGLUREOG3EGMICRBIVX5O445J3OQDQ2O475J7A, seg_pos={0 0}, upload=12571]: Done uploading piece. err: <nil> Aug 7 21:46:01.395257154 (12h6m0.984602141s) [noise=true , address=45.91.116.6:4973 , piece_id=7MZSPAEXC3Q7W7U3QMZUQRUI3OHPQI3XRAJW4OGU7UZ73FEI4LIQ, seg_pos={0 0}, upload=12571]: Done uploading piece. err: <nil> Aug 7 21:46:01.404312138 (12h6m0.993657125s) [noise=true , address=77.90.31.19:4843 , piece_id=S73TOGLK5HDCBE4BINZ72A3URDR6TUD43IVIFDY5V37EVCN6222A, seg_pos={0 0}, upload=12571]: Done uploading piece. err: <nil> Aug 7 21:46:01.409522908 (12h6m0.998867885s) [noise=true , address=45.91.116.13:4154 , piece_id=LDSWQ62RRGHTT544S6463WLDKW3WFBWIFQGQDHLYWJJD2ESMYJNQ, seg_pos={0 0}, upload=12571]: Done uploading piece. err: <nil> Aug 7 21:46:01.416315134 (12h6m1.005660121s) [noise=true , address=45.91.116.12:4227 , piece_id=MDQGJ65RT4QEUFX6OFXPEAOFSTG2VXBRC23DXVVD2IV6V52UFF4Q, seg_pos={0 0}, upload=12571]: Done uploading piece. err: <nil> Aug 7 21:46:01.421432725 (12h6m1.010777712s) [noise=true , address=45.91.116.8:4048 , piece_id=2LZRN76O3QS6FODVI52DIIMK7Z77QQBGIXYUAKZCSFI7WMGRTDUA, seg_pos={0 0}, upload=12571]: Done uploading piece. err: <nil> Aug 7 21:46:01.439709112 (12h6m1.029054099s) [noise=true , address=45.91.116.9:4262 , piece_id=D4ABUUAX4MHYLFPR6LN2REDC6VRT2W7D4VZHHYM7GZ6HWOWMG2LA, seg_pos={0 0}, upload=12571]: Done uploading piece. err: <nil> Aug 7 21:46:01.439735472 (12h6m1.029080459s) [seg_pos={0 0}, upload=12571]: Segment reached optimal threshold of 13 pieces. Aug 7 21:46:01.440013234 (12h6m1.029358221s) [noise=true , address=77.90.31.11:4554 , piece_id=LMSB62TMAJONFULH4DAOTPWQMGLUFRZYKVG4HNJ4AUBP7EBP5FFQ, seg_pos={0 0}, upload=12571]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek): protocol: expected piece hash; context canceled; EOF Aug 7 21:46:01.440020518 (12h6m1.029365505s) [noise=true , address=104.36.82.6:4874 , piece_id=T3W6RCAW4YPEU42CESOSUSFSNPDBFXR74WZUYQ2AHJNBZV776MVQ, seg_pos={0 0}, upload=12571]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71): protocol: write tcp 188.166.225.170:54068->104.36.82.6:4874: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 21:46:01.440109398 (12h6m1.029454385s) [noise=true , address=77.90.31.18:4807 , piece_id=63E5KXGYEJHGHYE4SKTNTMED5R4HSWIST4CR4EV6ZGON3YHV73AA, seg_pos={0 0}, upload=12571]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS): protocol: expected piece hash; context canceled; EOF Aug 7 21:46:01.440122312 (12h6m1.029467299s) [seg_pos={0 0}, upload=12571]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 7 21:46:01.440193058 (12h6m1.029538045s) [upload=12571]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 21:46:01.440197016 (12h6m1.029542003s) [upload=12571]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:48:14.557737130 (12h8m14.147082107s) [upload=12571]: Done uploading segments... Aug 7 21:48:14.557758951 (12h8m14.147103938s) [upload=12571]: Upload finished. err: metaclient: context deadline exceeded Duration: 6.023685464s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 21 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXX.X...X...................................] duration: 1.880900281s FAIL (2025-08-07 22:10:58 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== line exceeded Aug 7 21:37:22.572699237 (11h57m22.162044224s) [upload=12432]: Starting upload Aug 7 21:37:22.572782937 (11h57m22.162127924s) [upload=12432]: Done starting upload Aug 7 21:37:22.572842260 (11h57m22.162187247s) [upload=12432]: Uploading segments... Aug 7 21:37:22.572885313 (11h57m22.1622303s) [upload=12432]: Got next segment. Inline: true Aug 7 21:37:22.572889591 (11h57m22.162234578s) [upload=12432]: Waiting for error group managing segments... Aug 7 21:37:22.572930138 (11h57m22.162275125s) [upload=12432]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:37:22.572934296 (11h57m22.162279283s) [upload=12432]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:37:22.572937121 (11h57m22.162282118s) [upload=12432]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:37:51.933954276 (11h57m51.523299253s) [upload=12432]: Done uploading segments... Aug 7 21:37:51.933970927 (11h57m51.523315914s) [upload=12432]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:38:28.745940745 (11h58m28.335285732s) [upload=12448]: Starting upload Aug 7 21:38:28.745997504 (11h58m28.335342491s) [upload=12448]: Done starting upload Aug 7 21:38:28.746040176 (11h58m28.335385163s) [upload=12448]: Uploading segments... Aug 7 21:38:28.746078458 (11h58m28.335423446s) [upload=12448]: Got next segment. Inline: true Aug 7 21:38:28.746081766 (11h58m28.335426743s) [upload=12448]: Waiting for error group managing segments... Aug 7 21:38:28.746111923 (11h58m28.33545691s) [upload=12448]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:38:28.746115540 (11h58m28.335460527s) [upload=12448]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:38:28.746118205 (11h58m28.335463192s) [upload=12448]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:38:58.105315693 (11h58m57.694660691s) [upload=12448]: Done uploading segments... Aug 7 21:38:58.105342465 (11h58m57.694687452s) [upload=12448]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:39:27.137238449 (11h59m26.726583436s) [upload=12462]: Starting upload Aug 7 21:39:27.137294907 (11h59m26.726639894s) [upload=12462]: Done starting upload Aug 7 21:39:27.137337368 (11h59m26.726682355s) [upload=12462]: Uploading segments... Aug 7 21:39:27.137366043 (11h59m26.72671103s) [upload=12462]: Got next segment. Inline: true Aug 7 21:39:27.137370061 (11h59m26.726715048s) [upload=12462]: Waiting for error group managing segments... Aug 7 21:39:27.137401901 (11h59m26.726746879s) [upload=12462]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:39:27.137405518 (11h59m26.726750495s) [upload=12462]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:39:27.137407813 (11h59m26.72675279s) [upload=12462]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:39:56.495186447 (11h59m56.084531434s) [upload=12462]: Done uploading segments... Aug 7 21:39:56.495206415 (11h59m56.084551402s) [upload=12462]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:40:01.914615953 (12h0m1.50396094s) [upload=12475]: Starting upload Aug 7 21:40:01.914684295 (12h0m1.504029282s) [upload=12475]: Done starting upload Aug 7 21:40:01.914741163 (12h0m1.50408616s) [upload=12475]: Uploading segments... Aug 7 21:40:01.914783103 (12h0m1.50412809s) [upload=12475]: Got next segment. Inline: true Aug 7 21:40:01.914788143 (12h0m1.504133131s) [upload=12475]: Waiting for error group managing segments... Aug 7 21:40:01.914833950 (12h0m1.504178937s) [upload=12475]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:40:01.914840073 (12h0m1.50418507s) [upload=12475]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:40:01.914846014 (12h0m1.504191011s) [upload=12475]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:40:31.268196986 (12h0m30.857542023s) [upload=12475]: Done uploading segments... Aug 7 21:40:31.268223536 (12h0m30.857568533s) [upload=12475]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:40:46.248438399 (12h0m45.837783386s) [upload=12487]: Starting upload Aug 7 21:40:46.248491410 (12h0m45.837836387s) [upload=12487]: Done starting upload Aug 7 21:40:46.248540414 (12h0m45.837885401s) [upload=12487]: Uploading segments... Aug 7 21:40:46.248572345 (12h0m45.837917332s) [upload=12487]: Got next segment. Inline: true Aug 7 21:40:46.248576133 (12h0m45.83792112s) [upload=12487]: Waiting for error group managing segments... Aug 7 21:40:46.248613013 (12h0m45.837958s) [upload=12487]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:40:46.248616850 (12h0m45.837961837s) [upload=12487]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:40:46.248619255 (12h0m45.837964242s) [upload=12487]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:40:55.261300379 (12h0m54.850645366s) [upload=12487]: Done uploading segments... Aug 7 21:40:55.261318514 (12h0m54.850663501s) [upload=12487]: Upload finished. err: <nil> Aug 7 21:42:00.373248983 (12h1m59.962594s) [upload=12509]: Starting upload Aug 7 21:42:00.373363813 (12h1m59.96270884s) [upload=12509]: Done starting upload Aug 7 21:42:00.373438045 (12h1m59.962783062s) [upload=12509]: Uploading segments... Aug 7 21:42:00.373501075 (12h1m59.962846083s) [upload=12509]: Got next segment. Inline: true Aug 7 21:42:00.373507809 (12h1m59.962852816s) [upload=12509]: Waiting for error group managing segments... Aug 7 21:42:00.373577642 (12h1m59.962922669s) [upload=12509]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:42:00.373584325 (12h1m59.962929342s) [upload=12509]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:42:00.373590016 (12h1m59.962935053s) [upload=12509]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:42:29.724044552 (12h2m29.313389539s) [upload=12509]: Done uploading segments... Aug 7 21:42:29.724072415 (12h2m29.313417392s) [upload=12509]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:42:30.611135457 (12h2m30.200480444s) [upload=12516]: Starting upload Aug 7 21:42:30.611201894 (12h2m30.200546881s) [upload=12516]: Done starting upload Aug 7 21:42:30.611252732 (12h2m30.200597719s) [upload=12516]: Uploading segments... Aug 7 21:42:30.611338045 (12h2m30.200683032s) [upload=12516]: Got next segment. Inline: true Aug 7 21:42:30.611344076 (12h2m30.200689063s) [upload=12516]: Waiting for error group managing segments... Aug 7 21:42:30.611400083 (12h2m30.20074507s) [upload=12516]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:42:30.611404922 (12h2m30.200749909s) [upload=12516]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:42:30.611408178 (12h2m30.200753166s) [upload=12516]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:42:46.271341803 (12h2m45.860686781s) [upload=12516]: Done uploading segments... Aug 7 21:42:46.271358525 (12h2m45.860703512s) [upload=12516]: Upload finished. err: <nil> Aug 7 21:43:05.160958738 (12h3m4.750303725s) [upload=12528]: Starting upload Aug 7 21:43:05.161064639 (12h3m4.750409626s) [upload=12528]: Done starting upload Aug 7 21:43:05.161177936 (12h3m4.750522953s) [upload=12528]: Uploading segments... Aug 7 21:43:05.161261357 (12h3m4.750606344s) [upload=12528]: Got next segment. Inline: true Aug 7 21:43:05.161266376 (12h3m4.750611353s) [upload=12528]: Waiting for error group managing segments... Aug 7 21:43:05.161306081 (12h3m4.750651069s) [upload=12528]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:43:05.161310580 (12h3m4.750655567s) [upload=12528]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:43:05.161312864 (12h3m4.750657841s) [upload=12528]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:43:17.899724589 (12h3m17.489069576s) [upload=12528]: Done uploading segments... Aug 7 21:43:17.899747483 (12h3m17.48909251s) [upload=12528]: Upload finished. err: <nil> Aug 7 21:43:34.894582431 (12h3m34.483927418s) [upload=12537]: Starting upload Aug 7 21:43:34.894647576 (12h3m34.483992564s) [upload=12537]: Done starting upload Aug 7 21:43:34.894699496 (12h3m34.484044483s) [upload=12537]: Uploading segments... Aug 7 21:43:34.894732940 (12h3m34.484077927s) [upload=12537]: Got next segment. Inline: true Aug 7 21:43:34.894736967 (12h3m34.484081954s) [upload=12537]: Waiting for error group managing segments... Aug 7 21:43:34.894770662 (12h3m34.484115649s) [upload=12537]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:43:34.894775050 (12h3m34.484120027s) [upload=12537]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:43:34.894777795 (12h3m34.484122782s) [upload=12537]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:43:35.191263901 (12h3m34.780608888s) [upload=12537]: Done uploading segments... Aug 7 21:43:35.191280001 (12h3m34.780624978s) [upload=12537]: Upload finished. err: <nil> Aug 7 21:44:27.756622626 (12h4m27.345967613s) [upload=12553]: Starting upload Aug 7 21:44:27.756679294 (12h4m27.346024281s) [upload=12553]: Done starting upload Aug 7 21:44:27.756721205 (12h4m27.346066192s) [upload=12553]: Uploading segments... Aug 7 21:44:27.756751423 (12h4m27.34609641s) [upload=12553]: Got next segment. Inline: true Aug 7 21:44:27.756754779 (12h4m27.346099766s) [upload=12553]: Waiting for error group managing segments... Aug 7 21:44:27.756785678 (12h4m27.346130655s) [upload=12553]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:44:27.756789615 (12h4m27.346134602s) [upload=12553]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:44:27.756791790 (12h4m27.346136777s) [upload=12553]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:44:57.108386877 (12h4m56.697731864s) [upload=12553]: Done uploading segments... Aug 7 21:44:57.108405372 (12h4m56.697750359s) [upload=12553]: Upload finished. err: metaclient: context deadline exceeded Duration: 1.880927783s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 19 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXX.X........................................] duration: 2.342017905s FAIL (2025-08-07 22:10:56 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.342045508s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 5 out of 60 is failed (threshold=3) [X.............X..X....................X....................X] duration: 6.088769904s FAIL (2025-08-07 22:10:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/4M):: after 34e70bda2fe1d7206098dbfb59fc4576ab613f2b752f601da4a5b93473be374e != before ef9ba24509b0003982578e9daec1e6051f8d6ac0bef4f9769b005a8c713045cf ======LOGS====== ding segments... Aug 7 22:10:42.792858699 (12h30m42.382203686s) [upload=12997]: Got next segment. Inline: false Aug 7 22:10:42.792883667 (12h30m42.382228644s) [upload=12997]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:10:42.792887915 (12h30m42.382232902s) [upload=12997]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:10:43.239239407 (12h30m42.828584384s) [seg_pos={0 0}, upload=12997]: Begin upload segment... Aug 7 22:10:43.239300844 (12h30m42.828645831s) [seg_pos={0 0}, upload=12997]: Done begin upload segment. Aug 7 22:10:43.239329008 (12h30m42.828673995s) [noise=true , address=104.36.82.61:4394 , piece_id=BZ4OH34HU6COIEBD542MEZ5YGPXPLFMQKIO2EOV23UBTDTFK5XAA, seg_pos={0 0}, upload=12997]: Uploading piece... Aug 7 22:10:43.239541695 (12h30m42.828886692s) [noise=true , address=104.36.82.37:4687 , piece_id=I7ZESSKAS4ZUYX3CUI4ZRZWY2PLRATRNR6FGVGVAXQD6JT3DRUIA, seg_pos={0 0}, upload=12997]: Uploading piece... Aug 7 22:10:43.239923255 (12h30m42.829268242s) [noise=true , address=104.36.82.10:4087 , piece_id=PDZWS5ZZVKC5BCXJIY43NBQVXM7NCLKJZNSVTD3G2K6F2YEY4RVA, seg_pos={0 0}, upload=12997]: Uploading piece... Aug 7 22:10:43.240056509 (12h30m42.829401506s) [noise=true , address=45.91.116.6:4995 , piece_id=GYQIOSTEVRSY6VKX3GQCUC5YZY5S3ZV7R6X2LBG3QRTOOQD53NOA, seg_pos={0 0}, upload=12997]: Uploading piece... Aug 7 22:10:43.240324512 (12h30m42.829669499s) [noise=true , address=104.36.82.38:4705 , piece_id=M557M36RJRMMND4F33G7BOBYVMESQZZN7ZFESJ5CLHEKXWBCL5SA, seg_pos={0 0}, upload=12997]: Uploading piece... Aug 7 22:10:43.240397562 (12h30m42.829742549s) [noise=true , address=79.127.201.215:4351 , piece_id=367MVHSAIIGC6UZQY4EMH25DJCYYTAIUMPFF3PY426RZJHIO44DA, seg_pos={0 0}, upload=12997]: Uploading piece... Aug 7 22:10:43.240579942 (12h30m42.829924929s) [noise=true , address=104.36.82.47:4524 , piece_id=JDRSOKIHNV2RD3COKNJQHRYG2OXOPUBQOG3AHCTYOKKYC3LZISNQ, seg_pos={0 0}, upload=12997]: Uploading piece... Aug 7 22:10:43.240814029 (12h30m42.830159016s) [noise=true , address=64.49.13.169:4710 , piece_id=KOUJTCIUNCXFZSJWG6ZKR4KV42WV3ABECSARKOQ5YB5W4LAJTGAA, seg_pos={0 0}, upload=12997]: Uploading piece... Aug 7 22:10:43.240831132 (12h30m42.830176119s) [noise=true , address=104.36.82.59:4331 , piece_id=54OHTXRTR2J4S4HECSB7DFPHSC7T4UGBBC4X7WI72IVCMESI5BQQ, seg_pos={0 0}, upload=12997]: Uploading piece... Aug 7 22:10:43.241273939 (12h30m42.830618926s) [noise=true , address=104.36.82.44:4881 , piece_id=YWXSTJSJ46ERQLZ6A2E36CX3R2FI4XVVEZGUMDYJX4ZG3M3R5ACQ, seg_pos={0 0}, upload=12997]: Uploading piece... Aug 7 22:10:43.241589863 (12h30m42.83093485s) [noise=true , address=104.36.82.2:4750 , piece_id=24FHN36X2VIWLTYUDHSJNLSOHHEZ56K6DBIGSMI2ZCA26CPT7L7Q, seg_pos={0 0}, upload=12997]: Uploading piece... Aug 7 22:10:43.241938992 (12h30m42.831283979s) [noise=true , address=104.36.82.3:4791 , piece_id=DJ46BVH37DMTY3YD6VG2AEPNNTPYF7EH4IA3Y633UH3QO6XUTKPQ, seg_pos={0 0}, upload=12997]: Uploading piece... Aug 7 22:10:43.242014867 (12h30m42.831359854s) [noise=true , address=104.36.82.58:4270 , piece_id=HLYO6DZ75CNGFXJ6DA5SKQNVPD5COMDHFAETYUZTBOSKMPT4BUEQ, seg_pos={0 0}, upload=12997]: Uploading piece... Aug 7 22:10:43.242250207 (12h30m42.831595194s) [noise=true , address=104.36.82.44:4885 , piece_id=FJ726HA3PDRWAYDFGUEK2Y7D3B77ERCFPJSGWNKKPLGWXC3WK7GA, seg_pos={0 0}, upload=12997]: Uploading piece... Aug 7 22:10:43.242484636 (12h30m42.831829623s) [noise=true , address=79.127.205.246:4254 , piece_id=V3AT6KY72VDSIMMOZUC4JECVAQSPAHMAPC5CXDVCO46Z6RTPRUVA, seg_pos={0 0}, upload=12997]: Uploading piece... Aug 7 22:10:43.242691451 (12h30m42.832036438s) [noise=true , address=104.36.82.8:4995 , piece_id=BI77BF2ZGP3ZMB2ABMBUCKQCEF3Z6QHRC7IT6Q2ATIF6VON32WWQ, seg_pos={0 0}, upload=12997]: Uploading piece... Aug 7 22:10:43.242894410 (12h30m42.832239397s) [noise=true , address=104.36.82.43:4859 , piece_id=P2S2T5IIUTG4D6EQILFQ7OR5E5JOZHAYO563X7ED5UBN7THKW4QA, seg_pos={0 0}, upload=12997]: Uploading piece... Aug 7 22:10:43.243127415 (12h30m42.832472402s) [noise=true , address=104.36.82.46:4509 , piece_id=WQ3UWQPCMAZ7EDHTSURUQAKL6KWHUA3GSJI6U2G5ZUABRGCTFEXQ, seg_pos={0 0}, upload=12997]: Uploading piece... Aug 7 22:10:44.351806366 (12h30m43.941151353s) [upload=12997]: Next returned nil segment Aug 7 22:10:44.351819280 (12h30m43.941164257s) [upload=12997]: Waiting for error group managing segments... Aug 7 22:10:44.651888156 (12h30m44.241233144s) [noise=true , address=79.127.205.246:4254 , piece_id=V3AT6KY72VDSIMMOZUC4JECVAQSPAHMAPC5CXDVCO46Z6RTPRUVA, seg_pos={0 0}, upload=12997]: Done uploading piece. err: <nil> Aug 7 22:10:44.674573650 (12h30m44.263918637s) [noise=true , address=64.49.13.169:4710 , piece_id=KOUJTCIUNCXFZSJWG6ZKR4KV42WV3ABECSARKOQ5YB5W4LAJTGAA, seg_pos={0 0}, upload=12997]: Done uploading piece. err: <nil> Aug 7 22:10:45.186610734 (12h30m44.775955721s) [noise=true , address=104.36.82.38:4705 , piece_id=M557M36RJRMMND4F33G7BOBYVMESQZZN7ZFESJ5CLHEKXWBCL5SA, seg_pos={0 0}, upload=12997]: Done uploading piece. err: <nil> Aug 7 22:10:45.204653265 (12h30m44.793998252s) [noise=true , address=104.36.82.58:4270 , piece_id=HLYO6DZ75CNGFXJ6DA5SKQNVPD5COMDHFAETYUZTBOSKMPT4BUEQ, seg_pos={0 0}, upload=12997]: Done uploading piece. err: <nil> Aug 7 22:10:45.218003397 (12h30m44.807348374s) [noise=true , address=79.127.201.215:4351 , piece_id=367MVHSAIIGC6UZQY4EMH25DJCYYTAIUMPFF3PY426RZJHIO44DA, seg_pos={0 0}, upload=12997]: Done uploading piece. err: <nil> Aug 7 22:10:45.276599656 (12h30m44.865944643s) [noise=true , address=104.36.82.46:4509 , piece_id=WQ3UWQPCMAZ7EDHTSURUQAKL6KWHUA3GSJI6U2G5ZUABRGCTFEXQ, seg_pos={0 0}, upload=12997]: Done uploading piece. err: <nil> Aug 7 22:10:45.316254365 (12h30m44.905599352s) [noise=true , address=104.36.82.2:4750 , piece_id=24FHN36X2VIWLTYUDHSJNLSOHHEZ56K6DBIGSMI2ZCA26CPT7L7Q, seg_pos={0 0}, upload=12997]: Done uploading piece. err: <nil> Aug 7 22:10:45.349441480 (12h30m44.938786467s) [noise=true , address=104.36.82.3:4791 , piece_id=DJ46BVH37DMTY3YD6VG2AEPNNTPYF7EH4IA3Y633UH3QO6XUTKPQ, seg_pos={0 0}, upload=12997]: Done uploading piece. err: <nil> Aug 7 22:10:45.352466728 (12h30m44.941811715s) [noise=true , address=104.36.82.44:4881 , piece_id=YWXSTJSJ46ERQLZ6A2E36CX3R2FI4XVVEZGUMDYJX4ZG3M3R5ACQ, seg_pos={0 0}, upload=12997]: Done uploading piece. err: <nil> Aug 7 22:10:45.418760245 (12h30m45.008105232s) [noise=true , address=104.36.82.37:4687 , piece_id=I7ZESSKAS4ZUYX3CUI4ZRZWY2PLRATRNR6FGVGVAXQD6JT3DRUIA, seg_pos={0 0}, upload=12997]: Done uploading piece. err: <nil> Aug 7 22:10:45.540632266 (12h30m45.129977253s) [noise=true , address=104.36.82.10:4087 , piece_id=PDZWS5ZZVKC5BCXJIY43NBQVXM7NCLKJZNSVTD3G2K6F2YEY4RVA, seg_pos={0 0}, upload=12997]: Done uploading piece. err: <nil> Aug 7 22:10:45.567016576 (12h30m45.156361563s) [noise=true , address=104.36.82.59:4331 , piece_id=54OHTXRTR2J4S4HECSB7DFPHSC7T4UGBBC4X7WI72IVCMESI5BQQ, seg_pos={0 0}, upload=12997]: Done uploading piece. err: <nil> Aug 7 22:10:45.597615539 (12h30m45.186960526s) [noise=true , address=104.36.82.44:4885 , piece_id=FJ726HA3PDRWAYDFGUEK2Y7D3B77ERCFPJSGWNKKPLGWXC3WK7GA, seg_pos={0 0}, upload=12997]: Done uploading piece. err: <nil> Aug 7 22:10:45.676238976 (12h30m45.265584013s) [noise=true , address=45.91.116.6:4995 , piece_id=GYQIOSTEVRSY6VKX3GQCUC5YZY5S3ZV7R6X2LBG3QRTOOQD53NOA, seg_pos={0 0}, upload=12997]: Done uploading piece. err: <nil> Aug 7 22:10:45.676277709 (12h30m45.265622696s) [seg_pos={0 0}, upload=12997]: Segment reached optimal threshold of 14 pieces. Aug 7 22:10:45.676633781 (12h30m45.265978758s) [noise=true , address=104.36.82.8:4995 , piece_id=BI77BF2ZGP3ZMB2ABMBUCKQCEF3Z6QHRC7IT6Q2ATIF6VON32WWQ, seg_pos={0 0}, upload=12997]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tvoTD1LoSj3q2rLiDckfT56dXNKKwdVu1ZAsBZExNFVTom9tH): protocol: expected piece hash; context canceled; EOF Aug 7 22:10:45.676841539 (12h30m45.266186526s) [noise=true , address=104.36.82.43:4859 , piece_id=P2S2T5IIUTG4D6EQILFQ7OR5E5JOZHAYO563X7ED5UBN7THKW4QA, seg_pos={0 0}, upload=12997]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:126SqCtRGPziEeopRjA4nNkjkCZWvUwkt5XaMgcqZKJwoxuYWY7): protocol: rpc: dial tcp 104.36.82.43:4859: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 22:10:45.676869993 (12h30m45.26621498s) [noise=true , address=104.36.82.47:4524 , piece_id=JDRSOKIHNV2RD3COKNJQHRYG2OXOPUBQOG3AHCTYOKKYC3LZISNQ, seg_pos={0 0}, upload=12997]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xfbCzjSwFDRrR1gPggeN8tNCLbRRWDVzCJ6mqhZsbVSe8d4s4): protocol: read tcp 188.166.225.170:58322->104.36.82.47:4524: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 22:10:45.676908757 (12h30m45.266253744s) [noise=true , address=104.36.82.61:4394 , piece_id=BZ4OH34HU6COIEBD542MEZ5YGPXPLFMQKIO2EOV23UBTDTFK5XAA, seg_pos={0 0}, upload=12997]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1dSpTqCxeAcL9PVYbP7BhmuKE7papFPc9swB1Fy83WcFDgQbcF): protocol: rpc: dial tcp 104.36.82.61:4394: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 22:10:45.676927984 (12h30m45.266272961s) [seg_pos={0 0}, upload=12997]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 22:10:45.677012455 (12h30m45.266357443s) [upload=12997]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 22:10:45.677016543 (12h30m45.26636152s) [upload=12997]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:10:45.998268658 (12h30m45.587613635s) [upload=12997]: Done uploading segments... Aug 7 22:10:45.998285902 (12h30m45.587630889s) [upload=12997]: Upload finished. err: <nil> Duration: 6.088796657s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway) antiblip: 3 out of 60 is failed (threshold=3) [..X.X...X...................................................] duration: 753.301373ms remote: "tcp:45.38.24.2:443" target: https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410 FAIL (2025-08-07 22:10:48 UTC) orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": context deadline exceeded Duration: 753.323675ms edge gateway vhost us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXX.......X...............................] duration: 6.202384358s FAIL (2025-08-07 22:10:48 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.202407142s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 3 out of 60 is failed (threshold=3) [........X..........X.................X......................] duration: 1.900952175s FAIL (2025-08-07 22:10:44 UTC) orbiter warning: operation: ListObjects ======ERROR====== unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/1K ======LOGS====== 8m27.919686751s) [upload=12612]: Done uploading segments... Aug 7 21:48:28.330358527 (12h8m27.919703514s) [upload=12612]: Upload finished. err: <nil> Aug 7 21:48:54.943937458 (12h8m54.533282445s) [upload=12618]: Starting upload Aug 7 21:48:54.944002622 (12h8m54.533347609s) [upload=12618]: Done starting upload Aug 7 21:48:54.944106100 (12h8m54.533451087s) [upload=12618]: Uploading segments... Aug 7 21:48:54.944196242 (12h8m54.53354132s) [upload=12618]: Got next segment. Inline: true Aug 7 21:48:54.944206543 (12h8m54.53355158s) [upload=12618]: Waiting for error group managing segments... Aug 7 21:48:54.944254484 (12h8m54.533599461s) [upload=12618]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:48:54.944259223 (12h8m54.5336042s) [upload=12618]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:48:54.944262139 (12h8m54.533607126s) [upload=12618]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:48:55.330753187 (12h8m54.920098174s) [upload=12618]: Done uploading segments... Aug 7 21:48:55.330774959 (12h8m54.920119936s) [upload=12618]: Upload finished. err: <nil> Aug 7 21:49:36.650951942 (12h9m36.240296929s) [upload=12631]: Starting upload Aug 7 21:49:36.650993252 (12h9m36.240338239s) [upload=12631]: Done starting upload Aug 7 21:49:36.651049029 (12h9m36.240394016s) [upload=12631]: Uploading segments... Aug 7 21:49:36.651075309 (12h9m36.240420286s) [upload=12631]: Got next segment. Inline: true Aug 7 21:49:36.651078775 (12h9m36.240423763s) [upload=12631]: Waiting for error group managing segments... Aug 7 21:49:36.651115776 (12h9m36.240460773s) [upload=12631]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:49:36.651120254 (12h9m36.240465251s) [upload=12631]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:49:36.651124473 (12h9m36.24046947s) [upload=12631]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:49:36.920683881 (12h9m36.510028868s) [upload=12631]: Done uploading segments... Aug 7 21:49:36.920707246 (12h9m36.510052233s) [upload=12631]: Upload finished. err: <nil> Aug 7 21:50:03.363429937 (12h10m2.952774924s) [upload=12636]: Starting upload Aug 7 21:50:03.363493408 (12h10m2.952838405s) [upload=12636]: Done starting upload Aug 7 21:50:03.363582518 (12h10m2.952927515s) [upload=12636]: Uploading segments... Aug 7 21:50:03.363692078 (12h10m2.953037085s) [upload=12636]: Got next segment. Inline: true Aug 7 21:50:03.363699752 (12h10m2.953044739s) [upload=12636]: Waiting for error group managing segments... Aug 7 21:50:03.363764116 (12h10m2.953109103s) [upload=12636]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:50:03.363773323 (12h10m2.95311831s) [upload=12636]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:50:03.363777271 (12h10m2.953122258s) [upload=12636]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:50:03.636022787 (12h10m3.225367774s) [upload=12636]: Done uploading segments... Aug 7 21:50:03.636042204 (12h10m3.225387202s) [upload=12636]: Upload finished. err: <nil> Aug 7 21:50:20.944550175 (12h10m20.533895152s) [upload=12642]: Starting upload Aug 7 21:50:20.944594911 (12h10m20.533939898s) [upload=12642]: Done starting upload Aug 7 21:50:20.944644405 (12h10m20.533989392s) [upload=12642]: Uploading segments... Aug 7 21:50:20.944674473 (12h10m20.534019461s) [upload=12642]: Got next segment. Inline: true Aug 7 21:50:20.944677820 (12h10m20.534022807s) [upload=12642]: Waiting for error group managing segments... Aug 7 21:50:20.944709450 (12h10m20.534054437s) [upload=12642]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:50:20.944715131 (12h10m20.534060118s) [upload=12642]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:50:20.944717505 (12h10m20.534062493s) [upload=12642]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:50:21.212077476 (12h10m20.801422473s) [upload=12642]: Done uploading segments... Aug 7 21:50:21.212101602 (12h10m20.801446599s) [upload=12642]: Upload finished. err: <nil> Aug 7 21:51:22.649621444 (12h11m22.238966431s) [upload=12660]: Starting upload Aug 7 21:51:22.649696837 (12h11m22.239041824s) [upload=12660]: Done starting upload Aug 7 21:51:22.649777742 (12h11m22.239122729s) [upload=12660]: Uploading segments... Aug 7 21:51:22.649867044 (12h11m22.239212041s) [upload=12660]: Got next segment. Inline: true Aug 7 21:51:22.649872053 (12h11m22.23921704s) [upload=12660]: Waiting for error group managing segments... Aug 7 21:51:22.649940053 (12h11m22.23928504s) [upload=12660]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:51:22.649946785 (12h11m22.239291772s) [upload=12660]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:51:22.649949932 (12h11m22.239294919s) [upload=12660]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:51:22.914527578 (12h11m22.503872565s) [upload=12660]: Done uploading segments... Aug 7 21:51:22.914549130 (12h11m22.503894117s) [upload=12660]: Upload finished. err: <nil> Aug 7 21:51:38.628060070 (12h11m38.217405057s) [upload=12665]: Starting upload Aug 7 21:51:38.628100548 (12h11m38.217445535s) [upload=12665]: Done starting upload Aug 7 21:51:38.628155373 (12h11m38.21750036s) [upload=12665]: Uploading segments... Aug 7 21:51:38.628187814 (12h11m38.217532802s) [upload=12665]: Got next segment. Inline: true Aug 7 21:51:38.628191381 (12h11m38.217536368s) [upload=12665]: Waiting for error group managing segments... Aug 7 21:51:38.628222691 (12h11m38.217567678s) [upload=12665]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:51:38.628226268 (12h11m38.217571255s) [upload=12665]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:51:38.628228662 (12h11m38.217573649s) [upload=12665]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:51:38.940125222 (12h11m38.529470269s) [upload=12665]: Done uploading segments... Aug 7 21:51:38.940147124 (12h11m38.529492191s) [upload=12665]: Upload finished. err: <nil> Aug 7 21:52:16.262879794 (12h12m15.852224781s) [upload=12676]: Starting upload Aug 7 21:52:16.262939168 (12h12m15.852284155s) [upload=12676]: Done starting upload Aug 7 21:52:16.263047876 (12h12m15.852392873s) [upload=12676]: Uploading segments... Aug 7 21:52:16.263139601 (12h12m15.852484598s) [upload=12676]: Got next segment. Inline: true Aug 7 21:52:16.263144440 (12h12m15.852489417s) [upload=12676]: Waiting for error group managing segments... Aug 7 21:52:16.263191931 (12h12m15.852536918s) [upload=12676]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:52:16.263197472 (12h12m15.852542449s) [upload=12676]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:52:16.263200056 (12h12m15.852545043s) [upload=12676]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:52:16.570506954 (12h12m16.159851931s) [upload=12676]: Done uploading segments... Aug 7 21:52:16.570524978 (12h12m16.159869965s) [upload=12676]: Upload finished. err: <nil> Aug 7 21:53:00.574165558 (12h13m0.163510545s) [upload=12691]: Starting upload Aug 7 21:53:00.574212286 (12h13m0.163557273s) [upload=12691]: Done starting upload Aug 7 21:53:00.574248476 (12h13m0.163593463s) [upload=12691]: Uploading segments... Aug 7 21:53:00.574327307 (12h13m0.163672294s) [upload=12691]: Got next segment. Inline: true Aug 7 21:53:00.574332868 (12h13m0.163677855s) [upload=12691]: Waiting for error group managing segments... Aug 7 21:53:00.574373505 (12h13m0.163718492s) [upload=12691]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:53:00.574390598 (12h13m0.163735585s) [upload=12691]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:53:00.574393373 (12h13m0.16373835s) [upload=12691]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:53:00.840335180 (12h13m0.429680167s) [upload=12691]: Done uploading segments... Aug 7 21:53:00.840351500 (12h13m0.429696487s) [upload=12691]: Upload finished. err: <nil> Aug 7 21:53:46.136340579 (12h13m45.725685566s) [upload=12705]: Starting upload Aug 7 21:53:46.136382029 (12h13m45.725727016s) [upload=12705]: Done starting upload Aug 7 21:53:46.136495205 (12h13m45.725840192s) [upload=12705]: Uploading segments... Aug 7 21:53:46.136529601 (12h13m45.725874588s) [upload=12705]: Got next segment. Inline: true Aug 7 21:53:46.136533228 (12h13m45.725878215s) [upload=12705]: Waiting for error group managing segments... Aug 7 21:53:46.136581701 (12h13m45.725926688s) [upload=12705]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:53:46.136585949 (12h13m45.725930936s) [upload=12705]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:53:46.136588393 (12h13m45.72593338s) [upload=12705]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:53:46.427021148 (12h13m46.016366155s) [upload=12705]: Done uploading segments... Aug 7 21:53:46.427045194 (12h13m46.016390181s) [upload=12705]: Upload finished. err: <nil> Aug 7 21:54:41.670591536 (12h14m41.259936523s) [upload=12721]: Starting upload Aug 7 21:54:41.670632916 (12h14m41.259977903s) [upload=12721]: Done starting upload Aug 7 21:54:41.670677612 (12h14m41.260022599s) [upload=12721]: Uploading segments... Aug 7 21:54:41.670708571 (12h14m41.260053558s) [upload=12721]: Got next segment. Inline: true Aug 7 21:54:41.670713510 (12h14m41.260058497s) [upload=12721]: Waiting for error group managing segments... Aug 7 21:54:41.670746032 (12h14m41.260091019s) [upload=12721]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:54:41.670749979 (12h14m41.260094966s) [upload=12721]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:54:41.670752424 (12h14m41.260097411s) [upload=12721]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:54:41.940759391 (12h14m41.530104368s) [upload=12721]: Done uploading segments... Aug 7 21:54:41.940776734 (12h14m41.530121721s) [upload=12721]: Upload finished. err: <nil> Duration: 1.900975921s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway) antiblip: 7 out of 60 is failed (threshold=3) [.......XX.XX..X.XX..........................................] duration: 326.538221ms remote: "tcp:207.211.208.129:443" target: https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222 FAIL (2025-08-07 22:10:39 UTC) orbiter warning: Get "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222": context deadline exceeded Duration: 326.568788ms edge gateway vhost bgp us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 26 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXX..XX..X..............................] duration: 5.606449745s FAIL (2025-08-07 22:10:38 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 5.60783238s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 22 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXX.X...X..................................] duration: 1.823646629s FAIL (2025-08-07 22:10:13 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== line exceeded Aug 7 21:37:22.572699237 (11h57m22.162044224s) [upload=12432]: Starting upload Aug 7 21:37:22.572782937 (11h57m22.162127924s) [upload=12432]: Done starting upload Aug 7 21:37:22.572842260 (11h57m22.162187247s) [upload=12432]: Uploading segments... Aug 7 21:37:22.572885313 (11h57m22.1622303s) [upload=12432]: Got next segment. Inline: true Aug 7 21:37:22.572889591 (11h57m22.162234578s) [upload=12432]: Waiting for error group managing segments... Aug 7 21:37:22.572930138 (11h57m22.162275125s) [upload=12432]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:37:22.572934296 (11h57m22.162279283s) [upload=12432]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:37:22.572937121 (11h57m22.162282118s) [upload=12432]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:37:51.933954276 (11h57m51.523299253s) [upload=12432]: Done uploading segments... Aug 7 21:37:51.933970927 (11h57m51.523315914s) [upload=12432]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:38:28.745940745 (11h58m28.335285732s) [upload=12448]: Starting upload Aug 7 21:38:28.745997504 (11h58m28.335342491s) [upload=12448]: Done starting upload Aug 7 21:38:28.746040176 (11h58m28.335385163s) [upload=12448]: Uploading segments... Aug 7 21:38:28.746078458 (11h58m28.335423446s) [upload=12448]: Got next segment. Inline: true Aug 7 21:38:28.746081766 (11h58m28.335426743s) [upload=12448]: Waiting for error group managing segments... Aug 7 21:38:28.746111923 (11h58m28.33545691s) [upload=12448]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:38:28.746115540 (11h58m28.335460527s) [upload=12448]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:38:28.746118205 (11h58m28.335463192s) [upload=12448]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:38:58.105315693 (11h58m57.694660691s) [upload=12448]: Done uploading segments... Aug 7 21:38:58.105342465 (11h58m57.694687452s) [upload=12448]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:39:27.137238449 (11h59m26.726583436s) [upload=12462]: Starting upload Aug 7 21:39:27.137294907 (11h59m26.726639894s) [upload=12462]: Done starting upload Aug 7 21:39:27.137337368 (11h59m26.726682355s) [upload=12462]: Uploading segments... Aug 7 21:39:27.137366043 (11h59m26.72671103s) [upload=12462]: Got next segment. Inline: true Aug 7 21:39:27.137370061 (11h59m26.726715048s) [upload=12462]: Waiting for error group managing segments... Aug 7 21:39:27.137401901 (11h59m26.726746879s) [upload=12462]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:39:27.137405518 (11h59m26.726750495s) [upload=12462]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:39:27.137407813 (11h59m26.72675279s) [upload=12462]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:39:56.495186447 (11h59m56.084531434s) [upload=12462]: Done uploading segments... Aug 7 21:39:56.495206415 (11h59m56.084551402s) [upload=12462]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:40:01.914615953 (12h0m1.50396094s) [upload=12475]: Starting upload Aug 7 21:40:01.914684295 (12h0m1.504029282s) [upload=12475]: Done starting upload Aug 7 21:40:01.914741163 (12h0m1.50408616s) [upload=12475]: Uploading segments... Aug 7 21:40:01.914783103 (12h0m1.50412809s) [upload=12475]: Got next segment. Inline: true Aug 7 21:40:01.914788143 (12h0m1.504133131s) [upload=12475]: Waiting for error group managing segments... Aug 7 21:40:01.914833950 (12h0m1.504178937s) [upload=12475]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:40:01.914840073 (12h0m1.50418507s) [upload=12475]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:40:01.914846014 (12h0m1.504191011s) [upload=12475]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:40:31.268196986 (12h0m30.857542023s) [upload=12475]: Done uploading segments... Aug 7 21:40:31.268223536 (12h0m30.857568533s) [upload=12475]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:40:46.248438399 (12h0m45.837783386s) [upload=12487]: Starting upload Aug 7 21:40:46.248491410 (12h0m45.837836387s) [upload=12487]: Done starting upload Aug 7 21:40:46.248540414 (12h0m45.837885401s) [upload=12487]: Uploading segments... Aug 7 21:40:46.248572345 (12h0m45.837917332s) [upload=12487]: Got next segment. Inline: true Aug 7 21:40:46.248576133 (12h0m45.83792112s) [upload=12487]: Waiting for error group managing segments... Aug 7 21:40:46.248613013 (12h0m45.837958s) [upload=12487]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:40:46.248616850 (12h0m45.837961837s) [upload=12487]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:40:46.248619255 (12h0m45.837964242s) [upload=12487]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:40:55.261300379 (12h0m54.850645366s) [upload=12487]: Done uploading segments... Aug 7 21:40:55.261318514 (12h0m54.850663501s) [upload=12487]: Upload finished. err: <nil> Aug 7 21:42:00.373248983 (12h1m59.962594s) [upload=12509]: Starting upload Aug 7 21:42:00.373363813 (12h1m59.96270884s) [upload=12509]: Done starting upload Aug 7 21:42:00.373438045 (12h1m59.962783062s) [upload=12509]: Uploading segments... Aug 7 21:42:00.373501075 (12h1m59.962846083s) [upload=12509]: Got next segment. Inline: true Aug 7 21:42:00.373507809 (12h1m59.962852816s) [upload=12509]: Waiting for error group managing segments... Aug 7 21:42:00.373577642 (12h1m59.962922669s) [upload=12509]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:42:00.373584325 (12h1m59.962929342s) [upload=12509]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:42:00.373590016 (12h1m59.962935053s) [upload=12509]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:42:29.724044552 (12h2m29.313389539s) [upload=12509]: Done uploading segments... Aug 7 21:42:29.724072415 (12h2m29.313417392s) [upload=12509]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:42:30.611135457 (12h2m30.200480444s) [upload=12516]: Starting upload Aug 7 21:42:30.611201894 (12h2m30.200546881s) [upload=12516]: Done starting upload Aug 7 21:42:30.611252732 (12h2m30.200597719s) [upload=12516]: Uploading segments... Aug 7 21:42:30.611338045 (12h2m30.200683032s) [upload=12516]: Got next segment. Inline: true Aug 7 21:42:30.611344076 (12h2m30.200689063s) [upload=12516]: Waiting for error group managing segments... Aug 7 21:42:30.611400083 (12h2m30.20074507s) [upload=12516]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:42:30.611404922 (12h2m30.200749909s) [upload=12516]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:42:30.611408178 (12h2m30.200753166s) [upload=12516]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:42:46.271341803 (12h2m45.860686781s) [upload=12516]: Done uploading segments... Aug 7 21:42:46.271358525 (12h2m45.860703512s) [upload=12516]: Upload finished. err: <nil> Aug 7 21:43:05.160958738 (12h3m4.750303725s) [upload=12528]: Starting upload Aug 7 21:43:05.161064639 (12h3m4.750409626s) [upload=12528]: Done starting upload Aug 7 21:43:05.161177936 (12h3m4.750522953s) [upload=12528]: Uploading segments... Aug 7 21:43:05.161261357 (12h3m4.750606344s) [upload=12528]: Got next segment. Inline: true Aug 7 21:43:05.161266376 (12h3m4.750611353s) [upload=12528]: Waiting for error group managing segments... Aug 7 21:43:05.161306081 (12h3m4.750651069s) [upload=12528]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:43:05.161310580 (12h3m4.750655567s) [upload=12528]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:43:05.161312864 (12h3m4.750657841s) [upload=12528]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:43:17.899724589 (12h3m17.489069576s) [upload=12528]: Done uploading segments... Aug 7 21:43:17.899747483 (12h3m17.48909251s) [upload=12528]: Upload finished. err: <nil> Aug 7 21:43:34.894582431 (12h3m34.483927418s) [upload=12537]: Starting upload Aug 7 21:43:34.894647576 (12h3m34.483992564s) [upload=12537]: Done starting upload Aug 7 21:43:34.894699496 (12h3m34.484044483s) [upload=12537]: Uploading segments... Aug 7 21:43:34.894732940 (12h3m34.484077927s) [upload=12537]: Got next segment. Inline: true Aug 7 21:43:34.894736967 (12h3m34.484081954s) [upload=12537]: Waiting for error group managing segments... Aug 7 21:43:34.894770662 (12h3m34.484115649s) [upload=12537]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:43:34.894775050 (12h3m34.484120027s) [upload=12537]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:43:34.894777795 (12h3m34.484122782s) [upload=12537]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:43:35.191263901 (12h3m34.780608888s) [upload=12537]: Done uploading segments... Aug 7 21:43:35.191280001 (12h3m34.780624978s) [upload=12537]: Upload finished. err: <nil> Aug 7 21:44:27.756622626 (12h4m27.345967613s) [upload=12553]: Starting upload Aug 7 21:44:27.756679294 (12h4m27.346024281s) [upload=12553]: Done starting upload Aug 7 21:44:27.756721205 (12h4m27.346066192s) [upload=12553]: Uploading segments... Aug 7 21:44:27.756751423 (12h4m27.34609641s) [upload=12553]: Got next segment. Inline: true Aug 7 21:44:27.756754779 (12h4m27.346099766s) [upload=12553]: Waiting for error group managing segments... Aug 7 21:44:27.756785678 (12h4m27.346130655s) [upload=12553]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:44:27.756789615 (12h4m27.346134602s) [upload=12553]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:44:27.756791790 (12h4m27.346136777s) [upload=12553]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:44:57.108386877 (12h4m56.697731864s) [upload=12553]: Done uploading segments... Aug 7 21:44:57.108405372 (12h4m56.697750359s) [upload=12553]: Upload finished. err: metaclient: context deadline exceeded Duration: 1.823673702s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 27 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXX...X..X............................] duration: 5.732015422s FAIL (2025-08-07 22:10:12 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.732036393s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 24 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXX.....X...............................] duration: 6.841989848s FAIL (2025-08-07 22:10:06 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== t segment. Inline: false Aug 7 21:44:25.061589945 (12h4m24.650934922s) [upload=12552]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:44:25.061594724 (12h4m24.650939711s) [upload=12552]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:44:50.602885060 (12h4m50.192230047s) [seg_pos={0 0}, upload=12552]: Begin upload segment... Aug 7 21:44:50.602935446 (12h4m50.192280423s) [seg_pos={0 0}, upload=12552]: Done begin upload segment. Aug 7 21:44:50.603112565 (12h4m50.192457552s) [noise=true , address=79.127.205.233:4624 , piece_id=WEAYHGF7VE6W5ML3QAE3AITMEJGQFH4JQZ7RQKR7O4NKTPR6RZRA, seg_pos={0 0}, upload=12552]: Uploading piece... Aug 7 21:44:50.603695610 (12h4m50.193040597s) [noise=true , address=104.36.82.22:4529 , piece_id=ZTTBM5H6MHVMQ3OFVTVISN6WJ7WB3XBWI7YZKUXCFNQ6NO3D4CGA, seg_pos={0 0}, upload=12552]: Uploading piece... Aug 7 21:44:50.604300768 (12h4m50.193645755s) [noise=true , address=45.91.116.6:4002 , piece_id=AE5EL7NWD6FEQ7NXX2JRHB4MVZQDXRGQU3EER2X7COISFAX6P3XQ, seg_pos={0 0}, upload=12552]: Uploading piece... Aug 7 21:44:50.604588519 (12h4m50.193933496s) [noise=true , address=104.36.82.56:4249 , piece_id=RD4B7IDIJV25L7EJNSYCZGW5QKJEM435E2N3UFUD44Q4S3DCBOXQ, seg_pos={0 0}, upload=12552]: Uploading piece... Aug 7 21:44:50.605153860 (12h4m50.194498837s) [noise=true , address=77.90.31.11:4562 , piece_id=XC6F2WLSXKQB7E7MBRNZBJKUSUFL6XNA5TVX5M3ZJI3ZZMK7UBLA, seg_pos={0 0}, upload=12552]: Uploading piece... Aug 7 21:44:50.605455377 (12h4m50.194800364s) [noise=true , address=104.36.82.23:4466 , piece_id=5J5X2LSP6GCX2PNRIDZYFJPISWSX3YSRID5M2WU6Y5Z6LWRHLC4Q, seg_pos={0 0}, upload=12552]: Uploading piece... Aug 7 21:44:50.605743810 (12h4m50.195088797s) [noise=true , address=104.36.82.62:4420 , piece_id=UOO36PYVE5WFRRZLGYLJ6I6SV4EO7W2ZH2TIXJ2OIZZMY3LTHR2A, seg_pos={0 0}, upload=12552]: Uploading piece... Aug 7 21:44:50.606547457 (12h4m50.195892444s) [noise=true , address=104.36.82.5:4842 , piece_id=E5DSJXN63RFX7EQ4ZTBMUESFW77EB3HQNA5KCQGRVRAA5XO2GN3Q, seg_pos={0 0}, upload=12552]: Uploading piece... Aug 7 21:44:50.606982099 (12h4m50.196327086s) [noise=true , address=104.36.82.59:4301 , piece_id=VVR5CQWCPNPZUFEFCPB6ZL2KASWS5XO65FVDI474RKRZSJFDUQTA, seg_pos={0 0}, upload=12552]: Uploading piece... Aug 7 21:44:50.607295849 (12h4m50.196640836s) [noise=true , address=45.91.116.14:4198 , piece_id=2HRU3JZDQSNGD5NJZDLZSRG2WIO2Y2QBSOAY6KUEGUKDBWM53RSA, seg_pos={0 0}, upload=12552]: Uploading piece... Aug 7 21:44:50.607695593 (12h4m50.19704058s) [noise=true , address=104.36.82.6:4881 , piece_id=VUC33D7TLVFICDLA3334DEKBJE5E5P4HQTJU3VDQNMVYWVESSJWA, seg_pos={0 0}, upload=12552]: Uploading piece... Aug 7 21:44:50.608364103 (12h4m50.19770909s) [noise=true , address=104.36.82.2:4757 , piece_id=H6KNYP5HASPWQIHADTMEZDC5AE5HITCAY73XGNWVEXJOPDEZ3SHA, seg_pos={0 0}, upload=12552]: Uploading piece... Aug 7 21:44:50.608764288 (12h4m50.198109265s) [noise=true , address=104.36.82.46:4493 , piece_id=PS4S4OZ6HSITEJFPJDIBWQ5RGWTDRKMX3NCEWXCLC2PSP5Q2KPZQ, seg_pos={0 0}, upload=12552]: Uploading piece... Aug 7 21:44:50.609065284 (12h4m50.198410271s) [noise=true , address=104.36.82.47:4518 , piece_id=KJ3PJCT3EDH3WZBT7AWWKI6ABGVVAMKZTVJY7Q76FULC72TUWA6Q, seg_pos={0 0}, upload=12552]: Uploading piece... Aug 7 21:44:50.609312777 (12h4m50.198657755s) [noise=true , address=104.36.82.60:4349 , piece_id=GTTF32YMJWBY2GZEFZPIBZXZWNI7MBLUYCTQ3SRFG735PLWXTXBQ, seg_pos={0 0}, upload=12552]: Uploading piece... Aug 7 21:44:50.609589888 (12h4m50.198934875s) [noise=true , address=104.36.82.52:4116 , piece_id=7QA2Q5ACN43TQUOKWA34EZFSSQBE3OWNJTYODHHQ5GCVOA4QZOJA, seg_pos={0 0}, upload=12552]: Uploading piece... Aug 7 21:44:50.610295969 (12h4m50.199641056s) [noise=true , address=104.36.82.37:4678 , piece_id=VJO5JO3SJEXBQZC6PJ34DRKWEQKOPI33ZF4IHNNOOTSD7IBRR4PQ, seg_pos={0 0}, upload=12552]: Uploading piece... Aug 7 21:44:50.610811305 (12h4m50.200156302s) [noise=true , address=64.49.13.169:4714 , piece_id=A6O3H7QLE6KQTHWUKOEROVHPEHVLG22KNF2KCV3HK2KCLXP7SYVA, seg_pos={0 0}, upload=12552]: Uploading piece... Aug 7 21:44:51.715421583 (12h4m51.30476657s) [upload=12552]: Next returned nil segment Aug 7 21:44:51.715438125 (12h4m51.304783112s) [upload=12552]: Waiting for error group managing segments... Aug 7 21:44:51.998131281 (12h4m51.587476268s) [noise=true , address=64.49.13.169:4714 , piece_id=A6O3H7QLE6KQTHWUKOEROVHPEHVLG22KNF2KCV3HK2KCLXP7SYVA, seg_pos={0 0}, upload=12552]: Done uploading piece. err: <nil> Aug 7 21:44:51.999236335 (12h4m51.588581312s) [noise=true , address=79.127.205.233:4624 , piece_id=WEAYHGF7VE6W5ML3QAE3AITMEJGQFH4JQZ7RQKR7O4NKTPR6RZRA, seg_pos={0 0}, upload=12552]: Done uploading piece. err: <nil> Aug 7 21:44:52.638299991 (12h4m52.227644978s) [noise=true , address=45.91.116.6:4002 , piece_id=AE5EL7NWD6FEQ7NXX2JRHB4MVZQDXRGQU3EER2X7COISFAX6P3XQ, seg_pos={0 0}, upload=12552]: Done uploading piece. err: <nil> Aug 7 21:44:52.639927565 (12h4m52.229272542s) [noise=true , address=104.36.82.6:4881 , piece_id=VUC33D7TLVFICDLA3334DEKBJE5E5P4HQTJU3VDQNMVYWVESSJWA, seg_pos={0 0}, upload=12552]: Done uploading piece. err: <nil> Aug 7 21:44:52.643220585 (12h4m52.232565572s) [noise=true , address=104.36.82.2:4757 , piece_id=H6KNYP5HASPWQIHADTMEZDC5AE5HITCAY73XGNWVEXJOPDEZ3SHA, seg_pos={0 0}, upload=12552]: Done uploading piece. err: <nil> Aug 7 21:44:52.711872926 (12h4m52.301217913s) [noise=true , address=104.36.82.56:4249 , piece_id=RD4B7IDIJV25L7EJNSYCZGW5QKJEM435E2N3UFUD44Q4S3DCBOXQ, seg_pos={0 0}, upload=12552]: Done uploading piece. err: <nil> Aug 7 21:44:52.763601896 (12h4m52.352946883s) [noise=true , address=104.36.82.22:4529 , piece_id=ZTTBM5H6MHVMQ3OFVTVISN6WJ7WB3XBWI7YZKUXCFNQ6NO3D4CGA, seg_pos={0 0}, upload=12552]: Done uploading piece. err: <nil> Aug 7 21:44:52.768238797 (12h4m52.357583785s) [noise=true , address=104.36.82.60:4349 , piece_id=GTTF32YMJWBY2GZEFZPIBZXZWNI7MBLUYCTQ3SRFG735PLWXTXBQ, seg_pos={0 0}, upload=12552]: Done uploading piece. err: <nil> Aug 7 21:44:52.798543408 (12h4m52.387888395s) [noise=true , address=45.91.116.14:4198 , piece_id=2HRU3JZDQSNGD5NJZDLZSRG2WIO2Y2QBSOAY6KUEGUKDBWM53RSA, seg_pos={0 0}, upload=12552]: Done uploading piece. err: <nil> Aug 7 21:44:52.861192704 (12h4m52.450537691s) [noise=true , address=77.90.31.11:4562 , piece_id=XC6F2WLSXKQB7E7MBRNZBJKUSUFL6XNA5TVX5M3ZJI3ZZMK7UBLA, seg_pos={0 0}, upload=12552]: Done uploading piece. err: <nil> Aug 7 21:44:53.070218164 (12h4m52.659563151s) [noise=true , address=104.36.82.59:4301 , piece_id=VVR5CQWCPNPZUFEFCPB6ZL2KASWS5XO65FVDI474RKRZSJFDUQTA, seg_pos={0 0}, upload=12552]: Done uploading piece. err: <nil> Aug 7 21:44:53.558930346 (12h4m53.148275333s) [noise=true , address=104.36.82.46:4493 , piece_id=PS4S4OZ6HSITEJFPJDIBWQ5RGWTDRKMX3NCEWXCLC2PSP5Q2KPZQ, seg_pos={0 0}, upload=12552]: Done uploading piece. err: <nil> Aug 7 21:44:55.132342777 (12h4m54.721687765s) [noise=true , address=104.36.82.37:4678 , piece_id=VJO5JO3SJEXBQZC6PJ34DRKWEQKOPI33ZF4IHNNOOTSD7IBRR4PQ, seg_pos={0 0}, upload=12552]: Done uploading piece. err: <nil> Aug 7 21:44:55.265236558 (12h4m54.854581545s) [noise=true , address=104.36.82.52:4116 , piece_id=7QA2Q5ACN43TQUOKWA34EZFSSQBE3OWNJTYODHHQ5GCVOA4QZOJA, seg_pos={0 0}, upload=12552]: Done uploading piece. err: <nil> Aug 7 21:44:55.265254161 (12h4m54.854599138s) [seg_pos={0 0}, upload=12552]: Segment reached optimal threshold of 14 pieces. Aug 7 21:44:55.265482308 (12h4m54.854827295s) [noise=true , address=104.36.82.5:4842 , piece_id=E5DSJXN63RFX7EQ4ZTBMUESFW77EB3HQNA5KCQGRVRAA5XO2GN3Q, seg_pos={0 0}, upload=12552]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12q4VdT5BcjUb16PBYHK6zrnW8dqENKrcGx9zdLtNefDVB1MUSn): protocol: expected piece hash; write tcp 188.166.225.170:57020->104.36.82.5:4842: use of closed network connection; write tcp 188.166.225.170:57020->104.36.82.5:4842: use of closed network connection; EOF Aug 7 21:44:55.265551430 (12h4m54.854896417s) [noise=true , address=104.36.82.47:4518 , piece_id=KJ3PJCT3EDH3WZBT7AWWKI6ABGVVAMKZTVJY7Q76FULC72TUWA6Q, seg_pos={0 0}, upload=12552]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17TZ9JLdCivFF46guhSTW6hcoUmTz5uo5wvLkDqC4BGmpm9eLb): protocol: expected piece hash; context canceled; EOF Aug 7 21:44:55.265766391 (12h4m54.855111378s) [noise=true , address=104.36.82.23:4466 , piece_id=5J5X2LSP6GCX2PNRIDZYFJPISWSX3YSRID5M2WU6Y5Z6LWRHLC4Q, seg_pos={0 0}, upload=12552]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rkNnk9RUhKBJ3EsakqMRb3wYwrhRLhB4ZYbrdfKPDk8erRJpP): protocol: write tcp 188.166.225.170:49724->104.36.82.23:4466: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 21:44:55.265824172 (12h4m54.855169159s) [noise=true , address=104.36.82.62:4420 , piece_id=UOO36PYVE5WFRRZLGYLJ6I6SV4EO7W2ZH2TIXJ2OIZZMY3LTHR2A, seg_pos={0 0}, upload=12552]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SWVQwL5PxDLaSXADwTrAYS2hVcNBuYAA6ReWuRM3UZa57QyYi): protocol: read tcp 188.166.225.170:33092->104.36.82.62:4420: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 21:44:55.265833430 (12h4m54.855178417s) [seg_pos={0 0}, upload=12552]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 21:44:55.265888796 (12h4m54.855233783s) [upload=12552]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 21:44:55.265892303 (12h4m54.85523728s) [upload=12552]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:48:46.088275738 (12h8m45.677620725s) [upload=12552]: Done uploading segments... Aug 7 21:48:46.088301307 (12h8m45.677646284s) [upload=12552]: Upload finished. err: metaclient: context deadline exceeded Duration: 6.842007842s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 26 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXX.XX..X...............................] duration: 2.109794624s FAIL (2025-08-07 22:10:04 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 6664 (11h58m33.293011651s) [upload=12450]: Starting upload Aug 7 21:38:33.703724295 (11h58m33.293069282s) [upload=12450]: Done starting upload Aug 7 21:38:33.703770663 (11h58m33.29311565s) [upload=12450]: Uploading segments... Aug 7 21:38:33.703817212 (11h58m33.293162199s) [upload=12450]: Got next segment. Inline: true Aug 7 21:38:33.703820899 (11h58m33.293165886s) [upload=12450]: Waiting for error group managing segments... Aug 7 21:38:33.703853411 (11h58m33.293198388s) [upload=12450]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:38:33.703857378 (11h58m33.293202365s) [upload=12450]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:38:33.703859744 (11h58m33.293204731s) [upload=12450]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:39:33.059235349 (11h59m32.648580326s) [upload=12450]: Done uploading segments... Aug 7 21:39:33.059253302 (11h59m32.648598289s) [upload=12450]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:39:33.956833499 (11h59m33.546178486s) [upload=12464]: Starting upload Aug 7 21:39:33.956907240 (11h59m33.546252227s) [upload=12464]: Done starting upload Aug 7 21:39:33.956967195 (11h59m33.546312182s) [upload=12464]: Uploading segments... Aug 7 21:39:33.957011540 (11h59m33.546356527s) [upload=12464]: Got next segment. Inline: true Aug 7 21:39:33.957016229 (11h59m33.546361226s) [upload=12464]: Waiting for error group managing segments... Aug 7 21:39:33.957053820 (11h59m33.546398807s) [upload=12464]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:39:33.957058250 (11h59m33.546403237s) [upload=12464]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:39:33.957061736 (11h59m33.546406723s) [upload=12464]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:40:33.302006486 (12h0m32.891351473s) [upload=12464]: Done uploading segments... Aug 7 21:40:33.302025733 (12h0m32.89137071s) [upload=12464]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:40:34.211725785 (12h0m33.801070772s) [upload=12482]: Starting upload Aug 7 21:40:34.211806970 (12h0m33.801151957s) [upload=12482]: Done starting upload Aug 7 21:40:34.211965303 (12h0m33.8013103s) [upload=12482]: Uploading segments... Aug 7 21:40:34.212066617 (12h0m33.801411614s) [upload=12482]: Got next segment. Inline: true Aug 7 21:40:34.212072819 (12h0m33.801417806s) [upload=12482]: Waiting for error group managing segments... Aug 7 21:40:34.212126081 (12h0m33.801471068s) [upload=12482]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:40:34.212132573 (12h0m33.80147756s) [upload=12482]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:40:34.212136120 (12h0m33.801481107s) [upload=12482]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:41:33.557664583 (12h1m33.14700957s) [upload=12482]: Done uploading segments... Aug 7 21:41:33.557694410 (12h1m33.147039387s) [upload=12482]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:41:34.468736652 (12h1m34.058081639s) [upload=12501]: Starting upload Aug 7 21:41:34.468803340 (12h1m34.058148327s) [upload=12501]: Done starting upload Aug 7 21:41:34.468862643 (12h1m34.05820763s) [upload=12501]: Uploading segments... Aug 7 21:41:34.468901758 (12h1m34.058246735s) [upload=12501]: Got next segment. Inline: true Aug 7 21:41:34.468905907 (12h1m34.058250884s) [upload=12501]: Waiting for error group managing segments... Aug 7 21:41:34.468943147 (12h1m34.058288134s) [upload=12501]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:41:34.468947105 (12h1m34.058292092s) [upload=12501]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:41:34.468949690 (12h1m34.058294677s) [upload=12501]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:42:33.814395640 (12h2m33.403740627s) [upload=12501]: Done uploading segments... Aug 7 21:42:33.814423744 (12h2m33.403768731s) [upload=12501]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:42:46.083099596 (12h2m45.672444573s) [upload=12522]: Starting upload Aug 7 21:42:46.083172004 (12h2m45.672516981s) [upload=12522]: Done starting upload Aug 7 21:42:46.083231107 (12h2m45.672576094s) [upload=12522]: Uploading segments... Aug 7 21:42:46.083339093 (12h2m45.67268408s) [upload=12522]: Got next segment. Inline: true Aug 7 21:42:46.083344093 (12h2m45.67268908s) [upload=12522]: Waiting for error group managing segments... Aug 7 21:42:46.083408827 (12h2m45.672753814s) [upload=12522]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:42:46.083414187 (12h2m45.672759174s) [upload=12522]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:42:46.083416732 (12h2m45.672761719s) [upload=12522]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:42:47.411975663 (12h2m47.00132066s) [upload=12522]: Done uploading segments... Aug 7 21:42:47.411996032 (12h2m47.001341019s) [upload=12522]: Upload finished. err: <nil> Aug 7 21:43:11.548517821 (12h3m11.137862808s) [upload=12529]: Starting upload Aug 7 21:43:11.548568578 (12h3m11.137913565s) [upload=12529]: Done starting upload Aug 7 21:43:11.548617301 (12h3m11.137962288s) [upload=12529]: Uploading segments... Aug 7 21:43:11.548646878 (12h3m11.137991865s) [upload=12529]: Got next segment. Inline: true Aug 7 21:43:11.548650374 (12h3m11.137995362s) [upload=12529]: Waiting for error group managing segments... Aug 7 21:43:11.548681634 (12h3m11.138026621s) [upload=12529]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:43:11.548685361 (12h3m11.138030338s) [upload=12529]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:43:11.548687726 (12h3m11.138032713s) [upload=12529]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:44:10.906282559 (12h4m10.495627546s) [upload=12529]: Done uploading segments... Aug 7 21:44:10.906303038 (12h4m10.495648015s) [upload=12529]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:44:11.778510580 (12h4m11.367855567s) [upload=12548]: Starting upload Aug 7 21:44:11.778570013 (12h4m11.367915s) [upload=12548]: Done starting upload Aug 7 21:44:11.778662139 (12h4m11.368007126s) [upload=12548]: Uploading segments... Aug 7 21:44:11.778800213 (12h4m11.36814519s) [upload=12548]: Got next segment. Inline: true Aug 7 21:44:11.778804892 (12h4m11.368149869s) [upload=12548]: Waiting for error group managing segments... Aug 7 21:44:11.778861480 (12h4m11.368206467s) [upload=12548]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:44:11.778866230 (12h4m11.368211207s) [upload=12548]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:44:11.778868745 (12h4m11.368213732s) [upload=12548]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:45:11.131015060 (12h5m10.720360047s) [upload=12548]: Done uploading segments... Aug 7 21:45:11.131033725 (12h5m10.720378712s) [upload=12548]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:45:11.998653609 (12h5m11.587998596s) [upload=12563]: Starting upload Aug 7 21:45:11.998732921 (12h5m11.588077908s) [upload=12563]: Done starting upload Aug 7 21:45:11.998811802 (12h5m11.588156789s) [upload=12563]: Uploading segments... Aug 7 21:45:11.998922443 (12h5m11.58826743s) [upload=12563]: Got next segment. Inline: true Aug 7 21:45:11.998926992 (12h5m11.588271979s) [upload=12563]: Waiting for error group managing segments... Aug 7 21:45:11.998979913 (12h5m11.5883249s) [upload=12563]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:45:11.998987267 (12h5m11.588332254s) [upload=12563]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:45:11.998990603 (12h5m11.58833559s) [upload=12563]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:45:14.137150454 (12h5m13.726495441s) [upload=12563]: Done uploading segments... Aug 7 21:45:14.137168529 (12h5m13.726513516s) [upload=12563]: Upload finished. err: <nil> Aug 7 21:45:53.851988781 (12h5m53.441333768s) [upload=12569]: Starting upload Aug 7 21:45:53.852043686 (12h5m53.441388673s) [upload=12569]: Done starting upload Aug 7 21:45:53.852088772 (12h5m53.441433759s) [upload=12569]: Uploading segments... Aug 7 21:45:53.852141613 (12h5m53.4414866s) [upload=12569]: Got next segment. Inline: true Aug 7 21:45:53.852146192 (12h5m53.441491169s) [upload=12569]: Waiting for error group managing segments... Aug 7 21:45:53.852182612 (12h5m53.441527599s) [upload=12569]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:45:53.852186920 (12h5m53.441531907s) [upload=12569]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:45:53.852189195 (12h5m53.441534182s) [upload=12569]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:45:54.127817304 (12h5m53.717162291s) [upload=12569]: Done uploading segments... Aug 7 21:45:54.127834497 (12h5m53.717179484s) [upload=12569]: Upload finished. err: <nil> Aug 7 21:46:54.936546775 (12h6m54.525891762s) [upload=12585]: Starting upload Aug 7 21:46:54.936595597 (12h6m54.525940585s) [upload=12585]: Done starting upload Aug 7 21:46:54.936645253 (12h6m54.52599024s) [upload=12585]: Uploading segments... Aug 7 21:46:54.936675961 (12h6m54.526020948s) [upload=12585]: Got next segment. Inline: true Aug 7 21:46:54.936680381 (12h6m54.526025358s) [upload=12585]: Waiting for error group managing segments... Aug 7 21:46:54.936716089 (12h6m54.526061076s) [upload=12585]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:46:54.936720797 (12h6m54.526065774s) [upload=12585]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:46:54.936723172 (12h6m54.526068159s) [upload=12585]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:47:54.288569811 (12h7m53.877914798s) [upload=12585]: Done uploading segments... Aug 7 21:47:54.288620218 (12h7m53.877965205s) [upload=12585]: Upload finished. err: metaclient: context deadline exceeded Duration: 2.109846783s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 25 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXX...X................................] duration: 1.885082932s FAIL (2025-08-07 22:10:03 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 1.885111568s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 23 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXX.X...X.................................] duration: 1.906660326s FAIL (2025-08-07 22:09:54 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== line exceeded Aug 7 21:37:22.572699237 (11h57m22.162044224s) [upload=12432]: Starting upload Aug 7 21:37:22.572782937 (11h57m22.162127924s) [upload=12432]: Done starting upload Aug 7 21:37:22.572842260 (11h57m22.162187247s) [upload=12432]: Uploading segments... Aug 7 21:37:22.572885313 (11h57m22.1622303s) [upload=12432]: Got next segment. Inline: true Aug 7 21:37:22.572889591 (11h57m22.162234578s) [upload=12432]: Waiting for error group managing segments... Aug 7 21:37:22.572930138 (11h57m22.162275125s) [upload=12432]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:37:22.572934296 (11h57m22.162279283s) [upload=12432]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:37:22.572937121 (11h57m22.162282118s) [upload=12432]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:37:51.933954276 (11h57m51.523299253s) [upload=12432]: Done uploading segments... Aug 7 21:37:51.933970927 (11h57m51.523315914s) [upload=12432]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:38:28.745940745 (11h58m28.335285732s) [upload=12448]: Starting upload Aug 7 21:38:28.745997504 (11h58m28.335342491s) [upload=12448]: Done starting upload Aug 7 21:38:28.746040176 (11h58m28.335385163s) [upload=12448]: Uploading segments... Aug 7 21:38:28.746078458 (11h58m28.335423446s) [upload=12448]: Got next segment. Inline: true Aug 7 21:38:28.746081766 (11h58m28.335426743s) [upload=12448]: Waiting for error group managing segments... Aug 7 21:38:28.746111923 (11h58m28.33545691s) [upload=12448]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:38:28.746115540 (11h58m28.335460527s) [upload=12448]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:38:28.746118205 (11h58m28.335463192s) [upload=12448]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:38:58.105315693 (11h58m57.694660691s) [upload=12448]: Done uploading segments... Aug 7 21:38:58.105342465 (11h58m57.694687452s) [upload=12448]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:39:27.137238449 (11h59m26.726583436s) [upload=12462]: Starting upload Aug 7 21:39:27.137294907 (11h59m26.726639894s) [upload=12462]: Done starting upload Aug 7 21:39:27.137337368 (11h59m26.726682355s) [upload=12462]: Uploading segments... Aug 7 21:39:27.137366043 (11h59m26.72671103s) [upload=12462]: Got next segment. Inline: true Aug 7 21:39:27.137370061 (11h59m26.726715048s) [upload=12462]: Waiting for error group managing segments... Aug 7 21:39:27.137401901 (11h59m26.726746879s) [upload=12462]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:39:27.137405518 (11h59m26.726750495s) [upload=12462]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:39:27.137407813 (11h59m26.72675279s) [upload=12462]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:39:56.495186447 (11h59m56.084531434s) [upload=12462]: Done uploading segments... Aug 7 21:39:56.495206415 (11h59m56.084551402s) [upload=12462]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:40:01.914615953 (12h0m1.50396094s) [upload=12475]: Starting upload Aug 7 21:40:01.914684295 (12h0m1.504029282s) [upload=12475]: Done starting upload Aug 7 21:40:01.914741163 (12h0m1.50408616s) [upload=12475]: Uploading segments... Aug 7 21:40:01.914783103 (12h0m1.50412809s) [upload=12475]: Got next segment. Inline: true Aug 7 21:40:01.914788143 (12h0m1.504133131s) [upload=12475]: Waiting for error group managing segments... Aug 7 21:40:01.914833950 (12h0m1.504178937s) [upload=12475]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:40:01.914840073 (12h0m1.50418507s) [upload=12475]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:40:01.914846014 (12h0m1.504191011s) [upload=12475]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:40:31.268196986 (12h0m30.857542023s) [upload=12475]: Done uploading segments... Aug 7 21:40:31.268223536 (12h0m30.857568533s) [upload=12475]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:40:46.248438399 (12h0m45.837783386s) [upload=12487]: Starting upload Aug 7 21:40:46.248491410 (12h0m45.837836387s) [upload=12487]: Done starting upload Aug 7 21:40:46.248540414 (12h0m45.837885401s) [upload=12487]: Uploading segments... Aug 7 21:40:46.248572345 (12h0m45.837917332s) [upload=12487]: Got next segment. Inline: true Aug 7 21:40:46.248576133 (12h0m45.83792112s) [upload=12487]: Waiting for error group managing segments... Aug 7 21:40:46.248613013 (12h0m45.837958s) [upload=12487]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:40:46.248616850 (12h0m45.837961837s) [upload=12487]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:40:46.248619255 (12h0m45.837964242s) [upload=12487]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:40:55.261300379 (12h0m54.850645366s) [upload=12487]: Done uploading segments... Aug 7 21:40:55.261318514 (12h0m54.850663501s) [upload=12487]: Upload finished. err: <nil> Aug 7 21:42:00.373248983 (12h1m59.962594s) [upload=12509]: Starting upload Aug 7 21:42:00.373363813 (12h1m59.96270884s) [upload=12509]: Done starting upload Aug 7 21:42:00.373438045 (12h1m59.962783062s) [upload=12509]: Uploading segments... Aug 7 21:42:00.373501075 (12h1m59.962846083s) [upload=12509]: Got next segment. Inline: true Aug 7 21:42:00.373507809 (12h1m59.962852816s) [upload=12509]: Waiting for error group managing segments... Aug 7 21:42:00.373577642 (12h1m59.962922669s) [upload=12509]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:42:00.373584325 (12h1m59.962929342s) [upload=12509]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:42:00.373590016 (12h1m59.962935053s) [upload=12509]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:42:29.724044552 (12h2m29.313389539s) [upload=12509]: Done uploading segments... Aug 7 21:42:29.724072415 (12h2m29.313417392s) [upload=12509]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:42:30.611135457 (12h2m30.200480444s) [upload=12516]: Starting upload Aug 7 21:42:30.611201894 (12h2m30.200546881s) [upload=12516]: Done starting upload Aug 7 21:42:30.611252732 (12h2m30.200597719s) [upload=12516]: Uploading segments... Aug 7 21:42:30.611338045 (12h2m30.200683032s) [upload=12516]: Got next segment. Inline: true Aug 7 21:42:30.611344076 (12h2m30.200689063s) [upload=12516]: Waiting for error group managing segments... Aug 7 21:42:30.611400083 (12h2m30.20074507s) [upload=12516]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:42:30.611404922 (12h2m30.200749909s) [upload=12516]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:42:30.611408178 (12h2m30.200753166s) [upload=12516]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:42:46.271341803 (12h2m45.860686781s) [upload=12516]: Done uploading segments... Aug 7 21:42:46.271358525 (12h2m45.860703512s) [upload=12516]: Upload finished. err: <nil> Aug 7 21:43:05.160958738 (12h3m4.750303725s) [upload=12528]: Starting upload Aug 7 21:43:05.161064639 (12h3m4.750409626s) [upload=12528]: Done starting upload Aug 7 21:43:05.161177936 (12h3m4.750522953s) [upload=12528]: Uploading segments... Aug 7 21:43:05.161261357 (12h3m4.750606344s) [upload=12528]: Got next segment. Inline: true Aug 7 21:43:05.161266376 (12h3m4.750611353s) [upload=12528]: Waiting for error group managing segments... Aug 7 21:43:05.161306081 (12h3m4.750651069s) [upload=12528]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:43:05.161310580 (12h3m4.750655567s) [upload=12528]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:43:05.161312864 (12h3m4.750657841s) [upload=12528]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:43:17.899724589 (12h3m17.489069576s) [upload=12528]: Done uploading segments... Aug 7 21:43:17.899747483 (12h3m17.48909251s) [upload=12528]: Upload finished. err: <nil> Aug 7 21:43:34.894582431 (12h3m34.483927418s) [upload=12537]: Starting upload Aug 7 21:43:34.894647576 (12h3m34.483992564s) [upload=12537]: Done starting upload Aug 7 21:43:34.894699496 (12h3m34.484044483s) [upload=12537]: Uploading segments... Aug 7 21:43:34.894732940 (12h3m34.484077927s) [upload=12537]: Got next segment. Inline: true Aug 7 21:43:34.894736967 (12h3m34.484081954s) [upload=12537]: Waiting for error group managing segments... Aug 7 21:43:34.894770662 (12h3m34.484115649s) [upload=12537]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:43:34.894775050 (12h3m34.484120027s) [upload=12537]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:43:34.894777795 (12h3m34.484122782s) [upload=12537]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:43:35.191263901 (12h3m34.780608888s) [upload=12537]: Done uploading segments... Aug 7 21:43:35.191280001 (12h3m34.780624978s) [upload=12537]: Upload finished. err: <nil> Aug 7 21:44:27.756622626 (12h4m27.345967613s) [upload=12553]: Starting upload Aug 7 21:44:27.756679294 (12h4m27.346024281s) [upload=12553]: Done starting upload Aug 7 21:44:27.756721205 (12h4m27.346066192s) [upload=12553]: Uploading segments... Aug 7 21:44:27.756751423 (12h4m27.34609641s) [upload=12553]: Got next segment. Inline: true Aug 7 21:44:27.756754779 (12h4m27.346099766s) [upload=12553]: Waiting for error group managing segments... Aug 7 21:44:27.756785678 (12h4m27.346130655s) [upload=12553]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:44:27.756789615 (12h4m27.346134602s) [upload=12553]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:44:27.756791790 (12h4m27.346136777s) [upload=12553]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:44:57.108386877 (12h4m56.697731864s) [upload=12553]: Done uploading segments... Aug 7 21:44:57.108405372 (12h4m56.697750359s) [upload=12553]: Upload finished. err: metaclient: context deadline exceeded Duration: 1.906688981s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 26 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXX.X.X.X...............................] duration: 6.301560099s FAIL (2025-08-07 22:09:51 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== context canceled; ecclient: upload cut due to slow connection (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR): protocol: write tcp 188.166.225.170:46242->104.36.82.11:4115: use of closed network connection; write tcp 188.166.225.170:46242->104.36.82.11:4115: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46242->104.36.82.11:4115: use of closed network connection Aug 7 21:44:51.028127350 (12h4m50.617472327s) [seg_pos={0 0}, upload=12555]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 7 21:44:51.028226048 (12h4m50.617571035s) [upload=12555]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 21:44:51.028237751 (12h4m50.617582738s) [upload=12555]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:44:51.322864709 (12h4m50.912209697s) [upload=12555]: Done uploading segments... Aug 7 21:44:51.322882523 (12h4m50.9122275s) [upload=12555]: Upload finished. err: <nil> Aug 7 21:45:58.815791269 (12h5m58.405136256s) [upload=12571]: Starting upload Aug 7 21:45:58.815846996 (12h5m58.405191983s) [upload=12571]: Done starting upload Aug 7 21:45:58.815953540 (12h5m58.405298527s) [upload=12571]: Uploading segments... Aug 7 21:45:58.816065333 (12h5m58.405410321s) [upload=12571]: Got next segment. Inline: false Aug 7 21:45:58.816100661 (12h5m58.405445638s) [upload=12571]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:45:58.816106212 (12h5m58.405451199s) [upload=12571]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:45:59.064845605 (12h5m58.654190592s) [seg_pos={0 0}, upload=12571]: Begin upload segment... Aug 7 21:45:59.064885521 (12h5m58.654230508s) [seg_pos={0 0}, upload=12571]: Done begin upload segment. Aug 7 21:45:59.064910399 (12h5m58.654255376s) [noise=true , address=77.90.31.18:4807 , piece_id=63E5KXGYEJHGHYE4SKTNTMED5R4HSWIST4CR4EV6ZGON3YHV73AA, seg_pos={0 0}, upload=12571]: Uploading piece... Aug 7 21:45:59.064952560 (12h5m58.654297547s) [noise=true , address=45.91.116.12:4227 , piece_id=MDQGJ65RT4QEUFX6OFXPEAOFSTG2VXBRC23DXVVD2IV6V52UFF4Q, seg_pos={0 0}, upload=12571]: Uploading piece... Aug 7 21:45:59.065245920 (12h5m58.654590907s) [noise=true , address=45.91.116.8:4048 , piece_id=2LZRN76O3QS6FODVI52DIIMK7Z77QQBGIXYUAKZCSFI7WMGRTDUA, seg_pos={0 0}, upload=12571]: Uploading piece... Aug 7 21:45:59.065360070 (12h5m58.654705057s) [noise=true , address=45.91.116.9:4262 , piece_id=D4ABUUAX4MHYLFPR6LN2REDC6VRT2W7D4VZHHYM7GZ6HWOWMG2LA, seg_pos={0 0}, upload=12571]: Uploading piece... Aug 7 21:45:59.065517019 (12h5m58.654861996s) [noise=true , address=77.90.31.19:4843 , piece_id=S73TOGLK5HDCBE4BINZ72A3URDR6TUD43IVIFDY5V37EVCN6222A, seg_pos={0 0}, upload=12571]: Uploading piece... Aug 7 21:45:59.065615889 (12h5m58.654960876s) [noise=true , address=104.36.82.6:4875 , piece_id=W43EX62262JONZAGLUREOG3EGMICRBIVX5O445J3OQDQ2O475J7A, seg_pos={0 0}, upload=12571]: Uploading piece... Aug 7 21:45:59.065778009 (12h5m58.655122986s) [noise=true , address=104.36.82.11:4115 , piece_id=AH7EPSWFJFX7XDYUQOFCNXEHKQ5AMQX22NY6OFBMMHATCOVH4YJA, seg_pos={0 0}, upload=12571]: Uploading piece... Aug 7 21:45:59.065884293 (12h5m58.65522928s) [noise=true , address=45.91.116.8:4047 , piece_id=TEMT2SH64IMIPQ3GDE5XCF2R2XLY33HUKRFF6KUWEYH7CDVYD4YQ, seg_pos={0 0}, upload=12571]: Uploading piece... Aug 7 21:45:59.066024761 (12h5m58.655369748s) [noise=true , address=77.90.31.11:4554 , piece_id=LMSB62TMAJONFULH4DAOTPWQMGLUFRZYKVG4HNJ4AUBP7EBP5FFQ, seg_pos={0 0}, upload=12571]: Uploading piece... Aug 7 21:45:59.066243109 (12h5m58.655588096s) [noise=true , address=45.91.116.6:4973 , piece_id=7MZSPAEXC3Q7W7U3QMZUQRUI3OHPQI3XRAJW4OGU7UZ73FEI4LIQ, seg_pos={0 0}, upload=12571]: Uploading piece... Aug 7 21:45:59.066452850 (12h5m58.655797837s) [noise=true , address=104.36.82.6:4874 , piece_id=T3W6RCAW4YPEU42CESOSUSFSNPDBFXR74WZUYQ2AHJNBZV776MVQ, seg_pos={0 0}, upload=12571]: Uploading piece... Aug 7 21:45:59.066669685 (12h5m58.656014672s) [noise=true , address=45.91.116.6:4975 , piece_id=CI7SM56L5KPNQEV52PLIETEBQVH5CFXTKVQOQIZ73YOPEGYC3KOA, seg_pos={0 0}, upload=12571]: Uploading piece... Aug 7 21:45:59.066955041 (12h5m58.656300028s) [noise=true , address=104.36.82.7:4909 , piece_id=EKLBHF6P5BXX5VLSLL4QEOJXQXWM7MCOY67W5OON2UHRNMKOQFAA, seg_pos={0 0}, upload=12571]: Uploading piece... Aug 7 21:45:59.067149393 (12h5m58.65649438s) [noise=true , address=45.91.116.13:4154 , piece_id=LDSWQ62RRGHTT544S6463WLDKW3WFBWIFQGQDHLYWJJD2ESMYJNQ, seg_pos={0 0}, upload=12571]: Uploading piece... Aug 7 21:45:59.067334066 (12h5m58.656679053s) [noise=true , address=104.36.82.16:4294 , piece_id=KDCYWDEXSTKCHR23CAOKKTYUMRQS5Y4LH6KIJPQFF737WA5TST7Q, seg_pos={0 0}, upload=12571]: Uploading piece... Aug 7 21:45:59.067527215 (12h5m58.656872202s) [noise=true , address=77.90.31.15:4698 , piece_id=6L3IAVPA4PQGSMKOGSKWYG4NLM6TSA2YSIAM6S6MYSDUW26CP5CA, seg_pos={0 0}, upload=12571]: Uploading piece... Aug 7 21:46:00.855806043 (12h6m0.44515103s) [upload=12571]: Next returned nil segment Aug 7 21:46:00.855821061 (12h6m0.445166048s) [upload=12571]: Waiting for error group managing segments... Aug 7 21:46:01.180369497 (12h6m0.769714484s) [noise=true , address=77.90.31.15:4698 , piece_id=6L3IAVPA4PQGSMKOGSKWYG4NLM6TSA2YSIAM6S6MYSDUW26CP5CA, seg_pos={0 0}, upload=12571]: Done uploading piece. err: <nil> Aug 7 21:46:01.200492297 (12h6m0.789837284s) [noise=true , address=45.91.116.6:4975 , piece_id=CI7SM56L5KPNQEV52PLIETEBQVH5CFXTKVQOQIZ73YOPEGYC3KOA, seg_pos={0 0}, upload=12571]: Done uploading piece. err: <nil> Aug 7 21:46:01.210262691 (12h6m0.799607678s) [noise=true , address=104.36.82.16:4294 , piece_id=KDCYWDEXSTKCHR23CAOKKTYUMRQS5Y4LH6KIJPQFF737WA5TST7Q, seg_pos={0 0}, upload=12571]: Done uploading piece. err: <nil> Aug 7 21:46:01.220625478 (12h6m0.809970465s) [noise=true , address=104.36.82.7:4909 , piece_id=EKLBHF6P5BXX5VLSLL4QEOJXQXWM7MCOY67W5OON2UHRNMKOQFAA, seg_pos={0 0}, upload=12571]: Done uploading piece. err: <nil> Aug 7 21:46:01.304447281 (12h6m0.893792308s) [noise=true , address=104.36.82.11:4115 , piece_id=AH7EPSWFJFX7XDYUQOFCNXEHKQ5AMQX22NY6OFBMMHATCOVH4YJA, seg_pos={0 0}, upload=12571]: Done uploading piece. err: <nil> Aug 7 21:46:01.374462398 (12h6m0.963807385s) [noise=true , address=45.91.116.8:4047 , piece_id=TEMT2SH64IMIPQ3GDE5XCF2R2XLY33HUKRFF6KUWEYH7CDVYD4YQ, seg_pos={0 0}, upload=12571]: Done uploading piece. err: <nil> Aug 7 21:46:01.378351098 (12h6m0.967696085s) [noise=true , address=104.36.82.6:4875 , piece_id=W43EX62262JONZAGLUREOG3EGMICRBIVX5O445J3OQDQ2O475J7A, seg_pos={0 0}, upload=12571]: Done uploading piece. err: <nil> Aug 7 21:46:01.395257154 (12h6m0.984602141s) [noise=true , address=45.91.116.6:4973 , piece_id=7MZSPAEXC3Q7W7U3QMZUQRUI3OHPQI3XRAJW4OGU7UZ73FEI4LIQ, seg_pos={0 0}, upload=12571]: Done uploading piece. err: <nil> Aug 7 21:46:01.404312138 (12h6m0.993657125s) [noise=true , address=77.90.31.19:4843 , piece_id=S73TOGLK5HDCBE4BINZ72A3URDR6TUD43IVIFDY5V37EVCN6222A, seg_pos={0 0}, upload=12571]: Done uploading piece. err: <nil> Aug 7 21:46:01.409522908 (12h6m0.998867885s) [noise=true , address=45.91.116.13:4154 , piece_id=LDSWQ62RRGHTT544S6463WLDKW3WFBWIFQGQDHLYWJJD2ESMYJNQ, seg_pos={0 0}, upload=12571]: Done uploading piece. err: <nil> Aug 7 21:46:01.416315134 (12h6m1.005660121s) [noise=true , address=45.91.116.12:4227 , piece_id=MDQGJ65RT4QEUFX6OFXPEAOFSTG2VXBRC23DXVVD2IV6V52UFF4Q, seg_pos={0 0}, upload=12571]: Done uploading piece. err: <nil> Aug 7 21:46:01.421432725 (12h6m1.010777712s) [noise=true , address=45.91.116.8:4048 , piece_id=2LZRN76O3QS6FODVI52DIIMK7Z77QQBGIXYUAKZCSFI7WMGRTDUA, seg_pos={0 0}, upload=12571]: Done uploading piece. err: <nil> Aug 7 21:46:01.439709112 (12h6m1.029054099s) [noise=true , address=45.91.116.9:4262 , piece_id=D4ABUUAX4MHYLFPR6LN2REDC6VRT2W7D4VZHHYM7GZ6HWOWMG2LA, seg_pos={0 0}, upload=12571]: Done uploading piece. err: <nil> Aug 7 21:46:01.439735472 (12h6m1.029080459s) [seg_pos={0 0}, upload=12571]: Segment reached optimal threshold of 13 pieces. Aug 7 21:46:01.440013234 (12h6m1.029358221s) [noise=true , address=77.90.31.11:4554 , piece_id=LMSB62TMAJONFULH4DAOTPWQMGLUFRZYKVG4HNJ4AUBP7EBP5FFQ, seg_pos={0 0}, upload=12571]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek): protocol: expected piece hash; context canceled; EOF Aug 7 21:46:01.440020518 (12h6m1.029365505s) [noise=true , address=104.36.82.6:4874 , piece_id=T3W6RCAW4YPEU42CESOSUSFSNPDBFXR74WZUYQ2AHJNBZV776MVQ, seg_pos={0 0}, upload=12571]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71): protocol: write tcp 188.166.225.170:54068->104.36.82.6:4874: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 21:46:01.440109398 (12h6m1.029454385s) [noise=true , address=77.90.31.18:4807 , piece_id=63E5KXGYEJHGHYE4SKTNTMED5R4HSWIST4CR4EV6ZGON3YHV73AA, seg_pos={0 0}, upload=12571]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS): protocol: expected piece hash; context canceled; EOF Aug 7 21:46:01.440122312 (12h6m1.029467299s) [seg_pos={0 0}, upload=12571]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 7 21:46:01.440193058 (12h6m1.029538045s) [upload=12571]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 21:46:01.440197016 (12h6m1.029542003s) [upload=12571]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:48:14.557737130 (12h8m14.147082107s) [upload=12571]: Done uploading segments... Aug 7 21:48:14.557758951 (12h8m14.147103938s) [upload=12571]: Upload finished. err: metaclient: context deadline exceeded Duration: 6.301650994s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 4 out of 60 is failed (threshold=3) [.X.............X..X....................X....................] duration: 7.705890126s FAIL (2025-08-07 22:09:51 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== 57 (12h15m32.758009944s) [noise=true , address=104.36.82.54:4166 , piece_id=5SBUSGFVYPNFGILW6OTNLGJQG7CNI7GYYEJQJMITPKFZ2MAK5QRQ, seg_pos={0 0}, upload=12734]: Uploading piece... Aug 7 21:55:33.168945643 (12h15m32.75829063s) [noise=true , address=104.36.82.12:4179 , piece_id=KFVQX32X6I7QBU552L4NSPZG3JTVT24YESONPWGM6UCTR4ZWBUKQ, seg_pos={0 0}, upload=12734]: Uploading piece... Aug 7 21:55:33.168966093 (12h15m32.75831109s) [noise=true , address=104.36.82.43:4850 , piece_id=JH2ASD4NQAQMOBOZMDGFKVPPOS2P26QIFSYDNHYZN24RP52E4XLA, seg_pos={0 0}, upload=12734]: Uploading piece... Aug 7 21:55:33.169201393 (12h15m32.75854638s) [noise=true , address=104.36.82.36:4633 , piece_id=6RL3INCLYJCEH53JOFJDL2MMEXGQ2G4T33PKZPJ3W2FG5QMCFLCQ, seg_pos={0 0}, upload=12734]: Uploading piece... Aug 7 21:55:33.169453466 (12h15m32.758798453s) [noise=true , address=104.36.82.47:4520 , piece_id=K75KQHBYH7QTZDCFM7ZW55EUO2W3UN2NEIKUMNCMQWHYYNOWLL3A, seg_pos={0 0}, upload=12734]: Uploading piece... Aug 7 21:55:33.169588975 (12h15m32.758933962s) [noise=true , address=104.36.82.9:4014 , piece_id=D3NLTEDBRB4M7GIMPYCDNQ66HYLAZGSSBRM7SJWU45LPTCRMYMCQ, seg_pos={0 0}, upload=12734]: Uploading piece... Aug 7 21:55:33.170078472 (12h15m32.759423459s) [noise=true , address=104.36.82.39:4739 , piece_id=QGKQ3NJO7OTN5O2QQU43KYJXARJW6JQMSCONEJR35AEBBNRORJCA, seg_pos={0 0}, upload=12734]: Uploading piece... Aug 7 21:55:33.170313160 (12h15m32.759658147s) [noise=true , address=104.36.82.62:4429 , piece_id=RCPKZHWWS56QSEVG6L25B3FMQOU5ZES3BNAEYIUYQ3PLQV3FDMVQ, seg_pos={0 0}, upload=12734]: Uploading piece... Aug 7 21:55:33.170515587 (12h15m32.759860574s) [noise=true , address=79.127.201.215:4353 , piece_id=II3JPNVAATS6APC4JGPPOJRHQRMHS5EYGLLEEAMTOPHPEGRQUF7A, seg_pos={0 0}, upload=12734]: Uploading piece... Aug 7 21:55:33.170753382 (12h15m32.760098359s) [noise=true , address=45.91.116.8:4065 , piece_id=OEVAVKCHFYMJMGBKUKF5F34O3NH47DG45QJHHNWMOD7TDRMCQLAQ, seg_pos={0 0}, upload=12734]: Uploading piece... Aug 7 21:55:33.170965789 (12h15m32.760310776s) [noise=true , address=77.90.31.14:4678 , piece_id=7WZVWEAFKI4ZEIHTZOJNMAQWLBVPPLB7U4JNYZQNXCVRC2S4BUBA, seg_pos={0 0}, upload=12734]: Uploading piece... Aug 7 21:55:33.171167525 (12h15m32.760512512s) [noise=true , address=104.36.82.45:4926 , piece_id=S3EH75AGZ3QVGR6RM3RI3SCHXZJPE7LORX2QQA5SCEADV3RDYM6A, seg_pos={0 0}, upload=12734]: Uploading piece... Aug 7 21:55:33.171354372 (12h15m32.760699359s) [noise=true , address=64.49.13.154:4935 , piece_id=O6TZCBWBHVESLRX76EFTUT4U4OIKFFVXO5UKLMWAJA2BDTHIHN2A, seg_pos={0 0}, upload=12734]: Uploading piece... Aug 7 21:55:33.171604611 (12h15m32.760949598s) [noise=true , address=104.36.82.44:4894 , piece_id=WCLKE2VTORDUBR5II5IQ5OTYMNHCMP2OYAIBHG3GV5P5U3FCO7HQ, seg_pos={0 0}, upload=12734]: Uploading piece... Aug 7 21:55:33.171788884 (12h15m32.761133861s) [noise=true , address=104.36.82.37:4674 , piece_id=AN3RCVXXXNEJUD33D4W47FDFYJQ7X5ZK7H4BJD6NR4PJOJSCVV6A, seg_pos={0 0}, upload=12734]: Uploading piece... Aug 7 21:55:33.181382449 (12h15m32.770727426s) [noise=true , address=104.36.82.23:4474 , piece_id=WM2SVCDZEFAN6VXQ2PWKBZGUCHIHNJ52RNKJKUNBMJ3AHFXUT3PA, seg_pos={0 0}, upload=12734]: Uploading piece... Aug 7 21:55:34.281100029 (12h15m33.870445006s) [upload=12734]: Next returned nil segment Aug 7 21:55:34.281113294 (12h15m33.870458281s) [upload=12734]: Waiting for error group managing segments... Aug 7 21:55:34.581166654 (12h15m34.170511631s) [noise=true , address=64.49.13.154:4935 , piece_id=O6TZCBWBHVESLRX76EFTUT4U4OIKFFVXO5UKLMWAJA2BDTHIHN2A, seg_pos={0 0}, upload=12734]: Done uploading piece. err: <nil> Aug 7 21:55:35.075145374 (12h15m34.664490361s) [noise=true , address=79.127.201.215:4353 , piece_id=II3JPNVAATS6APC4JGPPOJRHQRMHS5EYGLLEEAMTOPHPEGRQUF7A, seg_pos={0 0}, upload=12734]: Done uploading piece. err: <nil> Aug 7 21:55:35.117523204 (12h15m34.706868191s) [noise=true , address=77.90.31.14:4678 , piece_id=7WZVWEAFKI4ZEIHTZOJNMAQWLBVPPLB7U4JNYZQNXCVRC2S4BUBA, seg_pos={0 0}, upload=12734]: Done uploading piece. err: <nil> Aug 7 21:55:35.189259324 (12h15m34.778604311s) [noise=true , address=104.36.82.9:4014 , piece_id=D3NLTEDBRB4M7GIMPYCDNQ66HYLAZGSSBRM7SJWU45LPTCRMYMCQ, seg_pos={0 0}, upload=12734]: Done uploading piece. err: <nil> Aug 7 21:55:35.208589096 (12h15m34.797934083s) [noise=true , address=104.36.82.23:4474 , piece_id=WM2SVCDZEFAN6VXQ2PWKBZGUCHIHNJ52RNKJKUNBMJ3AHFXUT3PA, seg_pos={0 0}, upload=12734]: Done uploading piece. err: <nil> Aug 7 21:55:35.240233681 (12h15m34.829578668s) [noise=true , address=104.36.82.58:4265 , piece_id=XMDZZK372B4DNLPJRKD2BM7OI7JNQSZ6ABLSXNPZEH2WT73AT7TA, seg_pos={0 0}, upload=12734]: Done uploading piece. err: <nil> Aug 7 21:55:35.249509658 (12h15m34.838854645s) [noise=true , address=104.36.82.39:4739 , piece_id=QGKQ3NJO7OTN5O2QQU43KYJXARJW6JQMSCONEJR35AEBBNRORJCA, seg_pos={0 0}, upload=12734]: Done uploading piece. err: <nil> Aug 7 21:55:35.254602933 (12h15m34.84394792s) [noise=true , address=104.36.82.12:4179 , piece_id=KFVQX32X6I7QBU552L4NSPZG3JTVT24YESONPWGM6UCTR4ZWBUKQ, seg_pos={0 0}, upload=12734]: Done uploading piece. err: <nil> Aug 7 21:55:35.255644015 (12h15m34.844989002s) [noise=true , address=104.36.82.59:4322 , piece_id=O6L7GOHWQOA6VAMNJATA232B7BEQS66XKSZXUAQLX3KYNQXTMUCQ, seg_pos={0 0}, upload=12734]: Done uploading piece. err: <nil> Aug 7 21:55:35.260038844 (12h15m34.849383831s) [noise=true , address=45.91.116.8:4065 , piece_id=OEVAVKCHFYMJMGBKUKF5F34O3NH47DG45QJHHNWMOD7TDRMCQLAQ, seg_pos={0 0}, upload=12734]: Done uploading piece. err: <nil> Aug 7 21:55:36.160031235 (12h15m35.749376222s) [noise=true , address=104.36.82.43:4850 , piece_id=JH2ASD4NQAQMOBOZMDGFKVPPOS2P26QIFSYDNHYZN24RP52E4XLA, seg_pos={0 0}, upload=12734]: Done uploading piece. err: <nil> Aug 7 21:55:36.204255979 (12h15m35.793600956s) [noise=true , address=104.36.82.62:4429 , piece_id=RCPKZHWWS56QSEVG6L25B3FMQOU5ZES3BNAEYIUYQ3PLQV3FDMVQ, seg_pos={0 0}, upload=12734]: Done uploading piece. err: <nil> Aug 7 21:55:36.608134744 (12h15m36.197479731s) [noise=true , address=104.36.82.37:4674 , piece_id=AN3RCVXXXNEJUD33D4W47FDFYJQ7X5ZK7H4BJD6NR4PJOJSCVV6A, seg_pos={0 0}, upload=12734]: Done uploading piece. err: <nil> Aug 7 21:55:36.985735083 (12h15m36.57508007s) [noise=true , address=104.36.82.45:4926 , piece_id=S3EH75AGZ3QVGR6RM3RI3SCHXZJPE7LORX2QQA5SCEADV3RDYM6A, seg_pos={0 0}, upload=12734]: Done uploading piece. err: <nil> Aug 7 21:55:36.985758668 (12h15m36.575103655s) [seg_pos={0 0}, upload=12734]: Segment reached optimal threshold of 14 pieces. Aug 7 21:55:36.985891332 (12h15m36.575236319s) [noise=true , address=104.36.82.36:4633 , piece_id=6RL3INCLYJCEH53JOFJDL2MMEXGQ2G4T33PKZPJ3W2FG5QMCFLCQ, seg_pos={0 0}, upload=12734]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12dvBPvb9YEtM4NieVCXL71Pak5zwiMg14MSXH4RuLZts4dByG8): protocol: expected piece hash; context canceled; EOF Aug 7 21:55:36.986091084 (12h15m36.575436071s) [noise=true , address=104.36.82.47:4520 , piece_id=K75KQHBYH7QTZDCFM7ZW55EUO2W3UN2NEIKUMNCMQWHYYNOWLL3A, seg_pos={0 0}, upload=12734]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12eNRsboM5zFiyBVPLuQfgb5QSAc5bZFizK9KJHudNd9nstVGyy): protocol: write tcp 188.166.225.170:35740->104.36.82.47:4520: use of closed network connection; write tcp 188.166.225.170:35740->104.36.82.47:4520: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35740->104.36.82.47:4520: use of closed network connection Aug 7 21:55:36.986139326 (12h15m36.575484333s) [noise=true , address=104.36.82.54:4166 , piece_id=5SBUSGFVYPNFGILW6OTNLGJQG7CNI7GYYEJQJMITPKFZ2MAK5QRQ, seg_pos={0 0}, upload=12734]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BV7mWGAwKMXzmZBc3qAQFUXYyWtgQC7QEfG54YJgyJ4U3tFiv): protocol: expected piece hash; context canceled; EOF Aug 7 21:55:36.986268794 (12h15m36.575613791s) [noise=true , address=104.36.82.44:4894 , piece_id=WCLKE2VTORDUBR5II5IQ5OTYMNHCMP2OYAIBHG3GV5P5U3FCO7HQ, seg_pos={0 0}, upload=12734]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KWNyruYhmSJVvgUF7fBpGf4H7eYmtAeEXToYjd2dcEPcj1E44): protocol: read tcp 188.166.225.170:34542->104.36.82.44:4894: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 21:55:36.986293281 (12h15m36.575638268s) [seg_pos={0 0}, upload=12734]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 21:55:36.986385397 (12h15m36.575730394s) [upload=12734]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 21:55:36.986391038 (12h15m36.575736025s) [upload=12734]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:55:37.437633211 (12h15m37.026978238s) [upload=12734]: Done uploading segments... Aug 7 21:55:37.437656937 (12h15m37.027001924s) [upload=12734]: Upload finished. err: <nil> Aug 7 21:55:48.800457325 (12h15m48.389802332s) [upload=12741]: Starting upload Aug 7 21:55:48.800589529 (12h15m48.389934526s) [upload=12741]: Done starting upload Aug 7 21:55:48.800688879 (12h15m48.390033866s) [upload=12741]: Uploading segments... Aug 7 21:55:48.801167825 (12h15m48.390512802s) [upload=12741]: Got next segment. Inline: false Aug 7 21:55:48.801200477 (12h15m48.390545464s) [upload=12741]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:55:48.801204775 (12h15m48.390549762s) [upload=12741]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:55:49.015107648 (12h15m48.604452635s) [upload=12741]: Next segment err: metaclient: Too Many Requests Aug 7 21:55:49.015124520 (12h15m48.604469507s) [upload=12741]: Done uploading segments... Aug 7 21:55:49.015136393 (12h15m48.60448138s) [upload=12741]: Upload finished. err: metaclient: Too Many Requests Duration: 7.70591361s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway) antiblip: 3 out of 60 is failed (threshold=3) [...X.X...X..................................................] duration: 735.155386ms remote: "tcp:45.38.24.2:443" target: https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410 FAIL (2025-08-07 22:09:48 UTC) orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": context deadline exceeded Duration: 735.185694ms edge gateway vhost us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway)

checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway)