Orbiter: EU/US/AP/EDGE | Code | Config | Playbook | Escalation Procedure
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)antiblip:
1 out of 60 is failed (threshold=3) [...............X............................................] duration:
645.67893ms target:
https://us1.storj.io/login
FAIL (2025-08-07 23:14:29 UTC)
orbiter warning: Get "https://us1.storj.io/login": dial tcp 34.150.199.48:443: connect: connection refused
Duration: 645.718376ms
core
satellite
us1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)antiblip:
1 out of 60 is failed (threshold=3) [.........................................................X..] duration:
1.864329085s
FAIL (2025-08-07 23:14:23 UTC)
orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== segments... Aug 7 23:06:49.229716945 (13h26m48.819061932s) [upload=13950]: Upload finished. err: <nil> Aug 7 23:07:42.104963855 (13h27m41.694308842s) [upload=13968]: Starting upload Aug 7 23:07:42.105031205 (13h27m41.694376192s) [upload=13968]: Done starting upload Aug 7 23:07:42.105104063 (13h27m41.69444905s) [upload=13968]: Uploading segments... Aug 7 23:07:42.105139652 (13h27m41.694484639s) [upload=13968]: Got next segment. Inline: true Aug 7 23:07:42.105143259 (13h27m41.694488246s) [upload=13968]: Waiting for error group managing segments... Aug 7 23:07:42.105190129 (13h27m41.694535126s) [upload=13968]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 23:07:42.105201120 (13h27m41.694546127s) [upload=13968]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 23:07:42.105207281 (13h27m41.694552278s) [upload=13968]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 23:07:42.417069033 (13h27m42.00641402s) [upload=13968]: Done uploading segments... Aug 7 23:07:42.417104130 (13h27m42.006449117s) [upload=13968]: Upload finished. err: <nil> Aug 7 23:08:34.230557646 (13h28m33.819902633s) [upload=13982]: Starting upload Aug 7 23:08:34.230604215 (13h28m33.819949193s) [upload=13982]: Done starting upload Aug 7 23:08:34.230655173 (13h28m33.82000016s) [upload=13982]: Uploading segments... Aug 7 23:08:34.230683928 (13h28m33.820028905s) [upload=13982]: Got next segment. Inline: true Aug 7 23:08:34.230687385 (13h28m33.820032362s) [upload=13982]: Waiting for error group managing segments... Aug 7 23:08:34.230719175 (13h28m33.820064162s) [upload=13982]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 23:08:34.230722832 (13h28m33.820067819s) [upload=13982]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 23:08:34.230725046 (13h28m33.820070033s) [upload=13982]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 23:08:34.528494026 (13h28m34.117839013s) [upload=13982]: Done uploading segments... Aug 7 23:08:34.528512322 (13h28m34.117857309s) [upload=13982]: Upload finished. err: <nil> Aug 7 23:09:05.508570966 (13h29m5.097915953s) [upload=13989]: Starting upload Aug 7 23:09:05.508638426 (13h29m5.097983423s) [upload=13989]: Done starting upload Aug 7 23:09:05.508701747 (13h29m5.098046764s) [upload=13989]: Uploading segments... Aug 7 23:09:05.508743536 (13h29m5.098088533s) [upload=13989]: Got next segment. Inline: true Aug 7 23:09:05.508749488 (13h29m5.098094515s) [upload=13989]: Waiting for error group managing segments... Aug 7 23:09:05.508795345 (13h29m5.098140362s) [upload=13989]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 23:09:05.508801848 (13h29m5.098146856s) [upload=13989]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 23:09:05.508805736 (13h29m5.098150733s) [upload=13989]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 23:09:05.806325549 (13h29m5.395670536s) [upload=13989]: Done uploading segments... Aug 7 23:09:05.806343342 (13h29m5.395688329s) [upload=13989]: Upload finished. err: <nil> Aug 7 23:09:38.090257337 (13h29m37.679602324s) [upload=14002]: Starting upload Aug 7 23:09:38.090300970 (13h29m37.679645957s) [upload=14002]: Done starting upload Aug 7 23:09:38.090372567 (13h29m37.679717554s) [upload=14002]: Uploading segments... Aug 7 23:09:38.090406682 (13h29m37.67975168s) [upload=14002]: Got next segment. Inline: true Aug 7 23:09:38.090410099 (13h29m37.679755076s) [upload=14002]: Waiting for error group managing segments... Aug 7 23:09:38.090443282 (13h29m37.679788269s) [upload=14002]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 23:09:38.090447210 (13h29m37.679792198s) [upload=14002]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 23:09:38.090449415 (13h29m37.679794392s) [upload=14002]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 23:09:38.661400996 (13h29m38.250746043s) [upload=14002]: Done uploading segments... Aug 7 23:09:38.661423068 (13h29m38.250768135s) [upload=14002]: Upload finished. err: <nil> Aug 7 23:09:58.827549308 (13h29m58.416894295s) [upload=14009]: Starting upload Aug 7 23:09:58.827598442 (13h29m58.416943429s) [upload=14009]: Done starting upload Aug 7 23:09:58.827644020 (13h29m58.416989007s) [upload=14009]: Uploading segments... Aug 7 23:09:58.827674438 (13h29m58.417019425s) [upload=14009]: Got next segment. Inline: true Aug 7 23:09:58.827678135 (13h29m58.417023112s) [upload=14009]: Waiting for error group managing segments... Aug 7 23:09:58.827711559 (13h29m58.417056546s) [upload=14009]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 23:09:58.827715526 (13h29m58.417060513s) [upload=14009]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 23:09:58.827719804 (13h29m58.417064781s) [upload=14009]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 23:09:59.099262618 (13h29m58.688607605s) [upload=14009]: Done uploading segments... Aug 7 23:09:59.099279320 (13h29m58.688624297s) [upload=14009]: Upload finished. err: <nil> Aug 7 23:10:50.147486507 (13h30m49.736831494s) [upload=14023]: Starting upload Aug 7 23:10:50.147530131 (13h30m49.736875118s) [upload=14023]: Done starting upload Aug 7 23:10:50.147587851 (13h30m49.736932868s) [upload=14023]: Uploading segments... Aug 7 23:10:50.147650551 (13h30m49.736995568s) [upload=14023]: Got next segment. Inline: true Aug 7 23:10:50.147659338 (13h30m49.737004355s) [upload=14023]: Waiting for error group managing segments... Aug 7 23:10:50.147719804 (13h30m49.737064811s) [upload=14023]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 23:10:50.147727708 (13h30m49.737072746s) [upload=14023]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 23:10:50.147734752 (13h30m49.737079739s) [upload=14023]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 23:10:50.421404977 (13h30m50.010749964s) [upload=14023]: Done uploading segments... Aug 7 23:10:50.421427519 (13h30m50.010772546s) [upload=14023]: Upload finished. err: <nil> Aug 7 23:11:30.587306004 (13h31m30.176650991s) [upload=14032]: Starting upload Aug 7 23:11:30.587348937 (13h31m30.176693924s) [upload=14032]: Done starting upload Aug 7 23:11:30.587398742 (13h31m30.17674373s) [upload=14032]: Uploading segments... Aug 7 23:11:30.587432447 (13h31m30.176777434s) [upload=14032]: Got next segment. Inline: true Aug 7 23:11:30.587437326 (13h31m30.176782313s) [upload=14032]: Waiting for error group managing segments... Aug 7 23:11:30.587469467 (13h31m30.176814454s) [upload=14032]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 23:11:30.587472974 (13h31m30.176817951s) [upload=14032]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 23:11:30.587475168 (13h31m30.176820155s) [upload=14032]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 23:11:30.892529848 (13h31m30.481874835s) [upload=14032]: Done uploading segments... Aug 7 23:11:30.892541890 (13h31m30.481886877s) [upload=14032]: Upload finished. err: <nil> Aug 7 23:11:48.390915893 (13h31m47.98026088s) [upload=14038]: Starting upload Aug 7 23:11:48.390965198 (13h31m47.980310185s) [upload=14038]: Done starting upload Aug 7 23:11:48.391014242 (13h31m47.980359229s) [upload=14038]: Uploading segments... Aug 7 23:11:48.391054007 (13h31m47.980399004s) [upload=14038]: Got next segment. Inline: true Aug 7 23:11:48.391057494 (13h31m47.980402471s) [upload=14038]: Waiting for error group managing segments... Aug 7 23:11:48.391089254 (13h31m47.980434231s) [upload=14038]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 23:11:48.391093213 (13h31m47.98043819s) [upload=14038]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 23:11:48.391095457 (13h31m47.980440444s) [upload=14038]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 23:11:48.701950724 (13h31m48.291295711s) [upload=14038]: Done uploading segments... Aug 7 23:11:48.701969449 (13h31m48.291314436s) [upload=14038]: Upload finished. err: <nil> Aug 7 23:12:06.430245154 (13h32m6.019590141s) [upload=14048]: Starting upload Aug 7 23:12:06.430293076 (13h32m6.019638063s) [upload=14048]: Done starting upload Aug 7 23:12:06.430361757 (13h32m6.019706745s) [upload=14048]: Uploading segments... Aug 7 23:12:06.430459223 (13h32m6.01980421s) [upload=14048]: Got next segment. Inline: true Aug 7 23:12:06.430464543 (13h32m6.01980953s) [upload=14048]: Waiting for error group managing segments... Aug 7 23:12:06.430527555 (13h32m6.019872542s) [upload=14048]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 23:12:06.430533796 (13h32m6.019878783s) [upload=14048]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 23:12:06.430536692 (13h32m6.019881679s) [upload=14048]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 23:12:06.729952062 (13h32m6.319297049s) [upload=14048]: Done uploading segments... Aug 7 23:12:06.729968844 (13h32m6.319313832s) [upload=14048]: Upload finished. err: <nil> Aug 7 23:12:37.551701338 (13h32m37.141046325s) [upload=14055]: Starting upload Aug 7 23:12:37.551746985 (13h32m37.141091972s) [upload=14055]: Done starting upload Aug 7 23:12:37.551807301 (13h32m37.141152288s) [upload=14055]: Uploading segments... Aug 7 23:12:37.551914826 (13h32m37.141259833s) [upload=14055]: Got next segment. Inline: true Aug 7 23:12:37.551921779 (13h32m37.141266766s) [upload=14055]: Waiting for error group managing segments... Aug 7 23:12:37.551977145 (13h32m37.141322122s) [upload=14055]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 23:12:37.551984059 (13h32m37.141329086s) [upload=14055]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 23:12:37.551988918 (13h32m37.141333936s) [upload=14055]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 23:12:37.767068918 (13h32m37.356413905s) [upload=14055]: Done uploading segments... Aug 7 23:12:37.767104365 (13h32m37.356449342s) [upload=14055]: Upload finished. err: metaclient: Too Many Requests
Duration: 1.864369674s
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)antiblip:
1 out of 60 is failed (threshold=3) [........................................................X...] duration:
1.888029815s
FAIL (2025-08-07 23:14:11 UTC)
orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded
Duration: 1.888057689s
core
uplink
us1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)antiblip:
2 out of 60 is failed (threshold=3) [..........................................................XX] duration:
1m0.001224982s
FAIL (2025-08-07 23:14:04 UTC)
orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 820965393 (13h25m4.41031038s) [upload=13917]: Upload finished. err: <nil> Aug 7 23:06:17.748054724 (13h26m17.337399711s) [upload=13938]: Starting upload Aug 7 23:06:17.748105922 (13h26m17.337450909s) [upload=13938]: Done starting upload Aug 7 23:06:17.748150697 (13h26m17.337495684s) [upload=13938]: Uploading segments... Aug 7 23:06:17.748177308 (13h26m17.337522295s) [upload=13938]: Got next segment. Inline: true Aug 7 23:06:17.748181095 (13h26m17.337526082s) [upload=13938]: Waiting for error group managing segments... Aug 7 23:06:17.748240640 (13h26m17.337585617s) [upload=13938]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 23:06:17.748244918 (13h26m17.337589905s) [upload=13938]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 23:06:17.748247122 (13h26m17.337592099s) [upload=13938]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 23:06:18.012343389 (13h26m17.601688427s) [upload=13938]: Done uploading segments... Aug 7 23:06:18.012365292 (13h26m17.601710319s) [upload=13938]: Upload finished. err: <nil> Aug 7 23:06:43.977164591 (13h26m43.566509578s) [upload=13946]: Starting upload Aug 7 23:06:43.977216861 (13h26m43.566561848s) [upload=13946]: Done starting upload Aug 7 23:06:43.977265654 (13h26m43.566610631s) [upload=13946]: Uploading segments... Aug 7 23:06:43.977299669 (13h26m43.566644646s) [upload=13946]: Got next segment. Inline: true Aug 7 23:06:43.977304398 (13h26m43.566649375s) [upload=13946]: Waiting for error group managing segments... Aug 7 23:06:43.977335879 (13h26m43.566680866s) [upload=13946]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 23:06:43.977342241 (13h26m43.566687228s) [upload=13946]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 23:06:43.977344626 (13h26m43.566689613s) [upload=13946]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 23:06:44.261001386 (13h26m43.850346383s) [upload=13946]: Done uploading segments... Aug 7 23:06:44.261036533 (13h26m43.85038152s) [upload=13946]: Upload finished. err: <nil> Aug 7 23:07:14.299823176 (13h27m13.889168163s) [upload=13956]: Starting upload Aug 7 23:07:14.299919721 (13h27m13.889264729s) [upload=13956]: Done starting upload Aug 7 23:07:14.300062905 (13h27m13.889407902s) [upload=13956]: Uploading segments... Aug 7 23:07:14.300136495 (13h27m13.889481482s) [upload=13956]: Got next segment. Inline: true Aug 7 23:07:14.300143029 (13h27m13.889488016s) [upload=13956]: Waiting for error group managing segments... Aug 7 23:07:14.300224825 (13h27m13.889569822s) [upload=13956]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 23:07:14.300231578 (13h27m13.889576565s) [upload=13956]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 23:07:14.300235044 (13h27m13.889580041s) [upload=13956]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 23:07:14.558772216 (13h27m14.148117193s) [upload=13956]: Done uploading segments... Aug 7 23:07:14.558788256 (13h27m14.148133243s) [upload=13956]: Upload finished. err: <nil> Aug 7 23:07:52.874987115 (13h27m52.464332102s) [upload=13971]: Starting upload Aug 7 23:07:52.875039626 (13h27m52.464384603s) [upload=13971]: Done starting upload Aug 7 23:07:52.875078921 (13h27m52.464423908s) [upload=13971]: Uploading segments... Aug 7 23:07:52.875111583 (13h27m52.46445657s) [upload=13971]: Got next segment. Inline: true Aug 7 23:07:52.875115000 (13h27m52.464459987s) [upload=13971]: Waiting for error group managing segments... Aug 7 23:07:52.875144226 (13h27m52.464489203s) [upload=13971]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 23:07:52.875147783 (13h27m52.46449277s) [upload=13971]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 23:07:52.875150268 (13h27m52.464495255s) [upload=13971]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 23:07:53.152523417 (13h27m52.741868404s) [upload=13971]: Done uploading segments... Aug 7 23:07:53.152567841 (13h27m52.741912828s) [upload=13971]: Upload finished. err: <nil> Aug 7 23:08:13.160315277 (13h28m12.749660264s) [upload=13978]: Starting upload Aug 7 23:08:13.160372306 (13h28m12.749717293s) [upload=13978]: Done starting upload Aug 7 23:08:13.160419306 (13h28m12.749764293s) [upload=13978]: Uploading segments... Aug 7 23:08:13.160446879 (13h28m12.749791866s) [upload=13978]: Got next segment. Inline: true Aug 7 23:08:13.160450095 (13h28m12.749795082s) [upload=13978]: Waiting for error group managing segments... Aug 7 23:08:13.160481565 (13h28m12.749826542s) [upload=13978]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 23:08:13.160485202 (13h28m12.749830189s) [upload=13978]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 23:08:13.160487416 (13h28m12.749832403s) [upload=13978]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 23:08:13.673421309 (13h28m13.262766296s) [upload=13978]: Done uploading segments... Aug 7 23:08:13.673439774 (13h28m13.262784761s) [upload=13978]: Upload finished. err: <nil> Aug 7 23:09:08.585854499 (13h29m8.175199486s) [upload=13993]: Starting upload Aug 7 23:09:08.585909965 (13h29m8.175254952s) [upload=13993]: Done starting upload Aug 7 23:09:08.585953217 (13h29m8.175298204s) [upload=13993]: Uploading segments... Aug 7 23:09:08.585980830 (13h29m8.175325817s) [upload=13993]: Got next segment. Inline: true Aug 7 23:09:08.585984206 (13h29m8.175329183s) [upload=13993]: Waiting for error group managing segments... Aug 7 23:09:08.586014684 (13h29m8.175359671s) [upload=13993]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 23:09:08.586018372 (13h29m8.175363359s) [upload=13993]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 23:09:08.586020767 (13h29m8.175365754s) [upload=13993]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 23:09:08.854829559 (13h29m8.444174546s) [upload=13993]: Done uploading segments... Aug 7 23:09:08.854846763 (13h29m8.44419175s) [upload=13993]: Upload finished. err: <nil> Aug 7 23:10:03.068961384 (13h30m2.658306371s) [upload=14012]: Starting upload Aug 7 23:10:03.069021258 (13h30m2.658366245s) [upload=14012]: Done starting upload Aug 7 23:10:03.069094389 (13h30m2.658439376s) [upload=14012]: Uploading segments... Aug 7 23:10:03.069128504 (13h30m2.658473491s) [upload=14012]: Got next segment. Inline: true Aug 7 23:10:03.069132782 (13h30m2.658477779s) [upload=14012]: Waiting for error group managing segments... Aug 7 23:10:03.069170935 (13h30m2.658515932s) [upload=14012]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 23:10:03.069175905 (13h30m2.658520892s) [upload=14012]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 23:10:03.069178650 (13h30m2.658523647s) [upload=14012]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 23:10:03.364714047 (13h30m2.954059064s) [upload=14012]: Done uploading segments... Aug 7 23:10:03.364761748 (13h30m2.954106745s) [upload=14012]: Upload finished. err: <nil> Aug 7 23:10:44.905335214 (13h30m44.494680201s) [upload=14022]: Starting upload Aug 7 23:10:44.905402192 (13h30m44.494747179s) [upload=14022]: Done starting upload Aug 7 23:10:44.905449983 (13h30m44.49479496s) [upload=14022]: Uploading segments... Aug 7 23:10:44.905479650 (13h30m44.494824628s) [upload=14022]: Got next segment. Inline: true Aug 7 23:10:44.905482787 (13h30m44.494827774s) [upload=14022]: Waiting for error group managing segments... Aug 7 23:10:44.905516160 (13h30m44.494861147s) [upload=14022]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 23:10:44.905519717 (13h30m44.494864694s) [upload=14022]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 23:10:44.905522222 (13h30m44.494867209s) [upload=14022]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 23:10:45.196197762 (13h30m44.785542749s) [upload=14022]: Done uploading segments... Aug 7 23:10:45.196218401 (13h30m44.785563389s) [upload=14022]: Upload finished. err: <nil> Aug 7 23:11:56.158319777 (13h31m55.747664764s) [upload=14041]: Starting upload Aug 7 23:11:56.158381214 (13h31m55.747726201s) [upload=14041]: Done starting upload Aug 7 23:11:56.158454435 (13h31m55.747799422s) [upload=14041]: Uploading segments... Aug 7 23:11:56.158485985 (13h31m55.747830972s) [upload=14041]: Got next segment. Inline: true Aug 7 23:11:56.158489752 (13h31m55.747834739s) [upload=14041]: Waiting for error group managing segments... Aug 7 23:11:56.158533225 (13h31m55.747878212s) [upload=14041]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 23:11:56.158536933 (13h31m55.74788192s) [upload=14041]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 23:11:56.158539077 (13h31m55.747884064s) [upload=14041]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 23:12:55.512693945 (13h32m55.102038922s) [upload=14041]: Done uploading segments... Aug 7 23:12:55.512714414 (13h32m55.102059401s) [upload=14041]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:13:04.863194286 (13h33m4.452539303s) [upload=14061]: Starting upload Aug 7 23:13:04.863346336 (13h33m4.452691313s) [upload=14061]: Done starting upload Aug 7 23:13:04.863414577 (13h33m4.452759614s) [upload=14061]: Uploading segments... Aug 7 23:13:04.863490322 (13h33m4.452835309s) [upload=14061]: Got next segment. Inline: true Aug 7 23:13:04.863497876 (13h33m4.452842863s) [upload=14061]: Waiting for error group managing segments... Aug 7 23:13:04.863564063 (13h33m4.45290905s) [upload=14061]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 23:13:04.863569523 (13h33m4.45291451s) [upload=14061]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 23:13:04.863571978 (13h33m4.452916965s) [upload=14061]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 23:14:04.218769699 (13h34m3.808114676s) [upload=14061]: Done uploading segments... Aug 7 23:14:04.218789687 (13h34m3.808134664s) [upload=14061]: Upload finished. err: metaclient: context deadline exceeded
Duration: 1m0.001243708s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [.......................................................X..X.] duration:
6.267777417s
FAIL (2025-08-07 23:13:57 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: encryption: context deadline exceeded
======LOGS======
59]: Uploading segments...
Aug 7 23:12:55.079822176 (13h32m54.669167153s) [upload=14059]: Got next segment. Inline: false
Aug 7 23:12:55.079845671 (13h32m54.669190658s) [upload=14059]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Aug 7 23:12:55.079851282 (13h32m54.669196269s) [upload=14059]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Aug 7 23:12:55.339717318 (13h32m54.929062306s) [seg_pos={0 0}, upload=14059]: Begin upload segment...
Aug 7 23:12:55.339761062 (13h32m54.929106049s) [seg_pos={0 0}, upload=14059]: Done begin upload segment.
Aug 7 23:12:55.339782853 (13h32m54.929127841s) [noise=true , address=45.91.116.5:4298 , piece_id=FZCSGN3DXVVYDTDLEMEA6MVDUXLDOSVSCWSGOLZXLQ42WNBJQ7LQ, seg_pos={0 0}, upload=14059]: Uploading piece...
Aug 7 23:12:55.339815726 (13h32m54.929160713s) [noise=true , address=77.90.31.12:4592 , piece_id=4S54QTFYN2YSSM72UCJH5MC5GWOOWX4RAPHHJWPF2VNWMZ2HDG4Q, seg_pos={0 0}, upload=14059]: Uploading piece...
Aug 7 23:12:55.340179824 (13h32m54.929524871s) [noise=true , address=45.91.116.7:4009 , piece_id=H3ULXVGYQJ6GRP55PQFVDPMWXJQVAXWJE5GNBPZKPABOENG634XQ, seg_pos={0 0}, upload=14059]: Uploading piece...
Aug 7 23:12:55.340270457 (13h32m54.929615444s) [noise=true , address=77.90.31.10:4519 , piece_id=IZABDSMWKHPKK54KQYLWEAOWZDVHZNHS3O66AFFBJKSCAJM2QY4A, seg_pos={0 0}, upload=14059]: Uploading piece...
Aug 7 23:12:55.340553708 (13h32m54.929898695s) [noise=true , address=77.90.31.16:4734 , piece_id=6ULIPHB5QUWSV434PCNMITOVPWUQZP4HKL5JE4M32K5P4RARFBVQ, seg_pos={0 0}, upload=14059]: Uploading piece...
Aug 7 23:12:55.340588515 (13h32m54.929933502s) [noise=true , address=77.90.31.21:4877 , piece_id=5DHXZ2DKV6GAYFXFHOZLKHJ6LUIAFFR7XVWAUH5LYQJJAS4SWEBA, seg_pos={0 0}, upload=14059]: Uploading piece...
Aug 7 23:12:55.340839815 (13h32m54.930184792s) [noise=true , address=104.36.82.12:4151 , piece_id=WXDRVIKI34JYP724A5DF4CFKD2PO22PLYPW77LVDOWR27TQMBS7A, seg_pos={0 0}, upload=14059]: Uploading piece...
Aug 7 23:12:55.340867328 (13h32m54.930212305s) [noise=true , address=104.36.82.5:4839 , piece_id=LYVOJW5SWSWWCUTVRFX26XILKICCN7XGBSF5FB6PDYTOKXYQBEXQ, seg_pos={0 0}, upload=14059]: Uploading piece...
Aug 7 23:12:55.341097419 (13h32m54.930442406s) [noise=true , address=45.91.116.6:4975 , piece_id=Y433WBZG54QY6IYOH2XCF6Y7ZB3YRNAW42DWKMGKB74TUUJR5RFA, seg_pos={0 0}, upload=14059]: Uploading piece...
Aug 7 23:12:55.341103560 (13h32m54.930448547s) [noise=true , address=77.90.31.11:4555 , piece_id=KZPXPCCSFZL26HSXD6HMCVMGMMKNLSBHHRG3M35XNOAS6QNAAXOQ, seg_pos={0 0}, upload=14059]: Uploading piece...
Aug 7 23:12:55.341383185 (13h32m54.930728182s) [noise=true , address=45.91.116.9:4263 , piece_id=ADYGCH44Y6PZBFDIJ7SOKZEWSJBDPOENOTMGL5QFEG2NHYTYVNFA, seg_pos={0 0}, upload=14059]: Uploading piece...
Aug 7 23:12:55.341641559 (13h32m54.930986546s) [noise=true , address=77.90.31.19:4841 , piece_id=YPAYRMJPGQ2M4ZUFIR7JRGLTTSYTXJEITBG4A6AZH4P5OZTZMJIA, seg_pos={0 0}, upload=14059]: Uploading piece...
Aug 7 23:12:55.341881939 (13h32m54.931226926s) [noise=true , address=77.90.31.21:4878 , piece_id=PAZ6SIHJZWUDWQDHOIGGAZO63GR57VKKSWTWW4BUAGII72ATEH5A, seg_pos={0 0}, upload=14059]: Uploading piece...
Aug 7 23:12:55.342085740 (13h32m54.931430727s) [noise=true , address=77.90.31.11:4554 , piece_id=LTEEAKJMEIBIP7AKZYEVABCV64MQOXZLOWYW75XOQKI6QLCWSMGQ, seg_pos={0 0}, upload=14059]: Uploading piece...
Aug 7 23:12:55.342331731 (13h32m54.931676718s) [noise=true , address=104.36.82.11:4114 , piece_id=EGTOXJ473TYB3AS5PRUHNNPQRLRV2FXKTBV6NUWQ2GQFG3SENZ7Q, seg_pos={0 0}, upload=14059]: Uploading piece...
Aug 7 23:12:55.342545298 (13h32m54.931890275s) [noise=true , address=77.90.31.17:4772 , piece_id=QGKDTU53DPOOYALG6Q7XL3MC4Q2RLVEQGWSNKBOW7K5W4MQDTR4A, seg_pos={0 0}, upload=14059]: Uploading piece...
Aug 7 23:12:57.106062881 (13h32m56.695407868s) [upload=14059]: Next returned nil segment
Aug 7 23:12:57.106080956 (13h32m56.695425943s) [upload=14059]: Waiting for error group managing segments...
Aug 7 23:12:57.617485773 (13h32m57.20683076s) [noise=true , address=77.90.31.21:4877 , piece_id=5DHXZ2DKV6GAYFXFHOZLKHJ6LUIAFFR7XVWAUH5LYQJJAS4SWEBA, seg_pos={0 0}, upload=14059]: Done uploading piece. err: <nil>
Aug 7 23:12:57.633195419 (13h32m57.222540396s) [noise=true , address=77.90.31.19:4841 , piece_id=YPAYRMJPGQ2M4ZUFIR7JRGLTTSYTXJEITBG4A6AZH4P5OZTZMJIA, seg_pos={0 0}, upload=14059]: Done uploading piece. err: <nil>
Aug 7 23:12:57.681886423 (13h32m57.271231411s) [noise=true , address=45.91.116.5:4298 , piece_id=FZCSGN3DXVVYDTDLEMEA6MVDUXLDOSVSCWSGOLZXLQ42WNBJQ7LQ, seg_pos={0 0}, upload=14059]: Done uploading piece. err: <nil>
Aug 7 23:12:57.704146110 (13h32m57.293491097s) [noise=true , address=77.90.31.12:4592 , piece_id=4S54QTFYN2YSSM72UCJH5MC5GWOOWX4RAPHHJWPF2VNWMZ2HDG4Q, seg_pos={0 0}, upload=14059]: Done uploading piece. err: <nil>
Aug 7 23:12:57.738020960 (13h32m57.327365947s) [noise=true , address=45.91.116.7:4009 , piece_id=H3ULXVGYQJ6GRP55PQFVDPMWXJQVAXWJE5GNBPZKPABOENG634XQ, seg_pos={0 0}, upload=14059]: Done uploading piece. err: <nil>
Aug 7 23:12:57.754950650 (13h32m57.344295677s) [noise=true , address=77.90.31.11:4554 , piece_id=LTEEAKJMEIBIP7AKZYEVABCV64MQOXZLOWYW75XOQKI6QLCWSMGQ, seg_pos={0 0}, upload=14059]: Done uploading piece. err: <nil>
Aug 7 23:12:57.771705445 (13h32m57.361050432s) [noise=true , address=77.90.31.11:4555 , piece_id=KZPXPCCSFZL26HSXD6HMCVMGMMKNLSBHHRG3M35XNOAS6QNAAXOQ, seg_pos={0 0}, upload=14059]: Done uploading piece. err: <nil>
Aug 7 23:12:57.783341697 (13h32m57.372686684s) [noise=true , address=45.91.116.9:4263 , piece_id=ADYGCH44Y6PZBFDIJ7SOKZEWSJBDPOENOTMGL5QFEG2NHYTYVNFA, seg_pos={0 0}, upload=14059]: Done uploading piece. err: <nil>
Aug 7 23:12:57.848987701 (13h32m57.438332688s) [noise=true , address=77.90.31.10:4519 , piece_id=IZABDSMWKHPKK54KQYLWEAOWZDVHZNHS3O66AFFBJKSCAJM2QY4A, seg_pos={0 0}, upload=14059]: Done uploading piece. err: <nil>
Aug 7 23:12:57.880838979 (13h32m57.470183956s) [noise=true , address=77.90.31.17:4772 , piece_id=QGKDTU53DPOOYALG6Q7XL3MC4Q2RLVEQGWSNKBOW7K5W4MQDTR4A, seg_pos={0 0}, upload=14059]: Done uploading piece. err: <nil>
Aug 7 23:12:57.906171455 (13h32m57.495516442s) [noise=true , address=77.90.31.16:4734 , piece_id=6ULIPHB5QUWSV434PCNMITOVPWUQZP4HKL5JE4M32K5P4RARFBVQ, seg_pos={0 0}, upload=14059]: Done uploading piece. err: <nil>
Aug 7 23:12:57.986749206 (13h32m57.576094193s) [noise=true , address=45.91.116.6:4975 , piece_id=Y433WBZG54QY6IYOH2XCF6Y7ZB3YRNAW42DWKMGKB74TUUJR5RFA, seg_pos={0 0}, upload=14059]: Done uploading piece. err: <nil>
Aug 7 23:12:58.149399618 (13h32m57.738744605s) [noise=true , address=77.90.31.21:4878 , piece_id=PAZ6SIHJZWUDWQDHOIGGAZO63GR57VKKSWTWW4BUAGII72ATEH5A, seg_pos={0 0}, upload=14059]: Done uploading piece. err: <nil>
Aug 7 23:12:58.149487245 (13h32m57.738832232s) [seg_pos={0 0}, upload=14059]: Segment reached optimal threshold of 13 pieces.
Aug 7 23:12:58.149728517 (13h32m57.739073494s) [noise=true , address=104.36.82.12:4151 , piece_id=WXDRVIKI34JYP724A5DF4CFKD2PO22PLYPW77LVDOWR27TQMBS7A, seg_pos={0 0}, upload=14059]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB): protocol: rpc: dial tcp 104.36.82.12:4151: operation was canceled; conn closed: op
Aug 7 23:12:58.149779455 (13h32m57.739124442s) [noise=true , address=104.36.82.5:4839 , piece_id=LYVOJW5SWSWWCUTVRFX26XILKICCN7XGBSF5FB6PDYTOKXYQBEXQ, seg_pos={0 0}, upload=14059]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv): protocol: write tcp 188.166.225.170:47614->104.36.82.5:4839: use of closed network connection; write tcp 188.166.225.170:47614->104.36.82.5:4839: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47614->104.36.82.5:4839: use of closed network connection
Aug 7 23:12:58.149939941 (13h32m57.739284928s) [noise=true , address=104.36.82.11:4114 , piece_id=EGTOXJ473TYB3AS5PRUHNNPQRLRV2FXKTBV6NUWQ2GQFG3SENZ7Q, seg_pos={0 0}, upload=14059]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN): protocol: read tcp 188.166.225.170:50326->104.36.82.11:4114: use of closed network connection; conn closed: op; piecestore: piecestore close: context canceled
Aug 7 23:12:58.149986811 (13h32m57.739331788s) [seg_pos={0 0}, upload=14059]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Aug 7 23:12:58.150065151 (13h32m57.739410128s) [upload=14059]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Aug 7 23:12:58.150068738 (13h32m57.739413725s) [upload=14059]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Aug 7 23:12:58.821141052 (13h32m58.410486039s) [upload=14059]: Done uploading segments...
Aug 7 23:12:58.821159126 (13h32m58.410504103s) [upload=14059]: Upload finished. err: <nil>
Aug 7 23:13:22.019610941 (13h33m21.608955939s) [upload=14067]: Starting upload
Aug 7 23:13:22.019685434 (13h33m21.609030421s) [upload=14067]: Done starting upload
Aug 7 23:13:22.019765757 (13h33m21.609110744s) [upload=14067]: Uploading segments...
Aug 7 23:13:22.019893803 (13h33m21.60923879s) [upload=14067]: Got next segment. Inline: false
Aug 7 23:13:22.019935211 (13h33m21.609280198s) [upload=14067]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Aug 7 23:13:22.019939930 (13h33m21.609284917s) [upload=14067]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Aug 7 23:13:51.369780933 (13h33m50.95912591s) [upload=14067]: Next segment err: context deadline exceeded
Aug 7 23:13:51.369934848 (13h33m50.959279835s) [upload=14067]: Done uploading segments...
Aug 7 23:13:51.369951439 (13h33m50.959296426s) [upload=14067]: Upload finished. err: context deadline exceeded
Duration: 6.267794661s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [X.......................................................X..X] duration:
30.001063688s
FAIL (2025-08-07 23:13:51 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: encryption: context deadline exceeded
======LOGS======
59]: Uploading segments...
Aug 7 23:12:55.079822176 (13h32m54.669167153s) [upload=14059]: Got next segment. Inline: false
Aug 7 23:12:55.079845671 (13h32m54.669190658s) [upload=14059]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Aug 7 23:12:55.079851282 (13h32m54.669196269s) [upload=14059]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Aug 7 23:12:55.339717318 (13h32m54.929062306s) [seg_pos={0 0}, upload=14059]: Begin upload segment...
Aug 7 23:12:55.339761062 (13h32m54.929106049s) [seg_pos={0 0}, upload=14059]: Done begin upload segment.
Aug 7 23:12:55.339782853 (13h32m54.929127841s) [noise=true , address=45.91.116.5:4298 , piece_id=FZCSGN3DXVVYDTDLEMEA6MVDUXLDOSVSCWSGOLZXLQ42WNBJQ7LQ, seg_pos={0 0}, upload=14059]: Uploading piece...
Aug 7 23:12:55.339815726 (13h32m54.929160713s) [noise=true , address=77.90.31.12:4592 , piece_id=4S54QTFYN2YSSM72UCJH5MC5GWOOWX4RAPHHJWPF2VNWMZ2HDG4Q, seg_pos={0 0}, upload=14059]: Uploading piece...
Aug 7 23:12:55.340179824 (13h32m54.929524871s) [noise=true , address=45.91.116.7:4009 , piece_id=H3ULXVGYQJ6GRP55PQFVDPMWXJQVAXWJE5GNBPZKPABOENG634XQ, seg_pos={0 0}, upload=14059]: Uploading piece...
Aug 7 23:12:55.340270457 (13h32m54.929615444s) [noise=true , address=77.90.31.10:4519 , piece_id=IZABDSMWKHPKK54KQYLWEAOWZDVHZNHS3O66AFFBJKSCAJM2QY4A, seg_pos={0 0}, upload=14059]: Uploading piece...
Aug 7 23:12:55.340553708 (13h32m54.929898695s) [noise=true , address=77.90.31.16:4734 , piece_id=6ULIPHB5QUWSV434PCNMITOVPWUQZP4HKL5JE4M32K5P4RARFBVQ, seg_pos={0 0}, upload=14059]: Uploading piece...
Aug 7 23:12:55.340588515 (13h32m54.929933502s) [noise=true , address=77.90.31.21:4877 , piece_id=5DHXZ2DKV6GAYFXFHOZLKHJ6LUIAFFR7XVWAUH5LYQJJAS4SWEBA, seg_pos={0 0}, upload=14059]: Uploading piece...
Aug 7 23:12:55.340839815 (13h32m54.930184792s) [noise=true , address=104.36.82.12:4151 , piece_id=WXDRVIKI34JYP724A5DF4CFKD2PO22PLYPW77LVDOWR27TQMBS7A, seg_pos={0 0}, upload=14059]: Uploading piece...
Aug 7 23:12:55.340867328 (13h32m54.930212305s) [noise=true , address=104.36.82.5:4839 , piece_id=LYVOJW5SWSWWCUTVRFX26XILKICCN7XGBSF5FB6PDYTOKXYQBEXQ, seg_pos={0 0}, upload=14059]: Uploading piece...
Aug 7 23:12:55.341097419 (13h32m54.930442406s) [noise=true , address=45.91.116.6:4975 , piece_id=Y433WBZG54QY6IYOH2XCF6Y7ZB3YRNAW42DWKMGKB74TUUJR5RFA, seg_pos={0 0}, upload=14059]: Uploading piece...
Aug 7 23:12:55.341103560 (13h32m54.930448547s) [noise=true , address=77.90.31.11:4555 , piece_id=KZPXPCCSFZL26HSXD6HMCVMGMMKNLSBHHRG3M35XNOAS6QNAAXOQ, seg_pos={0 0}, upload=14059]: Uploading piece...
Aug 7 23:12:55.341383185 (13h32m54.930728182s) [noise=true , address=45.91.116.9:4263 , piece_id=ADYGCH44Y6PZBFDIJ7SOKZEWSJBDPOENOTMGL5QFEG2NHYTYVNFA, seg_pos={0 0}, upload=14059]: Uploading piece...
Aug 7 23:12:55.341641559 (13h32m54.930986546s) [noise=true , address=77.90.31.19:4841 , piece_id=YPAYRMJPGQ2M4ZUFIR7JRGLTTSYTXJEITBG4A6AZH4P5OZTZMJIA, seg_pos={0 0}, upload=14059]: Uploading piece...
Aug 7 23:12:55.341881939 (13h32m54.931226926s) [noise=true , address=77.90.31.21:4878 , piece_id=PAZ6SIHJZWUDWQDHOIGGAZO63GR57VKKSWTWW4BUAGII72ATEH5A, seg_pos={0 0}, upload=14059]: Uploading piece...
Aug 7 23:12:55.342085740 (13h32m54.931430727s) [noise=true , address=77.90.31.11:4554 , piece_id=LTEEAKJMEIBIP7AKZYEVABCV64MQOXZLOWYW75XOQKI6QLCWSMGQ, seg_pos={0 0}, upload=14059]: Uploading piece...
Aug 7 23:12:55.342331731 (13h32m54.931676718s) [noise=true , address=104.36.82.11:4114 , piece_id=EGTOXJ473TYB3AS5PRUHNNPQRLRV2FXKTBV6NUWQ2GQFG3SENZ7Q, seg_pos={0 0}, upload=14059]: Uploading piece...
Aug 7 23:12:55.342545298 (13h32m54.931890275s) [noise=true , address=77.90.31.17:4772 , piece_id=QGKDTU53DPOOYALG6Q7XL3MC4Q2RLVEQGWSNKBOW7K5W4MQDTR4A, seg_pos={0 0}, upload=14059]: Uploading piece...
Aug 7 23:12:57.106062881 (13h32m56.695407868s) [upload=14059]: Next returned nil segment
Aug 7 23:12:57.106080956 (13h32m56.695425943s) [upload=14059]: Waiting for error group managing segments...
Aug 7 23:12:57.617485773 (13h32m57.20683076s) [noise=true , address=77.90.31.21:4877 , piece_id=5DHXZ2DKV6GAYFXFHOZLKHJ6LUIAFFR7XVWAUH5LYQJJAS4SWEBA, seg_pos={0 0}, upload=14059]: Done uploading piece. err: <nil>
Aug 7 23:12:57.633195419 (13h32m57.222540396s) [noise=true , address=77.90.31.19:4841 , piece_id=YPAYRMJPGQ2M4ZUFIR7JRGLTTSYTXJEITBG4A6AZH4P5OZTZMJIA, seg_pos={0 0}, upload=14059]: Done uploading piece. err: <nil>
Aug 7 23:12:57.681886423 (13h32m57.271231411s) [noise=true , address=45.91.116.5:4298 , piece_id=FZCSGN3DXVVYDTDLEMEA6MVDUXLDOSVSCWSGOLZXLQ42WNBJQ7LQ, seg_pos={0 0}, upload=14059]: Done uploading piece. err: <nil>
Aug 7 23:12:57.704146110 (13h32m57.293491097s) [noise=true , address=77.90.31.12:4592 , piece_id=4S54QTFYN2YSSM72UCJH5MC5GWOOWX4RAPHHJWPF2VNWMZ2HDG4Q, seg_pos={0 0}, upload=14059]: Done uploading piece. err: <nil>
Aug 7 23:12:57.738020960 (13h32m57.327365947s) [noise=true , address=45.91.116.7:4009 , piece_id=H3ULXVGYQJ6GRP55PQFVDPMWXJQVAXWJE5GNBPZKPABOENG634XQ, seg_pos={0 0}, upload=14059]: Done uploading piece. err: <nil>
Aug 7 23:12:57.754950650 (13h32m57.344295677s) [noise=true , address=77.90.31.11:4554 , piece_id=LTEEAKJMEIBIP7AKZYEVABCV64MQOXZLOWYW75XOQKI6QLCWSMGQ, seg_pos={0 0}, upload=14059]: Done uploading piece. err: <nil>
Aug 7 23:12:57.771705445 (13h32m57.361050432s) [noise=true , address=77.90.31.11:4555 , piece_id=KZPXPCCSFZL26HSXD6HMCVMGMMKNLSBHHRG3M35XNOAS6QNAAXOQ, seg_pos={0 0}, upload=14059]: Done uploading piece. err: <nil>
Aug 7 23:12:57.783341697 (13h32m57.372686684s) [noise=true , address=45.91.116.9:4263 , piece_id=ADYGCH44Y6PZBFDIJ7SOKZEWSJBDPOENOTMGL5QFEG2NHYTYVNFA, seg_pos={0 0}, upload=14059]: Done uploading piece. err: <nil>
Aug 7 23:12:57.848987701 (13h32m57.438332688s) [noise=true , address=77.90.31.10:4519 , piece_id=IZABDSMWKHPKK54KQYLWEAOWZDVHZNHS3O66AFFBJKSCAJM2QY4A, seg_pos={0 0}, upload=14059]: Done uploading piece. err: <nil>
Aug 7 23:12:57.880838979 (13h32m57.470183956s) [noise=true , address=77.90.31.17:4772 , piece_id=QGKDTU53DPOOYALG6Q7XL3MC4Q2RLVEQGWSNKBOW7K5W4MQDTR4A, seg_pos={0 0}, upload=14059]: Done uploading piece. err: <nil>
Aug 7 23:12:57.906171455 (13h32m57.495516442s) [noise=true , address=77.90.31.16:4734 , piece_id=6ULIPHB5QUWSV434PCNMITOVPWUQZP4HKL5JE4M32K5P4RARFBVQ, seg_pos={0 0}, upload=14059]: Done uploading piece. err: <nil>
Aug 7 23:12:57.986749206 (13h32m57.576094193s) [noise=true , address=45.91.116.6:4975 , piece_id=Y433WBZG54QY6IYOH2XCF6Y7ZB3YRNAW42DWKMGKB74TUUJR5RFA, seg_pos={0 0}, upload=14059]: Done uploading piece. err: <nil>
Aug 7 23:12:58.149399618 (13h32m57.738744605s) [noise=true , address=77.90.31.21:4878 , piece_id=PAZ6SIHJZWUDWQDHOIGGAZO63GR57VKKSWTWW4BUAGII72ATEH5A, seg_pos={0 0}, upload=14059]: Done uploading piece. err: <nil>
Aug 7 23:12:58.149487245 (13h32m57.738832232s) [seg_pos={0 0}, upload=14059]: Segment reached optimal threshold of 13 pieces.
Aug 7 23:12:58.149728517 (13h32m57.739073494s) [noise=true , address=104.36.82.12:4151 , piece_id=WXDRVIKI34JYP724A5DF4CFKD2PO22PLYPW77LVDOWR27TQMBS7A, seg_pos={0 0}, upload=14059]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB): protocol: rpc: dial tcp 104.36.82.12:4151: operation was canceled; conn closed: op
Aug 7 23:12:58.149779455 (13h32m57.739124442s) [noise=true , address=104.36.82.5:4839 , piece_id=LYVOJW5SWSWWCUTVRFX26XILKICCN7XGBSF5FB6PDYTOKXYQBEXQ, seg_pos={0 0}, upload=14059]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv): protocol: write tcp 188.166.225.170:47614->104.36.82.5:4839: use of closed network connection; write tcp 188.166.225.170:47614->104.36.82.5:4839: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47614->104.36.82.5:4839: use of closed network connection
Aug 7 23:12:58.149939941 (13h32m57.739284928s) [noise=true , address=104.36.82.11:4114 , piece_id=EGTOXJ473TYB3AS5PRUHNNPQRLRV2FXKTBV6NUWQ2GQFG3SENZ7Q, seg_pos={0 0}, upload=14059]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN): protocol: read tcp 188.166.225.170:50326->104.36.82.11:4114: use of closed network connection; conn closed: op; piecestore: piecestore close: context canceled
Aug 7 23:12:58.149986811 (13h32m57.739331788s) [seg_pos={0 0}, upload=14059]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Aug 7 23:12:58.150065151 (13h32m57.739410128s) [upload=14059]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Aug 7 23:12:58.150068738 (13h32m57.739413725s) [upload=14059]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Aug 7 23:12:58.821141052 (13h32m58.410486039s) [upload=14059]: Done uploading segments...
Aug 7 23:12:58.821159126 (13h32m58.410504103s) [upload=14059]: Upload finished. err: <nil>
Aug 7 23:13:22.019610941 (13h33m21.608955939s) [upload=14067]: Starting upload
Aug 7 23:13:22.019685434 (13h33m21.609030421s) [upload=14067]: Done starting upload
Aug 7 23:13:22.019765757 (13h33m21.609110744s) [upload=14067]: Uploading segments...
Aug 7 23:13:22.019893803 (13h33m21.60923879s) [upload=14067]: Got next segment. Inline: false
Aug 7 23:13:22.019935211 (13h33m21.609280198s) [upload=14067]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Aug 7 23:13:22.019939930 (13h33m21.609284917s) [upload=14067]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Aug 7 23:13:51.369780933 (13h33m50.95912591s) [upload=14067]: Next segment err: context deadline exceeded
Aug 7 23:13:51.369934848 (13h33m50.959279835s) [upload=14067]: Done uploading segments...
Aug 7 23:13:51.369951439 (13h33m50.959296426s) [upload=14067]: Upload finished. err: context deadline exceeded
Duration: 30.001091571s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [..X.......................................................XX] duration:
30.000836855s
FAIL (2025-08-07 23:13:50 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 30.000845962s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
6 out of 60 is failed (threshold=3) [XXX.X.....................................................XX] duration:
1m0.001656266s
FAIL (2025-08-07 23:13:50 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1m0.001670093s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
4 out of 60 is failed (threshold=3) [X.................X...................X....................X] duration:
6.047072956s
FAIL (2025-08-07 23:13:49 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/4M):: after e85db788dbeec38ad8084daa355e1bf59b8b718af998fc40c7451ca1ba6627b2 != before 1cad814b0341363008cdb7d22dd66bb6d87f0105cd39850888ced4405a0b4b43
======LOGS======
upload
Aug 7 23:13:43.967401320 (13h33m43.556746307s) [upload=14069]: Uploading segments...
Aug 7 23:13:43.967498206 (13h33m43.556843193s) [upload=14069]: Got next segment. Inline: false
Aug 7 23:13:43.967560455 (13h33m43.556905442s) [upload=14069]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Aug 7 23:13:43.967568550 (13h33m43.556913577s) [upload=14069]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Aug 7 23:13:44.414536988 (13h33m44.003881975s) [seg_pos={0 0}, upload=14069]: Begin upload segment...
Aug 7 23:13:44.414587104 (13h33m44.003932091s) [seg_pos={0 0}, upload=14069]: Done begin upload segment.
Aug 7 23:13:44.414606933 (13h33m44.00395192s) [noise=true , address=104.36.82.60:4362 , piece_id=LJMWWXQ4NXJY54S7WPOSCO37ALDF5PT2GEC5LKOAKB6YEW2PLEAA, seg_pos={0 0}, upload=14069]: Uploading piece...
Aug 7 23:13:44.414666236 (13h33m44.004011223s) [noise=true , address=104.36.82.43:4868 , piece_id=TOYHNT5XLP5YVN6ZFSWOS4UJOSIIOKUKWXP4CNSG56CHGEUZFQDQ, seg_pos={0 0}, upload=14069]: Uploading piece...
Aug 7 23:13:44.414966160 (13h33m44.004311147s) [noise=true , address=104.36.82.52:4086 , piece_id=DRDBLVU5QJKKQC5WCUJROGJYG7OQKTIRVAXDLFHQW3FIC2BV2WIA, seg_pos={0 0}, upload=14069]: Uploading piece...
Aug 7 23:13:44.415118381 (13h33m44.004463369s) [noise=true , address=104.36.82.16:4296 , piece_id=HCYT2XGA5RGQQGMWKTDKJIF6OQWK6NR4Z4SUPSG5IOA6ECK3EYPQ, seg_pos={0 0}, upload=14069]: Uploading piece...
Aug 7 23:13:44.415228682 (13h33m44.00457367s) [noise=true , address=79.127.205.233:4621 , piece_id=DSV45XV43BCLRGH7BJOICWBBDBCYFZXDC4CVACYRNT4LI745DK7Q, seg_pos={0 0}, upload=14069]: Uploading piece...
Aug 7 23:13:44.415496494 (13h33m44.004841471s) [noise=true , address=104.36.82.58:4291 , piece_id=DP64RXRSK2NLFM3N7TSZCZL4PEGZVHEQJHPJFXAIQXXLOEPQWY5Q, seg_pos={0 0}, upload=14069]: Uploading piece...
Aug 7 23:13:44.415745370 (13h33m44.005090357s) [noise=true , address=45.91.116.13:4158 , piece_id=62VB7AEIWCPAVJSLX3P2TOQRSALK5A6ZAD3ZFVQINR2IVT3AOXXQ, seg_pos={0 0}, upload=14069]: Uploading piece...
Aug 7 23:13:44.416026788 (13h33m44.005371766s) [noise=true , address=104.36.82.36:4642 , piece_id=3X7E6QOZKZ5KGPLWHIWL6WZRGOFHKZLFIZRKVZM3CGJM6VLA6OAA, seg_pos={0 0}, upload=14069]: Uploading piece...
Aug 7 23:13:44.416256569 (13h33m44.005601556s) [noise=true , address=104.36.82.14:4226 , piece_id=CLREJCNEC4WX6IGMUEJIV2IEZXFAVLSXLZVYMWJJBHHDHCQ6DVQA, seg_pos={0 0}, upload=14069]: Uploading piece...
Aug 7 23:13:44.416456080 (13h33m44.005801067s) [noise=true , address=64.49.13.137:4499 , piece_id=7D5JZSVLNEFT26CX3TBD4L27WIZU2IWH2JK3O7IHPAVDCUDBZAJQ, seg_pos={0 0}, upload=14069]: Uploading piece...
Aug 7 23:13:44.416552746 (13h33m44.005897753s) [noise=true , address=45.91.116.14:4194 , piece_id=2LO5TLHD4ZILDKAMW6EXNTMDHG4PCRFYS6F5EMLUN4EBP4RWEUPA, seg_pos={0 0}, upload=14069]: Uploading piece...
Aug 7 23:13:44.416803596 (13h33m44.006148583s) [noise=true , address=45.91.116.11:4150 , piece_id=2YORTWP2L67BV5ENJQ5CKPY3NK337SK6WM5XL77RUZ6LMQWXVORA, seg_pos={0 0}, upload=14069]: Uploading piece...
Aug 7 23:13:44.417042221 (13h33m44.006387208s) [noise=true , address=104.36.82.53:4126 , piece_id=A6LKRBLYVWJ7QN4UJ7JPDXKLCI5ZFAJ6I6ICCXDE7V7G2F5R65UA, seg_pos={0 0}, upload=14069]: Uploading piece...
Aug 7 23:13:44.417277101 (13h33m44.006622088s) [noise=true , address=104.36.82.59:4332 , piece_id=K5TGW4E726UEFXZQSKWGYU2PHFX2IYE2BYKHXUG2WQXTZV5GV5ZA, seg_pos={0 0}, upload=14069]: Uploading piece...
Aug 7 23:13:44.417316085 (13h33m44.006661072s) [noise=true , address=104.36.82.8:4977 , piece_id=PDXPXV3IJ7Y4P25A6BS3HTKCYLLNMRYIHAMLJCVV3QDMHGN4PXKQ, seg_pos={0 0}, upload=14069]: Uploading piece...
Aug 7 23:13:44.417537459 (13h33m44.006882436s) [noise=true , address=104.36.82.2:4757 , piece_id=LJ3JPH3HYJ7Z5QIWGSKWC3KGH7QJXYT3IDTFEN326A6QOLGAWVZA, seg_pos={0 0}, upload=14069]: Uploading piece...
Aug 7 23:13:44.417738975 (13h33m44.007083952s) [noise=true , address=64.49.13.26:4946 , piece_id=VYQT7NJAPN6XN6Z5PKEBJCPSYWRZHR5ZCGG6257JA3Q2BTK6S4FA, seg_pos={0 0}, upload=14069]: Uploading piece...
Aug 7 23:13:44.417952763 (13h33m44.00729774s) [noise=true , address=104.36.82.9:4037 , piece_id=RISOPBKZ6Y3UYM3HSBMJT6EI3ZXGQKGANDYNUSDK3IGPCNLC4QTQ, seg_pos={0 0}, upload=14069]: Uploading piece...
Aug 7 23:13:45.514309387 (13h33m45.103654374s) [upload=14069]: Next returned nil segment
Aug 7 23:13:45.514327020 (13h33m45.103672007s) [upload=14069]: Waiting for error group managing segments...
Aug 7 23:13:45.795258156 (13h33m45.384603143s) [noise=true , address=64.49.13.137:4499 , piece_id=7D5JZSVLNEFT26CX3TBD4L27WIZU2IWH2JK3O7IHPAVDCUDBZAJQ, seg_pos={0 0}, upload=14069]: Done uploading piece. err: <nil>
Aug 7 23:13:45.802263469 (13h33m45.391608456s) [noise=true , address=79.127.205.233:4621 , piece_id=DSV45XV43BCLRGH7BJOICWBBDBCYFZXDC4CVACYRNT4LI745DK7Q, seg_pos={0 0}, upload=14069]: Done uploading piece. err: <nil>
Aug 7 23:13:45.887877424 (13h33m45.477222401s) [noise=true , address=64.49.13.26:4946 , piece_id=VYQT7NJAPN6XN6Z5PKEBJCPSYWRZHR5ZCGG6257JA3Q2BTK6S4FA, seg_pos={0 0}, upload=14069]: Done uploading piece. err: <nil>
Aug 7 23:13:46.350209185 (13h33m45.939554172s) [noise=true , address=104.36.82.60:4362 , piece_id=LJMWWXQ4NXJY54S7WPOSCO37ALDF5PT2GEC5LKOAKB6YEW2PLEAA, seg_pos={0 0}, upload=14069]: Done uploading piece. err: <nil>
Aug 7 23:13:46.389672887 (13h33m45.979017874s) [noise=true , address=104.36.82.52:4086 , piece_id=DRDBLVU5QJKKQC5WCUJROGJYG7OQKTIRVAXDLFHQW3FIC2BV2WIA, seg_pos={0 0}, upload=14069]: Done uploading piece. err: <nil>
Aug 7 23:13:46.399409194 (13h33m45.988754171s) [noise=true , address=104.36.82.58:4291 , piece_id=DP64RXRSK2NLFM3N7TSZCZL4PEGZVHEQJHPJFXAIQXXLOEPQWY5Q, seg_pos={0 0}, upload=14069]: Done uploading piece. err: <nil>
Aug 7 23:13:46.424971119 (13h33m46.014316106s) [noise=true , address=104.36.82.36:4642 , piece_id=3X7E6QOZKZ5KGPLWHIWL6WZRGOFHKZLFIZRKVZM3CGJM6VLA6OAA, seg_pos={0 0}, upload=14069]: Done uploading piece. err: <nil>
Aug 7 23:13:46.450918913 (13h33m46.0402639s) [noise=true , address=45.91.116.13:4158 , piece_id=62VB7AEIWCPAVJSLX3P2TOQRSALK5A6ZAD3ZFVQINR2IVT3AOXXQ, seg_pos={0 0}, upload=14069]: Done uploading piece. err: <nil>
Aug 7 23:13:46.492609174 (13h33m46.081954161s) [noise=true , address=104.36.82.53:4126 , piece_id=A6LKRBLYVWJ7QN4UJ7JPDXKLCI5ZFAJ6I6ICCXDE7V7G2F5R65UA, seg_pos={0 0}, upload=14069]: Done uploading piece. err: <nil>
Aug 7 23:13:46.516716376 (13h33m46.106061353s) [noise=true , address=104.36.82.9:4037 , piece_id=RISOPBKZ6Y3UYM3HSBMJT6EI3ZXGQKGANDYNUSDK3IGPCNLC4QTQ, seg_pos={0 0}, upload=14069]: Done uploading piece. err: <nil>
Aug 7 23:13:46.592315332 (13h33m46.181660319s) [noise=true , address=104.36.82.43:4868 , piece_id=TOYHNT5XLP5YVN6ZFSWOS4UJOSIIOKUKWXP4CNSG56CHGEUZFQDQ, seg_pos={0 0}, upload=14069]: Done uploading piece. err: <nil>
Aug 7 23:13:46.746385819 (13h33m46.335730806s) [noise=true , address=45.91.116.11:4150 , piece_id=2YORTWP2L67BV5ENJQ5CKPY3NK337SK6WM5XL77RUZ6LMQWXVORA, seg_pos={0 0}, upload=14069]: Done uploading piece. err: <nil>
Aug 7 23:13:46.786539771 (13h33m46.375884758s) [noise=true , address=104.36.82.59:4332 , piece_id=K5TGW4E726UEFXZQSKWGYU2PHFX2IYE2BYKHXUG2WQXTZV5GV5ZA, seg_pos={0 0}, upload=14069]: Done uploading piece. err: <nil>
Aug 7 23:13:46.838001699 (13h33m46.427346676s) [noise=true , address=45.91.116.14:4194 , piece_id=2LO5TLHD4ZILDKAMW6EXNTMDHG4PCRFYS6F5EMLUN4EBP4RWEUPA, seg_pos={0 0}, upload=14069]: Done uploading piece. err: <nil>
Aug 7 23:13:46.838017648 (13h33m46.427362636s) [seg_pos={0 0}, upload=14069]: Segment reached optimal threshold of 14 pieces.
Aug 7 23:13:46.838217681 (13h33m46.427562658s) [noise=true , address=104.36.82.14:4226 , piece_id=CLREJCNEC4WX6IGMUEJIV2IEZXFAVLSXLZVYMWJJBHHDHCQ6DVQA, seg_pos={0 0}, upload=14069]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:126xWe3ELGmYQfSUk3tYtGTSa6LnGfDNWcGT164pR6tHEtFPAFt): protocol: expected piece hash; context canceled; EOF
Aug 7 23:13:46.838422624 (13h33m46.427767601s) [noise=true , address=104.36.82.2:4757 , piece_id=LJ3JPH3HYJ7Z5QIWGSKWC3KGH7QJXYT3IDTFEN326A6QOLGAWVZA, seg_pos={0 0}, upload=14069]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uVPbSLBfJdPZqh4yFcEMQTPUwbLhT4GAM813h2nK5qa9xCw9q): protocol: expected piece hash; context canceled; EOF
Aug 7 23:13:46.838575365 (13h33m46.427920352s) [noise=true , address=104.36.82.16:4296 , piece_id=HCYT2XGA5RGQQGMWKTDKJIF6OQWK6NR4Z4SUPSG5IOA6ECK3EYPQ, seg_pos={0 0}, upload=14069]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cKhDbA4ptwux2LYNPNAJ7Qh8jECsGWd1KWaK6T6pD38bJWdt1): protocol: rpc: dial tcp 104.36.82.16:4296: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 7 23:13:46.838627195 (13h33m46.427972172s) [noise=true , address=104.36.82.8:4977 , piece_id=PDXPXV3IJ7Y4P25A6BS3HTKCYLLNMRYIHAMLJCVV3QDMHGN4PXKQ, seg_pos={0 0}, upload=14069]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WCAm12tXwimFSPqz7Seh6A4oxWqQVxqsF459xux4a5CqjpbAT): protocol: write tcp 188.166.225.170:48060->104.36.82.8:4977: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 7 23:13:46.838635551 (13h33m46.427980538s) [seg_pos={0 0}, upload=14069]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Aug 7 23:13:46.838702128 (13h33m46.428047115s) [upload=14069]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Aug 7 23:13:46.838705895 (13h33m46.428050882s) [upload=14069]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Aug 7 23:13:47.127771955 (13h33m46.717116932s) [upload=14069]: Done uploading segments...
Aug 7 23:13:47.127791082 (13h33m46.717136069s) [upload=14069]: Upload finished. err: <nil>
Duration: 6.047103023s
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)antiblip:
1 out of 60 is failed (threshold=3) [..........................................................X.] duration:
1.840792415s
FAIL (2025-08-07 23:13:48 UTC)
orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== segments... Aug 7 23:06:49.229716945 (13h26m48.819061932s) [upload=13950]: Upload finished. err: <nil> Aug 7 23:07:42.104963855 (13h27m41.694308842s) [upload=13968]: Starting upload Aug 7 23:07:42.105031205 (13h27m41.694376192s) [upload=13968]: Done starting upload Aug 7 23:07:42.105104063 (13h27m41.69444905s) [upload=13968]: Uploading segments... Aug 7 23:07:42.105139652 (13h27m41.694484639s) [upload=13968]: Got next segment. Inline: true Aug 7 23:07:42.105143259 (13h27m41.694488246s) [upload=13968]: Waiting for error group managing segments... Aug 7 23:07:42.105190129 (13h27m41.694535126s) [upload=13968]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 23:07:42.105201120 (13h27m41.694546127s) [upload=13968]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 23:07:42.105207281 (13h27m41.694552278s) [upload=13968]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 23:07:42.417069033 (13h27m42.00641402s) [upload=13968]: Done uploading segments... Aug 7 23:07:42.417104130 (13h27m42.006449117s) [upload=13968]: Upload finished. err: <nil> Aug 7 23:08:34.230557646 (13h28m33.819902633s) [upload=13982]: Starting upload Aug 7 23:08:34.230604215 (13h28m33.819949193s) [upload=13982]: Done starting upload Aug 7 23:08:34.230655173 (13h28m33.82000016s) [upload=13982]: Uploading segments... Aug 7 23:08:34.230683928 (13h28m33.820028905s) [upload=13982]: Got next segment. Inline: true Aug 7 23:08:34.230687385 (13h28m33.820032362s) [upload=13982]: Waiting for error group managing segments... Aug 7 23:08:34.230719175 (13h28m33.820064162s) [upload=13982]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 23:08:34.230722832 (13h28m33.820067819s) [upload=13982]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 23:08:34.230725046 (13h28m33.820070033s) [upload=13982]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 23:08:34.528494026 (13h28m34.117839013s) [upload=13982]: Done uploading segments... Aug 7 23:08:34.528512322 (13h28m34.117857309s) [upload=13982]: Upload finished. err: <nil> Aug 7 23:09:05.508570966 (13h29m5.097915953s) [upload=13989]: Starting upload Aug 7 23:09:05.508638426 (13h29m5.097983423s) [upload=13989]: Done starting upload Aug 7 23:09:05.508701747 (13h29m5.098046764s) [upload=13989]: Uploading segments... Aug 7 23:09:05.508743536 (13h29m5.098088533s) [upload=13989]: Got next segment. Inline: true Aug 7 23:09:05.508749488 (13h29m5.098094515s) [upload=13989]: Waiting for error group managing segments... Aug 7 23:09:05.508795345 (13h29m5.098140362s) [upload=13989]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 23:09:05.508801848 (13h29m5.098146856s) [upload=13989]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 23:09:05.508805736 (13h29m5.098150733s) [upload=13989]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 23:09:05.806325549 (13h29m5.395670536s) [upload=13989]: Done uploading segments... Aug 7 23:09:05.806343342 (13h29m5.395688329s) [upload=13989]: Upload finished. err: <nil> Aug 7 23:09:38.090257337 (13h29m37.679602324s) [upload=14002]: Starting upload Aug 7 23:09:38.090300970 (13h29m37.679645957s) [upload=14002]: Done starting upload Aug 7 23:09:38.090372567 (13h29m37.679717554s) [upload=14002]: Uploading segments... Aug 7 23:09:38.090406682 (13h29m37.67975168s) [upload=14002]: Got next segment. Inline: true Aug 7 23:09:38.090410099 (13h29m37.679755076s) [upload=14002]: Waiting for error group managing segments... Aug 7 23:09:38.090443282 (13h29m37.679788269s) [upload=14002]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 23:09:38.090447210 (13h29m37.679792198s) [upload=14002]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 23:09:38.090449415 (13h29m37.679794392s) [upload=14002]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 23:09:38.661400996 (13h29m38.250746043s) [upload=14002]: Done uploading segments... Aug 7 23:09:38.661423068 (13h29m38.250768135s) [upload=14002]: Upload finished. err: <nil> Aug 7 23:09:58.827549308 (13h29m58.416894295s) [upload=14009]: Starting upload Aug 7 23:09:58.827598442 (13h29m58.416943429s) [upload=14009]: Done starting upload Aug 7 23:09:58.827644020 (13h29m58.416989007s) [upload=14009]: Uploading segments... Aug 7 23:09:58.827674438 (13h29m58.417019425s) [upload=14009]: Got next segment. Inline: true Aug 7 23:09:58.827678135 (13h29m58.417023112s) [upload=14009]: Waiting for error group managing segments... Aug 7 23:09:58.827711559 (13h29m58.417056546s) [upload=14009]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 23:09:58.827715526 (13h29m58.417060513s) [upload=14009]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 23:09:58.827719804 (13h29m58.417064781s) [upload=14009]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 23:09:59.099262618 (13h29m58.688607605s) [upload=14009]: Done uploading segments... Aug 7 23:09:59.099279320 (13h29m58.688624297s) [upload=14009]: Upload finished. err: <nil> Aug 7 23:10:50.147486507 (13h30m49.736831494s) [upload=14023]: Starting upload Aug 7 23:10:50.147530131 (13h30m49.736875118s) [upload=14023]: Done starting upload Aug 7 23:10:50.147587851 (13h30m49.736932868s) [upload=14023]: Uploading segments... Aug 7 23:10:50.147650551 (13h30m49.736995568s) [upload=14023]: Got next segment. Inline: true Aug 7 23:10:50.147659338 (13h30m49.737004355s) [upload=14023]: Waiting for error group managing segments... Aug 7 23:10:50.147719804 (13h30m49.737064811s) [upload=14023]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 23:10:50.147727708 (13h30m49.737072746s) [upload=14023]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 23:10:50.147734752 (13h30m49.737079739s) [upload=14023]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 23:10:50.421404977 (13h30m50.010749964s) [upload=14023]: Done uploading segments... Aug 7 23:10:50.421427519 (13h30m50.010772546s) [upload=14023]: Upload finished. err: <nil> Aug 7 23:11:30.587306004 (13h31m30.176650991s) [upload=14032]: Starting upload Aug 7 23:11:30.587348937 (13h31m30.176693924s) [upload=14032]: Done starting upload Aug 7 23:11:30.587398742 (13h31m30.17674373s) [upload=14032]: Uploading segments... Aug 7 23:11:30.587432447 (13h31m30.176777434s) [upload=14032]: Got next segment. Inline: true Aug 7 23:11:30.587437326 (13h31m30.176782313s) [upload=14032]: Waiting for error group managing segments... Aug 7 23:11:30.587469467 (13h31m30.176814454s) [upload=14032]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 23:11:30.587472974 (13h31m30.176817951s) [upload=14032]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 23:11:30.587475168 (13h31m30.176820155s) [upload=14032]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 23:11:30.892529848 (13h31m30.481874835s) [upload=14032]: Done uploading segments... Aug 7 23:11:30.892541890 (13h31m30.481886877s) [upload=14032]: Upload finished. err: <nil> Aug 7 23:11:48.390915893 (13h31m47.98026088s) [upload=14038]: Starting upload Aug 7 23:11:48.390965198 (13h31m47.980310185s) [upload=14038]: Done starting upload Aug 7 23:11:48.391014242 (13h31m47.980359229s) [upload=14038]: Uploading segments... Aug 7 23:11:48.391054007 (13h31m47.980399004s) [upload=14038]: Got next segment. Inline: true Aug 7 23:11:48.391057494 (13h31m47.980402471s) [upload=14038]: Waiting for error group managing segments... Aug 7 23:11:48.391089254 (13h31m47.980434231s) [upload=14038]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 23:11:48.391093213 (13h31m47.98043819s) [upload=14038]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 23:11:48.391095457 (13h31m47.980440444s) [upload=14038]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 23:11:48.701950724 (13h31m48.291295711s) [upload=14038]: Done uploading segments... Aug 7 23:11:48.701969449 (13h31m48.291314436s) [upload=14038]: Upload finished. err: <nil> Aug 7 23:12:06.430245154 (13h32m6.019590141s) [upload=14048]: Starting upload Aug 7 23:12:06.430293076 (13h32m6.019638063s) [upload=14048]: Done starting upload Aug 7 23:12:06.430361757 (13h32m6.019706745s) [upload=14048]: Uploading segments... Aug 7 23:12:06.430459223 (13h32m6.01980421s) [upload=14048]: Got next segment. Inline: true Aug 7 23:12:06.430464543 (13h32m6.01980953s) [upload=14048]: Waiting for error group managing segments... Aug 7 23:12:06.430527555 (13h32m6.019872542s) [upload=14048]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 23:12:06.430533796 (13h32m6.019878783s) [upload=14048]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 23:12:06.430536692 (13h32m6.019881679s) [upload=14048]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 23:12:06.729952062 (13h32m6.319297049s) [upload=14048]: Done uploading segments... Aug 7 23:12:06.729968844 (13h32m6.319313832s) [upload=14048]: Upload finished. err: <nil> Aug 7 23:12:37.551701338 (13h32m37.141046325s) [upload=14055]: Starting upload Aug 7 23:12:37.551746985 (13h32m37.141091972s) [upload=14055]: Done starting upload Aug 7 23:12:37.551807301 (13h32m37.141152288s) [upload=14055]: Uploading segments... Aug 7 23:12:37.551914826 (13h32m37.141259833s) [upload=14055]: Got next segment. Inline: true Aug 7 23:12:37.551921779 (13h32m37.141266766s) [upload=14055]: Waiting for error group managing segments... Aug 7 23:12:37.551977145 (13h32m37.141322122s) [upload=14055]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 23:12:37.551984059 (13h32m37.141329086s) [upload=14055]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 23:12:37.551988918 (13h32m37.141333936s) [upload=14055]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 23:12:37.767068918 (13h32m37.356413905s) [upload=14055]: Done uploading segments... Aug 7 23:12:37.767104365 (13h32m37.356449342s) [upload=14055]: Upload finished. err: metaclient: Too Many Requests
Duration: 1.840817351s
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline)antiblip:
2 out of 60 is failed (threshold=3) [X.X.........................................................] duration:
2.344637616s
FAIL (2025-08-07 23:13:39 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.344659898s
edge
gateway
us-select-1
|
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)antiblip:
1 out of 60 is failed (threshold=3) [................X...........................................] duration:
650.850562ms target:
https://us1.storj.io/login
FAIL (2025-08-07 23:13:36 UTC)
orbiter warning: Get "https://us1.storj.io/login": dial tcp 34.150.199.48:443: connect: connection refused
Duration: 650.883325ms
core
satellite
us1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [...X.......................................................X] duration:
30.000375843s
FAIL (2025-08-07 23:13:20 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 30.000388548s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)antiblip:
1 out of 60 is failed (threshold=3) [.........................................................X..] duration:
1.872213319s
FAIL (2025-08-07 23:13:07 UTC)
orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded
Duration: 1.872242625s
core
uplink
us1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [XX.......................................................X..] duration:
7.182174452s
FAIL (2025-08-07 23:13:01 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: encryption: context deadline exceeded
======LOGS======
30m19.119835718s) [upload=14014]: Got next segment. Inline: false
Aug 7 23:10:19.530521730 (13h30m19.119866718s) [upload=14014]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Aug 7 23:10:19.530526099 (13h30m19.119871086s) [upload=14014]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Aug 7 23:10:19.769147767 (13h30m19.358492754s) [seg_pos={0 0}, upload=14014]: Begin upload segment...
Aug 7 23:10:19.769196962 (13h30m19.358541949s) [seg_pos={0 0}, upload=14014]: Done begin upload segment.
Aug 7 23:10:19.769217901 (13h30m19.358562878s) [noise=true , address=77.90.31.19:4842 , piece_id=QUFDMBB7C5KALQOTIOQE6OOVFB3HTL23IPXC3SNT7TPUPBMENIYQ, seg_pos={0 0}, upload=14014]: Uploading piece...
Aug 7 23:10:19.769241967 (13h30m19.358586954s) [noise=true , address=45.91.116.9:4263 , piece_id=75NMKEGELJDZFSDE6L7BUOXIOCUZVZYZSCQFXL5AWFFN4L3B432Q, seg_pos={0 0}, upload=14014]: Uploading piece...
Aug 7 23:10:19.769614190 (13h30m19.358959177s) [noise=true , address=104.36.82.15:4257 , piece_id=DRDDV5DLUAAT2TFHHH4GMNHECM3LQHNH24Y2ZJALUXRCXL6QNRRA, seg_pos={0 0}, upload=14014]: Uploading piece...
Aug 7 23:10:19.769658985 (13h30m19.359003972s) [noise=true , address=77.90.31.16:4733 , piece_id=GRL5LY7JWUUYQFQ5CUZXDOZIYEG4GWWMVN76YTFPGKQOAP22XXNA, seg_pos={0 0}, upload=14014]: Uploading piece...
Aug 7 23:10:19.769909134 (13h30m19.359254111s) [noise=true , address=45.91.116.7:4011 , piece_id=ABOGCAOR5HL2MGGHQHXS3O3WLSEXD2B4UW4WGDJMDQYK5KBK2MSA, seg_pos={0 0}, upload=14014]: Uploading piece...
Aug 7 23:10:19.769942408 (13h30m19.359287395s) [noise=true , address=77.90.31.18:4807 , piece_id=7SNCXGGPT7RTAPPOX3LE526FIQU5G2VYYRJDFIPDFLA6ZPTFX7PQ, seg_pos={0 0}, upload=14014]: Uploading piece...
Aug 7 23:10:19.770178389 (13h30m19.359523366s) [noise=true , address=104.36.82.2:4730 , piece_id=3B4SQCLJ6FI6CRNRWCJRNA2CX4X5KZZELCFGOBIWVW7NQLLWKBRQ, seg_pos={0 0}, upload=14014]: Uploading piece...
Aug 7 23:10:19.770446231 (13h30m19.359791208s) [noise=true , address=104.36.82.8:4969 , piece_id=2EHLHQAWG3TA545TZV3QUAMRTLWBAYP2KQXXY7XSITAOUI5BJV7Q, seg_pos={0 0}, upload=14014]: Uploading piece...
Aug 7 23:10:19.770673426 (13h30m19.360018403s) [noise=true , address=45.91.116.11:4119 , piece_id=QMJWOHYHCVIVE7AVUUE3YWFTPZXQ2SK4QAAAS2ILP7J5OSVU2XTA, seg_pos={0 0}, upload=14014]: Uploading piece...
Aug 7 23:10:19.770876775 (13h30m19.360221763s) [noise=true , address=77.90.31.21:4877 , piece_id=DA742EGVEWSYO6PQZR7Z2IJTXFIBBYN5EVNKOGE2TZCM5RWOMTRA, seg_pos={0 0}, upload=14014]: Uploading piece...
Aug 7 23:10:19.771065907 (13h30m19.360410884s) [noise=true , address=77.90.31.14:4661 , piece_id=DX7EKUM3I6BRROMSBR3QMWOPBRXW3QCKDXL7RSCZ4MYGAOE62HEA, seg_pos={0 0}, upload=14014]: Uploading piece...
Aug 7 23:10:19.771256863 (13h30m19.36060185s) [noise=true , address=104.36.82.16:4295 , piece_id=NI7VPQGP7DXVVO6H7URWAGUWTHAIAQCXDZYWDMCD2EPIPLFDCGFQ, seg_pos={0 0}, upload=14014]: Uploading piece...
Aug 7 23:10:19.771456625 (13h30m19.360801612s) [noise=true , address=45.91.116.5:4298 , piece_id=ESZYBPWHWL2MOTBPVHK42M6CADJZNPJZBZXZLTYIUZHCH5XV5ZCQ, seg_pos={0 0}, upload=14014]: Uploading piece...
Aug 7 23:10:19.771646057 (13h30m19.360991034s) [noise=true , address=45.91.116.6:4973 , piece_id=MW66WBZULRRDBTQSZCHAOIC44YVAFFO2EKDEZLHF6QVZ55CHTHPQ, seg_pos={0 0}, upload=14014]: Uploading piece...
Aug 7 23:10:19.771840640 (13h30m19.361185627s) [noise=true , address=45.91.116.4:4939 , piece_id=XRYW6ZXHU4BCYNAKNWFAYBPJDOXQCAGTEIUJ7KJKNLVES73LD43Q, seg_pos={0 0}, upload=14014]: Uploading piece...
Aug 7 23:10:19.772068876 (13h30m19.361413863s) [noise=true , address=45.91.116.8:4045 , piece_id=XLOA7HZ7FRG5H5HMLMECR2PCSVOGD6XI5NS5HSTOQTZZ5KIMXIRA, seg_pos={0 0}, upload=14014]: Uploading piece...
Aug 7 23:10:20.020996815 (13h30m19.610341802s) [noise=true , address=77.90.31.16:4733 , piece_id=GRL5LY7JWUUYQFQ5CUZXDOZIYEG4GWWMVN76YTFPGKQOAP22XXNA, seg_pos={0 0}, upload=14014]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: write tcp 188.166.225.170:37310->77.90.31.16:4733: write: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 7 23:10:21.532433287 (13h30m21.121778274s) [upload=14014]: Next returned nil segment
Aug 7 23:10:21.532449418 (13h30m21.121794405s) [upload=14014]: Waiting for error group managing segments...
Aug 7 23:10:21.811529953 (13h30m21.40087494s) [noise=true , address=77.90.31.21:4877 , piece_id=DA742EGVEWSYO6PQZR7Z2IJTXFIBBYN5EVNKOGE2TZCM5RWOMTRA, seg_pos={0 0}, upload=14014]: Done uploading piece. err: <nil>
Aug 7 23:10:21.854297295 (13h30m21.443642272s) [noise=true , address=77.90.31.18:4807 , piece_id=7SNCXGGPT7RTAPPOX3LE526FIQU5G2VYYRJDFIPDFLA6ZPTFX7PQ, seg_pos={0 0}, upload=14014]: Done uploading piece. err: <nil>
Aug 7 23:10:21.921242465 (13h30m21.510587452s) [noise=true , address=104.36.82.8:4969 , piece_id=2EHLHQAWG3TA545TZV3QUAMRTLWBAYP2KQXXY7XSITAOUI5BJV7Q, seg_pos={0 0}, upload=14014]: Done uploading piece. err: <nil>
Aug 7 23:10:21.947359331 (13h30m21.536704318s) [noise=true , address=45.91.116.9:4263 , piece_id=75NMKEGELJDZFSDE6L7BUOXIOCUZVZYZSCQFXL5AWFFN4L3B432Q, seg_pos={0 0}, upload=14014]: Done uploading piece. err: <nil>
Aug 7 23:10:21.969454293 (13h30m21.55879928s) [noise=true , address=77.90.31.19:4842 , piece_id=QUFDMBB7C5KALQOTIOQE6OOVFB3HTL23IPXC3SNT7TPUPBMENIYQ, seg_pos={0 0}, upload=14014]: Done uploading piece. err: <nil>
Aug 7 23:10:22.042768958 (13h30m21.632113936s) [noise=true , address=45.91.116.5:4298 , piece_id=ESZYBPWHWL2MOTBPVHK42M6CADJZNPJZBZXZLTYIUZHCH5XV5ZCQ, seg_pos={0 0}, upload=14014]: Done uploading piece. err: <nil>
Aug 7 23:10:22.050439524 (13h30m21.639784501s) [noise=true , address=45.91.116.7:4011 , piece_id=ABOGCAOR5HL2MGGHQHXS3O3WLSEXD2B4UW4WGDJMDQYK5KBK2MSA, seg_pos={0 0}, upload=14014]: Done uploading piece. err: <nil>
Aug 7 23:10:22.063864939 (13h30m21.653209926s) [noise=true , address=45.91.116.8:4045 , piece_id=XLOA7HZ7FRG5H5HMLMECR2PCSVOGD6XI5NS5HSTOQTZZ5KIMXIRA, seg_pos={0 0}, upload=14014]: Done uploading piece. err: <nil>
Aug 7 23:10:22.108762327 (13h30m21.698107314s) [noise=true , address=45.91.116.11:4119 , piece_id=QMJWOHYHCVIVE7AVUUE3YWFTPZXQ2SK4QAAAS2ILP7J5OSVU2XTA, seg_pos={0 0}, upload=14014]: Done uploading piece. err: <nil>
Aug 7 23:10:22.196386158 (13h30m21.785731146s) [noise=true , address=104.36.82.16:4295 , piece_id=NI7VPQGP7DXVVO6H7URWAGUWTHAIAQCXDZYWDMCD2EPIPLFDCGFQ, seg_pos={0 0}, upload=14014]: Done uploading piece. err: <nil>
Aug 7 23:10:22.220196592 (13h30m21.809541579s) [noise=true , address=45.91.116.6:4973 , piece_id=MW66WBZULRRDBTQSZCHAOIC44YVAFFO2EKDEZLHF6QVZ55CHTHPQ, seg_pos={0 0}, upload=14014]: Done uploading piece. err: <nil>
Aug 7 23:10:22.247094513 (13h30m21.8364395s) [noise=true , address=45.91.116.4:4939 , piece_id=XRYW6ZXHU4BCYNAKNWFAYBPJDOXQCAGTEIUJ7KJKNLVES73LD43Q, seg_pos={0 0}, upload=14014]: Done uploading piece. err: <nil>
Aug 7 23:10:22.380553005 (13h30m21.969897992s) [noise=true , address=77.90.31.14:4661 , piece_id=DX7EKUM3I6BRROMSBR3QMWOPBRXW3QCKDXL7RSCZ4MYGAOE62HEA, seg_pos={0 0}, upload=14014]: Done uploading piece. err: <nil>
Aug 7 23:10:22.380581249 (13h30m21.969926236s) [seg_pos={0 0}, upload=14014]: Segment reached optimal threshold of 13 pieces.
Aug 7 23:10:22.380848780 (13h30m21.970193767s) [noise=true , address=104.36.82.2:4730 , piece_id=3B4SQCLJ6FI6CRNRWCJRNA2CX4X5KZZELCFGOBIWVW7NQLLWKBRQ, seg_pos={0 0}, upload=14014]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom): protocol: write tcp 188.166.225.170:44882->104.36.82.2:4730: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 7 23:10:22.380911371 (13h30m21.970256358s) [noise=true , address=104.36.82.15:4257 , piece_id=DRDDV5DLUAAT2TFHHH4GMNHECM3LQHNH24Y2ZJALUXRCXL6QNRRA, seg_pos={0 0}, upload=14014]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK): protocol: write tcp 188.166.225.170:60016->104.36.82.15:4257: use of closed network connection; write tcp 188.166.225.170:60016->104.36.82.15:4257: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60016->104.36.82.15:4257: use of closed network connection
Aug 7 23:10:22.380940176 (13h30m21.970285163s) [seg_pos={0 0}, upload=14014]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Aug 7 23:10:22.381026481 (13h30m21.970371468s) [upload=14014]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Aug 7 23:10:22.381030618 (13h30m21.970375606s) [upload=14014]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Aug 7 23:10:22.734882827 (13h30m22.324227804s) [upload=14014]: Done uploading segments...
Aug 7 23:10:22.734915910 (13h30m22.324260917s) [upload=14014]: Upload finished. err: <nil>
Aug 7 23:11:22.023156432 (13h31m21.612501409s) [upload=14030]: Starting upload
Aug 7 23:11:22.023212850 (13h31m21.612557837s) [upload=14030]: Done starting upload
Aug 7 23:11:22.023366314 (13h31m21.612711301s) [upload=14030]: Uploading segments...
Aug 7 23:11:22.023488728 (13h31m21.612833715s) [upload=14030]: Got next segment. Inline: false
Aug 7 23:11:22.023517703 (13h31m21.61286268s) [upload=14030]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Aug 7 23:11:22.023522112 (13h31m21.612867099s) [upload=14030]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Aug 7 23:11:51.377219283 (13h31m50.96656427s) [upload=14030]: Next segment err: context deadline exceeded
Aug 7 23:11:51.377486564 (13h31m50.966831552s) [upload=14030]: Done uploading segments...
Aug 7 23:11:51.377502725 (13h31m50.966847712s) [upload=14030]: Upload finished. err: context deadline exceeded
Duration: 7.182196877s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
5 out of 60 is failed (threshold=3) [XX..X.X....................................................X] duration:
1m0.0006532s
FAIL (2025-08-07 23:12:59 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: encryption: context deadline exceeded
======LOGS======
MAIE2WRIZU4V34X5RQ, seg_pos={0 0}, upload=14026]: Uploading piece...
Aug 7 23:10:58.326254256 (13h30m57.915599243s) [noise=true , address=77.90.31.17:4782 , piece_id=JH5WUV2V3EW4KY3E2PAGXMBPITLDNEHWOWQTHGGXMCEBNW6GNFOQ, seg_pos={0 0}, upload=14026]: Uploading piece...
Aug 7 23:10:58.326404964 (13h30m57.915749952s) [noise=true , address=104.36.82.8:4987 , piece_id=I6HVZPZKXJ7RQMJUCY6ZC7UTV3JSW4IJNXBEUVNOZL6WD32XBZFA, seg_pos={0 0}, upload=14026]: Uploading piece...
Aug 7 23:10:58.326511188 (13h30m57.915856175s) [noise=true , address=104.36.82.60:4337 , piece_id=GQWMRNCJK72UVH7ZJS2V3UMNO2GBQHJENSHWX3ZRLE747RTVP5TQ, seg_pos={0 0}, upload=14026]: Uploading piece...
Aug 7 23:10:58.326768891 (13h30m57.916113868s) [noise=true , address=104.36.82.59:4336 , piece_id=T57QBD57ALVLPCNKJ7RJREQI36POMQIIZ35RUMHQMJG2E3CSVSHQ, seg_pos={0 0}, upload=14026]: Uploading piece...
Aug 7 23:10:58.326941310 (13h30m57.916286297s) [noise=true , address=64.49.13.170:4718 , piece_id=YEYP7JBOXKOZ4NDHXYM7WHLFSQO6BZWUL36N4TTGOI2SUFAE5N2Q, seg_pos={0 0}, upload=14026]: Uploading piece...
Aug 7 23:10:58.326991376 (13h30m57.916336363s) [noise=true , address=104.36.82.58:4283 , piece_id=R7OXN3SBLXXCKCYGFMH23TS6PMT7KFY4OJXSYFPBR7QSQEHJ25SA, seg_pos={0 0}, upload=14026]: Uploading piece...
Aug 7 23:10:58.327230304 (13h30m57.916575291s) [noise=true , address=104.36.82.22:4541 , piece_id=XSFQJ75NP4IW4DWF7GPGHTVDJ3TUS2X4PTYTJZEA3X2CAUZ6PEGQ, seg_pos={0 0}, upload=14026]: Uploading piece...
Aug 7 23:10:58.327360833 (13h30m57.91670582s) [noise=true , address=104.36.82.62:4414 , piece_id=UD5F5SRJAL7Y3F7AZ4GH43QDY2MUI5YK6U6JR2KBAJC3VV3KWL4Q, seg_pos={0 0}, upload=14026]: Uploading piece...
Aug 7 23:10:58.327446957 (13h30m57.916791944s) [noise=true , address=104.36.82.2:4747 , piece_id=LP7YTSS4TZAHGSHTBC3K65YNKTCH4V56PWRXUADCHNQVJT2EXCXA, seg_pos={0 0}, upload=14026]: Uploading piece...
Aug 7 23:10:58.327664283 (13h30m57.91700927s) [noise=true , address=45.91.116.10:4106 , piece_id=QKAKLXEZ7F7ACE7HU4MMHBMKD6VCPH3UC3ZSVVPMAYSWZVKJY63Q, seg_pos={0 0}, upload=14026]: Uploading piece...
Aug 7 23:10:58.327814030 (13h30m57.917159027s) [noise=true , address=104.36.82.16:4327 , piece_id=5CNLHHOYGA2LNBHTJXJ3HOJXPZSORPBH5WZ4XNFP3OPL34KR2SBA, seg_pos={0 0}, upload=14026]: Uploading piece...
Aug 7 23:10:58.327882350 (13h30m57.917227337s) [noise=true , address=45.91.116.6:4008 , piece_id=KZXHMJ4BD7LVEBVKRKL7W6P3LD6IFZUZ54SDCP2DYM3HA4ERVZKA, seg_pos={0 0}, upload=14026]: Uploading piece...
Aug 7 23:10:58.328263840 (13h30m57.917608827s) [noise=true , address=104.36.82.45:4920 , piece_id=C2ATBJWXGD34B2GJZU7L4LH2RV3G6PR5EITUWDHW2BTQ62I57D7Q, seg_pos={0 0}, upload=14026]: Uploading piece...
Aug 7 23:10:58.328675428 (13h30m57.918020415s) [noise=true , address=64.49.13.137:4497 , piece_id=SNJXNZPUZZXQD5GQCWX6V6IXZJYNJ6Y57TYMYN2OPATTXPMVBS7A, seg_pos={0 0}, upload=14026]: Uploading piece...
Aug 7 23:10:58.329060845 (13h30m57.918405842s) [noise=true , address=104.36.82.12:4154 , piece_id=P3P4BKYNY57TORU6223ZMROZWP5ZXLIQTJ3LKGECIIYPQSKIHJMA, seg_pos={0 0}, upload=14026]: Uploading piece...
Aug 7 23:10:59.437540349 (13h30m59.026885337s) [upload=14026]: Next returned nil segment
Aug 7 23:10:59.437556841 (13h30m59.026901829s) [upload=14026]: Waiting for error group managing segments...
Aug 7 23:10:59.718841364 (13h30m59.308186351s) [noise=true , address=64.49.13.137:4497 , piece_id=SNJXNZPUZZXQD5GQCWX6V6IXZJYNJ6Y57TYMYN2OPATTXPMVBS7A, seg_pos={0 0}, upload=14026]: Done uploading piece. err: <nil>
Aug 7 23:10:59.825083015 (13h30m59.414428002s) [noise=true , address=64.49.13.170:4718 , piece_id=YEYP7JBOXKOZ4NDHXYM7WHLFSQO6BZWUL36N4TTGOI2SUFAE5N2Q, seg_pos={0 0}, upload=14026]: Done uploading piece. err: <nil>
Aug 7 23:11:00.325891629 (13h30m59.915236616s) [noise=true , address=104.36.82.15:4272 , piece_id=XZUEXYQ5TRJJOA4XHTF335X4ACIA5R6HHHLCFEJ4ZXDYZOSMUJPQ, seg_pos={0 0}, upload=14026]: Done uploading piece. err: <nil>
Aug 7 23:11:00.337428040 (13h30m59.926773027s) [noise=true , address=45.91.116.10:4106 , piece_id=QKAKLXEZ7F7ACE7HU4MMHBMKD6VCPH3UC3ZSVVPMAYSWZVKJY63Q, seg_pos={0 0}, upload=14026]: Done uploading piece. err: <nil>
Aug 7 23:11:00.364070432 (13h30m59.953415419s) [noise=true , address=104.36.82.8:4987 , piece_id=I6HVZPZKXJ7RQMJUCY6ZC7UTV3JSW4IJNXBEUVNOZL6WD32XBZFA, seg_pos={0 0}, upload=14026]: Done uploading piece. err: <nil>
Aug 7 23:11:00.400282073 (13h30m59.98962705s) [noise=true , address=104.36.82.13:4214 , piece_id=B3CWTI64GYO3WZ5DTDBS7JBPC3OV55PNX4MAIE2WRIZU4V34X5RQ, seg_pos={0 0}, upload=14026]: Done uploading piece. err: <nil>
Aug 7 23:11:00.456237196 (13h31m0.045582183s) [noise=true , address=104.36.82.9:4037 , piece_id=5HMUHVNDILDQUNOBEJRXP5PHVQV4W2GUVY5KEAN3NPJFUCQ232IA, seg_pos={0 0}, upload=14026]: Done uploading piece. err: <nil>
Aug 7 23:11:00.559884874 (13h31m0.149229861s) [noise=true , address=77.90.31.17:4782 , piece_id=JH5WUV2V3EW4KY3E2PAGXMBPITLDNEHWOWQTHGGXMCEBNW6GNFOQ, seg_pos={0 0}, upload=14026]: Done uploading piece. err: <nil>
Aug 7 23:11:00.606927676 (13h31m0.196272663s) [noise=true , address=104.36.82.58:4283 , piece_id=R7OXN3SBLXXCKCYGFMH23TS6PMT7KFY4OJXSYFPBR7QSQEHJ25SA, seg_pos={0 0}, upload=14026]: Done uploading piece. err: <nil>
Aug 7 23:11:00.658216512 (13h31m0.247561489s) [noise=true , address=104.36.82.2:4747 , piece_id=LP7YTSS4TZAHGSHTBC3K65YNKTCH4V56PWRXUADCHNQVJT2EXCXA, seg_pos={0 0}, upload=14026]: Done uploading piece. err: <nil>
Aug 7 23:11:00.690196575 (13h31m0.279541562s) [noise=true , address=45.91.116.6:4008 , piece_id=KZXHMJ4BD7LVEBVKRKL7W6P3LD6IFZUZ54SDCP2DYM3HA4ERVZKA, seg_pos={0 0}, upload=14026]: Done uploading piece. err: <nil>
Aug 7 23:11:00.917949522 (13h31m0.507294509s) [noise=true , address=104.36.82.60:4337 , piece_id=GQWMRNCJK72UVH7ZJS2V3UMNO2GBQHJENSHWX3ZRLE747RTVP5TQ, seg_pos={0 0}, upload=14026]: Done uploading piece. err: <nil>
Aug 7 23:11:01.791621637 (13h31m1.380966624s) [noise=true , address=104.36.82.59:4336 , piece_id=T57QBD57ALVLPCNKJ7RJREQI36POMQIIZ35RUMHQMJG2E3CSVSHQ, seg_pos={0 0}, upload=14026]: Done uploading piece. err: <nil>
Aug 7 23:11:01.886710571 (13h31m1.476055568s) [noise=true , address=104.36.82.62:4414 , piece_id=UD5F5SRJAL7Y3F7AZ4GH43QDY2MUI5YK6U6JR2KBAJC3VV3KWL4Q, seg_pos={0 0}, upload=14026]: Done uploading piece. err: <nil>
Aug 7 23:11:01.886746019 (13h31m1.476091006s) [seg_pos={0 0}, upload=14026]: Segment reached optimal threshold of 14 pieces.
Aug 7 23:11:01.886925121 (13h31m1.476270108s) [noise=true , address=104.36.82.12:4154 , piece_id=P3P4BKYNY57TORU6223ZMROZWP5ZXLIQTJ3LKGECIIYPQSKIHJMA, seg_pos={0 0}, upload=14026]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12S7Xsfnti3QuD9apBq1sgSjmahnAYA6LYbXqyckbLY4UU84ZQF): protocol: expected piece hash; write tcp 188.166.225.170:40722->104.36.82.12:4154: use of closed network connection; write tcp 188.166.225.170:40722->104.36.82.12:4154: use of closed network connection; EOF
Aug 7 23:11:01.887099706 (13h31m1.476444693s) [noise=true , address=104.36.82.45:4920 , piece_id=C2ATBJWXGD34B2GJZU7L4LH2RV3G6PR5EITUWDHW2BTQ62I57D7Q, seg_pos={0 0}, upload=14026]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ryY6XP6e8yUpMgwUmwf5z5syrjj6jBqzMpt2AUL9yV8tbd18S): protocol: context canceled; context canceled
Aug 7 23:11:01.887148139 (13h31m1.476493126s) [noise=true , address=104.36.82.16:4327 , piece_id=5CNLHHOYGA2LNBHTJXJ3HOJXPZSORPBH5WZ4XNFP3OPL34KR2SBA, seg_pos={0 0}, upload=14026]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1qZC1tmCvCadnm1fYZxR3kc4fonZUJPzvfTKkTFiDtccW5bo9T): protocol: read tcp 188.166.225.170:42114->104.36.82.16:4327: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 7 23:11:01.887241286 (13h31m1.476586273s) [noise=true , address=104.36.82.22:4541 , piece_id=XSFQJ75NP4IW4DWF7GPGHTVDJ3TUS2X4PTYTJZEA3X2CAUZ6PEGQ, seg_pos={0 0}, upload=14026]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zupYUhZUbuwMv5gwprk8M3kPYQ6qJiFK8ZsgGg3b1sqH6UwGr): protocol: write tcp 188.166.225.170:33974->104.36.82.22:4541: use of closed network connection; write tcp 188.166.225.170:33974->104.36.82.22:4541: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33974->104.36.82.22:4541: use of closed network connection
Aug 7 23:11:01.887268629 (13h31m1.476613616s) [seg_pos={0 0}, upload=14026]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Aug 7 23:11:01.887344244 (13h31m1.476689231s) [upload=14026]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Aug 7 23:11:01.887348952 (13h31m1.47669394s) [upload=14026]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Aug 7 23:11:02.179729203 (13h31m1.7690742s) [upload=14026]: Done uploading segments...
Aug 7 23:11:02.179751605 (13h31m1.769096602s) [upload=14026]: Upload finished. err: <nil>
Aug 7 23:12:00.553354812 (13h32m0.142699799s) [upload=14044]: Starting upload
Aug 7 23:12:00.553453822 (13h32m0.142798809s) [upload=14044]: Done starting upload
Aug 7 23:12:00.553533735 (13h32m0.142878722s) [upload=14044]: Uploading segments...
Aug 7 23:12:00.553787330 (13h32m0.143132327s) [upload=14044]: Got next segment. Inline: false
Aug 7 23:12:00.553874727 (13h32m0.143219754s) [upload=14044]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Aug 7 23:12:00.553882602 (13h32m0.143227589s) [upload=14044]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Aug 7 23:12:59.893877484 (13h32m59.483222471s) [upload=14044]: Next segment err: context deadline exceeded
Aug 7 23:12:59.893937389 (13h32m59.483282376s) [upload=14044]: Done uploading segments...
Aug 7 23:12:59.893949992 (13h32m59.483294979s) [upload=14044]: Upload finished. err: context deadline exceeded
Duration: 1m0.0006743s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)antiblip:
1 out of 60 is failed (threshold=3) [...........................................................X] duration:
1m0.00115161s
FAIL (2025-08-07 23:12:55 UTC)
orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ng segments... Aug 7 23:04:02.398549979 (13h24m1.987894966s) [upload=13900]: Upload finished. err: <nil> Aug 7 23:05:04.530978631 (13h25m4.120323618s) [upload=13917]: Starting upload Aug 7 23:05:04.531032664 (13h25m4.120377651s) [upload=13917]: Done starting upload Aug 7 23:05:04.531082430 (13h25m4.120427417s) [upload=13917]: Uploading segments... Aug 7 23:05:04.531113890 (13h25m4.120458877s) [upload=13917]: Got next segment. Inline: true Aug 7 23:05:04.531118148 (13h25m4.120463135s) [upload=13917]: Waiting for error group managing segments... Aug 7 23:05:04.531150901 (13h25m4.120495888s) [upload=13917]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 23:05:04.531154588 (13h25m4.120499575s) [upload=13917]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 23:05:04.531156893 (13h25m4.12050187s) [upload=13917]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 23:05:04.820945725 (13h25m4.410290722s) [upload=13917]: Done uploading segments... Aug 7 23:05:04.820965393 (13h25m4.41031038s) [upload=13917]: Upload finished. err: <nil> Aug 7 23:06:17.748054724 (13h26m17.337399711s) [upload=13938]: Starting upload Aug 7 23:06:17.748105922 (13h26m17.337450909s) [upload=13938]: Done starting upload Aug 7 23:06:17.748150697 (13h26m17.337495684s) [upload=13938]: Uploading segments... Aug 7 23:06:17.748177308 (13h26m17.337522295s) [upload=13938]: Got next segment. Inline: true Aug 7 23:06:17.748181095 (13h26m17.337526082s) [upload=13938]: Waiting for error group managing segments... Aug 7 23:06:17.748240640 (13h26m17.337585617s) [upload=13938]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 23:06:17.748244918 (13h26m17.337589905s) [upload=13938]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 23:06:17.748247122 (13h26m17.337592099s) [upload=13938]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 23:06:18.012343389 (13h26m17.601688427s) [upload=13938]: Done uploading segments... Aug 7 23:06:18.012365292 (13h26m17.601710319s) [upload=13938]: Upload finished. err: <nil> Aug 7 23:06:43.977164591 (13h26m43.566509578s) [upload=13946]: Starting upload Aug 7 23:06:43.977216861 (13h26m43.566561848s) [upload=13946]: Done starting upload Aug 7 23:06:43.977265654 (13h26m43.566610631s) [upload=13946]: Uploading segments... Aug 7 23:06:43.977299669 (13h26m43.566644646s) [upload=13946]: Got next segment. Inline: true Aug 7 23:06:43.977304398 (13h26m43.566649375s) [upload=13946]: Waiting for error group managing segments... Aug 7 23:06:43.977335879 (13h26m43.566680866s) [upload=13946]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 23:06:43.977342241 (13h26m43.566687228s) [upload=13946]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 23:06:43.977344626 (13h26m43.566689613s) [upload=13946]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 23:06:44.261001386 (13h26m43.850346383s) [upload=13946]: Done uploading segments... Aug 7 23:06:44.261036533 (13h26m43.85038152s) [upload=13946]: Upload finished. err: <nil> Aug 7 23:07:14.299823176 (13h27m13.889168163s) [upload=13956]: Starting upload Aug 7 23:07:14.299919721 (13h27m13.889264729s) [upload=13956]: Done starting upload Aug 7 23:07:14.300062905 (13h27m13.889407902s) [upload=13956]: Uploading segments... Aug 7 23:07:14.300136495 (13h27m13.889481482s) [upload=13956]: Got next segment. Inline: true Aug 7 23:07:14.300143029 (13h27m13.889488016s) [upload=13956]: Waiting for error group managing segments... Aug 7 23:07:14.300224825 (13h27m13.889569822s) [upload=13956]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 23:07:14.300231578 (13h27m13.889576565s) [upload=13956]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 23:07:14.300235044 (13h27m13.889580041s) [upload=13956]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 23:07:14.558772216 (13h27m14.148117193s) [upload=13956]: Done uploading segments... Aug 7 23:07:14.558788256 (13h27m14.148133243s) [upload=13956]: Upload finished. err: <nil> Aug 7 23:07:52.874987115 (13h27m52.464332102s) [upload=13971]: Starting upload Aug 7 23:07:52.875039626 (13h27m52.464384603s) [upload=13971]: Done starting upload Aug 7 23:07:52.875078921 (13h27m52.464423908s) [upload=13971]: Uploading segments... Aug 7 23:07:52.875111583 (13h27m52.46445657s) [upload=13971]: Got next segment. Inline: true Aug 7 23:07:52.875115000 (13h27m52.464459987s) [upload=13971]: Waiting for error group managing segments... Aug 7 23:07:52.875144226 (13h27m52.464489203s) [upload=13971]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 23:07:52.875147783 (13h27m52.46449277s) [upload=13971]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 23:07:52.875150268 (13h27m52.464495255s) [upload=13971]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 23:07:53.152523417 (13h27m52.741868404s) [upload=13971]: Done uploading segments... Aug 7 23:07:53.152567841 (13h27m52.741912828s) [upload=13971]: Upload finished. err: <nil> Aug 7 23:08:13.160315277 (13h28m12.749660264s) [upload=13978]: Starting upload Aug 7 23:08:13.160372306 (13h28m12.749717293s) [upload=13978]: Done starting upload Aug 7 23:08:13.160419306 (13h28m12.749764293s) [upload=13978]: Uploading segments... Aug 7 23:08:13.160446879 (13h28m12.749791866s) [upload=13978]: Got next segment. Inline: true Aug 7 23:08:13.160450095 (13h28m12.749795082s) [upload=13978]: Waiting for error group managing segments... Aug 7 23:08:13.160481565 (13h28m12.749826542s) [upload=13978]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 23:08:13.160485202 (13h28m12.749830189s) [upload=13978]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 23:08:13.160487416 (13h28m12.749832403s) [upload=13978]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 23:08:13.673421309 (13h28m13.262766296s) [upload=13978]: Done uploading segments... Aug 7 23:08:13.673439774 (13h28m13.262784761s) [upload=13978]: Upload finished. err: <nil> Aug 7 23:09:08.585854499 (13h29m8.175199486s) [upload=13993]: Starting upload Aug 7 23:09:08.585909965 (13h29m8.175254952s) [upload=13993]: Done starting upload Aug 7 23:09:08.585953217 (13h29m8.175298204s) [upload=13993]: Uploading segments... Aug 7 23:09:08.585980830 (13h29m8.175325817s) [upload=13993]: Got next segment. Inline: true Aug 7 23:09:08.585984206 (13h29m8.175329183s) [upload=13993]: Waiting for error group managing segments... Aug 7 23:09:08.586014684 (13h29m8.175359671s) [upload=13993]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 23:09:08.586018372 (13h29m8.175363359s) [upload=13993]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 23:09:08.586020767 (13h29m8.175365754s) [upload=13993]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 23:09:08.854829559 (13h29m8.444174546s) [upload=13993]: Done uploading segments... Aug 7 23:09:08.854846763 (13h29m8.44419175s) [upload=13993]: Upload finished. err: <nil> Aug 7 23:10:03.068961384 (13h30m2.658306371s) [upload=14012]: Starting upload Aug 7 23:10:03.069021258 (13h30m2.658366245s) [upload=14012]: Done starting upload Aug 7 23:10:03.069094389 (13h30m2.658439376s) [upload=14012]: Uploading segments... Aug 7 23:10:03.069128504 (13h30m2.658473491s) [upload=14012]: Got next segment. Inline: true Aug 7 23:10:03.069132782 (13h30m2.658477779s) [upload=14012]: Waiting for error group managing segments... Aug 7 23:10:03.069170935 (13h30m2.658515932s) [upload=14012]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 23:10:03.069175905 (13h30m2.658520892s) [upload=14012]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 23:10:03.069178650 (13h30m2.658523647s) [upload=14012]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 23:10:03.364714047 (13h30m2.954059064s) [upload=14012]: Done uploading segments... Aug 7 23:10:03.364761748 (13h30m2.954106745s) [upload=14012]: Upload finished. err: <nil> Aug 7 23:10:44.905335214 (13h30m44.494680201s) [upload=14022]: Starting upload Aug 7 23:10:44.905402192 (13h30m44.494747179s) [upload=14022]: Done starting upload Aug 7 23:10:44.905449983 (13h30m44.49479496s) [upload=14022]: Uploading segments... Aug 7 23:10:44.905479650 (13h30m44.494824628s) [upload=14022]: Got next segment. Inline: true Aug 7 23:10:44.905482787 (13h30m44.494827774s) [upload=14022]: Waiting for error group managing segments... Aug 7 23:10:44.905516160 (13h30m44.494861147s) [upload=14022]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 23:10:44.905519717 (13h30m44.494864694s) [upload=14022]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 23:10:44.905522222 (13h30m44.494867209s) [upload=14022]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 23:10:45.196197762 (13h30m44.785542749s) [upload=14022]: Done uploading segments... Aug 7 23:10:45.196218401 (13h30m44.785563389s) [upload=14022]: Upload finished. err: <nil> Aug 7 23:11:56.158319777 (13h31m55.747664764s) [upload=14041]: Starting upload Aug 7 23:11:56.158381214 (13h31m55.747726201s) [upload=14041]: Done starting upload Aug 7 23:11:56.158454435 (13h31m55.747799422s) [upload=14041]: Uploading segments... Aug 7 23:11:56.158485985 (13h31m55.747830972s) [upload=14041]: Got next segment. Inline: true Aug 7 23:11:56.158489752 (13h31m55.747834739s) [upload=14041]: Waiting for error group managing segments... Aug 7 23:11:56.158533225 (13h31m55.747878212s) [upload=14041]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 23:11:56.158536933 (13h31m55.74788192s) [upload=14041]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 23:11:56.158539077 (13h31m55.747884064s) [upload=14041]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 23:12:55.512693945 (13h32m55.102038922s) [upload=14041]: Done uploading segments... Aug 7 23:12:55.512714414 (13h32m55.102059401s) [upload=14041]: Upload finished. err: metaclient: context deadline exceeded
Duration: 1m0.001171849s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)antiblip:
2 out of 60 is failed (threshold=3) [XX..........................................................] duration:
5.600510448s
FAIL (2025-08-07 23:12:50 UTC)
orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded
Duration: 5.600533484s
core
uplink
us1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
6 out of 60 is failed (threshold=3) [XXXX.X.....................................................X] duration:
1m0.000905186s
FAIL (2025-08-07 23:12:50 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1m0.00093353s
core
uplink
us-select-1
us1-placement-20
|
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)antiblip:
1 out of 60 is failed (threshold=3) [.................X..........................................] duration:
645.905773ms target:
https://us1.storj.io/login
FAIL (2025-08-07 23:12:47 UTC)
orbiter warning: Get "https://us1.storj.io/login": dial tcp 34.150.199.48:443: connect: connection refused
Duration: 645.928245ms
core
satellite
us1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
3 out of 60 is failed (threshold=3) [.X.................X...................X....................] duration:
6.333410257s
FAIL (2025-08-07 23:12:42 UTC)
orbiter warning:
operation: ListObjects
======ERROR======
unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/4M
======LOGS======
172565s) [upload=13803]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Aug 7 22:58:33.457032807 (13h18m33.046377794s) [seg_pos={0 0}, upload=13803]: Begin upload segment...
Aug 7 22:58:33.457116296 (13h18m33.046461283s) [seg_pos={0 0}, upload=13803]: Done begin upload segment.
Aug 7 22:58:33.457172754 (13h18m33.046517741s) [noise=true , address=79.127.201.215:4349 , piece_id=XDVQZBXGHX65RHZBWUQRP3BAHRPRHOCWXXOLSJDQ2XDEYNFQSNAA, seg_pos={0 0}, upload=13803]: Uploading piece...
Aug 7 22:58:33.457191480 (13h18m33.046536468s) [noise=true , address=104.36.82.4:4818 , piece_id=T4K2OGU3TC27H5LE4CTDLAOYR6DOP3S2PBVV7KUX4KKVFNMZNXBQ, seg_pos={0 0}, upload=13803]: Uploading piece...
Aug 7 22:58:33.457735842 (13h18m33.047080829s) [noise=true , address=104.36.82.8:4003 , piece_id=C4M64YUD5J6CFLATRLBVPZY7P67VD5R2WQ4LPLIV6CNRAPDRDSMQ, seg_pos={0 0}, upload=13803]: Uploading piece...
Aug 7 22:58:33.457740962 (13h18m33.047085939s) [noise=true , address=104.36.82.15:4270 , piece_id=2MTKGO66NWBUQXDMTN4NRYKIWHT6GKRW3LAHTZHLOVBGE7OINGZA, seg_pos={0 0}, upload=13803]: Uploading piece...
Aug 7 22:58:33.458108976 (13h18m33.047453963s) [noise=true , address=104.36.82.39:4732 , piece_id=EJNQS3XUYBVPZ6SLENNNLQBJ5OEIF53XSM645SAV3DGTFQ7EK3BA, seg_pos={0 0}, upload=13803]: Uploading piece...
Aug 7 22:58:33.458181885 (13h18m33.047526872s) [noise=true , address=104.36.82.62:4430 , piece_id=SCXOGWUIS2KIBEWOQWYMVBS6EV3URUYFKYMV5PQV5HFPK5EGSRMQ, seg_pos={0 0}, upload=13803]: Uploading piece...
Aug 7 22:58:33.458456691 (13h18m33.047801678s) [noise=true , address=45.91.116.14:4221 , piece_id=J54CXKHJH2KOUV3MPT36F6JHOI67GMGCMSOKHNJUDL5I7NTRLCBA, seg_pos={0 0}, upload=13803]: Uploading piece...
Aug 7 22:58:33.458607449 (13h18m33.047952436s) [noise=true , address=104.36.82.44:4895 , piece_id=GWHMQEBTZ4MB4Z6AZUAUJGFGAQ7L567F7INSONYCQCSCKK3CC7JA, seg_pos={0 0}, upload=13803]: Uploading piece...
Aug 7 22:58:33.458794698 (13h18m33.048139685s) [noise=true , address=64.49.13.170:4719 , piece_id=5AWWDAKHOKZ4FJWUZTH4PM4XPIQA2L2MKX4L7XSBNVZ3EURMK2VA, seg_pos={0 0}, upload=13803]: Uploading piece...
Aug 7 22:58:33.459301818 (13h18m33.048646795s) [noise=true , address=45.91.116.4:4955 , piece_id=WYMA4QA3QF55ZHZAUTMHWPLXJCSBOTN3NYLRJUWEMRFVUKR47VLA, seg_pos={0 0}, upload=13803]: Uploading piece...
Aug 7 22:58:33.459553139 (13h18m33.048898126s) [noise=true , address=104.36.82.55:4202 , piece_id=UFXSOKDOHBYJYXJIKWF6AKQKXT6VLTVLKKCRBLYFOXQ554LF3OMA, seg_pos={0 0}, upload=13803]: Uploading piece...
Aug 7 22:58:33.459845177 (13h18m33.049190164s) [noise=true , address=104.36.82.46:4491 , piece_id=FE5LA53DP4JQFXYYMB7NV36E5I62EWJMFQ5ZVLUDYQX5QKLKV7MQ, seg_pos={0 0}, upload=13803]: Uploading piece...
Aug 7 22:58:33.460146223 (13h18m33.04949121s) [noise=true , address=104.36.82.45:4907 , piece_id=G33PUTRJPQZ2FHMYPFED3W6X6FOEDP5C7D572K7LADQYSFLEK2HQ, seg_pos={0 0}, upload=13803]: Uploading piece...
Aug 7 22:58:33.460635710 (13h18m33.049980697s) [noise=true , address=104.36.82.55:4205 , piece_id=VK2NPARLAXL5YR5XBVABX4YBAACSXXBUXUJJJAUTPD7XG7EBNFCQ, seg_pos={0 0}, upload=13803]: Uploading piece...
Aug 7 22:58:33.460916247 (13h18m33.050261234s) [noise=true , address=45.91.116.8:4068 , piece_id=UBEDD44644KW5VOQDYO4KSXMWOL33OQCYZ5UTY7RZ662RUMFVOVQ, seg_pos={0 0}, upload=13803]: Uploading piece...
Aug 7 22:58:33.461154122 (13h18m33.050499099s) [noise=true , address=45.91.116.6:4977 , piece_id=7LDMXXPDPT4Q6JCT6QPOIZBANDZA3RHBYBRZTGOWRBKZHLZEWHEQ, seg_pos={0 0}, upload=13803]: Uploading piece...
Aug 7 22:58:33.461485957 (13h18m33.050830934s) [noise=true , address=104.36.82.12:4162 , piece_id=GCO6IE3FJAO7V3XCNWB4IQQVN2CSX3IRSR4QTZO7ZH6AOB27CYJQ, seg_pos={0 0}, upload=13803]: Uploading piece...
Aug 7 22:58:33.461747909 (13h18m33.051092896s) [noise=true , address=104.36.82.23:4472 , piece_id=BIB3GU4R4VW7CCGT7S4VCDYJSV2I6FV4NKLLSCZZJ5QDOJGYHJKQ, seg_pos={0 0}, upload=13803]: Uploading piece...
Aug 7 22:58:34.610233566 (13h18m34.199578553s) [upload=13803]: Next returned nil segment
Aug 7 22:58:34.610251881 (13h18m34.199596868s) [upload=13803]: Waiting for error group managing segments...
Aug 7 22:58:34.912103574 (13h18m34.501448551s) [noise=true , address=64.49.13.170:4719 , piece_id=5AWWDAKHOKZ4FJWUZTH4PM4XPIQA2L2MKX4L7XSBNVZ3EURMK2VA, seg_pos={0 0}, upload=13803]: Done uploading piece. err: <nil>
Aug 7 22:58:35.384823542 (13h18m34.974168529s) [noise=true , address=104.36.82.8:4003 , piece_id=C4M64YUD5J6CFLATRLBVPZY7P67VD5R2WQ4LPLIV6CNRAPDRDSMQ, seg_pos={0 0}, upload=13803]: Done uploading piece. err: <nil>
Aug 7 22:58:35.418586980 (13h18m35.007931967s) [noise=true , address=79.127.201.215:4349 , piece_id=XDVQZBXGHX65RHZBWUQRP3BAHRPRHOCWXXOLSJDQ2XDEYNFQSNAA, seg_pos={0 0}, upload=13803]: Done uploading piece. err: <nil>
Aug 7 22:58:35.458066756 (13h18m35.047411743s) [noise=true , address=104.36.82.44:4895 , piece_id=GWHMQEBTZ4MB4Z6AZUAUJGFGAQ7L567F7INSONYCQCSCKK3CC7JA, seg_pos={0 0}, upload=13803]: Done uploading piece. err: <nil>
Aug 7 22:58:35.477223478 (13h18m35.066568455s) [noise=true , address=104.36.82.45:4907 , piece_id=G33PUTRJPQZ2FHMYPFED3W6X6FOEDP5C7D572K7LADQYSFLEK2HQ, seg_pos={0 0}, upload=13803]: Done uploading piece. err: <nil>
Aug 7 22:58:35.499730379 (13h18m35.089075366s) [noise=true , address=104.36.82.39:4732 , piece_id=EJNQS3XUYBVPZ6SLENNNLQBJ5OEIF53XSM645SAV3DGTFQ7EK3BA, seg_pos={0 0}, upload=13803]: Done uploading piece. err: <nil>
Aug 7 22:58:35.614098456 (13h18m35.203443443s) [noise=true , address=104.36.82.46:4491 , piece_id=FE5LA53DP4JQFXYYMB7NV36E5I62EWJMFQ5ZVLUDYQX5QKLKV7MQ, seg_pos={0 0}, upload=13803]: Done uploading piece. err: <nil>
Aug 7 22:58:35.748129262 (13h18m35.33747425s) [noise=true , address=104.36.82.62:4430 , piece_id=SCXOGWUIS2KIBEWOQWYMVBS6EV3URUYFKYMV5PQV5HFPK5EGSRMQ, seg_pos={0 0}, upload=13803]: Done uploading piece. err: <nil>
Aug 7 22:58:35.755387388 (13h18m35.344732376s) [noise=true , address=104.36.82.12:4162 , piece_id=GCO6IE3FJAO7V3XCNWB4IQQVN2CSX3IRSR4QTZO7ZH6AOB27CYJQ, seg_pos={0 0}, upload=13803]: Done uploading piece. err: <nil>
Aug 7 22:58:35.958468347 (13h18m35.547813334s) [noise=true , address=45.91.116.8:4068 , piece_id=UBEDD44644KW5VOQDYO4KSXMWOL33OQCYZ5UTY7RZ662RUMFVOVQ, seg_pos={0 0}, upload=13803]: Done uploading piece. err: <nil>
Aug 7 22:58:36.044513974 (13h18m35.633858961s) [noise=true , address=45.91.116.6:4977 , piece_id=7LDMXXPDPT4Q6JCT6QPOIZBANDZA3RHBYBRZTGOWRBKZHLZEWHEQ, seg_pos={0 0}, upload=13803]: Done uploading piece. err: <nil>
Aug 7 22:58:36.091808572 (13h18m35.681153549s) [noise=true , address=45.91.116.14:4221 , piece_id=J54CXKHJH2KOUV3MPT36F6JHOI67GMGCMSOKHNJUDL5I7NTRLCBA, seg_pos={0 0}, upload=13803]: Done uploading piece. err: <nil>
Aug 7 22:58:36.096744555 (13h18m35.686089543s) [noise=true , address=104.36.82.55:4205 , piece_id=VK2NPARLAXL5YR5XBVABX4YBAACSXXBUXUJJJAUTPD7XG7EBNFCQ, seg_pos={0 0}, upload=13803]: Done uploading piece. err: <nil>
Aug 7 22:58:36.171642074 (13h18m35.760987061s) [noise=true , address=104.36.82.55:4202 , piece_id=UFXSOKDOHBYJYXJIKWF6AKQKXT6VLTVLKKCRBLYFOXQ554LF3OMA, seg_pos={0 0}, upload=13803]: Done uploading piece. err: <nil>
Aug 7 22:58:36.171668725 (13h18m35.761013712s) [seg_pos={0 0}, upload=13803]: Segment reached optimal threshold of 14 pieces.
Aug 7 22:58:36.171958509 (13h18m35.761303496s) [noise=true , address=104.36.82.4:4818 , piece_id=T4K2OGU3TC27H5LE4CTDLAOYR6DOP3S2PBVV7KUX4KKVFNMZNXBQ, seg_pos={0 0}, upload=13803]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1235eFkz7CFwwiM8UUZU1vnqDneEic7RpFzeTU46gvTMZWhhox8): protocol: read tcp 188.166.225.170:34616->104.36.82.4:4818: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 7 22:58:36.172074250 (13h18m35.761419227s) [noise=true , address=45.91.116.4:4955 , piece_id=WYMA4QA3QF55ZHZAUTMHWPLXJCSBOTN3NYLRJUWEMRFVUKR47VLA, seg_pos={0 0}, upload=13803]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VULW731AQGUMf4KLbhPUDNAEiW2AxaXJQYHzqiPiBuWG17L2B): protocol: write tcp 188.166.225.170:39210->45.91.116.4:4955: use of closed network connection; write tcp 188.166.225.170:39210->45.91.116.4:4955: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39210->45.91.116.4:4955: use of closed network connection
Aug 7 22:58:36.172112223 (13h18m35.7614572s) [noise=true , address=104.36.82.15:4270 , piece_id=2MTKGO66NWBUQXDMTN4NRYKIWHT6GKRW3LAHTZHLOVBGE7OINGZA, seg_pos={0 0}, upload=13803]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nkuzy3vxeLdHhVs73Nceje7qXDiGJ1zEhpGgGKSk3cKb7vJbB): protocol: rpc: dial tcp 104.36.82.15:4270: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 7 22:58:36.172191986 (13h18m35.761536973s) [noise=true , address=104.36.82.23:4472 , piece_id=BIB3GU4R4VW7CCGT7S4VCDYJSV2I6FV4NKLLSCZZJ5QDOJGYHJKQ, seg_pos={0 0}, upload=13803]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12otbB1RmPUKSKo813hrKPQVAyeqmhjhApxi6ndtHBCfn9bGGct): protocol: write tcp 188.166.225.170:47876->104.36.82.23:4472: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 7 22:58:36.172216163 (13h18m35.76156115s) [seg_pos={0 0}, upload=13803]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Aug 7 22:58:36.172285585 (13h18m35.761630572s) [upload=13803]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Aug 7 22:58:36.172289833 (13h18m35.76163482s) [upload=13803]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Aug 7 22:58:36.490569647 (13h18m36.079914634s) [upload=13803]: Done uploading segments...
Aug 7 22:58:36.490587652 (13h18m36.079932639s) [upload=13803]: Upload finished. err: <nil>
Duration: 6.333431758s
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)antiblip:
1 out of 60 is failed (threshold=3) [...........................................................X] duration:
869.489164ms
FAIL (2025-08-07 23:12:37 UTC)
orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== segments... Aug 7 23:06:49.229716945 (13h26m48.819061932s) [upload=13950]: Upload finished. err: <nil> Aug 7 23:07:42.104963855 (13h27m41.694308842s) [upload=13968]: Starting upload Aug 7 23:07:42.105031205 (13h27m41.694376192s) [upload=13968]: Done starting upload Aug 7 23:07:42.105104063 (13h27m41.69444905s) [upload=13968]: Uploading segments... Aug 7 23:07:42.105139652 (13h27m41.694484639s) [upload=13968]: Got next segment. Inline: true Aug 7 23:07:42.105143259 (13h27m41.694488246s) [upload=13968]: Waiting for error group managing segments... Aug 7 23:07:42.105190129 (13h27m41.694535126s) [upload=13968]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 23:07:42.105201120 (13h27m41.694546127s) [upload=13968]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 23:07:42.105207281 (13h27m41.694552278s) [upload=13968]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 23:07:42.417069033 (13h27m42.00641402s) [upload=13968]: Done uploading segments... Aug 7 23:07:42.417104130 (13h27m42.006449117s) [upload=13968]: Upload finished. err: <nil> Aug 7 23:08:34.230557646 (13h28m33.819902633s) [upload=13982]: Starting upload Aug 7 23:08:34.230604215 (13h28m33.819949193s) [upload=13982]: Done starting upload Aug 7 23:08:34.230655173 (13h28m33.82000016s) [upload=13982]: Uploading segments... Aug 7 23:08:34.230683928 (13h28m33.820028905s) [upload=13982]: Got next segment. Inline: true Aug 7 23:08:34.230687385 (13h28m33.820032362s) [upload=13982]: Waiting for error group managing segments... Aug 7 23:08:34.230719175 (13h28m33.820064162s) [upload=13982]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 23:08:34.230722832 (13h28m33.820067819s) [upload=13982]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 23:08:34.230725046 (13h28m33.820070033s) [upload=13982]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 23:08:34.528494026 (13h28m34.117839013s) [upload=13982]: Done uploading segments... Aug 7 23:08:34.528512322 (13h28m34.117857309s) [upload=13982]: Upload finished. err: <nil> Aug 7 23:09:05.508570966 (13h29m5.097915953s) [upload=13989]: Starting upload Aug 7 23:09:05.508638426 (13h29m5.097983423s) [upload=13989]: Done starting upload Aug 7 23:09:05.508701747 (13h29m5.098046764s) [upload=13989]: Uploading segments... Aug 7 23:09:05.508743536 (13h29m5.098088533s) [upload=13989]: Got next segment. Inline: true Aug 7 23:09:05.508749488 (13h29m5.098094515s) [upload=13989]: Waiting for error group managing segments... Aug 7 23:09:05.508795345 (13h29m5.098140362s) [upload=13989]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 23:09:05.508801848 (13h29m5.098146856s) [upload=13989]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 23:09:05.508805736 (13h29m5.098150733s) [upload=13989]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 23:09:05.806325549 (13h29m5.395670536s) [upload=13989]: Done uploading segments... Aug 7 23:09:05.806343342 (13h29m5.395688329s) [upload=13989]: Upload finished. err: <nil> Aug 7 23:09:38.090257337 (13h29m37.679602324s) [upload=14002]: Starting upload Aug 7 23:09:38.090300970 (13h29m37.679645957s) [upload=14002]: Done starting upload Aug 7 23:09:38.090372567 (13h29m37.679717554s) [upload=14002]: Uploading segments... Aug 7 23:09:38.090406682 (13h29m37.67975168s) [upload=14002]: Got next segment. Inline: true Aug 7 23:09:38.090410099 (13h29m37.679755076s) [upload=14002]: Waiting for error group managing segments... Aug 7 23:09:38.090443282 (13h29m37.679788269s) [upload=14002]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 23:09:38.090447210 (13h29m37.679792198s) [upload=14002]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 23:09:38.090449415 (13h29m37.679794392s) [upload=14002]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 23:09:38.661400996 (13h29m38.250746043s) [upload=14002]: Done uploading segments... Aug 7 23:09:38.661423068 (13h29m38.250768135s) [upload=14002]: Upload finished. err: <nil> Aug 7 23:09:58.827549308 (13h29m58.416894295s) [upload=14009]: Starting upload Aug 7 23:09:58.827598442 (13h29m58.416943429s) [upload=14009]: Done starting upload Aug 7 23:09:58.827644020 (13h29m58.416989007s) [upload=14009]: Uploading segments... Aug 7 23:09:58.827674438 (13h29m58.417019425s) [upload=14009]: Got next segment. Inline: true Aug 7 23:09:58.827678135 (13h29m58.417023112s) [upload=14009]: Waiting for error group managing segments... Aug 7 23:09:58.827711559 (13h29m58.417056546s) [upload=14009]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 23:09:58.827715526 (13h29m58.417060513s) [upload=14009]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 23:09:58.827719804 (13h29m58.417064781s) [upload=14009]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 23:09:59.099262618 (13h29m58.688607605s) [upload=14009]: Done uploading segments... Aug 7 23:09:59.099279320 (13h29m58.688624297s) [upload=14009]: Upload finished. err: <nil> Aug 7 23:10:50.147486507 (13h30m49.736831494s) [upload=14023]: Starting upload Aug 7 23:10:50.147530131 (13h30m49.736875118s) [upload=14023]: Done starting upload Aug 7 23:10:50.147587851 (13h30m49.736932868s) [upload=14023]: Uploading segments... Aug 7 23:10:50.147650551 (13h30m49.736995568s) [upload=14023]: Got next segment. Inline: true Aug 7 23:10:50.147659338 (13h30m49.737004355s) [upload=14023]: Waiting for error group managing segments... Aug 7 23:10:50.147719804 (13h30m49.737064811s) [upload=14023]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 23:10:50.147727708 (13h30m49.737072746s) [upload=14023]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 23:10:50.147734752 (13h30m49.737079739s) [upload=14023]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 23:10:50.421404977 (13h30m50.010749964s) [upload=14023]: Done uploading segments... Aug 7 23:10:50.421427519 (13h30m50.010772546s) [upload=14023]: Upload finished. err: <nil> Aug 7 23:11:30.587306004 (13h31m30.176650991s) [upload=14032]: Starting upload Aug 7 23:11:30.587348937 (13h31m30.176693924s) [upload=14032]: Done starting upload Aug 7 23:11:30.587398742 (13h31m30.17674373s) [upload=14032]: Uploading segments... Aug 7 23:11:30.587432447 (13h31m30.176777434s) [upload=14032]: Got next segment. Inline: true Aug 7 23:11:30.587437326 (13h31m30.176782313s) [upload=14032]: Waiting for error group managing segments... Aug 7 23:11:30.587469467 (13h31m30.176814454s) [upload=14032]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 23:11:30.587472974 (13h31m30.176817951s) [upload=14032]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 23:11:30.587475168 (13h31m30.176820155s) [upload=14032]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 23:11:30.892529848 (13h31m30.481874835s) [upload=14032]: Done uploading segments... Aug 7 23:11:30.892541890 (13h31m30.481886877s) [upload=14032]: Upload finished. err: <nil> Aug 7 23:11:48.390915893 (13h31m47.98026088s) [upload=14038]: Starting upload Aug 7 23:11:48.390965198 (13h31m47.980310185s) [upload=14038]: Done starting upload Aug 7 23:11:48.391014242 (13h31m47.980359229s) [upload=14038]: Uploading segments... Aug 7 23:11:48.391054007 (13h31m47.980399004s) [upload=14038]: Got next segment. Inline: true Aug 7 23:11:48.391057494 (13h31m47.980402471s) [upload=14038]: Waiting for error group managing segments... Aug 7 23:11:48.391089254 (13h31m47.980434231s) [upload=14038]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 23:11:48.391093213 (13h31m47.98043819s) [upload=14038]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 23:11:48.391095457 (13h31m47.980440444s) [upload=14038]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 23:11:48.701950724 (13h31m48.291295711s) [upload=14038]: Done uploading segments... Aug 7 23:11:48.701969449 (13h31m48.291314436s) [upload=14038]: Upload finished. err: <nil> Aug 7 23:12:06.430245154 (13h32m6.019590141s) [upload=14048]: Starting upload Aug 7 23:12:06.430293076 (13h32m6.019638063s) [upload=14048]: Done starting upload Aug 7 23:12:06.430361757 (13h32m6.019706745s) [upload=14048]: Uploading segments... Aug 7 23:12:06.430459223 (13h32m6.01980421s) [upload=14048]: Got next segment. Inline: true Aug 7 23:12:06.430464543 (13h32m6.01980953s) [upload=14048]: Waiting for error group managing segments... Aug 7 23:12:06.430527555 (13h32m6.019872542s) [upload=14048]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 23:12:06.430533796 (13h32m6.019878783s) [upload=14048]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 23:12:06.430536692 (13h32m6.019881679s) [upload=14048]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 23:12:06.729952062 (13h32m6.319297049s) [upload=14048]: Done uploading segments... Aug 7 23:12:06.729968844 (13h32m6.319313832s) [upload=14048]: Upload finished. err: <nil> Aug 7 23:12:37.551701338 (13h32m37.141046325s) [upload=14055]: Starting upload Aug 7 23:12:37.551746985 (13h32m37.141091972s) [upload=14055]: Done starting upload Aug 7 23:12:37.551807301 (13h32m37.141152288s) [upload=14055]: Uploading segments... Aug 7 23:12:37.551914826 (13h32m37.141259833s) [upload=14055]: Got next segment. Inline: true Aug 7 23:12:37.551921779 (13h32m37.141266766s) [upload=14055]: Waiting for error group managing segments... Aug 7 23:12:37.551977145 (13h32m37.141322122s) [upload=14055]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 23:12:37.551984059 (13h32m37.141329086s) [upload=14055]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 23:12:37.551988918 (13h32m37.141333936s) [upload=14055]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 23:12:37.767068918 (13h32m37.356413905s) [upload=14055]: Done uploading segments... Aug 7 23:12:37.767104365 (13h32m37.356449342s) [upload=14055]: Upload finished. err: metaclient: Too Many Requests
Duration: 869.509682ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)antiblip:
2 out of 60 is failed (threshold=3) [X.........................................................X.] duration:
1.862433341s
FAIL (2025-08-07 23:12:33 UTC)
orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded
Duration: 1.862450274s
core
uplink
us1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)antiblip:
2 out of 60 is failed (threshold=3) [.X.........................................................X] duration:
30.001334303s
FAIL (2025-08-07 23:12:31 UTC)
orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded
Duration: 30.001355172s
core
uplink
us1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline)antiblip:
2 out of 60 is failed (threshold=3) [.X.X........................................................] duration:
2.277529114s
FAIL (2025-08-07 23:12:28 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.277555745s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)antiblip:
3 out of 60 is failed (threshold=3) [XXX.........................................................] duration:
5.58092813s
FAIL (2025-08-07 23:12:09 UTC)
orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded
Duration: 5.580957175s
core
uplink
us1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [X...X.......................................................] duration:
5.834707077s
FAIL (2025-08-07 23:12:08 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.834725103s
edge
gateway
us-select-1
|
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)antiblip:
1 out of 60 is failed (threshold=3) [..................X.........................................] duration:
656.436632ms target:
https://us1.storj.io/login
FAIL (2025-08-07 23:12:06 UTC)
orbiter warning: Get "https://us1.storj.io/login": dial tcp 34.150.199.48:443: connect: connection refused
Duration: 656.479553ms
core
satellite
us1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [XXX.......................................................X.] duration:
6.407310048s
FAIL (2025-08-07 23:11:58 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: encryption: context deadline exceeded
======LOGS======
30m19.119835718s) [upload=14014]: Got next segment. Inline: false
Aug 7 23:10:19.530521730 (13h30m19.119866718s) [upload=14014]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Aug 7 23:10:19.530526099 (13h30m19.119871086s) [upload=14014]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Aug 7 23:10:19.769147767 (13h30m19.358492754s) [seg_pos={0 0}, upload=14014]: Begin upload segment...
Aug 7 23:10:19.769196962 (13h30m19.358541949s) [seg_pos={0 0}, upload=14014]: Done begin upload segment.
Aug 7 23:10:19.769217901 (13h30m19.358562878s) [noise=true , address=77.90.31.19:4842 , piece_id=QUFDMBB7C5KALQOTIOQE6OOVFB3HTL23IPXC3SNT7TPUPBMENIYQ, seg_pos={0 0}, upload=14014]: Uploading piece...
Aug 7 23:10:19.769241967 (13h30m19.358586954s) [noise=true , address=45.91.116.9:4263 , piece_id=75NMKEGELJDZFSDE6L7BUOXIOCUZVZYZSCQFXL5AWFFN4L3B432Q, seg_pos={0 0}, upload=14014]: Uploading piece...
Aug 7 23:10:19.769614190 (13h30m19.358959177s) [noise=true , address=104.36.82.15:4257 , piece_id=DRDDV5DLUAAT2TFHHH4GMNHECM3LQHNH24Y2ZJALUXRCXL6QNRRA, seg_pos={0 0}, upload=14014]: Uploading piece...
Aug 7 23:10:19.769658985 (13h30m19.359003972s) [noise=true , address=77.90.31.16:4733 , piece_id=GRL5LY7JWUUYQFQ5CUZXDOZIYEG4GWWMVN76YTFPGKQOAP22XXNA, seg_pos={0 0}, upload=14014]: Uploading piece...
Aug 7 23:10:19.769909134 (13h30m19.359254111s) [noise=true , address=45.91.116.7:4011 , piece_id=ABOGCAOR5HL2MGGHQHXS3O3WLSEXD2B4UW4WGDJMDQYK5KBK2MSA, seg_pos={0 0}, upload=14014]: Uploading piece...
Aug 7 23:10:19.769942408 (13h30m19.359287395s) [noise=true , address=77.90.31.18:4807 , piece_id=7SNCXGGPT7RTAPPOX3LE526FIQU5G2VYYRJDFIPDFLA6ZPTFX7PQ, seg_pos={0 0}, upload=14014]: Uploading piece...
Aug 7 23:10:19.770178389 (13h30m19.359523366s) [noise=true , address=104.36.82.2:4730 , piece_id=3B4SQCLJ6FI6CRNRWCJRNA2CX4X5KZZELCFGOBIWVW7NQLLWKBRQ, seg_pos={0 0}, upload=14014]: Uploading piece...
Aug 7 23:10:19.770446231 (13h30m19.359791208s) [noise=true , address=104.36.82.8:4969 , piece_id=2EHLHQAWG3TA545TZV3QUAMRTLWBAYP2KQXXY7XSITAOUI5BJV7Q, seg_pos={0 0}, upload=14014]: Uploading piece...
Aug 7 23:10:19.770673426 (13h30m19.360018403s) [noise=true , address=45.91.116.11:4119 , piece_id=QMJWOHYHCVIVE7AVUUE3YWFTPZXQ2SK4QAAAS2ILP7J5OSVU2XTA, seg_pos={0 0}, upload=14014]: Uploading piece...
Aug 7 23:10:19.770876775 (13h30m19.360221763s) [noise=true , address=77.90.31.21:4877 , piece_id=DA742EGVEWSYO6PQZR7Z2IJTXFIBBYN5EVNKOGE2TZCM5RWOMTRA, seg_pos={0 0}, upload=14014]: Uploading piece...
Aug 7 23:10:19.771065907 (13h30m19.360410884s) [noise=true , address=77.90.31.14:4661 , piece_id=DX7EKUM3I6BRROMSBR3QMWOPBRXW3QCKDXL7RSCZ4MYGAOE62HEA, seg_pos={0 0}, upload=14014]: Uploading piece...
Aug 7 23:10:19.771256863 (13h30m19.36060185s) [noise=true , address=104.36.82.16:4295 , piece_id=NI7VPQGP7DXVVO6H7URWAGUWTHAIAQCXDZYWDMCD2EPIPLFDCGFQ, seg_pos={0 0}, upload=14014]: Uploading piece...
Aug 7 23:10:19.771456625 (13h30m19.360801612s) [noise=true , address=45.91.116.5:4298 , piece_id=ESZYBPWHWL2MOTBPVHK42M6CADJZNPJZBZXZLTYIUZHCH5XV5ZCQ, seg_pos={0 0}, upload=14014]: Uploading piece...
Aug 7 23:10:19.771646057 (13h30m19.360991034s) [noise=true , address=45.91.116.6:4973 , piece_id=MW66WBZULRRDBTQSZCHAOIC44YVAFFO2EKDEZLHF6QVZ55CHTHPQ, seg_pos={0 0}, upload=14014]: Uploading piece...
Aug 7 23:10:19.771840640 (13h30m19.361185627s) [noise=true , address=45.91.116.4:4939 , piece_id=XRYW6ZXHU4BCYNAKNWFAYBPJDOXQCAGTEIUJ7KJKNLVES73LD43Q, seg_pos={0 0}, upload=14014]: Uploading piece...
Aug 7 23:10:19.772068876 (13h30m19.361413863s) [noise=true , address=45.91.116.8:4045 , piece_id=XLOA7HZ7FRG5H5HMLMECR2PCSVOGD6XI5NS5HSTOQTZZ5KIMXIRA, seg_pos={0 0}, upload=14014]: Uploading piece...
Aug 7 23:10:20.020996815 (13h30m19.610341802s) [noise=true , address=77.90.31.16:4733 , piece_id=GRL5LY7JWUUYQFQ5CUZXDOZIYEG4GWWMVN76YTFPGKQOAP22XXNA, seg_pos={0 0}, upload=14014]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: write tcp 188.166.225.170:37310->77.90.31.16:4733: write: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 7 23:10:21.532433287 (13h30m21.121778274s) [upload=14014]: Next returned nil segment
Aug 7 23:10:21.532449418 (13h30m21.121794405s) [upload=14014]: Waiting for error group managing segments...
Aug 7 23:10:21.811529953 (13h30m21.40087494s) [noise=true , address=77.90.31.21:4877 , piece_id=DA742EGVEWSYO6PQZR7Z2IJTXFIBBYN5EVNKOGE2TZCM5RWOMTRA, seg_pos={0 0}, upload=14014]: Done uploading piece. err: <nil>
Aug 7 23:10:21.854297295 (13h30m21.443642272s) [noise=true , address=77.90.31.18:4807 , piece_id=7SNCXGGPT7RTAPPOX3LE526FIQU5G2VYYRJDFIPDFLA6ZPTFX7PQ, seg_pos={0 0}, upload=14014]: Done uploading piece. err: <nil>
Aug 7 23:10:21.921242465 (13h30m21.510587452s) [noise=true , address=104.36.82.8:4969 , piece_id=2EHLHQAWG3TA545TZV3QUAMRTLWBAYP2KQXXY7XSITAOUI5BJV7Q, seg_pos={0 0}, upload=14014]: Done uploading piece. err: <nil>
Aug 7 23:10:21.947359331 (13h30m21.536704318s) [noise=true , address=45.91.116.9:4263 , piece_id=75NMKEGELJDZFSDE6L7BUOXIOCUZVZYZSCQFXL5AWFFN4L3B432Q, seg_pos={0 0}, upload=14014]: Done uploading piece. err: <nil>
Aug 7 23:10:21.969454293 (13h30m21.55879928s) [noise=true , address=77.90.31.19:4842 , piece_id=QUFDMBB7C5KALQOTIOQE6OOVFB3HTL23IPXC3SNT7TPUPBMENIYQ, seg_pos={0 0}, upload=14014]: Done uploading piece. err: <nil>
Aug 7 23:10:22.042768958 (13h30m21.632113936s) [noise=true , address=45.91.116.5:4298 , piece_id=ESZYBPWHWL2MOTBPVHK42M6CADJZNPJZBZXZLTYIUZHCH5XV5ZCQ, seg_pos={0 0}, upload=14014]: Done uploading piece. err: <nil>
Aug 7 23:10:22.050439524 (13h30m21.639784501s) [noise=true , address=45.91.116.7:4011 , piece_id=ABOGCAOR5HL2MGGHQHXS3O3WLSEXD2B4UW4WGDJMDQYK5KBK2MSA, seg_pos={0 0}, upload=14014]: Done uploading piece. err: <nil>
Aug 7 23:10:22.063864939 (13h30m21.653209926s) [noise=true , address=45.91.116.8:4045 , piece_id=XLOA7HZ7FRG5H5HMLMECR2PCSVOGD6XI5NS5HSTOQTZZ5KIMXIRA, seg_pos={0 0}, upload=14014]: Done uploading piece. err: <nil>
Aug 7 23:10:22.108762327 (13h30m21.698107314s) [noise=true , address=45.91.116.11:4119 , piece_id=QMJWOHYHCVIVE7AVUUE3YWFTPZXQ2SK4QAAAS2ILP7J5OSVU2XTA, seg_pos={0 0}, upload=14014]: Done uploading piece. err: <nil>
Aug 7 23:10:22.196386158 (13h30m21.785731146s) [noise=true , address=104.36.82.16:4295 , piece_id=NI7VPQGP7DXVVO6H7URWAGUWTHAIAQCXDZYWDMCD2EPIPLFDCGFQ, seg_pos={0 0}, upload=14014]: Done uploading piece. err: <nil>
Aug 7 23:10:22.220196592 (13h30m21.809541579s) [noise=true , address=45.91.116.6:4973 , piece_id=MW66WBZULRRDBTQSZCHAOIC44YVAFFO2EKDEZLHF6QVZ55CHTHPQ, seg_pos={0 0}, upload=14014]: Done uploading piece. err: <nil>
Aug 7 23:10:22.247094513 (13h30m21.8364395s) [noise=true , address=45.91.116.4:4939 , piece_id=XRYW6ZXHU4BCYNAKNWFAYBPJDOXQCAGTEIUJ7KJKNLVES73LD43Q, seg_pos={0 0}, upload=14014]: Done uploading piece. err: <nil>
Aug 7 23:10:22.380553005 (13h30m21.969897992s) [noise=true , address=77.90.31.14:4661 , piece_id=DX7EKUM3I6BRROMSBR3QMWOPBRXW3QCKDXL7RSCZ4MYGAOE62HEA, seg_pos={0 0}, upload=14014]: Done uploading piece. err: <nil>
Aug 7 23:10:22.380581249 (13h30m21.969926236s) [seg_pos={0 0}, upload=14014]: Segment reached optimal threshold of 13 pieces.
Aug 7 23:10:22.380848780 (13h30m21.970193767s) [noise=true , address=104.36.82.2:4730 , piece_id=3B4SQCLJ6FI6CRNRWCJRNA2CX4X5KZZELCFGOBIWVW7NQLLWKBRQ, seg_pos={0 0}, upload=14014]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom): protocol: write tcp 188.166.225.170:44882->104.36.82.2:4730: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 7 23:10:22.380911371 (13h30m21.970256358s) [noise=true , address=104.36.82.15:4257 , piece_id=DRDDV5DLUAAT2TFHHH4GMNHECM3LQHNH24Y2ZJALUXRCXL6QNRRA, seg_pos={0 0}, upload=14014]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK): protocol: write tcp 188.166.225.170:60016->104.36.82.15:4257: use of closed network connection; write tcp 188.166.225.170:60016->104.36.82.15:4257: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60016->104.36.82.15:4257: use of closed network connection
Aug 7 23:10:22.380940176 (13h30m21.970285163s) [seg_pos={0 0}, upload=14014]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Aug 7 23:10:22.381026481 (13h30m21.970371468s) [upload=14014]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Aug 7 23:10:22.381030618 (13h30m21.970375606s) [upload=14014]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Aug 7 23:10:22.734882827 (13h30m22.324227804s) [upload=14014]: Done uploading segments...
Aug 7 23:10:22.734915910 (13h30m22.324260917s) [upload=14014]: Upload finished. err: <nil>
Aug 7 23:11:22.023156432 (13h31m21.612501409s) [upload=14030]: Starting upload
Aug 7 23:11:22.023212850 (13h31m21.612557837s) [upload=14030]: Done starting upload
Aug 7 23:11:22.023366314 (13h31m21.612711301s) [upload=14030]: Uploading segments...
Aug 7 23:11:22.023488728 (13h31m21.612833715s) [upload=14030]: Got next segment. Inline: false
Aug 7 23:11:22.023517703 (13h31m21.61286268s) [upload=14030]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Aug 7 23:11:22.023522112 (13h31m21.612867099s) [upload=14030]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Aug 7 23:11:51.377219283 (13h31m50.96656427s) [upload=14030]: Next segment err: context deadline exceeded
Aug 7 23:11:51.377486564 (13h31m50.966831552s) [upload=14030]: Done uploading segments...
Aug 7 23:11:51.377502725 (13h31m50.966847712s) [upload=14030]: Upload finished. err: context deadline exceeded
Duration: 6.407338182s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline)antiblip:
3 out of 60 is failed (threshold=3) [X.X.X.......................................................] duration:
2.345221574s
FAIL (2025-08-07 23:11:56 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.345244839s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [XXXX.......................................................X] duration:
30.001456921s
FAIL (2025-08-07 23:11:51 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: encryption: context deadline exceeded
======LOGS======
30m19.119835718s) [upload=14014]: Got next segment. Inline: false
Aug 7 23:10:19.530521730 (13h30m19.119866718s) [upload=14014]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Aug 7 23:10:19.530526099 (13h30m19.119871086s) [upload=14014]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Aug 7 23:10:19.769147767 (13h30m19.358492754s) [seg_pos={0 0}, upload=14014]: Begin upload segment...
Aug 7 23:10:19.769196962 (13h30m19.358541949s) [seg_pos={0 0}, upload=14014]: Done begin upload segment.
Aug 7 23:10:19.769217901 (13h30m19.358562878s) [noise=true , address=77.90.31.19:4842 , piece_id=QUFDMBB7C5KALQOTIOQE6OOVFB3HTL23IPXC3SNT7TPUPBMENIYQ, seg_pos={0 0}, upload=14014]: Uploading piece...
Aug 7 23:10:19.769241967 (13h30m19.358586954s) [noise=true , address=45.91.116.9:4263 , piece_id=75NMKEGELJDZFSDE6L7BUOXIOCUZVZYZSCQFXL5AWFFN4L3B432Q, seg_pos={0 0}, upload=14014]: Uploading piece...
Aug 7 23:10:19.769614190 (13h30m19.358959177s) [noise=true , address=104.36.82.15:4257 , piece_id=DRDDV5DLUAAT2TFHHH4GMNHECM3LQHNH24Y2ZJALUXRCXL6QNRRA, seg_pos={0 0}, upload=14014]: Uploading piece...
Aug 7 23:10:19.769658985 (13h30m19.359003972s) [noise=true , address=77.90.31.16:4733 , piece_id=GRL5LY7JWUUYQFQ5CUZXDOZIYEG4GWWMVN76YTFPGKQOAP22XXNA, seg_pos={0 0}, upload=14014]: Uploading piece...
Aug 7 23:10:19.769909134 (13h30m19.359254111s) [noise=true , address=45.91.116.7:4011 , piece_id=ABOGCAOR5HL2MGGHQHXS3O3WLSEXD2B4UW4WGDJMDQYK5KBK2MSA, seg_pos={0 0}, upload=14014]: Uploading piece...
Aug 7 23:10:19.769942408 (13h30m19.359287395s) [noise=true , address=77.90.31.18:4807 , piece_id=7SNCXGGPT7RTAPPOX3LE526FIQU5G2VYYRJDFIPDFLA6ZPTFX7PQ, seg_pos={0 0}, upload=14014]: Uploading piece...
Aug 7 23:10:19.770178389 (13h30m19.359523366s) [noise=true , address=104.36.82.2:4730 , piece_id=3B4SQCLJ6FI6CRNRWCJRNA2CX4X5KZZELCFGOBIWVW7NQLLWKBRQ, seg_pos={0 0}, upload=14014]: Uploading piece...
Aug 7 23:10:19.770446231 (13h30m19.359791208s) [noise=true , address=104.36.82.8:4969 , piece_id=2EHLHQAWG3TA545TZV3QUAMRTLWBAYP2KQXXY7XSITAOUI5BJV7Q, seg_pos={0 0}, upload=14014]: Uploading piece...
Aug 7 23:10:19.770673426 (13h30m19.360018403s) [noise=true , address=45.91.116.11:4119 , piece_id=QMJWOHYHCVIVE7AVUUE3YWFTPZXQ2SK4QAAAS2ILP7J5OSVU2XTA, seg_pos={0 0}, upload=14014]: Uploading piece...
Aug 7 23:10:19.770876775 (13h30m19.360221763s) [noise=true , address=77.90.31.21:4877 , piece_id=DA742EGVEWSYO6PQZR7Z2IJTXFIBBYN5EVNKOGE2TZCM5RWOMTRA, seg_pos={0 0}, upload=14014]: Uploading piece...
Aug 7 23:10:19.771065907 (13h30m19.360410884s) [noise=true , address=77.90.31.14:4661 , piece_id=DX7EKUM3I6BRROMSBR3QMWOPBRXW3QCKDXL7RSCZ4MYGAOE62HEA, seg_pos={0 0}, upload=14014]: Uploading piece...
Aug 7 23:10:19.771256863 (13h30m19.36060185s) [noise=true , address=104.36.82.16:4295 , piece_id=NI7VPQGP7DXVVO6H7URWAGUWTHAIAQCXDZYWDMCD2EPIPLFDCGFQ, seg_pos={0 0}, upload=14014]: Uploading piece...
Aug 7 23:10:19.771456625 (13h30m19.360801612s) [noise=true , address=45.91.116.5:4298 , piece_id=ESZYBPWHWL2MOTBPVHK42M6CADJZNPJZBZXZLTYIUZHCH5XV5ZCQ, seg_pos={0 0}, upload=14014]: Uploading piece...
Aug 7 23:10:19.771646057 (13h30m19.360991034s) [noise=true , address=45.91.116.6:4973 , piece_id=MW66WBZULRRDBTQSZCHAOIC44YVAFFO2EKDEZLHF6QVZ55CHTHPQ, seg_pos={0 0}, upload=14014]: Uploading piece...
Aug 7 23:10:19.771840640 (13h30m19.361185627s) [noise=true , address=45.91.116.4:4939 , piece_id=XRYW6ZXHU4BCYNAKNWFAYBPJDOXQCAGTEIUJ7KJKNLVES73LD43Q, seg_pos={0 0}, upload=14014]: Uploading piece...
Aug 7 23:10:19.772068876 (13h30m19.361413863s) [noise=true , address=45.91.116.8:4045 , piece_id=XLOA7HZ7FRG5H5HMLMECR2PCSVOGD6XI5NS5HSTOQTZZ5KIMXIRA, seg_pos={0 0}, upload=14014]: Uploading piece...
Aug 7 23:10:20.020996815 (13h30m19.610341802s) [noise=true , address=77.90.31.16:4733 , piece_id=GRL5LY7JWUUYQFQ5CUZXDOZIYEG4GWWMVN76YTFPGKQOAP22XXNA, seg_pos={0 0}, upload=14014]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: write tcp 188.166.225.170:37310->77.90.31.16:4733: write: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 7 23:10:21.532433287 (13h30m21.121778274s) [upload=14014]: Next returned nil segment
Aug 7 23:10:21.532449418 (13h30m21.121794405s) [upload=14014]: Waiting for error group managing segments...
Aug 7 23:10:21.811529953 (13h30m21.40087494s) [noise=true , address=77.90.31.21:4877 , piece_id=DA742EGVEWSYO6PQZR7Z2IJTXFIBBYN5EVNKOGE2TZCM5RWOMTRA, seg_pos={0 0}, upload=14014]: Done uploading piece. err: <nil>
Aug 7 23:10:21.854297295 (13h30m21.443642272s) [noise=true , address=77.90.31.18:4807 , piece_id=7SNCXGGPT7RTAPPOX3LE526FIQU5G2VYYRJDFIPDFLA6ZPTFX7PQ, seg_pos={0 0}, upload=14014]: Done uploading piece. err: <nil>
Aug 7 23:10:21.921242465 (13h30m21.510587452s) [noise=true , address=104.36.82.8:4969 , piece_id=2EHLHQAWG3TA545TZV3QUAMRTLWBAYP2KQXXY7XSITAOUI5BJV7Q, seg_pos={0 0}, upload=14014]: Done uploading piece. err: <nil>
Aug 7 23:10:21.947359331 (13h30m21.536704318s) [noise=true , address=45.91.116.9:4263 , piece_id=75NMKEGELJDZFSDE6L7BUOXIOCUZVZYZSCQFXL5AWFFN4L3B432Q, seg_pos={0 0}, upload=14014]: Done uploading piece. err: <nil>
Aug 7 23:10:21.969454293 (13h30m21.55879928s) [noise=true , address=77.90.31.19:4842 , piece_id=QUFDMBB7C5KALQOTIOQE6OOVFB3HTL23IPXC3SNT7TPUPBMENIYQ, seg_pos={0 0}, upload=14014]: Done uploading piece. err: <nil>
Aug 7 23:10:22.042768958 (13h30m21.632113936s) [noise=true , address=45.91.116.5:4298 , piece_id=ESZYBPWHWL2MOTBPVHK42M6CADJZNPJZBZXZLTYIUZHCH5XV5ZCQ, seg_pos={0 0}, upload=14014]: Done uploading piece. err: <nil>
Aug 7 23:10:22.050439524 (13h30m21.639784501s) [noise=true , address=45.91.116.7:4011 , piece_id=ABOGCAOR5HL2MGGHQHXS3O3WLSEXD2B4UW4WGDJMDQYK5KBK2MSA, seg_pos={0 0}, upload=14014]: Done uploading piece. err: <nil>
Aug 7 23:10:22.063864939 (13h30m21.653209926s) [noise=true , address=45.91.116.8:4045 , piece_id=XLOA7HZ7FRG5H5HMLMECR2PCSVOGD6XI5NS5HSTOQTZZ5KIMXIRA, seg_pos={0 0}, upload=14014]: Done uploading piece. err: <nil>
Aug 7 23:10:22.108762327 (13h30m21.698107314s) [noise=true , address=45.91.116.11:4119 , piece_id=QMJWOHYHCVIVE7AVUUE3YWFTPZXQ2SK4QAAAS2ILP7J5OSVU2XTA, seg_pos={0 0}, upload=14014]: Done uploading piece. err: <nil>
Aug 7 23:10:22.196386158 (13h30m21.785731146s) [noise=true , address=104.36.82.16:4295 , piece_id=NI7VPQGP7DXVVO6H7URWAGUWTHAIAQCXDZYWDMCD2EPIPLFDCGFQ, seg_pos={0 0}, upload=14014]: Done uploading piece. err: <nil>
Aug 7 23:10:22.220196592 (13h30m21.809541579s) [noise=true , address=45.91.116.6:4973 , piece_id=MW66WBZULRRDBTQSZCHAOIC44YVAFFO2EKDEZLHF6QVZ55CHTHPQ, seg_pos={0 0}, upload=14014]: Done uploading piece. err: <nil>
Aug 7 23:10:22.247094513 (13h30m21.8364395s) [noise=true , address=45.91.116.4:4939 , piece_id=XRYW6ZXHU4BCYNAKNWFAYBPJDOXQCAGTEIUJ7KJKNLVES73LD43Q, seg_pos={0 0}, upload=14014]: Done uploading piece. err: <nil>
Aug 7 23:10:22.380553005 (13h30m21.969897992s) [noise=true , address=77.90.31.14:4661 , piece_id=DX7EKUM3I6BRROMSBR3QMWOPBRXW3QCKDXL7RSCZ4MYGAOE62HEA, seg_pos={0 0}, upload=14014]: Done uploading piece. err: <nil>
Aug 7 23:10:22.380581249 (13h30m21.969926236s) [seg_pos={0 0}, upload=14014]: Segment reached optimal threshold of 13 pieces.
Aug 7 23:10:22.380848780 (13h30m21.970193767s) [noise=true , address=104.36.82.2:4730 , piece_id=3B4SQCLJ6FI6CRNRWCJRNA2CX4X5KZZELCFGOBIWVW7NQLLWKBRQ, seg_pos={0 0}, upload=14014]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom): protocol: write tcp 188.166.225.170:44882->104.36.82.2:4730: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 7 23:10:22.380911371 (13h30m21.970256358s) [noise=true , address=104.36.82.15:4257 , piece_id=DRDDV5DLUAAT2TFHHH4GMNHECM3LQHNH24Y2ZJALUXRCXL6QNRRA, seg_pos={0 0}, upload=14014]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK): protocol: write tcp 188.166.225.170:60016->104.36.82.15:4257: use of closed network connection; write tcp 188.166.225.170:60016->104.36.82.15:4257: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60016->104.36.82.15:4257: use of closed network connection
Aug 7 23:10:22.380940176 (13h30m21.970285163s) [seg_pos={0 0}, upload=14014]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Aug 7 23:10:22.381026481 (13h30m21.970371468s) [upload=14014]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Aug 7 23:10:22.381030618 (13h30m21.970375606s) [upload=14014]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Aug 7 23:10:22.734882827 (13h30m22.324227804s) [upload=14014]: Done uploading segments...
Aug 7 23:10:22.734915910 (13h30m22.324260917s) [upload=14014]: Upload finished. err: <nil>
Aug 7 23:11:22.023156432 (13h31m21.612501409s) [upload=14030]: Starting upload
Aug 7 23:11:22.023212850 (13h31m21.612557837s) [upload=14030]: Done starting upload
Aug 7 23:11:22.023366314 (13h31m21.612711301s) [upload=14030]: Uploading segments...
Aug 7 23:11:22.023488728 (13h31m21.612833715s) [upload=14030]: Got next segment. Inline: false
Aug 7 23:11:22.023517703 (13h31m21.61286268s) [upload=14030]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Aug 7 23:11:22.023522112 (13h31m21.612867099s) [upload=14030]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Aug 7 23:11:51.377219283 (13h31m50.96656427s) [upload=14030]: Next segment err: context deadline exceeded
Aug 7 23:11:51.377486564 (13h31m50.966831552s) [upload=14030]: Done uploading segments...
Aug 7 23:11:51.377502725 (13h31m50.966847712s) [upload=14030]: Upload finished. err: context deadline exceeded
Duration: 30.001476468s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)antiblip:
3 out of 60 is failed (threshold=3) [.XXX........................................................] duration:
5.800949043s
FAIL (2025-08-07 23:11:50 UTC)
orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded
Duration: 5.800969373s
core
uplink
us1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
3 out of 60 is failed (threshold=3) [..X.................X...................X...................] duration:
5.970255791s
FAIL (2025-08-07 23:11:38 UTC)
orbiter warning:
operation: ListObjects
======ERROR======
unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/4M
======LOGS======
172565s) [upload=13803]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Aug 7 22:58:33.457032807 (13h18m33.046377794s) [seg_pos={0 0}, upload=13803]: Begin upload segment...
Aug 7 22:58:33.457116296 (13h18m33.046461283s) [seg_pos={0 0}, upload=13803]: Done begin upload segment.
Aug 7 22:58:33.457172754 (13h18m33.046517741s) [noise=true , address=79.127.201.215:4349 , piece_id=XDVQZBXGHX65RHZBWUQRP3BAHRPRHOCWXXOLSJDQ2XDEYNFQSNAA, seg_pos={0 0}, upload=13803]: Uploading piece...
Aug 7 22:58:33.457191480 (13h18m33.046536468s) [noise=true , address=104.36.82.4:4818 , piece_id=T4K2OGU3TC27H5LE4CTDLAOYR6DOP3S2PBVV7KUX4KKVFNMZNXBQ, seg_pos={0 0}, upload=13803]: Uploading piece...
Aug 7 22:58:33.457735842 (13h18m33.047080829s) [noise=true , address=104.36.82.8:4003 , piece_id=C4M64YUD5J6CFLATRLBVPZY7P67VD5R2WQ4LPLIV6CNRAPDRDSMQ, seg_pos={0 0}, upload=13803]: Uploading piece...
Aug 7 22:58:33.457740962 (13h18m33.047085939s) [noise=true , address=104.36.82.15:4270 , piece_id=2MTKGO66NWBUQXDMTN4NRYKIWHT6GKRW3LAHTZHLOVBGE7OINGZA, seg_pos={0 0}, upload=13803]: Uploading piece...
Aug 7 22:58:33.458108976 (13h18m33.047453963s) [noise=true , address=104.36.82.39:4732 , piece_id=EJNQS3XUYBVPZ6SLENNNLQBJ5OEIF53XSM645SAV3DGTFQ7EK3BA, seg_pos={0 0}, upload=13803]: Uploading piece...
Aug 7 22:58:33.458181885 (13h18m33.047526872s) [noise=true , address=104.36.82.62:4430 , piece_id=SCXOGWUIS2KIBEWOQWYMVBS6EV3URUYFKYMV5PQV5HFPK5EGSRMQ, seg_pos={0 0}, upload=13803]: Uploading piece...
Aug 7 22:58:33.458456691 (13h18m33.047801678s) [noise=true , address=45.91.116.14:4221 , piece_id=J54CXKHJH2KOUV3MPT36F6JHOI67GMGCMSOKHNJUDL5I7NTRLCBA, seg_pos={0 0}, upload=13803]: Uploading piece...
Aug 7 22:58:33.458607449 (13h18m33.047952436s) [noise=true , address=104.36.82.44:4895 , piece_id=GWHMQEBTZ4MB4Z6AZUAUJGFGAQ7L567F7INSONYCQCSCKK3CC7JA, seg_pos={0 0}, upload=13803]: Uploading piece...
Aug 7 22:58:33.458794698 (13h18m33.048139685s) [noise=true , address=64.49.13.170:4719 , piece_id=5AWWDAKHOKZ4FJWUZTH4PM4XPIQA2L2MKX4L7XSBNVZ3EURMK2VA, seg_pos={0 0}, upload=13803]: Uploading piece...
Aug 7 22:58:33.459301818 (13h18m33.048646795s) [noise=true , address=45.91.116.4:4955 , piece_id=WYMA4QA3QF55ZHZAUTMHWPLXJCSBOTN3NYLRJUWEMRFVUKR47VLA, seg_pos={0 0}, upload=13803]: Uploading piece...
Aug 7 22:58:33.459553139 (13h18m33.048898126s) [noise=true , address=104.36.82.55:4202 , piece_id=UFXSOKDOHBYJYXJIKWF6AKQKXT6VLTVLKKCRBLYFOXQ554LF3OMA, seg_pos={0 0}, upload=13803]: Uploading piece...
Aug 7 22:58:33.459845177 (13h18m33.049190164s) [noise=true , address=104.36.82.46:4491 , piece_id=FE5LA53DP4JQFXYYMB7NV36E5I62EWJMFQ5ZVLUDYQX5QKLKV7MQ, seg_pos={0 0}, upload=13803]: Uploading piece...
Aug 7 22:58:33.460146223 (13h18m33.04949121s) [noise=true , address=104.36.82.45:4907 , piece_id=G33PUTRJPQZ2FHMYPFED3W6X6FOEDP5C7D572K7LADQYSFLEK2HQ, seg_pos={0 0}, upload=13803]: Uploading piece...
Aug 7 22:58:33.460635710 (13h18m33.049980697s) [noise=true , address=104.36.82.55:4205 , piece_id=VK2NPARLAXL5YR5XBVABX4YBAACSXXBUXUJJJAUTPD7XG7EBNFCQ, seg_pos={0 0}, upload=13803]: Uploading piece...
Aug 7 22:58:33.460916247 (13h18m33.050261234s) [noise=true , address=45.91.116.8:4068 , piece_id=UBEDD44644KW5VOQDYO4KSXMWOL33OQCYZ5UTY7RZ662RUMFVOVQ, seg_pos={0 0}, upload=13803]: Uploading piece...
Aug 7 22:58:33.461154122 (13h18m33.050499099s) [noise=true , address=45.91.116.6:4977 , piece_id=7LDMXXPDPT4Q6JCT6QPOIZBANDZA3RHBYBRZTGOWRBKZHLZEWHEQ, seg_pos={0 0}, upload=13803]: Uploading piece...
Aug 7 22:58:33.461485957 (13h18m33.050830934s) [noise=true , address=104.36.82.12:4162 , piece_id=GCO6IE3FJAO7V3XCNWB4IQQVN2CSX3IRSR4QTZO7ZH6AOB27CYJQ, seg_pos={0 0}, upload=13803]: Uploading piece...
Aug 7 22:58:33.461747909 (13h18m33.051092896s) [noise=true , address=104.36.82.23:4472 , piece_id=BIB3GU4R4VW7CCGT7S4VCDYJSV2I6FV4NKLLSCZZJ5QDOJGYHJKQ, seg_pos={0 0}, upload=13803]: Uploading piece...
Aug 7 22:58:34.610233566 (13h18m34.199578553s) [upload=13803]: Next returned nil segment
Aug 7 22:58:34.610251881 (13h18m34.199596868s) [upload=13803]: Waiting for error group managing segments...
Aug 7 22:58:34.912103574 (13h18m34.501448551s) [noise=true , address=64.49.13.170:4719 , piece_id=5AWWDAKHOKZ4FJWUZTH4PM4XPIQA2L2MKX4L7XSBNVZ3EURMK2VA, seg_pos={0 0}, upload=13803]: Done uploading piece. err: <nil>
Aug 7 22:58:35.384823542 (13h18m34.974168529s) [noise=true , address=104.36.82.8:4003 , piece_id=C4M64YUD5J6CFLATRLBVPZY7P67VD5R2WQ4LPLIV6CNRAPDRDSMQ, seg_pos={0 0}, upload=13803]: Done uploading piece. err: <nil>
Aug 7 22:58:35.418586980 (13h18m35.007931967s) [noise=true , address=79.127.201.215:4349 , piece_id=XDVQZBXGHX65RHZBWUQRP3BAHRPRHOCWXXOLSJDQ2XDEYNFQSNAA, seg_pos={0 0}, upload=13803]: Done uploading piece. err: <nil>
Aug 7 22:58:35.458066756 (13h18m35.047411743s) [noise=true , address=104.36.82.44:4895 , piece_id=GWHMQEBTZ4MB4Z6AZUAUJGFGAQ7L567F7INSONYCQCSCKK3CC7JA, seg_pos={0 0}, upload=13803]: Done uploading piece. err: <nil>
Aug 7 22:58:35.477223478 (13h18m35.066568455s) [noise=true , address=104.36.82.45:4907 , piece_id=G33PUTRJPQZ2FHMYPFED3W6X6FOEDP5C7D572K7LADQYSFLEK2HQ, seg_pos={0 0}, upload=13803]: Done uploading piece. err: <nil>
Aug 7 22:58:35.499730379 (13h18m35.089075366s) [noise=true , address=104.36.82.39:4732 , piece_id=EJNQS3XUYBVPZ6SLENNNLQBJ5OEIF53XSM645SAV3DGTFQ7EK3BA, seg_pos={0 0}, upload=13803]: Done uploading piece. err: <nil>
Aug 7 22:58:35.614098456 (13h18m35.203443443s) [noise=true , address=104.36.82.46:4491 , piece_id=FE5LA53DP4JQFXYYMB7NV36E5I62EWJMFQ5ZVLUDYQX5QKLKV7MQ, seg_pos={0 0}, upload=13803]: Done uploading piece. err: <nil>
Aug 7 22:58:35.748129262 (13h18m35.33747425s) [noise=true , address=104.36.82.62:4430 , piece_id=SCXOGWUIS2KIBEWOQWYMVBS6EV3URUYFKYMV5PQV5HFPK5EGSRMQ, seg_pos={0 0}, upload=13803]: Done uploading piece. err: <nil>
Aug 7 22:58:35.755387388 (13h18m35.344732376s) [noise=true , address=104.36.82.12:4162 , piece_id=GCO6IE3FJAO7V3XCNWB4IQQVN2CSX3IRSR4QTZO7ZH6AOB27CYJQ, seg_pos={0 0}, upload=13803]: Done uploading piece. err: <nil>
Aug 7 22:58:35.958468347 (13h18m35.547813334s) [noise=true , address=45.91.116.8:4068 , piece_id=UBEDD44644KW5VOQDYO4KSXMWOL33OQCYZ5UTY7RZ662RUMFVOVQ, seg_pos={0 0}, upload=13803]: Done uploading piece. err: <nil>
Aug 7 22:58:36.044513974 (13h18m35.633858961s) [noise=true , address=45.91.116.6:4977 , piece_id=7LDMXXPDPT4Q6JCT6QPOIZBANDZA3RHBYBRZTGOWRBKZHLZEWHEQ, seg_pos={0 0}, upload=13803]: Done uploading piece. err: <nil>
Aug 7 22:58:36.091808572 (13h18m35.681153549s) [noise=true , address=45.91.116.14:4221 , piece_id=J54CXKHJH2KOUV3MPT36F6JHOI67GMGCMSOKHNJUDL5I7NTRLCBA, seg_pos={0 0}, upload=13803]: Done uploading piece. err: <nil>
Aug 7 22:58:36.096744555 (13h18m35.686089543s) [noise=true , address=104.36.82.55:4205 , piece_id=VK2NPARLAXL5YR5XBVABX4YBAACSXXBUXUJJJAUTPD7XG7EBNFCQ, seg_pos={0 0}, upload=13803]: Done uploading piece. err: <nil>
Aug 7 22:58:36.171642074 (13h18m35.760987061s) [noise=true , address=104.36.82.55:4202 , piece_id=UFXSOKDOHBYJYXJIKWF6AKQKXT6VLTVLKKCRBLYFOXQ554LF3OMA, seg_pos={0 0}, upload=13803]: Done uploading piece. err: <nil>
Aug 7 22:58:36.171668725 (13h18m35.761013712s) [seg_pos={0 0}, upload=13803]: Segment reached optimal threshold of 14 pieces.
Aug 7 22:58:36.171958509 (13h18m35.761303496s) [noise=true , address=104.36.82.4:4818 , piece_id=T4K2OGU3TC27H5LE4CTDLAOYR6DOP3S2PBVV7KUX4KKVFNMZNXBQ, seg_pos={0 0}, upload=13803]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1235eFkz7CFwwiM8UUZU1vnqDneEic7RpFzeTU46gvTMZWhhox8): protocol: read tcp 188.166.225.170:34616->104.36.82.4:4818: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 7 22:58:36.172074250 (13h18m35.761419227s) [noise=true , address=45.91.116.4:4955 , piece_id=WYMA4QA3QF55ZHZAUTMHWPLXJCSBOTN3NYLRJUWEMRFVUKR47VLA, seg_pos={0 0}, upload=13803]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VULW731AQGUMf4KLbhPUDNAEiW2AxaXJQYHzqiPiBuWG17L2B): protocol: write tcp 188.166.225.170:39210->45.91.116.4:4955: use of closed network connection; write tcp 188.166.225.170:39210->45.91.116.4:4955: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39210->45.91.116.4:4955: use of closed network connection
Aug 7 22:58:36.172112223 (13h18m35.7614572s) [noise=true , address=104.36.82.15:4270 , piece_id=2MTKGO66NWBUQXDMTN4NRYKIWHT6GKRW3LAHTZHLOVBGE7OINGZA, seg_pos={0 0}, upload=13803]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nkuzy3vxeLdHhVs73Nceje7qXDiGJ1zEhpGgGKSk3cKb7vJbB): protocol: rpc: dial tcp 104.36.82.15:4270: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 7 22:58:36.172191986 (13h18m35.761536973s) [noise=true , address=104.36.82.23:4472 , piece_id=BIB3GU4R4VW7CCGT7S4VCDYJSV2I6FV4NKLLSCZZJ5QDOJGYHJKQ, seg_pos={0 0}, upload=13803]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12otbB1RmPUKSKo813hrKPQVAyeqmhjhApxi6ndtHBCfn9bGGct): protocol: write tcp 188.166.225.170:47876->104.36.82.23:4472: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 7 22:58:36.172216163 (13h18m35.76156115s) [seg_pos={0 0}, upload=13803]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Aug 7 22:58:36.172285585 (13h18m35.761630572s) [upload=13803]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Aug 7 22:58:36.172289833 (13h18m35.76163482s) [upload=13803]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Aug 7 22:58:36.490569647 (13h18m36.079914634s) [upload=13803]: Done uploading segments...
Aug 7 22:58:36.490587652 (13h18m36.079932639s) [upload=13803]: Upload finished. err: <nil>
Duration: 5.970301949s
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)antiblip:
1 out of 60 is failed (threshold=3) [...................X........................................] duration:
644.672143ms target:
https://us1.storj.io/login
FAIL (2025-08-07 23:11:36 UTC)
orbiter warning: Get "https://us1.storj.io/login": dial tcp 34.150.199.48:443: connect: connection refused
Duration: 644.688465ms
core
satellite
us1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
6 out of 60 is failed (threshold=3) [XXXXX.X.....................................................] duration:
5.439510872s
FAIL (2025-08-07 23:11:35 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.439723427s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)antiblip:
1 out of 60 is failed (threshold=3) [..X.........................................................] duration:
1.856248607s
FAIL (2025-08-07 23:11:20 UTC)
orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded
Duration: 1.856273605s
core
uplink
us1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [XX...X......................................................] duration:
6.278573454s
FAIL (2025-08-07 23:11:08 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.278599294s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
5 out of 60 is failed (threshold=3) [XXX..X.X....................................................] duration:
7.510756151s
FAIL (2025-08-07 23:11:04 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
load=13236]: Begin upload segment...
Aug 7 22:25:52.992307080 (12h45m52.581652067s) [seg_pos={0 0}, upload=13236]: Done begin upload segment.
Aug 7 22:25:52.992331798 (12h45m52.581676775s) [noise=true , address=104.36.82.13:4213 , piece_id=5W5FD4WMMSZ7BKHFQI4GYOODXZ2P2MMAIQNK6CD6XILH22C53JGQ, seg_pos={0 0}, upload=13236]: Uploading piece...
Aug 7 22:25:52.992683770 (12h45m52.582028757s) [noise=true , address=104.36.82.9:4024 , piece_id=VBFOHRWDKWRY2774KJVK3JXJSMG4CPVDBBECCSZVJHR4274KLRMA, seg_pos={0 0}, upload=13236]: Uploading piece...
Aug 7 22:25:52.992785475 (12h45m52.582130462s) [noise=true , address=104.36.82.22:4527 , piece_id=JBNJOSETDAB6VG7PGVXIBUP6GPF2FBPJURBZNBR3BJOGMRK3EIQA, seg_pos={0 0}, upload=13236]: Uploading piece...
Aug 7 22:25:52.992942396 (12h45m52.582287373s) [noise=true , address=45.91.116.11:4124 , piece_id=TLATR2WNDVRR3WWTT3FZARUJ2QWWQFZBDYXLZ7PUPFOAEGJI4BTQ, seg_pos={0 0}, upload=13236]: Uploading piece...
Aug 7 22:25:52.993132439 (12h45m52.582477457s) [noise=true , address=104.36.82.5:4855 , piece_id=6EW7GNJ6KVWKNFHGFZVPEYPD4TDIVVZUXRW6AGJ3IGVANEAHYDUA, seg_pos={0 0}, upload=13236]: Uploading piece...
Aug 7 22:25:52.993337862 (12h45m52.582682849s) [noise=true , address=64.49.13.170:4717 , piece_id=52FN65VRQ24LBOGHJ22EQQM3MNI3EB2UXB57P5LD5GPLSABNS22Q, seg_pos={0 0}, upload=13236]: Uploading piece...
Aug 7 22:25:52.993511144 (12h45m52.582856131s) [noise=true , address=104.36.82.60:4361 , piece_id=EPWXHNXPVAUX75FUAXY57QZDKRXIEDHRXFL6PEBQGXY2A3Q4KNHA, seg_pos={0 0}, upload=13236]: Uploading piece...
Aug 7 22:25:52.993715204 (12h45m52.583060191s) [noise=true , address=104.36.82.58:4281 , piece_id=5UH27M25HE2LEZJ53H7QBKU6I6N2JE6F4PZTLOE6XP3GEDCZLEEQ, seg_pos={0 0}, upload=13236]: Uploading piece...
Aug 7 22:25:52.993820806 (12h45m52.583165784s) [noise=true , address=45.91.116.9:4284 , piece_id=LT3RP2EBVHIFIMAIGM3DW6QMPNLKZCK5FD4GQEQI6BFWYD56ZUBQ, seg_pos={0 0}, upload=13236]: Uploading piece...
Aug 7 22:25:52.994068250 (12h45m52.583413237s) [noise=true , address=79.127.205.233:4633 , piece_id=LBO5XLAXBRDURMDMN6MZ4HLVSG4TTUZPWCFTOTB6QMWAS5IIRLBA, seg_pos={0 0}, upload=13236]: Uploading piece...
Aug 7 22:25:52.994092416 (12h45m52.583437393s) [noise=true , address=104.36.82.60:4346 , piece_id=3NU3QBO5DAW5ZB7NTAHWP4C7RE4BYDPHZ77JOUQT3R3CBTOMONLA, seg_pos={0 0}, upload=13236]: Uploading piece...
Aug 7 22:25:52.994295394 (12h45m52.583640371s) [noise=true , address=45.91.116.7:4037 , piece_id=2NDGWBVYO3D2RYJ57QSXT632TT24SMRNWA3YZ2YW2BN6TBBLMRQA, seg_pos={0 0}, upload=13236]: Uploading piece...
Aug 7 22:25:52.994307056 (12h45m52.583652033s) [noise=true , address=104.36.82.46:4505 , piece_id=A7XPGAYUTNB26A6DJQNQ4NZ4FG27FUBQNEH5AQ7IFLYR3DLGG6SQ, seg_pos={0 0}, upload=13236]: Uploading piece...
Aug 7 22:25:52.994547126 (12h45m52.583892103s) [noise=true , address=104.36.82.61:4389 , piece_id=WBB2Y7YYCEML23D3CKJHGIEOUELRTVG3NXVGI4JDD7XPNQ4STENQ, seg_pos={0 0}, upload=13236]: Uploading piece...
Aug 7 22:25:52.994782436 (12h45m52.584127423s) [noise=true , address=64.49.13.154:4934 , piece_id=LDQTK4SFJPTL7NAY5XXKBIFCVLNRSLHLACTJ65MWCHFSBV4EKJYA, seg_pos={0 0}, upload=13236]: Uploading piece...
Aug 7 22:25:52.998117076 (12h45m52.587462053s) [noise=true , address=104.36.82.56:4252 , piece_id=XQWAMPOYTVGKYEHIEEEIDDST5BOIN2XCT7B3SFUR4U7PQBROEELQ, seg_pos={0 0}, upload=13236]: Uploading piece...
Aug 7 22:25:52.998596283 (12h45m52.58794127s) [noise=true , address=104.36.82.15:4261 , piece_id=C3RUC2F5LVRKH4MYTXDYOAP5KJGRZ6TAX57XULVBAKOS7ZK4QONQ, seg_pos={0 0}, upload=13236]: Uploading piece...
Aug 7 22:25:52.998917628 (12h45m52.588262615s) [noise=true , address=104.36.82.23:4473 , piece_id=N54OVKSO4KKX4NPORULBLAPVE5UT5RU6BBIEJCCMKG5553GVY6JA, seg_pos={0 0}, upload=13236]: Uploading piece...
Aug 7 22:25:54.121532197 (12h45m53.710877184s) [upload=13236]: Next returned nil segment
Aug 7 22:25:54.121549150 (12h45m53.710894137s) [upload=13236]: Waiting for error group managing segments...
Aug 7 22:25:54.447812563 (12h45m54.03715755s) [noise=true , address=79.127.205.233:4633 , piece_id=LBO5XLAXBRDURMDMN6MZ4HLVSG4TTUZPWCFTOTB6QMWAS5IIRLBA, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil>
Aug 7 22:25:54.450658769 (12h45m54.040003756s) [noise=true , address=64.49.13.170:4717 , piece_id=52FN65VRQ24LBOGHJ22EQQM3MNI3EB2UXB57P5LD5GPLSABNS22Q, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil>
Aug 7 22:25:54.505775361 (12h45m54.095120349s) [noise=true , address=64.49.13.154:4934 , piece_id=LDQTK4SFJPTL7NAY5XXKBIFCVLNRSLHLACTJ65MWCHFSBV4EKJYA, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil>
Aug 7 22:25:55.074032067 (12h45m54.663377054s) [noise=true , address=104.36.82.60:4346 , piece_id=3NU3QBO5DAW5ZB7NTAHWP4C7RE4BYDPHZ77JOUQT3R3CBTOMONLA, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil>
Aug 7 22:25:55.160612428 (12h45m54.749957415s) [noise=true , address=104.36.82.61:4389 , piece_id=WBB2Y7YYCEML23D3CKJHGIEOUELRTVG3NXVGI4JDD7XPNQ4STENQ, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil>
Aug 7 22:25:55.161069252 (12h45m54.750414239s) [noise=true , address=104.36.82.22:4527 , piece_id=JBNJOSETDAB6VG7PGVXIBUP6GPF2FBPJURBZNBR3BJOGMRK3EIQA, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil>
Aug 7 22:25:55.172460447 (12h45m54.761805434s) [noise=true , address=104.36.82.60:4361 , piece_id=EPWXHNXPVAUX75FUAXY57QZDKRXIEDHRXFL6PEBQGXY2A3Q4KNHA, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil>
Aug 7 22:25:55.217919931 (12h45m54.807264908s) [noise=true , address=104.36.82.56:4252 , piece_id=XQWAMPOYTVGKYEHIEEEIDDST5BOIN2XCT7B3SFUR4U7PQBROEELQ, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil>
Aug 7 22:25:55.226637230 (12h45m54.815982217s) [noise=true , address=104.36.82.58:4281 , piece_id=5UH27M25HE2LEZJ53H7QBKU6I6N2JE6F4PZTLOE6XP3GEDCZLEEQ, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil>
Aug 7 22:25:55.271492068 (12h45m54.860837055s) [noise=true , address=45.91.116.11:4124 , piece_id=TLATR2WNDVRR3WWTT3FZARUJ2QWWQFZBDYXLZ7PUPFOAEGJI4BTQ, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil>
Aug 7 22:25:55.281497411 (12h45m54.870842388s) [noise=true , address=45.91.116.9:4284 , piece_id=LT3RP2EBVHIFIMAIGM3DW6QMPNLKZCK5FD4GQEQI6BFWYD56ZUBQ, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil>
Aug 7 22:25:55.402216807 (12h45m54.991561794s) [noise=true , address=104.36.82.9:4024 , piece_id=VBFOHRWDKWRY2774KJVK3JXJSMG4CPVDBBECCSZVJHR4274KLRMA, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil>
Aug 7 22:25:55.632857570 (12h45m55.222202558s) [noise=true , address=104.36.82.5:4855 , piece_id=6EW7GNJ6KVWKNFHGFZVPEYPD4TDIVVZUXRW6AGJ3IGVANEAHYDUA, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil>
Aug 7 22:25:56.838313689 (12h45m56.427658676s) [noise=true , address=104.36.82.13:4213 , piece_id=5W5FD4WMMSZ7BKHFQI4GYOODXZ2P2MMAIQNK6CD6XILH22C53JGQ, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil>
Aug 7 22:25:56.838338066 (12h45m56.427683053s) [seg_pos={0 0}, upload=13236]: Segment reached optimal threshold of 14 pieces.
Aug 7 22:25:56.838585389 (12h45m56.427930366s) [noise=true , address=45.91.116.7:4037 , piece_id=2NDGWBVYO3D2RYJ57QSXT632TT24SMRNWA3YZ2YW2BN6TBBLMRQA, seg_pos={0 0}, upload=13236]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iNPwE2XSfXcaXAXcrH2jcyNGYeDiiWvzAcDQffrmjPwTVqVmK): protocol: expected piece hash; write tcp 188.166.225.170:55570->45.91.116.7:4037: use of closed network connection; write tcp 188.166.225.170:55570->45.91.116.7:4037: use of closed network connection; EOF
Aug 7 22:25:56.838630465 (12h45m56.427975452s) [noise=true , address=104.36.82.46:4505 , piece_id=A7XPGAYUTNB26A6DJQNQ4NZ4FG27FUBQNEH5AQ7IFLYR3DLGG6SQ, seg_pos={0 0}, upload=13236]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i6itGoBhLsLiqC8et8Y5L62fLjcWxy18u39NAr4R3nXNMWcsf): protocol: write tcp 188.166.225.170:56556->104.36.82.46:4505: use of closed network connection; write tcp 188.166.225.170:56556->104.36.82.46:4505: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56556->104.36.82.46:4505: use of closed network connection
Aug 7 22:25:56.838682564 (12h45m56.428027551s) [noise=true , address=104.36.82.23:4473 , piece_id=N54OVKSO4KKX4NPORULBLAPVE5UT5RU6BBIEJCCMKG5553GVY6JA, seg_pos={0 0}, upload=13236]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1AKpkGZ6XbeqCTAYe273MACVvMCLcPPJ4By5EX2MkNuxC6zRpC): protocol: write tcp 188.166.225.170:50948->104.36.82.23:4473: use of closed network connection; write tcp 188.166.225.170:50948->104.36.82.23:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50948->104.36.82.23:4473: use of closed network connection
Aug 7 22:25:56.838697754 (12h45m56.428042731s) [noise=true , address=104.36.82.15:4261 , piece_id=C3RUC2F5LVRKH4MYTXDYOAP5KJGRZ6TAX57XULVBAKOS7ZK4QONQ, seg_pos={0 0}, upload=13236]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i1jcJdEP7Ch9uaGcjDaVxpodx55dJEHUKFbcr1GeKQ9waWjYz): protocol: expected piece hash; context canceled; EOF
Aug 7 22:25:56.838720558 (12h45m56.428065535s) [seg_pos={0 0}, upload=13236]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Aug 7 22:25:56.838823714 (12h45m56.428168701s) [upload=13236]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Aug 7 22:25:56.838828013 (12h45m56.428173s) [upload=13236]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Aug 7 22:30:17.177125535 (12h50m16.766470522s) [upload=13236]: Done uploading segments...
Aug 7 22:30:17.177153268 (12h50m16.766498255s) [upload=13236]: Upload finished. err: metaclient: context deadline exceeded
Duration: 7.510782822s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
3 out of 60 is failed (threshold=3) [...X.................X...................X..................] duration:
6.36715603s
FAIL (2025-08-07 23:10:58 UTC)
orbiter warning:
operation: ListObjects
======ERROR======
unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/4M
======LOGS======
172565s) [upload=13803]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Aug 7 22:58:33.457032807 (13h18m33.046377794s) [seg_pos={0 0}, upload=13803]: Begin upload segment...
Aug 7 22:58:33.457116296 (13h18m33.046461283s) [seg_pos={0 0}, upload=13803]: Done begin upload segment.
Aug 7 22:58:33.457172754 (13h18m33.046517741s) [noise=true , address=79.127.201.215:4349 , piece_id=XDVQZBXGHX65RHZBWUQRP3BAHRPRHOCWXXOLSJDQ2XDEYNFQSNAA, seg_pos={0 0}, upload=13803]: Uploading piece...
Aug 7 22:58:33.457191480 (13h18m33.046536468s) [noise=true , address=104.36.82.4:4818 , piece_id=T4K2OGU3TC27H5LE4CTDLAOYR6DOP3S2PBVV7KUX4KKVFNMZNXBQ, seg_pos={0 0}, upload=13803]: Uploading piece...
Aug 7 22:58:33.457735842 (13h18m33.047080829s) [noise=true , address=104.36.82.8:4003 , piece_id=C4M64YUD5J6CFLATRLBVPZY7P67VD5R2WQ4LPLIV6CNRAPDRDSMQ, seg_pos={0 0}, upload=13803]: Uploading piece...
Aug 7 22:58:33.457740962 (13h18m33.047085939s) [noise=true , address=104.36.82.15:4270 , piece_id=2MTKGO66NWBUQXDMTN4NRYKIWHT6GKRW3LAHTZHLOVBGE7OINGZA, seg_pos={0 0}, upload=13803]: Uploading piece...
Aug 7 22:58:33.458108976 (13h18m33.047453963s) [noise=true , address=104.36.82.39:4732 , piece_id=EJNQS3XUYBVPZ6SLENNNLQBJ5OEIF53XSM645SAV3DGTFQ7EK3BA, seg_pos={0 0}, upload=13803]: Uploading piece...
Aug 7 22:58:33.458181885 (13h18m33.047526872s) [noise=true , address=104.36.82.62:4430 , piece_id=SCXOGWUIS2KIBEWOQWYMVBS6EV3URUYFKYMV5PQV5HFPK5EGSRMQ, seg_pos={0 0}, upload=13803]: Uploading piece...
Aug 7 22:58:33.458456691 (13h18m33.047801678s) [noise=true , address=45.91.116.14:4221 , piece_id=J54CXKHJH2KOUV3MPT36F6JHOI67GMGCMSOKHNJUDL5I7NTRLCBA, seg_pos={0 0}, upload=13803]: Uploading piece...
Aug 7 22:58:33.458607449 (13h18m33.047952436s) [noise=true , address=104.36.82.44:4895 , piece_id=GWHMQEBTZ4MB4Z6AZUAUJGFGAQ7L567F7INSONYCQCSCKK3CC7JA, seg_pos={0 0}, upload=13803]: Uploading piece...
Aug 7 22:58:33.458794698 (13h18m33.048139685s) [noise=true , address=64.49.13.170:4719 , piece_id=5AWWDAKHOKZ4FJWUZTH4PM4XPIQA2L2MKX4L7XSBNVZ3EURMK2VA, seg_pos={0 0}, upload=13803]: Uploading piece...
Aug 7 22:58:33.459301818 (13h18m33.048646795s) [noise=true , address=45.91.116.4:4955 , piece_id=WYMA4QA3QF55ZHZAUTMHWPLXJCSBOTN3NYLRJUWEMRFVUKR47VLA, seg_pos={0 0}, upload=13803]: Uploading piece...
Aug 7 22:58:33.459553139 (13h18m33.048898126s) [noise=true , address=104.36.82.55:4202 , piece_id=UFXSOKDOHBYJYXJIKWF6AKQKXT6VLTVLKKCRBLYFOXQ554LF3OMA, seg_pos={0 0}, upload=13803]: Uploading piece...
Aug 7 22:58:33.459845177 (13h18m33.049190164s) [noise=true , address=104.36.82.46:4491 , piece_id=FE5LA53DP4JQFXYYMB7NV36E5I62EWJMFQ5ZVLUDYQX5QKLKV7MQ, seg_pos={0 0}, upload=13803]: Uploading piece...
Aug 7 22:58:33.460146223 (13h18m33.04949121s) [noise=true , address=104.36.82.45:4907 , piece_id=G33PUTRJPQZ2FHMYPFED3W6X6FOEDP5C7D572K7LADQYSFLEK2HQ, seg_pos={0 0}, upload=13803]: Uploading piece...
Aug 7 22:58:33.460635710 (13h18m33.049980697s) [noise=true , address=104.36.82.55:4205 , piece_id=VK2NPARLAXL5YR5XBVABX4YBAACSXXBUXUJJJAUTPD7XG7EBNFCQ, seg_pos={0 0}, upload=13803]: Uploading piece...
Aug 7 22:58:33.460916247 (13h18m33.050261234s) [noise=true , address=45.91.116.8:4068 , piece_id=UBEDD44644KW5VOQDYO4KSXMWOL33OQCYZ5UTY7RZ662RUMFVOVQ, seg_pos={0 0}, upload=13803]: Uploading piece...
Aug 7 22:58:33.461154122 (13h18m33.050499099s) [noise=true , address=45.91.116.6:4977 , piece_id=7LDMXXPDPT4Q6JCT6QPOIZBANDZA3RHBYBRZTGOWRBKZHLZEWHEQ, seg_pos={0 0}, upload=13803]: Uploading piece...
Aug 7 22:58:33.461485957 (13h18m33.050830934s) [noise=true , address=104.36.82.12:4162 , piece_id=GCO6IE3FJAO7V3XCNWB4IQQVN2CSX3IRSR4QTZO7ZH6AOB27CYJQ, seg_pos={0 0}, upload=13803]: Uploading piece...
Aug 7 22:58:33.461747909 (13h18m33.051092896s) [noise=true , address=104.36.82.23:4472 , piece_id=BIB3GU4R4VW7CCGT7S4VCDYJSV2I6FV4NKLLSCZZJ5QDOJGYHJKQ, seg_pos={0 0}, upload=13803]: Uploading piece...
Aug 7 22:58:34.610233566 (13h18m34.199578553s) [upload=13803]: Next returned nil segment
Aug 7 22:58:34.610251881 (13h18m34.199596868s) [upload=13803]: Waiting for error group managing segments...
Aug 7 22:58:34.912103574 (13h18m34.501448551s) [noise=true , address=64.49.13.170:4719 , piece_id=5AWWDAKHOKZ4FJWUZTH4PM4XPIQA2L2MKX4L7XSBNVZ3EURMK2VA, seg_pos={0 0}, upload=13803]: Done uploading piece. err: <nil>
Aug 7 22:58:35.384823542 (13h18m34.974168529s) [noise=true , address=104.36.82.8:4003 , piece_id=C4M64YUD5J6CFLATRLBVPZY7P67VD5R2WQ4LPLIV6CNRAPDRDSMQ, seg_pos={0 0}, upload=13803]: Done uploading piece. err: <nil>
Aug 7 22:58:35.418586980 (13h18m35.007931967s) [noise=true , address=79.127.201.215:4349 , piece_id=XDVQZBXGHX65RHZBWUQRP3BAHRPRHOCWXXOLSJDQ2XDEYNFQSNAA, seg_pos={0 0}, upload=13803]: Done uploading piece. err: <nil>
Aug 7 22:58:35.458066756 (13h18m35.047411743s) [noise=true , address=104.36.82.44:4895 , piece_id=GWHMQEBTZ4MB4Z6AZUAUJGFGAQ7L567F7INSONYCQCSCKK3CC7JA, seg_pos={0 0}, upload=13803]: Done uploading piece. err: <nil>
Aug 7 22:58:35.477223478 (13h18m35.066568455s) [noise=true , address=104.36.82.45:4907 , piece_id=G33PUTRJPQZ2FHMYPFED3W6X6FOEDP5C7D572K7LADQYSFLEK2HQ, seg_pos={0 0}, upload=13803]: Done uploading piece. err: <nil>
Aug 7 22:58:35.499730379 (13h18m35.089075366s) [noise=true , address=104.36.82.39:4732 , piece_id=EJNQS3XUYBVPZ6SLENNNLQBJ5OEIF53XSM645SAV3DGTFQ7EK3BA, seg_pos={0 0}, upload=13803]: Done uploading piece. err: <nil>
Aug 7 22:58:35.614098456 (13h18m35.203443443s) [noise=true , address=104.36.82.46:4491 , piece_id=FE5LA53DP4JQFXYYMB7NV36E5I62EWJMFQ5ZVLUDYQX5QKLKV7MQ, seg_pos={0 0}, upload=13803]: Done uploading piece. err: <nil>
Aug 7 22:58:35.748129262 (13h18m35.33747425s) [noise=true , address=104.36.82.62:4430 , piece_id=SCXOGWUIS2KIBEWOQWYMVBS6EV3URUYFKYMV5PQV5HFPK5EGSRMQ, seg_pos={0 0}, upload=13803]: Done uploading piece. err: <nil>
Aug 7 22:58:35.755387388 (13h18m35.344732376s) [noise=true , address=104.36.82.12:4162 , piece_id=GCO6IE3FJAO7V3XCNWB4IQQVN2CSX3IRSR4QTZO7ZH6AOB27CYJQ, seg_pos={0 0}, upload=13803]: Done uploading piece. err: <nil>
Aug 7 22:58:35.958468347 (13h18m35.547813334s) [noise=true , address=45.91.116.8:4068 , piece_id=UBEDD44644KW5VOQDYO4KSXMWOL33OQCYZ5UTY7RZ662RUMFVOVQ, seg_pos={0 0}, upload=13803]: Done uploading piece. err: <nil>
Aug 7 22:58:36.044513974 (13h18m35.633858961s) [noise=true , address=45.91.116.6:4977 , piece_id=7LDMXXPDPT4Q6JCT6QPOIZBANDZA3RHBYBRZTGOWRBKZHLZEWHEQ, seg_pos={0 0}, upload=13803]: Done uploading piece. err: <nil>
Aug 7 22:58:36.091808572 (13h18m35.681153549s) [noise=true , address=45.91.116.14:4221 , piece_id=J54CXKHJH2KOUV3MPT36F6JHOI67GMGCMSOKHNJUDL5I7NTRLCBA, seg_pos={0 0}, upload=13803]: Done uploading piece. err: <nil>
Aug 7 22:58:36.096744555 (13h18m35.686089543s) [noise=true , address=104.36.82.55:4205 , piece_id=VK2NPARLAXL5YR5XBVABX4YBAACSXXBUXUJJJAUTPD7XG7EBNFCQ, seg_pos={0 0}, upload=13803]: Done uploading piece. err: <nil>
Aug 7 22:58:36.171642074 (13h18m35.760987061s) [noise=true , address=104.36.82.55:4202 , piece_id=UFXSOKDOHBYJYXJIKWF6AKQKXT6VLTVLKKCRBLYFOXQ554LF3OMA, seg_pos={0 0}, upload=13803]: Done uploading piece. err: <nil>
Aug 7 22:58:36.171668725 (13h18m35.761013712s) [seg_pos={0 0}, upload=13803]: Segment reached optimal threshold of 14 pieces.
Aug 7 22:58:36.171958509 (13h18m35.761303496s) [noise=true , address=104.36.82.4:4818 , piece_id=T4K2OGU3TC27H5LE4CTDLAOYR6DOP3S2PBVV7KUX4KKVFNMZNXBQ, seg_pos={0 0}, upload=13803]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1235eFkz7CFwwiM8UUZU1vnqDneEic7RpFzeTU46gvTMZWhhox8): protocol: read tcp 188.166.225.170:34616->104.36.82.4:4818: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 7 22:58:36.172074250 (13h18m35.761419227s) [noise=true , address=45.91.116.4:4955 , piece_id=WYMA4QA3QF55ZHZAUTMHWPLXJCSBOTN3NYLRJUWEMRFVUKR47VLA, seg_pos={0 0}, upload=13803]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VULW731AQGUMf4KLbhPUDNAEiW2AxaXJQYHzqiPiBuWG17L2B): protocol: write tcp 188.166.225.170:39210->45.91.116.4:4955: use of closed network connection; write tcp 188.166.225.170:39210->45.91.116.4:4955: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39210->45.91.116.4:4955: use of closed network connection
Aug 7 22:58:36.172112223 (13h18m35.7614572s) [noise=true , address=104.36.82.15:4270 , piece_id=2MTKGO66NWBUQXDMTN4NRYKIWHT6GKRW3LAHTZHLOVBGE7OINGZA, seg_pos={0 0}, upload=13803]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nkuzy3vxeLdHhVs73Nceje7qXDiGJ1zEhpGgGKSk3cKb7vJbB): protocol: rpc: dial tcp 104.36.82.15:4270: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 7 22:58:36.172191986 (13h18m35.761536973s) [noise=true , address=104.36.82.23:4472 , piece_id=BIB3GU4R4VW7CCGT7S4VCDYJSV2I6FV4NKLLSCZZJ5QDOJGYHJKQ, seg_pos={0 0}, upload=13803]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12otbB1RmPUKSKo813hrKPQVAyeqmhjhApxi6ndtHBCfn9bGGct): protocol: write tcp 188.166.225.170:47876->104.36.82.23:4472: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 7 22:58:36.172216163 (13h18m35.76156115s) [seg_pos={0 0}, upload=13803]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Aug 7 22:58:36.172285585 (13h18m35.761630572s) [upload=13803]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Aug 7 22:58:36.172289833 (13h18m35.76163482s) [upload=13803]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Aug 7 22:58:36.490569647 (13h18m36.079914634s) [upload=13803]: Done uploading segments...
Aug 7 22:58:36.490587652 (13h18m36.079932639s) [upload=13803]: Upload finished. err: <nil>
Duration: 6.367182482s
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)antiblip:
1 out of 60 is failed (threshold=3) [....................X.......................................] duration:
650.763718ms target:
https://us1.storj.io/login
FAIL (2025-08-07 23:10:56 UTC)
orbiter warning: Get "https://us1.storj.io/login": dial tcp 34.150.199.48:443: connect: connection refused
Duration: 650.789457ms
core
satellite
us1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline)antiblip:
4 out of 60 is failed (threshold=3) [XX.X.X......................................................] duration:
2.414352346s
FAIL (2025-08-07 23:10:56 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.414367634s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)antiblip:
2 out of 60 is failed (threshold=3) [X..X........................................................] duration:
1.985313792s
FAIL (2025-08-07 23:10:54 UTC)
orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded
Duration: 1.985339582s
core
uplink
us1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
7 out of 60 is failed (threshold=3) [XXXXXX.X....................................................] duration:
5.788513732s
FAIL (2025-08-07 23:10:42 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.788525015s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)antiblip:
3 out of 60 is failed (threshold=3) [..XXX.......................................................] duration:
5.970142066s
FAIL (2025-08-07 23:10:39 UTC)
orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded
Duration: 5.970164009s
core
uplink
us1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline)antiblip:
5 out of 60 is failed (threshold=3) [XXX.X.X.....................................................] duration:
2.258762414s
FAIL (2025-08-07 23:10:37 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.258789015s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [XXX...X.....................................................] duration:
6.083850762s
FAIL (2025-08-07 23:10:34 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.083871402s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
6 out of 60 is failed (threshold=3) [XXXX..X.X...................................................] duration:
6.389331594s
FAIL (2025-08-07 23:10:27 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
load=13236]: Begin upload segment...
Aug 7 22:25:52.992307080 (12h45m52.581652067s) [seg_pos={0 0}, upload=13236]: Done begin upload segment.
Aug 7 22:25:52.992331798 (12h45m52.581676775s) [noise=true , address=104.36.82.13:4213 , piece_id=5W5FD4WMMSZ7BKHFQI4GYOODXZ2P2MMAIQNK6CD6XILH22C53JGQ, seg_pos={0 0}, upload=13236]: Uploading piece...
Aug 7 22:25:52.992683770 (12h45m52.582028757s) [noise=true , address=104.36.82.9:4024 , piece_id=VBFOHRWDKWRY2774KJVK3JXJSMG4CPVDBBECCSZVJHR4274KLRMA, seg_pos={0 0}, upload=13236]: Uploading piece...
Aug 7 22:25:52.992785475 (12h45m52.582130462s) [noise=true , address=104.36.82.22:4527 , piece_id=JBNJOSETDAB6VG7PGVXIBUP6GPF2FBPJURBZNBR3BJOGMRK3EIQA, seg_pos={0 0}, upload=13236]: Uploading piece...
Aug 7 22:25:52.992942396 (12h45m52.582287373s) [noise=true , address=45.91.116.11:4124 , piece_id=TLATR2WNDVRR3WWTT3FZARUJ2QWWQFZBDYXLZ7PUPFOAEGJI4BTQ, seg_pos={0 0}, upload=13236]: Uploading piece...
Aug 7 22:25:52.993132439 (12h45m52.582477457s) [noise=true , address=104.36.82.5:4855 , piece_id=6EW7GNJ6KVWKNFHGFZVPEYPD4TDIVVZUXRW6AGJ3IGVANEAHYDUA, seg_pos={0 0}, upload=13236]: Uploading piece...
Aug 7 22:25:52.993337862 (12h45m52.582682849s) [noise=true , address=64.49.13.170:4717 , piece_id=52FN65VRQ24LBOGHJ22EQQM3MNI3EB2UXB57P5LD5GPLSABNS22Q, seg_pos={0 0}, upload=13236]: Uploading piece...
Aug 7 22:25:52.993511144 (12h45m52.582856131s) [noise=true , address=104.36.82.60:4361 , piece_id=EPWXHNXPVAUX75FUAXY57QZDKRXIEDHRXFL6PEBQGXY2A3Q4KNHA, seg_pos={0 0}, upload=13236]: Uploading piece...
Aug 7 22:25:52.993715204 (12h45m52.583060191s) [noise=true , address=104.36.82.58:4281 , piece_id=5UH27M25HE2LEZJ53H7QBKU6I6N2JE6F4PZTLOE6XP3GEDCZLEEQ, seg_pos={0 0}, upload=13236]: Uploading piece...
Aug 7 22:25:52.993820806 (12h45m52.583165784s) [noise=true , address=45.91.116.9:4284 , piece_id=LT3RP2EBVHIFIMAIGM3DW6QMPNLKZCK5FD4GQEQI6BFWYD56ZUBQ, seg_pos={0 0}, upload=13236]: Uploading piece...
Aug 7 22:25:52.994068250 (12h45m52.583413237s) [noise=true , address=79.127.205.233:4633 , piece_id=LBO5XLAXBRDURMDMN6MZ4HLVSG4TTUZPWCFTOTB6QMWAS5IIRLBA, seg_pos={0 0}, upload=13236]: Uploading piece...
Aug 7 22:25:52.994092416 (12h45m52.583437393s) [noise=true , address=104.36.82.60:4346 , piece_id=3NU3QBO5DAW5ZB7NTAHWP4C7RE4BYDPHZ77JOUQT3R3CBTOMONLA, seg_pos={0 0}, upload=13236]: Uploading piece...
Aug 7 22:25:52.994295394 (12h45m52.583640371s) [noise=true , address=45.91.116.7:4037 , piece_id=2NDGWBVYO3D2RYJ57QSXT632TT24SMRNWA3YZ2YW2BN6TBBLMRQA, seg_pos={0 0}, upload=13236]: Uploading piece...
Aug 7 22:25:52.994307056 (12h45m52.583652033s) [noise=true , address=104.36.82.46:4505 , piece_id=A7XPGAYUTNB26A6DJQNQ4NZ4FG27FUBQNEH5AQ7IFLYR3DLGG6SQ, seg_pos={0 0}, upload=13236]: Uploading piece...
Aug 7 22:25:52.994547126 (12h45m52.583892103s) [noise=true , address=104.36.82.61:4389 , piece_id=WBB2Y7YYCEML23D3CKJHGIEOUELRTVG3NXVGI4JDD7XPNQ4STENQ, seg_pos={0 0}, upload=13236]: Uploading piece...
Aug 7 22:25:52.994782436 (12h45m52.584127423s) [noise=true , address=64.49.13.154:4934 , piece_id=LDQTK4SFJPTL7NAY5XXKBIFCVLNRSLHLACTJ65MWCHFSBV4EKJYA, seg_pos={0 0}, upload=13236]: Uploading piece...
Aug 7 22:25:52.998117076 (12h45m52.587462053s) [noise=true , address=104.36.82.56:4252 , piece_id=XQWAMPOYTVGKYEHIEEEIDDST5BOIN2XCT7B3SFUR4U7PQBROEELQ, seg_pos={0 0}, upload=13236]: Uploading piece...
Aug 7 22:25:52.998596283 (12h45m52.58794127s) [noise=true , address=104.36.82.15:4261 , piece_id=C3RUC2F5LVRKH4MYTXDYOAP5KJGRZ6TAX57XULVBAKOS7ZK4QONQ, seg_pos={0 0}, upload=13236]: Uploading piece...
Aug 7 22:25:52.998917628 (12h45m52.588262615s) [noise=true , address=104.36.82.23:4473 , piece_id=N54OVKSO4KKX4NPORULBLAPVE5UT5RU6BBIEJCCMKG5553GVY6JA, seg_pos={0 0}, upload=13236]: Uploading piece...
Aug 7 22:25:54.121532197 (12h45m53.710877184s) [upload=13236]: Next returned nil segment
Aug 7 22:25:54.121549150 (12h45m53.710894137s) [upload=13236]: Waiting for error group managing segments...
Aug 7 22:25:54.447812563 (12h45m54.03715755s) [noise=true , address=79.127.205.233:4633 , piece_id=LBO5XLAXBRDURMDMN6MZ4HLVSG4TTUZPWCFTOTB6QMWAS5IIRLBA, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil>
Aug 7 22:25:54.450658769 (12h45m54.040003756s) [noise=true , address=64.49.13.170:4717 , piece_id=52FN65VRQ24LBOGHJ22EQQM3MNI3EB2UXB57P5LD5GPLSABNS22Q, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil>
Aug 7 22:25:54.505775361 (12h45m54.095120349s) [noise=true , address=64.49.13.154:4934 , piece_id=LDQTK4SFJPTL7NAY5XXKBIFCVLNRSLHLACTJ65MWCHFSBV4EKJYA, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil>
Aug 7 22:25:55.074032067 (12h45m54.663377054s) [noise=true , address=104.36.82.60:4346 , piece_id=3NU3QBO5DAW5ZB7NTAHWP4C7RE4BYDPHZ77JOUQT3R3CBTOMONLA, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil>
Aug 7 22:25:55.160612428 (12h45m54.749957415s) [noise=true , address=104.36.82.61:4389 , piece_id=WBB2Y7YYCEML23D3CKJHGIEOUELRTVG3NXVGI4JDD7XPNQ4STENQ, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil>
Aug 7 22:25:55.161069252 (12h45m54.750414239s) [noise=true , address=104.36.82.22:4527 , piece_id=JBNJOSETDAB6VG7PGVXIBUP6GPF2FBPJURBZNBR3BJOGMRK3EIQA, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil>
Aug 7 22:25:55.172460447 (12h45m54.761805434s) [noise=true , address=104.36.82.60:4361 , piece_id=EPWXHNXPVAUX75FUAXY57QZDKRXIEDHRXFL6PEBQGXY2A3Q4KNHA, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil>
Aug 7 22:25:55.217919931 (12h45m54.807264908s) [noise=true , address=104.36.82.56:4252 , piece_id=XQWAMPOYTVGKYEHIEEEIDDST5BOIN2XCT7B3SFUR4U7PQBROEELQ, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil>
Aug 7 22:25:55.226637230 (12h45m54.815982217s) [noise=true , address=104.36.82.58:4281 , piece_id=5UH27M25HE2LEZJ53H7QBKU6I6N2JE6F4PZTLOE6XP3GEDCZLEEQ, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil>
Aug 7 22:25:55.271492068 (12h45m54.860837055s) [noise=true , address=45.91.116.11:4124 , piece_id=TLATR2WNDVRR3WWTT3FZARUJ2QWWQFZBDYXLZ7PUPFOAEGJI4BTQ, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil>
Aug 7 22:25:55.281497411 (12h45m54.870842388s) [noise=true , address=45.91.116.9:4284 , piece_id=LT3RP2EBVHIFIMAIGM3DW6QMPNLKZCK5FD4GQEQI6BFWYD56ZUBQ, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil>
Aug 7 22:25:55.402216807 (12h45m54.991561794s) [noise=true , address=104.36.82.9:4024 , piece_id=VBFOHRWDKWRY2774KJVK3JXJSMG4CPVDBBECCSZVJHR4274KLRMA, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil>
Aug 7 22:25:55.632857570 (12h45m55.222202558s) [noise=true , address=104.36.82.5:4855 , piece_id=6EW7GNJ6KVWKNFHGFZVPEYPD4TDIVVZUXRW6AGJ3IGVANEAHYDUA, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil>
Aug 7 22:25:56.838313689 (12h45m56.427658676s) [noise=true , address=104.36.82.13:4213 , piece_id=5W5FD4WMMSZ7BKHFQI4GYOODXZ2P2MMAIQNK6CD6XILH22C53JGQ, seg_pos={0 0}, upload=13236]: Done uploading piece. err: <nil>
Aug 7 22:25:56.838338066 (12h45m56.427683053s) [seg_pos={0 0}, upload=13236]: Segment reached optimal threshold of 14 pieces.
Aug 7 22:25:56.838585389 (12h45m56.427930366s) [noise=true , address=45.91.116.7:4037 , piece_id=2NDGWBVYO3D2RYJ57QSXT632TT24SMRNWA3YZ2YW2BN6TBBLMRQA, seg_pos={0 0}, upload=13236]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iNPwE2XSfXcaXAXcrH2jcyNGYeDiiWvzAcDQffrmjPwTVqVmK): protocol: expected piece hash; write tcp 188.166.225.170:55570->45.91.116.7:4037: use of closed network connection; write tcp 188.166.225.170:55570->45.91.116.7:4037: use of closed network connection; EOF
Aug 7 22:25:56.838630465 (12h45m56.427975452s) [noise=true , address=104.36.82.46:4505 , piece_id=A7XPGAYUTNB26A6DJQNQ4NZ4FG27FUBQNEH5AQ7IFLYR3DLGG6SQ, seg_pos={0 0}, upload=13236]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i6itGoBhLsLiqC8et8Y5L62fLjcWxy18u39NAr4R3nXNMWcsf): protocol: write tcp 188.166.225.170:56556->104.36.82.46:4505: use of closed network connection; write tcp 188.166.225.170:56556->104.36.82.46:4505: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56556->104.36.82.46:4505: use of closed network connection
Aug 7 22:25:56.838682564 (12h45m56.428027551s) [noise=true , address=104.36.82.23:4473 , piece_id=N54OVKSO4KKX4NPORULBLAPVE5UT5RU6BBIEJCCMKG5553GVY6JA, seg_pos={0 0}, upload=13236]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1AKpkGZ6XbeqCTAYe273MACVvMCLcPPJ4By5EX2MkNuxC6zRpC): protocol: write tcp 188.166.225.170:50948->104.36.82.23:4473: use of closed network connection; write tcp 188.166.225.170:50948->104.36.82.23:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50948->104.36.82.23:4473: use of closed network connection
Aug 7 22:25:56.838697754 (12h45m56.428042731s) [noise=true , address=104.36.82.15:4261 , piece_id=C3RUC2F5LVRKH4MYTXDYOAP5KJGRZ6TAX57XULVBAKOS7ZK4QONQ, seg_pos={0 0}, upload=13236]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i1jcJdEP7Ch9uaGcjDaVxpodx55dJEHUKFbcr1GeKQ9waWjYz): protocol: expected piece hash; context canceled; EOF
Aug 7 22:25:56.838720558 (12h45m56.428065535s) [seg_pos={0 0}, upload=13236]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Aug 7 22:25:56.838823714 (12h45m56.428168701s) [upload=13236]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Aug 7 22:25:56.838828013 (12h45m56.428173s) [upload=13236]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Aug 7 22:30:17.177125535 (12h50m16.766470522s) [upload=13236]: Done uploading segments...
Aug 7 22:30:17.177153268 (12h50m16.766498255s) [upload=13236]: Upload finished. err: metaclient: context deadline exceeded
Duration: 6.389356642s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [XXXXX.......................................................] duration:
6.564546334s
FAIL (2025-08-07 23:10:25 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
:32.805219872 (12h43m32.394564859s) [upload=13200]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Aug 7 22:23:32.805223589 (12h43m32.394568576s) [upload=13200]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Aug 7 22:24:02.160684947 (12h44m1.750029944s) [upload=13200]: Next segment err: context deadline exceeded
Aug 7 22:24:02.160927551 (12h44m1.750272538s) [upload=13200]: Done uploading segments...
Aug 7 22:24:02.160942019 (12h44m1.750287007s) [upload=13200]: Upload finished. err: context deadline exceeded
Aug 7 22:24:12.500861314 (12h44m12.090206301s) [upload=13211]: Starting upload
Aug 7 22:24:12.500912582 (12h44m12.090257569s) [upload=13211]: Done starting upload
Aug 7 22:24:12.500973068 (12h44m12.090318045s) [upload=13211]: Uploading segments...
Aug 7 22:24:12.501078069 (12h44m12.090423056s) [upload=13211]: Got next segment. Inline: false
Aug 7 22:24:12.501106834 (12h44m12.090451821s) [upload=13211]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Aug 7 22:24:12.501110571 (12h44m12.090455558s) [upload=13211]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Aug 7 22:24:25.371267525 (12h44m24.960612512s) [seg_pos={0 0}, upload=13211]: Begin upload segment...
Aug 7 22:24:25.371330936 (12h44m24.960675953s) [seg_pos={0 0}, upload=13211]: Done begin upload segment.
Aug 7 22:24:25.371383426 (12h44m24.960728413s) [noise=true , address=77.90.31.14:4661 , piece_id=73RH6546ZSA46KNTPMI3YPHM65Y3DJG5PCAAPGYJJFXDQ6KM5MRA, seg_pos={0 0}, upload=13211]: Uploading piece...
Aug 7 22:24:25.371396932 (12h44m24.96074194s) [noise=true , address=77.90.31.19:4841 , piece_id=GPGSM3FHS76WAHMKYCXKIU57NEENBDZFAMIBSFP6JWV5K33BDCOQ, seg_pos={0 0}, upload=13211]: Uploading piece...
Aug 7 22:24:25.371863044 (12h44m24.961208021s) [noise=true , address=45.91.116.9:4261 , piece_id=F3JBOLQO76A5WQKYNQAKPAWVWZO5U5EYX7ZZIKQMOAUZUQCRZ7GA, seg_pos={0 0}, upload=13211]: Uploading piece...
Aug 7 22:24:25.372024263 (12h44m24.96136925s) [noise=true , address=45.91.116.12:4225 , piece_id=47OO6QUMLHWH54IAXRLTQEN56Z5ITGXCGKSTQFGSHTF32X3Z7HSQ, seg_pos={0 0}, upload=13211]: Uploading piece...
Aug 7 22:24:25.372219627 (12h44m24.961564614s) [noise=true , address=104.36.82.2:4729 , piece_id=JYG3SHRCFAE37242ADYQA2CXJRRGO5APTOBZ5725GMZ2FRMPYLKA, seg_pos={0 0}, upload=13211]: Uploading piece...
Aug 7 22:24:25.372406654 (12h44m24.961751641s) [noise=true , address=45.91.116.7:4009 , piece_id=N25P6VDHR6RAW44ZNMAPNYJUX4AUV2UI3ROEX67KTWLK7RRB4K7Q, seg_pos={0 0}, upload=13211]: Uploading piece...
Aug 7 22:24:25.372852217 (12h44m24.962197204s) [noise=true , address=45.91.116.6:4973 , piece_id=WLZJLHSER36K377KZCHHCLWNANIVKGHVKF2I3FBC5UDDLSIUCHFQ, seg_pos={0 0}, upload=13211]: Uploading piece...
Aug 7 22:24:25.373233576 (12h44m24.962578563s) [noise=true , address=45.91.116.4:4937 , piece_id=ALO2UUBMXABZGWARBYTKGYY5DHLJEGULYGIZ7E4JQCYNFGP4JG7Q, seg_pos={0 0}, upload=13211]: Uploading piece...
Aug 7 22:24:25.373581912 (12h44m24.9629269s) [noise=true , address=45.91.116.5:4299 , piece_id=LMJE5NXY2LTDFI3BZY2YGMAXFURMI5NOSZWW2ABZK4NFKQK5JROQ, seg_pos={0 0}, upload=13211]: Uploading piece...
Aug 7 22:24:25.373901074 (12h44m24.963246061s) [noise=true , address=104.36.82.11:4113 , piece_id=7UYIPJZWWCEIMHEOPGC3DKXXTULAHTNF5QMPJJYIHXTKPSDOMM5A, seg_pos={0 0}, upload=13211]: Uploading piece...
Aug 7 22:24:25.374042614 (12h44m24.963387601s) [noise=true , address=77.90.31.17:4772 , piece_id=UEGCYD3SF4UF3CHRT5GEAZ6B77VKMG7S7PMSSRLZT6IP4ZKOTHMA, seg_pos={0 0}, upload=13211]: Uploading piece...
Aug 7 22:24:25.374483207 (12h44m24.963828194s) [noise=true , address=104.36.82.7:4912 , piece_id=H6DSVHM5ZX2GWEPHGRDH2WEDRHCLEYLZ62ISEHBLEITT4NGNDT7A, seg_pos={0 0}, upload=13211]: Uploading piece...
Aug 7 22:24:25.374810343 (12h44m24.96415533s) [noise=true , address=45.91.116.4:4939 , piece_id=S4PTGHW7FTLW5Q7W2VQZHLAI4YQZ25U35J7LUYSS5EXK552GUE2A, seg_pos={0 0}, upload=13211]: Uploading piece...
Aug 7 22:24:25.375117380 (12h44m24.964462368s) [noise=true , address=77.90.31.11:4555 , piece_id=CTQZTDFEGF756WLYI6U6NGMGBLFOKXZX3QWTQE7KBAITW43WVIJA, seg_pos={0 0}, upload=13211]: Uploading piece...
Aug 7 22:24:25.375526834 (12h44m24.964871821s) [noise=true , address=45.91.116.10:4082 , piece_id=4QZEPLP5I47FAOHIYQWJEBODVUEWTM25IQ652LDWLFZERCPTEXHA, seg_pos={0 0}, upload=13211]: Uploading piece...
Aug 7 22:24:25.375955474 (12h44m24.965300461s) [noise=true , address=104.36.82.10:4078 , piece_id=QKGPABRCDU26EDHDNRMCHMUQ3XTOQSN47WG5RKF6S4LD7OG2HY6Q, seg_pos={0 0}, upload=13211]: Uploading piece...
Aug 7 22:24:27.072833335 (12h44m26.662178332s) [upload=13211]: Next returned nil segment
Aug 7 22:24:27.072902037 (12h44m26.662247034s) [upload=13211]: Waiting for error group managing segments...
Aug 7 22:24:27.436037535 (12h44m27.025382522s) [noise=true , address=104.36.82.7:4912 , piece_id=H6DSVHM5ZX2GWEPHGRDH2WEDRHCLEYLZ62ISEHBLEITT4NGNDT7A, seg_pos={0 0}, upload=13211]: Done uploading piece. err: <nil>
Aug 7 22:24:27.446916249 (12h44m27.036261236s) [noise=true , address=104.36.82.2:4729 , piece_id=JYG3SHRCFAE37242ADYQA2CXJRRGO5APTOBZ5725GMZ2FRMPYLKA, seg_pos={0 0}, upload=13211]: Done uploading piece. err: <nil>
Aug 7 22:24:27.566730713 (12h44m27.15607569s) [noise=true , address=45.91.116.9:4261 , piece_id=F3JBOLQO76A5WQKYNQAKPAWVWZO5U5EYX7ZZIKQMOAUZUQCRZ7GA, seg_pos={0 0}, upload=13211]: Done uploading piece. err: <nil>
Aug 7 22:24:27.594802733 (12h44m27.18414772s) [noise=true , address=45.91.116.4:4937 , piece_id=ALO2UUBMXABZGWARBYTKGYY5DHLJEGULYGIZ7E4JQCYNFGP4JG7Q, seg_pos={0 0}, upload=13211]: Done uploading piece. err: <nil>
Aug 7 22:24:27.613347363 (12h44m27.20269235s) [noise=true , address=77.90.31.14:4661 , piece_id=73RH6546ZSA46KNTPMI3YPHM65Y3DJG5PCAAPGYJJFXDQ6KM5MRA, seg_pos={0 0}, upload=13211]: Done uploading piece. err: <nil>
Aug 7 22:24:27.623084653 (12h44m27.21242964s) [noise=true , address=45.91.116.10:4082 , piece_id=4QZEPLP5I47FAOHIYQWJEBODVUEWTM25IQ652LDWLFZERCPTEXHA, seg_pos={0 0}, upload=13211]: Done uploading piece. err: <nil>
Aug 7 22:24:27.661186031 (12h44m27.250531018s) [noise=true , address=77.90.31.19:4841 , piece_id=GPGSM3FHS76WAHMKYCXKIU57NEENBDZFAMIBSFP6JWV5K33BDCOQ, seg_pos={0 0}, upload=13211]: Done uploading piece. err: <nil>
Aug 7 22:24:27.698198064 (12h44m27.287543051s) [noise=true , address=45.91.116.6:4973 , piece_id=WLZJLHSER36K377KZCHHCLWNANIVKGHVKF2I3FBC5UDDLSIUCHFQ, seg_pos={0 0}, upload=13211]: Done uploading piece. err: <nil>
Aug 7 22:24:27.724830297 (12h44m27.314175285s) [noise=true , address=45.91.116.4:4939 , piece_id=S4PTGHW7FTLW5Q7W2VQZHLAI4YQZ25U35J7LUYSS5EXK552GUE2A, seg_pos={0 0}, upload=13211]: Done uploading piece. err: <nil>
Aug 7 22:24:27.734317339 (12h44m27.323662326s) [noise=true , address=45.91.116.5:4299 , piece_id=LMJE5NXY2LTDFI3BZY2YGMAXFURMI5NOSZWW2ABZK4NFKQK5JROQ, seg_pos={0 0}, upload=13211]: Done uploading piece. err: <nil>
Aug 7 22:24:27.908078026 (12h44m27.497423013s) [noise=true , address=77.90.31.17:4772 , piece_id=UEGCYD3SF4UF3CHRT5GEAZ6B77VKMG7S7PMSSRLZT6IP4ZKOTHMA, seg_pos={0 0}, upload=13211]: Done uploading piece. err: <nil>
Aug 7 22:24:27.985050925 (12h44m27.574395912s) [noise=true , address=45.91.116.7:4009 , piece_id=N25P6VDHR6RAW44ZNMAPNYJUX4AUV2UI3ROEX67KTWLK7RRB4K7Q, seg_pos={0 0}, upload=13211]: Done uploading piece. err: <nil>
Aug 7 22:24:28.048748935 (12h44m27.638093922s) [noise=true , address=77.90.31.11:4555 , piece_id=CTQZTDFEGF756WLYI6U6NGMGBLFOKXZX3QWTQE7KBAITW43WVIJA, seg_pos={0 0}, upload=13211]: Done uploading piece. err: <nil>
Aug 7 22:24:28.048775115 (12h44m27.638120102s) [seg_pos={0 0}, upload=13211]: Segment reached optimal threshold of 13 pieces.
Aug 7 22:24:28.048900375 (12h44m27.638245362s) [noise=true , address=45.91.116.12:4225 , piece_id=47OO6QUMLHWH54IAXRLTQEN56Z5ITGXCGKSTQFGSHTF32X3Z7HSQ, seg_pos={0 0}, upload=13211]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: expected piece hash; context canceled; EOF
Aug 7 22:24:28.049006828 (12h44m27.638351815s) [noise=true , address=104.36.82.11:4113 , piece_id=7UYIPJZWWCEIMHEOPGC3DKXXTULAHTNF5QMPJJYIHXTKPSDOMM5A, seg_pos={0 0}, upload=13211]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV): protocol: write tcp 188.166.225.170:45632->104.36.82.11:4113: use of closed network connection; write tcp 188.166.225.170:45632->104.36.82.11:4113: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45632->104.36.82.11:4113: use of closed network connection
Aug 7 22:24:28.049050902 (12h44m27.638395889s) [noise=true , address=104.36.82.10:4078 , piece_id=QKGPABRCDU26EDHDNRMCHMUQ3XTOQSN47WG5RKF6S4LD7OG2HY6Q, seg_pos={0 0}, upload=13211]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr): protocol: write tcp 188.166.225.170:52940->104.36.82.10:4078: use of closed network connection; write tcp 188.166.225.170:52940->104.36.82.10:4078: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52940->104.36.82.10:4078: use of closed network connection
Aug 7 22:24:28.049061182 (12h44m27.638406169s) [seg_pos={0 0}, upload=13211]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Aug 7 22:24:28.049126247 (12h44m27.638471224s) [upload=13211]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Aug 7 22:24:28.049130154 (12h44m27.638475141s) [upload=13211]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Aug 7 22:27:23.601003390 (12h47m23.190348377s) [upload=13211]: Done uploading segments...
Aug 7 22:27:23.601027807 (12h47m23.190372794s) [upload=13211]: Upload finished. err: metaclient: context deadline exceeded
Duration: 6.564572555s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [XXXX...X....................................................] duration:
6.190889705s
FAIL (2025-08-07 23:10:18 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.190920735s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)antiblip:
2 out of 60 is failed (threshold=3) [.X..X.......................................................] duration:
1.859369277s
FAIL (2025-08-07 23:10:09 UTC)
orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded
Duration: 1.859410205s
core
uplink
us1
|