Orbiter: EU/US/AP/EDGE | Code | Config | Playbook | Escalation Procedure
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)antiblip:
5 out of 60 is failed (threshold=3) [.XXXXX......................................................] duration:
2.015867697s
FAIL (2025-08-08 00:40:06 UTC)
orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== m24.001855996s) [upload=14742]: Done starting upload Aug 7 23:53:24.412569290 (14h13m24.001914277s) [upload=14742]: Uploading segments... Aug 7 23:53:24.412638803 (14h13m24.001983831s) [upload=14742]: Got next segment. Inline: true Aug 7 23:53:24.412644354 (14h13m24.001989341s) [upload=14742]: Waiting for error group managing segments... Aug 7 23:53:24.412680903 (14h13m24.002025891s) [upload=14742]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 23:53:24.412685442 (14h13m24.002030429s) [upload=14742]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 23:53:24.412688137 (14h13m24.002033114s) [upload=14742]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 23:53:53.772057432 (14h13m53.361402419s) [upload=14742]: Done uploading segments... Aug 7 23:53:53.772119881 (14h13m53.361464908s) [upload=14742]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:54:30.496202349 (14h14m30.085547336s) [upload=14761]: Starting upload Aug 7 23:54:30.496270259 (14h14m30.085615246s) [upload=14761]: Done starting upload Aug 7 23:54:30.496341565 (14h14m30.085686552s) [upload=14761]: Uploading segments... Aug 7 23:54:30.496378957 (14h14m30.085723944s) [upload=14761]: Got next segment. Inline: true Aug 7 23:54:30.496383836 (14h14m30.085728823s) [upload=14761]: Waiting for error group managing segments... Aug 7 23:54:30.496441346 (14h14m30.085786333s) [upload=14761]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 23:54:30.496446426 (14h14m30.085791413s) [upload=14761]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 23:54:30.496449551 (14h14m30.085794539s) [upload=14761]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 23:54:59.847423574 (14h14m59.436768561s) [upload=14761]: Done uploading segments... Aug 7 23:54:59.847456066 (14h14m59.436801043s) [upload=14761]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:55:40.612588843 (14h15m40.20193383s) [upload=14781]: Starting upload Aug 7 23:55:40.612648929 (14h15m40.201993906s) [upload=14781]: Done starting upload Aug 7 23:55:40.612717029 (14h15m40.202062016s) [upload=14781]: Uploading segments... Aug 7 23:55:40.612750823 (14h15m40.20209581s) [upload=14781]: Got next segment. Inline: true Aug 7 23:55:40.612754510 (14h15m40.202099487s) [upload=14781]: Waiting for error group managing segments... Aug 7 23:55:40.612786883 (14h15m40.20213186s) [upload=14781]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 23:55:40.612790891 (14h15m40.202135868s) [upload=14781]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 23:55:40.612793075 (14h15m40.202138062s) [upload=14781]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 23:55:40.883469345 (14h15m40.472814333s) [upload=14781]: Done uploading segments... Aug 7 23:55:40.883486387 (14h15m40.472831375s) [upload=14781]: Upload finished. err: <nil> Aug 7 23:56:17.423712239 (14h16m17.013057226s) [upload=14789]: Starting upload Aug 7 23:56:17.423767895 (14h16m17.013112882s) [upload=14789]: Done starting upload Aug 7 23:56:17.423814835 (14h16m17.013159822s) [upload=14789]: Uploading segments... Aug 7 23:56:17.423847678 (14h16m17.013192665s) [upload=14789]: Got next segment. Inline: true Aug 7 23:56:17.423851335 (14h16m17.013196322s) [upload=14789]: Waiting for error group managing segments... Aug 7 23:56:17.423883377 (14h16m17.013228364s) [upload=14789]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 23:56:17.423887054 (14h16m17.013232041s) [upload=14789]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 23:56:17.423889258 (14h16m17.013234245s) [upload=14789]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 23:56:17.689327115 (14h16m17.278672102s) [upload=14789]: Done uploading segments... Aug 7 23:56:17.689345711 (14h16m17.278690688s) [upload=14789]: Upload finished. err: <nil> Aug 7 23:56:58.288741281 (14h16m57.878086268s) [upload=14801]: Starting upload Aug 7 23:56:58.288797098 (14h16m57.878142085s) [upload=14801]: Done starting upload Aug 7 23:56:58.288851010 (14h16m57.878195987s) [upload=14801]: Uploading segments... Aug 7 23:56:58.288882281 (14h16m57.878227268s) [upload=14801]: Got next segment. Inline: true Aug 7 23:56:58.288887911 (14h16m57.878232898s) [upload=14801]: Waiting for error group managing segments... Aug 7 23:56:58.288919792 (14h16m57.878264769s) [upload=14801]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 23:56:58.288923970 (14h16m57.878268957s) [upload=14801]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 23:56:58.288926485 (14h16m57.878271462s) [upload=14801]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 23:56:58.556529816 (14h16m58.145874803s) [upload=14801]: Done uploading segments... Aug 7 23:56:58.556544975 (14h16m58.145889962s) [upload=14801]: Upload finished. err: <nil> Aug 7 23:57:18.362794018 (14h17m17.952138995s) [upload=14806]: Starting upload Aug 7 23:57:18.362883679 (14h17m17.952228666s) [upload=14806]: Done starting upload Aug 7 23:57:18.362928104 (14h17m17.952273092s) [upload=14806]: Uploading segments... Aug 7 23:57:18.362957320 (14h17m17.952302307s) [upload=14806]: Got next segment. Inline: true Aug 7 23:57:18.362961077 (14h17m17.952306064s) [upload=14806]: Waiting for error group managing segments... Aug 7 23:57:18.363018407 (14h17m17.952363394s) [upload=14806]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 23:57:18.363022404 (14h17m17.952367392s) [upload=14806]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 23:57:18.363024739 (14h17m17.952369716s) [upload=14806]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 23:57:47.715297444 (14h17m47.304642431s) [upload=14806]: Done uploading segments... Aug 7 23:57:47.715410370 (14h17m47.304755357s) [upload=14806]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:57:53.375540080 (14h17m52.964885068s) [upload=14813]: Starting upload Aug 7 23:57:53.375600245 (14h17m52.964945232s) [upload=14813]: Done starting upload Aug 7 23:57:53.375654389 (14h17m52.964999376s) [upload=14813]: Uploading segments... Aug 7 23:57:53.375686622 (14h17m52.965031609s) [upload=14813]: Got next segment. Inline: true Aug 7 23:57:53.375690759 (14h17m52.965035746s) [upload=14813]: Waiting for error group managing segments... Aug 7 23:57:53.375725876 (14h17m52.965070863s) [upload=14813]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 23:57:53.375730134 (14h17m52.965075121s) [upload=14813]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 23:57:53.375734372 (14h17m52.965079359s) [upload=14813]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 23:58:22.726838464 (14h18m22.316183451s) [upload=14813]: Done uploading segments... Aug 7 23:58:22.726860075 (14h18m22.316205062s) [upload=14813]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:58:23.621870384 (14h18m23.211215371s) [upload=14822]: Starting upload Aug 7 23:58:23.621952000 (14h18m23.211296987s) [upload=14822]: Done starting upload Aug 7 23:58:23.622003359 (14h18m23.211348346s) [upload=14822]: Uploading segments... Aug 7 23:58:23.622043436 (14h18m23.211388413s) [upload=14822]: Got next segment. Inline: true Aug 7 23:58:23.622046772 (14h18m23.211391759s) [upload=14822]: Waiting for error group managing segments... Aug 7 23:58:23.622078573 (14h18m23.21142356s) [upload=14822]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 23:58:23.622082230 (14h18m23.211427217s) [upload=14822]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 23:58:23.622084534 (14h18m23.211429511s) [upload=14822]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 23:58:52.983677879 (14h18m52.573022876s) [upload=14822]: Done uploading segments... Aug 7 23:58:52.983713708 (14h18m52.573058685s) [upload=14822]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:59:11.446858470 (14h19m11.036203457s) [upload=14835]: Starting upload Aug 7 23:59:11.446906493 (14h19m11.03625148s) [upload=14835]: Done starting upload Aug 7 23:59:11.446947491 (14h19m11.036292479s) [upload=14835]: Uploading segments... Aug 7 23:59:11.446975475 (14h19m11.036320462s) [upload=14835]: Got next segment. Inline: true Aug 7 23:59:11.446978751 (14h19m11.036323738s) [upload=14835]: Waiting for error group managing segments... Aug 7 23:59:11.447008889 (14h19m11.036353876s) [upload=14835]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 23:59:11.447013638 (14h19m11.036358625s) [upload=14835]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 23:59:11.447016212 (14h19m11.0363612s) [upload=14835]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 23:59:40.799397635 (14h19m40.388742622s) [upload=14835]: Done uploading segments... Aug 7 23:59:40.799413266 (14h19m40.388758243s) [upload=14835]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:59:59.523080643 (14h19m59.11242563s) [upload=14847]: Starting upload Aug 7 23:59:59.523152089 (14h19m59.112497076s) [upload=14847]: Done starting upload Aug 7 23:59:59.523213727 (14h19m59.112558714s) [upload=14847]: Uploading segments... Aug 7 23:59:59.523251349 (14h19m59.112596336s) [upload=14847]: Got next segment. Inline: true Aug 7 23:59:59.523255287 (14h19m59.112600274s) [upload=14847]: Waiting for error group managing segments... Aug 7 23:59:59.523300013 (14h19m59.112645s) [upload=14847]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 23:59:59.523304251 (14h19m59.112649238s) [upload=14847]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 23:59:59.523307146 (14h19m59.112652133s) [upload=14847]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 8 00:00:28.877190143 (14h20m28.46653516s) [upload=14847]: Done uploading segments... Aug 8 00:00:28.877218477 (14h20m28.466563505s) [upload=14847]: Upload finished. err: metaclient: context deadline exceeded
Duration: 2.015898817s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)antiblip:
1 out of 60 is failed (threshold=3) [.............................X..............................] duration:
1.924859877s
FAIL (2025-08-08 00:40:05 UTC)
orbiter warning:
operation: DownloadObject
======ERROR======
uplink: object not found ("orbiter_us_0/1K")
======LOGS======
=14964]: Done uploading segments...
Aug 8 00:06:41.669168196 (14h26m41.258513183s) [upload=14964]: Upload finished. err: <nil>
Aug 8 00:07:06.258431851 (14h27m5.847776838s) [upload=14976]: Starting upload
Aug 8 00:07:06.258508298 (14h27m5.847853285s) [upload=14976]: Done starting upload
Aug 8 00:07:06.258574986 (14h27m5.847919983s) [upload=14976]: Uploading segments...
Aug 8 00:07:06.258618599 (14h27m5.847963616s) [upload=14976]: Got next segment. Inline: true
Aug 8 00:07:06.258624600 (14h27m5.847969587s) [upload=14976]: Waiting for error group managing segments...
Aug 8 00:07:06.258705555 (14h27m5.848050542s) [upload=14976]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Aug 8 00:07:06.258712268 (14h27m5.848057296s) [upload=14976]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline
Aug 8 00:07:06.258716226 (14h27m5.848061253s) [upload=14976]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Aug 8 00:07:06.522670976 (14h27m6.112015963s) [upload=14976]: Done uploading segments...
Aug 8 00:07:06.522717766 (14h27m6.112062773s) [upload=14976]: Upload finished. err: <nil>
Aug 8 00:07:36.371369594 (14h27m35.960714581s) [upload=14980]: Starting upload
Aug 8 00:07:36.371417074 (14h27m35.960762061s) [upload=14980]: Done starting upload
Aug 8 00:07:36.371466890 (14h27m35.960811877s) [upload=14980]: Uploading segments...
Aug 8 00:07:36.371508951 (14h27m35.960853938s) [upload=14980]: Got next segment. Inline: true
Aug 8 00:07:36.371512888 (14h27m35.960857875s) [upload=14980]: Waiting for error group managing segments...
Aug 8 00:07:36.371543446 (14h27m35.960888433s) [upload=14980]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Aug 8 00:07:36.371547434 (14h27m35.960892421s) [upload=14980]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline
Aug 8 00:07:36.371549708 (14h27m35.960894695s) [upload=14980]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Aug 8 00:07:36.789946570 (14h27m36.379291557s) [upload=14980]: Done uploading segments...
Aug 8 00:07:36.789964876 (14h27m36.379309863s) [upload=14980]: Upload finished. err: <nil>
Aug 8 00:08:16.866388064 (14h28m16.455733051s) [upload=14994]: Starting upload
Aug 8 00:08:16.866430886 (14h28m16.455775873s) [upload=14994]: Done starting upload
Aug 8 00:08:16.866496882 (14h28m16.455841869s) [upload=14994]: Uploading segments...
Aug 8 00:08:16.866525727 (14h28m16.455870714s) [upload=14994]: Got next segment. Inline: true
Aug 8 00:08:16.866529494 (14h28m16.455874481s) [upload=14994]: Waiting for error group managing segments...
Aug 8 00:08:16.866570884 (14h28m16.455915871s) [upload=14994]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Aug 8 00:08:16.866575212 (14h28m16.455920199s) [upload=14994]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline
Aug 8 00:08:16.866577626 (14h28m16.455922613s) [upload=14994]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Aug 8 00:08:17.138322091 (14h28m16.727667078s) [upload=14994]: Done uploading segments...
Aug 8 00:08:17.138338402 (14h28m16.727683389s) [upload=14994]: Upload finished. err: <nil>
Aug 8 00:08:57.633625328 (14h28m57.222970315s) [upload=15006]: Starting upload
Aug 8 00:08:57.633689390 (14h28m57.223034378s) [upload=15006]: Done starting upload
Aug 8 00:08:57.633744837 (14h28m57.223089824s) [upload=15006]: Uploading segments...
Aug 8 00:08:57.633775525 (14h28m57.223120512s) [upload=15006]: Got next segment. Inline: true
Aug 8 00:08:57.633779222 (14h28m57.223124199s) [upload=15006]: Waiting for error group managing segments...
Aug 8 00:08:57.633951743 (14h28m57.22329672s) [upload=15006]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Aug 8 00:08:57.633956161 (14h28m57.223301148s) [upload=15006]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline
Aug 8 00:08:57.633958886 (14h28m57.223303873s) [upload=15006]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Aug 8 00:08:57.920842105 (14h28m57.510187102s) [upload=15006]: Done uploading segments...
Aug 8 00:08:57.920865330 (14h28m57.510210327s) [upload=15006]: Upload finished. err: <nil>
Aug 8 00:09:43.341371442 (14h29m42.930716449s) [upload=15015]: Starting upload
Aug 8 00:09:43.341457767 (14h29m42.930802774s) [upload=15015]: Done starting upload
Aug 8 00:09:43.341511951 (14h29m42.930856938s) [upload=15015]: Uploading segments...
Aug 8 00:09:43.341579249 (14h29m42.930924236s) [upload=15015]: Got next segment. Inline: true
Aug 8 00:09:43.341585642 (14h29m42.930930639s) [upload=15015]: Waiting for error group managing segments...
Aug 8 00:09:43.341640437 (14h29m42.930985454s) [upload=15015]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Aug 8 00:09:43.341648422 (14h29m42.930993469s) [upload=15015]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline
Aug 8 00:09:43.341653391 (14h29m42.930998408s) [upload=15015]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Aug 8 00:09:43.627931533 (14h29m43.21727652s) [upload=15015]: Done uploading segments...
Aug 8 00:09:43.627948425 (14h29m43.217293402s) [upload=15015]: Upload finished. err: <nil>
Aug 8 00:10:45.166177580 (14h30m44.755522568s) [upload=15031]: Starting upload
Aug 8 00:10:45.166220523 (14h30m44.75556551s) [upload=15031]: Done starting upload
Aug 8 00:10:45.166273034 (14h30m44.755618011s) [upload=15031]: Uploading segments...
Aug 8 00:10:45.166297991 (14h30m44.755642979s) [upload=15031]: Got next segment. Inline: true
Aug 8 00:10:45.166301568 (14h30m44.755646555s) [upload=15031]: Waiting for error group managing segments...
Aug 8 00:10:45.166332277 (14h30m44.755677264s) [upload=15031]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Aug 8 00:10:45.166335783 (14h30m44.75568077s) [upload=15031]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline
Aug 8 00:10:45.166337998 (14h30m44.755682985s) [upload=15031]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Aug 8 00:10:45.444515471 (14h30m45.033860458s) [upload=15031]: Done uploading segments...
Aug 8 00:10:45.444533305 (14h30m45.033878282s) [upload=15031]: Upload finished. err: <nil>
Aug 8 00:11:54.979425886 (14h31m54.568770863s) [upload=15052]: Starting upload
Aug 8 00:11:54.979471082 (14h31m54.568816069s) [upload=15052]: Done starting upload
Aug 8 00:11:54.979517260 (14h31m54.568862237s) [upload=15052]: Uploading segments...
Aug 8 00:11:54.979549422 (14h31m54.568894399s) [upload=15052]: Got next segment. Inline: true
Aug 8 00:11:54.979553008 (14h31m54.568897995s) [upload=15052]: Waiting for error group managing segments...
Aug 8 00:11:54.979584328 (14h31m54.568929315s) [upload=15052]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Aug 8 00:11:54.979588356 (14h31m54.568933333s) [upload=15052]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline
Aug 8 00:11:54.979590661 (14h31m54.568935648s) [upload=15052]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Aug 8 00:11:55.340902393 (14h31m54.93024738s) [upload=15052]: Done uploading segments...
Aug 8 00:11:55.340920908 (14h31m54.930265895s) [upload=15052]: Upload finished. err: <nil>
Aug 8 00:12:44.503574128 (14h32m44.092919115s) [upload=15065]: Starting upload
Aug 8 00:12:44.503634083 (14h32m44.09297907s) [upload=15065]: Done starting upload
Aug 8 00:12:44.503710068 (14h32m44.093055055s) [upload=15065]: Uploading segments...
Aug 8 00:12:44.503762669 (14h32m44.093107656s) [upload=15065]: Got next segment. Inline: true
Aug 8 00:12:44.503767698 (14h32m44.093112685s) [upload=15065]: Waiting for error group managing segments...
Aug 8 00:12:44.503805531 (14h32m44.093150508s) [upload=15065]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Aug 8 00:12:44.503809379 (14h32m44.093154366s) [upload=15065]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline
Aug 8 00:12:44.503811633 (14h32m44.09315662s) [upload=15065]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Aug 8 00:12:44.792734473 (14h32m44.38207946s) [upload=15065]: Done uploading segments...
Aug 8 00:12:44.792752288 (14h32m44.382097275s) [upload=15065]: Upload finished. err: <nil>
Aug 8 00:13:19.961146705 (14h33m19.550491693s) [upload=15079]: Starting upload
Aug 8 00:13:19.961187835 (14h33m19.550532822s) [upload=15079]: Done starting upload
Aug 8 00:13:19.961236618 (14h33m19.550581605s) [upload=15079]: Uploading segments...
Aug 8 00:13:19.961268168 (14h33m19.550613145s) [upload=15079]: Got next segment. Inline: true
Aug 8 00:13:19.961273849 (14h33m19.550618836s) [upload=15079]: Waiting for error group managing segments...
Aug 8 00:13:19.961304548 (14h33m19.550649525s) [upload=15079]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Aug 8 00:13:19.961308345 (14h33m19.550653332s) [upload=15079]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline
Aug 8 00:13:19.961310910 (14h33m19.550655887s) [upload=15079]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Aug 8 00:13:20.385237160 (14h33m19.974582137s) [upload=15079]: Done uploading segments...
Aug 8 00:13:20.385255485 (14h33m19.974600472s) [upload=15079]: Upload finished. err: <nil>
Aug 8 00:14:06.785741109 (14h34m6.375086096s) [upload=15091]: Starting upload
Aug 8 00:14:06.785899151 (14h34m6.375244138s) [upload=15091]: Done starting upload
Aug 8 00:14:06.786002628 (14h34m6.375347615s) [upload=15091]: Uploading segments...
Aug 8 00:14:06.786034078 (14h34m6.375379065s) [upload=15091]: Got next segment. Inline: true
Aug 8 00:14:06.786037786 (14h34m6.375382773s) [upload=15091]: Waiting for error group managing segments...
Aug 8 00:14:06.786073094 (14h34m6.375418081s) [upload=15091]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Aug 8 00:14:06.786078945 (14h34m6.375423922s) [upload=15091]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline
Aug 8 00:14:06.786081339 (14h34m6.375426326s) [upload=15091]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Aug 8 00:14:07.092788097 (14h34m6.682133084s) [upload=15091]: Done uploading segments...
Aug 8 00:14:07.092803166 (14h34m6.682148153s) [upload=15091]: Upload finished. err: <nil>
Duration: 1.924891648s
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
6 out of 60 is failed (threshold=3) [...............X...X..........X.......X...X............X....] duration:
6.157994459s
FAIL (2025-08-08 00:39:59 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/4M):: after 7dd7aed873cca87b7cd76d50ee66fe90f8c443893d6355bc8519d2fde31a068f != before c1a76dc0bec8105ec080db0072ab886792f3545b344045a49a190b10eaa4f282
======LOGS======
[upload=15508]: Uploading segments...
Aug 8 00:38:24.258804844 (14h58m23.848149831s) [upload=15508]: Got next segment. Inline: false
Aug 8 00:38:24.258847265 (14h58m23.848192252s) [upload=15508]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Aug 8 00:38:24.258872243 (14h58m23.84821723s) [upload=15508]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Aug 8 00:38:24.709304765 (14h58m24.298649752s) [seg_pos={0 0}, upload=15508]: Begin upload segment...
Aug 8 00:38:24.709363377 (14h58m24.298708364s) [seg_pos={0 0}, upload=15508]: Done begin upload segment.
Aug 8 00:38:24.709365892 (14h58m24.298710879s) [noise=true , address=104.36.82.47:4534 , piece_id=G56O4SR7DQ6YHATRCIMLOARHFQBILTCEUU2S35Y5NQKYIQX22X6A, seg_pos={0 0}, upload=15508]: Uploading piece...
Aug 8 00:38:24.709380991 (14h58m24.298725978s) [noise=true , address=104.36.82.39:4734 , piece_id=2ZZDYY5W5NHZ7OFXZ3X566EWP5F6GV32KNCIGTAEMGWDZI3ZHNNQ, seg_pos={0 0}, upload=15508]: Uploading piece...
Aug 8 00:38:24.709715681 (14h58m24.299060668s) [noise=true , address=104.36.82.61:4386 , piece_id=3YTRS2CUXJ3X3AVS6SWI234NQD7G6XPSV7ONZWT22WS23P6GEUOA, seg_pos={0 0}, upload=15508]: Uploading piece...
Aug 8 00:38:24.710014734 (14h58m24.299359721s) [noise=true , address=45.91.116.11:4142 , piece_id=GWGM46SJ4ZLCAW2TTFUX4R3GXX6C4QAMBN62BUKXED6YVROHQ6PQ, seg_pos={0 0}, upload=15508]: Uploading piece...
Aug 8 00:38:24.710261737 (14h58m24.299606724s) [noise=true , address=104.36.82.9:4038 , piece_id=IUFHRP22JUSOJLB66HKMEIK6OJJJPR3NLRA4JHTLQZDMD5IEKQEA, seg_pos={0 0}, upload=15508]: Uploading piece...
Aug 8 00:38:24.710536713 (14h58m24.2998817s) [noise=true , address=45.91.116.10:4113 , piece_id=QNB2CD3IINPCU6F3NCCPV7MNKY2GBV7ZBWUXY6UAYT7IIPJDGTCQ, seg_pos={0 0}, upload=15508]: Uploading piece...
Aug 8 00:38:24.710790879 (14h58m24.300135866s) [noise=true , address=64.49.13.26:4948 , piece_id=QSDEMGP65XIE3CLDSG6ASMKJCOBSSDZN2GWEVFJB3QQEGVCHTUIQ, seg_pos={0 0}, upload=15508]: Uploading piece...
Aug 8 00:38:24.711027322 (14h58m24.300372309s) [noise=true , address=104.36.82.37:4656 , piece_id=QGCIU7D4AG66MLGC7XA435DKIIBGLSLSWHQDVIZ75NFJKTONV4UQ, seg_pos={0 0}, upload=15508]: Uploading piece...
Aug 8 00:38:24.711279785 (14h58m24.300624772s) [noise=true , address=104.36.82.45:4931 , piece_id=IBPB2JL6J2JDMHYYNI3UVNEZ5DUOOTA2SKQKRGV255F6NTCWUDWQ, seg_pos={0 0}, upload=15508]: Uploading piece...
Aug 8 00:38:24.711373895 (14h58m24.300718882s) [noise=true , address=104.36.82.56:4256 , piece_id=3JG4YZGAK5JV6U3HTKEFSOVAKKLN5H66SBTSCUAL6KDS5YELSUWA, seg_pos={0 0}, upload=15508]: Uploading piece...
Aug 8 00:38:24.711652839 (14h58m24.300997826s) [noise=true , address=104.36.82.58:4288 , piece_id=XBQQBKEJMENF6WMGIFEFC4CP2FA4NQ2LW22RUBS4DWTGBOWNHO7A, seg_pos={0 0}, upload=15508]: Uploading piece...
Aug 8 00:38:24.711891356 (14h58m24.301236343s) [noise=true , address=104.36.82.2:4745 , piece_id=JYMY6EJ2ZG6XEJUUUJOIFNNMI4QMINNS4Q77QYXPVLTSC7BLQEXA, seg_pos={0 0}, upload=15508]: Uploading piece...
Aug 8 00:38:24.712120744 (14h58m24.301465732s) [noise=true , address=104.36.82.54:4166 , piece_id=WTEO3MZJMTHGA5PTY5CCQ42SKTJI43M43EULZGGUXAQVIY7C46IA, seg_pos={0 0}, upload=15508]: Uploading piece...
Aug 8 00:38:24.712325797 (14h58m24.301670784s) [noise=true , address=104.36.82.3:4781 , piece_id=A4ZYEHF6HFDP6CX5MALR26F6RA7BE5CST5GDDULJBJR3ALUJAFVQ, seg_pos={0 0}, upload=15508]: Uploading piece...
Aug 8 00:38:24.712565686 (14h58m24.301910673s) [noise=true , address=104.36.82.16:4320 , piece_id=2OP3TMF3FNQE2WTV6HEY6HVNO6KR2FCMPXRZJYCUI3QJIYIRLBOQ, seg_pos={0 0}, upload=15508]: Uploading piece...
Aug 8 00:38:24.712743346 (14h58m24.302088333s) [noise=true , address=45.91.116.14:4192 , piece_id=XIT5DARFYND632CV46LHR3UBVDQ6GAZ2DH3HQFUCTUFMW2Y4VUOQ, seg_pos={0 0}, upload=15508]: Uploading piece...
Aug 8 00:38:24.712959169 (14h58m24.302304156s) [noise=true , address=45.91.116.5:4324 , piece_id=TP3YSP3WXJKXA3DKPDXBRHTAKQBC6EXMZCRG7HBCELRWY6TFVXAA, seg_pos={0 0}, upload=15508]: Uploading piece...
Aug 8 00:38:24.713184781 (14h58m24.302529768s) [noise=true , address=104.36.82.52:4097 , piece_id=DFJXCTAXSGUPKENZVYE6G4EAQF5SE5FKGRHI62QHCCVHYRAQDA7Q, seg_pos={0 0}, upload=15508]: Uploading piece...
Aug 8 00:38:25.841668344 (14h58m25.431013331s) [upload=15508]: Next returned nil segment
Aug 8 00:38:25.841685085 (14h58m25.431030072s) [upload=15508]: Waiting for error group managing segments...
Aug 8 00:38:26.112321232 (14h58m25.701666219s) [noise=true , address=64.49.13.26:4948 , piece_id=QSDEMGP65XIE3CLDSG6ASMKJCOBSSDZN2GWEVFJB3QQEGVCHTUIQ, seg_pos={0 0}, upload=15508]: Done uploading piece. err: <nil>
Aug 8 00:38:26.667567146 (14h58m26.256912133s) [noise=true , address=104.36.82.56:4256 , piece_id=3JG4YZGAK5JV6U3HTKEFSOVAKKLN5H66SBTSCUAL6KDS5YELSUWA, seg_pos={0 0}, upload=15508]: Done uploading piece. err: <nil>
Aug 8 00:38:26.669941781 (14h58m26.259286768s) [noise=true , address=104.36.82.3:4781 , piece_id=A4ZYEHF6HFDP6CX5MALR26F6RA7BE5CST5GDDULJBJR3ALUJAFVQ, seg_pos={0 0}, upload=15508]: Done uploading piece. err: <nil>
Aug 8 00:38:26.699819391 (14h58m26.289164378s) [noise=true , address=45.91.116.10:4113 , piece_id=QNB2CD3IINPCU6F3NCCPV7MNKY2GBV7ZBWUXY6UAYT7IIPJDGTCQ, seg_pos={0 0}, upload=15508]: Done uploading piece. err: <nil>
Aug 8 00:38:26.738125452 (14h58m26.327470439s) [noise=true , address=104.36.82.2:4745 , piece_id=JYMY6EJ2ZG6XEJUUUJOIFNNMI4QMINNS4Q77QYXPVLTSC7BLQEXA, seg_pos={0 0}, upload=15508]: Done uploading piece. err: <nil>
Aug 8 00:38:26.745536716 (14h58m26.334881703s) [noise=true , address=104.36.82.39:4734 , piece_id=2ZZDYY5W5NHZ7OFXZ3X566EWP5F6GV32KNCIGTAEMGWDZI3ZHNNQ, seg_pos={0 0}, upload=15508]: Done uploading piece. err: <nil>
Aug 8 00:38:26.768528297 (14h58m26.357873294s) [noise=true , address=45.91.116.11:4142 , piece_id=GWGM46SJ4ZLCAW2TTFUX4R3GXX6C4QAMBN62BUKXED6YVROHQ6PQ, seg_pos={0 0}, upload=15508]: Done uploading piece. err: <nil>
Aug 8 00:38:26.818308437 (14h58m26.407653414s) [noise=true , address=104.36.82.37:4656 , piece_id=QGCIU7D4AG66MLGC7XA435DKIIBGLSLSWHQDVIZ75NFJKTONV4UQ, seg_pos={0 0}, upload=15508]: Done uploading piece. err: <nil>
Aug 8 00:38:26.832514054 (14h58m26.421859041s) [noise=true , address=45.91.116.14:4192 , piece_id=XIT5DARFYND632CV46LHR3UBVDQ6GAZ2DH3HQFUCTUFMW2Y4VUOQ, seg_pos={0 0}, upload=15508]: Done uploading piece. err: <nil>
Aug 8 00:38:26.869135760 (14h58m26.458480748s) [noise=true , address=104.36.82.54:4166 , piece_id=WTEO3MZJMTHGA5PTY5CCQ42SKTJI43M43EULZGGUXAQVIY7C46IA, seg_pos={0 0}, upload=15508]: Done uploading piece. err: <nil>
Aug 8 00:38:27.131612271 (14h58m26.720957258s) [noise=true , address=45.91.116.5:4324 , piece_id=TP3YSP3WXJKXA3DKPDXBRHTAKQBC6EXMZCRG7HBCELRWY6TFVXAA, seg_pos={0 0}, upload=15508]: Done uploading piece. err: <nil>
Aug 8 00:38:27.152099560 (14h58m26.741444547s) [noise=true , address=104.36.82.58:4288 , piece_id=XBQQBKEJMENF6WMGIFEFC4CP2FA4NQ2LW22RUBS4DWTGBOWNHO7A, seg_pos={0 0}, upload=15508]: Done uploading piece. err: <nil>
Aug 8 00:38:27.501729825 (14h58m27.091074813s) [noise=true , address=104.36.82.16:4320 , piece_id=2OP3TMF3FNQE2WTV6HEY6HVNO6KR2FCMPXRZJYCUI3QJIYIRLBOQ, seg_pos={0 0}, upload=15508]: Done uploading piece. err: <nil>
Aug 8 00:38:27.778376595 (14h58m27.367721582s) [noise=true , address=104.36.82.9:4038 , piece_id=IUFHRP22JUSOJLB66HKMEIK6OJJJPR3NLRA4JHTLQZDMD5IEKQEA, seg_pos={0 0}, upload=15508]: Done uploading piece. err: <nil>
Aug 8 00:38:27.778417313 (14h58m27.3677623s) [seg_pos={0 0}, upload=15508]: Segment reached optimal threshold of 14 pieces.
Aug 8 00:38:27.778642765 (14h58m27.367987752s) [noise=true , address=104.36.82.47:4534 , piece_id=G56O4SR7DQ6YHATRCIMLOARHFQBILTCEUU2S35Y5NQKYIQX22X6A, seg_pos={0 0}, upload=15508]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VDrNGcCnaKEmG5NN4nex1JoZAgdCEfX5CdsKbxytfLfKzPbr): protocol: expected piece hash; context canceled; EOF
Aug 8 00:38:27.778734049 (14h58m27.368079036s) [noise=true , address=104.36.82.61:4386 , piece_id=3YTRS2CUXJ3X3AVS6SWI234NQD7G6XPSV7ONZWT22WS23P6GEUOA, seg_pos={0 0}, upload=15508]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19aPjJnya2Q3stEhiedDSth3NHV8phLRj5yY79hZv81UgDitMv): protocol: write tcp 188.166.225.170:42102->104.36.82.61:4386: use of closed network connection; write tcp 188.166.225.170:42102->104.36.82.61:4386: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42102->104.36.82.61:4386: use of closed network connection
Aug 8 00:38:27.778843298 (14h58m27.368188285s) [noise=true , address=104.36.82.45:4931 , piece_id=IBPB2JL6J2JDMHYYNI3UVNEZ5DUOOTA2SKQKRGV255F6NTCWUDWQ, seg_pos={0 0}, upload=15508]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pCgoSfxjtXPw5jpbz9RJaxRiGoYPdPFdoMPtD19F1ZWY8UwPu): protocol: expected piece hash; context canceled; EOF
Aug 8 00:38:27.778889437 (14h58m27.368234424s) [noise=true , address=104.36.82.52:4097 , piece_id=DFJXCTAXSGUPKENZVYE6G4EAQF5SE5FKGRHI62QHCCVHYRAQDA7Q, seg_pos={0 0}, upload=15508]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1sMhTktqf2H4aZSyFV3m9Z6eMVxL2U52bvUHVHUh2gUCVp32AM): protocol: expected piece hash; context canceled; EOF
Aug 8 00:38:27.778906359 (14h58m27.368251346s) [seg_pos={0 0}, upload=15508]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Aug 8 00:38:27.778981493 (14h58m27.36832647s) [upload=15508]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Aug 8 00:38:27.778985781 (14h58m27.368330768s) [upload=15508]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Aug 8 00:38:28.078873078 (14h58m27.668218065s) [upload=15508]: Done uploading segments...
Aug 8 00:38:28.078891813 (14h58m27.6682368s) [upload=15508]: Upload finished. err: <nil>
Duration: 6.158019215s
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline)antiblip:
3 out of 60 is failed (threshold=3) [X.X..X......................................................] duration:
1.84087826s
FAIL (2025-08-08 00:39:55 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1.840898129s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
4 out of 60 is failed (threshold=3) [......X.XX...X..............................................] duration:
6.954277224s
FAIL (2025-08-08 00:39:51 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: encryption: context deadline exceeded
======LOGS======
ZYSDFBHFTDOKPS73M57QA, seg_pos={0 0}, upload=14870]: Uploading piece...
Aug 8 00:01:16.246137011 (14h21m15.835481998s) [noise=true , address=104.36.82.9:4024 , piece_id=FM7MVIPG7C264O6Q64RUFOKZ2E5NMTIPWSIVMH5SWZUSCMFI7UGQ, seg_pos={0 0}, upload=14870]: Uploading piece...
Aug 8 00:01:16.246560380 (14h21m15.835905367s) [noise=true , address=64.49.13.169:4705 , piece_id=3MVXPRNKBBUSYAH2E7PCIWXDCUFQQ547IPOYIDLQ3T7QDYPBJA7A, seg_pos={0 0}, upload=14870]: Uploading piece...
Aug 8 00:01:16.247408974 (14h21m15.836753961s) [noise=true , address=45.91.116.5:4302 , piece_id=6RMPHN4LYSZVM6XKAUGSU7GY3TIKBGLKK4KNFMZQ2FMVEEVWV2LA, seg_pos={0 0}, upload=14870]: Uploading piece...
Aug 8 00:01:16.247678410 (14h21m15.837023397s) [noise=true , address=79.127.219.51:4181 , piece_id=4LYG6AL2BWD3UYQQAUUGZNJ7HUFXSHZ2TULWJOYSWTJLNVGHVKLQ, seg_pos={0 0}, upload=14870]: Uploading piece...
Aug 8 00:01:16.247919882 (14h21m15.837264869s) [noise=true , address=104.36.82.22:4512 , piece_id=7VOWVUPU3QVXIKML65NHX3NL7AC26QSDGHSSCRLUXIVZRNUPPKRA, seg_pos={0 0}, upload=14870]: Uploading piece...
Aug 8 00:01:16.247943888 (14h21m15.837288875s) [noise=true , address=104.36.82.20:4466 , piece_id=PLG7Z2WTYKUNB347NPT4ESIDK2NHCP66NPZJOXYL4A6L5OMFQIOQ, seg_pos={0 0}, upload=14870]: Uploading piece...
Aug 8 00:01:16.248293266 (14h21m15.837638253s) [noise=true , address=104.36.82.11:4143 , piece_id=E4AW52SZYZVRWPU4L56INK3IRNG7UNO5RRIRJANLMEPZQVDUHCEQ, seg_pos={0 0}, upload=14870]: Uploading piece...
Aug 8 00:01:16.248420540 (14h21m15.837765527s) [noise=true , address=64.49.13.139:4471 , piece_id=FPVXYIRNZXODU7N5RPCDOVVUECXWRTXCLINEXMOJTKNHF6YN46YQ, seg_pos={0 0}, upload=14870]: Uploading piece...
Aug 8 00:01:16.248557311 (14h21m15.837902288s) [noise=true , address=104.36.82.37:4659 , piece_id=627LA2AVL53NXFMDWSLGBO5GUKDAIGGSOPBFRPYIPYC5EAQC6P7Q, seg_pos={0 0}, upload=14870]: Uploading piece...
Aug 8 00:01:16.248758506 (14h21m15.838103493s) [noise=true , address=104.36.82.58:4275 , piece_id=VBV72SK36XIHMWSZTJJE4MZPLD5VZ6Z6UPBHK4A2ZI2C3O3RCVXA, seg_pos={0 0}, upload=14870]: Uploading piece...
Aug 8 00:01:16.248897792 (14h21m15.838242779s) [noise=true , address=104.36.82.55:4209 , piece_id=PJ4RK52LARBDG6WEVKXPZPX3RKJVIRSDCZY3B6CVCIUZO6BTWLBQ, seg_pos={0 0}, upload=14870]: Uploading piece...
Aug 8 00:01:16.248989629 (14h21m15.838334616s) [noise=true , address=104.36.82.43:4855 , piece_id=MJYC5ZGE6P72UYNYDRMH3BAZ6KIY57SA3EKJR54W6HPZNO3AE4IA, seg_pos={0 0}, upload=14870]: Uploading piece...
Aug 8 00:01:16.249186575 (14h21m15.838531552s) [noise=true , address=104.36.82.46:4487 , piece_id=6BZYNRXKBHFL3RQTGN3ZN4RV2O4USZ24GGCVSKFZWNS6N32L3NGQ, seg_pos={0 0}, upload=14870]: Uploading piece...
Aug 8 00:01:16.249273772 (14h21m15.838618759s) [noise=true , address=104.36.82.7:4920 , piece_id=BN5NCMLEGDVRIHO6YOWM5FNPCGFYC3BDKLOU7DIP6K6DYZXJ5A5A, seg_pos={0 0}, upload=14870]: Uploading piece...
Aug 8 00:01:16.249459598 (14h21m15.838804585s) [noise=true , address=77.90.31.14:4672 , piece_id=3ZYE5SLRNLY2ALXRR4QGCZT4SQ37U7FIAUOCVT63HQSBKPKV4UGQ, seg_pos={0 0}, upload=14870]: Uploading piece...
Aug 8 00:01:16.249717240 (14h21m15.839062227s) [noise=true , address=104.36.82.5:4844 , piece_id=REDGG4W6TJYEPEEPR65O5AVQDU2QMBNRA6HQQRUJA6JP7IS5T7YQ, seg_pos={0 0}, upload=14870]: Uploading piece...
Aug 8 00:01:17.413257645 (14h21m17.002602632s) [upload=14870]: Next returned nil segment
Aug 8 00:01:17.413273324 (14h21m17.002618311s) [upload=14870]: Waiting for error group managing segments...
Aug 8 00:01:17.710897294 (14h21m17.300242281s) [noise=true , address=64.49.13.169:4705 , piece_id=3MVXPRNKBBUSYAH2E7PCIWXDCUFQQ547IPOYIDLQ3T7QDYPBJA7A, seg_pos={0 0}, upload=14870]: Done uploading piece. err: <nil>
Aug 8 00:01:17.715058896 (14h21m17.304403883s) [noise=true , address=64.49.13.139:4471 , piece_id=FPVXYIRNZXODU7N5RPCDOVVUECXWRTXCLINEXMOJTKNHF6YN46YQ, seg_pos={0 0}, upload=14870]: Done uploading piece. err: <nil>
Aug 8 00:01:18.126218161 (14h21m17.715563148s) [noise=true , address=79.127.201.215:4356 , piece_id=VWNEIHNI4KPPF4QLW36INIWTUI2CD6MZYSDFBHFTDOKPS73M57QA, seg_pos={0 0}, upload=14870]: Done uploading piece. err: <nil>
Aug 8 00:01:18.139272367 (14h21m17.728617354s) [noise=true , address=79.127.219.51:4181 , piece_id=4LYG6AL2BWD3UYQQAUUGZNJ7HUFXSHZ2TULWJOYSWTJLNVGHVKLQ, seg_pos={0 0}, upload=14870]: Done uploading piece. err: <nil>
Aug 8 00:01:18.221269031 (14h21m17.810614018s) [noise=true , address=104.36.82.43:4855 , piece_id=MJYC5ZGE6P72UYNYDRMH3BAZ6KIY57SA3EKJR54W6HPZNO3AE4IA, seg_pos={0 0}, upload=14870]: Done uploading piece. err: <nil>
Aug 8 00:01:18.257350144 (14h21m17.846695121s) [noise=true , address=104.36.82.37:4659 , piece_id=627LA2AVL53NXFMDWSLGBO5GUKDAIGGSOPBFRPYIPYC5EAQC6P7Q, seg_pos={0 0}, upload=14870]: Done uploading piece. err: <nil>
Aug 8 00:01:18.258435089 (14h21m17.847780076s) [noise=true , address=104.36.82.44:4876 , piece_id=BA2G42OXQD3NLONUVD475S2BNC7T37T5PHPE3K3E5HJAJP4W5MKQ, seg_pos={0 0}, upload=14870]: Done uploading piece. err: <nil>
Aug 8 00:01:18.287010923 (14h21m17.87635591s) [noise=true , address=104.36.82.7:4920 , piece_id=BN5NCMLEGDVRIHO6YOWM5FNPCGFYC3BDKLOU7DIP6K6DYZXJ5A5A, seg_pos={0 0}, upload=14870]: Done uploading piece. err: <nil>
Aug 8 00:01:18.348793001 (14h21m17.938137988s) [noise=true , address=104.36.82.11:4143 , piece_id=E4AW52SZYZVRWPU4L56INK3IRNG7UNO5RRIRJANLMEPZQVDUHCEQ, seg_pos={0 0}, upload=14870]: Done uploading piece. err: <nil>
Aug 8 00:01:18.378129459 (14h21m17.967474436s) [noise=true , address=104.36.82.58:4275 , piece_id=VBV72SK36XIHMWSZTJJE4MZPLD5VZ6Z6UPBHK4A2ZI2C3O3RCVXA, seg_pos={0 0}, upload=14870]: Done uploading piece. err: <nil>
Aug 8 00:01:18.386414901 (14h21m17.975759888s) [noise=true , address=104.36.82.46:4487 , piece_id=6BZYNRXKBHFL3RQTGN3ZN4RV2O4USZ24GGCVSKFZWNS6N32L3NGQ, seg_pos={0 0}, upload=14870]: Done uploading piece. err: <nil>
Aug 8 00:01:18.516150823 (14h21m18.10549581s) [noise=true , address=104.36.82.9:4024 , piece_id=FM7MVIPG7C264O6Q64RUFOKZ2E5NMTIPWSIVMH5SWZUSCMFI7UGQ, seg_pos={0 0}, upload=14870]: Done uploading piece. err: <nil>
Aug 8 00:01:18.626647204 (14h21m18.215992191s) [noise=true , address=77.90.31.14:4672 , piece_id=3ZYE5SLRNLY2ALXRR4QGCZT4SQ37U7FIAUOCVT63HQSBKPKV4UGQ, seg_pos={0 0}, upload=14870]: Done uploading piece. err: <nil>
Aug 8 00:01:19.042108811 (14h21m18.631453798s) [noise=true , address=104.36.82.22:4512 , piece_id=7VOWVUPU3QVXIKML65NHX3NL7AC26QSDGHSSCRLUXIVZRNUPPKRA, seg_pos={0 0}, upload=14870]: Done uploading piece. err: <nil>
Aug 8 00:01:19.042136654 (14h21m18.631481641s) [seg_pos={0 0}, upload=14870]: Segment reached optimal threshold of 14 pieces.
Aug 8 00:01:19.042375541 (14h21m18.631720518s) [noise=true , address=45.91.116.5:4302 , piece_id=6RMPHN4LYSZVM6XKAUGSU7GY3TIKBGLKK4KNFMZQ2FMVEEVWV2LA, seg_pos={0 0}, upload=14870]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pJWhSLza8WaMuvvDwH4nae57UxQmfCdDvAkQ3v25MgPYHoZXx): protocol: expected piece hash; context canceled; EOF
Aug 8 00:01:19.042485912 (14h21m18.6318309s) [noise=true , address=104.36.82.55:4209 , piece_id=PJ4RK52LARBDG6WEVKXPZPX3RKJVIRSDCZY3B6CVCIUZO6BTWLBQ, seg_pos={0 0}, upload=14870]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12EmE2BgEZTG1LTsZ5p9ghqDSfg5ZQATt31Tqg5jZ5KQE2UjDnR): protocol: rpc: dial tcp 104.36.82.55:4209: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 8 00:01:19.042532361 (14h21m18.631877348s) [noise=true , address=104.36.82.20:4466 , piece_id=PLG7Z2WTYKUNB347NPT4ESIDK2NHCP66NPZJOXYL4A6L5OMFQIOQ, seg_pos={0 0}, upload=14870]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SqNLPRWY7fJRS6F9h4B4qVFv6U5jyWkCy3Um7jfjEdCShEjsD): protocol: write tcp 188.166.225.170:37056->104.36.82.20:4466: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 8 00:01:19.042616462 (14h21m18.631961449s) [noise=true , address=104.36.82.5:4844 , piece_id=REDGG4W6TJYEPEEPR65O5AVQDU2QMBNRA6HQQRUJA6JP7IS5T7YQ, seg_pos={0 0}, upload=14870]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12iovpNdbidkiv9mEHaAPjBubGmoqr4sUJWm8Cb64LzS6wuwLWa): protocol: rpc: dial tcp 104.36.82.5:4844: operation was canceled; conn closed: op; piecestore: piecestore close: context canceled
Aug 8 00:01:19.042628876 (14h21m18.631973863s) [seg_pos={0 0}, upload=14870]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Aug 8 00:01:19.042699029 (14h21m18.632044017s) [upload=14870]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Aug 8 00:01:19.042703128 (14h21m18.632048115s) [upload=14870]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Aug 8 00:01:19.344335316 (14h21m18.933680293s) [upload=14870]: Done uploading segments...
Aug 8 00:01:19.344353089 (14h21m18.933698077s) [upload=14870]: Upload finished. err: <nil>
Aug 8 00:02:30.778225454 (14h22m30.367570441s) [upload=14892]: Starting upload
Aug 8 00:02:30.778283114 (14h22m30.367628151s) [upload=14892]: Done starting upload
Aug 8 00:02:30.778411841 (14h22m30.367756828s) [upload=14892]: Uploading segments...
Aug 8 00:02:30.778539616 (14h22m30.367884623s) [upload=14892]: Got next segment. Inline: false
Aug 8 00:02:30.778583389 (14h22m30.367928376s) [upload=14892]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Aug 8 00:02:30.778587717 (14h22m30.367932704s) [upload=14892]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Aug 8 00:03:30.135980490 (14h23m29.725325467s) [upload=14892]: Next segment err: context deadline exceeded
Aug 8 00:03:30.136177137 (14h23m29.725522124s) [upload=14892]: Done uploading segments...
Aug 8 00:03:30.136196834 (14h23m29.725541821s) [upload=14892]: Upload finished. err: context deadline exceeded
Duration: 6.954308714s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)antiblip:
6 out of 60 is failed (threshold=3) [........X.XXXX....X.........................................] duration:
2.09611331s
FAIL (2025-08-08 00:39:51 UTC)
orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 9109253s) [upload=14753]: Starting upload Aug 7 23:54:04.489828037 (14h14m4.079173014s) [upload=14753]: Done starting upload Aug 7 23:54:04.489877612 (14h14m4.079222589s) [upload=14753]: Uploading segments... Aug 7 23:54:04.489903933 (14h14m4.07924892s) [upload=14753]: Got next segment. Inline: true Aug 7 23:54:04.489907469 (14h14m4.079252446s) [upload=14753]: Waiting for error group managing segments... Aug 7 23:54:04.489939651 (14h14m4.079284638s) [upload=14753]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 23:54:04.489943298 (14h14m4.079288285s) [upload=14753]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 23:54:04.489945472 (14h14m4.079290459s) [upload=14753]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 23:54:04.778726062 (14h14m4.368071049s) [upload=14753]: Done uploading segments... Aug 7 23:54:04.778744989 (14h14m4.368089976s) [upload=14753]: Upload finished. err: <nil> Aug 7 23:55:14.199719855 (14h15m13.789064892s) [upload=14770]: Starting upload Aug 7 23:55:14.199777645 (14h15m13.789122632s) [upload=14770]: Done starting upload Aug 7 23:55:14.199824415 (14h15m13.789169392s) [upload=14770]: Uploading segments... Aug 7 23:55:14.199852819 (14h15m13.789197806s) [upload=14770]: Got next segment. Inline: true Aug 7 23:55:14.199856436 (14h15m13.789201423s) [upload=14770]: Waiting for error group managing segments... Aug 7 23:55:14.199889419 (14h15m13.789234406s) [upload=14770]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 23:55:14.199893557 (14h15m13.789238544s) [upload=14770]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 23:55:14.199895951 (14h15m13.789240938s) [upload=14770]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 23:56:13.542516296 (14h16m13.131861283s) [upload=14770]: Done uploading segments... Aug 7 23:56:13.542536334 (14h16m13.131881321s) [upload=14770]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:56:14.428677736 (14h16m14.018022723s) [upload=14788]: Starting upload Aug 7 23:56:14.428730978 (14h16m14.018075965s) [upload=14788]: Done starting upload Aug 7 23:56:14.428789289 (14h16m14.018134276s) [upload=14788]: Uploading segments... Aug 7 23:56:14.428819026 (14h16m14.018164013s) [upload=14788]: Got next segment. Inline: true Aug 7 23:56:14.428822873 (14h16m14.01816785s) [upload=14788]: Waiting for error group managing segments... Aug 7 23:56:14.428855215 (14h16m14.018200202s) [upload=14788]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 23:56:14.428858562 (14h16m14.018203549s) [upload=14788]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 23:56:14.428860736 (14h16m14.018205723s) [upload=14788]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 23:57:13.781984834 (14h17m13.371329821s) [upload=14788]: Done uploading segments... Aug 7 23:57:13.782003370 (14h17m13.371348357s) [upload=14788]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:57:29.079359478 (14h17m28.668704466s) [upload=14808]: Starting upload Aug 7 23:57:29.079416147 (14h17m28.668761134s) [upload=14808]: Done starting upload Aug 7 23:57:29.079461845 (14h17m28.668806822s) [upload=14808]: Uploading segments... Aug 7 23:57:29.079488876 (14h17m28.668833863s) [upload=14808]: Got next segment. Inline: true Aug 7 23:57:29.079492263 (14h17m28.66883725s) [upload=14808]: Waiting for error group managing segments... Aug 7 23:57:29.079523392 (14h17m28.668868379s) [upload=14808]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 23:57:29.079527049 (14h17m28.668872036s) [upload=14808]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 23:57:29.079529484 (14h17m28.668874461s) [upload=14808]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 23:58:28.439377526 (14h18m28.028722513s) [upload=14808]: Done uploading segments... Aug 7 23:58:28.439395721 (14h18m28.028740708s) [upload=14808]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:58:29.333669502 (14h18m28.923014489s) [upload=14825]: Starting upload Aug 7 23:58:29.333742581 (14h18m28.923087568s) [upload=14825]: Done starting upload Aug 7 23:58:29.333835740 (14h18m28.923180727s) [upload=14825]: Uploading segments... Aug 7 23:58:29.333878000 (14h18m28.923222987s) [upload=14825]: Got next segment. Inline: true Aug 7 23:58:29.333881998 (14h18m28.923226975s) [upload=14825]: Waiting for error group managing segments... Aug 7 23:58:29.333911455 (14h18m28.923256442s) [upload=14825]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 23:58:29.333917496 (14h18m28.923262473s) [upload=14825]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 23:58:29.333919741 (14h18m28.923264728s) [upload=14825]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 23:59:28.692415040 (14h19m28.281760027s) [upload=14825]: Done uploading segments... Aug 7 23:59:28.692444777 (14h19m28.281789764s) [upload=14825]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:59:29.562470053 (14h19m29.15181504s) [upload=14840]: Starting upload Aug 7 23:59:29.562538985 (14h19m29.151883972s) [upload=14840]: Done starting upload Aug 7 23:59:29.562614629 (14h19m29.151959616s) [upload=14840]: Uploading segments... Aug 7 23:59:29.562650589 (14h19m29.151995566s) [upload=14840]: Got next segment. Inline: true Aug 7 23:59:29.562655498 (14h19m29.152000485s) [upload=14840]: Waiting for error group managing segments... Aug 7 23:59:29.562686507 (14h19m29.152031494s) [upload=14840]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 23:59:29.562690264 (14h19m29.152035241s) [upload=14840]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 23:59:29.562692428 (14h19m29.152037405s) [upload=14840]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 23:59:29.822497550 (14h19m29.411842537s) [upload=14840]: Done uploading segments... Aug 7 23:59:29.822514693 (14h19m29.41185968s) [upload=14840]: Upload finished. err: <nil> Aug 8 00:00:35.820554363 (14h20m35.40989935s) [upload=14856]: Starting upload Aug 8 00:00:35.820610611 (14h20m35.409955598s) [upload=14856]: Done starting upload Aug 8 00:00:35.820669503 (14h20m35.4100145s) [upload=14856]: Uploading segments... Aug 8 00:00:35.820742333 (14h20m35.41008732s) [upload=14856]: Got next segment. Inline: true Aug 8 00:00:35.820747282 (14h20m35.410092269s) [upload=14856]: Waiting for error group managing segments... Aug 8 00:00:35.820788371 (14h20m35.410133358s) [upload=14856]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 8 00:00:35.820793050 (14h20m35.410138037s) [upload=14856]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 8 00:00:35.820795485 (14h20m35.410140472s) [upload=14856]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 8 00:00:36.084321816 (14h20m35.673666803s) [upload=14856]: Done uploading segments... Aug 8 00:00:36.084338618 (14h20m35.673683615s) [upload=14856]: Upload finished. err: <nil> Aug 8 00:01:14.496466774 (14h21m14.085811781s) [upload=14868]: Starting upload Aug 8 00:01:14.496556174 (14h21m14.085901162s) [upload=14868]: Done starting upload Aug 8 00:01:14.496609206 (14h21m14.085954183s) [upload=14868]: Uploading segments... Aug 8 00:01:14.496639985 (14h21m14.085984972s) [upload=14868]: Got next segment. Inline: true Aug 8 00:01:14.496643592 (14h21m14.085988579s) [upload=14868]: Waiting for error group managing segments... Aug 8 00:01:14.496678638 (14h21m14.086023625s) [upload=14868]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 8 00:01:14.496682386 (14h21m14.086027373s) [upload=14868]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 8 00:01:14.496684681 (14h21m14.086029658s) [upload=14868]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 8 00:01:14.760681029 (14h21m14.350026016s) [upload=14868]: Done uploading segments... Aug 8 00:01:14.760702580 (14h21m14.350047567s) [upload=14868]: Upload finished. err: <nil> Aug 8 00:01:50.577049881 (14h21m50.166394858s) [upload=14878]: Starting upload Aug 8 00:01:50.577100538 (14h21m50.166445525s) [upload=14878]: Done starting upload Aug 8 00:01:50.577180091 (14h21m50.166525078s) [upload=14878]: Uploading segments... Aug 8 00:01:50.577228012 (14h21m50.166573s) [upload=14878]: Got next segment. Inline: true Aug 8 00:01:50.577231469 (14h21m50.166576456s) [upload=14878]: Waiting for error group managing segments... Aug 8 00:01:50.577270633 (14h21m50.166615621s) [upload=14878]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 8 00:01:50.577274721 (14h21m50.166619708s) [upload=14878]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 8 00:01:50.577276906 (14h21m50.166621893s) [upload=14878]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 8 00:01:50.863237339 (14h21m50.452582326s) [upload=14878]: Done uploading segments... Aug 8 00:01:50.863254973 (14h21m50.452599961s) [upload=14878]: Upload finished. err: <nil> Aug 8 00:02:41.474648957 (14h22m41.063993944s) [upload=14899]: Starting upload Aug 8 00:02:41.474716297 (14h22m41.064061274s) [upload=14899]: Done starting upload Aug 8 00:02:41.474771743 (14h22m41.06411673s) [upload=14899]: Uploading segments... Aug 8 00:02:41.474801099 (14h22m41.064146086s) [upload=14899]: Got next segment. Inline: true Aug 8 00:02:41.474804966 (14h22m41.064149953s) [upload=14899]: Waiting for error group managing segments... Aug 8 00:02:41.474841336 (14h22m41.064186323s) [upload=14899]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 8 00:02:41.474845214 (14h22m41.064190201s) [upload=14899]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 8 00:02:41.474847558 (14h22m41.064192535s) [upload=14899]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 8 00:03:40.835731182 (14h23m40.425076189s) [upload=14899]: Done uploading segments... Aug 8 00:03:40.835778212 (14h23m40.425123199s) [upload=14899]: Upload finished. err: metaclient: context deadline exceeded
Duration: 2.096143119s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
6 out of 60 is failed (threshold=3) [................X...X..........X.......X...X............X...] duration:
6.099775599s
FAIL (2025-08-08 00:39:37 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/4M):: after 7dd7aed873cca87b7cd76d50ee66fe90f8c443893d6355bc8519d2fde31a068f != before c1a76dc0bec8105ec080db0072ab886792f3545b344045a49a190b10eaa4f282
======LOGS======
[upload=15508]: Uploading segments...
Aug 8 00:38:24.258804844 (14h58m23.848149831s) [upload=15508]: Got next segment. Inline: false
Aug 8 00:38:24.258847265 (14h58m23.848192252s) [upload=15508]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Aug 8 00:38:24.258872243 (14h58m23.84821723s) [upload=15508]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Aug 8 00:38:24.709304765 (14h58m24.298649752s) [seg_pos={0 0}, upload=15508]: Begin upload segment...
Aug 8 00:38:24.709363377 (14h58m24.298708364s) [seg_pos={0 0}, upload=15508]: Done begin upload segment.
Aug 8 00:38:24.709365892 (14h58m24.298710879s) [noise=true , address=104.36.82.47:4534 , piece_id=G56O4SR7DQ6YHATRCIMLOARHFQBILTCEUU2S35Y5NQKYIQX22X6A, seg_pos={0 0}, upload=15508]: Uploading piece...
Aug 8 00:38:24.709380991 (14h58m24.298725978s) [noise=true , address=104.36.82.39:4734 , piece_id=2ZZDYY5W5NHZ7OFXZ3X566EWP5F6GV32KNCIGTAEMGWDZI3ZHNNQ, seg_pos={0 0}, upload=15508]: Uploading piece...
Aug 8 00:38:24.709715681 (14h58m24.299060668s) [noise=true , address=104.36.82.61:4386 , piece_id=3YTRS2CUXJ3X3AVS6SWI234NQD7G6XPSV7ONZWT22WS23P6GEUOA, seg_pos={0 0}, upload=15508]: Uploading piece...
Aug 8 00:38:24.710014734 (14h58m24.299359721s) [noise=true , address=45.91.116.11:4142 , piece_id=GWGM46SJ4ZLCAW2TTFUX4R3GXX6C4QAMBN62BUKXED6YVROHQ6PQ, seg_pos={0 0}, upload=15508]: Uploading piece...
Aug 8 00:38:24.710261737 (14h58m24.299606724s) [noise=true , address=104.36.82.9:4038 , piece_id=IUFHRP22JUSOJLB66HKMEIK6OJJJPR3NLRA4JHTLQZDMD5IEKQEA, seg_pos={0 0}, upload=15508]: Uploading piece...
Aug 8 00:38:24.710536713 (14h58m24.2998817s) [noise=true , address=45.91.116.10:4113 , piece_id=QNB2CD3IINPCU6F3NCCPV7MNKY2GBV7ZBWUXY6UAYT7IIPJDGTCQ, seg_pos={0 0}, upload=15508]: Uploading piece...
Aug 8 00:38:24.710790879 (14h58m24.300135866s) [noise=true , address=64.49.13.26:4948 , piece_id=QSDEMGP65XIE3CLDSG6ASMKJCOBSSDZN2GWEVFJB3QQEGVCHTUIQ, seg_pos={0 0}, upload=15508]: Uploading piece...
Aug 8 00:38:24.711027322 (14h58m24.300372309s) [noise=true , address=104.36.82.37:4656 , piece_id=QGCIU7D4AG66MLGC7XA435DKIIBGLSLSWHQDVIZ75NFJKTONV4UQ, seg_pos={0 0}, upload=15508]: Uploading piece...
Aug 8 00:38:24.711279785 (14h58m24.300624772s) [noise=true , address=104.36.82.45:4931 , piece_id=IBPB2JL6J2JDMHYYNI3UVNEZ5DUOOTA2SKQKRGV255F6NTCWUDWQ, seg_pos={0 0}, upload=15508]: Uploading piece...
Aug 8 00:38:24.711373895 (14h58m24.300718882s) [noise=true , address=104.36.82.56:4256 , piece_id=3JG4YZGAK5JV6U3HTKEFSOVAKKLN5H66SBTSCUAL6KDS5YELSUWA, seg_pos={0 0}, upload=15508]: Uploading piece...
Aug 8 00:38:24.711652839 (14h58m24.300997826s) [noise=true , address=104.36.82.58:4288 , piece_id=XBQQBKEJMENF6WMGIFEFC4CP2FA4NQ2LW22RUBS4DWTGBOWNHO7A, seg_pos={0 0}, upload=15508]: Uploading piece...
Aug 8 00:38:24.711891356 (14h58m24.301236343s) [noise=true , address=104.36.82.2:4745 , piece_id=JYMY6EJ2ZG6XEJUUUJOIFNNMI4QMINNS4Q77QYXPVLTSC7BLQEXA, seg_pos={0 0}, upload=15508]: Uploading piece...
Aug 8 00:38:24.712120744 (14h58m24.301465732s) [noise=true , address=104.36.82.54:4166 , piece_id=WTEO3MZJMTHGA5PTY5CCQ42SKTJI43M43EULZGGUXAQVIY7C46IA, seg_pos={0 0}, upload=15508]: Uploading piece...
Aug 8 00:38:24.712325797 (14h58m24.301670784s) [noise=true , address=104.36.82.3:4781 , piece_id=A4ZYEHF6HFDP6CX5MALR26F6RA7BE5CST5GDDULJBJR3ALUJAFVQ, seg_pos={0 0}, upload=15508]: Uploading piece...
Aug 8 00:38:24.712565686 (14h58m24.301910673s) [noise=true , address=104.36.82.16:4320 , piece_id=2OP3TMF3FNQE2WTV6HEY6HVNO6KR2FCMPXRZJYCUI3QJIYIRLBOQ, seg_pos={0 0}, upload=15508]: Uploading piece...
Aug 8 00:38:24.712743346 (14h58m24.302088333s) [noise=true , address=45.91.116.14:4192 , piece_id=XIT5DARFYND632CV46LHR3UBVDQ6GAZ2DH3HQFUCTUFMW2Y4VUOQ, seg_pos={0 0}, upload=15508]: Uploading piece...
Aug 8 00:38:24.712959169 (14h58m24.302304156s) [noise=true , address=45.91.116.5:4324 , piece_id=TP3YSP3WXJKXA3DKPDXBRHTAKQBC6EXMZCRG7HBCELRWY6TFVXAA, seg_pos={0 0}, upload=15508]: Uploading piece...
Aug 8 00:38:24.713184781 (14h58m24.302529768s) [noise=true , address=104.36.82.52:4097 , piece_id=DFJXCTAXSGUPKENZVYE6G4EAQF5SE5FKGRHI62QHCCVHYRAQDA7Q, seg_pos={0 0}, upload=15508]: Uploading piece...
Aug 8 00:38:25.841668344 (14h58m25.431013331s) [upload=15508]: Next returned nil segment
Aug 8 00:38:25.841685085 (14h58m25.431030072s) [upload=15508]: Waiting for error group managing segments...
Aug 8 00:38:26.112321232 (14h58m25.701666219s) [noise=true , address=64.49.13.26:4948 , piece_id=QSDEMGP65XIE3CLDSG6ASMKJCOBSSDZN2GWEVFJB3QQEGVCHTUIQ, seg_pos={0 0}, upload=15508]: Done uploading piece. err: <nil>
Aug 8 00:38:26.667567146 (14h58m26.256912133s) [noise=true , address=104.36.82.56:4256 , piece_id=3JG4YZGAK5JV6U3HTKEFSOVAKKLN5H66SBTSCUAL6KDS5YELSUWA, seg_pos={0 0}, upload=15508]: Done uploading piece. err: <nil>
Aug 8 00:38:26.669941781 (14h58m26.259286768s) [noise=true , address=104.36.82.3:4781 , piece_id=A4ZYEHF6HFDP6CX5MALR26F6RA7BE5CST5GDDULJBJR3ALUJAFVQ, seg_pos={0 0}, upload=15508]: Done uploading piece. err: <nil>
Aug 8 00:38:26.699819391 (14h58m26.289164378s) [noise=true , address=45.91.116.10:4113 , piece_id=QNB2CD3IINPCU6F3NCCPV7MNKY2GBV7ZBWUXY6UAYT7IIPJDGTCQ, seg_pos={0 0}, upload=15508]: Done uploading piece. err: <nil>
Aug 8 00:38:26.738125452 (14h58m26.327470439s) [noise=true , address=104.36.82.2:4745 , piece_id=JYMY6EJ2ZG6XEJUUUJOIFNNMI4QMINNS4Q77QYXPVLTSC7BLQEXA, seg_pos={0 0}, upload=15508]: Done uploading piece. err: <nil>
Aug 8 00:38:26.745536716 (14h58m26.334881703s) [noise=true , address=104.36.82.39:4734 , piece_id=2ZZDYY5W5NHZ7OFXZ3X566EWP5F6GV32KNCIGTAEMGWDZI3ZHNNQ, seg_pos={0 0}, upload=15508]: Done uploading piece. err: <nil>
Aug 8 00:38:26.768528297 (14h58m26.357873294s) [noise=true , address=45.91.116.11:4142 , piece_id=GWGM46SJ4ZLCAW2TTFUX4R3GXX6C4QAMBN62BUKXED6YVROHQ6PQ, seg_pos={0 0}, upload=15508]: Done uploading piece. err: <nil>
Aug 8 00:38:26.818308437 (14h58m26.407653414s) [noise=true , address=104.36.82.37:4656 , piece_id=QGCIU7D4AG66MLGC7XA435DKIIBGLSLSWHQDVIZ75NFJKTONV4UQ, seg_pos={0 0}, upload=15508]: Done uploading piece. err: <nil>
Aug 8 00:38:26.832514054 (14h58m26.421859041s) [noise=true , address=45.91.116.14:4192 , piece_id=XIT5DARFYND632CV46LHR3UBVDQ6GAZ2DH3HQFUCTUFMW2Y4VUOQ, seg_pos={0 0}, upload=15508]: Done uploading piece. err: <nil>
Aug 8 00:38:26.869135760 (14h58m26.458480748s) [noise=true , address=104.36.82.54:4166 , piece_id=WTEO3MZJMTHGA5PTY5CCQ42SKTJI43M43EULZGGUXAQVIY7C46IA, seg_pos={0 0}, upload=15508]: Done uploading piece. err: <nil>
Aug 8 00:38:27.131612271 (14h58m26.720957258s) [noise=true , address=45.91.116.5:4324 , piece_id=TP3YSP3WXJKXA3DKPDXBRHTAKQBC6EXMZCRG7HBCELRWY6TFVXAA, seg_pos={0 0}, upload=15508]: Done uploading piece. err: <nil>
Aug 8 00:38:27.152099560 (14h58m26.741444547s) [noise=true , address=104.36.82.58:4288 , piece_id=XBQQBKEJMENF6WMGIFEFC4CP2FA4NQ2LW22RUBS4DWTGBOWNHO7A, seg_pos={0 0}, upload=15508]: Done uploading piece. err: <nil>
Aug 8 00:38:27.501729825 (14h58m27.091074813s) [noise=true , address=104.36.82.16:4320 , piece_id=2OP3TMF3FNQE2WTV6HEY6HVNO6KR2FCMPXRZJYCUI3QJIYIRLBOQ, seg_pos={0 0}, upload=15508]: Done uploading piece. err: <nil>
Aug 8 00:38:27.778376595 (14h58m27.367721582s) [noise=true , address=104.36.82.9:4038 , piece_id=IUFHRP22JUSOJLB66HKMEIK6OJJJPR3NLRA4JHTLQZDMD5IEKQEA, seg_pos={0 0}, upload=15508]: Done uploading piece. err: <nil>
Aug 8 00:38:27.778417313 (14h58m27.3677623s) [seg_pos={0 0}, upload=15508]: Segment reached optimal threshold of 14 pieces.
Aug 8 00:38:27.778642765 (14h58m27.367987752s) [noise=true , address=104.36.82.47:4534 , piece_id=G56O4SR7DQ6YHATRCIMLOARHFQBILTCEUU2S35Y5NQKYIQX22X6A, seg_pos={0 0}, upload=15508]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VDrNGcCnaKEmG5NN4nex1JoZAgdCEfX5CdsKbxytfLfKzPbr): protocol: expected piece hash; context canceled; EOF
Aug 8 00:38:27.778734049 (14h58m27.368079036s) [noise=true , address=104.36.82.61:4386 , piece_id=3YTRS2CUXJ3X3AVS6SWI234NQD7G6XPSV7ONZWT22WS23P6GEUOA, seg_pos={0 0}, upload=15508]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19aPjJnya2Q3stEhiedDSth3NHV8phLRj5yY79hZv81UgDitMv): protocol: write tcp 188.166.225.170:42102->104.36.82.61:4386: use of closed network connection; write tcp 188.166.225.170:42102->104.36.82.61:4386: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42102->104.36.82.61:4386: use of closed network connection
Aug 8 00:38:27.778843298 (14h58m27.368188285s) [noise=true , address=104.36.82.45:4931 , piece_id=IBPB2JL6J2JDMHYYNI3UVNEZ5DUOOTA2SKQKRGV255F6NTCWUDWQ, seg_pos={0 0}, upload=15508]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pCgoSfxjtXPw5jpbz9RJaxRiGoYPdPFdoMPtD19F1ZWY8UwPu): protocol: expected piece hash; context canceled; EOF
Aug 8 00:38:27.778889437 (14h58m27.368234424s) [noise=true , address=104.36.82.52:4097 , piece_id=DFJXCTAXSGUPKENZVYE6G4EAQF5SE5FKGRHI62QHCCVHYRAQDA7Q, seg_pos={0 0}, upload=15508]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1sMhTktqf2H4aZSyFV3m9Z6eMVxL2U52bvUHVHUh2gUCVp32AM): protocol: expected piece hash; context canceled; EOF
Aug 8 00:38:27.778906359 (14h58m27.368251346s) [seg_pos={0 0}, upload=15508]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Aug 8 00:38:27.778981493 (14h58m27.36832647s) [upload=15508]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Aug 8 00:38:27.778985781 (14h58m27.368330768s) [upload=15508]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Aug 8 00:38:28.078873078 (14h58m27.668218065s) [upload=15508]: Done uploading segments...
Aug 8 00:38:28.078891813 (14h58m27.6682368s) [upload=15508]: Upload finished. err: <nil>
Duration: 6.099802782s
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)antiblip:
5 out of 60 is failed (threshold=3) [..XXXXX.....................................................] duration:
1.861258414s
FAIL (2025-08-08 00:39:33 UTC)
orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== m24.001855996s) [upload=14742]: Done starting upload Aug 7 23:53:24.412569290 (14h13m24.001914277s) [upload=14742]: Uploading segments... Aug 7 23:53:24.412638803 (14h13m24.001983831s) [upload=14742]: Got next segment. Inline: true Aug 7 23:53:24.412644354 (14h13m24.001989341s) [upload=14742]: Waiting for error group managing segments... Aug 7 23:53:24.412680903 (14h13m24.002025891s) [upload=14742]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 23:53:24.412685442 (14h13m24.002030429s) [upload=14742]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 23:53:24.412688137 (14h13m24.002033114s) [upload=14742]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 23:53:53.772057432 (14h13m53.361402419s) [upload=14742]: Done uploading segments... Aug 7 23:53:53.772119881 (14h13m53.361464908s) [upload=14742]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:54:30.496202349 (14h14m30.085547336s) [upload=14761]: Starting upload Aug 7 23:54:30.496270259 (14h14m30.085615246s) [upload=14761]: Done starting upload Aug 7 23:54:30.496341565 (14h14m30.085686552s) [upload=14761]: Uploading segments... Aug 7 23:54:30.496378957 (14h14m30.085723944s) [upload=14761]: Got next segment. Inline: true Aug 7 23:54:30.496383836 (14h14m30.085728823s) [upload=14761]: Waiting for error group managing segments... Aug 7 23:54:30.496441346 (14h14m30.085786333s) [upload=14761]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 23:54:30.496446426 (14h14m30.085791413s) [upload=14761]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 23:54:30.496449551 (14h14m30.085794539s) [upload=14761]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 23:54:59.847423574 (14h14m59.436768561s) [upload=14761]: Done uploading segments... Aug 7 23:54:59.847456066 (14h14m59.436801043s) [upload=14761]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:55:40.612588843 (14h15m40.20193383s) [upload=14781]: Starting upload Aug 7 23:55:40.612648929 (14h15m40.201993906s) [upload=14781]: Done starting upload Aug 7 23:55:40.612717029 (14h15m40.202062016s) [upload=14781]: Uploading segments... Aug 7 23:55:40.612750823 (14h15m40.20209581s) [upload=14781]: Got next segment. Inline: true Aug 7 23:55:40.612754510 (14h15m40.202099487s) [upload=14781]: Waiting for error group managing segments... Aug 7 23:55:40.612786883 (14h15m40.20213186s) [upload=14781]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 23:55:40.612790891 (14h15m40.202135868s) [upload=14781]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 23:55:40.612793075 (14h15m40.202138062s) [upload=14781]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 23:55:40.883469345 (14h15m40.472814333s) [upload=14781]: Done uploading segments... Aug 7 23:55:40.883486387 (14h15m40.472831375s) [upload=14781]: Upload finished. err: <nil> Aug 7 23:56:17.423712239 (14h16m17.013057226s) [upload=14789]: Starting upload Aug 7 23:56:17.423767895 (14h16m17.013112882s) [upload=14789]: Done starting upload Aug 7 23:56:17.423814835 (14h16m17.013159822s) [upload=14789]: Uploading segments... Aug 7 23:56:17.423847678 (14h16m17.013192665s) [upload=14789]: Got next segment. Inline: true Aug 7 23:56:17.423851335 (14h16m17.013196322s) [upload=14789]: Waiting for error group managing segments... Aug 7 23:56:17.423883377 (14h16m17.013228364s) [upload=14789]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 23:56:17.423887054 (14h16m17.013232041s) [upload=14789]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 23:56:17.423889258 (14h16m17.013234245s) [upload=14789]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 23:56:17.689327115 (14h16m17.278672102s) [upload=14789]: Done uploading segments... Aug 7 23:56:17.689345711 (14h16m17.278690688s) [upload=14789]: Upload finished. err: <nil> Aug 7 23:56:58.288741281 (14h16m57.878086268s) [upload=14801]: Starting upload Aug 7 23:56:58.288797098 (14h16m57.878142085s) [upload=14801]: Done starting upload Aug 7 23:56:58.288851010 (14h16m57.878195987s) [upload=14801]: Uploading segments... Aug 7 23:56:58.288882281 (14h16m57.878227268s) [upload=14801]: Got next segment. Inline: true Aug 7 23:56:58.288887911 (14h16m57.878232898s) [upload=14801]: Waiting for error group managing segments... Aug 7 23:56:58.288919792 (14h16m57.878264769s) [upload=14801]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 23:56:58.288923970 (14h16m57.878268957s) [upload=14801]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 23:56:58.288926485 (14h16m57.878271462s) [upload=14801]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 23:56:58.556529816 (14h16m58.145874803s) [upload=14801]: Done uploading segments... Aug 7 23:56:58.556544975 (14h16m58.145889962s) [upload=14801]: Upload finished. err: <nil> Aug 7 23:57:18.362794018 (14h17m17.952138995s) [upload=14806]: Starting upload Aug 7 23:57:18.362883679 (14h17m17.952228666s) [upload=14806]: Done starting upload Aug 7 23:57:18.362928104 (14h17m17.952273092s) [upload=14806]: Uploading segments... Aug 7 23:57:18.362957320 (14h17m17.952302307s) [upload=14806]: Got next segment. Inline: true Aug 7 23:57:18.362961077 (14h17m17.952306064s) [upload=14806]: Waiting for error group managing segments... Aug 7 23:57:18.363018407 (14h17m17.952363394s) [upload=14806]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 23:57:18.363022404 (14h17m17.952367392s) [upload=14806]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 23:57:18.363024739 (14h17m17.952369716s) [upload=14806]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 23:57:47.715297444 (14h17m47.304642431s) [upload=14806]: Done uploading segments... Aug 7 23:57:47.715410370 (14h17m47.304755357s) [upload=14806]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:57:53.375540080 (14h17m52.964885068s) [upload=14813]: Starting upload Aug 7 23:57:53.375600245 (14h17m52.964945232s) [upload=14813]: Done starting upload Aug 7 23:57:53.375654389 (14h17m52.964999376s) [upload=14813]: Uploading segments... Aug 7 23:57:53.375686622 (14h17m52.965031609s) [upload=14813]: Got next segment. Inline: true Aug 7 23:57:53.375690759 (14h17m52.965035746s) [upload=14813]: Waiting for error group managing segments... Aug 7 23:57:53.375725876 (14h17m52.965070863s) [upload=14813]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 23:57:53.375730134 (14h17m52.965075121s) [upload=14813]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 23:57:53.375734372 (14h17m52.965079359s) [upload=14813]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 23:58:22.726838464 (14h18m22.316183451s) [upload=14813]: Done uploading segments... Aug 7 23:58:22.726860075 (14h18m22.316205062s) [upload=14813]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:58:23.621870384 (14h18m23.211215371s) [upload=14822]: Starting upload Aug 7 23:58:23.621952000 (14h18m23.211296987s) [upload=14822]: Done starting upload Aug 7 23:58:23.622003359 (14h18m23.211348346s) [upload=14822]: Uploading segments... Aug 7 23:58:23.622043436 (14h18m23.211388413s) [upload=14822]: Got next segment. Inline: true Aug 7 23:58:23.622046772 (14h18m23.211391759s) [upload=14822]: Waiting for error group managing segments... Aug 7 23:58:23.622078573 (14h18m23.21142356s) [upload=14822]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 23:58:23.622082230 (14h18m23.211427217s) [upload=14822]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 23:58:23.622084534 (14h18m23.211429511s) [upload=14822]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 23:58:52.983677879 (14h18m52.573022876s) [upload=14822]: Done uploading segments... Aug 7 23:58:52.983713708 (14h18m52.573058685s) [upload=14822]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:59:11.446858470 (14h19m11.036203457s) [upload=14835]: Starting upload Aug 7 23:59:11.446906493 (14h19m11.03625148s) [upload=14835]: Done starting upload Aug 7 23:59:11.446947491 (14h19m11.036292479s) [upload=14835]: Uploading segments... Aug 7 23:59:11.446975475 (14h19m11.036320462s) [upload=14835]: Got next segment. Inline: true Aug 7 23:59:11.446978751 (14h19m11.036323738s) [upload=14835]: Waiting for error group managing segments... Aug 7 23:59:11.447008889 (14h19m11.036353876s) [upload=14835]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 23:59:11.447013638 (14h19m11.036358625s) [upload=14835]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 23:59:11.447016212 (14h19m11.0363612s) [upload=14835]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 23:59:40.799397635 (14h19m40.388742622s) [upload=14835]: Done uploading segments... Aug 7 23:59:40.799413266 (14h19m40.388758243s) [upload=14835]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:59:59.523080643 (14h19m59.11242563s) [upload=14847]: Starting upload Aug 7 23:59:59.523152089 (14h19m59.112497076s) [upload=14847]: Done starting upload Aug 7 23:59:59.523213727 (14h19m59.112558714s) [upload=14847]: Uploading segments... Aug 7 23:59:59.523251349 (14h19m59.112596336s) [upload=14847]: Got next segment. Inline: true Aug 7 23:59:59.523255287 (14h19m59.112600274s) [upload=14847]: Waiting for error group managing segments... Aug 7 23:59:59.523300013 (14h19m59.112645s) [upload=14847]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 23:59:59.523304251 (14h19m59.112649238s) [upload=14847]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 23:59:59.523307146 (14h19m59.112652133s) [upload=14847]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 8 00:00:28.877190143 (14h20m28.46653516s) [upload=14847]: Done uploading segments... Aug 8 00:00:28.877218477 (14h20m28.466563505s) [upload=14847]: Upload finished. err: metaclient: context deadline exceeded
Duration: 1.861295837s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
6 out of 60 is failed (threshold=3) [..XXXX....XX................................................] duration:
6.064894126s
FAIL (2025-08-08 00:39:17 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
3.700898256s) [upload=14855]: Done starting upload
Aug 8 00:00:34.111602513 (14h20m33.7009475s) [upload=14855]: Uploading segments...
Aug 8 00:00:34.111774201 (14h20m33.701119218s) [upload=14855]: Got next segment. Inline: false
Aug 8 00:00:34.111816783 (14h20m33.70116178s) [upload=14855]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Aug 8 00:00:34.111822464 (14h20m33.701167461s) [upload=14855]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Aug 8 00:01:03.465979471 (14h21m3.055324468s) [upload=14855]: Next segment err: context deadline exceeded
Aug 8 00:01:03.466148254 (14h21m3.055493251s) [upload=14855]: Done uploading segments...
Aug 8 00:01:03.466174945 (14h21m3.055519932s) [upload=14855]: Upload finished. err: context deadline exceeded
Aug 8 00:01:19.453976971 (14h21m19.043321958s) [upload=14872]: Starting upload
Aug 8 00:01:19.454042106 (14h21m19.043387123s) [upload=14872]: Done starting upload
Aug 8 00:01:19.454156104 (14h21m19.043501091s) [upload=14872]: Uploading segments...
Aug 8 00:01:19.454244503 (14h21m19.04358949s) [upload=14872]: Got next segment. Inline: false
Aug 8 00:01:19.454272567 (14h21m19.043617554s) [upload=14872]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Aug 8 00:01:19.454276825 (14h21m19.043621812s) [upload=14872]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Aug 8 00:01:39.582519114 (14h21m39.171864101s) [seg_pos={0 0}, upload=14872]: Begin upload segment...
Aug 8 00:01:39.582565882 (14h21m39.17191087s) [seg_pos={0 0}, upload=14872]: Done begin upload segment.
Aug 8 00:01:39.582581332 (14h21m39.17192632s) [noise=true , address=45.91.116.5:4299 , piece_id=HRKD77XA4SHZKSEW2WA3AROWPLHOWEXG5GEOWJCG7NTFOYYZ4DFA, seg_pos={0 0}, upload=14872]: Uploading piece...
Aug 8 00:01:39.582590870 (14h21m39.171935857s) [noise=true , address=104.36.82.2:4731 , piece_id=2BTN7TE7VVSPN5BUCDVKHWZPPATMNIJLXNDXUO46Z7CRB3QBCUEQ, seg_pos={0 0}, upload=14872]: Uploading piece...
Aug 8 00:01:39.582951871 (14h21m39.172296858s) [noise=true , address=45.91.116.12:4225 , piece_id=MUZ4UWUJSWGYKIO3BSTEOJYBBXSK4R2PURNDXHOYLTZ7VZ466IGA, seg_pos={0 0}, upload=14872]: Uploading piece...
Aug 8 00:01:39.583059888 (14h21m39.172404905s) [noise=true , address=104.36.82.13:4185 , piece_id=O26DWLA45AEOUZOW2CZTLJNPVRWB5KVDQW6FH2ANTKLUMYNWRVKA, seg_pos={0 0}, upload=14872]: Uploading piece...
Aug 8 00:01:39.583214905 (14h21m39.172559892s) [noise=true , address=45.91.116.8:4047 , piece_id=4LUGFRA2QPUK5LPKZGNTURXXSGY64AZU4Q3BHBAM3SPBZ7M2FFKQ, seg_pos={0 0}, upload=14872]: Uploading piece...
Aug 8 00:01:39.583427882 (14h21m39.172772869s) [noise=true , address=77.90.31.21:4877 , piece_id=MZ4WJDOFWH56JIAGXJZ7FOAPMNCHVEBUUM2G5IY2KKPRHPJH5MDA, seg_pos={0 0}, upload=14872]: Uploading piece...
Aug 8 00:01:39.583689032 (14h21m39.173034019s) [noise=true , address=77.90.31.11:4555 , piece_id=OCZBWUJHEVLMUD6BNADJ7BOSHHZPESQJOSKDZ3T2MUSQRFTNI7KA, seg_pos={0 0}, upload=14872]: Uploading piece...
Aug 8 00:01:39.583925344 (14h21m39.173270331s) [noise=true , address=45.91.116.9:4263 , piece_id=F4QDS3EOSLFFCEDU6N56AMMQR4GHPWZ6TIAMO5SGVIMVMFA47EQQ, seg_pos={0 0}, upload=14872]: Uploading piece...
Aug 8 00:01:39.584272317 (14h21m39.173617294s) [noise=true , address=104.36.82.8:4970 , piece_id=N52DS4AB7Z6VSFYWHN5MXPUS3AP7UKN2K7U5D5JZWETM7ZLROO7Q, seg_pos={0 0}, upload=14872]: Uploading piece...
Aug 8 00:01:39.584579135 (14h21m39.173924122s) [noise=true , address=77.90.31.18:4807 , piece_id=V7EZJW5PYJCWWNNIRAJCFTT42DQJY2C4RDHB5645TKOSXJJXE2WA, seg_pos={0 0}, upload=14872]: Uploading piece...
Aug 8 00:01:39.584852117 (14h21m39.174197104s) [noise=true , address=104.36.82.11:4115 , piece_id=S34RF36SERFPPSHBPYJ5V3FCR5ZE7RFIVEXBOTFRHPEEIUCHO2LQ, seg_pos={0 0}, upload=14872]: Uploading piece...
Aug 8 00:01:39.585105202 (14h21m39.174450179s) [noise=true , address=104.36.82.9:4005 , piece_id=ZYZ6VK5MCW7QE6N2GOYW2WMWDEWGONNK2Y7QBE7FPVPLY2OZGIMQ, seg_pos={0 0}, upload=14872]: Uploading piece...
Aug 8 00:01:39.585298762 (14h21m39.174643739s) [noise=true , address=45.91.116.4:4940 , piece_id=5S5JLBIGSBIVTPVJT5MIRAE42PRCIFWDNWLY7NQXSWPMHT5WSTRA, seg_pos={0 0}, upload=14872]: Uploading piece...
Aug 8 00:01:39.585506369 (14h21m39.174851346s) [noise=true , address=104.36.82.10:4079 , piece_id=262SP4DREL7FYDG6YOBFWDPWKYHZIQGDJKSX7XAIUJUTQWUIEEZQ, seg_pos={0 0}, upload=14872]: Uploading piece...
Aug 8 00:01:39.585763882 (14h21m39.175108869s) [noise=true , address=45.91.116.14:4190 , piece_id=FSO3UN2D24KPDVSWWPI3ROBXN7RN6D2HAZKOXEQ2JOW2M6RQ5MDA, seg_pos={0 0}, upload=14872]: Uploading piece...
Aug 8 00:01:39.586021575 (14h21m39.175366562s) [noise=true , address=104.36.82.5:4839 , piece_id=EHHLJG3INARPORGO6YVXU7QJAK23ADRVWBGNT5IKL4N6Y4ASM3XA, seg_pos={0 0}, upload=14872]: Uploading piece...
Aug 8 00:01:41.370103161 (14h21m40.959448138s) [upload=14872]: Next returned nil segment
Aug 8 00:01:41.370130243 (14h21m40.95947523s) [upload=14872]: Waiting for error group managing segments...
Aug 8 00:01:41.673384945 (14h21m41.262729922s) [noise=true , address=104.36.82.2:4731 , piece_id=2BTN7TE7VVSPN5BUCDVKHWZPPATMNIJLXNDXUO46Z7CRB3QBCUEQ, seg_pos={0 0}, upload=14872]: Done uploading piece. err: <nil>
Aug 8 00:01:41.739890780 (14h21m41.329235767s) [noise=true , address=104.36.82.9:4005 , piece_id=ZYZ6VK5MCW7QE6N2GOYW2WMWDEWGONNK2Y7QBE7FPVPLY2OZGIMQ, seg_pos={0 0}, upload=14872]: Done uploading piece. err: <nil>
Aug 8 00:01:41.743845828 (14h21m41.333190815s) [noise=true , address=104.36.82.8:4970 , piece_id=N52DS4AB7Z6VSFYWHN5MXPUS3AP7UKN2K7U5D5JZWETM7ZLROO7Q, seg_pos={0 0}, upload=14872]: Done uploading piece. err: <nil>
Aug 8 00:01:41.786086095 (14h21m41.375431072s) [noise=true , address=45.91.116.8:4047 , piece_id=4LUGFRA2QPUK5LPKZGNTURXXSGY64AZU4Q3BHBAM3SPBZ7M2FFKQ, seg_pos={0 0}, upload=14872]: Done uploading piece. err: <nil>
Aug 8 00:01:41.791525722 (14h21m41.380870699s) [noise=true , address=104.36.82.5:4839 , piece_id=EHHLJG3INARPORGO6YVXU7QJAK23ADRVWBGNT5IKL4N6Y4ASM3XA, seg_pos={0 0}, upload=14872]: Done uploading piece. err: <nil>
Aug 8 00:01:41.959657137 (14h21m41.549002124s) [noise=true , address=104.36.82.13:4185 , piece_id=O26DWLA45AEOUZOW2CZTLJNPVRWB5KVDQW6FH2ANTKLUMYNWRVKA, seg_pos={0 0}, upload=14872]: Done uploading piece. err: <nil>
Aug 8 00:01:42.000743686 (14h21m41.590088693s) [noise=true , address=45.91.116.12:4225 , piece_id=MUZ4UWUJSWGYKIO3BSTEOJYBBXSK4R2PURNDXHOYLTZ7VZ466IGA, seg_pos={0 0}, upload=14872]: Done uploading piece. err: <nil>
Aug 8 00:01:42.160166408 (14h21m41.749511395s) [noise=true , address=45.91.116.4:4940 , piece_id=5S5JLBIGSBIVTPVJT5MIRAE42PRCIFWDNWLY7NQXSWPMHT5WSTRA, seg_pos={0 0}, upload=14872]: Done uploading piece. err: <nil>
Aug 8 00:01:42.175399653 (14h21m41.764744641s) [noise=true , address=45.91.116.14:4190 , piece_id=FSO3UN2D24KPDVSWWPI3ROBXN7RN6D2HAZKOXEQ2JOW2M6RQ5MDA, seg_pos={0 0}, upload=14872]: Done uploading piece. err: <nil>
Aug 8 00:01:42.175880162 (14h21m41.765225139s) [noise=true , address=45.91.116.5:4299 , piece_id=HRKD77XA4SHZKSEW2WA3AROWPLHOWEXG5GEOWJCG7NTFOYYZ4DFA, seg_pos={0 0}, upload=14872]: Done uploading piece. err: <nil>
Aug 8 00:01:42.278394081 (14h21m41.867739068s) [noise=true , address=77.90.31.21:4877 , piece_id=MZ4WJDOFWH56JIAGXJZ7FOAPMNCHVEBUUM2G5IY2KKPRHPJH5MDA, seg_pos={0 0}, upload=14872]: Done uploading piece. err: <nil>
Aug 8 00:01:42.293663936 (14h21m41.883008933s) [noise=true , address=77.90.31.18:4807 , piece_id=V7EZJW5PYJCWWNNIRAJCFTT42DQJY2C4RDHB5645TKOSXJJXE2WA, seg_pos={0 0}, upload=14872]: Done uploading piece. err: <nil>
Aug 8 00:01:42.309113927 (14h21m41.898458904s) [noise=true , address=45.91.116.9:4263 , piece_id=F4QDS3EOSLFFCEDU6N56AMMQR4GHPWZ6TIAMO5SGVIMVMFA47EQQ, seg_pos={0 0}, upload=14872]: Done uploading piece. err: <nil>
Aug 8 00:01:42.309137942 (14h21m41.898482919s) [seg_pos={0 0}, upload=14872]: Segment reached optimal threshold of 13 pieces.
Aug 8 00:01:42.309315973 (14h21m41.89866099s) [noise=true , address=77.90.31.11:4555 , piece_id=OCZBWUJHEVLMUD6BNADJ7BOSHHZPESQJOSKDZ3T2MUSQRFTNI7KA, seg_pos={0 0}, upload=14872]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ): protocol: expected piece hash; context canceled; EOF
Aug 8 00:01:42.309402328 (14h21m41.898747305s) [noise=true , address=104.36.82.10:4079 , piece_id=262SP4DREL7FYDG6YOBFWDPWKYHZIQGDJKSX7XAIUJUTQWUIEEZQ, seg_pos={0 0}, upload=14872]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud): protocol: expected piece hash; context canceled; EOF
Aug 8 00:01:42.309425222 (14h21m41.898770229s) [noise=true , address=104.36.82.11:4115 , piece_id=S34RF36SERFPPSHBPYJ5V3FCR5ZE7RFIVEXBOTFRHPEEIUCHO2LQ, seg_pos={0 0}, upload=14872]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR): protocol: write tcp 188.166.225.170:42778->104.36.82.11:4115: use of closed network connection; write tcp 188.166.225.170:42778->104.36.82.11:4115: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42778->104.36.82.11:4115: use of closed network connection
Aug 8 00:01:42.309450360 (14h21m41.898795347s) [seg_pos={0 0}, upload=14872]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Aug 8 00:01:42.309540663 (14h21m41.89888565s) [upload=14872]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Aug 8 00:01:42.309548688 (14h21m41.898893735s) [upload=14872]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Aug 8 00:01:57.644545293 (14h21m57.2338903s) [upload=14872]: Done uploading segments...
Aug 8 00:01:57.644581281 (14h21m57.233926288s) [upload=14872]: Upload finished. err: metaclient: context deadline exceeded
Duration: 6.064920145s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)antiblip:
6 out of 60 is failed (threshold=3) [XXX.XX....X.................................................] duration:
5.848173333s
FAIL (2025-08-08 00:39:15 UTC)
orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded
Duration: 5.848201778s
core
uplink
us1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [....X.......................................................] duration:
6.043013549s
FAIL (2025-08-08 00:39:13 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.043069458s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline)antiblip:
3 out of 60 is failed (threshold=3) [.X.X..X.....................................................] duration:
2.003252224s
FAIL (2025-08-08 00:39:10 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.003273093s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
4 out of 60 is failed (threshold=3) [.......X.XX...X.............................................] duration:
5.856916647s
FAIL (2025-08-08 00:39:05 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: encryption: context deadline exceeded
======LOGS======
ZYSDFBHFTDOKPS73M57QA, seg_pos={0 0}, upload=14870]: Uploading piece...
Aug 8 00:01:16.246137011 (14h21m15.835481998s) [noise=true , address=104.36.82.9:4024 , piece_id=FM7MVIPG7C264O6Q64RUFOKZ2E5NMTIPWSIVMH5SWZUSCMFI7UGQ, seg_pos={0 0}, upload=14870]: Uploading piece...
Aug 8 00:01:16.246560380 (14h21m15.835905367s) [noise=true , address=64.49.13.169:4705 , piece_id=3MVXPRNKBBUSYAH2E7PCIWXDCUFQQ547IPOYIDLQ3T7QDYPBJA7A, seg_pos={0 0}, upload=14870]: Uploading piece...
Aug 8 00:01:16.247408974 (14h21m15.836753961s) [noise=true , address=45.91.116.5:4302 , piece_id=6RMPHN4LYSZVM6XKAUGSU7GY3TIKBGLKK4KNFMZQ2FMVEEVWV2LA, seg_pos={0 0}, upload=14870]: Uploading piece...
Aug 8 00:01:16.247678410 (14h21m15.837023397s) [noise=true , address=79.127.219.51:4181 , piece_id=4LYG6AL2BWD3UYQQAUUGZNJ7HUFXSHZ2TULWJOYSWTJLNVGHVKLQ, seg_pos={0 0}, upload=14870]: Uploading piece...
Aug 8 00:01:16.247919882 (14h21m15.837264869s) [noise=true , address=104.36.82.22:4512 , piece_id=7VOWVUPU3QVXIKML65NHX3NL7AC26QSDGHSSCRLUXIVZRNUPPKRA, seg_pos={0 0}, upload=14870]: Uploading piece...
Aug 8 00:01:16.247943888 (14h21m15.837288875s) [noise=true , address=104.36.82.20:4466 , piece_id=PLG7Z2WTYKUNB347NPT4ESIDK2NHCP66NPZJOXYL4A6L5OMFQIOQ, seg_pos={0 0}, upload=14870]: Uploading piece...
Aug 8 00:01:16.248293266 (14h21m15.837638253s) [noise=true , address=104.36.82.11:4143 , piece_id=E4AW52SZYZVRWPU4L56INK3IRNG7UNO5RRIRJANLMEPZQVDUHCEQ, seg_pos={0 0}, upload=14870]: Uploading piece...
Aug 8 00:01:16.248420540 (14h21m15.837765527s) [noise=true , address=64.49.13.139:4471 , piece_id=FPVXYIRNZXODU7N5RPCDOVVUECXWRTXCLINEXMOJTKNHF6YN46YQ, seg_pos={0 0}, upload=14870]: Uploading piece...
Aug 8 00:01:16.248557311 (14h21m15.837902288s) [noise=true , address=104.36.82.37:4659 , piece_id=627LA2AVL53NXFMDWSLGBO5GUKDAIGGSOPBFRPYIPYC5EAQC6P7Q, seg_pos={0 0}, upload=14870]: Uploading piece...
Aug 8 00:01:16.248758506 (14h21m15.838103493s) [noise=true , address=104.36.82.58:4275 , piece_id=VBV72SK36XIHMWSZTJJE4MZPLD5VZ6Z6UPBHK4A2ZI2C3O3RCVXA, seg_pos={0 0}, upload=14870]: Uploading piece...
Aug 8 00:01:16.248897792 (14h21m15.838242779s) [noise=true , address=104.36.82.55:4209 , piece_id=PJ4RK52LARBDG6WEVKXPZPX3RKJVIRSDCZY3B6CVCIUZO6BTWLBQ, seg_pos={0 0}, upload=14870]: Uploading piece...
Aug 8 00:01:16.248989629 (14h21m15.838334616s) [noise=true , address=104.36.82.43:4855 , piece_id=MJYC5ZGE6P72UYNYDRMH3BAZ6KIY57SA3EKJR54W6HPZNO3AE4IA, seg_pos={0 0}, upload=14870]: Uploading piece...
Aug 8 00:01:16.249186575 (14h21m15.838531552s) [noise=true , address=104.36.82.46:4487 , piece_id=6BZYNRXKBHFL3RQTGN3ZN4RV2O4USZ24GGCVSKFZWNS6N32L3NGQ, seg_pos={0 0}, upload=14870]: Uploading piece...
Aug 8 00:01:16.249273772 (14h21m15.838618759s) [noise=true , address=104.36.82.7:4920 , piece_id=BN5NCMLEGDVRIHO6YOWM5FNPCGFYC3BDKLOU7DIP6K6DYZXJ5A5A, seg_pos={0 0}, upload=14870]: Uploading piece...
Aug 8 00:01:16.249459598 (14h21m15.838804585s) [noise=true , address=77.90.31.14:4672 , piece_id=3ZYE5SLRNLY2ALXRR4QGCZT4SQ37U7FIAUOCVT63HQSBKPKV4UGQ, seg_pos={0 0}, upload=14870]: Uploading piece...
Aug 8 00:01:16.249717240 (14h21m15.839062227s) [noise=true , address=104.36.82.5:4844 , piece_id=REDGG4W6TJYEPEEPR65O5AVQDU2QMBNRA6HQQRUJA6JP7IS5T7YQ, seg_pos={0 0}, upload=14870]: Uploading piece...
Aug 8 00:01:17.413257645 (14h21m17.002602632s) [upload=14870]: Next returned nil segment
Aug 8 00:01:17.413273324 (14h21m17.002618311s) [upload=14870]: Waiting for error group managing segments...
Aug 8 00:01:17.710897294 (14h21m17.300242281s) [noise=true , address=64.49.13.169:4705 , piece_id=3MVXPRNKBBUSYAH2E7PCIWXDCUFQQ547IPOYIDLQ3T7QDYPBJA7A, seg_pos={0 0}, upload=14870]: Done uploading piece. err: <nil>
Aug 8 00:01:17.715058896 (14h21m17.304403883s) [noise=true , address=64.49.13.139:4471 , piece_id=FPVXYIRNZXODU7N5RPCDOVVUECXWRTXCLINEXMOJTKNHF6YN46YQ, seg_pos={0 0}, upload=14870]: Done uploading piece. err: <nil>
Aug 8 00:01:18.126218161 (14h21m17.715563148s) [noise=true , address=79.127.201.215:4356 , piece_id=VWNEIHNI4KPPF4QLW36INIWTUI2CD6MZYSDFBHFTDOKPS73M57QA, seg_pos={0 0}, upload=14870]: Done uploading piece. err: <nil>
Aug 8 00:01:18.139272367 (14h21m17.728617354s) [noise=true , address=79.127.219.51:4181 , piece_id=4LYG6AL2BWD3UYQQAUUGZNJ7HUFXSHZ2TULWJOYSWTJLNVGHVKLQ, seg_pos={0 0}, upload=14870]: Done uploading piece. err: <nil>
Aug 8 00:01:18.221269031 (14h21m17.810614018s) [noise=true , address=104.36.82.43:4855 , piece_id=MJYC5ZGE6P72UYNYDRMH3BAZ6KIY57SA3EKJR54W6HPZNO3AE4IA, seg_pos={0 0}, upload=14870]: Done uploading piece. err: <nil>
Aug 8 00:01:18.257350144 (14h21m17.846695121s) [noise=true , address=104.36.82.37:4659 , piece_id=627LA2AVL53NXFMDWSLGBO5GUKDAIGGSOPBFRPYIPYC5EAQC6P7Q, seg_pos={0 0}, upload=14870]: Done uploading piece. err: <nil>
Aug 8 00:01:18.258435089 (14h21m17.847780076s) [noise=true , address=104.36.82.44:4876 , piece_id=BA2G42OXQD3NLONUVD475S2BNC7T37T5PHPE3K3E5HJAJP4W5MKQ, seg_pos={0 0}, upload=14870]: Done uploading piece. err: <nil>
Aug 8 00:01:18.287010923 (14h21m17.87635591s) [noise=true , address=104.36.82.7:4920 , piece_id=BN5NCMLEGDVRIHO6YOWM5FNPCGFYC3BDKLOU7DIP6K6DYZXJ5A5A, seg_pos={0 0}, upload=14870]: Done uploading piece. err: <nil>
Aug 8 00:01:18.348793001 (14h21m17.938137988s) [noise=true , address=104.36.82.11:4143 , piece_id=E4AW52SZYZVRWPU4L56INK3IRNG7UNO5RRIRJANLMEPZQVDUHCEQ, seg_pos={0 0}, upload=14870]: Done uploading piece. err: <nil>
Aug 8 00:01:18.378129459 (14h21m17.967474436s) [noise=true , address=104.36.82.58:4275 , piece_id=VBV72SK36XIHMWSZTJJE4MZPLD5VZ6Z6UPBHK4A2ZI2C3O3RCVXA, seg_pos={0 0}, upload=14870]: Done uploading piece. err: <nil>
Aug 8 00:01:18.386414901 (14h21m17.975759888s) [noise=true , address=104.36.82.46:4487 , piece_id=6BZYNRXKBHFL3RQTGN3ZN4RV2O4USZ24GGCVSKFZWNS6N32L3NGQ, seg_pos={0 0}, upload=14870]: Done uploading piece. err: <nil>
Aug 8 00:01:18.516150823 (14h21m18.10549581s) [noise=true , address=104.36.82.9:4024 , piece_id=FM7MVIPG7C264O6Q64RUFOKZ2E5NMTIPWSIVMH5SWZUSCMFI7UGQ, seg_pos={0 0}, upload=14870]: Done uploading piece. err: <nil>
Aug 8 00:01:18.626647204 (14h21m18.215992191s) [noise=true , address=77.90.31.14:4672 , piece_id=3ZYE5SLRNLY2ALXRR4QGCZT4SQ37U7FIAUOCVT63HQSBKPKV4UGQ, seg_pos={0 0}, upload=14870]: Done uploading piece. err: <nil>
Aug 8 00:01:19.042108811 (14h21m18.631453798s) [noise=true , address=104.36.82.22:4512 , piece_id=7VOWVUPU3QVXIKML65NHX3NL7AC26QSDGHSSCRLUXIVZRNUPPKRA, seg_pos={0 0}, upload=14870]: Done uploading piece. err: <nil>
Aug 8 00:01:19.042136654 (14h21m18.631481641s) [seg_pos={0 0}, upload=14870]: Segment reached optimal threshold of 14 pieces.
Aug 8 00:01:19.042375541 (14h21m18.631720518s) [noise=true , address=45.91.116.5:4302 , piece_id=6RMPHN4LYSZVM6XKAUGSU7GY3TIKBGLKK4KNFMZQ2FMVEEVWV2LA, seg_pos={0 0}, upload=14870]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pJWhSLza8WaMuvvDwH4nae57UxQmfCdDvAkQ3v25MgPYHoZXx): protocol: expected piece hash; context canceled; EOF
Aug 8 00:01:19.042485912 (14h21m18.6318309s) [noise=true , address=104.36.82.55:4209 , piece_id=PJ4RK52LARBDG6WEVKXPZPX3RKJVIRSDCZY3B6CVCIUZO6BTWLBQ, seg_pos={0 0}, upload=14870]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12EmE2BgEZTG1LTsZ5p9ghqDSfg5ZQATt31Tqg5jZ5KQE2UjDnR): protocol: rpc: dial tcp 104.36.82.55:4209: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 8 00:01:19.042532361 (14h21m18.631877348s) [noise=true , address=104.36.82.20:4466 , piece_id=PLG7Z2WTYKUNB347NPT4ESIDK2NHCP66NPZJOXYL4A6L5OMFQIOQ, seg_pos={0 0}, upload=14870]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SqNLPRWY7fJRS6F9h4B4qVFv6U5jyWkCy3Um7jfjEdCShEjsD): protocol: write tcp 188.166.225.170:37056->104.36.82.20:4466: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 8 00:01:19.042616462 (14h21m18.631961449s) [noise=true , address=104.36.82.5:4844 , piece_id=REDGG4W6TJYEPEEPR65O5AVQDU2QMBNRA6HQQRUJA6JP7IS5T7YQ, seg_pos={0 0}, upload=14870]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12iovpNdbidkiv9mEHaAPjBubGmoqr4sUJWm8Cb64LzS6wuwLWa): protocol: rpc: dial tcp 104.36.82.5:4844: operation was canceled; conn closed: op; piecestore: piecestore close: context canceled
Aug 8 00:01:19.042628876 (14h21m18.631973863s) [seg_pos={0 0}, upload=14870]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Aug 8 00:01:19.042699029 (14h21m18.632044017s) [upload=14870]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Aug 8 00:01:19.042703128 (14h21m18.632048115s) [upload=14870]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Aug 8 00:01:19.344335316 (14h21m18.933680293s) [upload=14870]: Done uploading segments...
Aug 8 00:01:19.344353089 (14h21m18.933698077s) [upload=14870]: Upload finished. err: <nil>
Aug 8 00:02:30.778225454 (14h22m30.367570441s) [upload=14892]: Starting upload
Aug 8 00:02:30.778283114 (14h22m30.367628151s) [upload=14892]: Done starting upload
Aug 8 00:02:30.778411841 (14h22m30.367756828s) [upload=14892]: Uploading segments...
Aug 8 00:02:30.778539616 (14h22m30.367884623s) [upload=14892]: Got next segment. Inline: false
Aug 8 00:02:30.778583389 (14h22m30.367928376s) [upload=14892]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Aug 8 00:02:30.778587717 (14h22m30.367932704s) [upload=14892]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Aug 8 00:03:30.135980490 (14h23m29.725325467s) [upload=14892]: Next segment err: context deadline exceeded
Aug 8 00:03:30.136177137 (14h23m29.725522124s) [upload=14892]: Done uploading segments...
Aug 8 00:03:30.136196834 (14h23m29.725541821s) [upload=14892]: Upload finished. err: context deadline exceeded
Duration: 5.856941293s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
6 out of 60 is failed (threshold=3) [.................X...X..........X.......X...X............X..] duration:
6.194847884s
FAIL (2025-08-08 00:39:01 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/4M):: after 7dd7aed873cca87b7cd76d50ee66fe90f8c443893d6355bc8519d2fde31a068f != before c1a76dc0bec8105ec080db0072ab886792f3545b344045a49a190b10eaa4f282
======LOGS======
[upload=15508]: Uploading segments...
Aug 8 00:38:24.258804844 (14h58m23.848149831s) [upload=15508]: Got next segment. Inline: false
Aug 8 00:38:24.258847265 (14h58m23.848192252s) [upload=15508]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Aug 8 00:38:24.258872243 (14h58m23.84821723s) [upload=15508]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Aug 8 00:38:24.709304765 (14h58m24.298649752s) [seg_pos={0 0}, upload=15508]: Begin upload segment...
Aug 8 00:38:24.709363377 (14h58m24.298708364s) [seg_pos={0 0}, upload=15508]: Done begin upload segment.
Aug 8 00:38:24.709365892 (14h58m24.298710879s) [noise=true , address=104.36.82.47:4534 , piece_id=G56O4SR7DQ6YHATRCIMLOARHFQBILTCEUU2S35Y5NQKYIQX22X6A, seg_pos={0 0}, upload=15508]: Uploading piece...
Aug 8 00:38:24.709380991 (14h58m24.298725978s) [noise=true , address=104.36.82.39:4734 , piece_id=2ZZDYY5W5NHZ7OFXZ3X566EWP5F6GV32KNCIGTAEMGWDZI3ZHNNQ, seg_pos={0 0}, upload=15508]: Uploading piece...
Aug 8 00:38:24.709715681 (14h58m24.299060668s) [noise=true , address=104.36.82.61:4386 , piece_id=3YTRS2CUXJ3X3AVS6SWI234NQD7G6XPSV7ONZWT22WS23P6GEUOA, seg_pos={0 0}, upload=15508]: Uploading piece...
Aug 8 00:38:24.710014734 (14h58m24.299359721s) [noise=true , address=45.91.116.11:4142 , piece_id=GWGM46SJ4ZLCAW2TTFUX4R3GXX6C4QAMBN62BUKXED6YVROHQ6PQ, seg_pos={0 0}, upload=15508]: Uploading piece...
Aug 8 00:38:24.710261737 (14h58m24.299606724s) [noise=true , address=104.36.82.9:4038 , piece_id=IUFHRP22JUSOJLB66HKMEIK6OJJJPR3NLRA4JHTLQZDMD5IEKQEA, seg_pos={0 0}, upload=15508]: Uploading piece...
Aug 8 00:38:24.710536713 (14h58m24.2998817s) [noise=true , address=45.91.116.10:4113 , piece_id=QNB2CD3IINPCU6F3NCCPV7MNKY2GBV7ZBWUXY6UAYT7IIPJDGTCQ, seg_pos={0 0}, upload=15508]: Uploading piece...
Aug 8 00:38:24.710790879 (14h58m24.300135866s) [noise=true , address=64.49.13.26:4948 , piece_id=QSDEMGP65XIE3CLDSG6ASMKJCOBSSDZN2GWEVFJB3QQEGVCHTUIQ, seg_pos={0 0}, upload=15508]: Uploading piece...
Aug 8 00:38:24.711027322 (14h58m24.300372309s) [noise=true , address=104.36.82.37:4656 , piece_id=QGCIU7D4AG66MLGC7XA435DKIIBGLSLSWHQDVIZ75NFJKTONV4UQ, seg_pos={0 0}, upload=15508]: Uploading piece...
Aug 8 00:38:24.711279785 (14h58m24.300624772s) [noise=true , address=104.36.82.45:4931 , piece_id=IBPB2JL6J2JDMHYYNI3UVNEZ5DUOOTA2SKQKRGV255F6NTCWUDWQ, seg_pos={0 0}, upload=15508]: Uploading piece...
Aug 8 00:38:24.711373895 (14h58m24.300718882s) [noise=true , address=104.36.82.56:4256 , piece_id=3JG4YZGAK5JV6U3HTKEFSOVAKKLN5H66SBTSCUAL6KDS5YELSUWA, seg_pos={0 0}, upload=15508]: Uploading piece...
Aug 8 00:38:24.711652839 (14h58m24.300997826s) [noise=true , address=104.36.82.58:4288 , piece_id=XBQQBKEJMENF6WMGIFEFC4CP2FA4NQ2LW22RUBS4DWTGBOWNHO7A, seg_pos={0 0}, upload=15508]: Uploading piece...
Aug 8 00:38:24.711891356 (14h58m24.301236343s) [noise=true , address=104.36.82.2:4745 , piece_id=JYMY6EJ2ZG6XEJUUUJOIFNNMI4QMINNS4Q77QYXPVLTSC7BLQEXA, seg_pos={0 0}, upload=15508]: Uploading piece...
Aug 8 00:38:24.712120744 (14h58m24.301465732s) [noise=true , address=104.36.82.54:4166 , piece_id=WTEO3MZJMTHGA5PTY5CCQ42SKTJI43M43EULZGGUXAQVIY7C46IA, seg_pos={0 0}, upload=15508]: Uploading piece...
Aug 8 00:38:24.712325797 (14h58m24.301670784s) [noise=true , address=104.36.82.3:4781 , piece_id=A4ZYEHF6HFDP6CX5MALR26F6RA7BE5CST5GDDULJBJR3ALUJAFVQ, seg_pos={0 0}, upload=15508]: Uploading piece...
Aug 8 00:38:24.712565686 (14h58m24.301910673s) [noise=true , address=104.36.82.16:4320 , piece_id=2OP3TMF3FNQE2WTV6HEY6HVNO6KR2FCMPXRZJYCUI3QJIYIRLBOQ, seg_pos={0 0}, upload=15508]: Uploading piece...
Aug 8 00:38:24.712743346 (14h58m24.302088333s) [noise=true , address=45.91.116.14:4192 , piece_id=XIT5DARFYND632CV46LHR3UBVDQ6GAZ2DH3HQFUCTUFMW2Y4VUOQ, seg_pos={0 0}, upload=15508]: Uploading piece...
Aug 8 00:38:24.712959169 (14h58m24.302304156s) [noise=true , address=45.91.116.5:4324 , piece_id=TP3YSP3WXJKXA3DKPDXBRHTAKQBC6EXMZCRG7HBCELRWY6TFVXAA, seg_pos={0 0}, upload=15508]: Uploading piece...
Aug 8 00:38:24.713184781 (14h58m24.302529768s) [noise=true , address=104.36.82.52:4097 , piece_id=DFJXCTAXSGUPKENZVYE6G4EAQF5SE5FKGRHI62QHCCVHYRAQDA7Q, seg_pos={0 0}, upload=15508]: Uploading piece...
Aug 8 00:38:25.841668344 (14h58m25.431013331s) [upload=15508]: Next returned nil segment
Aug 8 00:38:25.841685085 (14h58m25.431030072s) [upload=15508]: Waiting for error group managing segments...
Aug 8 00:38:26.112321232 (14h58m25.701666219s) [noise=true , address=64.49.13.26:4948 , piece_id=QSDEMGP65XIE3CLDSG6ASMKJCOBSSDZN2GWEVFJB3QQEGVCHTUIQ, seg_pos={0 0}, upload=15508]: Done uploading piece. err: <nil>
Aug 8 00:38:26.667567146 (14h58m26.256912133s) [noise=true , address=104.36.82.56:4256 , piece_id=3JG4YZGAK5JV6U3HTKEFSOVAKKLN5H66SBTSCUAL6KDS5YELSUWA, seg_pos={0 0}, upload=15508]: Done uploading piece. err: <nil>
Aug 8 00:38:26.669941781 (14h58m26.259286768s) [noise=true , address=104.36.82.3:4781 , piece_id=A4ZYEHF6HFDP6CX5MALR26F6RA7BE5CST5GDDULJBJR3ALUJAFVQ, seg_pos={0 0}, upload=15508]: Done uploading piece. err: <nil>
Aug 8 00:38:26.699819391 (14h58m26.289164378s) [noise=true , address=45.91.116.10:4113 , piece_id=QNB2CD3IINPCU6F3NCCPV7MNKY2GBV7ZBWUXY6UAYT7IIPJDGTCQ, seg_pos={0 0}, upload=15508]: Done uploading piece. err: <nil>
Aug 8 00:38:26.738125452 (14h58m26.327470439s) [noise=true , address=104.36.82.2:4745 , piece_id=JYMY6EJ2ZG6XEJUUUJOIFNNMI4QMINNS4Q77QYXPVLTSC7BLQEXA, seg_pos={0 0}, upload=15508]: Done uploading piece. err: <nil>
Aug 8 00:38:26.745536716 (14h58m26.334881703s) [noise=true , address=104.36.82.39:4734 , piece_id=2ZZDYY5W5NHZ7OFXZ3X566EWP5F6GV32KNCIGTAEMGWDZI3ZHNNQ, seg_pos={0 0}, upload=15508]: Done uploading piece. err: <nil>
Aug 8 00:38:26.768528297 (14h58m26.357873294s) [noise=true , address=45.91.116.11:4142 , piece_id=GWGM46SJ4ZLCAW2TTFUX4R3GXX6C4QAMBN62BUKXED6YVROHQ6PQ, seg_pos={0 0}, upload=15508]: Done uploading piece. err: <nil>
Aug 8 00:38:26.818308437 (14h58m26.407653414s) [noise=true , address=104.36.82.37:4656 , piece_id=QGCIU7D4AG66MLGC7XA435DKIIBGLSLSWHQDVIZ75NFJKTONV4UQ, seg_pos={0 0}, upload=15508]: Done uploading piece. err: <nil>
Aug 8 00:38:26.832514054 (14h58m26.421859041s) [noise=true , address=45.91.116.14:4192 , piece_id=XIT5DARFYND632CV46LHR3UBVDQ6GAZ2DH3HQFUCTUFMW2Y4VUOQ, seg_pos={0 0}, upload=15508]: Done uploading piece. err: <nil>
Aug 8 00:38:26.869135760 (14h58m26.458480748s) [noise=true , address=104.36.82.54:4166 , piece_id=WTEO3MZJMTHGA5PTY5CCQ42SKTJI43M43EULZGGUXAQVIY7C46IA, seg_pos={0 0}, upload=15508]: Done uploading piece. err: <nil>
Aug 8 00:38:27.131612271 (14h58m26.720957258s) [noise=true , address=45.91.116.5:4324 , piece_id=TP3YSP3WXJKXA3DKPDXBRHTAKQBC6EXMZCRG7HBCELRWY6TFVXAA, seg_pos={0 0}, upload=15508]: Done uploading piece. err: <nil>
Aug 8 00:38:27.152099560 (14h58m26.741444547s) [noise=true , address=104.36.82.58:4288 , piece_id=XBQQBKEJMENF6WMGIFEFC4CP2FA4NQ2LW22RUBS4DWTGBOWNHO7A, seg_pos={0 0}, upload=15508]: Done uploading piece. err: <nil>
Aug 8 00:38:27.501729825 (14h58m27.091074813s) [noise=true , address=104.36.82.16:4320 , piece_id=2OP3TMF3FNQE2WTV6HEY6HVNO6KR2FCMPXRZJYCUI3QJIYIRLBOQ, seg_pos={0 0}, upload=15508]: Done uploading piece. err: <nil>
Aug 8 00:38:27.778376595 (14h58m27.367721582s) [noise=true , address=104.36.82.9:4038 , piece_id=IUFHRP22JUSOJLB66HKMEIK6OJJJPR3NLRA4JHTLQZDMD5IEKQEA, seg_pos={0 0}, upload=15508]: Done uploading piece. err: <nil>
Aug 8 00:38:27.778417313 (14h58m27.3677623s) [seg_pos={0 0}, upload=15508]: Segment reached optimal threshold of 14 pieces.
Aug 8 00:38:27.778642765 (14h58m27.367987752s) [noise=true , address=104.36.82.47:4534 , piece_id=G56O4SR7DQ6YHATRCIMLOARHFQBILTCEUU2S35Y5NQKYIQX22X6A, seg_pos={0 0}, upload=15508]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VDrNGcCnaKEmG5NN4nex1JoZAgdCEfX5CdsKbxytfLfKzPbr): protocol: expected piece hash; context canceled; EOF
Aug 8 00:38:27.778734049 (14h58m27.368079036s) [noise=true , address=104.36.82.61:4386 , piece_id=3YTRS2CUXJ3X3AVS6SWI234NQD7G6XPSV7ONZWT22WS23P6GEUOA, seg_pos={0 0}, upload=15508]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19aPjJnya2Q3stEhiedDSth3NHV8phLRj5yY79hZv81UgDitMv): protocol: write tcp 188.166.225.170:42102->104.36.82.61:4386: use of closed network connection; write tcp 188.166.225.170:42102->104.36.82.61:4386: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42102->104.36.82.61:4386: use of closed network connection
Aug 8 00:38:27.778843298 (14h58m27.368188285s) [noise=true , address=104.36.82.45:4931 , piece_id=IBPB2JL6J2JDMHYYNI3UVNEZ5DUOOTA2SKQKRGV255F6NTCWUDWQ, seg_pos={0 0}, upload=15508]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pCgoSfxjtXPw5jpbz9RJaxRiGoYPdPFdoMPtD19F1ZWY8UwPu): protocol: expected piece hash; context canceled; EOF
Aug 8 00:38:27.778889437 (14h58m27.368234424s) [noise=true , address=104.36.82.52:4097 , piece_id=DFJXCTAXSGUPKENZVYE6G4EAQF5SE5FKGRHI62QHCCVHYRAQDA7Q, seg_pos={0 0}, upload=15508]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1sMhTktqf2H4aZSyFV3m9Z6eMVxL2U52bvUHVHUh2gUCVp32AM): protocol: expected piece hash; context canceled; EOF
Aug 8 00:38:27.778906359 (14h58m27.368251346s) [seg_pos={0 0}, upload=15508]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Aug 8 00:38:27.778981493 (14h58m27.36832647s) [upload=15508]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Aug 8 00:38:27.778985781 (14h58m27.368330768s) [upload=15508]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Aug 8 00:38:28.078873078 (14h58m27.668218065s) [upload=15508]: Done uploading segments...
Aug 8 00:38:28.078891813 (14h58m27.6682368s) [upload=15508]: Upload finished. err: <nil>
Duration: 6.194877823s
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)antiblip:
1 out of 60 is failed (threshold=3) [X...........................................................] duration:
1.8403804s
FAIL (2025-08-08 00:38:57 UTC)
orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded
Duration: 1.840403776s
core
uplink
us1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)antiblip:
1 out of 60 is failed (threshold=3) [..............................X.............................] duration:
1.822006605s
FAIL (2025-08-08 00:38:54 UTC)
orbiter warning:
operation: DownloadObject
======ERROR======
uplink: object not found ("orbiter_us_0/1K")
======LOGS======
=14964]: Done uploading segments...
Aug 8 00:06:41.669168196 (14h26m41.258513183s) [upload=14964]: Upload finished. err: <nil>
Aug 8 00:07:06.258431851 (14h27m5.847776838s) [upload=14976]: Starting upload
Aug 8 00:07:06.258508298 (14h27m5.847853285s) [upload=14976]: Done starting upload
Aug 8 00:07:06.258574986 (14h27m5.847919983s) [upload=14976]: Uploading segments...
Aug 8 00:07:06.258618599 (14h27m5.847963616s) [upload=14976]: Got next segment. Inline: true
Aug 8 00:07:06.258624600 (14h27m5.847969587s) [upload=14976]: Waiting for error group managing segments...
Aug 8 00:07:06.258705555 (14h27m5.848050542s) [upload=14976]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Aug 8 00:07:06.258712268 (14h27m5.848057296s) [upload=14976]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline
Aug 8 00:07:06.258716226 (14h27m5.848061253s) [upload=14976]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Aug 8 00:07:06.522670976 (14h27m6.112015963s) [upload=14976]: Done uploading segments...
Aug 8 00:07:06.522717766 (14h27m6.112062773s) [upload=14976]: Upload finished. err: <nil>
Aug 8 00:07:36.371369594 (14h27m35.960714581s) [upload=14980]: Starting upload
Aug 8 00:07:36.371417074 (14h27m35.960762061s) [upload=14980]: Done starting upload
Aug 8 00:07:36.371466890 (14h27m35.960811877s) [upload=14980]: Uploading segments...
Aug 8 00:07:36.371508951 (14h27m35.960853938s) [upload=14980]: Got next segment. Inline: true
Aug 8 00:07:36.371512888 (14h27m35.960857875s) [upload=14980]: Waiting for error group managing segments...
Aug 8 00:07:36.371543446 (14h27m35.960888433s) [upload=14980]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Aug 8 00:07:36.371547434 (14h27m35.960892421s) [upload=14980]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline
Aug 8 00:07:36.371549708 (14h27m35.960894695s) [upload=14980]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Aug 8 00:07:36.789946570 (14h27m36.379291557s) [upload=14980]: Done uploading segments...
Aug 8 00:07:36.789964876 (14h27m36.379309863s) [upload=14980]: Upload finished. err: <nil>
Aug 8 00:08:16.866388064 (14h28m16.455733051s) [upload=14994]: Starting upload
Aug 8 00:08:16.866430886 (14h28m16.455775873s) [upload=14994]: Done starting upload
Aug 8 00:08:16.866496882 (14h28m16.455841869s) [upload=14994]: Uploading segments...
Aug 8 00:08:16.866525727 (14h28m16.455870714s) [upload=14994]: Got next segment. Inline: true
Aug 8 00:08:16.866529494 (14h28m16.455874481s) [upload=14994]: Waiting for error group managing segments...
Aug 8 00:08:16.866570884 (14h28m16.455915871s) [upload=14994]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Aug 8 00:08:16.866575212 (14h28m16.455920199s) [upload=14994]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline
Aug 8 00:08:16.866577626 (14h28m16.455922613s) [upload=14994]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Aug 8 00:08:17.138322091 (14h28m16.727667078s) [upload=14994]: Done uploading segments...
Aug 8 00:08:17.138338402 (14h28m16.727683389s) [upload=14994]: Upload finished. err: <nil>
Aug 8 00:08:57.633625328 (14h28m57.222970315s) [upload=15006]: Starting upload
Aug 8 00:08:57.633689390 (14h28m57.223034378s) [upload=15006]: Done starting upload
Aug 8 00:08:57.633744837 (14h28m57.223089824s) [upload=15006]: Uploading segments...
Aug 8 00:08:57.633775525 (14h28m57.223120512s) [upload=15006]: Got next segment. Inline: true
Aug 8 00:08:57.633779222 (14h28m57.223124199s) [upload=15006]: Waiting for error group managing segments...
Aug 8 00:08:57.633951743 (14h28m57.22329672s) [upload=15006]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Aug 8 00:08:57.633956161 (14h28m57.223301148s) [upload=15006]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline
Aug 8 00:08:57.633958886 (14h28m57.223303873s) [upload=15006]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Aug 8 00:08:57.920842105 (14h28m57.510187102s) [upload=15006]: Done uploading segments...
Aug 8 00:08:57.920865330 (14h28m57.510210327s) [upload=15006]: Upload finished. err: <nil>
Aug 8 00:09:43.341371442 (14h29m42.930716449s) [upload=15015]: Starting upload
Aug 8 00:09:43.341457767 (14h29m42.930802774s) [upload=15015]: Done starting upload
Aug 8 00:09:43.341511951 (14h29m42.930856938s) [upload=15015]: Uploading segments...
Aug 8 00:09:43.341579249 (14h29m42.930924236s) [upload=15015]: Got next segment. Inline: true
Aug 8 00:09:43.341585642 (14h29m42.930930639s) [upload=15015]: Waiting for error group managing segments...
Aug 8 00:09:43.341640437 (14h29m42.930985454s) [upload=15015]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Aug 8 00:09:43.341648422 (14h29m42.930993469s) [upload=15015]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline
Aug 8 00:09:43.341653391 (14h29m42.930998408s) [upload=15015]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Aug 8 00:09:43.627931533 (14h29m43.21727652s) [upload=15015]: Done uploading segments...
Aug 8 00:09:43.627948425 (14h29m43.217293402s) [upload=15015]: Upload finished. err: <nil>
Aug 8 00:10:45.166177580 (14h30m44.755522568s) [upload=15031]: Starting upload
Aug 8 00:10:45.166220523 (14h30m44.75556551s) [upload=15031]: Done starting upload
Aug 8 00:10:45.166273034 (14h30m44.755618011s) [upload=15031]: Uploading segments...
Aug 8 00:10:45.166297991 (14h30m44.755642979s) [upload=15031]: Got next segment. Inline: true
Aug 8 00:10:45.166301568 (14h30m44.755646555s) [upload=15031]: Waiting for error group managing segments...
Aug 8 00:10:45.166332277 (14h30m44.755677264s) [upload=15031]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Aug 8 00:10:45.166335783 (14h30m44.75568077s) [upload=15031]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline
Aug 8 00:10:45.166337998 (14h30m44.755682985s) [upload=15031]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Aug 8 00:10:45.444515471 (14h30m45.033860458s) [upload=15031]: Done uploading segments...
Aug 8 00:10:45.444533305 (14h30m45.033878282s) [upload=15031]: Upload finished. err: <nil>
Aug 8 00:11:54.979425886 (14h31m54.568770863s) [upload=15052]: Starting upload
Aug 8 00:11:54.979471082 (14h31m54.568816069s) [upload=15052]: Done starting upload
Aug 8 00:11:54.979517260 (14h31m54.568862237s) [upload=15052]: Uploading segments...
Aug 8 00:11:54.979549422 (14h31m54.568894399s) [upload=15052]: Got next segment. Inline: true
Aug 8 00:11:54.979553008 (14h31m54.568897995s) [upload=15052]: Waiting for error group managing segments...
Aug 8 00:11:54.979584328 (14h31m54.568929315s) [upload=15052]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Aug 8 00:11:54.979588356 (14h31m54.568933333s) [upload=15052]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline
Aug 8 00:11:54.979590661 (14h31m54.568935648s) [upload=15052]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Aug 8 00:11:55.340902393 (14h31m54.93024738s) [upload=15052]: Done uploading segments...
Aug 8 00:11:55.340920908 (14h31m54.930265895s) [upload=15052]: Upload finished. err: <nil>
Aug 8 00:12:44.503574128 (14h32m44.092919115s) [upload=15065]: Starting upload
Aug 8 00:12:44.503634083 (14h32m44.09297907s) [upload=15065]: Done starting upload
Aug 8 00:12:44.503710068 (14h32m44.093055055s) [upload=15065]: Uploading segments...
Aug 8 00:12:44.503762669 (14h32m44.093107656s) [upload=15065]: Got next segment. Inline: true
Aug 8 00:12:44.503767698 (14h32m44.093112685s) [upload=15065]: Waiting for error group managing segments...
Aug 8 00:12:44.503805531 (14h32m44.093150508s) [upload=15065]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Aug 8 00:12:44.503809379 (14h32m44.093154366s) [upload=15065]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline
Aug 8 00:12:44.503811633 (14h32m44.09315662s) [upload=15065]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Aug 8 00:12:44.792734473 (14h32m44.38207946s) [upload=15065]: Done uploading segments...
Aug 8 00:12:44.792752288 (14h32m44.382097275s) [upload=15065]: Upload finished. err: <nil>
Aug 8 00:13:19.961146705 (14h33m19.550491693s) [upload=15079]: Starting upload
Aug 8 00:13:19.961187835 (14h33m19.550532822s) [upload=15079]: Done starting upload
Aug 8 00:13:19.961236618 (14h33m19.550581605s) [upload=15079]: Uploading segments...
Aug 8 00:13:19.961268168 (14h33m19.550613145s) [upload=15079]: Got next segment. Inline: true
Aug 8 00:13:19.961273849 (14h33m19.550618836s) [upload=15079]: Waiting for error group managing segments...
Aug 8 00:13:19.961304548 (14h33m19.550649525s) [upload=15079]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Aug 8 00:13:19.961308345 (14h33m19.550653332s) [upload=15079]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline
Aug 8 00:13:19.961310910 (14h33m19.550655887s) [upload=15079]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Aug 8 00:13:20.385237160 (14h33m19.974582137s) [upload=15079]: Done uploading segments...
Aug 8 00:13:20.385255485 (14h33m19.974600472s) [upload=15079]: Upload finished. err: <nil>
Aug 8 00:14:06.785741109 (14h34m6.375086096s) [upload=15091]: Starting upload
Aug 8 00:14:06.785899151 (14h34m6.375244138s) [upload=15091]: Done starting upload
Aug 8 00:14:06.786002628 (14h34m6.375347615s) [upload=15091]: Uploading segments...
Aug 8 00:14:06.786034078 (14h34m6.375379065s) [upload=15091]: Got next segment. Inline: true
Aug 8 00:14:06.786037786 (14h34m6.375382773s) [upload=15091]: Waiting for error group managing segments...
Aug 8 00:14:06.786073094 (14h34m6.375418081s) [upload=15091]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Aug 8 00:14:06.786078945 (14h34m6.375423922s) [upload=15091]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline
Aug 8 00:14:06.786081339 (14h34m6.375426326s) [upload=15091]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Aug 8 00:14:07.092788097 (14h34m6.682133084s) [upload=15091]: Done uploading segments...
Aug 8 00:14:07.092803166 (14h34m6.682148153s) [upload=15091]: Upload finished. err: <nil>
Duration: 1.822031401s
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
7 out of 60 is failed (threshold=3) [X..XXXX....XX...............................................] duration:
5.855126598s
FAIL (2025-08-08 00:38:51 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
3.700898256s) [upload=14855]: Done starting upload
Aug 8 00:00:34.111602513 (14h20m33.7009475s) [upload=14855]: Uploading segments...
Aug 8 00:00:34.111774201 (14h20m33.701119218s) [upload=14855]: Got next segment. Inline: false
Aug 8 00:00:34.111816783 (14h20m33.70116178s) [upload=14855]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Aug 8 00:00:34.111822464 (14h20m33.701167461s) [upload=14855]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Aug 8 00:01:03.465979471 (14h21m3.055324468s) [upload=14855]: Next segment err: context deadline exceeded
Aug 8 00:01:03.466148254 (14h21m3.055493251s) [upload=14855]: Done uploading segments...
Aug 8 00:01:03.466174945 (14h21m3.055519932s) [upload=14855]: Upload finished. err: context deadline exceeded
Aug 8 00:01:19.453976971 (14h21m19.043321958s) [upload=14872]: Starting upload
Aug 8 00:01:19.454042106 (14h21m19.043387123s) [upload=14872]: Done starting upload
Aug 8 00:01:19.454156104 (14h21m19.043501091s) [upload=14872]: Uploading segments...
Aug 8 00:01:19.454244503 (14h21m19.04358949s) [upload=14872]: Got next segment. Inline: false
Aug 8 00:01:19.454272567 (14h21m19.043617554s) [upload=14872]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Aug 8 00:01:19.454276825 (14h21m19.043621812s) [upload=14872]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Aug 8 00:01:39.582519114 (14h21m39.171864101s) [seg_pos={0 0}, upload=14872]: Begin upload segment...
Aug 8 00:01:39.582565882 (14h21m39.17191087s) [seg_pos={0 0}, upload=14872]: Done begin upload segment.
Aug 8 00:01:39.582581332 (14h21m39.17192632s) [noise=true , address=45.91.116.5:4299 , piece_id=HRKD77XA4SHZKSEW2WA3AROWPLHOWEXG5GEOWJCG7NTFOYYZ4DFA, seg_pos={0 0}, upload=14872]: Uploading piece...
Aug 8 00:01:39.582590870 (14h21m39.171935857s) [noise=true , address=104.36.82.2:4731 , piece_id=2BTN7TE7VVSPN5BUCDVKHWZPPATMNIJLXNDXUO46Z7CRB3QBCUEQ, seg_pos={0 0}, upload=14872]: Uploading piece...
Aug 8 00:01:39.582951871 (14h21m39.172296858s) [noise=true , address=45.91.116.12:4225 , piece_id=MUZ4UWUJSWGYKIO3BSTEOJYBBXSK4R2PURNDXHOYLTZ7VZ466IGA, seg_pos={0 0}, upload=14872]: Uploading piece...
Aug 8 00:01:39.583059888 (14h21m39.172404905s) [noise=true , address=104.36.82.13:4185 , piece_id=O26DWLA45AEOUZOW2CZTLJNPVRWB5KVDQW6FH2ANTKLUMYNWRVKA, seg_pos={0 0}, upload=14872]: Uploading piece...
Aug 8 00:01:39.583214905 (14h21m39.172559892s) [noise=true , address=45.91.116.8:4047 , piece_id=4LUGFRA2QPUK5LPKZGNTURXXSGY64AZU4Q3BHBAM3SPBZ7M2FFKQ, seg_pos={0 0}, upload=14872]: Uploading piece...
Aug 8 00:01:39.583427882 (14h21m39.172772869s) [noise=true , address=77.90.31.21:4877 , piece_id=MZ4WJDOFWH56JIAGXJZ7FOAPMNCHVEBUUM2G5IY2KKPRHPJH5MDA, seg_pos={0 0}, upload=14872]: Uploading piece...
Aug 8 00:01:39.583689032 (14h21m39.173034019s) [noise=true , address=77.90.31.11:4555 , piece_id=OCZBWUJHEVLMUD6BNADJ7BOSHHZPESQJOSKDZ3T2MUSQRFTNI7KA, seg_pos={0 0}, upload=14872]: Uploading piece...
Aug 8 00:01:39.583925344 (14h21m39.173270331s) [noise=true , address=45.91.116.9:4263 , piece_id=F4QDS3EOSLFFCEDU6N56AMMQR4GHPWZ6TIAMO5SGVIMVMFA47EQQ, seg_pos={0 0}, upload=14872]: Uploading piece...
Aug 8 00:01:39.584272317 (14h21m39.173617294s) [noise=true , address=104.36.82.8:4970 , piece_id=N52DS4AB7Z6VSFYWHN5MXPUS3AP7UKN2K7U5D5JZWETM7ZLROO7Q, seg_pos={0 0}, upload=14872]: Uploading piece...
Aug 8 00:01:39.584579135 (14h21m39.173924122s) [noise=true , address=77.90.31.18:4807 , piece_id=V7EZJW5PYJCWWNNIRAJCFTT42DQJY2C4RDHB5645TKOSXJJXE2WA, seg_pos={0 0}, upload=14872]: Uploading piece...
Aug 8 00:01:39.584852117 (14h21m39.174197104s) [noise=true , address=104.36.82.11:4115 , piece_id=S34RF36SERFPPSHBPYJ5V3FCR5ZE7RFIVEXBOTFRHPEEIUCHO2LQ, seg_pos={0 0}, upload=14872]: Uploading piece...
Aug 8 00:01:39.585105202 (14h21m39.174450179s) [noise=true , address=104.36.82.9:4005 , piece_id=ZYZ6VK5MCW7QE6N2GOYW2WMWDEWGONNK2Y7QBE7FPVPLY2OZGIMQ, seg_pos={0 0}, upload=14872]: Uploading piece...
Aug 8 00:01:39.585298762 (14h21m39.174643739s) [noise=true , address=45.91.116.4:4940 , piece_id=5S5JLBIGSBIVTPVJT5MIRAE42PRCIFWDNWLY7NQXSWPMHT5WSTRA, seg_pos={0 0}, upload=14872]: Uploading piece...
Aug 8 00:01:39.585506369 (14h21m39.174851346s) [noise=true , address=104.36.82.10:4079 , piece_id=262SP4DREL7FYDG6YOBFWDPWKYHZIQGDJKSX7XAIUJUTQWUIEEZQ, seg_pos={0 0}, upload=14872]: Uploading piece...
Aug 8 00:01:39.585763882 (14h21m39.175108869s) [noise=true , address=45.91.116.14:4190 , piece_id=FSO3UN2D24KPDVSWWPI3ROBXN7RN6D2HAZKOXEQ2JOW2M6RQ5MDA, seg_pos={0 0}, upload=14872]: Uploading piece...
Aug 8 00:01:39.586021575 (14h21m39.175366562s) [noise=true , address=104.36.82.5:4839 , piece_id=EHHLJG3INARPORGO6YVXU7QJAK23ADRVWBGNT5IKL4N6Y4ASM3XA, seg_pos={0 0}, upload=14872]: Uploading piece...
Aug 8 00:01:41.370103161 (14h21m40.959448138s) [upload=14872]: Next returned nil segment
Aug 8 00:01:41.370130243 (14h21m40.95947523s) [upload=14872]: Waiting for error group managing segments...
Aug 8 00:01:41.673384945 (14h21m41.262729922s) [noise=true , address=104.36.82.2:4731 , piece_id=2BTN7TE7VVSPN5BUCDVKHWZPPATMNIJLXNDXUO46Z7CRB3QBCUEQ, seg_pos={0 0}, upload=14872]: Done uploading piece. err: <nil>
Aug 8 00:01:41.739890780 (14h21m41.329235767s) [noise=true , address=104.36.82.9:4005 , piece_id=ZYZ6VK5MCW7QE6N2GOYW2WMWDEWGONNK2Y7QBE7FPVPLY2OZGIMQ, seg_pos={0 0}, upload=14872]: Done uploading piece. err: <nil>
Aug 8 00:01:41.743845828 (14h21m41.333190815s) [noise=true , address=104.36.82.8:4970 , piece_id=N52DS4AB7Z6VSFYWHN5MXPUS3AP7UKN2K7U5D5JZWETM7ZLROO7Q, seg_pos={0 0}, upload=14872]: Done uploading piece. err: <nil>
Aug 8 00:01:41.786086095 (14h21m41.375431072s) [noise=true , address=45.91.116.8:4047 , piece_id=4LUGFRA2QPUK5LPKZGNTURXXSGY64AZU4Q3BHBAM3SPBZ7M2FFKQ, seg_pos={0 0}, upload=14872]: Done uploading piece. err: <nil>
Aug 8 00:01:41.791525722 (14h21m41.380870699s) [noise=true , address=104.36.82.5:4839 , piece_id=EHHLJG3INARPORGO6YVXU7QJAK23ADRVWBGNT5IKL4N6Y4ASM3XA, seg_pos={0 0}, upload=14872]: Done uploading piece. err: <nil>
Aug 8 00:01:41.959657137 (14h21m41.549002124s) [noise=true , address=104.36.82.13:4185 , piece_id=O26DWLA45AEOUZOW2CZTLJNPVRWB5KVDQW6FH2ANTKLUMYNWRVKA, seg_pos={0 0}, upload=14872]: Done uploading piece. err: <nil>
Aug 8 00:01:42.000743686 (14h21m41.590088693s) [noise=true , address=45.91.116.12:4225 , piece_id=MUZ4UWUJSWGYKIO3BSTEOJYBBXSK4R2PURNDXHOYLTZ7VZ466IGA, seg_pos={0 0}, upload=14872]: Done uploading piece. err: <nil>
Aug 8 00:01:42.160166408 (14h21m41.749511395s) [noise=true , address=45.91.116.4:4940 , piece_id=5S5JLBIGSBIVTPVJT5MIRAE42PRCIFWDNWLY7NQXSWPMHT5WSTRA, seg_pos={0 0}, upload=14872]: Done uploading piece. err: <nil>
Aug 8 00:01:42.175399653 (14h21m41.764744641s) [noise=true , address=45.91.116.14:4190 , piece_id=FSO3UN2D24KPDVSWWPI3ROBXN7RN6D2HAZKOXEQ2JOW2M6RQ5MDA, seg_pos={0 0}, upload=14872]: Done uploading piece. err: <nil>
Aug 8 00:01:42.175880162 (14h21m41.765225139s) [noise=true , address=45.91.116.5:4299 , piece_id=HRKD77XA4SHZKSEW2WA3AROWPLHOWEXG5GEOWJCG7NTFOYYZ4DFA, seg_pos={0 0}, upload=14872]: Done uploading piece. err: <nil>
Aug 8 00:01:42.278394081 (14h21m41.867739068s) [noise=true , address=77.90.31.21:4877 , piece_id=MZ4WJDOFWH56JIAGXJZ7FOAPMNCHVEBUUM2G5IY2KKPRHPJH5MDA, seg_pos={0 0}, upload=14872]: Done uploading piece. err: <nil>
Aug 8 00:01:42.293663936 (14h21m41.883008933s) [noise=true , address=77.90.31.18:4807 , piece_id=V7EZJW5PYJCWWNNIRAJCFTT42DQJY2C4RDHB5645TKOSXJJXE2WA, seg_pos={0 0}, upload=14872]: Done uploading piece. err: <nil>
Aug 8 00:01:42.309113927 (14h21m41.898458904s) [noise=true , address=45.91.116.9:4263 , piece_id=F4QDS3EOSLFFCEDU6N56AMMQR4GHPWZ6TIAMO5SGVIMVMFA47EQQ, seg_pos={0 0}, upload=14872]: Done uploading piece. err: <nil>
Aug 8 00:01:42.309137942 (14h21m41.898482919s) [seg_pos={0 0}, upload=14872]: Segment reached optimal threshold of 13 pieces.
Aug 8 00:01:42.309315973 (14h21m41.89866099s) [noise=true , address=77.90.31.11:4555 , piece_id=OCZBWUJHEVLMUD6BNADJ7BOSHHZPESQJOSKDZ3T2MUSQRFTNI7KA, seg_pos={0 0}, upload=14872]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ): protocol: expected piece hash; context canceled; EOF
Aug 8 00:01:42.309402328 (14h21m41.898747305s) [noise=true , address=104.36.82.10:4079 , piece_id=262SP4DREL7FYDG6YOBFWDPWKYHZIQGDJKSX7XAIUJUTQWUIEEZQ, seg_pos={0 0}, upload=14872]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud): protocol: expected piece hash; context canceled; EOF
Aug 8 00:01:42.309425222 (14h21m41.898770229s) [noise=true , address=104.36.82.11:4115 , piece_id=S34RF36SERFPPSHBPYJ5V3FCR5ZE7RFIVEXBOTFRHPEEIUCHO2LQ, seg_pos={0 0}, upload=14872]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR): protocol: write tcp 188.166.225.170:42778->104.36.82.11:4115: use of closed network connection; write tcp 188.166.225.170:42778->104.36.82.11:4115: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42778->104.36.82.11:4115: use of closed network connection
Aug 8 00:01:42.309450360 (14h21m41.898795347s) [seg_pos={0 0}, upload=14872]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Aug 8 00:01:42.309540663 (14h21m41.89888565s) [upload=14872]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Aug 8 00:01:42.309548688 (14h21m41.898893735s) [upload=14872]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Aug 8 00:01:57.644545293 (14h21m57.2338903s) [upload=14872]: Done uploading segments...
Aug 8 00:01:57.644581281 (14h21m57.233926288s) [upload=14872]: Upload finished. err: metaclient: context deadline exceeded
Duration: 5.855151796s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
6 out of 60 is failed (threshold=3) [..................X...X..........X.......X...X............X.] duration:
6.219178356s
FAIL (2025-08-08 00:38:45 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/4M):: after 7dd7aed873cca87b7cd76d50ee66fe90f8c443893d6355bc8519d2fde31a068f != before c1a76dc0bec8105ec080db0072ab886792f3545b344045a49a190b10eaa4f282
======LOGS======
[upload=15508]: Uploading segments...
Aug 8 00:38:24.258804844 (14h58m23.848149831s) [upload=15508]: Got next segment. Inline: false
Aug 8 00:38:24.258847265 (14h58m23.848192252s) [upload=15508]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Aug 8 00:38:24.258872243 (14h58m23.84821723s) [upload=15508]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Aug 8 00:38:24.709304765 (14h58m24.298649752s) [seg_pos={0 0}, upload=15508]: Begin upload segment...
Aug 8 00:38:24.709363377 (14h58m24.298708364s) [seg_pos={0 0}, upload=15508]: Done begin upload segment.
Aug 8 00:38:24.709365892 (14h58m24.298710879s) [noise=true , address=104.36.82.47:4534 , piece_id=G56O4SR7DQ6YHATRCIMLOARHFQBILTCEUU2S35Y5NQKYIQX22X6A, seg_pos={0 0}, upload=15508]: Uploading piece...
Aug 8 00:38:24.709380991 (14h58m24.298725978s) [noise=true , address=104.36.82.39:4734 , piece_id=2ZZDYY5W5NHZ7OFXZ3X566EWP5F6GV32KNCIGTAEMGWDZI3ZHNNQ, seg_pos={0 0}, upload=15508]: Uploading piece...
Aug 8 00:38:24.709715681 (14h58m24.299060668s) [noise=true , address=104.36.82.61:4386 , piece_id=3YTRS2CUXJ3X3AVS6SWI234NQD7G6XPSV7ONZWT22WS23P6GEUOA, seg_pos={0 0}, upload=15508]: Uploading piece...
Aug 8 00:38:24.710014734 (14h58m24.299359721s) [noise=true , address=45.91.116.11:4142 , piece_id=GWGM46SJ4ZLCAW2TTFUX4R3GXX6C4QAMBN62BUKXED6YVROHQ6PQ, seg_pos={0 0}, upload=15508]: Uploading piece...
Aug 8 00:38:24.710261737 (14h58m24.299606724s) [noise=true , address=104.36.82.9:4038 , piece_id=IUFHRP22JUSOJLB66HKMEIK6OJJJPR3NLRA4JHTLQZDMD5IEKQEA, seg_pos={0 0}, upload=15508]: Uploading piece...
Aug 8 00:38:24.710536713 (14h58m24.2998817s) [noise=true , address=45.91.116.10:4113 , piece_id=QNB2CD3IINPCU6F3NCCPV7MNKY2GBV7ZBWUXY6UAYT7IIPJDGTCQ, seg_pos={0 0}, upload=15508]: Uploading piece...
Aug 8 00:38:24.710790879 (14h58m24.300135866s) [noise=true , address=64.49.13.26:4948 , piece_id=QSDEMGP65XIE3CLDSG6ASMKJCOBSSDZN2GWEVFJB3QQEGVCHTUIQ, seg_pos={0 0}, upload=15508]: Uploading piece...
Aug 8 00:38:24.711027322 (14h58m24.300372309s) [noise=true , address=104.36.82.37:4656 , piece_id=QGCIU7D4AG66MLGC7XA435DKIIBGLSLSWHQDVIZ75NFJKTONV4UQ, seg_pos={0 0}, upload=15508]: Uploading piece...
Aug 8 00:38:24.711279785 (14h58m24.300624772s) [noise=true , address=104.36.82.45:4931 , piece_id=IBPB2JL6J2JDMHYYNI3UVNEZ5DUOOTA2SKQKRGV255F6NTCWUDWQ, seg_pos={0 0}, upload=15508]: Uploading piece...
Aug 8 00:38:24.711373895 (14h58m24.300718882s) [noise=true , address=104.36.82.56:4256 , piece_id=3JG4YZGAK5JV6U3HTKEFSOVAKKLN5H66SBTSCUAL6KDS5YELSUWA, seg_pos={0 0}, upload=15508]: Uploading piece...
Aug 8 00:38:24.711652839 (14h58m24.300997826s) [noise=true , address=104.36.82.58:4288 , piece_id=XBQQBKEJMENF6WMGIFEFC4CP2FA4NQ2LW22RUBS4DWTGBOWNHO7A, seg_pos={0 0}, upload=15508]: Uploading piece...
Aug 8 00:38:24.711891356 (14h58m24.301236343s) [noise=true , address=104.36.82.2:4745 , piece_id=JYMY6EJ2ZG6XEJUUUJOIFNNMI4QMINNS4Q77QYXPVLTSC7BLQEXA, seg_pos={0 0}, upload=15508]: Uploading piece...
Aug 8 00:38:24.712120744 (14h58m24.301465732s) [noise=true , address=104.36.82.54:4166 , piece_id=WTEO3MZJMTHGA5PTY5CCQ42SKTJI43M43EULZGGUXAQVIY7C46IA, seg_pos={0 0}, upload=15508]: Uploading piece...
Aug 8 00:38:24.712325797 (14h58m24.301670784s) [noise=true , address=104.36.82.3:4781 , piece_id=A4ZYEHF6HFDP6CX5MALR26F6RA7BE5CST5GDDULJBJR3ALUJAFVQ, seg_pos={0 0}, upload=15508]: Uploading piece...
Aug 8 00:38:24.712565686 (14h58m24.301910673s) [noise=true , address=104.36.82.16:4320 , piece_id=2OP3TMF3FNQE2WTV6HEY6HVNO6KR2FCMPXRZJYCUI3QJIYIRLBOQ, seg_pos={0 0}, upload=15508]: Uploading piece...
Aug 8 00:38:24.712743346 (14h58m24.302088333s) [noise=true , address=45.91.116.14:4192 , piece_id=XIT5DARFYND632CV46LHR3UBVDQ6GAZ2DH3HQFUCTUFMW2Y4VUOQ, seg_pos={0 0}, upload=15508]: Uploading piece...
Aug 8 00:38:24.712959169 (14h58m24.302304156s) [noise=true , address=45.91.116.5:4324 , piece_id=TP3YSP3WXJKXA3DKPDXBRHTAKQBC6EXMZCRG7HBCELRWY6TFVXAA, seg_pos={0 0}, upload=15508]: Uploading piece...
Aug 8 00:38:24.713184781 (14h58m24.302529768s) [noise=true , address=104.36.82.52:4097 , piece_id=DFJXCTAXSGUPKENZVYE6G4EAQF5SE5FKGRHI62QHCCVHYRAQDA7Q, seg_pos={0 0}, upload=15508]: Uploading piece...
Aug 8 00:38:25.841668344 (14h58m25.431013331s) [upload=15508]: Next returned nil segment
Aug 8 00:38:25.841685085 (14h58m25.431030072s) [upload=15508]: Waiting for error group managing segments...
Aug 8 00:38:26.112321232 (14h58m25.701666219s) [noise=true , address=64.49.13.26:4948 , piece_id=QSDEMGP65XIE3CLDSG6ASMKJCOBSSDZN2GWEVFJB3QQEGVCHTUIQ, seg_pos={0 0}, upload=15508]: Done uploading piece. err: <nil>
Aug 8 00:38:26.667567146 (14h58m26.256912133s) [noise=true , address=104.36.82.56:4256 , piece_id=3JG4YZGAK5JV6U3HTKEFSOVAKKLN5H66SBTSCUAL6KDS5YELSUWA, seg_pos={0 0}, upload=15508]: Done uploading piece. err: <nil>
Aug 8 00:38:26.669941781 (14h58m26.259286768s) [noise=true , address=104.36.82.3:4781 , piece_id=A4ZYEHF6HFDP6CX5MALR26F6RA7BE5CST5GDDULJBJR3ALUJAFVQ, seg_pos={0 0}, upload=15508]: Done uploading piece. err: <nil>
Aug 8 00:38:26.699819391 (14h58m26.289164378s) [noise=true , address=45.91.116.10:4113 , piece_id=QNB2CD3IINPCU6F3NCCPV7MNKY2GBV7ZBWUXY6UAYT7IIPJDGTCQ, seg_pos={0 0}, upload=15508]: Done uploading piece. err: <nil>
Aug 8 00:38:26.738125452 (14h58m26.327470439s) [noise=true , address=104.36.82.2:4745 , piece_id=JYMY6EJ2ZG6XEJUUUJOIFNNMI4QMINNS4Q77QYXPVLTSC7BLQEXA, seg_pos={0 0}, upload=15508]: Done uploading piece. err: <nil>
Aug 8 00:38:26.745536716 (14h58m26.334881703s) [noise=true , address=104.36.82.39:4734 , piece_id=2ZZDYY5W5NHZ7OFXZ3X566EWP5F6GV32KNCIGTAEMGWDZI3ZHNNQ, seg_pos={0 0}, upload=15508]: Done uploading piece. err: <nil>
Aug 8 00:38:26.768528297 (14h58m26.357873294s) [noise=true , address=45.91.116.11:4142 , piece_id=GWGM46SJ4ZLCAW2TTFUX4R3GXX6C4QAMBN62BUKXED6YVROHQ6PQ, seg_pos={0 0}, upload=15508]: Done uploading piece. err: <nil>
Aug 8 00:38:26.818308437 (14h58m26.407653414s) [noise=true , address=104.36.82.37:4656 , piece_id=QGCIU7D4AG66MLGC7XA435DKIIBGLSLSWHQDVIZ75NFJKTONV4UQ, seg_pos={0 0}, upload=15508]: Done uploading piece. err: <nil>
Aug 8 00:38:26.832514054 (14h58m26.421859041s) [noise=true , address=45.91.116.14:4192 , piece_id=XIT5DARFYND632CV46LHR3UBVDQ6GAZ2DH3HQFUCTUFMW2Y4VUOQ, seg_pos={0 0}, upload=15508]: Done uploading piece. err: <nil>
Aug 8 00:38:26.869135760 (14h58m26.458480748s) [noise=true , address=104.36.82.54:4166 , piece_id=WTEO3MZJMTHGA5PTY5CCQ42SKTJI43M43EULZGGUXAQVIY7C46IA, seg_pos={0 0}, upload=15508]: Done uploading piece. err: <nil>
Aug 8 00:38:27.131612271 (14h58m26.720957258s) [noise=true , address=45.91.116.5:4324 , piece_id=TP3YSP3WXJKXA3DKPDXBRHTAKQBC6EXMZCRG7HBCELRWY6TFVXAA, seg_pos={0 0}, upload=15508]: Done uploading piece. err: <nil>
Aug 8 00:38:27.152099560 (14h58m26.741444547s) [noise=true , address=104.36.82.58:4288 , piece_id=XBQQBKEJMENF6WMGIFEFC4CP2FA4NQ2LW22RUBS4DWTGBOWNHO7A, seg_pos={0 0}, upload=15508]: Done uploading piece. err: <nil>
Aug 8 00:38:27.501729825 (14h58m27.091074813s) [noise=true , address=104.36.82.16:4320 , piece_id=2OP3TMF3FNQE2WTV6HEY6HVNO6KR2FCMPXRZJYCUI3QJIYIRLBOQ, seg_pos={0 0}, upload=15508]: Done uploading piece. err: <nil>
Aug 8 00:38:27.778376595 (14h58m27.367721582s) [noise=true , address=104.36.82.9:4038 , piece_id=IUFHRP22JUSOJLB66HKMEIK6OJJJPR3NLRA4JHTLQZDMD5IEKQEA, seg_pos={0 0}, upload=15508]: Done uploading piece. err: <nil>
Aug 8 00:38:27.778417313 (14h58m27.3677623s) [seg_pos={0 0}, upload=15508]: Segment reached optimal threshold of 14 pieces.
Aug 8 00:38:27.778642765 (14h58m27.367987752s) [noise=true , address=104.36.82.47:4534 , piece_id=G56O4SR7DQ6YHATRCIMLOARHFQBILTCEUU2S35Y5NQKYIQX22X6A, seg_pos={0 0}, upload=15508]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VDrNGcCnaKEmG5NN4nex1JoZAgdCEfX5CdsKbxytfLfKzPbr): protocol: expected piece hash; context canceled; EOF
Aug 8 00:38:27.778734049 (14h58m27.368079036s) [noise=true , address=104.36.82.61:4386 , piece_id=3YTRS2CUXJ3X3AVS6SWI234NQD7G6XPSV7ONZWT22WS23P6GEUOA, seg_pos={0 0}, upload=15508]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19aPjJnya2Q3stEhiedDSth3NHV8phLRj5yY79hZv81UgDitMv): protocol: write tcp 188.166.225.170:42102->104.36.82.61:4386: use of closed network connection; write tcp 188.166.225.170:42102->104.36.82.61:4386: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42102->104.36.82.61:4386: use of closed network connection
Aug 8 00:38:27.778843298 (14h58m27.368188285s) [noise=true , address=104.36.82.45:4931 , piece_id=IBPB2JL6J2JDMHYYNI3UVNEZ5DUOOTA2SKQKRGV255F6NTCWUDWQ, seg_pos={0 0}, upload=15508]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pCgoSfxjtXPw5jpbz9RJaxRiGoYPdPFdoMPtD19F1ZWY8UwPu): protocol: expected piece hash; context canceled; EOF
Aug 8 00:38:27.778889437 (14h58m27.368234424s) [noise=true , address=104.36.82.52:4097 , piece_id=DFJXCTAXSGUPKENZVYE6G4EAQF5SE5FKGRHI62QHCCVHYRAQDA7Q, seg_pos={0 0}, upload=15508]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1sMhTktqf2H4aZSyFV3m9Z6eMVxL2U52bvUHVHUh2gUCVp32AM): protocol: expected piece hash; context canceled; EOF
Aug 8 00:38:27.778906359 (14h58m27.368251346s) [seg_pos={0 0}, upload=15508]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Aug 8 00:38:27.778981493 (14h58m27.36832647s) [upload=15508]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Aug 8 00:38:27.778985781 (14h58m27.368330768s) [upload=15508]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Aug 8 00:38:28.078873078 (14h58m27.668218065s) [upload=15508]: Done uploading segments...
Aug 8 00:38:28.078891813 (14h58m27.6682368s) [upload=15508]: Upload finished. err: <nil>
Duration: 6.219208974s
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)antiblip:
6 out of 60 is failed (threshold=3) [.........X.XXXX....X........................................] duration:
1.909627053s
FAIL (2025-08-08 00:38:42 UTC)
orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 9109253s) [upload=14753]: Starting upload Aug 7 23:54:04.489828037 (14h14m4.079173014s) [upload=14753]: Done starting upload Aug 7 23:54:04.489877612 (14h14m4.079222589s) [upload=14753]: Uploading segments... Aug 7 23:54:04.489903933 (14h14m4.07924892s) [upload=14753]: Got next segment. Inline: true Aug 7 23:54:04.489907469 (14h14m4.079252446s) [upload=14753]: Waiting for error group managing segments... Aug 7 23:54:04.489939651 (14h14m4.079284638s) [upload=14753]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 23:54:04.489943298 (14h14m4.079288285s) [upload=14753]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 23:54:04.489945472 (14h14m4.079290459s) [upload=14753]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 23:54:04.778726062 (14h14m4.368071049s) [upload=14753]: Done uploading segments... Aug 7 23:54:04.778744989 (14h14m4.368089976s) [upload=14753]: Upload finished. err: <nil> Aug 7 23:55:14.199719855 (14h15m13.789064892s) [upload=14770]: Starting upload Aug 7 23:55:14.199777645 (14h15m13.789122632s) [upload=14770]: Done starting upload Aug 7 23:55:14.199824415 (14h15m13.789169392s) [upload=14770]: Uploading segments... Aug 7 23:55:14.199852819 (14h15m13.789197806s) [upload=14770]: Got next segment. Inline: true Aug 7 23:55:14.199856436 (14h15m13.789201423s) [upload=14770]: Waiting for error group managing segments... Aug 7 23:55:14.199889419 (14h15m13.789234406s) [upload=14770]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 23:55:14.199893557 (14h15m13.789238544s) [upload=14770]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 23:55:14.199895951 (14h15m13.789240938s) [upload=14770]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 23:56:13.542516296 (14h16m13.131861283s) [upload=14770]: Done uploading segments... Aug 7 23:56:13.542536334 (14h16m13.131881321s) [upload=14770]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:56:14.428677736 (14h16m14.018022723s) [upload=14788]: Starting upload Aug 7 23:56:14.428730978 (14h16m14.018075965s) [upload=14788]: Done starting upload Aug 7 23:56:14.428789289 (14h16m14.018134276s) [upload=14788]: Uploading segments... Aug 7 23:56:14.428819026 (14h16m14.018164013s) [upload=14788]: Got next segment. Inline: true Aug 7 23:56:14.428822873 (14h16m14.01816785s) [upload=14788]: Waiting for error group managing segments... Aug 7 23:56:14.428855215 (14h16m14.018200202s) [upload=14788]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 23:56:14.428858562 (14h16m14.018203549s) [upload=14788]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 23:56:14.428860736 (14h16m14.018205723s) [upload=14788]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 23:57:13.781984834 (14h17m13.371329821s) [upload=14788]: Done uploading segments... Aug 7 23:57:13.782003370 (14h17m13.371348357s) [upload=14788]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:57:29.079359478 (14h17m28.668704466s) [upload=14808]: Starting upload Aug 7 23:57:29.079416147 (14h17m28.668761134s) [upload=14808]: Done starting upload Aug 7 23:57:29.079461845 (14h17m28.668806822s) [upload=14808]: Uploading segments... Aug 7 23:57:29.079488876 (14h17m28.668833863s) [upload=14808]: Got next segment. Inline: true Aug 7 23:57:29.079492263 (14h17m28.66883725s) [upload=14808]: Waiting for error group managing segments... Aug 7 23:57:29.079523392 (14h17m28.668868379s) [upload=14808]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 23:57:29.079527049 (14h17m28.668872036s) [upload=14808]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 23:57:29.079529484 (14h17m28.668874461s) [upload=14808]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 23:58:28.439377526 (14h18m28.028722513s) [upload=14808]: Done uploading segments... Aug 7 23:58:28.439395721 (14h18m28.028740708s) [upload=14808]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:58:29.333669502 (14h18m28.923014489s) [upload=14825]: Starting upload Aug 7 23:58:29.333742581 (14h18m28.923087568s) [upload=14825]: Done starting upload Aug 7 23:58:29.333835740 (14h18m28.923180727s) [upload=14825]: Uploading segments... Aug 7 23:58:29.333878000 (14h18m28.923222987s) [upload=14825]: Got next segment. Inline: true Aug 7 23:58:29.333881998 (14h18m28.923226975s) [upload=14825]: Waiting for error group managing segments... Aug 7 23:58:29.333911455 (14h18m28.923256442s) [upload=14825]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 23:58:29.333917496 (14h18m28.923262473s) [upload=14825]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 23:58:29.333919741 (14h18m28.923264728s) [upload=14825]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 23:59:28.692415040 (14h19m28.281760027s) [upload=14825]: Done uploading segments... Aug 7 23:59:28.692444777 (14h19m28.281789764s) [upload=14825]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:59:29.562470053 (14h19m29.15181504s) [upload=14840]: Starting upload Aug 7 23:59:29.562538985 (14h19m29.151883972s) [upload=14840]: Done starting upload Aug 7 23:59:29.562614629 (14h19m29.151959616s) [upload=14840]: Uploading segments... Aug 7 23:59:29.562650589 (14h19m29.151995566s) [upload=14840]: Got next segment. Inline: true Aug 7 23:59:29.562655498 (14h19m29.152000485s) [upload=14840]: Waiting for error group managing segments... Aug 7 23:59:29.562686507 (14h19m29.152031494s) [upload=14840]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 23:59:29.562690264 (14h19m29.152035241s) [upload=14840]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 23:59:29.562692428 (14h19m29.152037405s) [upload=14840]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 23:59:29.822497550 (14h19m29.411842537s) [upload=14840]: Done uploading segments... Aug 7 23:59:29.822514693 (14h19m29.41185968s) [upload=14840]: Upload finished. err: <nil> Aug 8 00:00:35.820554363 (14h20m35.40989935s) [upload=14856]: Starting upload Aug 8 00:00:35.820610611 (14h20m35.409955598s) [upload=14856]: Done starting upload Aug 8 00:00:35.820669503 (14h20m35.4100145s) [upload=14856]: Uploading segments... Aug 8 00:00:35.820742333 (14h20m35.41008732s) [upload=14856]: Got next segment. Inline: true Aug 8 00:00:35.820747282 (14h20m35.410092269s) [upload=14856]: Waiting for error group managing segments... Aug 8 00:00:35.820788371 (14h20m35.410133358s) [upload=14856]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 8 00:00:35.820793050 (14h20m35.410138037s) [upload=14856]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 8 00:00:35.820795485 (14h20m35.410140472s) [upload=14856]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 8 00:00:36.084321816 (14h20m35.673666803s) [upload=14856]: Done uploading segments... Aug 8 00:00:36.084338618 (14h20m35.673683615s) [upload=14856]: Upload finished. err: <nil> Aug 8 00:01:14.496466774 (14h21m14.085811781s) [upload=14868]: Starting upload Aug 8 00:01:14.496556174 (14h21m14.085901162s) [upload=14868]: Done starting upload Aug 8 00:01:14.496609206 (14h21m14.085954183s) [upload=14868]: Uploading segments... Aug 8 00:01:14.496639985 (14h21m14.085984972s) [upload=14868]: Got next segment. Inline: true Aug 8 00:01:14.496643592 (14h21m14.085988579s) [upload=14868]: Waiting for error group managing segments... Aug 8 00:01:14.496678638 (14h21m14.086023625s) [upload=14868]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 8 00:01:14.496682386 (14h21m14.086027373s) [upload=14868]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 8 00:01:14.496684681 (14h21m14.086029658s) [upload=14868]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 8 00:01:14.760681029 (14h21m14.350026016s) [upload=14868]: Done uploading segments... Aug 8 00:01:14.760702580 (14h21m14.350047567s) [upload=14868]: Upload finished. err: <nil> Aug 8 00:01:50.577049881 (14h21m50.166394858s) [upload=14878]: Starting upload Aug 8 00:01:50.577100538 (14h21m50.166445525s) [upload=14878]: Done starting upload Aug 8 00:01:50.577180091 (14h21m50.166525078s) [upload=14878]: Uploading segments... Aug 8 00:01:50.577228012 (14h21m50.166573s) [upload=14878]: Got next segment. Inline: true Aug 8 00:01:50.577231469 (14h21m50.166576456s) [upload=14878]: Waiting for error group managing segments... Aug 8 00:01:50.577270633 (14h21m50.166615621s) [upload=14878]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 8 00:01:50.577274721 (14h21m50.166619708s) [upload=14878]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 8 00:01:50.577276906 (14h21m50.166621893s) [upload=14878]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 8 00:01:50.863237339 (14h21m50.452582326s) [upload=14878]: Done uploading segments... Aug 8 00:01:50.863254973 (14h21m50.452599961s) [upload=14878]: Upload finished. err: <nil> Aug 8 00:02:41.474648957 (14h22m41.063993944s) [upload=14899]: Starting upload Aug 8 00:02:41.474716297 (14h22m41.064061274s) [upload=14899]: Done starting upload Aug 8 00:02:41.474771743 (14h22m41.06411673s) [upload=14899]: Uploading segments... Aug 8 00:02:41.474801099 (14h22m41.064146086s) [upload=14899]: Got next segment. Inline: true Aug 8 00:02:41.474804966 (14h22m41.064149953s) [upload=14899]: Waiting for error group managing segments... Aug 8 00:02:41.474841336 (14h22m41.064186323s) [upload=14899]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 8 00:02:41.474845214 (14h22m41.064190201s) [upload=14899]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 8 00:02:41.474847558 (14h22m41.064192535s) [upload=14899]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 8 00:03:40.835731182 (14h23m40.425076189s) [upload=14899]: Done uploading segments... Aug 8 00:03:40.835778212 (14h23m40.425123199s) [upload=14899]: Upload finished. err: metaclient: context deadline exceeded
Duration: 1.909653385s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)antiblip:
6 out of 60 is failed (threshold=3) [.XXX.XX....X................................................] duration:
6.07881047s
FAIL (2025-08-08 00:38:35 UTC)
orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded
Duration: 6.078845757s
core
uplink
us1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
6 out of 60 is failed (threshold=3) [...................X...X..........X.......X...X............X] duration:
6.723998254s
FAIL (2025-08-08 00:38:30 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/4M):: after 7dd7aed873cca87b7cd76d50ee66fe90f8c443893d6355bc8519d2fde31a068f != before c1a76dc0bec8105ec080db0072ab886792f3545b344045a49a190b10eaa4f282
======LOGS======
[upload=15508]: Uploading segments...
Aug 8 00:38:24.258804844 (14h58m23.848149831s) [upload=15508]: Got next segment. Inline: false
Aug 8 00:38:24.258847265 (14h58m23.848192252s) [upload=15508]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Aug 8 00:38:24.258872243 (14h58m23.84821723s) [upload=15508]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Aug 8 00:38:24.709304765 (14h58m24.298649752s) [seg_pos={0 0}, upload=15508]: Begin upload segment...
Aug 8 00:38:24.709363377 (14h58m24.298708364s) [seg_pos={0 0}, upload=15508]: Done begin upload segment.
Aug 8 00:38:24.709365892 (14h58m24.298710879s) [noise=true , address=104.36.82.47:4534 , piece_id=G56O4SR7DQ6YHATRCIMLOARHFQBILTCEUU2S35Y5NQKYIQX22X6A, seg_pos={0 0}, upload=15508]: Uploading piece...
Aug 8 00:38:24.709380991 (14h58m24.298725978s) [noise=true , address=104.36.82.39:4734 , piece_id=2ZZDYY5W5NHZ7OFXZ3X566EWP5F6GV32KNCIGTAEMGWDZI3ZHNNQ, seg_pos={0 0}, upload=15508]: Uploading piece...
Aug 8 00:38:24.709715681 (14h58m24.299060668s) [noise=true , address=104.36.82.61:4386 , piece_id=3YTRS2CUXJ3X3AVS6SWI234NQD7G6XPSV7ONZWT22WS23P6GEUOA, seg_pos={0 0}, upload=15508]: Uploading piece...
Aug 8 00:38:24.710014734 (14h58m24.299359721s) [noise=true , address=45.91.116.11:4142 , piece_id=GWGM46SJ4ZLCAW2TTFUX4R3GXX6C4QAMBN62BUKXED6YVROHQ6PQ, seg_pos={0 0}, upload=15508]: Uploading piece...
Aug 8 00:38:24.710261737 (14h58m24.299606724s) [noise=true , address=104.36.82.9:4038 , piece_id=IUFHRP22JUSOJLB66HKMEIK6OJJJPR3NLRA4JHTLQZDMD5IEKQEA, seg_pos={0 0}, upload=15508]: Uploading piece...
Aug 8 00:38:24.710536713 (14h58m24.2998817s) [noise=true , address=45.91.116.10:4113 , piece_id=QNB2CD3IINPCU6F3NCCPV7MNKY2GBV7ZBWUXY6UAYT7IIPJDGTCQ, seg_pos={0 0}, upload=15508]: Uploading piece...
Aug 8 00:38:24.710790879 (14h58m24.300135866s) [noise=true , address=64.49.13.26:4948 , piece_id=QSDEMGP65XIE3CLDSG6ASMKJCOBSSDZN2GWEVFJB3QQEGVCHTUIQ, seg_pos={0 0}, upload=15508]: Uploading piece...
Aug 8 00:38:24.711027322 (14h58m24.300372309s) [noise=true , address=104.36.82.37:4656 , piece_id=QGCIU7D4AG66MLGC7XA435DKIIBGLSLSWHQDVIZ75NFJKTONV4UQ, seg_pos={0 0}, upload=15508]: Uploading piece...
Aug 8 00:38:24.711279785 (14h58m24.300624772s) [noise=true , address=104.36.82.45:4931 , piece_id=IBPB2JL6J2JDMHYYNI3UVNEZ5DUOOTA2SKQKRGV255F6NTCWUDWQ, seg_pos={0 0}, upload=15508]: Uploading piece...
Aug 8 00:38:24.711373895 (14h58m24.300718882s) [noise=true , address=104.36.82.56:4256 , piece_id=3JG4YZGAK5JV6U3HTKEFSOVAKKLN5H66SBTSCUAL6KDS5YELSUWA, seg_pos={0 0}, upload=15508]: Uploading piece...
Aug 8 00:38:24.711652839 (14h58m24.300997826s) [noise=true , address=104.36.82.58:4288 , piece_id=XBQQBKEJMENF6WMGIFEFC4CP2FA4NQ2LW22RUBS4DWTGBOWNHO7A, seg_pos={0 0}, upload=15508]: Uploading piece...
Aug 8 00:38:24.711891356 (14h58m24.301236343s) [noise=true , address=104.36.82.2:4745 , piece_id=JYMY6EJ2ZG6XEJUUUJOIFNNMI4QMINNS4Q77QYXPVLTSC7BLQEXA, seg_pos={0 0}, upload=15508]: Uploading piece...
Aug 8 00:38:24.712120744 (14h58m24.301465732s) [noise=true , address=104.36.82.54:4166 , piece_id=WTEO3MZJMTHGA5PTY5CCQ42SKTJI43M43EULZGGUXAQVIY7C46IA, seg_pos={0 0}, upload=15508]: Uploading piece...
Aug 8 00:38:24.712325797 (14h58m24.301670784s) [noise=true , address=104.36.82.3:4781 , piece_id=A4ZYEHF6HFDP6CX5MALR26F6RA7BE5CST5GDDULJBJR3ALUJAFVQ, seg_pos={0 0}, upload=15508]: Uploading piece...
Aug 8 00:38:24.712565686 (14h58m24.301910673s) [noise=true , address=104.36.82.16:4320 , piece_id=2OP3TMF3FNQE2WTV6HEY6HVNO6KR2FCMPXRZJYCUI3QJIYIRLBOQ, seg_pos={0 0}, upload=15508]: Uploading piece...
Aug 8 00:38:24.712743346 (14h58m24.302088333s) [noise=true , address=45.91.116.14:4192 , piece_id=XIT5DARFYND632CV46LHR3UBVDQ6GAZ2DH3HQFUCTUFMW2Y4VUOQ, seg_pos={0 0}, upload=15508]: Uploading piece...
Aug 8 00:38:24.712959169 (14h58m24.302304156s) [noise=true , address=45.91.116.5:4324 , piece_id=TP3YSP3WXJKXA3DKPDXBRHTAKQBC6EXMZCRG7HBCELRWY6TFVXAA, seg_pos={0 0}, upload=15508]: Uploading piece...
Aug 8 00:38:24.713184781 (14h58m24.302529768s) [noise=true , address=104.36.82.52:4097 , piece_id=DFJXCTAXSGUPKENZVYE6G4EAQF5SE5FKGRHI62QHCCVHYRAQDA7Q, seg_pos={0 0}, upload=15508]: Uploading piece...
Aug 8 00:38:25.841668344 (14h58m25.431013331s) [upload=15508]: Next returned nil segment
Aug 8 00:38:25.841685085 (14h58m25.431030072s) [upload=15508]: Waiting for error group managing segments...
Aug 8 00:38:26.112321232 (14h58m25.701666219s) [noise=true , address=64.49.13.26:4948 , piece_id=QSDEMGP65XIE3CLDSG6ASMKJCOBSSDZN2GWEVFJB3QQEGVCHTUIQ, seg_pos={0 0}, upload=15508]: Done uploading piece. err: <nil>
Aug 8 00:38:26.667567146 (14h58m26.256912133s) [noise=true , address=104.36.82.56:4256 , piece_id=3JG4YZGAK5JV6U3HTKEFSOVAKKLN5H66SBTSCUAL6KDS5YELSUWA, seg_pos={0 0}, upload=15508]: Done uploading piece. err: <nil>
Aug 8 00:38:26.669941781 (14h58m26.259286768s) [noise=true , address=104.36.82.3:4781 , piece_id=A4ZYEHF6HFDP6CX5MALR26F6RA7BE5CST5GDDULJBJR3ALUJAFVQ, seg_pos={0 0}, upload=15508]: Done uploading piece. err: <nil>
Aug 8 00:38:26.699819391 (14h58m26.289164378s) [noise=true , address=45.91.116.10:4113 , piece_id=QNB2CD3IINPCU6F3NCCPV7MNKY2GBV7ZBWUXY6UAYT7IIPJDGTCQ, seg_pos={0 0}, upload=15508]: Done uploading piece. err: <nil>
Aug 8 00:38:26.738125452 (14h58m26.327470439s) [noise=true , address=104.36.82.2:4745 , piece_id=JYMY6EJ2ZG6XEJUUUJOIFNNMI4QMINNS4Q77QYXPVLTSC7BLQEXA, seg_pos={0 0}, upload=15508]: Done uploading piece. err: <nil>
Aug 8 00:38:26.745536716 (14h58m26.334881703s) [noise=true , address=104.36.82.39:4734 , piece_id=2ZZDYY5W5NHZ7OFXZ3X566EWP5F6GV32KNCIGTAEMGWDZI3ZHNNQ, seg_pos={0 0}, upload=15508]: Done uploading piece. err: <nil>
Aug 8 00:38:26.768528297 (14h58m26.357873294s) [noise=true , address=45.91.116.11:4142 , piece_id=GWGM46SJ4ZLCAW2TTFUX4R3GXX6C4QAMBN62BUKXED6YVROHQ6PQ, seg_pos={0 0}, upload=15508]: Done uploading piece. err: <nil>
Aug 8 00:38:26.818308437 (14h58m26.407653414s) [noise=true , address=104.36.82.37:4656 , piece_id=QGCIU7D4AG66MLGC7XA435DKIIBGLSLSWHQDVIZ75NFJKTONV4UQ, seg_pos={0 0}, upload=15508]: Done uploading piece. err: <nil>
Aug 8 00:38:26.832514054 (14h58m26.421859041s) [noise=true , address=45.91.116.14:4192 , piece_id=XIT5DARFYND632CV46LHR3UBVDQ6GAZ2DH3HQFUCTUFMW2Y4VUOQ, seg_pos={0 0}, upload=15508]: Done uploading piece. err: <nil>
Aug 8 00:38:26.869135760 (14h58m26.458480748s) [noise=true , address=104.36.82.54:4166 , piece_id=WTEO3MZJMTHGA5PTY5CCQ42SKTJI43M43EULZGGUXAQVIY7C46IA, seg_pos={0 0}, upload=15508]: Done uploading piece. err: <nil>
Aug 8 00:38:27.131612271 (14h58m26.720957258s) [noise=true , address=45.91.116.5:4324 , piece_id=TP3YSP3WXJKXA3DKPDXBRHTAKQBC6EXMZCRG7HBCELRWY6TFVXAA, seg_pos={0 0}, upload=15508]: Done uploading piece. err: <nil>
Aug 8 00:38:27.152099560 (14h58m26.741444547s) [noise=true , address=104.36.82.58:4288 , piece_id=XBQQBKEJMENF6WMGIFEFC4CP2FA4NQ2LW22RUBS4DWTGBOWNHO7A, seg_pos={0 0}, upload=15508]: Done uploading piece. err: <nil>
Aug 8 00:38:27.501729825 (14h58m27.091074813s) [noise=true , address=104.36.82.16:4320 , piece_id=2OP3TMF3FNQE2WTV6HEY6HVNO6KR2FCMPXRZJYCUI3QJIYIRLBOQ, seg_pos={0 0}, upload=15508]: Done uploading piece. err: <nil>
Aug 8 00:38:27.778376595 (14h58m27.367721582s) [noise=true , address=104.36.82.9:4038 , piece_id=IUFHRP22JUSOJLB66HKMEIK6OJJJPR3NLRA4JHTLQZDMD5IEKQEA, seg_pos={0 0}, upload=15508]: Done uploading piece. err: <nil>
Aug 8 00:38:27.778417313 (14h58m27.3677623s) [seg_pos={0 0}, upload=15508]: Segment reached optimal threshold of 14 pieces.
Aug 8 00:38:27.778642765 (14h58m27.367987752s) [noise=true , address=104.36.82.47:4534 , piece_id=G56O4SR7DQ6YHATRCIMLOARHFQBILTCEUU2S35Y5NQKYIQX22X6A, seg_pos={0 0}, upload=15508]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VDrNGcCnaKEmG5NN4nex1JoZAgdCEfX5CdsKbxytfLfKzPbr): protocol: expected piece hash; context canceled; EOF
Aug 8 00:38:27.778734049 (14h58m27.368079036s) [noise=true , address=104.36.82.61:4386 , piece_id=3YTRS2CUXJ3X3AVS6SWI234NQD7G6XPSV7ONZWT22WS23P6GEUOA, seg_pos={0 0}, upload=15508]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19aPjJnya2Q3stEhiedDSth3NHV8phLRj5yY79hZv81UgDitMv): protocol: write tcp 188.166.225.170:42102->104.36.82.61:4386: use of closed network connection; write tcp 188.166.225.170:42102->104.36.82.61:4386: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42102->104.36.82.61:4386: use of closed network connection
Aug 8 00:38:27.778843298 (14h58m27.368188285s) [noise=true , address=104.36.82.45:4931 , piece_id=IBPB2JL6J2JDMHYYNI3UVNEZ5DUOOTA2SKQKRGV255F6NTCWUDWQ, seg_pos={0 0}, upload=15508]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pCgoSfxjtXPw5jpbz9RJaxRiGoYPdPFdoMPtD19F1ZWY8UwPu): protocol: expected piece hash; context canceled; EOF
Aug 8 00:38:27.778889437 (14h58m27.368234424s) [noise=true , address=104.36.82.52:4097 , piece_id=DFJXCTAXSGUPKENZVYE6G4EAQF5SE5FKGRHI62QHCCVHYRAQDA7Q, seg_pos={0 0}, upload=15508]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1sMhTktqf2H4aZSyFV3m9Z6eMVxL2U52bvUHVHUh2gUCVp32AM): protocol: expected piece hash; context canceled; EOF
Aug 8 00:38:27.778906359 (14h58m27.368251346s) [seg_pos={0 0}, upload=15508]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Aug 8 00:38:27.778981493 (14h58m27.36832647s) [upload=15508]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Aug 8 00:38:27.778985781 (14h58m27.368330768s) [upload=15508]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Aug 8 00:38:28.078873078 (14h58m27.668218065s) [upload=15508]: Done uploading segments...
Aug 8 00:38:28.078891813 (14h58m27.6682368s) [upload=15508]: Upload finished. err: <nil>
Duration: 6.724025566s
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)antiblip:
5 out of 60 is failed (threshold=3) [...XXXXX....................................................] duration:
1.857808402s
FAIL (2025-08-08 00:38:27 UTC)
orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== m24.001855996s) [upload=14742]: Done starting upload Aug 7 23:53:24.412569290 (14h13m24.001914277s) [upload=14742]: Uploading segments... Aug 7 23:53:24.412638803 (14h13m24.001983831s) [upload=14742]: Got next segment. Inline: true Aug 7 23:53:24.412644354 (14h13m24.001989341s) [upload=14742]: Waiting for error group managing segments... Aug 7 23:53:24.412680903 (14h13m24.002025891s) [upload=14742]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 23:53:24.412685442 (14h13m24.002030429s) [upload=14742]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 23:53:24.412688137 (14h13m24.002033114s) [upload=14742]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 23:53:53.772057432 (14h13m53.361402419s) [upload=14742]: Done uploading segments... Aug 7 23:53:53.772119881 (14h13m53.361464908s) [upload=14742]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:54:30.496202349 (14h14m30.085547336s) [upload=14761]: Starting upload Aug 7 23:54:30.496270259 (14h14m30.085615246s) [upload=14761]: Done starting upload Aug 7 23:54:30.496341565 (14h14m30.085686552s) [upload=14761]: Uploading segments... Aug 7 23:54:30.496378957 (14h14m30.085723944s) [upload=14761]: Got next segment. Inline: true Aug 7 23:54:30.496383836 (14h14m30.085728823s) [upload=14761]: Waiting for error group managing segments... Aug 7 23:54:30.496441346 (14h14m30.085786333s) [upload=14761]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 23:54:30.496446426 (14h14m30.085791413s) [upload=14761]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 23:54:30.496449551 (14h14m30.085794539s) [upload=14761]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 23:54:59.847423574 (14h14m59.436768561s) [upload=14761]: Done uploading segments... Aug 7 23:54:59.847456066 (14h14m59.436801043s) [upload=14761]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:55:40.612588843 (14h15m40.20193383s) [upload=14781]: Starting upload Aug 7 23:55:40.612648929 (14h15m40.201993906s) [upload=14781]: Done starting upload Aug 7 23:55:40.612717029 (14h15m40.202062016s) [upload=14781]: Uploading segments... Aug 7 23:55:40.612750823 (14h15m40.20209581s) [upload=14781]: Got next segment. Inline: true Aug 7 23:55:40.612754510 (14h15m40.202099487s) [upload=14781]: Waiting for error group managing segments... Aug 7 23:55:40.612786883 (14h15m40.20213186s) [upload=14781]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 23:55:40.612790891 (14h15m40.202135868s) [upload=14781]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 23:55:40.612793075 (14h15m40.202138062s) [upload=14781]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 23:55:40.883469345 (14h15m40.472814333s) [upload=14781]: Done uploading segments... Aug 7 23:55:40.883486387 (14h15m40.472831375s) [upload=14781]: Upload finished. err: <nil> Aug 7 23:56:17.423712239 (14h16m17.013057226s) [upload=14789]: Starting upload Aug 7 23:56:17.423767895 (14h16m17.013112882s) [upload=14789]: Done starting upload Aug 7 23:56:17.423814835 (14h16m17.013159822s) [upload=14789]: Uploading segments... Aug 7 23:56:17.423847678 (14h16m17.013192665s) [upload=14789]: Got next segment. Inline: true Aug 7 23:56:17.423851335 (14h16m17.013196322s) [upload=14789]: Waiting for error group managing segments... Aug 7 23:56:17.423883377 (14h16m17.013228364s) [upload=14789]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 23:56:17.423887054 (14h16m17.013232041s) [upload=14789]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 23:56:17.423889258 (14h16m17.013234245s) [upload=14789]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 23:56:17.689327115 (14h16m17.278672102s) [upload=14789]: Done uploading segments... Aug 7 23:56:17.689345711 (14h16m17.278690688s) [upload=14789]: Upload finished. err: <nil> Aug 7 23:56:58.288741281 (14h16m57.878086268s) [upload=14801]: Starting upload Aug 7 23:56:58.288797098 (14h16m57.878142085s) [upload=14801]: Done starting upload Aug 7 23:56:58.288851010 (14h16m57.878195987s) [upload=14801]: Uploading segments... Aug 7 23:56:58.288882281 (14h16m57.878227268s) [upload=14801]: Got next segment. Inline: true Aug 7 23:56:58.288887911 (14h16m57.878232898s) [upload=14801]: Waiting for error group managing segments... Aug 7 23:56:58.288919792 (14h16m57.878264769s) [upload=14801]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 23:56:58.288923970 (14h16m57.878268957s) [upload=14801]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 23:56:58.288926485 (14h16m57.878271462s) [upload=14801]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 23:56:58.556529816 (14h16m58.145874803s) [upload=14801]: Done uploading segments... Aug 7 23:56:58.556544975 (14h16m58.145889962s) [upload=14801]: Upload finished. err: <nil> Aug 7 23:57:18.362794018 (14h17m17.952138995s) [upload=14806]: Starting upload Aug 7 23:57:18.362883679 (14h17m17.952228666s) [upload=14806]: Done starting upload Aug 7 23:57:18.362928104 (14h17m17.952273092s) [upload=14806]: Uploading segments... Aug 7 23:57:18.362957320 (14h17m17.952302307s) [upload=14806]: Got next segment. Inline: true Aug 7 23:57:18.362961077 (14h17m17.952306064s) [upload=14806]: Waiting for error group managing segments... Aug 7 23:57:18.363018407 (14h17m17.952363394s) [upload=14806]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 23:57:18.363022404 (14h17m17.952367392s) [upload=14806]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 23:57:18.363024739 (14h17m17.952369716s) [upload=14806]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 23:57:47.715297444 (14h17m47.304642431s) [upload=14806]: Done uploading segments... Aug 7 23:57:47.715410370 (14h17m47.304755357s) [upload=14806]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:57:53.375540080 (14h17m52.964885068s) [upload=14813]: Starting upload Aug 7 23:57:53.375600245 (14h17m52.964945232s) [upload=14813]: Done starting upload Aug 7 23:57:53.375654389 (14h17m52.964999376s) [upload=14813]: Uploading segments... Aug 7 23:57:53.375686622 (14h17m52.965031609s) [upload=14813]: Got next segment. Inline: true Aug 7 23:57:53.375690759 (14h17m52.965035746s) [upload=14813]: Waiting for error group managing segments... Aug 7 23:57:53.375725876 (14h17m52.965070863s) [upload=14813]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 23:57:53.375730134 (14h17m52.965075121s) [upload=14813]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 23:57:53.375734372 (14h17m52.965079359s) [upload=14813]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 23:58:22.726838464 (14h18m22.316183451s) [upload=14813]: Done uploading segments... Aug 7 23:58:22.726860075 (14h18m22.316205062s) [upload=14813]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:58:23.621870384 (14h18m23.211215371s) [upload=14822]: Starting upload Aug 7 23:58:23.621952000 (14h18m23.211296987s) [upload=14822]: Done starting upload Aug 7 23:58:23.622003359 (14h18m23.211348346s) [upload=14822]: Uploading segments... Aug 7 23:58:23.622043436 (14h18m23.211388413s) [upload=14822]: Got next segment. Inline: true Aug 7 23:58:23.622046772 (14h18m23.211391759s) [upload=14822]: Waiting for error group managing segments... Aug 7 23:58:23.622078573 (14h18m23.21142356s) [upload=14822]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 23:58:23.622082230 (14h18m23.211427217s) [upload=14822]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 23:58:23.622084534 (14h18m23.211429511s) [upload=14822]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 23:58:52.983677879 (14h18m52.573022876s) [upload=14822]: Done uploading segments... Aug 7 23:58:52.983713708 (14h18m52.573058685s) [upload=14822]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:59:11.446858470 (14h19m11.036203457s) [upload=14835]: Starting upload Aug 7 23:59:11.446906493 (14h19m11.03625148s) [upload=14835]: Done starting upload Aug 7 23:59:11.446947491 (14h19m11.036292479s) [upload=14835]: Uploading segments... Aug 7 23:59:11.446975475 (14h19m11.036320462s) [upload=14835]: Got next segment. Inline: true Aug 7 23:59:11.446978751 (14h19m11.036323738s) [upload=14835]: Waiting for error group managing segments... Aug 7 23:59:11.447008889 (14h19m11.036353876s) [upload=14835]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 23:59:11.447013638 (14h19m11.036358625s) [upload=14835]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 23:59:11.447016212 (14h19m11.0363612s) [upload=14835]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 23:59:40.799397635 (14h19m40.388742622s) [upload=14835]: Done uploading segments... Aug 7 23:59:40.799413266 (14h19m40.388758243s) [upload=14835]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:59:59.523080643 (14h19m59.11242563s) [upload=14847]: Starting upload Aug 7 23:59:59.523152089 (14h19m59.112497076s) [upload=14847]: Done starting upload Aug 7 23:59:59.523213727 (14h19m59.112558714s) [upload=14847]: Uploading segments... Aug 7 23:59:59.523251349 (14h19m59.112596336s) [upload=14847]: Got next segment. Inline: true Aug 7 23:59:59.523255287 (14h19m59.112600274s) [upload=14847]: Waiting for error group managing segments... Aug 7 23:59:59.523300013 (14h19m59.112645s) [upload=14847]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 23:59:59.523304251 (14h19m59.112649238s) [upload=14847]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 23:59:59.523307146 (14h19m59.112652133s) [upload=14847]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 8 00:00:28.877190143 (14h20m28.46653516s) [upload=14847]: Done uploading segments... Aug 8 00:00:28.877218477 (14h20m28.466563505s) [upload=14847]: Upload finished. err: metaclient: context deadline exceeded
Duration: 1.8578282s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [X...........................................................] duration:
5.55305061s
FAIL (2025-08-08 00:38:26 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.553070257s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [X....X......................................................] duration:
6.519471634s
FAIL (2025-08-08 00:38:12 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.519494467s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline)antiblip:
3 out of 60 is failed (threshold=3) [..X.X..X....................................................] duration:
1.854307545s
FAIL (2025-08-08 00:38:08 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1.854322112s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)antiblip:
6 out of 60 is failed (threshold=3) [..........X.XXXX....X.......................................] duration:
1.846852425s
FAIL (2025-08-08 00:38:06 UTC)
orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 9109253s) [upload=14753]: Starting upload Aug 7 23:54:04.489828037 (14h14m4.079173014s) [upload=14753]: Done starting upload Aug 7 23:54:04.489877612 (14h14m4.079222589s) [upload=14753]: Uploading segments... Aug 7 23:54:04.489903933 (14h14m4.07924892s) [upload=14753]: Got next segment. Inline: true Aug 7 23:54:04.489907469 (14h14m4.079252446s) [upload=14753]: Waiting for error group managing segments... Aug 7 23:54:04.489939651 (14h14m4.079284638s) [upload=14753]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 23:54:04.489943298 (14h14m4.079288285s) [upload=14753]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 23:54:04.489945472 (14h14m4.079290459s) [upload=14753]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 23:54:04.778726062 (14h14m4.368071049s) [upload=14753]: Done uploading segments... Aug 7 23:54:04.778744989 (14h14m4.368089976s) [upload=14753]: Upload finished. err: <nil> Aug 7 23:55:14.199719855 (14h15m13.789064892s) [upload=14770]: Starting upload Aug 7 23:55:14.199777645 (14h15m13.789122632s) [upload=14770]: Done starting upload Aug 7 23:55:14.199824415 (14h15m13.789169392s) [upload=14770]: Uploading segments... Aug 7 23:55:14.199852819 (14h15m13.789197806s) [upload=14770]: Got next segment. Inline: true Aug 7 23:55:14.199856436 (14h15m13.789201423s) [upload=14770]: Waiting for error group managing segments... Aug 7 23:55:14.199889419 (14h15m13.789234406s) [upload=14770]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 23:55:14.199893557 (14h15m13.789238544s) [upload=14770]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 23:55:14.199895951 (14h15m13.789240938s) [upload=14770]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 23:56:13.542516296 (14h16m13.131861283s) [upload=14770]: Done uploading segments... Aug 7 23:56:13.542536334 (14h16m13.131881321s) [upload=14770]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:56:14.428677736 (14h16m14.018022723s) [upload=14788]: Starting upload Aug 7 23:56:14.428730978 (14h16m14.018075965s) [upload=14788]: Done starting upload Aug 7 23:56:14.428789289 (14h16m14.018134276s) [upload=14788]: Uploading segments... Aug 7 23:56:14.428819026 (14h16m14.018164013s) [upload=14788]: Got next segment. Inline: true Aug 7 23:56:14.428822873 (14h16m14.01816785s) [upload=14788]: Waiting for error group managing segments... Aug 7 23:56:14.428855215 (14h16m14.018200202s) [upload=14788]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 23:56:14.428858562 (14h16m14.018203549s) [upload=14788]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 23:56:14.428860736 (14h16m14.018205723s) [upload=14788]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 23:57:13.781984834 (14h17m13.371329821s) [upload=14788]: Done uploading segments... Aug 7 23:57:13.782003370 (14h17m13.371348357s) [upload=14788]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:57:29.079359478 (14h17m28.668704466s) [upload=14808]: Starting upload Aug 7 23:57:29.079416147 (14h17m28.668761134s) [upload=14808]: Done starting upload Aug 7 23:57:29.079461845 (14h17m28.668806822s) [upload=14808]: Uploading segments... Aug 7 23:57:29.079488876 (14h17m28.668833863s) [upload=14808]: Got next segment. Inline: true Aug 7 23:57:29.079492263 (14h17m28.66883725s) [upload=14808]: Waiting for error group managing segments... Aug 7 23:57:29.079523392 (14h17m28.668868379s) [upload=14808]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 23:57:29.079527049 (14h17m28.668872036s) [upload=14808]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 23:57:29.079529484 (14h17m28.668874461s) [upload=14808]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 23:58:28.439377526 (14h18m28.028722513s) [upload=14808]: Done uploading segments... Aug 7 23:58:28.439395721 (14h18m28.028740708s) [upload=14808]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:58:29.333669502 (14h18m28.923014489s) [upload=14825]: Starting upload Aug 7 23:58:29.333742581 (14h18m28.923087568s) [upload=14825]: Done starting upload Aug 7 23:58:29.333835740 (14h18m28.923180727s) [upload=14825]: Uploading segments... Aug 7 23:58:29.333878000 (14h18m28.923222987s) [upload=14825]: Got next segment. Inline: true Aug 7 23:58:29.333881998 (14h18m28.923226975s) [upload=14825]: Waiting for error group managing segments... Aug 7 23:58:29.333911455 (14h18m28.923256442s) [upload=14825]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 23:58:29.333917496 (14h18m28.923262473s) [upload=14825]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 23:58:29.333919741 (14h18m28.923264728s) [upload=14825]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 23:59:28.692415040 (14h19m28.281760027s) [upload=14825]: Done uploading segments... Aug 7 23:59:28.692444777 (14h19m28.281789764s) [upload=14825]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:59:29.562470053 (14h19m29.15181504s) [upload=14840]: Starting upload Aug 7 23:59:29.562538985 (14h19m29.151883972s) [upload=14840]: Done starting upload Aug 7 23:59:29.562614629 (14h19m29.151959616s) [upload=14840]: Uploading segments... Aug 7 23:59:29.562650589 (14h19m29.151995566s) [upload=14840]: Got next segment. Inline: true Aug 7 23:59:29.562655498 (14h19m29.152000485s) [upload=14840]: Waiting for error group managing segments... Aug 7 23:59:29.562686507 (14h19m29.152031494s) [upload=14840]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 23:59:29.562690264 (14h19m29.152035241s) [upload=14840]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 23:59:29.562692428 (14h19m29.152037405s) [upload=14840]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 23:59:29.822497550 (14h19m29.411842537s) [upload=14840]: Done uploading segments... Aug 7 23:59:29.822514693 (14h19m29.41185968s) [upload=14840]: Upload finished. err: <nil> Aug 8 00:00:35.820554363 (14h20m35.40989935s) [upload=14856]: Starting upload Aug 8 00:00:35.820610611 (14h20m35.409955598s) [upload=14856]: Done starting upload Aug 8 00:00:35.820669503 (14h20m35.4100145s) [upload=14856]: Uploading segments... Aug 8 00:00:35.820742333 (14h20m35.41008732s) [upload=14856]: Got next segment. Inline: true Aug 8 00:00:35.820747282 (14h20m35.410092269s) [upload=14856]: Waiting for error group managing segments... Aug 8 00:00:35.820788371 (14h20m35.410133358s) [upload=14856]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 8 00:00:35.820793050 (14h20m35.410138037s) [upload=14856]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 8 00:00:35.820795485 (14h20m35.410140472s) [upload=14856]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 8 00:00:36.084321816 (14h20m35.673666803s) [upload=14856]: Done uploading segments... Aug 8 00:00:36.084338618 (14h20m35.673683615s) [upload=14856]: Upload finished. err: <nil> Aug 8 00:01:14.496466774 (14h21m14.085811781s) [upload=14868]: Starting upload Aug 8 00:01:14.496556174 (14h21m14.085901162s) [upload=14868]: Done starting upload Aug 8 00:01:14.496609206 (14h21m14.085954183s) [upload=14868]: Uploading segments... Aug 8 00:01:14.496639985 (14h21m14.085984972s) [upload=14868]: Got next segment. Inline: true Aug 8 00:01:14.496643592 (14h21m14.085988579s) [upload=14868]: Waiting for error group managing segments... Aug 8 00:01:14.496678638 (14h21m14.086023625s) [upload=14868]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 8 00:01:14.496682386 (14h21m14.086027373s) [upload=14868]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 8 00:01:14.496684681 (14h21m14.086029658s) [upload=14868]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 8 00:01:14.760681029 (14h21m14.350026016s) [upload=14868]: Done uploading segments... Aug 8 00:01:14.760702580 (14h21m14.350047567s) [upload=14868]: Upload finished. err: <nil> Aug 8 00:01:50.577049881 (14h21m50.166394858s) [upload=14878]: Starting upload Aug 8 00:01:50.577100538 (14h21m50.166445525s) [upload=14878]: Done starting upload Aug 8 00:01:50.577180091 (14h21m50.166525078s) [upload=14878]: Uploading segments... Aug 8 00:01:50.577228012 (14h21m50.166573s) [upload=14878]: Got next segment. Inline: true Aug 8 00:01:50.577231469 (14h21m50.166576456s) [upload=14878]: Waiting for error group managing segments... Aug 8 00:01:50.577270633 (14h21m50.166615621s) [upload=14878]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 8 00:01:50.577274721 (14h21m50.166619708s) [upload=14878]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 8 00:01:50.577276906 (14h21m50.166621893s) [upload=14878]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 8 00:01:50.863237339 (14h21m50.452582326s) [upload=14878]: Done uploading segments... Aug 8 00:01:50.863254973 (14h21m50.452599961s) [upload=14878]: Upload finished. err: <nil> Aug 8 00:02:41.474648957 (14h22m41.063993944s) [upload=14899]: Starting upload Aug 8 00:02:41.474716297 (14h22m41.064061274s) [upload=14899]: Done starting upload Aug 8 00:02:41.474771743 (14h22m41.06411673s) [upload=14899]: Uploading segments... Aug 8 00:02:41.474801099 (14h22m41.064146086s) [upload=14899]: Got next segment. Inline: true Aug 8 00:02:41.474804966 (14h22m41.064149953s) [upload=14899]: Waiting for error group managing segments... Aug 8 00:02:41.474841336 (14h22m41.064186323s) [upload=14899]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 8 00:02:41.474845214 (14h22m41.064190201s) [upload=14899]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 8 00:02:41.474847558 (14h22m41.064192535s) [upload=14899]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 8 00:03:40.835731182 (14h23m40.425076189s) [upload=14899]: Done uploading segments... Aug 8 00:03:40.835778212 (14h23m40.425123199s) [upload=14899]: Upload finished. err: metaclient: context deadline exceeded
Duration: 1.8468759s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [XX..........................................................] duration:
5.602178475s
FAIL (2025-08-08 00:38:06 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.602199125s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
8 out of 60 is failed (threshold=3) [XX..XXXX....XX..............................................] duration:
6.614488523s
FAIL (2025-08-08 00:38:03 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
3.700898256s) [upload=14855]: Done starting upload
Aug 8 00:00:34.111602513 (14h20m33.7009475s) [upload=14855]: Uploading segments...
Aug 8 00:00:34.111774201 (14h20m33.701119218s) [upload=14855]: Got next segment. Inline: false
Aug 8 00:00:34.111816783 (14h20m33.70116178s) [upload=14855]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Aug 8 00:00:34.111822464 (14h20m33.701167461s) [upload=14855]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Aug 8 00:01:03.465979471 (14h21m3.055324468s) [upload=14855]: Next segment err: context deadline exceeded
Aug 8 00:01:03.466148254 (14h21m3.055493251s) [upload=14855]: Done uploading segments...
Aug 8 00:01:03.466174945 (14h21m3.055519932s) [upload=14855]: Upload finished. err: context deadline exceeded
Aug 8 00:01:19.453976971 (14h21m19.043321958s) [upload=14872]: Starting upload
Aug 8 00:01:19.454042106 (14h21m19.043387123s) [upload=14872]: Done starting upload
Aug 8 00:01:19.454156104 (14h21m19.043501091s) [upload=14872]: Uploading segments...
Aug 8 00:01:19.454244503 (14h21m19.04358949s) [upload=14872]: Got next segment. Inline: false
Aug 8 00:01:19.454272567 (14h21m19.043617554s) [upload=14872]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Aug 8 00:01:19.454276825 (14h21m19.043621812s) [upload=14872]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Aug 8 00:01:39.582519114 (14h21m39.171864101s) [seg_pos={0 0}, upload=14872]: Begin upload segment...
Aug 8 00:01:39.582565882 (14h21m39.17191087s) [seg_pos={0 0}, upload=14872]: Done begin upload segment.
Aug 8 00:01:39.582581332 (14h21m39.17192632s) [noise=true , address=45.91.116.5:4299 , piece_id=HRKD77XA4SHZKSEW2WA3AROWPLHOWEXG5GEOWJCG7NTFOYYZ4DFA, seg_pos={0 0}, upload=14872]: Uploading piece...
Aug 8 00:01:39.582590870 (14h21m39.171935857s) [noise=true , address=104.36.82.2:4731 , piece_id=2BTN7TE7VVSPN5BUCDVKHWZPPATMNIJLXNDXUO46Z7CRB3QBCUEQ, seg_pos={0 0}, upload=14872]: Uploading piece...
Aug 8 00:01:39.582951871 (14h21m39.172296858s) [noise=true , address=45.91.116.12:4225 , piece_id=MUZ4UWUJSWGYKIO3BSTEOJYBBXSK4R2PURNDXHOYLTZ7VZ466IGA, seg_pos={0 0}, upload=14872]: Uploading piece...
Aug 8 00:01:39.583059888 (14h21m39.172404905s) [noise=true , address=104.36.82.13:4185 , piece_id=O26DWLA45AEOUZOW2CZTLJNPVRWB5KVDQW6FH2ANTKLUMYNWRVKA, seg_pos={0 0}, upload=14872]: Uploading piece...
Aug 8 00:01:39.583214905 (14h21m39.172559892s) [noise=true , address=45.91.116.8:4047 , piece_id=4LUGFRA2QPUK5LPKZGNTURXXSGY64AZU4Q3BHBAM3SPBZ7M2FFKQ, seg_pos={0 0}, upload=14872]: Uploading piece...
Aug 8 00:01:39.583427882 (14h21m39.172772869s) [noise=true , address=77.90.31.21:4877 , piece_id=MZ4WJDOFWH56JIAGXJZ7FOAPMNCHVEBUUM2G5IY2KKPRHPJH5MDA, seg_pos={0 0}, upload=14872]: Uploading piece...
Aug 8 00:01:39.583689032 (14h21m39.173034019s) [noise=true , address=77.90.31.11:4555 , piece_id=OCZBWUJHEVLMUD6BNADJ7BOSHHZPESQJOSKDZ3T2MUSQRFTNI7KA, seg_pos={0 0}, upload=14872]: Uploading piece...
Aug 8 00:01:39.583925344 (14h21m39.173270331s) [noise=true , address=45.91.116.9:4263 , piece_id=F4QDS3EOSLFFCEDU6N56AMMQR4GHPWZ6TIAMO5SGVIMVMFA47EQQ, seg_pos={0 0}, upload=14872]: Uploading piece...
Aug 8 00:01:39.584272317 (14h21m39.173617294s) [noise=true , address=104.36.82.8:4970 , piece_id=N52DS4AB7Z6VSFYWHN5MXPUS3AP7UKN2K7U5D5JZWETM7ZLROO7Q, seg_pos={0 0}, upload=14872]: Uploading piece...
Aug 8 00:01:39.584579135 (14h21m39.173924122s) [noise=true , address=77.90.31.18:4807 , piece_id=V7EZJW5PYJCWWNNIRAJCFTT42DQJY2C4RDHB5645TKOSXJJXE2WA, seg_pos={0 0}, upload=14872]: Uploading piece...
Aug 8 00:01:39.584852117 (14h21m39.174197104s) [noise=true , address=104.36.82.11:4115 , piece_id=S34RF36SERFPPSHBPYJ5V3FCR5ZE7RFIVEXBOTFRHPEEIUCHO2LQ, seg_pos={0 0}, upload=14872]: Uploading piece...
Aug 8 00:01:39.585105202 (14h21m39.174450179s) [noise=true , address=104.36.82.9:4005 , piece_id=ZYZ6VK5MCW7QE6N2GOYW2WMWDEWGONNK2Y7QBE7FPVPLY2OZGIMQ, seg_pos={0 0}, upload=14872]: Uploading piece...
Aug 8 00:01:39.585298762 (14h21m39.174643739s) [noise=true , address=45.91.116.4:4940 , piece_id=5S5JLBIGSBIVTPVJT5MIRAE42PRCIFWDNWLY7NQXSWPMHT5WSTRA, seg_pos={0 0}, upload=14872]: Uploading piece...
Aug 8 00:01:39.585506369 (14h21m39.174851346s) [noise=true , address=104.36.82.10:4079 , piece_id=262SP4DREL7FYDG6YOBFWDPWKYHZIQGDJKSX7XAIUJUTQWUIEEZQ, seg_pos={0 0}, upload=14872]: Uploading piece...
Aug 8 00:01:39.585763882 (14h21m39.175108869s) [noise=true , address=45.91.116.14:4190 , piece_id=FSO3UN2D24KPDVSWWPI3ROBXN7RN6D2HAZKOXEQ2JOW2M6RQ5MDA, seg_pos={0 0}, upload=14872]: Uploading piece...
Aug 8 00:01:39.586021575 (14h21m39.175366562s) [noise=true , address=104.36.82.5:4839 , piece_id=EHHLJG3INARPORGO6YVXU7QJAK23ADRVWBGNT5IKL4N6Y4ASM3XA, seg_pos={0 0}, upload=14872]: Uploading piece...
Aug 8 00:01:41.370103161 (14h21m40.959448138s) [upload=14872]: Next returned nil segment
Aug 8 00:01:41.370130243 (14h21m40.95947523s) [upload=14872]: Waiting for error group managing segments...
Aug 8 00:01:41.673384945 (14h21m41.262729922s) [noise=true , address=104.36.82.2:4731 , piece_id=2BTN7TE7VVSPN5BUCDVKHWZPPATMNIJLXNDXUO46Z7CRB3QBCUEQ, seg_pos={0 0}, upload=14872]: Done uploading piece. err: <nil>
Aug 8 00:01:41.739890780 (14h21m41.329235767s) [noise=true , address=104.36.82.9:4005 , piece_id=ZYZ6VK5MCW7QE6N2GOYW2WMWDEWGONNK2Y7QBE7FPVPLY2OZGIMQ, seg_pos={0 0}, upload=14872]: Done uploading piece. err: <nil>
Aug 8 00:01:41.743845828 (14h21m41.333190815s) [noise=true , address=104.36.82.8:4970 , piece_id=N52DS4AB7Z6VSFYWHN5MXPUS3AP7UKN2K7U5D5JZWETM7ZLROO7Q, seg_pos={0 0}, upload=14872]: Done uploading piece. err: <nil>
Aug 8 00:01:41.786086095 (14h21m41.375431072s) [noise=true , address=45.91.116.8:4047 , piece_id=4LUGFRA2QPUK5LPKZGNTURXXSGY64AZU4Q3BHBAM3SPBZ7M2FFKQ, seg_pos={0 0}, upload=14872]: Done uploading piece. err: <nil>
Aug 8 00:01:41.791525722 (14h21m41.380870699s) [noise=true , address=104.36.82.5:4839 , piece_id=EHHLJG3INARPORGO6YVXU7QJAK23ADRVWBGNT5IKL4N6Y4ASM3XA, seg_pos={0 0}, upload=14872]: Done uploading piece. err: <nil>
Aug 8 00:01:41.959657137 (14h21m41.549002124s) [noise=true , address=104.36.82.13:4185 , piece_id=O26DWLA45AEOUZOW2CZTLJNPVRWB5KVDQW6FH2ANTKLUMYNWRVKA, seg_pos={0 0}, upload=14872]: Done uploading piece. err: <nil>
Aug 8 00:01:42.000743686 (14h21m41.590088693s) [noise=true , address=45.91.116.12:4225 , piece_id=MUZ4UWUJSWGYKIO3BSTEOJYBBXSK4R2PURNDXHOYLTZ7VZ466IGA, seg_pos={0 0}, upload=14872]: Done uploading piece. err: <nil>
Aug 8 00:01:42.160166408 (14h21m41.749511395s) [noise=true , address=45.91.116.4:4940 , piece_id=5S5JLBIGSBIVTPVJT5MIRAE42PRCIFWDNWLY7NQXSWPMHT5WSTRA, seg_pos={0 0}, upload=14872]: Done uploading piece. err: <nil>
Aug 8 00:01:42.175399653 (14h21m41.764744641s) [noise=true , address=45.91.116.14:4190 , piece_id=FSO3UN2D24KPDVSWWPI3ROBXN7RN6D2HAZKOXEQ2JOW2M6RQ5MDA, seg_pos={0 0}, upload=14872]: Done uploading piece. err: <nil>
Aug 8 00:01:42.175880162 (14h21m41.765225139s) [noise=true , address=45.91.116.5:4299 , piece_id=HRKD77XA4SHZKSEW2WA3AROWPLHOWEXG5GEOWJCG7NTFOYYZ4DFA, seg_pos={0 0}, upload=14872]: Done uploading piece. err: <nil>
Aug 8 00:01:42.278394081 (14h21m41.867739068s) [noise=true , address=77.90.31.21:4877 , piece_id=MZ4WJDOFWH56JIAGXJZ7FOAPMNCHVEBUUM2G5IY2KKPRHPJH5MDA, seg_pos={0 0}, upload=14872]: Done uploading piece. err: <nil>
Aug 8 00:01:42.293663936 (14h21m41.883008933s) [noise=true , address=77.90.31.18:4807 , piece_id=V7EZJW5PYJCWWNNIRAJCFTT42DQJY2C4RDHB5645TKOSXJJXE2WA, seg_pos={0 0}, upload=14872]: Done uploading piece. err: <nil>
Aug 8 00:01:42.309113927 (14h21m41.898458904s) [noise=true , address=45.91.116.9:4263 , piece_id=F4QDS3EOSLFFCEDU6N56AMMQR4GHPWZ6TIAMO5SGVIMVMFA47EQQ, seg_pos={0 0}, upload=14872]: Done uploading piece. err: <nil>
Aug 8 00:01:42.309137942 (14h21m41.898482919s) [seg_pos={0 0}, upload=14872]: Segment reached optimal threshold of 13 pieces.
Aug 8 00:01:42.309315973 (14h21m41.89866099s) [noise=true , address=77.90.31.11:4555 , piece_id=OCZBWUJHEVLMUD6BNADJ7BOSHHZPESQJOSKDZ3T2MUSQRFTNI7KA, seg_pos={0 0}, upload=14872]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ): protocol: expected piece hash; context canceled; EOF
Aug 8 00:01:42.309402328 (14h21m41.898747305s) [noise=true , address=104.36.82.10:4079 , piece_id=262SP4DREL7FYDG6YOBFWDPWKYHZIQGDJKSX7XAIUJUTQWUIEEZQ, seg_pos={0 0}, upload=14872]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud): protocol: expected piece hash; context canceled; EOF
Aug 8 00:01:42.309425222 (14h21m41.898770229s) [noise=true , address=104.36.82.11:4115 , piece_id=S34RF36SERFPPSHBPYJ5V3FCR5ZE7RFIVEXBOTFRHPEEIUCHO2LQ, seg_pos={0 0}, upload=14872]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR): protocol: write tcp 188.166.225.170:42778->104.36.82.11:4115: use of closed network connection; write tcp 188.166.225.170:42778->104.36.82.11:4115: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42778->104.36.82.11:4115: use of closed network connection
Aug 8 00:01:42.309450360 (14h21m41.898795347s) [seg_pos={0 0}, upload=14872]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Aug 8 00:01:42.309540663 (14h21m41.89888565s) [upload=14872]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Aug 8 00:01:42.309548688 (14h21m41.898893735s) [upload=14872]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Aug 8 00:01:57.644545293 (14h21m57.2338903s) [upload=14872]: Done uploading segments...
Aug 8 00:01:57.644581281 (14h21m57.233926288s) [upload=14872]: Upload finished. err: metaclient: context deadline exceeded
Duration: 6.614509201s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)antiblip:
6 out of 60 is failed (threshold=3) [..XXX.XX....X...............................................] duration:
5.779843781s
FAIL (2025-08-08 00:38:01 UTC)
orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded
Duration: 5.77986355s
core
uplink
us1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)antiblip:
1 out of 60 is failed (threshold=3) [...............................X............................] duration:
1.917930383s
FAIL (2025-08-08 00:37:59 UTC)
orbiter warning:
operation: DownloadObject
======ERROR======
uplink: object not found ("orbiter_us_0/1K")
======LOGS======
=14964]: Done uploading segments...
Aug 8 00:06:41.669168196 (14h26m41.258513183s) [upload=14964]: Upload finished. err: <nil>
Aug 8 00:07:06.258431851 (14h27m5.847776838s) [upload=14976]: Starting upload
Aug 8 00:07:06.258508298 (14h27m5.847853285s) [upload=14976]: Done starting upload
Aug 8 00:07:06.258574986 (14h27m5.847919983s) [upload=14976]: Uploading segments...
Aug 8 00:07:06.258618599 (14h27m5.847963616s) [upload=14976]: Got next segment. Inline: true
Aug 8 00:07:06.258624600 (14h27m5.847969587s) [upload=14976]: Waiting for error group managing segments...
Aug 8 00:07:06.258705555 (14h27m5.848050542s) [upload=14976]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Aug 8 00:07:06.258712268 (14h27m5.848057296s) [upload=14976]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline
Aug 8 00:07:06.258716226 (14h27m5.848061253s) [upload=14976]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Aug 8 00:07:06.522670976 (14h27m6.112015963s) [upload=14976]: Done uploading segments...
Aug 8 00:07:06.522717766 (14h27m6.112062773s) [upload=14976]: Upload finished. err: <nil>
Aug 8 00:07:36.371369594 (14h27m35.960714581s) [upload=14980]: Starting upload
Aug 8 00:07:36.371417074 (14h27m35.960762061s) [upload=14980]: Done starting upload
Aug 8 00:07:36.371466890 (14h27m35.960811877s) [upload=14980]: Uploading segments...
Aug 8 00:07:36.371508951 (14h27m35.960853938s) [upload=14980]: Got next segment. Inline: true
Aug 8 00:07:36.371512888 (14h27m35.960857875s) [upload=14980]: Waiting for error group managing segments...
Aug 8 00:07:36.371543446 (14h27m35.960888433s) [upload=14980]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Aug 8 00:07:36.371547434 (14h27m35.960892421s) [upload=14980]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline
Aug 8 00:07:36.371549708 (14h27m35.960894695s) [upload=14980]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Aug 8 00:07:36.789946570 (14h27m36.379291557s) [upload=14980]: Done uploading segments...
Aug 8 00:07:36.789964876 (14h27m36.379309863s) [upload=14980]: Upload finished. err: <nil>
Aug 8 00:08:16.866388064 (14h28m16.455733051s) [upload=14994]: Starting upload
Aug 8 00:08:16.866430886 (14h28m16.455775873s) [upload=14994]: Done starting upload
Aug 8 00:08:16.866496882 (14h28m16.455841869s) [upload=14994]: Uploading segments...
Aug 8 00:08:16.866525727 (14h28m16.455870714s) [upload=14994]: Got next segment. Inline: true
Aug 8 00:08:16.866529494 (14h28m16.455874481s) [upload=14994]: Waiting for error group managing segments...
Aug 8 00:08:16.866570884 (14h28m16.455915871s) [upload=14994]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Aug 8 00:08:16.866575212 (14h28m16.455920199s) [upload=14994]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline
Aug 8 00:08:16.866577626 (14h28m16.455922613s) [upload=14994]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Aug 8 00:08:17.138322091 (14h28m16.727667078s) [upload=14994]: Done uploading segments...
Aug 8 00:08:17.138338402 (14h28m16.727683389s) [upload=14994]: Upload finished. err: <nil>
Aug 8 00:08:57.633625328 (14h28m57.222970315s) [upload=15006]: Starting upload
Aug 8 00:08:57.633689390 (14h28m57.223034378s) [upload=15006]: Done starting upload
Aug 8 00:08:57.633744837 (14h28m57.223089824s) [upload=15006]: Uploading segments...
Aug 8 00:08:57.633775525 (14h28m57.223120512s) [upload=15006]: Got next segment. Inline: true
Aug 8 00:08:57.633779222 (14h28m57.223124199s) [upload=15006]: Waiting for error group managing segments...
Aug 8 00:08:57.633951743 (14h28m57.22329672s) [upload=15006]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Aug 8 00:08:57.633956161 (14h28m57.223301148s) [upload=15006]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline
Aug 8 00:08:57.633958886 (14h28m57.223303873s) [upload=15006]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Aug 8 00:08:57.920842105 (14h28m57.510187102s) [upload=15006]: Done uploading segments...
Aug 8 00:08:57.920865330 (14h28m57.510210327s) [upload=15006]: Upload finished. err: <nil>
Aug 8 00:09:43.341371442 (14h29m42.930716449s) [upload=15015]: Starting upload
Aug 8 00:09:43.341457767 (14h29m42.930802774s) [upload=15015]: Done starting upload
Aug 8 00:09:43.341511951 (14h29m42.930856938s) [upload=15015]: Uploading segments...
Aug 8 00:09:43.341579249 (14h29m42.930924236s) [upload=15015]: Got next segment. Inline: true
Aug 8 00:09:43.341585642 (14h29m42.930930639s) [upload=15015]: Waiting for error group managing segments...
Aug 8 00:09:43.341640437 (14h29m42.930985454s) [upload=15015]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Aug 8 00:09:43.341648422 (14h29m42.930993469s) [upload=15015]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline
Aug 8 00:09:43.341653391 (14h29m42.930998408s) [upload=15015]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Aug 8 00:09:43.627931533 (14h29m43.21727652s) [upload=15015]: Done uploading segments...
Aug 8 00:09:43.627948425 (14h29m43.217293402s) [upload=15015]: Upload finished. err: <nil>
Aug 8 00:10:45.166177580 (14h30m44.755522568s) [upload=15031]: Starting upload
Aug 8 00:10:45.166220523 (14h30m44.75556551s) [upload=15031]: Done starting upload
Aug 8 00:10:45.166273034 (14h30m44.755618011s) [upload=15031]: Uploading segments...
Aug 8 00:10:45.166297991 (14h30m44.755642979s) [upload=15031]: Got next segment. Inline: true
Aug 8 00:10:45.166301568 (14h30m44.755646555s) [upload=15031]: Waiting for error group managing segments...
Aug 8 00:10:45.166332277 (14h30m44.755677264s) [upload=15031]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Aug 8 00:10:45.166335783 (14h30m44.75568077s) [upload=15031]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline
Aug 8 00:10:45.166337998 (14h30m44.755682985s) [upload=15031]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Aug 8 00:10:45.444515471 (14h30m45.033860458s) [upload=15031]: Done uploading segments...
Aug 8 00:10:45.444533305 (14h30m45.033878282s) [upload=15031]: Upload finished. err: <nil>
Aug 8 00:11:54.979425886 (14h31m54.568770863s) [upload=15052]: Starting upload
Aug 8 00:11:54.979471082 (14h31m54.568816069s) [upload=15052]: Done starting upload
Aug 8 00:11:54.979517260 (14h31m54.568862237s) [upload=15052]: Uploading segments...
Aug 8 00:11:54.979549422 (14h31m54.568894399s) [upload=15052]: Got next segment. Inline: true
Aug 8 00:11:54.979553008 (14h31m54.568897995s) [upload=15052]: Waiting for error group managing segments...
Aug 8 00:11:54.979584328 (14h31m54.568929315s) [upload=15052]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Aug 8 00:11:54.979588356 (14h31m54.568933333s) [upload=15052]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline
Aug 8 00:11:54.979590661 (14h31m54.568935648s) [upload=15052]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Aug 8 00:11:55.340902393 (14h31m54.93024738s) [upload=15052]: Done uploading segments...
Aug 8 00:11:55.340920908 (14h31m54.930265895s) [upload=15052]: Upload finished. err: <nil>
Aug 8 00:12:44.503574128 (14h32m44.092919115s) [upload=15065]: Starting upload
Aug 8 00:12:44.503634083 (14h32m44.09297907s) [upload=15065]: Done starting upload
Aug 8 00:12:44.503710068 (14h32m44.093055055s) [upload=15065]: Uploading segments...
Aug 8 00:12:44.503762669 (14h32m44.093107656s) [upload=15065]: Got next segment. Inline: true
Aug 8 00:12:44.503767698 (14h32m44.093112685s) [upload=15065]: Waiting for error group managing segments...
Aug 8 00:12:44.503805531 (14h32m44.093150508s) [upload=15065]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Aug 8 00:12:44.503809379 (14h32m44.093154366s) [upload=15065]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline
Aug 8 00:12:44.503811633 (14h32m44.09315662s) [upload=15065]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Aug 8 00:12:44.792734473 (14h32m44.38207946s) [upload=15065]: Done uploading segments...
Aug 8 00:12:44.792752288 (14h32m44.382097275s) [upload=15065]: Upload finished. err: <nil>
Aug 8 00:13:19.961146705 (14h33m19.550491693s) [upload=15079]: Starting upload
Aug 8 00:13:19.961187835 (14h33m19.550532822s) [upload=15079]: Done starting upload
Aug 8 00:13:19.961236618 (14h33m19.550581605s) [upload=15079]: Uploading segments...
Aug 8 00:13:19.961268168 (14h33m19.550613145s) [upload=15079]: Got next segment. Inline: true
Aug 8 00:13:19.961273849 (14h33m19.550618836s) [upload=15079]: Waiting for error group managing segments...
Aug 8 00:13:19.961304548 (14h33m19.550649525s) [upload=15079]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Aug 8 00:13:19.961308345 (14h33m19.550653332s) [upload=15079]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline
Aug 8 00:13:19.961310910 (14h33m19.550655887s) [upload=15079]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Aug 8 00:13:20.385237160 (14h33m19.974582137s) [upload=15079]: Done uploading segments...
Aug 8 00:13:20.385255485 (14h33m19.974600472s) [upload=15079]: Upload finished. err: <nil>
Aug 8 00:14:06.785741109 (14h34m6.375086096s) [upload=15091]: Starting upload
Aug 8 00:14:06.785899151 (14h34m6.375244138s) [upload=15091]: Done starting upload
Aug 8 00:14:06.786002628 (14h34m6.375347615s) [upload=15091]: Uploading segments...
Aug 8 00:14:06.786034078 (14h34m6.375379065s) [upload=15091]: Got next segment. Inline: true
Aug 8 00:14:06.786037786 (14h34m6.375382773s) [upload=15091]: Waiting for error group managing segments...
Aug 8 00:14:06.786073094 (14h34m6.375418081s) [upload=15091]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Aug 8 00:14:06.786078945 (14h34m6.375423922s) [upload=15091]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline
Aug 8 00:14:06.786081339 (14h34m6.375426326s) [upload=15091]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Aug 8 00:14:07.092788097 (14h34m6.682133084s) [upload=15091]: Done uploading segments...
Aug 8 00:14:07.092803166 (14h34m6.682148153s) [upload=15091]: Upload finished. err: <nil>
Duration: 1.91795927s
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
4 out of 60 is failed (threshold=3) [........X.XX...X............................................] duration:
5.725097147s
FAIL (2025-08-08 00:37:56 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: encryption: context deadline exceeded
======LOGS======
ZYSDFBHFTDOKPS73M57QA, seg_pos={0 0}, upload=14870]: Uploading piece...
Aug 8 00:01:16.246137011 (14h21m15.835481998s) [noise=true , address=104.36.82.9:4024 , piece_id=FM7MVIPG7C264O6Q64RUFOKZ2E5NMTIPWSIVMH5SWZUSCMFI7UGQ, seg_pos={0 0}, upload=14870]: Uploading piece...
Aug 8 00:01:16.246560380 (14h21m15.835905367s) [noise=true , address=64.49.13.169:4705 , piece_id=3MVXPRNKBBUSYAH2E7PCIWXDCUFQQ547IPOYIDLQ3T7QDYPBJA7A, seg_pos={0 0}, upload=14870]: Uploading piece...
Aug 8 00:01:16.247408974 (14h21m15.836753961s) [noise=true , address=45.91.116.5:4302 , piece_id=6RMPHN4LYSZVM6XKAUGSU7GY3TIKBGLKK4KNFMZQ2FMVEEVWV2LA, seg_pos={0 0}, upload=14870]: Uploading piece...
Aug 8 00:01:16.247678410 (14h21m15.837023397s) [noise=true , address=79.127.219.51:4181 , piece_id=4LYG6AL2BWD3UYQQAUUGZNJ7HUFXSHZ2TULWJOYSWTJLNVGHVKLQ, seg_pos={0 0}, upload=14870]: Uploading piece...
Aug 8 00:01:16.247919882 (14h21m15.837264869s) [noise=true , address=104.36.82.22:4512 , piece_id=7VOWVUPU3QVXIKML65NHX3NL7AC26QSDGHSSCRLUXIVZRNUPPKRA, seg_pos={0 0}, upload=14870]: Uploading piece...
Aug 8 00:01:16.247943888 (14h21m15.837288875s) [noise=true , address=104.36.82.20:4466 , piece_id=PLG7Z2WTYKUNB347NPT4ESIDK2NHCP66NPZJOXYL4A6L5OMFQIOQ, seg_pos={0 0}, upload=14870]: Uploading piece...
Aug 8 00:01:16.248293266 (14h21m15.837638253s) [noise=true , address=104.36.82.11:4143 , piece_id=E4AW52SZYZVRWPU4L56INK3IRNG7UNO5RRIRJANLMEPZQVDUHCEQ, seg_pos={0 0}, upload=14870]: Uploading piece...
Aug 8 00:01:16.248420540 (14h21m15.837765527s) [noise=true , address=64.49.13.139:4471 , piece_id=FPVXYIRNZXODU7N5RPCDOVVUECXWRTXCLINEXMOJTKNHF6YN46YQ, seg_pos={0 0}, upload=14870]: Uploading piece...
Aug 8 00:01:16.248557311 (14h21m15.837902288s) [noise=true , address=104.36.82.37:4659 , piece_id=627LA2AVL53NXFMDWSLGBO5GUKDAIGGSOPBFRPYIPYC5EAQC6P7Q, seg_pos={0 0}, upload=14870]: Uploading piece...
Aug 8 00:01:16.248758506 (14h21m15.838103493s) [noise=true , address=104.36.82.58:4275 , piece_id=VBV72SK36XIHMWSZTJJE4MZPLD5VZ6Z6UPBHK4A2ZI2C3O3RCVXA, seg_pos={0 0}, upload=14870]: Uploading piece...
Aug 8 00:01:16.248897792 (14h21m15.838242779s) [noise=true , address=104.36.82.55:4209 , piece_id=PJ4RK52LARBDG6WEVKXPZPX3RKJVIRSDCZY3B6CVCIUZO6BTWLBQ, seg_pos={0 0}, upload=14870]: Uploading piece...
Aug 8 00:01:16.248989629 (14h21m15.838334616s) [noise=true , address=104.36.82.43:4855 , piece_id=MJYC5ZGE6P72UYNYDRMH3BAZ6KIY57SA3EKJR54W6HPZNO3AE4IA, seg_pos={0 0}, upload=14870]: Uploading piece...
Aug 8 00:01:16.249186575 (14h21m15.838531552s) [noise=true , address=104.36.82.46:4487 , piece_id=6BZYNRXKBHFL3RQTGN3ZN4RV2O4USZ24GGCVSKFZWNS6N32L3NGQ, seg_pos={0 0}, upload=14870]: Uploading piece...
Aug 8 00:01:16.249273772 (14h21m15.838618759s) [noise=true , address=104.36.82.7:4920 , piece_id=BN5NCMLEGDVRIHO6YOWM5FNPCGFYC3BDKLOU7DIP6K6DYZXJ5A5A, seg_pos={0 0}, upload=14870]: Uploading piece...
Aug 8 00:01:16.249459598 (14h21m15.838804585s) [noise=true , address=77.90.31.14:4672 , piece_id=3ZYE5SLRNLY2ALXRR4QGCZT4SQ37U7FIAUOCVT63HQSBKPKV4UGQ, seg_pos={0 0}, upload=14870]: Uploading piece...
Aug 8 00:01:16.249717240 (14h21m15.839062227s) [noise=true , address=104.36.82.5:4844 , piece_id=REDGG4W6TJYEPEEPR65O5AVQDU2QMBNRA6HQQRUJA6JP7IS5T7YQ, seg_pos={0 0}, upload=14870]: Uploading piece...
Aug 8 00:01:17.413257645 (14h21m17.002602632s) [upload=14870]: Next returned nil segment
Aug 8 00:01:17.413273324 (14h21m17.002618311s) [upload=14870]: Waiting for error group managing segments...
Aug 8 00:01:17.710897294 (14h21m17.300242281s) [noise=true , address=64.49.13.169:4705 , piece_id=3MVXPRNKBBUSYAH2E7PCIWXDCUFQQ547IPOYIDLQ3T7QDYPBJA7A, seg_pos={0 0}, upload=14870]: Done uploading piece. err: <nil>
Aug 8 00:01:17.715058896 (14h21m17.304403883s) [noise=true , address=64.49.13.139:4471 , piece_id=FPVXYIRNZXODU7N5RPCDOVVUECXWRTXCLINEXMOJTKNHF6YN46YQ, seg_pos={0 0}, upload=14870]: Done uploading piece. err: <nil>
Aug 8 00:01:18.126218161 (14h21m17.715563148s) [noise=true , address=79.127.201.215:4356 , piece_id=VWNEIHNI4KPPF4QLW36INIWTUI2CD6MZYSDFBHFTDOKPS73M57QA, seg_pos={0 0}, upload=14870]: Done uploading piece. err: <nil>
Aug 8 00:01:18.139272367 (14h21m17.728617354s) [noise=true , address=79.127.219.51:4181 , piece_id=4LYG6AL2BWD3UYQQAUUGZNJ7HUFXSHZ2TULWJOYSWTJLNVGHVKLQ, seg_pos={0 0}, upload=14870]: Done uploading piece. err: <nil>
Aug 8 00:01:18.221269031 (14h21m17.810614018s) [noise=true , address=104.36.82.43:4855 , piece_id=MJYC5ZGE6P72UYNYDRMH3BAZ6KIY57SA3EKJR54W6HPZNO3AE4IA, seg_pos={0 0}, upload=14870]: Done uploading piece. err: <nil>
Aug 8 00:01:18.257350144 (14h21m17.846695121s) [noise=true , address=104.36.82.37:4659 , piece_id=627LA2AVL53NXFMDWSLGBO5GUKDAIGGSOPBFRPYIPYC5EAQC6P7Q, seg_pos={0 0}, upload=14870]: Done uploading piece. err: <nil>
Aug 8 00:01:18.258435089 (14h21m17.847780076s) [noise=true , address=104.36.82.44:4876 , piece_id=BA2G42OXQD3NLONUVD475S2BNC7T37T5PHPE3K3E5HJAJP4W5MKQ, seg_pos={0 0}, upload=14870]: Done uploading piece. err: <nil>
Aug 8 00:01:18.287010923 (14h21m17.87635591s) [noise=true , address=104.36.82.7:4920 , piece_id=BN5NCMLEGDVRIHO6YOWM5FNPCGFYC3BDKLOU7DIP6K6DYZXJ5A5A, seg_pos={0 0}, upload=14870]: Done uploading piece. err: <nil>
Aug 8 00:01:18.348793001 (14h21m17.938137988s) [noise=true , address=104.36.82.11:4143 , piece_id=E4AW52SZYZVRWPU4L56INK3IRNG7UNO5RRIRJANLMEPZQVDUHCEQ, seg_pos={0 0}, upload=14870]: Done uploading piece. err: <nil>
Aug 8 00:01:18.378129459 (14h21m17.967474436s) [noise=true , address=104.36.82.58:4275 , piece_id=VBV72SK36XIHMWSZTJJE4MZPLD5VZ6Z6UPBHK4A2ZI2C3O3RCVXA, seg_pos={0 0}, upload=14870]: Done uploading piece. err: <nil>
Aug 8 00:01:18.386414901 (14h21m17.975759888s) [noise=true , address=104.36.82.46:4487 , piece_id=6BZYNRXKBHFL3RQTGN3ZN4RV2O4USZ24GGCVSKFZWNS6N32L3NGQ, seg_pos={0 0}, upload=14870]: Done uploading piece. err: <nil>
Aug 8 00:01:18.516150823 (14h21m18.10549581s) [noise=true , address=104.36.82.9:4024 , piece_id=FM7MVIPG7C264O6Q64RUFOKZ2E5NMTIPWSIVMH5SWZUSCMFI7UGQ, seg_pos={0 0}, upload=14870]: Done uploading piece. err: <nil>
Aug 8 00:01:18.626647204 (14h21m18.215992191s) [noise=true , address=77.90.31.14:4672 , piece_id=3ZYE5SLRNLY2ALXRR4QGCZT4SQ37U7FIAUOCVT63HQSBKPKV4UGQ, seg_pos={0 0}, upload=14870]: Done uploading piece. err: <nil>
Aug 8 00:01:19.042108811 (14h21m18.631453798s) [noise=true , address=104.36.82.22:4512 , piece_id=7VOWVUPU3QVXIKML65NHX3NL7AC26QSDGHSSCRLUXIVZRNUPPKRA, seg_pos={0 0}, upload=14870]: Done uploading piece. err: <nil>
Aug 8 00:01:19.042136654 (14h21m18.631481641s) [seg_pos={0 0}, upload=14870]: Segment reached optimal threshold of 14 pieces.
Aug 8 00:01:19.042375541 (14h21m18.631720518s) [noise=true , address=45.91.116.5:4302 , piece_id=6RMPHN4LYSZVM6XKAUGSU7GY3TIKBGLKK4KNFMZQ2FMVEEVWV2LA, seg_pos={0 0}, upload=14870]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pJWhSLza8WaMuvvDwH4nae57UxQmfCdDvAkQ3v25MgPYHoZXx): protocol: expected piece hash; context canceled; EOF
Aug 8 00:01:19.042485912 (14h21m18.6318309s) [noise=true , address=104.36.82.55:4209 , piece_id=PJ4RK52LARBDG6WEVKXPZPX3RKJVIRSDCZY3B6CVCIUZO6BTWLBQ, seg_pos={0 0}, upload=14870]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12EmE2BgEZTG1LTsZ5p9ghqDSfg5ZQATt31Tqg5jZ5KQE2UjDnR): protocol: rpc: dial tcp 104.36.82.55:4209: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 8 00:01:19.042532361 (14h21m18.631877348s) [noise=true , address=104.36.82.20:4466 , piece_id=PLG7Z2WTYKUNB347NPT4ESIDK2NHCP66NPZJOXYL4A6L5OMFQIOQ, seg_pos={0 0}, upload=14870]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SqNLPRWY7fJRS6F9h4B4qVFv6U5jyWkCy3Um7jfjEdCShEjsD): protocol: write tcp 188.166.225.170:37056->104.36.82.20:4466: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 8 00:01:19.042616462 (14h21m18.631961449s) [noise=true , address=104.36.82.5:4844 , piece_id=REDGG4W6TJYEPEEPR65O5AVQDU2QMBNRA6HQQRUJA6JP7IS5T7YQ, seg_pos={0 0}, upload=14870]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12iovpNdbidkiv9mEHaAPjBubGmoqr4sUJWm8Cb64LzS6wuwLWa): protocol: rpc: dial tcp 104.36.82.5:4844: operation was canceled; conn closed: op; piecestore: piecestore close: context canceled
Aug 8 00:01:19.042628876 (14h21m18.631973863s) [seg_pos={0 0}, upload=14870]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Aug 8 00:01:19.042699029 (14h21m18.632044017s) [upload=14870]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Aug 8 00:01:19.042703128 (14h21m18.632048115s) [upload=14870]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Aug 8 00:01:19.344335316 (14h21m18.933680293s) [upload=14870]: Done uploading segments...
Aug 8 00:01:19.344353089 (14h21m18.933698077s) [upload=14870]: Upload finished. err: <nil>
Aug 8 00:02:30.778225454 (14h22m30.367570441s) [upload=14892]: Starting upload
Aug 8 00:02:30.778283114 (14h22m30.367628151s) [upload=14892]: Done starting upload
Aug 8 00:02:30.778411841 (14h22m30.367756828s) [upload=14892]: Uploading segments...
Aug 8 00:02:30.778539616 (14h22m30.367884623s) [upload=14892]: Got next segment. Inline: false
Aug 8 00:02:30.778583389 (14h22m30.367928376s) [upload=14892]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Aug 8 00:02:30.778587717 (14h22m30.367932704s) [upload=14892]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Aug 8 00:03:30.135980490 (14h23m29.725325467s) [upload=14892]: Next segment err: context deadline exceeded
Aug 8 00:03:30.136177137 (14h23m29.725522124s) [upload=14892]: Done uploading segments...
Aug 8 00:03:30.136196834 (14h23m29.725541821s) [upload=14892]: Upload finished. err: context deadline exceeded
Duration: 5.725115202s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
5 out of 60 is failed (threshold=3) [....................X...X..........X.......X...X............] duration:
6.064392834s
FAIL (2025-08-08 00:37:56 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/4M):: after a4399ade4fa2878d720011f9f1d393a7a350a6f1820ab363fc7588b62888e2c7 != before 879367dc26e769e454df7bfa7c99b4b3877306fa37488cef7c83620ea9a98ba6
======LOGS======
.BatchRequestItem_ObjectBegin
Aug 8 00:28:47.094828147 (14h48m46.684173134s) [upload=15346]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Aug 8 00:28:47.551561608 (14h48m47.140906595s) [seg_pos={0 0}, upload=15346]: Begin upload segment...
Aug 8 00:28:47.551608808 (14h48m47.140953795s) [seg_pos={0 0}, upload=15346]: Done begin upload segment.
Aug 8 00:28:47.551629437 (14h48m47.140974424s) [noise=true , address=104.36.82.18:4367 , piece_id=PVNJQVZ3QW5ZVHGQG3QSDLRRZV5BFGN6J2MC62JQ5P5ZJW65WW5Q, seg_pos={0 0}, upload=15346]: Uploading piece...
Aug 8 00:28:47.554243379 (14h48m47.143588366s) [noise=true , address=104.36.82.7:4937 , piece_id=HI26KBJIH6YFKH2NU655SRLUCFTDHNA4QR2THVJVIEDMHQTZSJLQ, seg_pos={0 0}, upload=15346]: Uploading piece...
Aug 8 00:28:47.554910236 (14h48m47.144255223s) [noise=true , address=104.36.82.20:4445 , piece_id=LOWLB4F4N7LCZ5C3JE6HRSSU4FUOKJNIFTHIHRNCJCEKHGG667GA, seg_pos={0 0}, upload=15346]: Uploading piece...
Aug 8 00:28:47.555304531 (14h48m47.144649518s) [noise=true , address=104.36.82.62:4413 , piece_id=XNYCVTWNEL27VZYCYSA6F5JIIEZMNUPHGLCOX5LVH6LHAFSNB7DA, seg_pos={0 0}, upload=15346]: Uploading piece...
Aug 8 00:28:47.555953072 (14h48m47.145298059s) [noise=true , address=104.36.82.43:4861 , piece_id=JMUXDXGFTHJHGHMNKXIWYNWG4QALWELUKH6DODKRKVUS53YBEYAQ, seg_pos={0 0}, upload=15346]: Uploading piece...
Aug 8 00:28:47.556452077 (14h48m47.145797064s) [noise=true , address=104.36.82.37:4680 , piece_id=XZNKDG5FITGWMR7U7WSVE5F3KYFMNXZ73AFFCB5NC2RF3CKCK7RQ, seg_pos={0 0}, upload=15346]: Uploading piece...
Aug 8 00:28:47.557051404 (14h48m47.146396391s) [noise=true , address=104.36.82.10:4085 , piece_id=RI6KTTGHEXBFGMW54NCT7MJFUYRBIGB62AOUS3DASGOA2D55P6SQ, seg_pos={0 0}, upload=15346]: Uploading piece...
Aug 8 00:28:47.557476658 (14h48m47.146821645s) [noise=true , address=104.36.82.8:4979 , piece_id=Z6YJ5QYHJJJLGUXGCIPGGE25OLKO25MPC5AQ53LMZ22RCB25QWXQ, seg_pos={0 0}, upload=15346]: Uploading piece...
Aug 8 00:28:47.557973529 (14h48m47.147318516s) [noise=true , address=104.36.82.61:4376 , piece_id=PRM6GANCIR4W32MFVJY5UQ7YBMXHVMNXBUYDNQQUHA45HR5JWQAQ, seg_pos={0 0}, upload=15346]: Uploading piece...
Aug 8 00:28:47.558963734 (14h48m47.148308721s) [noise=true , address=104.36.82.52:4105 , piece_id=XT46OYR6SIAZQ6L4GFYNKXQCNIETX6FYVM3SZRMUYFSICP66YXKA, seg_pos={0 0}, upload=15346]: Uploading piece...
Aug 8 00:28:47.559345425 (14h48m47.148690412s) [noise=true , address=104.36.82.4:4836 , piece_id=M2Y7QYLPP4FNSUTJZOF3U5ZYDBUDJL25O7ILENA47OYIHJIE7CJA, seg_pos={0 0}, upload=15346]: Uploading piece...
Aug 8 00:28:47.559751782 (14h48m47.149096759s) [noise=true , address=104.36.82.23:4464 , piece_id=KSV4RQE2YF4TL2VB466SPMQTWWO2P7TD624GQQYYHZDIUUBKAJUQ, seg_pos={0 0}, upload=15346]: Uploading piece...
Aug 8 00:28:47.560180823 (14h48m47.14952581s) [noise=true , address=104.36.82.60:4366 , piece_id=U2HK37YA6BWLVE7ABJ465K7PTEXGEWI72ICQWZRSXZ3KGASXRL3Q, seg_pos={0 0}, upload=15346]: Uploading piece...
Aug 8 00:28:47.561215644 (14h48m47.150560631s) [noise=true , address=104.36.82.47:4520 , piece_id=QGOCU74LKBEZ5IA76WKIM2ZY57MAUMSJDYF3U3L7WCM3JPPTV7YQ, seg_pos={0 0}, upload=15346]: Uploading piece...
Aug 8 00:28:47.561560894 (14h48m47.150905881s) [noise=true , address=104.36.82.4:4813 , piece_id=37ZR56FKL5MJB4ZBBNGFUTVCXX6WEDJTYKEVSWLL7JLRNVONV2CQ, seg_pos={0 0}, upload=15346]: Uploading piece...
Aug 8 00:28:47.561844247 (14h48m47.151189234s) [noise=true , address=104.36.82.59:4325 , piece_id=EIEJTAIZHKY4SWUZPA7EXYUWYQV3SU4NHVA7IUDRTW2CJHSIPQPA, seg_pos={0 0}, upload=15346]: Uploading piece...
Aug 8 00:28:47.562154400 (14h48m47.151499387s) [noise=true , address=79.127.201.215:4352 , piece_id=EC3UMETYKU6KUMG3VJQOJ2WJCTCDWQ4WBHKPHQAKT7NC3F6RWDXQ, seg_pos={0 0}, upload=15346]: Uploading piece...
Aug 8 00:28:47.562457500 (14h48m47.151802487s) [noise=true , address=104.36.82.44:4895 , piece_id=E364QDW2NRBMQPZGZHJWRZO6GJN3SHL2CRVAAD4KRFYS5YDLOYZA, seg_pos={0 0}, upload=15346]: Uploading piece...
Aug 8 00:28:49.073642290 (14h48m48.662987277s) [upload=15346]: Next returned nil segment
Aug 8 00:28:49.073651728 (14h48m48.662996715s) [upload=15346]: Waiting for error group managing segments...
Aug 8 00:28:49.485127304 (14h48m49.074472291s) [noise=true , address=79.127.201.215:4352 , piece_id=EC3UMETYKU6KUMG3VJQOJ2WJCTCDWQ4WBHKPHQAKT7NC3F6RWDXQ, seg_pos={0 0}, upload=15346]: Done uploading piece. err: <nil>
Aug 8 00:28:49.569397892 (14h48m49.158742879s) [noise=true , address=104.36.82.23:4464 , piece_id=KSV4RQE2YF4TL2VB466SPMQTWWO2P7TD624GQQYYHZDIUUBKAJUQ, seg_pos={0 0}, upload=15346]: Done uploading piece. err: <nil>
Aug 8 00:28:49.589957680 (14h48m49.179302667s) [noise=true , address=104.36.82.7:4937 , piece_id=HI26KBJIH6YFKH2NU655SRLUCFTDHNA4QR2THVJVIEDMHQTZSJLQ, seg_pos={0 0}, upload=15346]: Done uploading piece. err: <nil>
Aug 8 00:28:49.659215316 (14h48m49.248560303s) [noise=true , address=104.36.82.20:4445 , piece_id=LOWLB4F4N7LCZ5C3JE6HRSSU4FUOKJNIFTHIHRNCJCEKHGG667GA, seg_pos={0 0}, upload=15346]: Done uploading piece. err: <nil>
Aug 8 00:28:49.659539797 (14h48m49.248884784s) [noise=true , address=104.36.82.8:4979 , piece_id=Z6YJ5QYHJJJLGUXGCIPGGE25OLKO25MPC5AQ53LMZ22RCB25QWXQ, seg_pos={0 0}, upload=15346]: Done uploading piece. err: <nil>
Aug 8 00:28:49.699454667 (14h48m49.288799654s) [noise=true , address=104.36.82.4:4813 , piece_id=37ZR56FKL5MJB4ZBBNGFUTVCXX6WEDJTYKEVSWLL7JLRNVONV2CQ, seg_pos={0 0}, upload=15346]: Done uploading piece. err: <nil>
Aug 8 00:28:49.734069373 (14h48m49.32341436s) [noise=true , address=104.36.82.44:4895 , piece_id=E364QDW2NRBMQPZGZHJWRZO6GJN3SHL2CRVAAD4KRFYS5YDLOYZA, seg_pos={0 0}, upload=15346]: Done uploading piece. err: <nil>
Aug 8 00:28:49.759950614 (14h48m49.349295601s) [noise=true , address=104.36.82.4:4836 , piece_id=M2Y7QYLPP4FNSUTJZOF3U5ZYDBUDJL25O7ILENA47OYIHJIE7CJA, seg_pos={0 0}, upload=15346]: Done uploading piece. err: <nil>
Aug 8 00:28:49.787498666 (14h48m49.376843653s) [noise=true , address=104.36.82.10:4085 , piece_id=RI6KTTGHEXBFGMW54NCT7MJFUYRBIGB62AOUS3DASGOA2D55P6SQ, seg_pos={0 0}, upload=15346]: Done uploading piece. err: <nil>
Aug 8 00:28:49.874182563 (14h48m49.46352758s) [noise=true , address=104.36.82.52:4105 , piece_id=XT46OYR6SIAZQ6L4GFYNKXQCNIETX6FYVM3SZRMUYFSICP66YXKA, seg_pos={0 0}, upload=15346]: Done uploading piece. err: <nil>
Aug 8 00:28:49.960972762 (14h48m49.550317749s) [noise=true , address=104.36.82.59:4325 , piece_id=EIEJTAIZHKY4SWUZPA7EXYUWYQV3SU4NHVA7IUDRTW2CJHSIPQPA, seg_pos={0 0}, upload=15346]: Done uploading piece. err: <nil>
Aug 8 00:28:50.001273120 (14h48m49.590618117s) [noise=true , address=104.36.82.37:4680 , piece_id=XZNKDG5FITGWMR7U7WSVE5F3KYFMNXZ73AFFCB5NC2RF3CKCK7RQ, seg_pos={0 0}, upload=15346]: Done uploading piece. err: <nil>
Aug 8 00:28:50.571026048 (14h48m50.160371025s) [noise=true , address=104.36.82.62:4413 , piece_id=XNYCVTWNEL27VZYCYSA6F5JIIEZMNUPHGLCOX5LVH6LHAFSNB7DA, seg_pos={0 0}, upload=15346]: Done uploading piece. err: <nil>
Aug 8 00:28:51.294616675 (14h48m50.883961662s) [noise=true , address=104.36.82.61:4376 , piece_id=PRM6GANCIR4W32MFVJY5UQ7YBMXHVMNXBUYDNQQUHA45HR5JWQAQ, seg_pos={0 0}, upload=15346]: Done uploading piece. err: <nil>
Aug 8 00:28:51.294639959 (14h48m50.883984936s) [seg_pos={0 0}, upload=15346]: Segment reached optimal threshold of 14 pieces.
Aug 8 00:28:51.294832458 (14h48m50.884177445s) [noise=true , address=104.36.82.47:4520 , piece_id=QGOCU74LKBEZ5IA76WKIM2ZY57MAUMSJDYF3U3L7WCM3JPPTV7YQ, seg_pos={0 0}, upload=15346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12eNRsboM5zFiyBVPLuQfgb5QSAc5bZFizK9KJHudNd9nstVGyy): protocol: expected piece hash; context canceled; EOF
Aug 8 00:28:51.295062047 (14h48m50.884407034s) [noise=true , address=104.36.82.60:4366 , piece_id=U2HK37YA6BWLVE7ABJ465K7PTEXGEWI72ICQWZRSXZ3KGASXRL3Q, seg_pos={0 0}, upload=15346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12NBUjBGU2MCjDx8iZKvzCvhL6GHCfdXfihC5H2uoBUmzhz9uoV): protocol: write tcp 188.166.225.170:57292->104.36.82.60:4366: use of closed network connection; write tcp 188.166.225.170:57292->104.36.82.60:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57292->104.36.82.60:4366: use of closed network connection
Aug 8 00:28:51.295101312 (14h48m50.884446299s) [noise=true , address=104.36.82.43:4861 , piece_id=JMUXDXGFTHJHGHMNKXIWYNWG4QALWELUKH6DODKRKVUS53YBEYAQ, seg_pos={0 0}, upload=15346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122HC6BetMaBDNGEaHVwmadWtV1nCHx3vGBK2SWVDEQS1iBCz4o): protocol: write tcp 188.166.225.170:53912->104.36.82.43:4861: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 8 00:28:51.295131720 (14h48m50.884476707s) [noise=true , address=104.36.82.18:4367 , piece_id=PVNJQVZ3QW5ZVHGQG3QSDLRRZV5BFGN6J2MC62JQ5P5ZJW65WW5Q, seg_pos={0 0}, upload=15346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6): protocol: read tcp 188.166.225.170:32868->104.36.82.18:4367: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 8 00:28:51.295147570 (14h48m50.884492557s) [seg_pos={0 0}, upload=15346]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Aug 8 00:28:51.295216993 (14h48m50.88456198s) [upload=15346]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Aug 8 00:28:51.295226662 (14h48m50.884571649s) [upload=15346]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Aug 8 00:28:51.599168829 (14h48m51.188513806s) [upload=15346]: Done uploading segments...
Aug 8 00:28:51.599190280 (14h48m51.188535267s) [upload=15346]: Upload finished. err: <nil>
Duration: 6.064416619s
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)antiblip:
1 out of 60 is failed (threshold=3) [.X..........................................................] duration:
1.893473108s
FAIL (2025-08-08 00:37:53 UTC)
orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded
Duration: 1.893499537s
core
uplink
us1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [XXX.........................................................] duration:
5.672315514s
FAIL (2025-08-08 00:37:47 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.672362504s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)antiblip:
1 out of 60 is failed (threshold=3) [..X.........................................................] duration:
1.878210929s
FAIL (2025-08-08 00:37:36 UTC)
orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded
Duration: 1.878233512s
core
uplink
us1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)antiblip:
6 out of 60 is failed (threshold=3) [X...XXXXX...................................................] duration:
1.843532671s
FAIL (2025-08-08 00:37:28 UTC)
orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== m24.001855996s) [upload=14742]: Done starting upload Aug 7 23:53:24.412569290 (14h13m24.001914277s) [upload=14742]: Uploading segments... Aug 7 23:53:24.412638803 (14h13m24.001983831s) [upload=14742]: Got next segment. Inline: true Aug 7 23:53:24.412644354 (14h13m24.001989341s) [upload=14742]: Waiting for error group managing segments... Aug 7 23:53:24.412680903 (14h13m24.002025891s) [upload=14742]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 23:53:24.412685442 (14h13m24.002030429s) [upload=14742]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 23:53:24.412688137 (14h13m24.002033114s) [upload=14742]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 23:53:53.772057432 (14h13m53.361402419s) [upload=14742]: Done uploading segments... Aug 7 23:53:53.772119881 (14h13m53.361464908s) [upload=14742]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:54:30.496202349 (14h14m30.085547336s) [upload=14761]: Starting upload Aug 7 23:54:30.496270259 (14h14m30.085615246s) [upload=14761]: Done starting upload Aug 7 23:54:30.496341565 (14h14m30.085686552s) [upload=14761]: Uploading segments... Aug 7 23:54:30.496378957 (14h14m30.085723944s) [upload=14761]: Got next segment. Inline: true Aug 7 23:54:30.496383836 (14h14m30.085728823s) [upload=14761]: Waiting for error group managing segments... Aug 7 23:54:30.496441346 (14h14m30.085786333s) [upload=14761]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 23:54:30.496446426 (14h14m30.085791413s) [upload=14761]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 23:54:30.496449551 (14h14m30.085794539s) [upload=14761]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 23:54:59.847423574 (14h14m59.436768561s) [upload=14761]: Done uploading segments... Aug 7 23:54:59.847456066 (14h14m59.436801043s) [upload=14761]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:55:40.612588843 (14h15m40.20193383s) [upload=14781]: Starting upload Aug 7 23:55:40.612648929 (14h15m40.201993906s) [upload=14781]: Done starting upload Aug 7 23:55:40.612717029 (14h15m40.202062016s) [upload=14781]: Uploading segments... Aug 7 23:55:40.612750823 (14h15m40.20209581s) [upload=14781]: Got next segment. Inline: true Aug 7 23:55:40.612754510 (14h15m40.202099487s) [upload=14781]: Waiting for error group managing segments... Aug 7 23:55:40.612786883 (14h15m40.20213186s) [upload=14781]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 23:55:40.612790891 (14h15m40.202135868s) [upload=14781]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 23:55:40.612793075 (14h15m40.202138062s) [upload=14781]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 23:55:40.883469345 (14h15m40.472814333s) [upload=14781]: Done uploading segments... Aug 7 23:55:40.883486387 (14h15m40.472831375s) [upload=14781]: Upload finished. err: <nil> Aug 7 23:56:17.423712239 (14h16m17.013057226s) [upload=14789]: Starting upload Aug 7 23:56:17.423767895 (14h16m17.013112882s) [upload=14789]: Done starting upload Aug 7 23:56:17.423814835 (14h16m17.013159822s) [upload=14789]: Uploading segments... Aug 7 23:56:17.423847678 (14h16m17.013192665s) [upload=14789]: Got next segment. Inline: true Aug 7 23:56:17.423851335 (14h16m17.013196322s) [upload=14789]: Waiting for error group managing segments... Aug 7 23:56:17.423883377 (14h16m17.013228364s) [upload=14789]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 23:56:17.423887054 (14h16m17.013232041s) [upload=14789]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 23:56:17.423889258 (14h16m17.013234245s) [upload=14789]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 23:56:17.689327115 (14h16m17.278672102s) [upload=14789]: Done uploading segments... Aug 7 23:56:17.689345711 (14h16m17.278690688s) [upload=14789]: Upload finished. err: <nil> Aug 7 23:56:58.288741281 (14h16m57.878086268s) [upload=14801]: Starting upload Aug 7 23:56:58.288797098 (14h16m57.878142085s) [upload=14801]: Done starting upload Aug 7 23:56:58.288851010 (14h16m57.878195987s) [upload=14801]: Uploading segments... Aug 7 23:56:58.288882281 (14h16m57.878227268s) [upload=14801]: Got next segment. Inline: true Aug 7 23:56:58.288887911 (14h16m57.878232898s) [upload=14801]: Waiting for error group managing segments... Aug 7 23:56:58.288919792 (14h16m57.878264769s) [upload=14801]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 23:56:58.288923970 (14h16m57.878268957s) [upload=14801]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 23:56:58.288926485 (14h16m57.878271462s) [upload=14801]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 23:56:58.556529816 (14h16m58.145874803s) [upload=14801]: Done uploading segments... Aug 7 23:56:58.556544975 (14h16m58.145889962s) [upload=14801]: Upload finished. err: <nil> Aug 7 23:57:18.362794018 (14h17m17.952138995s) [upload=14806]: Starting upload Aug 7 23:57:18.362883679 (14h17m17.952228666s) [upload=14806]: Done starting upload Aug 7 23:57:18.362928104 (14h17m17.952273092s) [upload=14806]: Uploading segments... Aug 7 23:57:18.362957320 (14h17m17.952302307s) [upload=14806]: Got next segment. Inline: true Aug 7 23:57:18.362961077 (14h17m17.952306064s) [upload=14806]: Waiting for error group managing segments... Aug 7 23:57:18.363018407 (14h17m17.952363394s) [upload=14806]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 23:57:18.363022404 (14h17m17.952367392s) [upload=14806]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 23:57:18.363024739 (14h17m17.952369716s) [upload=14806]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 23:57:47.715297444 (14h17m47.304642431s) [upload=14806]: Done uploading segments... Aug 7 23:57:47.715410370 (14h17m47.304755357s) [upload=14806]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:57:53.375540080 (14h17m52.964885068s) [upload=14813]: Starting upload Aug 7 23:57:53.375600245 (14h17m52.964945232s) [upload=14813]: Done starting upload Aug 7 23:57:53.375654389 (14h17m52.964999376s) [upload=14813]: Uploading segments... Aug 7 23:57:53.375686622 (14h17m52.965031609s) [upload=14813]: Got next segment. Inline: true Aug 7 23:57:53.375690759 (14h17m52.965035746s) [upload=14813]: Waiting for error group managing segments... Aug 7 23:57:53.375725876 (14h17m52.965070863s) [upload=14813]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 23:57:53.375730134 (14h17m52.965075121s) [upload=14813]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 23:57:53.375734372 (14h17m52.965079359s) [upload=14813]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 23:58:22.726838464 (14h18m22.316183451s) [upload=14813]: Done uploading segments... Aug 7 23:58:22.726860075 (14h18m22.316205062s) [upload=14813]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:58:23.621870384 (14h18m23.211215371s) [upload=14822]: Starting upload Aug 7 23:58:23.621952000 (14h18m23.211296987s) [upload=14822]: Done starting upload Aug 7 23:58:23.622003359 (14h18m23.211348346s) [upload=14822]: Uploading segments... Aug 7 23:58:23.622043436 (14h18m23.211388413s) [upload=14822]: Got next segment. Inline: true Aug 7 23:58:23.622046772 (14h18m23.211391759s) [upload=14822]: Waiting for error group managing segments... Aug 7 23:58:23.622078573 (14h18m23.21142356s) [upload=14822]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 23:58:23.622082230 (14h18m23.211427217s) [upload=14822]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 23:58:23.622084534 (14h18m23.211429511s) [upload=14822]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 23:58:52.983677879 (14h18m52.573022876s) [upload=14822]: Done uploading segments... Aug 7 23:58:52.983713708 (14h18m52.573058685s) [upload=14822]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:59:11.446858470 (14h19m11.036203457s) [upload=14835]: Starting upload Aug 7 23:59:11.446906493 (14h19m11.03625148s) [upload=14835]: Done starting upload Aug 7 23:59:11.446947491 (14h19m11.036292479s) [upload=14835]: Uploading segments... Aug 7 23:59:11.446975475 (14h19m11.036320462s) [upload=14835]: Got next segment. Inline: true Aug 7 23:59:11.446978751 (14h19m11.036323738s) [upload=14835]: Waiting for error group managing segments... Aug 7 23:59:11.447008889 (14h19m11.036353876s) [upload=14835]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 23:59:11.447013638 (14h19m11.036358625s) [upload=14835]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 23:59:11.447016212 (14h19m11.0363612s) [upload=14835]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 23:59:40.799397635 (14h19m40.388742622s) [upload=14835]: Done uploading segments... Aug 7 23:59:40.799413266 (14h19m40.388758243s) [upload=14835]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:59:59.523080643 (14h19m59.11242563s) [upload=14847]: Starting upload Aug 7 23:59:59.523152089 (14h19m59.112497076s) [upload=14847]: Done starting upload Aug 7 23:59:59.523213727 (14h19m59.112558714s) [upload=14847]: Uploading segments... Aug 7 23:59:59.523251349 (14h19m59.112596336s) [upload=14847]: Got next segment. Inline: true Aug 7 23:59:59.523255287 (14h19m59.112600274s) [upload=14847]: Waiting for error group managing segments... Aug 7 23:59:59.523300013 (14h19m59.112645s) [upload=14847]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 23:59:59.523304251 (14h19m59.112649238s) [upload=14847]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 23:59:59.523307146 (14h19m59.112652133s) [upload=14847]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 8 00:00:28.877190143 (14h20m28.46653516s) [upload=14847]: Done uploading segments... Aug 8 00:00:28.877218477 (14h20m28.466563505s) [upload=14847]: Upload finished. err: metaclient: context deadline exceeded
Duration: 1.843556466s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)antiblip:
6 out of 60 is failed (threshold=3) [...XXX.XX....X..............................................] duration:
5.804870653s
FAIL (2025-08-08 00:37:26 UTC)
orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded
Duration: 5.804896142s
core
uplink
us1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)antiblip:
1 out of 60 is failed (threshold=3) [................................X...........................] duration:
1.86804763s
FAIL (2025-08-08 00:37:24 UTC)
orbiter warning:
operation: DownloadObject
======ERROR======
uplink: object not found ("orbiter_us_0/1K")
======LOGS======
=14964]: Done uploading segments...
Aug 8 00:06:41.669168196 (14h26m41.258513183s) [upload=14964]: Upload finished. err: <nil>
Aug 8 00:07:06.258431851 (14h27m5.847776838s) [upload=14976]: Starting upload
Aug 8 00:07:06.258508298 (14h27m5.847853285s) [upload=14976]: Done starting upload
Aug 8 00:07:06.258574986 (14h27m5.847919983s) [upload=14976]: Uploading segments...
Aug 8 00:07:06.258618599 (14h27m5.847963616s) [upload=14976]: Got next segment. Inline: true
Aug 8 00:07:06.258624600 (14h27m5.847969587s) [upload=14976]: Waiting for error group managing segments...
Aug 8 00:07:06.258705555 (14h27m5.848050542s) [upload=14976]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Aug 8 00:07:06.258712268 (14h27m5.848057296s) [upload=14976]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline
Aug 8 00:07:06.258716226 (14h27m5.848061253s) [upload=14976]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Aug 8 00:07:06.522670976 (14h27m6.112015963s) [upload=14976]: Done uploading segments...
Aug 8 00:07:06.522717766 (14h27m6.112062773s) [upload=14976]: Upload finished. err: <nil>
Aug 8 00:07:36.371369594 (14h27m35.960714581s) [upload=14980]: Starting upload
Aug 8 00:07:36.371417074 (14h27m35.960762061s) [upload=14980]: Done starting upload
Aug 8 00:07:36.371466890 (14h27m35.960811877s) [upload=14980]: Uploading segments...
Aug 8 00:07:36.371508951 (14h27m35.960853938s) [upload=14980]: Got next segment. Inline: true
Aug 8 00:07:36.371512888 (14h27m35.960857875s) [upload=14980]: Waiting for error group managing segments...
Aug 8 00:07:36.371543446 (14h27m35.960888433s) [upload=14980]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Aug 8 00:07:36.371547434 (14h27m35.960892421s) [upload=14980]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline
Aug 8 00:07:36.371549708 (14h27m35.960894695s) [upload=14980]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Aug 8 00:07:36.789946570 (14h27m36.379291557s) [upload=14980]: Done uploading segments...
Aug 8 00:07:36.789964876 (14h27m36.379309863s) [upload=14980]: Upload finished. err: <nil>
Aug 8 00:08:16.866388064 (14h28m16.455733051s) [upload=14994]: Starting upload
Aug 8 00:08:16.866430886 (14h28m16.455775873s) [upload=14994]: Done starting upload
Aug 8 00:08:16.866496882 (14h28m16.455841869s) [upload=14994]: Uploading segments...
Aug 8 00:08:16.866525727 (14h28m16.455870714s) [upload=14994]: Got next segment. Inline: true
Aug 8 00:08:16.866529494 (14h28m16.455874481s) [upload=14994]: Waiting for error group managing segments...
Aug 8 00:08:16.866570884 (14h28m16.455915871s) [upload=14994]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Aug 8 00:08:16.866575212 (14h28m16.455920199s) [upload=14994]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline
Aug 8 00:08:16.866577626 (14h28m16.455922613s) [upload=14994]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Aug 8 00:08:17.138322091 (14h28m16.727667078s) [upload=14994]: Done uploading segments...
Aug 8 00:08:17.138338402 (14h28m16.727683389s) [upload=14994]: Upload finished. err: <nil>
Aug 8 00:08:57.633625328 (14h28m57.222970315s) [upload=15006]: Starting upload
Aug 8 00:08:57.633689390 (14h28m57.223034378s) [upload=15006]: Done starting upload
Aug 8 00:08:57.633744837 (14h28m57.223089824s) [upload=15006]: Uploading segments...
Aug 8 00:08:57.633775525 (14h28m57.223120512s) [upload=15006]: Got next segment. Inline: true
Aug 8 00:08:57.633779222 (14h28m57.223124199s) [upload=15006]: Waiting for error group managing segments...
Aug 8 00:08:57.633951743 (14h28m57.22329672s) [upload=15006]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Aug 8 00:08:57.633956161 (14h28m57.223301148s) [upload=15006]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline
Aug 8 00:08:57.633958886 (14h28m57.223303873s) [upload=15006]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Aug 8 00:08:57.920842105 (14h28m57.510187102s) [upload=15006]: Done uploading segments...
Aug 8 00:08:57.920865330 (14h28m57.510210327s) [upload=15006]: Upload finished. err: <nil>
Aug 8 00:09:43.341371442 (14h29m42.930716449s) [upload=15015]: Starting upload
Aug 8 00:09:43.341457767 (14h29m42.930802774s) [upload=15015]: Done starting upload
Aug 8 00:09:43.341511951 (14h29m42.930856938s) [upload=15015]: Uploading segments...
Aug 8 00:09:43.341579249 (14h29m42.930924236s) [upload=15015]: Got next segment. Inline: true
Aug 8 00:09:43.341585642 (14h29m42.930930639s) [upload=15015]: Waiting for error group managing segments...
Aug 8 00:09:43.341640437 (14h29m42.930985454s) [upload=15015]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Aug 8 00:09:43.341648422 (14h29m42.930993469s) [upload=15015]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline
Aug 8 00:09:43.341653391 (14h29m42.930998408s) [upload=15015]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Aug 8 00:09:43.627931533 (14h29m43.21727652s) [upload=15015]: Done uploading segments...
Aug 8 00:09:43.627948425 (14h29m43.217293402s) [upload=15015]: Upload finished. err: <nil>
Aug 8 00:10:45.166177580 (14h30m44.755522568s) [upload=15031]: Starting upload
Aug 8 00:10:45.166220523 (14h30m44.75556551s) [upload=15031]: Done starting upload
Aug 8 00:10:45.166273034 (14h30m44.755618011s) [upload=15031]: Uploading segments...
Aug 8 00:10:45.166297991 (14h30m44.755642979s) [upload=15031]: Got next segment. Inline: true
Aug 8 00:10:45.166301568 (14h30m44.755646555s) [upload=15031]: Waiting for error group managing segments...
Aug 8 00:10:45.166332277 (14h30m44.755677264s) [upload=15031]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Aug 8 00:10:45.166335783 (14h30m44.75568077s) [upload=15031]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline
Aug 8 00:10:45.166337998 (14h30m44.755682985s) [upload=15031]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Aug 8 00:10:45.444515471 (14h30m45.033860458s) [upload=15031]: Done uploading segments...
Aug 8 00:10:45.444533305 (14h30m45.033878282s) [upload=15031]: Upload finished. err: <nil>
Aug 8 00:11:54.979425886 (14h31m54.568770863s) [upload=15052]: Starting upload
Aug 8 00:11:54.979471082 (14h31m54.568816069s) [upload=15052]: Done starting upload
Aug 8 00:11:54.979517260 (14h31m54.568862237s) [upload=15052]: Uploading segments...
Aug 8 00:11:54.979549422 (14h31m54.568894399s) [upload=15052]: Got next segment. Inline: true
Aug 8 00:11:54.979553008 (14h31m54.568897995s) [upload=15052]: Waiting for error group managing segments...
Aug 8 00:11:54.979584328 (14h31m54.568929315s) [upload=15052]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Aug 8 00:11:54.979588356 (14h31m54.568933333s) [upload=15052]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline
Aug 8 00:11:54.979590661 (14h31m54.568935648s) [upload=15052]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Aug 8 00:11:55.340902393 (14h31m54.93024738s) [upload=15052]: Done uploading segments...
Aug 8 00:11:55.340920908 (14h31m54.930265895s) [upload=15052]: Upload finished. err: <nil>
Aug 8 00:12:44.503574128 (14h32m44.092919115s) [upload=15065]: Starting upload
Aug 8 00:12:44.503634083 (14h32m44.09297907s) [upload=15065]: Done starting upload
Aug 8 00:12:44.503710068 (14h32m44.093055055s) [upload=15065]: Uploading segments...
Aug 8 00:12:44.503762669 (14h32m44.093107656s) [upload=15065]: Got next segment. Inline: true
Aug 8 00:12:44.503767698 (14h32m44.093112685s) [upload=15065]: Waiting for error group managing segments...
Aug 8 00:12:44.503805531 (14h32m44.093150508s) [upload=15065]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Aug 8 00:12:44.503809379 (14h32m44.093154366s) [upload=15065]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline
Aug 8 00:12:44.503811633 (14h32m44.09315662s) [upload=15065]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Aug 8 00:12:44.792734473 (14h32m44.38207946s) [upload=15065]: Done uploading segments...
Aug 8 00:12:44.792752288 (14h32m44.382097275s) [upload=15065]: Upload finished. err: <nil>
Aug 8 00:13:19.961146705 (14h33m19.550491693s) [upload=15079]: Starting upload
Aug 8 00:13:19.961187835 (14h33m19.550532822s) [upload=15079]: Done starting upload
Aug 8 00:13:19.961236618 (14h33m19.550581605s) [upload=15079]: Uploading segments...
Aug 8 00:13:19.961268168 (14h33m19.550613145s) [upload=15079]: Got next segment. Inline: true
Aug 8 00:13:19.961273849 (14h33m19.550618836s) [upload=15079]: Waiting for error group managing segments...
Aug 8 00:13:19.961304548 (14h33m19.550649525s) [upload=15079]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Aug 8 00:13:19.961308345 (14h33m19.550653332s) [upload=15079]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline
Aug 8 00:13:19.961310910 (14h33m19.550655887s) [upload=15079]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Aug 8 00:13:20.385237160 (14h33m19.974582137s) [upload=15079]: Done uploading segments...
Aug 8 00:13:20.385255485 (14h33m19.974600472s) [upload=15079]: Upload finished. err: <nil>
Aug 8 00:14:06.785741109 (14h34m6.375086096s) [upload=15091]: Starting upload
Aug 8 00:14:06.785899151 (14h34m6.375244138s) [upload=15091]: Done starting upload
Aug 8 00:14:06.786002628 (14h34m6.375347615s) [upload=15091]: Uploading segments...
Aug 8 00:14:06.786034078 (14h34m6.375379065s) [upload=15091]: Got next segment. Inline: true
Aug 8 00:14:06.786037786 (14h34m6.375382773s) [upload=15091]: Waiting for error group managing segments...
Aug 8 00:14:06.786073094 (14h34m6.375418081s) [upload=15091]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Aug 8 00:14:06.786078945 (14h34m6.375423922s) [upload=15091]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline
Aug 8 00:14:06.786081339 (14h34m6.375426326s) [upload=15091]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Aug 8 00:14:07.092788097 (14h34m6.682133084s) [upload=15091]: Done uploading segments...
Aug 8 00:14:07.092803166 (14h34m6.682148153s) [upload=15091]: Upload finished. err: <nil>
Duration: 1.86808427s
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline)antiblip:
3 out of 60 is failed (threshold=3) [...X.X..X...................................................] duration:
2.559845947s
FAIL (2025-08-08 00:37:21 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.559863571s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [.X....X.....................................................] duration:
6.062146533s
FAIL (2025-08-08 00:37:18 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.062166061s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
8 out of 60 is failed (threshold=3) [.XX..XXXX....XX.............................................] duration:
5.833075799s
FAIL (2025-08-08 00:37:09 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
3.700898256s) [upload=14855]: Done starting upload
Aug 8 00:00:34.111602513 (14h20m33.7009475s) [upload=14855]: Uploading segments...
Aug 8 00:00:34.111774201 (14h20m33.701119218s) [upload=14855]: Got next segment. Inline: false
Aug 8 00:00:34.111816783 (14h20m33.70116178s) [upload=14855]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Aug 8 00:00:34.111822464 (14h20m33.701167461s) [upload=14855]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Aug 8 00:01:03.465979471 (14h21m3.055324468s) [upload=14855]: Next segment err: context deadline exceeded
Aug 8 00:01:03.466148254 (14h21m3.055493251s) [upload=14855]: Done uploading segments...
Aug 8 00:01:03.466174945 (14h21m3.055519932s) [upload=14855]: Upload finished. err: context deadline exceeded
Aug 8 00:01:19.453976971 (14h21m19.043321958s) [upload=14872]: Starting upload
Aug 8 00:01:19.454042106 (14h21m19.043387123s) [upload=14872]: Done starting upload
Aug 8 00:01:19.454156104 (14h21m19.043501091s) [upload=14872]: Uploading segments...
Aug 8 00:01:19.454244503 (14h21m19.04358949s) [upload=14872]: Got next segment. Inline: false
Aug 8 00:01:19.454272567 (14h21m19.043617554s) [upload=14872]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Aug 8 00:01:19.454276825 (14h21m19.043621812s) [upload=14872]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Aug 8 00:01:39.582519114 (14h21m39.171864101s) [seg_pos={0 0}, upload=14872]: Begin upload segment...
Aug 8 00:01:39.582565882 (14h21m39.17191087s) [seg_pos={0 0}, upload=14872]: Done begin upload segment.
Aug 8 00:01:39.582581332 (14h21m39.17192632s) [noise=true , address=45.91.116.5:4299 , piece_id=HRKD77XA4SHZKSEW2WA3AROWPLHOWEXG5GEOWJCG7NTFOYYZ4DFA, seg_pos={0 0}, upload=14872]: Uploading piece...
Aug 8 00:01:39.582590870 (14h21m39.171935857s) [noise=true , address=104.36.82.2:4731 , piece_id=2BTN7TE7VVSPN5BUCDVKHWZPPATMNIJLXNDXUO46Z7CRB3QBCUEQ, seg_pos={0 0}, upload=14872]: Uploading piece...
Aug 8 00:01:39.582951871 (14h21m39.172296858s) [noise=true , address=45.91.116.12:4225 , piece_id=MUZ4UWUJSWGYKIO3BSTEOJYBBXSK4R2PURNDXHOYLTZ7VZ466IGA, seg_pos={0 0}, upload=14872]: Uploading piece...
Aug 8 00:01:39.583059888 (14h21m39.172404905s) [noise=true , address=104.36.82.13:4185 , piece_id=O26DWLA45AEOUZOW2CZTLJNPVRWB5KVDQW6FH2ANTKLUMYNWRVKA, seg_pos={0 0}, upload=14872]: Uploading piece...
Aug 8 00:01:39.583214905 (14h21m39.172559892s) [noise=true , address=45.91.116.8:4047 , piece_id=4LUGFRA2QPUK5LPKZGNTURXXSGY64AZU4Q3BHBAM3SPBZ7M2FFKQ, seg_pos={0 0}, upload=14872]: Uploading piece...
Aug 8 00:01:39.583427882 (14h21m39.172772869s) [noise=true , address=77.90.31.21:4877 , piece_id=MZ4WJDOFWH56JIAGXJZ7FOAPMNCHVEBUUM2G5IY2KKPRHPJH5MDA, seg_pos={0 0}, upload=14872]: Uploading piece...
Aug 8 00:01:39.583689032 (14h21m39.173034019s) [noise=true , address=77.90.31.11:4555 , piece_id=OCZBWUJHEVLMUD6BNADJ7BOSHHZPESQJOSKDZ3T2MUSQRFTNI7KA, seg_pos={0 0}, upload=14872]: Uploading piece...
Aug 8 00:01:39.583925344 (14h21m39.173270331s) [noise=true , address=45.91.116.9:4263 , piece_id=F4QDS3EOSLFFCEDU6N56AMMQR4GHPWZ6TIAMO5SGVIMVMFA47EQQ, seg_pos={0 0}, upload=14872]: Uploading piece...
Aug 8 00:01:39.584272317 (14h21m39.173617294s) [noise=true , address=104.36.82.8:4970 , piece_id=N52DS4AB7Z6VSFYWHN5MXPUS3AP7UKN2K7U5D5JZWETM7ZLROO7Q, seg_pos={0 0}, upload=14872]: Uploading piece...
Aug 8 00:01:39.584579135 (14h21m39.173924122s) [noise=true , address=77.90.31.18:4807 , piece_id=V7EZJW5PYJCWWNNIRAJCFTT42DQJY2C4RDHB5645TKOSXJJXE2WA, seg_pos={0 0}, upload=14872]: Uploading piece...
Aug 8 00:01:39.584852117 (14h21m39.174197104s) [noise=true , address=104.36.82.11:4115 , piece_id=S34RF36SERFPPSHBPYJ5V3FCR5ZE7RFIVEXBOTFRHPEEIUCHO2LQ, seg_pos={0 0}, upload=14872]: Uploading piece...
Aug 8 00:01:39.585105202 (14h21m39.174450179s) [noise=true , address=104.36.82.9:4005 , piece_id=ZYZ6VK5MCW7QE6N2GOYW2WMWDEWGONNK2Y7QBE7FPVPLY2OZGIMQ, seg_pos={0 0}, upload=14872]: Uploading piece...
Aug 8 00:01:39.585298762 (14h21m39.174643739s) [noise=true , address=45.91.116.4:4940 , piece_id=5S5JLBIGSBIVTPVJT5MIRAE42PRCIFWDNWLY7NQXSWPMHT5WSTRA, seg_pos={0 0}, upload=14872]: Uploading piece...
Aug 8 00:01:39.585506369 (14h21m39.174851346s) [noise=true , address=104.36.82.10:4079 , piece_id=262SP4DREL7FYDG6YOBFWDPWKYHZIQGDJKSX7XAIUJUTQWUIEEZQ, seg_pos={0 0}, upload=14872]: Uploading piece...
Aug 8 00:01:39.585763882 (14h21m39.175108869s) [noise=true , address=45.91.116.14:4190 , piece_id=FSO3UN2D24KPDVSWWPI3ROBXN7RN6D2HAZKOXEQ2JOW2M6RQ5MDA, seg_pos={0 0}, upload=14872]: Uploading piece...
Aug 8 00:01:39.586021575 (14h21m39.175366562s) [noise=true , address=104.36.82.5:4839 , piece_id=EHHLJG3INARPORGO6YVXU7QJAK23ADRVWBGNT5IKL4N6Y4ASM3XA, seg_pos={0 0}, upload=14872]: Uploading piece...
Aug 8 00:01:41.370103161 (14h21m40.959448138s) [upload=14872]: Next returned nil segment
Aug 8 00:01:41.370130243 (14h21m40.95947523s) [upload=14872]: Waiting for error group managing segments...
Aug 8 00:01:41.673384945 (14h21m41.262729922s) [noise=true , address=104.36.82.2:4731 , piece_id=2BTN7TE7VVSPN5BUCDVKHWZPPATMNIJLXNDXUO46Z7CRB3QBCUEQ, seg_pos={0 0}, upload=14872]: Done uploading piece. err: <nil>
Aug 8 00:01:41.739890780 (14h21m41.329235767s) [noise=true , address=104.36.82.9:4005 , piece_id=ZYZ6VK5MCW7QE6N2GOYW2WMWDEWGONNK2Y7QBE7FPVPLY2OZGIMQ, seg_pos={0 0}, upload=14872]: Done uploading piece. err: <nil>
Aug 8 00:01:41.743845828 (14h21m41.333190815s) [noise=true , address=104.36.82.8:4970 , piece_id=N52DS4AB7Z6VSFYWHN5MXPUS3AP7UKN2K7U5D5JZWETM7ZLROO7Q, seg_pos={0 0}, upload=14872]: Done uploading piece. err: <nil>
Aug 8 00:01:41.786086095 (14h21m41.375431072s) [noise=true , address=45.91.116.8:4047 , piece_id=4LUGFRA2QPUK5LPKZGNTURXXSGY64AZU4Q3BHBAM3SPBZ7M2FFKQ, seg_pos={0 0}, upload=14872]: Done uploading piece. err: <nil>
Aug 8 00:01:41.791525722 (14h21m41.380870699s) [noise=true , address=104.36.82.5:4839 , piece_id=EHHLJG3INARPORGO6YVXU7QJAK23ADRVWBGNT5IKL4N6Y4ASM3XA, seg_pos={0 0}, upload=14872]: Done uploading piece. err: <nil>
Aug 8 00:01:41.959657137 (14h21m41.549002124s) [noise=true , address=104.36.82.13:4185 , piece_id=O26DWLA45AEOUZOW2CZTLJNPVRWB5KVDQW6FH2ANTKLUMYNWRVKA, seg_pos={0 0}, upload=14872]: Done uploading piece. err: <nil>
Aug 8 00:01:42.000743686 (14h21m41.590088693s) [noise=true , address=45.91.116.12:4225 , piece_id=MUZ4UWUJSWGYKIO3BSTEOJYBBXSK4R2PURNDXHOYLTZ7VZ466IGA, seg_pos={0 0}, upload=14872]: Done uploading piece. err: <nil>
Aug 8 00:01:42.160166408 (14h21m41.749511395s) [noise=true , address=45.91.116.4:4940 , piece_id=5S5JLBIGSBIVTPVJT5MIRAE42PRCIFWDNWLY7NQXSWPMHT5WSTRA, seg_pos={0 0}, upload=14872]: Done uploading piece. err: <nil>
Aug 8 00:01:42.175399653 (14h21m41.764744641s) [noise=true , address=45.91.116.14:4190 , piece_id=FSO3UN2D24KPDVSWWPI3ROBXN7RN6D2HAZKOXEQ2JOW2M6RQ5MDA, seg_pos={0 0}, upload=14872]: Done uploading piece. err: <nil>
Aug 8 00:01:42.175880162 (14h21m41.765225139s) [noise=true , address=45.91.116.5:4299 , piece_id=HRKD77XA4SHZKSEW2WA3AROWPLHOWEXG5GEOWJCG7NTFOYYZ4DFA, seg_pos={0 0}, upload=14872]: Done uploading piece. err: <nil>
Aug 8 00:01:42.278394081 (14h21m41.867739068s) [noise=true , address=77.90.31.21:4877 , piece_id=MZ4WJDOFWH56JIAGXJZ7FOAPMNCHVEBUUM2G5IY2KKPRHPJH5MDA, seg_pos={0 0}, upload=14872]: Done uploading piece. err: <nil>
Aug 8 00:01:42.293663936 (14h21m41.883008933s) [noise=true , address=77.90.31.18:4807 , piece_id=V7EZJW5PYJCWWNNIRAJCFTT42DQJY2C4RDHB5645TKOSXJJXE2WA, seg_pos={0 0}, upload=14872]: Done uploading piece. err: <nil>
Aug 8 00:01:42.309113927 (14h21m41.898458904s) [noise=true , address=45.91.116.9:4263 , piece_id=F4QDS3EOSLFFCEDU6N56AMMQR4GHPWZ6TIAMO5SGVIMVMFA47EQQ, seg_pos={0 0}, upload=14872]: Done uploading piece. err: <nil>
Aug 8 00:01:42.309137942 (14h21m41.898482919s) [seg_pos={0 0}, upload=14872]: Segment reached optimal threshold of 13 pieces.
Aug 8 00:01:42.309315973 (14h21m41.89866099s) [noise=true , address=77.90.31.11:4555 , piece_id=OCZBWUJHEVLMUD6BNADJ7BOSHHZPESQJOSKDZ3T2MUSQRFTNI7KA, seg_pos={0 0}, upload=14872]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ): protocol: expected piece hash; context canceled; EOF
Aug 8 00:01:42.309402328 (14h21m41.898747305s) [noise=true , address=104.36.82.10:4079 , piece_id=262SP4DREL7FYDG6YOBFWDPWKYHZIQGDJKSX7XAIUJUTQWUIEEZQ, seg_pos={0 0}, upload=14872]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud): protocol: expected piece hash; context canceled; EOF
Aug 8 00:01:42.309425222 (14h21m41.898770229s) [noise=true , address=104.36.82.11:4115 , piece_id=S34RF36SERFPPSHBPYJ5V3FCR5ZE7RFIVEXBOTFRHPEEIUCHO2LQ, seg_pos={0 0}, upload=14872]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR): protocol: write tcp 188.166.225.170:42778->104.36.82.11:4115: use of closed network connection; write tcp 188.166.225.170:42778->104.36.82.11:4115: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42778->104.36.82.11:4115: use of closed network connection
Aug 8 00:01:42.309450360 (14h21m41.898795347s) [seg_pos={0 0}, upload=14872]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Aug 8 00:01:42.309540663 (14h21m41.89888565s) [upload=14872]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Aug 8 00:01:42.309548688 (14h21m41.898893735s) [upload=14872]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Aug 8 00:01:57.644545293 (14h21m57.2338903s) [upload=14872]: Done uploading segments...
Aug 8 00:01:57.644581281 (14h21m57.233926288s) [upload=14872]: Upload finished. err: metaclient: context deadline exceeded
Duration: 5.833103072s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
4 out of 60 is failed (threshold=3) [XXXX........................................................] duration:
5.667617841s
FAIL (2025-08-08 00:37:03 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.667640294s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [..X....X....................................................] duration:
6.159608967s
FAIL (2025-08-08 00:37:02 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.159660456s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
5 out of 60 is failed (threshold=3) [.....................X...X..........X.......X...X...........] duration:
5.913938289s
FAIL (2025-08-08 00:37:00 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/4M):: after a4399ade4fa2878d720011f9f1d393a7a350a6f1820ab363fc7588b62888e2c7 != before 879367dc26e769e454df7bfa7c99b4b3877306fa37488cef7c83620ea9a98ba6
======LOGS======
.BatchRequestItem_ObjectBegin
Aug 8 00:28:47.094828147 (14h48m46.684173134s) [upload=15346]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Aug 8 00:28:47.551561608 (14h48m47.140906595s) [seg_pos={0 0}, upload=15346]: Begin upload segment...
Aug 8 00:28:47.551608808 (14h48m47.140953795s) [seg_pos={0 0}, upload=15346]: Done begin upload segment.
Aug 8 00:28:47.551629437 (14h48m47.140974424s) [noise=true , address=104.36.82.18:4367 , piece_id=PVNJQVZ3QW5ZVHGQG3QSDLRRZV5BFGN6J2MC62JQ5P5ZJW65WW5Q, seg_pos={0 0}, upload=15346]: Uploading piece...
Aug 8 00:28:47.554243379 (14h48m47.143588366s) [noise=true , address=104.36.82.7:4937 , piece_id=HI26KBJIH6YFKH2NU655SRLUCFTDHNA4QR2THVJVIEDMHQTZSJLQ, seg_pos={0 0}, upload=15346]: Uploading piece...
Aug 8 00:28:47.554910236 (14h48m47.144255223s) [noise=true , address=104.36.82.20:4445 , piece_id=LOWLB4F4N7LCZ5C3JE6HRSSU4FUOKJNIFTHIHRNCJCEKHGG667GA, seg_pos={0 0}, upload=15346]: Uploading piece...
Aug 8 00:28:47.555304531 (14h48m47.144649518s) [noise=true , address=104.36.82.62:4413 , piece_id=XNYCVTWNEL27VZYCYSA6F5JIIEZMNUPHGLCOX5LVH6LHAFSNB7DA, seg_pos={0 0}, upload=15346]: Uploading piece...
Aug 8 00:28:47.555953072 (14h48m47.145298059s) [noise=true , address=104.36.82.43:4861 , piece_id=JMUXDXGFTHJHGHMNKXIWYNWG4QALWELUKH6DODKRKVUS53YBEYAQ, seg_pos={0 0}, upload=15346]: Uploading piece...
Aug 8 00:28:47.556452077 (14h48m47.145797064s) [noise=true , address=104.36.82.37:4680 , piece_id=XZNKDG5FITGWMR7U7WSVE5F3KYFMNXZ73AFFCB5NC2RF3CKCK7RQ, seg_pos={0 0}, upload=15346]: Uploading piece...
Aug 8 00:28:47.557051404 (14h48m47.146396391s) [noise=true , address=104.36.82.10:4085 , piece_id=RI6KTTGHEXBFGMW54NCT7MJFUYRBIGB62AOUS3DASGOA2D55P6SQ, seg_pos={0 0}, upload=15346]: Uploading piece...
Aug 8 00:28:47.557476658 (14h48m47.146821645s) [noise=true , address=104.36.82.8:4979 , piece_id=Z6YJ5QYHJJJLGUXGCIPGGE25OLKO25MPC5AQ53LMZ22RCB25QWXQ, seg_pos={0 0}, upload=15346]: Uploading piece...
Aug 8 00:28:47.557973529 (14h48m47.147318516s) [noise=true , address=104.36.82.61:4376 , piece_id=PRM6GANCIR4W32MFVJY5UQ7YBMXHVMNXBUYDNQQUHA45HR5JWQAQ, seg_pos={0 0}, upload=15346]: Uploading piece...
Aug 8 00:28:47.558963734 (14h48m47.148308721s) [noise=true , address=104.36.82.52:4105 , piece_id=XT46OYR6SIAZQ6L4GFYNKXQCNIETX6FYVM3SZRMUYFSICP66YXKA, seg_pos={0 0}, upload=15346]: Uploading piece...
Aug 8 00:28:47.559345425 (14h48m47.148690412s) [noise=true , address=104.36.82.4:4836 , piece_id=M2Y7QYLPP4FNSUTJZOF3U5ZYDBUDJL25O7ILENA47OYIHJIE7CJA, seg_pos={0 0}, upload=15346]: Uploading piece...
Aug 8 00:28:47.559751782 (14h48m47.149096759s) [noise=true , address=104.36.82.23:4464 , piece_id=KSV4RQE2YF4TL2VB466SPMQTWWO2P7TD624GQQYYHZDIUUBKAJUQ, seg_pos={0 0}, upload=15346]: Uploading piece...
Aug 8 00:28:47.560180823 (14h48m47.14952581s) [noise=true , address=104.36.82.60:4366 , piece_id=U2HK37YA6BWLVE7ABJ465K7PTEXGEWI72ICQWZRSXZ3KGASXRL3Q, seg_pos={0 0}, upload=15346]: Uploading piece...
Aug 8 00:28:47.561215644 (14h48m47.150560631s) [noise=true , address=104.36.82.47:4520 , piece_id=QGOCU74LKBEZ5IA76WKIM2ZY57MAUMSJDYF3U3L7WCM3JPPTV7YQ, seg_pos={0 0}, upload=15346]: Uploading piece...
Aug 8 00:28:47.561560894 (14h48m47.150905881s) [noise=true , address=104.36.82.4:4813 , piece_id=37ZR56FKL5MJB4ZBBNGFUTVCXX6WEDJTYKEVSWLL7JLRNVONV2CQ, seg_pos={0 0}, upload=15346]: Uploading piece...
Aug 8 00:28:47.561844247 (14h48m47.151189234s) [noise=true , address=104.36.82.59:4325 , piece_id=EIEJTAIZHKY4SWUZPA7EXYUWYQV3SU4NHVA7IUDRTW2CJHSIPQPA, seg_pos={0 0}, upload=15346]: Uploading piece...
Aug 8 00:28:47.562154400 (14h48m47.151499387s) [noise=true , address=79.127.201.215:4352 , piece_id=EC3UMETYKU6KUMG3VJQOJ2WJCTCDWQ4WBHKPHQAKT7NC3F6RWDXQ, seg_pos={0 0}, upload=15346]: Uploading piece...
Aug 8 00:28:47.562457500 (14h48m47.151802487s) [noise=true , address=104.36.82.44:4895 , piece_id=E364QDW2NRBMQPZGZHJWRZO6GJN3SHL2CRVAAD4KRFYS5YDLOYZA, seg_pos={0 0}, upload=15346]: Uploading piece...
Aug 8 00:28:49.073642290 (14h48m48.662987277s) [upload=15346]: Next returned nil segment
Aug 8 00:28:49.073651728 (14h48m48.662996715s) [upload=15346]: Waiting for error group managing segments...
Aug 8 00:28:49.485127304 (14h48m49.074472291s) [noise=true , address=79.127.201.215:4352 , piece_id=EC3UMETYKU6KUMG3VJQOJ2WJCTCDWQ4WBHKPHQAKT7NC3F6RWDXQ, seg_pos={0 0}, upload=15346]: Done uploading piece. err: <nil>
Aug 8 00:28:49.569397892 (14h48m49.158742879s) [noise=true , address=104.36.82.23:4464 , piece_id=KSV4RQE2YF4TL2VB466SPMQTWWO2P7TD624GQQYYHZDIUUBKAJUQ, seg_pos={0 0}, upload=15346]: Done uploading piece. err: <nil>
Aug 8 00:28:49.589957680 (14h48m49.179302667s) [noise=true , address=104.36.82.7:4937 , piece_id=HI26KBJIH6YFKH2NU655SRLUCFTDHNA4QR2THVJVIEDMHQTZSJLQ, seg_pos={0 0}, upload=15346]: Done uploading piece. err: <nil>
Aug 8 00:28:49.659215316 (14h48m49.248560303s) [noise=true , address=104.36.82.20:4445 , piece_id=LOWLB4F4N7LCZ5C3JE6HRSSU4FUOKJNIFTHIHRNCJCEKHGG667GA, seg_pos={0 0}, upload=15346]: Done uploading piece. err: <nil>
Aug 8 00:28:49.659539797 (14h48m49.248884784s) [noise=true , address=104.36.82.8:4979 , piece_id=Z6YJ5QYHJJJLGUXGCIPGGE25OLKO25MPC5AQ53LMZ22RCB25QWXQ, seg_pos={0 0}, upload=15346]: Done uploading piece. err: <nil>
Aug 8 00:28:49.699454667 (14h48m49.288799654s) [noise=true , address=104.36.82.4:4813 , piece_id=37ZR56FKL5MJB4ZBBNGFUTVCXX6WEDJTYKEVSWLL7JLRNVONV2CQ, seg_pos={0 0}, upload=15346]: Done uploading piece. err: <nil>
Aug 8 00:28:49.734069373 (14h48m49.32341436s) [noise=true , address=104.36.82.44:4895 , piece_id=E364QDW2NRBMQPZGZHJWRZO6GJN3SHL2CRVAAD4KRFYS5YDLOYZA, seg_pos={0 0}, upload=15346]: Done uploading piece. err: <nil>
Aug 8 00:28:49.759950614 (14h48m49.349295601s) [noise=true , address=104.36.82.4:4836 , piece_id=M2Y7QYLPP4FNSUTJZOF3U5ZYDBUDJL25O7ILENA47OYIHJIE7CJA, seg_pos={0 0}, upload=15346]: Done uploading piece. err: <nil>
Aug 8 00:28:49.787498666 (14h48m49.376843653s) [noise=true , address=104.36.82.10:4085 , piece_id=RI6KTTGHEXBFGMW54NCT7MJFUYRBIGB62AOUS3DASGOA2D55P6SQ, seg_pos={0 0}, upload=15346]: Done uploading piece. err: <nil>
Aug 8 00:28:49.874182563 (14h48m49.46352758s) [noise=true , address=104.36.82.52:4105 , piece_id=XT46OYR6SIAZQ6L4GFYNKXQCNIETX6FYVM3SZRMUYFSICP66YXKA, seg_pos={0 0}, upload=15346]: Done uploading piece. err: <nil>
Aug 8 00:28:49.960972762 (14h48m49.550317749s) [noise=true , address=104.36.82.59:4325 , piece_id=EIEJTAIZHKY4SWUZPA7EXYUWYQV3SU4NHVA7IUDRTW2CJHSIPQPA, seg_pos={0 0}, upload=15346]: Done uploading piece. err: <nil>
Aug 8 00:28:50.001273120 (14h48m49.590618117s) [noise=true , address=104.36.82.37:4680 , piece_id=XZNKDG5FITGWMR7U7WSVE5F3KYFMNXZ73AFFCB5NC2RF3CKCK7RQ, seg_pos={0 0}, upload=15346]: Done uploading piece. err: <nil>
Aug 8 00:28:50.571026048 (14h48m50.160371025s) [noise=true , address=104.36.82.62:4413 , piece_id=XNYCVTWNEL27VZYCYSA6F5JIIEZMNUPHGLCOX5LVH6LHAFSNB7DA, seg_pos={0 0}, upload=15346]: Done uploading piece. err: <nil>
Aug 8 00:28:51.294616675 (14h48m50.883961662s) [noise=true , address=104.36.82.61:4376 , piece_id=PRM6GANCIR4W32MFVJY5UQ7YBMXHVMNXBUYDNQQUHA45HR5JWQAQ, seg_pos={0 0}, upload=15346]: Done uploading piece. err: <nil>
Aug 8 00:28:51.294639959 (14h48m50.883984936s) [seg_pos={0 0}, upload=15346]: Segment reached optimal threshold of 14 pieces.
Aug 8 00:28:51.294832458 (14h48m50.884177445s) [noise=true , address=104.36.82.47:4520 , piece_id=QGOCU74LKBEZ5IA76WKIM2ZY57MAUMSJDYF3U3L7WCM3JPPTV7YQ, seg_pos={0 0}, upload=15346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12eNRsboM5zFiyBVPLuQfgb5QSAc5bZFizK9KJHudNd9nstVGyy): protocol: expected piece hash; context canceled; EOF
Aug 8 00:28:51.295062047 (14h48m50.884407034s) [noise=true , address=104.36.82.60:4366 , piece_id=U2HK37YA6BWLVE7ABJ465K7PTEXGEWI72ICQWZRSXZ3KGASXRL3Q, seg_pos={0 0}, upload=15346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12NBUjBGU2MCjDx8iZKvzCvhL6GHCfdXfihC5H2uoBUmzhz9uoV): protocol: write tcp 188.166.225.170:57292->104.36.82.60:4366: use of closed network connection; write tcp 188.166.225.170:57292->104.36.82.60:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57292->104.36.82.60:4366: use of closed network connection
Aug 8 00:28:51.295101312 (14h48m50.884446299s) [noise=true , address=104.36.82.43:4861 , piece_id=JMUXDXGFTHJHGHMNKXIWYNWG4QALWELUKH6DODKRKVUS53YBEYAQ, seg_pos={0 0}, upload=15346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122HC6BetMaBDNGEaHVwmadWtV1nCHx3vGBK2SWVDEQS1iBCz4o): protocol: write tcp 188.166.225.170:53912->104.36.82.43:4861: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 8 00:28:51.295131720 (14h48m50.884476707s) [noise=true , address=104.36.82.18:4367 , piece_id=PVNJQVZ3QW5ZVHGQG3QSDLRRZV5BFGN6J2MC62JQ5P5ZJW65WW5Q, seg_pos={0 0}, upload=15346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6): protocol: read tcp 188.166.225.170:32868->104.36.82.18:4367: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 8 00:28:51.295147570 (14h48m50.884492557s) [seg_pos={0 0}, upload=15346]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Aug 8 00:28:51.295216993 (14h48m50.88456198s) [upload=15346]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Aug 8 00:28:51.295226662 (14h48m50.884571649s) [upload=15346]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Aug 8 00:28:51.599168829 (14h48m51.188513806s) [upload=15346]: Done uploading segments...
Aug 8 00:28:51.599190280 (14h48m51.188535267s) [upload=15346]: Upload finished. err: <nil>
Duration: 5.913961083s
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
4 out of 60 is failed (threshold=3) [.........X.XX...X...........................................] duration:
6.303843973s
FAIL (2025-08-08 00:36:56 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: encryption: context deadline exceeded
======LOGS======
ZYSDFBHFTDOKPS73M57QA, seg_pos={0 0}, upload=14870]: Uploading piece...
Aug 8 00:01:16.246137011 (14h21m15.835481998s) [noise=true , address=104.36.82.9:4024 , piece_id=FM7MVIPG7C264O6Q64RUFOKZ2E5NMTIPWSIVMH5SWZUSCMFI7UGQ, seg_pos={0 0}, upload=14870]: Uploading piece...
Aug 8 00:01:16.246560380 (14h21m15.835905367s) [noise=true , address=64.49.13.169:4705 , piece_id=3MVXPRNKBBUSYAH2E7PCIWXDCUFQQ547IPOYIDLQ3T7QDYPBJA7A, seg_pos={0 0}, upload=14870]: Uploading piece...
Aug 8 00:01:16.247408974 (14h21m15.836753961s) [noise=true , address=45.91.116.5:4302 , piece_id=6RMPHN4LYSZVM6XKAUGSU7GY3TIKBGLKK4KNFMZQ2FMVEEVWV2LA, seg_pos={0 0}, upload=14870]: Uploading piece...
Aug 8 00:01:16.247678410 (14h21m15.837023397s) [noise=true , address=79.127.219.51:4181 , piece_id=4LYG6AL2BWD3UYQQAUUGZNJ7HUFXSHZ2TULWJOYSWTJLNVGHVKLQ, seg_pos={0 0}, upload=14870]: Uploading piece...
Aug 8 00:01:16.247919882 (14h21m15.837264869s) [noise=true , address=104.36.82.22:4512 , piece_id=7VOWVUPU3QVXIKML65NHX3NL7AC26QSDGHSSCRLUXIVZRNUPPKRA, seg_pos={0 0}, upload=14870]: Uploading piece...
Aug 8 00:01:16.247943888 (14h21m15.837288875s) [noise=true , address=104.36.82.20:4466 , piece_id=PLG7Z2WTYKUNB347NPT4ESIDK2NHCP66NPZJOXYL4A6L5OMFQIOQ, seg_pos={0 0}, upload=14870]: Uploading piece...
Aug 8 00:01:16.248293266 (14h21m15.837638253s) [noise=true , address=104.36.82.11:4143 , piece_id=E4AW52SZYZVRWPU4L56INK3IRNG7UNO5RRIRJANLMEPZQVDUHCEQ, seg_pos={0 0}, upload=14870]: Uploading piece...
Aug 8 00:01:16.248420540 (14h21m15.837765527s) [noise=true , address=64.49.13.139:4471 , piece_id=FPVXYIRNZXODU7N5RPCDOVVUECXWRTXCLINEXMOJTKNHF6YN46YQ, seg_pos={0 0}, upload=14870]: Uploading piece...
Aug 8 00:01:16.248557311 (14h21m15.837902288s) [noise=true , address=104.36.82.37:4659 , piece_id=627LA2AVL53NXFMDWSLGBO5GUKDAIGGSOPBFRPYIPYC5EAQC6P7Q, seg_pos={0 0}, upload=14870]: Uploading piece...
Aug 8 00:01:16.248758506 (14h21m15.838103493s) [noise=true , address=104.36.82.58:4275 , piece_id=VBV72SK36XIHMWSZTJJE4MZPLD5VZ6Z6UPBHK4A2ZI2C3O3RCVXA, seg_pos={0 0}, upload=14870]: Uploading piece...
Aug 8 00:01:16.248897792 (14h21m15.838242779s) [noise=true , address=104.36.82.55:4209 , piece_id=PJ4RK52LARBDG6WEVKXPZPX3RKJVIRSDCZY3B6CVCIUZO6BTWLBQ, seg_pos={0 0}, upload=14870]: Uploading piece...
Aug 8 00:01:16.248989629 (14h21m15.838334616s) [noise=true , address=104.36.82.43:4855 , piece_id=MJYC5ZGE6P72UYNYDRMH3BAZ6KIY57SA3EKJR54W6HPZNO3AE4IA, seg_pos={0 0}, upload=14870]: Uploading piece...
Aug 8 00:01:16.249186575 (14h21m15.838531552s) [noise=true , address=104.36.82.46:4487 , piece_id=6BZYNRXKBHFL3RQTGN3ZN4RV2O4USZ24GGCVSKFZWNS6N32L3NGQ, seg_pos={0 0}, upload=14870]: Uploading piece...
Aug 8 00:01:16.249273772 (14h21m15.838618759s) [noise=true , address=104.36.82.7:4920 , piece_id=BN5NCMLEGDVRIHO6YOWM5FNPCGFYC3BDKLOU7DIP6K6DYZXJ5A5A, seg_pos={0 0}, upload=14870]: Uploading piece...
Aug 8 00:01:16.249459598 (14h21m15.838804585s) [noise=true , address=77.90.31.14:4672 , piece_id=3ZYE5SLRNLY2ALXRR4QGCZT4SQ37U7FIAUOCVT63HQSBKPKV4UGQ, seg_pos={0 0}, upload=14870]: Uploading piece...
Aug 8 00:01:16.249717240 (14h21m15.839062227s) [noise=true , address=104.36.82.5:4844 , piece_id=REDGG4W6TJYEPEEPR65O5AVQDU2QMBNRA6HQQRUJA6JP7IS5T7YQ, seg_pos={0 0}, upload=14870]: Uploading piece...
Aug 8 00:01:17.413257645 (14h21m17.002602632s) [upload=14870]: Next returned nil segment
Aug 8 00:01:17.413273324 (14h21m17.002618311s) [upload=14870]: Waiting for error group managing segments...
Aug 8 00:01:17.710897294 (14h21m17.300242281s) [noise=true , address=64.49.13.169:4705 , piece_id=3MVXPRNKBBUSYAH2E7PCIWXDCUFQQ547IPOYIDLQ3T7QDYPBJA7A, seg_pos={0 0}, upload=14870]: Done uploading piece. err: <nil>
Aug 8 00:01:17.715058896 (14h21m17.304403883s) [noise=true , address=64.49.13.139:4471 , piece_id=FPVXYIRNZXODU7N5RPCDOVVUECXWRTXCLINEXMOJTKNHF6YN46YQ, seg_pos={0 0}, upload=14870]: Done uploading piece. err: <nil>
Aug 8 00:01:18.126218161 (14h21m17.715563148s) [noise=true , address=79.127.201.215:4356 , piece_id=VWNEIHNI4KPPF4QLW36INIWTUI2CD6MZYSDFBHFTDOKPS73M57QA, seg_pos={0 0}, upload=14870]: Done uploading piece. err: <nil>
Aug 8 00:01:18.139272367 (14h21m17.728617354s) [noise=true , address=79.127.219.51:4181 , piece_id=4LYG6AL2BWD3UYQQAUUGZNJ7HUFXSHZ2TULWJOYSWTJLNVGHVKLQ, seg_pos={0 0}, upload=14870]: Done uploading piece. err: <nil>
Aug 8 00:01:18.221269031 (14h21m17.810614018s) [noise=true , address=104.36.82.43:4855 , piece_id=MJYC5ZGE6P72UYNYDRMH3BAZ6KIY57SA3EKJR54W6HPZNO3AE4IA, seg_pos={0 0}, upload=14870]: Done uploading piece. err: <nil>
Aug 8 00:01:18.257350144 (14h21m17.846695121s) [noise=true , address=104.36.82.37:4659 , piece_id=627LA2AVL53NXFMDWSLGBO5GUKDAIGGSOPBFRPYIPYC5EAQC6P7Q, seg_pos={0 0}, upload=14870]: Done uploading piece. err: <nil>
Aug 8 00:01:18.258435089 (14h21m17.847780076s) [noise=true , address=104.36.82.44:4876 , piece_id=BA2G42OXQD3NLONUVD475S2BNC7T37T5PHPE3K3E5HJAJP4W5MKQ, seg_pos={0 0}, upload=14870]: Done uploading piece. err: <nil>
Aug 8 00:01:18.287010923 (14h21m17.87635591s) [noise=true , address=104.36.82.7:4920 , piece_id=BN5NCMLEGDVRIHO6YOWM5FNPCGFYC3BDKLOU7DIP6K6DYZXJ5A5A, seg_pos={0 0}, upload=14870]: Done uploading piece. err: <nil>
Aug 8 00:01:18.348793001 (14h21m17.938137988s) [noise=true , address=104.36.82.11:4143 , piece_id=E4AW52SZYZVRWPU4L56INK3IRNG7UNO5RRIRJANLMEPZQVDUHCEQ, seg_pos={0 0}, upload=14870]: Done uploading piece. err: <nil>
Aug 8 00:01:18.378129459 (14h21m17.967474436s) [noise=true , address=104.36.82.58:4275 , piece_id=VBV72SK36XIHMWSZTJJE4MZPLD5VZ6Z6UPBHK4A2ZI2C3O3RCVXA, seg_pos={0 0}, upload=14870]: Done uploading piece. err: <nil>
Aug 8 00:01:18.386414901 (14h21m17.975759888s) [noise=true , address=104.36.82.46:4487 , piece_id=6BZYNRXKBHFL3RQTGN3ZN4RV2O4USZ24GGCVSKFZWNS6N32L3NGQ, seg_pos={0 0}, upload=14870]: Done uploading piece. err: <nil>
Aug 8 00:01:18.516150823 (14h21m18.10549581s) [noise=true , address=104.36.82.9:4024 , piece_id=FM7MVIPG7C264O6Q64RUFOKZ2E5NMTIPWSIVMH5SWZUSCMFI7UGQ, seg_pos={0 0}, upload=14870]: Done uploading piece. err: <nil>
Aug 8 00:01:18.626647204 (14h21m18.215992191s) [noise=true , address=77.90.31.14:4672 , piece_id=3ZYE5SLRNLY2ALXRR4QGCZT4SQ37U7FIAUOCVT63HQSBKPKV4UGQ, seg_pos={0 0}, upload=14870]: Done uploading piece. err: <nil>
Aug 8 00:01:19.042108811 (14h21m18.631453798s) [noise=true , address=104.36.82.22:4512 , piece_id=7VOWVUPU3QVXIKML65NHX3NL7AC26QSDGHSSCRLUXIVZRNUPPKRA, seg_pos={0 0}, upload=14870]: Done uploading piece. err: <nil>
Aug 8 00:01:19.042136654 (14h21m18.631481641s) [seg_pos={0 0}, upload=14870]: Segment reached optimal threshold of 14 pieces.
Aug 8 00:01:19.042375541 (14h21m18.631720518s) [noise=true , address=45.91.116.5:4302 , piece_id=6RMPHN4LYSZVM6XKAUGSU7GY3TIKBGLKK4KNFMZQ2FMVEEVWV2LA, seg_pos={0 0}, upload=14870]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pJWhSLza8WaMuvvDwH4nae57UxQmfCdDvAkQ3v25MgPYHoZXx): protocol: expected piece hash; context canceled; EOF
Aug 8 00:01:19.042485912 (14h21m18.6318309s) [noise=true , address=104.36.82.55:4209 , piece_id=PJ4RK52LARBDG6WEVKXPZPX3RKJVIRSDCZY3B6CVCIUZO6BTWLBQ, seg_pos={0 0}, upload=14870]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12EmE2BgEZTG1LTsZ5p9ghqDSfg5ZQATt31Tqg5jZ5KQE2UjDnR): protocol: rpc: dial tcp 104.36.82.55:4209: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 8 00:01:19.042532361 (14h21m18.631877348s) [noise=true , address=104.36.82.20:4466 , piece_id=PLG7Z2WTYKUNB347NPT4ESIDK2NHCP66NPZJOXYL4A6L5OMFQIOQ, seg_pos={0 0}, upload=14870]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SqNLPRWY7fJRS6F9h4B4qVFv6U5jyWkCy3Um7jfjEdCShEjsD): protocol: write tcp 188.166.225.170:37056->104.36.82.20:4466: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 8 00:01:19.042616462 (14h21m18.631961449s) [noise=true , address=104.36.82.5:4844 , piece_id=REDGG4W6TJYEPEEPR65O5AVQDU2QMBNRA6HQQRUJA6JP7IS5T7YQ, seg_pos={0 0}, upload=14870]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12iovpNdbidkiv9mEHaAPjBubGmoqr4sUJWm8Cb64LzS6wuwLWa): protocol: rpc: dial tcp 104.36.82.5:4844: operation was canceled; conn closed: op; piecestore: piecestore close: context canceled
Aug 8 00:01:19.042628876 (14h21m18.631973863s) [seg_pos={0 0}, upload=14870]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Aug 8 00:01:19.042699029 (14h21m18.632044017s) [upload=14870]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Aug 8 00:01:19.042703128 (14h21m18.632048115s) [upload=14870]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Aug 8 00:01:19.344335316 (14h21m18.933680293s) [upload=14870]: Done uploading segments...
Aug 8 00:01:19.344353089 (14h21m18.933698077s) [upload=14870]: Upload finished. err: <nil>
Aug 8 00:02:30.778225454 (14h22m30.367570441s) [upload=14892]: Starting upload
Aug 8 00:02:30.778283114 (14h22m30.367628151s) [upload=14892]: Done starting upload
Aug 8 00:02:30.778411841 (14h22m30.367756828s) [upload=14892]: Uploading segments...
Aug 8 00:02:30.778539616 (14h22m30.367884623s) [upload=14892]: Got next segment. Inline: false
Aug 8 00:02:30.778583389 (14h22m30.367928376s) [upload=14892]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Aug 8 00:02:30.778587717 (14h22m30.367932704s) [upload=14892]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Aug 8 00:03:30.135980490 (14h23m29.725325467s) [upload=14892]: Next segment err: context deadline exceeded
Aug 8 00:03:30.136177137 (14h23m29.725522124s) [upload=14892]: Done uploading segments...
Aug 8 00:03:30.136196834 (14h23m29.725541821s) [upload=14892]: Upload finished. err: context deadline exceeded
Duration: 6.303867508s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)antiblip:
2 out of 60 is failed (threshold=3) [X..X........................................................] duration:
1.865648418s
FAIL (2025-08-08 00:36:53 UTC)
orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded
Duration: 1.865681342s
core
uplink
us1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)antiblip:
6 out of 60 is failed (threshold=3) [...........X.XXXX....X......................................] duration:
1.859371723s
FAIL (2025-08-08 00:36:52 UTC)
orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 9109253s) [upload=14753]: Starting upload Aug 7 23:54:04.489828037 (14h14m4.079173014s) [upload=14753]: Done starting upload Aug 7 23:54:04.489877612 (14h14m4.079222589s) [upload=14753]: Uploading segments... Aug 7 23:54:04.489903933 (14h14m4.07924892s) [upload=14753]: Got next segment. Inline: true Aug 7 23:54:04.489907469 (14h14m4.079252446s) [upload=14753]: Waiting for error group managing segments... Aug 7 23:54:04.489939651 (14h14m4.079284638s) [upload=14753]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 23:54:04.489943298 (14h14m4.079288285s) [upload=14753]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 23:54:04.489945472 (14h14m4.079290459s) [upload=14753]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 23:54:04.778726062 (14h14m4.368071049s) [upload=14753]: Done uploading segments... Aug 7 23:54:04.778744989 (14h14m4.368089976s) [upload=14753]: Upload finished. err: <nil> Aug 7 23:55:14.199719855 (14h15m13.789064892s) [upload=14770]: Starting upload Aug 7 23:55:14.199777645 (14h15m13.789122632s) [upload=14770]: Done starting upload Aug 7 23:55:14.199824415 (14h15m13.789169392s) [upload=14770]: Uploading segments... Aug 7 23:55:14.199852819 (14h15m13.789197806s) [upload=14770]: Got next segment. Inline: true Aug 7 23:55:14.199856436 (14h15m13.789201423s) [upload=14770]: Waiting for error group managing segments... Aug 7 23:55:14.199889419 (14h15m13.789234406s) [upload=14770]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 23:55:14.199893557 (14h15m13.789238544s) [upload=14770]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 23:55:14.199895951 (14h15m13.789240938s) [upload=14770]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 23:56:13.542516296 (14h16m13.131861283s) [upload=14770]: Done uploading segments... Aug 7 23:56:13.542536334 (14h16m13.131881321s) [upload=14770]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:56:14.428677736 (14h16m14.018022723s) [upload=14788]: Starting upload Aug 7 23:56:14.428730978 (14h16m14.018075965s) [upload=14788]: Done starting upload Aug 7 23:56:14.428789289 (14h16m14.018134276s) [upload=14788]: Uploading segments... Aug 7 23:56:14.428819026 (14h16m14.018164013s) [upload=14788]: Got next segment. Inline: true Aug 7 23:56:14.428822873 (14h16m14.01816785s) [upload=14788]: Waiting for error group managing segments... Aug 7 23:56:14.428855215 (14h16m14.018200202s) [upload=14788]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 23:56:14.428858562 (14h16m14.018203549s) [upload=14788]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 23:56:14.428860736 (14h16m14.018205723s) [upload=14788]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 23:57:13.781984834 (14h17m13.371329821s) [upload=14788]: Done uploading segments... Aug 7 23:57:13.782003370 (14h17m13.371348357s) [upload=14788]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:57:29.079359478 (14h17m28.668704466s) [upload=14808]: Starting upload Aug 7 23:57:29.079416147 (14h17m28.668761134s) [upload=14808]: Done starting upload Aug 7 23:57:29.079461845 (14h17m28.668806822s) [upload=14808]: Uploading segments... Aug 7 23:57:29.079488876 (14h17m28.668833863s) [upload=14808]: Got next segment. Inline: true Aug 7 23:57:29.079492263 (14h17m28.66883725s) [upload=14808]: Waiting for error group managing segments... Aug 7 23:57:29.079523392 (14h17m28.668868379s) [upload=14808]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 23:57:29.079527049 (14h17m28.668872036s) [upload=14808]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 23:57:29.079529484 (14h17m28.668874461s) [upload=14808]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 23:58:28.439377526 (14h18m28.028722513s) [upload=14808]: Done uploading segments... Aug 7 23:58:28.439395721 (14h18m28.028740708s) [upload=14808]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:58:29.333669502 (14h18m28.923014489s) [upload=14825]: Starting upload Aug 7 23:58:29.333742581 (14h18m28.923087568s) [upload=14825]: Done starting upload Aug 7 23:58:29.333835740 (14h18m28.923180727s) [upload=14825]: Uploading segments... Aug 7 23:58:29.333878000 (14h18m28.923222987s) [upload=14825]: Got next segment. Inline: true Aug 7 23:58:29.333881998 (14h18m28.923226975s) [upload=14825]: Waiting for error group managing segments... Aug 7 23:58:29.333911455 (14h18m28.923256442s) [upload=14825]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 23:58:29.333917496 (14h18m28.923262473s) [upload=14825]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 23:58:29.333919741 (14h18m28.923264728s) [upload=14825]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 23:59:28.692415040 (14h19m28.281760027s) [upload=14825]: Done uploading segments... Aug 7 23:59:28.692444777 (14h19m28.281789764s) [upload=14825]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:59:29.562470053 (14h19m29.15181504s) [upload=14840]: Starting upload Aug 7 23:59:29.562538985 (14h19m29.151883972s) [upload=14840]: Done starting upload Aug 7 23:59:29.562614629 (14h19m29.151959616s) [upload=14840]: Uploading segments... Aug 7 23:59:29.562650589 (14h19m29.151995566s) [upload=14840]: Got next segment. Inline: true Aug 7 23:59:29.562655498 (14h19m29.152000485s) [upload=14840]: Waiting for error group managing segments... Aug 7 23:59:29.562686507 (14h19m29.152031494s) [upload=14840]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 23:59:29.562690264 (14h19m29.152035241s) [upload=14840]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 23:59:29.562692428 (14h19m29.152037405s) [upload=14840]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 23:59:29.822497550 (14h19m29.411842537s) [upload=14840]: Done uploading segments... Aug 7 23:59:29.822514693 (14h19m29.41185968s) [upload=14840]: Upload finished. err: <nil> Aug 8 00:00:35.820554363 (14h20m35.40989935s) [upload=14856]: Starting upload Aug 8 00:00:35.820610611 (14h20m35.409955598s) [upload=14856]: Done starting upload Aug 8 00:00:35.820669503 (14h20m35.4100145s) [upload=14856]: Uploading segments... Aug 8 00:00:35.820742333 (14h20m35.41008732s) [upload=14856]: Got next segment. Inline: true Aug 8 00:00:35.820747282 (14h20m35.410092269s) [upload=14856]: Waiting for error group managing segments... Aug 8 00:00:35.820788371 (14h20m35.410133358s) [upload=14856]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 8 00:00:35.820793050 (14h20m35.410138037s) [upload=14856]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 8 00:00:35.820795485 (14h20m35.410140472s) [upload=14856]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 8 00:00:36.084321816 (14h20m35.673666803s) [upload=14856]: Done uploading segments... Aug 8 00:00:36.084338618 (14h20m35.673683615s) [upload=14856]: Upload finished. err: <nil> Aug 8 00:01:14.496466774 (14h21m14.085811781s) [upload=14868]: Starting upload Aug 8 00:01:14.496556174 (14h21m14.085901162s) [upload=14868]: Done starting upload Aug 8 00:01:14.496609206 (14h21m14.085954183s) [upload=14868]: Uploading segments... Aug 8 00:01:14.496639985 (14h21m14.085984972s) [upload=14868]: Got next segment. Inline: true Aug 8 00:01:14.496643592 (14h21m14.085988579s) [upload=14868]: Waiting for error group managing segments... Aug 8 00:01:14.496678638 (14h21m14.086023625s) [upload=14868]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 8 00:01:14.496682386 (14h21m14.086027373s) [upload=14868]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 8 00:01:14.496684681 (14h21m14.086029658s) [upload=14868]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 8 00:01:14.760681029 (14h21m14.350026016s) [upload=14868]: Done uploading segments... Aug 8 00:01:14.760702580 (14h21m14.350047567s) [upload=14868]: Upload finished. err: <nil> Aug 8 00:01:50.577049881 (14h21m50.166394858s) [upload=14878]: Starting upload Aug 8 00:01:50.577100538 (14h21m50.166445525s) [upload=14878]: Done starting upload Aug 8 00:01:50.577180091 (14h21m50.166525078s) [upload=14878]: Uploading segments... Aug 8 00:01:50.577228012 (14h21m50.166573s) [upload=14878]: Got next segment. Inline: true Aug 8 00:01:50.577231469 (14h21m50.166576456s) [upload=14878]: Waiting for error group managing segments... Aug 8 00:01:50.577270633 (14h21m50.166615621s) [upload=14878]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 8 00:01:50.577274721 (14h21m50.166619708s) [upload=14878]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 8 00:01:50.577276906 (14h21m50.166621893s) [upload=14878]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 8 00:01:50.863237339 (14h21m50.452582326s) [upload=14878]: Done uploading segments... Aug 8 00:01:50.863254973 (14h21m50.452599961s) [upload=14878]: Upload finished. err: <nil> Aug 8 00:02:41.474648957 (14h22m41.063993944s) [upload=14899]: Starting upload Aug 8 00:02:41.474716297 (14h22m41.064061274s) [upload=14899]: Done starting upload Aug 8 00:02:41.474771743 (14h22m41.06411673s) [upload=14899]: Uploading segments... Aug 8 00:02:41.474801099 (14h22m41.064146086s) [upload=14899]: Got next segment. Inline: true Aug 8 00:02:41.474804966 (14h22m41.064149953s) [upload=14899]: Waiting for error group managing segments... Aug 8 00:02:41.474841336 (14h22m41.064186323s) [upload=14899]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 8 00:02:41.474845214 (14h22m41.064190201s) [upload=14899]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 8 00:02:41.474847558 (14h22m41.064192535s) [upload=14899]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 8 00:03:40.835731182 (14h23m40.425076189s) [upload=14899]: Done uploading segments... Aug 8 00:03:40.835778212 (14h23m40.425123199s) [upload=14899]: Upload finished. err: metaclient: context deadline exceeded
Duration: 1.859412682s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)antiblip:
7 out of 60 is failed (threshold=3) [XX...XXXXX..................................................] duration:
1.893036639s
FAIL (2025-08-08 00:36:40 UTC)
orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== m24.001855996s) [upload=14742]: Done starting upload Aug 7 23:53:24.412569290 (14h13m24.001914277s) [upload=14742]: Uploading segments... Aug 7 23:53:24.412638803 (14h13m24.001983831s) [upload=14742]: Got next segment. Inline: true Aug 7 23:53:24.412644354 (14h13m24.001989341s) [upload=14742]: Waiting for error group managing segments... Aug 7 23:53:24.412680903 (14h13m24.002025891s) [upload=14742]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 23:53:24.412685442 (14h13m24.002030429s) [upload=14742]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 23:53:24.412688137 (14h13m24.002033114s) [upload=14742]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 23:53:53.772057432 (14h13m53.361402419s) [upload=14742]: Done uploading segments... Aug 7 23:53:53.772119881 (14h13m53.361464908s) [upload=14742]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:54:30.496202349 (14h14m30.085547336s) [upload=14761]: Starting upload Aug 7 23:54:30.496270259 (14h14m30.085615246s) [upload=14761]: Done starting upload Aug 7 23:54:30.496341565 (14h14m30.085686552s) [upload=14761]: Uploading segments... Aug 7 23:54:30.496378957 (14h14m30.085723944s) [upload=14761]: Got next segment. Inline: true Aug 7 23:54:30.496383836 (14h14m30.085728823s) [upload=14761]: Waiting for error group managing segments... Aug 7 23:54:30.496441346 (14h14m30.085786333s) [upload=14761]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 23:54:30.496446426 (14h14m30.085791413s) [upload=14761]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 23:54:30.496449551 (14h14m30.085794539s) [upload=14761]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 23:54:59.847423574 (14h14m59.436768561s) [upload=14761]: Done uploading segments... Aug 7 23:54:59.847456066 (14h14m59.436801043s) [upload=14761]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:55:40.612588843 (14h15m40.20193383s) [upload=14781]: Starting upload Aug 7 23:55:40.612648929 (14h15m40.201993906s) [upload=14781]: Done starting upload Aug 7 23:55:40.612717029 (14h15m40.202062016s) [upload=14781]: Uploading segments... Aug 7 23:55:40.612750823 (14h15m40.20209581s) [upload=14781]: Got next segment. Inline: true Aug 7 23:55:40.612754510 (14h15m40.202099487s) [upload=14781]: Waiting for error group managing segments... Aug 7 23:55:40.612786883 (14h15m40.20213186s) [upload=14781]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 23:55:40.612790891 (14h15m40.202135868s) [upload=14781]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 23:55:40.612793075 (14h15m40.202138062s) [upload=14781]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 23:55:40.883469345 (14h15m40.472814333s) [upload=14781]: Done uploading segments... Aug 7 23:55:40.883486387 (14h15m40.472831375s) [upload=14781]: Upload finished. err: <nil> Aug 7 23:56:17.423712239 (14h16m17.013057226s) [upload=14789]: Starting upload Aug 7 23:56:17.423767895 (14h16m17.013112882s) [upload=14789]: Done starting upload Aug 7 23:56:17.423814835 (14h16m17.013159822s) [upload=14789]: Uploading segments... Aug 7 23:56:17.423847678 (14h16m17.013192665s) [upload=14789]: Got next segment. Inline: true Aug 7 23:56:17.423851335 (14h16m17.013196322s) [upload=14789]: Waiting for error group managing segments... Aug 7 23:56:17.423883377 (14h16m17.013228364s) [upload=14789]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 23:56:17.423887054 (14h16m17.013232041s) [upload=14789]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 23:56:17.423889258 (14h16m17.013234245s) [upload=14789]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 23:56:17.689327115 (14h16m17.278672102s) [upload=14789]: Done uploading segments... Aug 7 23:56:17.689345711 (14h16m17.278690688s) [upload=14789]: Upload finished. err: <nil> Aug 7 23:56:58.288741281 (14h16m57.878086268s) [upload=14801]: Starting upload Aug 7 23:56:58.288797098 (14h16m57.878142085s) [upload=14801]: Done starting upload Aug 7 23:56:58.288851010 (14h16m57.878195987s) [upload=14801]: Uploading segments... Aug 7 23:56:58.288882281 (14h16m57.878227268s) [upload=14801]: Got next segment. Inline: true Aug 7 23:56:58.288887911 (14h16m57.878232898s) [upload=14801]: Waiting for error group managing segments... Aug 7 23:56:58.288919792 (14h16m57.878264769s) [upload=14801]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 23:56:58.288923970 (14h16m57.878268957s) [upload=14801]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 23:56:58.288926485 (14h16m57.878271462s) [upload=14801]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 23:56:58.556529816 (14h16m58.145874803s) [upload=14801]: Done uploading segments... Aug 7 23:56:58.556544975 (14h16m58.145889962s) [upload=14801]: Upload finished. err: <nil> Aug 7 23:57:18.362794018 (14h17m17.952138995s) [upload=14806]: Starting upload Aug 7 23:57:18.362883679 (14h17m17.952228666s) [upload=14806]: Done starting upload Aug 7 23:57:18.362928104 (14h17m17.952273092s) [upload=14806]: Uploading segments... Aug 7 23:57:18.362957320 (14h17m17.952302307s) [upload=14806]: Got next segment. Inline: true Aug 7 23:57:18.362961077 (14h17m17.952306064s) [upload=14806]: Waiting for error group managing segments... Aug 7 23:57:18.363018407 (14h17m17.952363394s) [upload=14806]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 23:57:18.363022404 (14h17m17.952367392s) [upload=14806]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 23:57:18.363024739 (14h17m17.952369716s) [upload=14806]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 23:57:47.715297444 (14h17m47.304642431s) [upload=14806]: Done uploading segments... Aug 7 23:57:47.715410370 (14h17m47.304755357s) [upload=14806]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:57:53.375540080 (14h17m52.964885068s) [upload=14813]: Starting upload Aug 7 23:57:53.375600245 (14h17m52.964945232s) [upload=14813]: Done starting upload Aug 7 23:57:53.375654389 (14h17m52.964999376s) [upload=14813]: Uploading segments... Aug 7 23:57:53.375686622 (14h17m52.965031609s) [upload=14813]: Got next segment. Inline: true Aug 7 23:57:53.375690759 (14h17m52.965035746s) [upload=14813]: Waiting for error group managing segments... Aug 7 23:57:53.375725876 (14h17m52.965070863s) [upload=14813]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 23:57:53.375730134 (14h17m52.965075121s) [upload=14813]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 23:57:53.375734372 (14h17m52.965079359s) [upload=14813]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 23:58:22.726838464 (14h18m22.316183451s) [upload=14813]: Done uploading segments... Aug 7 23:58:22.726860075 (14h18m22.316205062s) [upload=14813]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:58:23.621870384 (14h18m23.211215371s) [upload=14822]: Starting upload Aug 7 23:58:23.621952000 (14h18m23.211296987s) [upload=14822]: Done starting upload Aug 7 23:58:23.622003359 (14h18m23.211348346s) [upload=14822]: Uploading segments... Aug 7 23:58:23.622043436 (14h18m23.211388413s) [upload=14822]: Got next segment. Inline: true Aug 7 23:58:23.622046772 (14h18m23.211391759s) [upload=14822]: Waiting for error group managing segments... Aug 7 23:58:23.622078573 (14h18m23.21142356s) [upload=14822]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 23:58:23.622082230 (14h18m23.211427217s) [upload=14822]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 23:58:23.622084534 (14h18m23.211429511s) [upload=14822]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 23:58:52.983677879 (14h18m52.573022876s) [upload=14822]: Done uploading segments... Aug 7 23:58:52.983713708 (14h18m52.573058685s) [upload=14822]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:59:11.446858470 (14h19m11.036203457s) [upload=14835]: Starting upload Aug 7 23:59:11.446906493 (14h19m11.03625148s) [upload=14835]: Done starting upload Aug 7 23:59:11.446947491 (14h19m11.036292479s) [upload=14835]: Uploading segments... Aug 7 23:59:11.446975475 (14h19m11.036320462s) [upload=14835]: Got next segment. Inline: true Aug 7 23:59:11.446978751 (14h19m11.036323738s) [upload=14835]: Waiting for error group managing segments... Aug 7 23:59:11.447008889 (14h19m11.036353876s) [upload=14835]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 23:59:11.447013638 (14h19m11.036358625s) [upload=14835]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 23:59:11.447016212 (14h19m11.0363612s) [upload=14835]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 23:59:40.799397635 (14h19m40.388742622s) [upload=14835]: Done uploading segments... Aug 7 23:59:40.799413266 (14h19m40.388758243s) [upload=14835]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:59:59.523080643 (14h19m59.11242563s) [upload=14847]: Starting upload Aug 7 23:59:59.523152089 (14h19m59.112497076s) [upload=14847]: Done starting upload Aug 7 23:59:59.523213727 (14h19m59.112558714s) [upload=14847]: Uploading segments... Aug 7 23:59:59.523251349 (14h19m59.112596336s) [upload=14847]: Got next segment. Inline: true Aug 7 23:59:59.523255287 (14h19m59.112600274s) [upload=14847]: Waiting for error group managing segments... Aug 7 23:59:59.523300013 (14h19m59.112645s) [upload=14847]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 23:59:59.523304251 (14h19m59.112649238s) [upload=14847]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 23:59:59.523307146 (14h19m59.112652133s) [upload=14847]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 8 00:00:28.877190143 (14h20m28.46653516s) [upload=14847]: Done uploading segments... Aug 8 00:00:28.877218477 (14h20m28.466563505s) [upload=14847]: Upload finished. err: metaclient: context deadline exceeded
Duration: 1.893067679s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)antiblip:
6 out of 60 is failed (threshold=3) [....XXX.XX....X.............................................] duration:
5.846048865s
FAIL (2025-08-08 00:36:31 UTC)
orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded
Duration: 5.846078211s
core
uplink
us1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
5 out of 60 is failed (threshold=3) [......................X...X..........X.......X...X..........] duration:
6.399815383s
FAIL (2025-08-08 00:36:22 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/4M):: after a4399ade4fa2878d720011f9f1d393a7a350a6f1820ab363fc7588b62888e2c7 != before 879367dc26e769e454df7bfa7c99b4b3877306fa37488cef7c83620ea9a98ba6
======LOGS======
.BatchRequestItem_ObjectBegin
Aug 8 00:28:47.094828147 (14h48m46.684173134s) [upload=15346]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Aug 8 00:28:47.551561608 (14h48m47.140906595s) [seg_pos={0 0}, upload=15346]: Begin upload segment...
Aug 8 00:28:47.551608808 (14h48m47.140953795s) [seg_pos={0 0}, upload=15346]: Done begin upload segment.
Aug 8 00:28:47.551629437 (14h48m47.140974424s) [noise=true , address=104.36.82.18:4367 , piece_id=PVNJQVZ3QW5ZVHGQG3QSDLRRZV5BFGN6J2MC62JQ5P5ZJW65WW5Q, seg_pos={0 0}, upload=15346]: Uploading piece...
Aug 8 00:28:47.554243379 (14h48m47.143588366s) [noise=true , address=104.36.82.7:4937 , piece_id=HI26KBJIH6YFKH2NU655SRLUCFTDHNA4QR2THVJVIEDMHQTZSJLQ, seg_pos={0 0}, upload=15346]: Uploading piece...
Aug 8 00:28:47.554910236 (14h48m47.144255223s) [noise=true , address=104.36.82.20:4445 , piece_id=LOWLB4F4N7LCZ5C3JE6HRSSU4FUOKJNIFTHIHRNCJCEKHGG667GA, seg_pos={0 0}, upload=15346]: Uploading piece...
Aug 8 00:28:47.555304531 (14h48m47.144649518s) [noise=true , address=104.36.82.62:4413 , piece_id=XNYCVTWNEL27VZYCYSA6F5JIIEZMNUPHGLCOX5LVH6LHAFSNB7DA, seg_pos={0 0}, upload=15346]: Uploading piece...
Aug 8 00:28:47.555953072 (14h48m47.145298059s) [noise=true , address=104.36.82.43:4861 , piece_id=JMUXDXGFTHJHGHMNKXIWYNWG4QALWELUKH6DODKRKVUS53YBEYAQ, seg_pos={0 0}, upload=15346]: Uploading piece...
Aug 8 00:28:47.556452077 (14h48m47.145797064s) [noise=true , address=104.36.82.37:4680 , piece_id=XZNKDG5FITGWMR7U7WSVE5F3KYFMNXZ73AFFCB5NC2RF3CKCK7RQ, seg_pos={0 0}, upload=15346]: Uploading piece...
Aug 8 00:28:47.557051404 (14h48m47.146396391s) [noise=true , address=104.36.82.10:4085 , piece_id=RI6KTTGHEXBFGMW54NCT7MJFUYRBIGB62AOUS3DASGOA2D55P6SQ, seg_pos={0 0}, upload=15346]: Uploading piece...
Aug 8 00:28:47.557476658 (14h48m47.146821645s) [noise=true , address=104.36.82.8:4979 , piece_id=Z6YJ5QYHJJJLGUXGCIPGGE25OLKO25MPC5AQ53LMZ22RCB25QWXQ, seg_pos={0 0}, upload=15346]: Uploading piece...
Aug 8 00:28:47.557973529 (14h48m47.147318516s) [noise=true , address=104.36.82.61:4376 , piece_id=PRM6GANCIR4W32MFVJY5UQ7YBMXHVMNXBUYDNQQUHA45HR5JWQAQ, seg_pos={0 0}, upload=15346]: Uploading piece...
Aug 8 00:28:47.558963734 (14h48m47.148308721s) [noise=true , address=104.36.82.52:4105 , piece_id=XT46OYR6SIAZQ6L4GFYNKXQCNIETX6FYVM3SZRMUYFSICP66YXKA, seg_pos={0 0}, upload=15346]: Uploading piece...
Aug 8 00:28:47.559345425 (14h48m47.148690412s) [noise=true , address=104.36.82.4:4836 , piece_id=M2Y7QYLPP4FNSUTJZOF3U5ZYDBUDJL25O7ILENA47OYIHJIE7CJA, seg_pos={0 0}, upload=15346]: Uploading piece...
Aug 8 00:28:47.559751782 (14h48m47.149096759s) [noise=true , address=104.36.82.23:4464 , piece_id=KSV4RQE2YF4TL2VB466SPMQTWWO2P7TD624GQQYYHZDIUUBKAJUQ, seg_pos={0 0}, upload=15346]: Uploading piece...
Aug 8 00:28:47.560180823 (14h48m47.14952581s) [noise=true , address=104.36.82.60:4366 , piece_id=U2HK37YA6BWLVE7ABJ465K7PTEXGEWI72ICQWZRSXZ3KGASXRL3Q, seg_pos={0 0}, upload=15346]: Uploading piece...
Aug 8 00:28:47.561215644 (14h48m47.150560631s) [noise=true , address=104.36.82.47:4520 , piece_id=QGOCU74LKBEZ5IA76WKIM2ZY57MAUMSJDYF3U3L7WCM3JPPTV7YQ, seg_pos={0 0}, upload=15346]: Uploading piece...
Aug 8 00:28:47.561560894 (14h48m47.150905881s) [noise=true , address=104.36.82.4:4813 , piece_id=37ZR56FKL5MJB4ZBBNGFUTVCXX6WEDJTYKEVSWLL7JLRNVONV2CQ, seg_pos={0 0}, upload=15346]: Uploading piece...
Aug 8 00:28:47.561844247 (14h48m47.151189234s) [noise=true , address=104.36.82.59:4325 , piece_id=EIEJTAIZHKY4SWUZPA7EXYUWYQV3SU4NHVA7IUDRTW2CJHSIPQPA, seg_pos={0 0}, upload=15346]: Uploading piece...
Aug 8 00:28:47.562154400 (14h48m47.151499387s) [noise=true , address=79.127.201.215:4352 , piece_id=EC3UMETYKU6KUMG3VJQOJ2WJCTCDWQ4WBHKPHQAKT7NC3F6RWDXQ, seg_pos={0 0}, upload=15346]: Uploading piece...
Aug 8 00:28:47.562457500 (14h48m47.151802487s) [noise=true , address=104.36.82.44:4895 , piece_id=E364QDW2NRBMQPZGZHJWRZO6GJN3SHL2CRVAAD4KRFYS5YDLOYZA, seg_pos={0 0}, upload=15346]: Uploading piece...
Aug 8 00:28:49.073642290 (14h48m48.662987277s) [upload=15346]: Next returned nil segment
Aug 8 00:28:49.073651728 (14h48m48.662996715s) [upload=15346]: Waiting for error group managing segments...
Aug 8 00:28:49.485127304 (14h48m49.074472291s) [noise=true , address=79.127.201.215:4352 , piece_id=EC3UMETYKU6KUMG3VJQOJ2WJCTCDWQ4WBHKPHQAKT7NC3F6RWDXQ, seg_pos={0 0}, upload=15346]: Done uploading piece. err: <nil>
Aug 8 00:28:49.569397892 (14h48m49.158742879s) [noise=true , address=104.36.82.23:4464 , piece_id=KSV4RQE2YF4TL2VB466SPMQTWWO2P7TD624GQQYYHZDIUUBKAJUQ, seg_pos={0 0}, upload=15346]: Done uploading piece. err: <nil>
Aug 8 00:28:49.589957680 (14h48m49.179302667s) [noise=true , address=104.36.82.7:4937 , piece_id=HI26KBJIH6YFKH2NU655SRLUCFTDHNA4QR2THVJVIEDMHQTZSJLQ, seg_pos={0 0}, upload=15346]: Done uploading piece. err: <nil>
Aug 8 00:28:49.659215316 (14h48m49.248560303s) [noise=true , address=104.36.82.20:4445 , piece_id=LOWLB4F4N7LCZ5C3JE6HRSSU4FUOKJNIFTHIHRNCJCEKHGG667GA, seg_pos={0 0}, upload=15346]: Done uploading piece. err: <nil>
Aug 8 00:28:49.659539797 (14h48m49.248884784s) [noise=true , address=104.36.82.8:4979 , piece_id=Z6YJ5QYHJJJLGUXGCIPGGE25OLKO25MPC5AQ53LMZ22RCB25QWXQ, seg_pos={0 0}, upload=15346]: Done uploading piece. err: <nil>
Aug 8 00:28:49.699454667 (14h48m49.288799654s) [noise=true , address=104.36.82.4:4813 , piece_id=37ZR56FKL5MJB4ZBBNGFUTVCXX6WEDJTYKEVSWLL7JLRNVONV2CQ, seg_pos={0 0}, upload=15346]: Done uploading piece. err: <nil>
Aug 8 00:28:49.734069373 (14h48m49.32341436s) [noise=true , address=104.36.82.44:4895 , piece_id=E364QDW2NRBMQPZGZHJWRZO6GJN3SHL2CRVAAD4KRFYS5YDLOYZA, seg_pos={0 0}, upload=15346]: Done uploading piece. err: <nil>
Aug 8 00:28:49.759950614 (14h48m49.349295601s) [noise=true , address=104.36.82.4:4836 , piece_id=M2Y7QYLPP4FNSUTJZOF3U5ZYDBUDJL25O7ILENA47OYIHJIE7CJA, seg_pos={0 0}, upload=15346]: Done uploading piece. err: <nil>
Aug 8 00:28:49.787498666 (14h48m49.376843653s) [noise=true , address=104.36.82.10:4085 , piece_id=RI6KTTGHEXBFGMW54NCT7MJFUYRBIGB62AOUS3DASGOA2D55P6SQ, seg_pos={0 0}, upload=15346]: Done uploading piece. err: <nil>
Aug 8 00:28:49.874182563 (14h48m49.46352758s) [noise=true , address=104.36.82.52:4105 , piece_id=XT46OYR6SIAZQ6L4GFYNKXQCNIETX6FYVM3SZRMUYFSICP66YXKA, seg_pos={0 0}, upload=15346]: Done uploading piece. err: <nil>
Aug 8 00:28:49.960972762 (14h48m49.550317749s) [noise=true , address=104.36.82.59:4325 , piece_id=EIEJTAIZHKY4SWUZPA7EXYUWYQV3SU4NHVA7IUDRTW2CJHSIPQPA, seg_pos={0 0}, upload=15346]: Done uploading piece. err: <nil>
Aug 8 00:28:50.001273120 (14h48m49.590618117s) [noise=true , address=104.36.82.37:4680 , piece_id=XZNKDG5FITGWMR7U7WSVE5F3KYFMNXZ73AFFCB5NC2RF3CKCK7RQ, seg_pos={0 0}, upload=15346]: Done uploading piece. err: <nil>
Aug 8 00:28:50.571026048 (14h48m50.160371025s) [noise=true , address=104.36.82.62:4413 , piece_id=XNYCVTWNEL27VZYCYSA6F5JIIEZMNUPHGLCOX5LVH6LHAFSNB7DA, seg_pos={0 0}, upload=15346]: Done uploading piece. err: <nil>
Aug 8 00:28:51.294616675 (14h48m50.883961662s) [noise=true , address=104.36.82.61:4376 , piece_id=PRM6GANCIR4W32MFVJY5UQ7YBMXHVMNXBUYDNQQUHA45HR5JWQAQ, seg_pos={0 0}, upload=15346]: Done uploading piece. err: <nil>
Aug 8 00:28:51.294639959 (14h48m50.883984936s) [seg_pos={0 0}, upload=15346]: Segment reached optimal threshold of 14 pieces.
Aug 8 00:28:51.294832458 (14h48m50.884177445s) [noise=true , address=104.36.82.47:4520 , piece_id=QGOCU74LKBEZ5IA76WKIM2ZY57MAUMSJDYF3U3L7WCM3JPPTV7YQ, seg_pos={0 0}, upload=15346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12eNRsboM5zFiyBVPLuQfgb5QSAc5bZFizK9KJHudNd9nstVGyy): protocol: expected piece hash; context canceled; EOF
Aug 8 00:28:51.295062047 (14h48m50.884407034s) [noise=true , address=104.36.82.60:4366 , piece_id=U2HK37YA6BWLVE7ABJ465K7PTEXGEWI72ICQWZRSXZ3KGASXRL3Q, seg_pos={0 0}, upload=15346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12NBUjBGU2MCjDx8iZKvzCvhL6GHCfdXfihC5H2uoBUmzhz9uoV): protocol: write tcp 188.166.225.170:57292->104.36.82.60:4366: use of closed network connection; write tcp 188.166.225.170:57292->104.36.82.60:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57292->104.36.82.60:4366: use of closed network connection
Aug 8 00:28:51.295101312 (14h48m50.884446299s) [noise=true , address=104.36.82.43:4861 , piece_id=JMUXDXGFTHJHGHMNKXIWYNWG4QALWELUKH6DODKRKVUS53YBEYAQ, seg_pos={0 0}, upload=15346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122HC6BetMaBDNGEaHVwmadWtV1nCHx3vGBK2SWVDEQS1iBCz4o): protocol: write tcp 188.166.225.170:53912->104.36.82.43:4861: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 8 00:28:51.295131720 (14h48m50.884476707s) [noise=true , address=104.36.82.18:4367 , piece_id=PVNJQVZ3QW5ZVHGQG3QSDLRRZV5BFGN6J2MC62JQ5P5ZJW65WW5Q, seg_pos={0 0}, upload=15346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6): protocol: read tcp 188.166.225.170:32868->104.36.82.18:4367: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Aug 8 00:28:51.295147570 (14h48m50.884492557s) [seg_pos={0 0}, upload=15346]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Aug 8 00:28:51.295216993 (14h48m50.88456198s) [upload=15346]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Aug 8 00:28:51.295226662 (14h48m50.884571649s) [upload=15346]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Aug 8 00:28:51.599168829 (14h48m51.188513806s) [upload=15346]: Done uploading segments...
Aug 8 00:28:51.599190280 (14h48m51.188535267s) [upload=15346]: Upload finished. err: <nil>
Duration: 6.399842373s
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|