Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 30 out of 60 is failed (threshold=3) [..............................XXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.000927376s FAIL (2025-08-07 21:39:56 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 940 (11h50m3.727433937s) [upload=12310]: Uploading segments... Aug 7 21:30:04.138138645 (11h50m3.727483642s) [upload=12310]: Got next segment. Inline: true Aug 7 21:30:04.138144005 (11h50m3.727488992s) [upload=12310]: Waiting for error group managing segments... Aug 7 21:30:04.138191466 (11h50m3.727536453s) [upload=12310]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:30:04.138197808 (11h50m3.727542795s) [upload=12310]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:30:04.138201486 (11h50m3.727546483s) [upload=12310]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:30:33.459773387 (11h50m33.049118384s) [upload=12310]: Done uploading segments... Aug 7 21:30:33.459805027 (11h50m33.049150014s) [upload=12310]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:31:10.105493540 (11h51m9.694838527s) [upload=12327]: Starting upload Aug 7 21:31:10.105548566 (11h51m9.694893543s) [upload=12327]: Done starting upload Aug 7 21:31:10.105626315 (11h51m9.694971302s) [upload=12327]: Uploading segments... Aug 7 21:31:10.105718581 (11h51m9.695063568s) [upload=12327]: Got next segment. Inline: true Aug 7 21:31:10.105723311 (11h51m9.695068298s) [upload=12327]: Waiting for error group managing segments... Aug 7 21:31:10.105777513 (11h51m9.69512249s) [upload=12327]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:31:10.105782744 (11h51m9.695127731s) [upload=12327]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:31:10.105785079 (11h51m9.695130066s) [upload=12327]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:31:39.460569760 (11h51m39.049914747s) [upload=12327]: Done uploading segments... Aug 7 21:31:39.460597573 (11h51m39.04994255s) [upload=12327]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:32:24.324148762 (11h52m23.913493749s) [upload=12347]: Starting upload Aug 7 21:32:24.324206403 (11h52m23.91355139s) [upload=12347]: Done starting upload Aug 7 21:32:24.324252100 (11h52m23.913597087s) [upload=12347]: Uploading segments... Aug 7 21:32:24.324280415 (11h52m23.913625402s) [upload=12347]: Got next segment. Inline: true Aug 7 21:32:24.324284021 (11h52m23.913628998s) [upload=12347]: Waiting for error group managing segments... Aug 7 21:32:24.324329689 (11h52m23.913674686s) [upload=12347]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:32:24.324336372 (11h52m23.913681379s) [upload=12347]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:32:24.324341050 (11h52m23.913686057s) [upload=12347]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:32:53.678084170 (11h52m53.267429207s) [upload=12347]: Done uploading segments... Aug 7 21:32:53.678107405 (11h52m53.267452392s) [upload=12347]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:33:17.618427342 (11h53m17.207772389s) [upload=12364]: Starting upload Aug 7 21:33:17.618576718 (11h53m17.207921745s) [upload=12364]: Done starting upload Aug 7 21:33:17.618658725 (11h53m17.208003742s) [upload=12364]: Uploading segments... Aug 7 21:33:17.618713580 (11h53m17.208058597s) [upload=12364]: Got next segment. Inline: true Aug 7 21:33:17.618720533 (11h53m17.20806553s) [upload=12364]: Waiting for error group managing segments... Aug 7 21:33:17.618772433 (11h53m17.20811743s) [upload=12364]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:33:17.618779636 (11h53m17.208124643s) [upload=12364]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:33:17.618784776 (11h53m17.208129793s) [upload=12364]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:33:46.957323530 (11h53m46.546668577s) [upload=12364]: Done uploading segments... Aug 7 21:33:46.957345773 (11h53m46.54669075s) [upload=12364]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:34:19.197336641 (11h54m18.786681628s) [upload=12382]: Starting upload Aug 7 21:34:19.197461931 (11h54m18.786806918s) [upload=12382]: Done starting upload Aug 7 21:34:19.197512228 (11h54m18.786857225s) [upload=12382]: Uploading segments... Aug 7 21:34:19.197561282 (11h54m18.786906269s) [upload=12382]: Got next segment. Inline: true Aug 7 21:34:19.197565189 (11h54m18.786910166s) [upload=12382]: Waiting for error group managing segments... Aug 7 21:34:19.197610725 (11h54m18.786955723s) [upload=12382]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:34:19.197615966 (11h54m18.786960953s) [upload=12382]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:34:19.197619142 (11h54m18.786964139s) [upload=12382]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:34:48.535954936 (11h54m48.125299923s) [upload=12382]: Done uploading segments... Aug 7 21:34:48.535978650 (11h54m48.125323637s) [upload=12382]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:35:05.991757765 (11h55m5.581102752s) [upload=12393]: Starting upload Aug 7 21:35:05.991822780 (11h55m5.581167767s) [upload=12393]: Done starting upload Aug 7 21:35:05.991946456 (11h55m5.581291443s) [upload=12393]: Uploading segments... Aug 7 21:35:05.991988116 (11h55m5.581333104s) [upload=12393]: Got next segment. Inline: true Aug 7 21:35:05.991994929 (11h55m5.581339996s) [upload=12393]: Waiting for error group managing segments... Aug 7 21:35:05.992042741 (11h55m5.581387728s) [upload=12393]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:35:05.992048021 (11h55m5.581393018s) [upload=12393]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:35:05.992051768 (11h55m5.581396755s) [upload=12393]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:35:35.339368736 (11h55m34.928713723s) [upload=12393]: Done uploading segments... Aug 7 21:35:35.339503534 (11h55m34.928848522s) [upload=12393]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:36:12.952326182 (11h56m12.541671159s) [upload=12411]: Starting upload Aug 7 21:36:12.952385075 (11h56m12.541730052s) [upload=12411]: Done starting upload Aug 7 21:36:12.952428238 (11h56m12.541773225s) [upload=12411]: Uploading segments... Aug 7 21:36:12.952455510 (11h56m12.541800497s) [upload=12411]: Got next segment. Inline: true Aug 7 21:36:12.952459417 (11h56m12.541804404s) [upload=12411]: Waiting for error group managing segments... Aug 7 21:36:12.952507650 (11h56m12.541852637s) [upload=12411]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:36:12.952514352 (11h56m12.541859349s) [upload=12411]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:36:12.952518550 (11h56m12.541863537s) [upload=12411]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:36:42.299158316 (11h56m41.888503304s) [upload=12411]: Done uploading segments... Aug 7 21:36:42.299175039 (11h56m41.888520026s) [upload=12411]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:37:22.572699237 (11h57m22.162044224s) [upload=12432]: Starting upload Aug 7 21:37:22.572782937 (11h57m22.162127924s) [upload=12432]: Done starting upload Aug 7 21:37:22.572842260 (11h57m22.162187247s) [upload=12432]: Uploading segments... Aug 7 21:37:22.572885313 (11h57m22.1622303s) [upload=12432]: Got next segment. Inline: true Aug 7 21:37:22.572889591 (11h57m22.162234578s) [upload=12432]: Waiting for error group managing segments... Aug 7 21:37:22.572930138 (11h57m22.162275125s) [upload=12432]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:37:22.572934296 (11h57m22.162279283s) [upload=12432]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:37:22.572937121 (11h57m22.162282118s) [upload=12432]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:37:51.933954276 (11h57m51.523299253s) [upload=12432]: Done uploading segments... Aug 7 21:37:51.933970927 (11h57m51.523315914s) [upload=12432]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:38:28.745940745 (11h58m28.335285732s) [upload=12448]: Starting upload Aug 7 21:38:28.745997504 (11h58m28.335342491s) [upload=12448]: Done starting upload Aug 7 21:38:28.746040176 (11h58m28.335385163s) [upload=12448]: Uploading segments... Aug 7 21:38:28.746078458 (11h58m28.335423446s) [upload=12448]: Got next segment. Inline: true Aug 7 21:38:28.746081766 (11h58m28.335426743s) [upload=12448]: Waiting for error group managing segments... Aug 7 21:38:28.746111923 (11h58m28.33545691s) [upload=12448]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:38:28.746115540 (11h58m28.335460527s) [upload=12448]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:38:28.746118205 (11h58m28.335463192s) [upload=12448]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:38:58.105315693 (11h58m57.694660691s) [upload=12448]: Done uploading segments... Aug 7 21:38:58.105342465 (11h58m57.694687452s) [upload=12448]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:39:27.137238449 (11h59m26.726583436s) [upload=12462]: Starting upload Aug 7 21:39:27.137294907 (11h59m26.726639894s) [upload=12462]: Done starting upload Aug 7 21:39:27.137337368 (11h59m26.726682355s) [upload=12462]: Uploading segments... Aug 7 21:39:27.137366043 (11h59m26.72671103s) [upload=12462]: Got next segment. Inline: true Aug 7 21:39:27.137370061 (11h59m26.726715048s) [upload=12462]: Waiting for error group managing segments... Aug 7 21:39:27.137401901 (11h59m26.726746879s) [upload=12462]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:39:27.137405518 (11h59m26.726750495s) [upload=12462]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:39:27.137407813 (11h59m26.72675279s) [upload=12462]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:39:56.495186447 (11h59m56.084531434s) [upload=12462]: Done uploading segments... Aug 7 21:39:56.495206415 (11h59m56.084551402s) [upload=12462]: Upload finished. err: metaclient: context deadline exceeded Duration: 30.000947254s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [..................................X...............X.........] duration: 2.047890781s FAIL (2025-08-07 21:39:56 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/1K") ======LOGS====== 083457s) [upload=12258]: Done uploading segments... Aug 7 21:26:43.142756605 (11h46m42.732101593s) [upload=12258]: Upload finished. err: <nil> Aug 7 21:27:16.645746716 (11h47m16.235091703s) [upload=12264]: Starting upload Aug 7 21:27:16.645808424 (11h47m16.235153411s) [upload=12264]: Done starting upload Aug 7 21:27:16.645853019 (11h47m16.235198006s) [upload=12264]: Uploading segments... Aug 7 21:27:16.645881223 (11h47m16.23522621s) [upload=12264]: Got next segment. Inline: true Aug 7 21:27:16.645884680 (11h47m16.235229667s) [upload=12264]: Waiting for error group managing segments... Aug 7 21:27:16.645915299 (11h47m16.235260286s) [upload=12264]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:27:16.645919216 (11h47m16.235264203s) [upload=12264]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:27:16.645921510 (11h47m16.235266498s) [upload=12264]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:27:16.954194817 (11h47m16.543539804s) [upload=12264]: Done uploading segments... Aug 7 21:27:16.954212471 (11h47m16.543557459s) [upload=12264]: Upload finished. err: <nil> Aug 7 21:27:46.945635033 (11h47m46.53498005s) [upload=12274]: Starting upload Aug 7 21:27:46.945720126 (11h47m46.535065163s) [upload=12274]: Done starting upload Aug 7 21:27:46.945850435 (11h47m46.535195422s) [upload=12274]: Uploading segments... Aug 7 21:27:46.945918004 (11h47m46.535262991s) [upload=12274]: Got next segment. Inline: true Aug 7 21:27:46.945922443 (11h47m46.53526742s) [upload=12274]: Waiting for error group managing segments... Aug 7 21:27:46.945960726 (11h47m46.535305713s) [upload=12274]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:27:46.945965345 (11h47m46.535310332s) [upload=12274]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:27:46.945999029 (11h47m46.535344026s) [upload=12274]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:27:47.238086931 (11h47m46.827431908s) [upload=12274]: Done uploading segments... Aug 7 21:27:47.238104565 (11h47m46.827449552s) [upload=12274]: Upload finished. err: <nil> Aug 7 21:28:27.597717226 (11h48m27.187062213s) [upload=12282]: Starting upload Aug 7 21:28:27.597768604 (11h48m27.187113581s) [upload=12282]: Done starting upload Aug 7 21:28:27.597851232 (11h48m27.187196219s) [upload=12282]: Uploading segments... Aug 7 21:28:27.597881140 (11h48m27.187226127s) [upload=12282]: Got next segment. Inline: true Aug 7 21:28:27.597885238 (11h48m27.187230225s) [upload=12282]: Waiting for error group managing segments... Aug 7 21:28:27.597916337 (11h48m27.187261324s) [upload=12282]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:28:27.597920134 (11h48m27.187265111s) [upload=12282]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:28:27.597922509 (11h48m27.187267496s) [upload=12282]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:28:27.890743040 (11h48m27.480088057s) [upload=12282]: Done uploading segments... Aug 7 21:28:27.890764190 (11h48m27.480109177s) [upload=12282]: Upload finished. err: <nil> Aug 7 21:29:22.617930722 (11h49m22.207275709s) [upload=12295]: Starting upload Aug 7 21:29:22.617969135 (11h49m22.207314122s) [upload=12295]: Done starting upload Aug 7 21:29:22.618026967 (11h49m22.207371954s) [upload=12295]: Uploading segments... Aug 7 21:29:22.618059649 (11h49m22.207404636s) [upload=12295]: Got next segment. Inline: true Aug 7 21:29:22.618063226 (11h49m22.207408213s) [upload=12295]: Waiting for error group managing segments... Aug 7 21:29:22.618096049 (11h49m22.207441026s) [upload=12295]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:29:22.618100216 (11h49m22.207445193s) [upload=12295]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:29:22.618104334 (11h49m22.207449321s) [upload=12295]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:29:23.935417449 (11h49m23.524762486s) [upload=12295]: Done uploading segments... Aug 7 21:29:23.935440473 (11h49m23.52478548s) [upload=12295]: Upload finished. err: <nil> Aug 7 21:29:59.675674818 (11h49m59.265019805s) [upload=12308]: Starting upload Aug 7 21:29:59.675709946 (11h49m59.265054933s) [upload=12308]: Done starting upload Aug 7 21:29:59.675775310 (11h49m59.265120287s) [upload=12308]: Uploading segments... Aug 7 21:29:59.675809485 (11h49m59.265154462s) [upload=12308]: Got next segment. Inline: true Aug 7 21:29:59.675813022 (11h49m59.265158009s) [upload=12308]: Waiting for error group managing segments... Aug 7 21:29:59.675846747 (11h49m59.265191724s) [upload=12308]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:29:59.675850584 (11h49m59.265195572s) [upload=12308]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:29:59.675852909 (11h49m59.265197896s) [upload=12308]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:29:59.960382218 (11h49m59.549727205s) [upload=12308]: Done uploading segments... Aug 7 21:29:59.960402026 (11h49m59.549747013s) [upload=12308]: Upload finished. err: <nil> Aug 7 21:31:01.204540565 (11h51m0.793885552s) [upload=12324]: Starting upload Aug 7 21:31:01.204590701 (11h51m0.793935688s) [upload=12324]: Done starting upload Aug 7 21:31:01.204646679 (11h51m0.793991666s) [upload=12324]: Uploading segments... Aug 7 21:31:01.204679222 (11h51m0.794024209s) [upload=12324]: Got next segment. Inline: true Aug 7 21:31:01.204683269 (11h51m0.794028256s) [upload=12324]: Waiting for error group managing segments... Aug 7 21:31:01.204721322 (11h51m0.794066309s) [upload=12324]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:31:01.204725289 (11h51m0.794070276s) [upload=12324]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:31:01.204728024 (11h51m0.794073011s) [upload=12324]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:31:01.494645083 (11h51m1.08399007s) [upload=12324]: Done uploading segments... Aug 7 21:31:01.494665120 (11h51m1.084010107s) [upload=12324]: Upload finished. err: <nil> Aug 7 21:31:28.376153869 (11h51m27.965498856s) [upload=12334]: Starting upload Aug 7 21:31:28.376197382 (11h51m27.965542369s) [upload=12334]: Done starting upload Aug 7 21:31:28.376245204 (11h51m27.965590181s) [upload=12334]: Uploading segments... Aug 7 21:31:28.376273257 (11h51m27.965618245s) [upload=12334]: Got next segment. Inline: true Aug 7 21:31:28.376276624 (11h51m27.965621611s) [upload=12334]: Waiting for error group managing segments... Aug 7 21:31:28.376306892 (11h51m27.965651869s) [upload=12334]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:31:28.376312712 (11h51m27.9656577s) [upload=12334]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:31:28.376315419 (11h51m27.965660406s) [upload=12334]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:31:28.645925163 (11h51m28.23527015s) [upload=12334]: Done uploading segments... Aug 7 21:31:28.645942786 (11h51m28.235287773s) [upload=12334]: Upload finished. err: <nil> Aug 7 21:32:07.331024173 (11h52m6.92036916s) [upload=12342]: Starting upload Aug 7 21:32:07.331074259 (11h52m6.920419246s) [upload=12342]: Done starting upload Aug 7 21:32:07.331122251 (11h52m6.920467239s) [upload=12342]: Uploading segments... Aug 7 21:32:07.331154433 (11h52m6.92049942s) [upload=12342]: Got next segment. Inline: true Aug 7 21:32:07.331159583 (11h52m6.92050456s) [upload=12342]: Waiting for error group managing segments... Aug 7 21:32:07.331191624 (11h52m6.920536611s) [upload=12342]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:32:07.331195581 (11h52m6.920540558s) [upload=12342]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:32:07.331197946 (11h52m6.920542933s) [upload=12342]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:32:07.722093791 (11h52m7.311438768s) [upload=12342]: Done uploading segments... Aug 7 21:32:07.722110794 (11h52m7.311455781s) [upload=12342]: Upload finished. err: <nil> Aug 7 21:32:38.771578054 (11h52m38.360923031s) [upload=12353]: Starting upload Aug 7 21:32:38.771650211 (11h52m38.360995188s) [upload=12353]: Done starting upload Aug 7 21:32:38.771738161 (11h52m38.361083148s) [upload=12353]: Uploading segments... Aug 7 21:32:38.771779349 (11h52m38.361124336s) [upload=12353]: Got next segment. Inline: true Aug 7 21:32:38.771782916 (11h52m38.361127893s) [upload=12353]: Waiting for error group managing segments... Aug 7 21:32:38.771815357 (11h52m38.361160334s) [upload=12353]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:32:38.771819266 (11h52m38.361164253s) [upload=12353]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:32:38.771821730 (11h52m38.361166717s) [upload=12353]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:32:39.111523769 (11h52m38.700868796s) [upload=12353]: Done uploading segments... Aug 7 21:32:39.111547013 (11h52m38.700892031s) [upload=12353]: Upload finished. err: <nil> Aug 7 21:33:06.222382527 (11h53m5.811727514s) [upload=12362]: Starting upload Aug 7 21:33:06.222432713 (11h53m5.8117777s) [upload=12362]: Done starting upload Aug 7 21:33:06.222489962 (11h53m5.811834949s) [upload=12362]: Uploading segments... Aug 7 21:33:06.222536221 (11h53m5.811881208s) [upload=12362]: Got next segment. Inline: true Aug 7 21:33:06.222540199 (11h53m5.811885176s) [upload=12362]: Waiting for error group managing segments... Aug 7 21:33:06.222573302 (11h53m5.811918289s) [upload=12362]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:33:06.222576979 (11h53m5.811921966s) [upload=12362]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:33:06.222579153 (11h53m5.81192413s) [upload=12362]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:33:06.561908138 (11h53m6.151253125s) [upload=12362]: Done uploading segments... Aug 7 21:33:06.561986849 (11h53m6.151331836s) [upload=12362]: Upload finished. err: <nil> Duration: 2.047917503s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 30 out of 60 is failed (threshold=3) [............................X..XXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.000529323s FAIL (2025-08-07 21:39:55 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 30.000545375s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 24 out of 60 is failed (threshold=3) [....................................XXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.000856758s FAIL (2025-08-07 21:39:41 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== 26.428567841 (11h51m26.017912818s) [noise=true , address=104.36.82.38:4714 , piece_id=7AYVJ5FRQUIZRQ2NTCUN43CWJWWNQZEHMD7UE3MCYXSRBFQOL56A, seg_pos={0 0}, upload=12321]: Done uploading piece. err: <nil> Aug 7 21:31:26.430706232 (11h51m26.020051219s) [noise=true , address=104.36.82.59:4313 , piece_id=BLGX433PKGAVB5WCNKXMA6P3EYPPWCPCB6WYHQ5RZ4I6WRUH4GHQ, seg_pos={0 0}, upload=12321]: Done uploading piece. err: <nil> Aug 7 21:31:26.612329184 (11h51m26.201674171s) [noise=true , address=104.36.82.55:4222 , piece_id=4W52URT2PUTVOIVFFYOFSE3RDOVBHMS56PG57MEEKCFU7QP3CZGQ, seg_pos={0 0}, upload=12321]: Done uploading piece. err: <nil> Aug 7 21:31:26.658377737 (11h51m26.247722724s) [noise=true , address=104.36.82.54:4162 , piece_id=6RDDMFE4ZG2YJG5GKRFLSANPJXNKADJXLY6H64A53RPJPSXEC4BA, seg_pos={0 0}, upload=12321]: Done uploading piece. err: <nil> Aug 7 21:31:26.674652975 (11h51m26.263997962s) [noise=true , address=104.36.82.45:4937 , piece_id=EBUAJKXBK6RZK7T46KXJT5P5LINBFVZCEPGVGG5OZ66SYNOA2RSQ, seg_pos={0 0}, upload=12321]: Done uploading piece. err: <nil> Aug 7 21:31:26.680556480 (11h51m26.269901467s) [noise=true , address=45.91.116.10:4110 , piece_id=F4ZMNLCCFQCBFB6FOPOJFTGPIL5BFYT6TPGL5C7SVLIHMTY4EJJA, seg_pos={0 0}, upload=12321]: Done uploading piece. err: <nil> Aug 7 21:31:26.680580406 (11h51m26.269925393s) [seg_pos={0 0}, upload=12321]: Segment reached optimal threshold of 14 pieces. Aug 7 21:31:26.680752626 (11h51m26.270097613s) [noise=true , address=45.91.116.6:4983 , piece_id=ZWH64KJWCVSTEGVN25WW35HRHTAO3BE3SPKKALNRT7NNLUKB4L4Q, seg_pos={0 0}, upload=12321]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122trJbDHuqeBMQHZT97HP2PgkeuLyUiDiuRm4nayu6ZTH7tat6): protocol: expected piece hash; context canceled; EOF Aug 7 21:31:26.680804515 (11h51m26.270149492s) [noise=true , address=45.91.116.7:4037 , piece_id=WY2F64PFXEGFVE4M6PUHHUYIMTI2KB4CB73DQADZK4QGIEZ2YCYQ, seg_pos={0 0}, upload=12321]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iNPwE2XSfXcaXAXcrH2jcyNGYeDiiWvzAcDQffrmjPwTVqVmK): protocol: expected piece hash; context canceled; EOF Aug 7 21:31:26.680954802 (11h51m26.270299789s) [noise=true , address=104.36.82.61:4375 , piece_id=5LQOQ3MR3EA36DIYLOVO6YDKF6EP2HPHMPG4P6MN76A4IA5BTBEQ, seg_pos={0 0}, upload=12321]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121fzX9v4rgEd78xnDAwcQtthyQHV3yUZUQYLXZ9byPDucGK4T5): protocol: write tcp 188.166.225.170:45604->104.36.82.61:4375: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 21:31:26.680994198 (11h51m26.270339185s) [noise=true , address=104.36.82.8:4977 , piece_id=VY47HC4FF6JU4QQTDVZ62NOIEGWSROTWYMGCB7XXNHJ3GZNJATMQ, seg_pos={0 0}, upload=12321]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WCAm12tXwimFSPqz7Seh6A4oxWqQVxqsF459xux4a5CqjpbAT): protocol: write tcp 188.166.225.170:36156->104.36.82.8:4977: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 21:31:26.681016300 (11h51m26.270361287s) [seg_pos={0 0}, upload=12321]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 21:31:26.681067097 (11h51m26.270412074s) [upload=12321]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 21:31:26.681072137 (11h51m26.270417114s) [upload=12321]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:32:39.816785277 (11h52m39.406130264s) [upload=12321]: Done uploading segments... Aug 7 21:32:39.816815415 (11h52m39.406160402s) [upload=12321]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:32:40.684664778 (11h52m40.274009755s) [upload=12355]: Starting upload Aug 7 21:32:40.684721386 (11h52m40.274066373s) [upload=12355]: Done starting upload Aug 7 21:32:40.684776000 (11h52m40.274121017s) [upload=12355]: Uploading segments... Aug 7 21:32:40.684893314 (11h52m40.274238292s) [upload=12355]: Got next segment. Inline: false Aug 7 21:32:40.684916699 (11h52m40.274261686s) [upload=12355]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:32:40.684920466 (11h52m40.274265443s) [upload=12355]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:33:40.036921304 (11h53m39.626266281s) [upload=12355]: Next segment err: context deadline exceeded Aug 7 21:33:40.037128099 (11h53m39.626473086s) [upload=12355]: Done uploading segments... Aug 7 21:33:40.037148919 (11h53m39.626493896s) [upload=12355]: Upload finished. err: context deadline exceeded Aug 7 21:33:40.899438535 (11h53m40.488783522s) [upload=12372]: Starting upload Aug 7 21:33:40.899490825 (11h53m40.488835832s) [upload=12372]: Done starting upload Aug 7 21:33:40.899590144 (11h53m40.488935131s) [upload=12372]: Uploading segments... Aug 7 21:33:40.899771832 (11h53m40.489116819s) [upload=12372]: Got next segment. Inline: false Aug 7 21:33:40.899795227 (11h53m40.489140214s) [upload=12372]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:33:40.899799275 (11h53m40.489144252s) [upload=12372]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:34:40.257761531 (11h54m39.847106518s) [upload=12372]: Next segment err: context deadline exceeded Aug 7 21:34:40.257885859 (11h54m39.847230856s) [upload=12372]: Done uploading segments... Aug 7 21:34:40.257904695 (11h54m39.847249672s) [upload=12372]: Upload finished. err: context deadline exceeded Aug 7 21:34:41.144124896 (11h54m40.733469873s) [upload=12386]: Starting upload Aug 7 21:34:41.144233103 (11h54m40.73357813s) [upload=12386]: Done starting upload Aug 7 21:34:41.144343113 (11h54m40.733688101s) [upload=12386]: Uploading segments... Aug 7 21:34:41.144700337 (11h54m40.734045364s) [upload=12386]: Got next segment. Inline: false Aug 7 21:34:41.144760612 (11h54m40.734105639s) [upload=12386]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:34:41.144768206 (11h54m40.734113223s) [upload=12386]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:35:40.491924814 (11h55m40.081269811s) [upload=12386]: Next segment err: context deadline exceeded Aug 7 21:35:40.492048891 (11h55m40.081393878s) [upload=12386]: Done uploading segments... Aug 7 21:35:40.492066364 (11h55m40.081411351s) [upload=12386]: Upload finished. err: context deadline exceeded Aug 7 21:35:41.353263670 (11h55m40.942608657s) [upload=12405]: Starting upload Aug 7 21:35:41.353369352 (11h55m40.942714339s) [upload=12405]: Done starting upload Aug 7 21:35:41.353452601 (11h55m40.942797588s) [upload=12405]: Uploading segments... Aug 7 21:35:41.353579473 (11h55m40.942924511s) [upload=12405]: Got next segment. Inline: false Aug 7 21:35:41.353621805 (11h55m40.942966792s) [upload=12405]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:35:41.353626053 (11h55m40.94297104s) [upload=12405]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:36:40.713868241 (11h56m40.303213228s) [upload=12405]: Next segment err: context deadline exceeded Aug 7 21:36:40.713958674 (11h56m40.303303651s) [upload=12405]: Done uploading segments... Aug 7 21:36:40.713972460 (11h56m40.303317447s) [upload=12405]: Upload finished. err: context deadline exceeded Aug 7 21:36:41.596698707 (11h56m41.186043744s) [upload=12418]: Starting upload Aug 7 21:36:41.596808026 (11h56m41.186153063s) [upload=12418]: Done starting upload Aug 7 21:36:41.596885475 (11h56m41.186230462s) [upload=12418]: Uploading segments... Aug 7 21:36:41.597098722 (11h56m41.186443709s) [upload=12418]: Got next segment. Inline: false Aug 7 21:36:41.597141925 (11h56m41.186486912s) [upload=12418]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:36:41.597146604 (11h56m41.186491591s) [upload=12418]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:37:40.947744552 (11h57m40.537089539s) [upload=12418]: Next segment err: context deadline exceeded Aug 7 21:37:40.947825697 (11h57m40.537170684s) [upload=12418]: Done uploading segments... Aug 7 21:37:40.947843592 (11h57m40.537188579s) [upload=12418]: Upload finished. err: context deadline exceeded Aug 7 21:37:41.823629920 (11h57m41.412974907s) [upload=12437]: Starting upload Aug 7 21:37:41.823680908 (11h57m41.413025895s) [upload=12437]: Done starting upload Aug 7 21:37:41.823826305 (11h57m41.413171293s) [upload=12437]: Uploading segments... Aug 7 21:37:41.823857055 (11h57m41.413202052s) [upload=12437]: Got next segment. Inline: false Aug 7 21:37:41.823914255 (11h57m41.413259242s) [upload=12437]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:37:41.823918503 (11h57m41.41326348s) [upload=12437]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:38:41.179029912 (11h58m40.768374899s) [upload=12437]: Next segment err: context deadline exceeded Aug 7 21:38:41.179128490 (11h58m40.768473477s) [upload=12437]: Done uploading segments... Aug 7 21:38:41.179142979 (11h58m40.768487966s) [upload=12437]: Upload finished. err: context deadline exceeded Aug 7 21:38:42.074438453 (11h58m41.66378344s) [upload=12451]: Starting upload Aug 7 21:38:42.074516963 (11h58m41.66386195s) [upload=12451]: Done starting upload Aug 7 21:38:42.074605202 (11h58m41.663950189s) [upload=12451]: Uploading segments... Aug 7 21:38:42.074689222 (11h58m41.664034209s) [upload=12451]: Got next segment. Inline: false Aug 7 21:38:42.074715102 (11h58m41.664060089s) [upload=12451]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:38:42.074718829 (11h58m41.664063816s) [upload=12451]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:39:41.428394360 (11h59m41.017739347s) [upload=12451]: Next segment err: context deadline exceeded Aug 7 21:39:41.428496375 (11h59m41.017841372s) [upload=12451]: Done uploading segments... Aug 7 21:39:41.428519450 (11h59m41.017864437s) [upload=12451]: Upload finished. err: context deadline exceeded Duration: 1m0.000875253s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 31 out of 60 is failed (threshold=3) [...........................X..XXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m6.609778899s FAIL (2025-08-07 21:39:38 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== uploading segments... Aug 7 21:37:14.238799287 (11h57m13.828144274s) [upload=12420]: Upload finished. err: context deadline exceeded Aug 7 21:37:32.234403187 (11h57m31.823748174s) [upload=12433]: Starting upload Aug 7 21:37:32.234480776 (11h57m31.823825763s) [upload=12433]: Done starting upload Aug 7 21:37:32.234611054 (11h57m31.823956041s) [upload=12433]: Uploading segments... Aug 7 21:37:32.234741895 (11h57m31.824086882s) [upload=12433]: Got next segment. Inline: false Aug 7 21:37:32.234780930 (11h57m31.824125917s) [upload=12433]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:37:32.234785328 (11h57m31.824130305s) [upload=12433]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:38:01.579014502 (11h58m1.168359489s) [upload=12433]: Next segment err: context deadline exceeded Aug 7 21:38:01.579278507 (11h58m1.168623494s) [upload=12433]: Done uploading segments... Aug 7 21:38:01.579294227 (11h58m1.168639214s) [upload=12433]: Upload finished. err: context deadline exceeded Aug 7 21:38:32.979307013 (11h58m32.56865199s) [upload=12449]: Starting upload Aug 7 21:38:32.979360956 (11h58m32.568705943s) [upload=12449]: Done starting upload Aug 7 21:38:32.979501265 (11h58m32.568846252s) [upload=12449]: Uploading segments... Aug 7 21:38:32.979631284 (11h58m32.568976271s) [upload=12449]: Got next segment. Inline: false Aug 7 21:38:32.979691148 (11h58m32.569036135s) [upload=12449]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:38:32.979696198 (11h58m32.569041185s) [upload=12449]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:38:48.686817210 (11h58m48.276162267s) [seg_pos={0 0}, upload=12449]: Begin upload segment... Aug 7 21:38:48.686871564 (11h58m48.276216581s) [seg_pos={0 0}, upload=12449]: Done begin upload segment. Aug 7 21:38:48.686907212 (11h58m48.276252239s) [noise=true , address=104.36.82.2:4731 , piece_id=EA5DJOHUKB4JMDX56BNKNC3Z3SP5MLHLYN5PTJHOL4PC3DC2YZQA, seg_pos={0 0}, upload=12449]: Uploading piece... Aug 7 21:38:48.687437808 (11h58m48.276782795s) [noise=true , address=104.36.82.13:4185 , piece_id=GCSEQS5RYA7DIYXXEYRSVTCZWM4GII3CCOWDZXCWGK73LSPH6VDQ, seg_pos={0 0}, upload=12449]: Uploading piece... Aug 7 21:38:48.687923266 (11h58m48.277268253s) [noise=true , address=77.90.31.11:4554 , piece_id=Y5X5HMEMMOAD5QMICFFJFWSBHXSVZKTD54AP2NB4I2DJT42KJGVQ, seg_pos={0 0}, upload=12449]: Uploading piece... Aug 7 21:38:48.688304686 (11h58m48.277649673s) [noise=true , address=45.91.116.8:4045 , piece_id=JUV7DENMC32B4EJWTW6BJM4TEQRQ2U4FR7IFQAEJMCHRSV5JBXVQ, seg_pos={0 0}, upload=12449]: Uploading piece... Aug 7 21:38:48.688674213 (11h58m48.278019201s) [noise=true , address=104.36.82.15:4257 , piece_id=U7SFDCW5WBXVOCSJLGFOMXVNIMLQB7CY6BVPA2YUV63LBEC4IWAQ, seg_pos={0 0}, upload=12449]: Uploading piece... Aug 7 21:38:48.689020405 (11h58m48.278365392s) [noise=true , address=104.36.82.13:4186 , piece_id=XY22B7MTHZJ4MJFMJCTVDS4JAYQ3TWQDOPVMBUDZE3EF5VUAQTWQ, seg_pos={0 0}, upload=12449]: Uploading piece... Aug 7 21:38:48.689281786 (11h58m48.278626773s) [noise=true , address=104.36.82.10:4078 , piece_id=XYCM4QHQ7UU2S34Q5DUF7G5YGD2Z7KIDHHUMBRQNBEZUMD4GKNZQ, seg_pos={0 0}, upload=12449]: Uploading piece... Aug 7 21:38:48.689581169 (11h58m48.278926156s) [noise=true , address=45.91.116.5:4299 , piece_id=ED5YULTZVTYLCHQDFVZJIBUSEAYQUJMXGHHB7TTO7GQ5CSBBHYQA, seg_pos={0 0}, upload=12449]: Uploading piece... Aug 7 21:38:48.689893847 (11h58m48.279238824s) [noise=true , address=45.91.116.12:4225 , piece_id=INA2TNZ6KIYF45J4UFGRSLNRELTKISWF7IYIGPNXFDLLGZ7QVJWQ, seg_pos={0 0}, upload=12449]: Uploading piece... Aug 7 21:38:48.690256691 (11h58m48.279601678s) [noise=true , address=104.36.82.11:4114 , piece_id=QVVBKLBIFQC6RRFOJW7MM6AMHL2ZGLQAMH5ZE73XLBGY3VERYEVA, seg_pos={0 0}, upload=12449]: Uploading piece... Aug 7 21:38:48.690626529 (11h58m48.279971516s) [noise=true , address=45.91.116.10:4082 , piece_id=QX3QMQSXP5XBAWE4QY4P5A5QK2QXFBSV67YFPNIQV2BFIH2RVEKQ, seg_pos={0 0}, upload=12449]: Uploading piece... Aug 7 21:38:48.690901335 (11h58m48.280246322s) [noise=true , address=104.36.82.11:4113 , piece_id=2EGFZD54ZFDWU5SP6RQCC4YPP3USNQYBZNFSEI3V7CPMYLLB2A5Q, seg_pos={0 0}, upload=12449]: Uploading piece... Aug 7 21:38:48.691203833 (11h58m48.280548841s) [noise=true , address=77.90.31.12:4592 , piece_id=WW3UFNYQSC4MDLIZ2AJPTLTQN33HWQ2423TRFV5KEUNJ2R6XWKSQ, seg_pos={0 0}, upload=12449]: Uploading piece... Aug 7 21:38:48.691489149 (11h58m48.280834126s) [noise=true , address=77.90.31.21:4879 , piece_id=VB2RZ4HV4VI6WNNSHXZZRHZEAINPP5KU4LO6HVR2FLSGI5H3HP5A, seg_pos={0 0}, upload=12449]: Uploading piece... Aug 7 21:38:48.691733436 (11h58m48.281078424s) [noise=true , address=45.91.116.9:4263 , piece_id=EEBTJKDOQWVNQSWOUI47IY3RHKY5FDAMYQL4LJSUOS6GULR52TGA, seg_pos={0 0}, upload=12449]: Uploading piece... Aug 7 21:38:48.692068177 (11h58m48.281413164s) [noise=true , address=104.36.82.14:4221 , piece_id=W24SNKHE75CCO7XJFHUIH53M26MCEM7ILYZEGH672SQVK6ZCRBLQ, seg_pos={0 0}, upload=12449]: Uploading piece... Aug 7 21:38:50.513228864 (11h58m50.102573851s) [upload=12449]: Next returned nil segment Aug 7 21:38:50.513249563 (11h58m50.10259454s) [upload=12449]: Waiting for error group managing segments... Aug 7 21:38:50.810863968 (11h58m50.400208945s) [noise=true , address=104.36.82.2:4731 , piece_id=EA5DJOHUKB4JMDX56BNKNC3Z3SP5MLHLYN5PTJHOL4PC3DC2YZQA, seg_pos={0 0}, upload=12449]: Done uploading piece. err: <nil> Aug 7 21:38:50.873747961 (11h58m50.463092948s) [noise=true , address=104.36.82.11:4113 , piece_id=2EGFZD54ZFDWU5SP6RQCC4YPP3USNQYBZNFSEI3V7CPMYLLB2A5Q, seg_pos={0 0}, upload=12449]: Done uploading piece. err: <nil> Aug 7 21:38:50.944657320 (11h58m50.534002307s) [noise=true , address=45.91.116.8:4045 , piece_id=JUV7DENMC32B4EJWTW6BJM4TEQRQ2U4FR7IFQAEJMCHRSV5JBXVQ, seg_pos={0 0}, upload=12449]: Done uploading piece. err: <nil> Aug 7 21:38:50.972481134 (11h58m50.561826121s) [noise=true , address=104.36.82.14:4221 , piece_id=W24SNKHE75CCO7XJFHUIH53M26MCEM7ILYZEGH672SQVK6ZCRBLQ, seg_pos={0 0}, upload=12449]: Done uploading piece. err: <nil> Aug 7 21:38:50.980065425 (11h58m50.569410412s) [noise=true , address=104.36.82.13:4186 , piece_id=XY22B7MTHZJ4MJFMJCTVDS4JAYQ3TWQDOPVMBUDZE3EF5VUAQTWQ, seg_pos={0 0}, upload=12449]: Done uploading piece. err: <nil> Aug 7 21:38:50.981707937 (11h58m50.571052925s) [noise=true , address=104.36.82.11:4114 , piece_id=QVVBKLBIFQC6RRFOJW7MM6AMHL2ZGLQAMH5ZE73XLBGY3VERYEVA, seg_pos={0 0}, upload=12449]: Done uploading piece. err: <nil> Aug 7 21:38:51.023162491 (11h58m50.612507508s) [noise=true , address=104.36.82.13:4185 , piece_id=GCSEQS5RYA7DIYXXEYRSVTCZWM4GII3CCOWDZXCWGK73LSPH6VDQ, seg_pos={0 0}, upload=12449]: Done uploading piece. err: <nil> Aug 7 21:38:51.075895351 (11h58m50.665240338s) [noise=true , address=77.90.31.11:4554 , piece_id=Y5X5HMEMMOAD5QMICFFJFWSBHXSVZKTD54AP2NB4I2DJT42KJGVQ, seg_pos={0 0}, upload=12449]: Done uploading piece. err: <nil> Aug 7 21:38:51.095722626 (11h58m50.685067613s) [noise=true , address=45.91.116.5:4299 , piece_id=ED5YULTZVTYLCHQDFVZJIBUSEAYQUJMXGHHB7TTO7GQ5CSBBHYQA, seg_pos={0 0}, upload=12449]: Done uploading piece. err: <nil> Aug 7 21:38:51.098735141 (11h58m50.688080128s) [noise=true , address=104.36.82.15:4257 , piece_id=U7SFDCW5WBXVOCSJLGFOMXVNIMLQB7CY6BVPA2YUV63LBEC4IWAQ, seg_pos={0 0}, upload=12449]: Done uploading piece. err: <nil> Aug 7 21:38:51.107956113 (11h58m50.6973011s) [noise=true , address=77.90.31.21:4879 , piece_id=VB2RZ4HV4VI6WNNSHXZZRHZEAINPP5KU4LO6HVR2FLSGI5H3HP5A, seg_pos={0 0}, upload=12449]: Done uploading piece. err: <nil> Aug 7 21:38:51.133243133 (11h58m50.72258812s) [noise=true , address=77.90.31.12:4592 , piece_id=WW3UFNYQSC4MDLIZ2AJPTLTQN33HWQ2423TRFV5KEUNJ2R6XWKSQ, seg_pos={0 0}, upload=12449]: Done uploading piece. err: <nil> Aug 7 21:38:51.249092953 (11h58m50.83843794s) [noise=true , address=45.91.116.9:4263 , piece_id=EEBTJKDOQWVNQSWOUI47IY3RHKY5FDAMYQL4LJSUOS6GULR52TGA, seg_pos={0 0}, upload=12449]: Done uploading piece. err: <nil> Aug 7 21:38:51.249118061 (11h58m50.838463048s) [seg_pos={0 0}, upload=12449]: Segment reached optimal threshold of 13 pieces. Aug 7 21:38:51.249306762 (11h58m50.838651749s) [noise=true , address=45.91.116.12:4225 , piece_id=INA2TNZ6KIYF45J4UFGRSLNRELTKISWF7IYIGPNXFDLLGZ7QVJWQ, seg_pos={0 0}, upload=12449]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: expected piece hash; context canceled; EOF Aug 7 21:38:51.249323974 (11h58m50.838668961s) [noise=true , address=45.91.116.10:4082 , piece_id=QX3QMQSXP5XBAWE4QY4P5A5QK2QXFBSV67YFPNIQV2BFIH2RVEKQ, seg_pos={0 0}, upload=12449]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Aug 7 21:38:51.249493108 (11h58m50.838838095s) [noise=true , address=104.36.82.10:4078 , piece_id=XYCM4QHQ7UU2S34Q5DUF7G5YGD2Z7KIDHHUMBRQNBEZUMD4GKNZQ, seg_pos={0 0}, upload=12449]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr): protocol: context canceled; context canceled Aug 7 21:38:51.249505242 (11h58m50.838850229s) [seg_pos={0 0}, upload=12449]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 7 21:38:51.249581758 (11h58m50.838926735s) [upload=12449]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 21:38:51.249586157 (11h58m50.838931144s) [upload=12449]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:39:38.936884982 (11h59m38.526229969s) [upload=12449]: Done uploading segments... Aug 7 21:39:38.936908165 (11h59m38.526253152s) [upload=12449]: Upload finished. err: metaclient: context deadline exceeded Duration: 1m6.609801042s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 5 out of 60 is failed (threshold=3) [.......................X.........X......X..X.............X..] duration: 6.852735778s FAIL (2025-08-07 21:39:36 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/4M):: after 3c7be29e56ef0a348f54a62643273e5baec016fc19e64e2fa674114dac8dddcf != before 2decbc1b39ad0a3120f73e14d303cc93862fc082b00150c3e706fb25a679be76 ======LOGS====== d=12436]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:37:41.241548466 (11h57m40.830893473s) [upload=12436]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:37:41.699604637 (11h57m41.288949624s) [seg_pos={0 0}, upload=12436]: Begin upload segment... Aug 7 21:37:41.699642941 (11h57m41.288987928s) [seg_pos={0 0}, upload=12436]: Done begin upload segment. Aug 7 21:37:41.699665653 (11h57m41.28901063s) [noise=true , address=64.49.13.137:4494 , piece_id=27I45T5P6HE5MB7KLKHIDYF3GKGB4X5EYUQEQCZAIPO56TWSUIXQ, seg_pos={0 0}, upload=12436]: Uploading piece... Aug 7 21:37:41.699677036 (11h57m41.289022023s) [noise=true , address=104.36.82.37:4671 , piece_id=GAEIZGDLAYYQ2ATH4PCFWG7ZDNDVLY3RWQNDKPZNVPBYZ4EPCCBQ, seg_pos={0 0}, upload=12436]: Uploading piece... Aug 7 21:37:41.700019711 (11h57m41.289364698s) [noise=true , address=104.36.82.58:4271 , piece_id=VBXDX65YCLKD7WWXXCKLDKNOCUBMA4PTK53IMIJKCM5M5AOQRRYQ, seg_pos={0 0}, upload=12436]: Uploading piece... Aug 7 21:37:41.700313864 (11h57m41.289658851s) [noise=true , address=104.36.82.55:4193 , piece_id=YO4COWLA4546LGDVL7E3X7CGY2TLKVGEMVBI6A3D7LCI226ZQGMA, seg_pos={0 0}, upload=12436]: Uploading piece... Aug 7 21:37:41.700321118 (11h57m41.289666105s) [noise=true , address=104.36.82.8:4975 , piece_id=MPDCSPMDGY4MEJHSXYOTT3VFFAZGRABQH7QR5VLSD2OWB25C2DBQ, seg_pos={0 0}, upload=12436]: Uploading piece... Aug 7 21:37:41.700555927 (11h57m41.289900914s) [noise=true , address=104.36.82.37:4683 , piece_id=ZYQOB47RWRIT4GC36XROSHCGKBKTYNE5PRDTECJJ4RIEWRXLNZ2Q, seg_pos={0 0}, upload=12436]: Uploading piece... Aug 7 21:37:41.700769827 (11h57m41.290114814s) [noise=true , address=79.127.201.215:4356 , piece_id=VA6MG6UXZ7VUH5LAMLMBPIK7C2GLTDXA33LCFDIZMBPXDDPLVXIA, seg_pos={0 0}, upload=12436]: Uploading piece... Aug 7 21:37:41.700792900 (11h57m41.290137887s) [noise=true , address=104.36.82.23:4447 , piece_id=ZPOANDX46722LIXWQSDNXMFDB4ZHSS765MZQ3NOF4JXPUNZ4TFPQ, seg_pos={0 0}, upload=12436]: Uploading piece... Aug 7 21:37:41.701033912 (11h57m41.290378889s) [noise=true , address=104.36.82.43:4861 , piece_id=AXOCVGDO4GD7PWM43RYNWUPGX2H7BK4X4QCWTAJ4C3CNFL7UICRA, seg_pos={0 0}, upload=12436]: Uploading piece... Aug 7 21:37:41.701082625 (11h57m41.290427642s) [noise=true , address=104.36.82.59:4314 , piece_id=RF5LCHJAO7EOIT4AVOUVKKYVVBST4F3S5BO63CPYVJBI4CAWLGJA, seg_pos={0 0}, upload=12436]: Uploading piece... Aug 7 21:37:41.701257619 (11h57m41.290602606s) [noise=true , address=104.36.82.60:4358 , piece_id=MQTJBJP572O6M7CC756QSNAU4LUWO67OD7FJDNHIXOVANN7GDPQQ, seg_pos={0 0}, upload=12436]: Uploading piece... Aug 7 21:37:41.701341520 (11h57m41.290686507s) [noise=true , address=104.36.82.58:4285 , piece_id=E7G4SBPGKX3OAR3J2Y2PG67ZWXLKAJWJ5L34KVWLDX44IAQHZFNA, seg_pos={0 0}, upload=12436]: Uploading piece... Aug 7 21:37:41.701453143 (11h57m41.29079813s) [noise=true , address=104.36.82.14:4242 , piece_id=BOU64PUQXF4JOVE7GEVOYSXQKQPRAI75BRRHO577Q3BVXES5C7EQ, seg_pos={0 0}, upload=12436]: Uploading piece... Aug 7 21:37:41.701589534 (11h57m41.290934521s) [noise=true , address=45.91.116.11:4150 , piece_id=CUAREN64B6BVTXHF3RIJPH6LRPR3QEPVHRKLZDVXVMZFWBEXVBSA, seg_pos={0 0}, upload=12436]: Uploading piece... Aug 7 21:37:41.701638548 (11h57m41.290983535s) [noise=true , address=104.36.82.4:4830 , piece_id=MKXIFYHW4KNTLS4AIHCEI53CE7STNGL5ITSZA7IKWF36QMF2MSMQ, seg_pos={0 0}, upload=12436]: Uploading piece... Aug 7 21:37:41.701846857 (11h57m41.291191844s) [noise=true , address=104.36.82.46:4508 , piece_id=LVE3T3E6XA5TR3W4XANXE44O54BMSBSAHLBRS2XROOEAXVF2K6DA, seg_pos={0 0}, upload=12436]: Uploading piece... Aug 7 21:37:41.701850394 (11h57m41.291195381s) [noise=true , address=45.91.116.14:4205 , piece_id=PSGW5VTZ3JNFGHFGEO2IWQN4A6DUYEFOYGLOCJHCX625AZGSJHZA, seg_pos={0 0}, upload=12436]: Uploading piece... Aug 7 21:37:41.702036911 (11h57m41.291381898s) [noise=true , address=45.91.116.7:4040 , piece_id=ILATCWOLO5UO5RPRF4IRL7Y4JNMYLUIBFNLZBUDUAJKQDVLTXCCQ, seg_pos={0 0}, upload=12436]: Uploading piece... Aug 7 21:37:42.901972883 (11h57m42.49131787s) [upload=12436]: Next returned nil segment Aug 7 21:37:42.901987461 (11h57m42.491332448s) [upload=12436]: Waiting for error group managing segments... Aug 7 21:37:43.221358460 (11h57m42.810703437s) [noise=true , address=64.49.13.137:4494 , piece_id=27I45T5P6HE5MB7KLKHIDYF3GKGB4X5EYUQEQCZAIPO56TWSUIXQ, seg_pos={0 0}, upload=12436]: Done uploading piece. err: <nil> Aug 7 21:37:43.635176843 (11h57m43.22452183s) [noise=true , address=45.91.116.7:4040 , piece_id=ILATCWOLO5UO5RPRF4IRL7Y4JNMYLUIBFNLZBUDUAJKQDVLTXCCQ, seg_pos={0 0}, upload=12436]: Done uploading piece. err: <nil> Aug 7 21:37:43.691989821 (11h57m43.281334828s) [noise=true , address=79.127.201.215:4356 , piece_id=VA6MG6UXZ7VUH5LAMLMBPIK7C2GLTDXA33LCFDIZMBPXDDPLVXIA, seg_pos={0 0}, upload=12436]: Done uploading piece. err: <nil> Aug 7 21:37:43.715706768 (11h57m43.305051755s) [noise=true , address=104.36.82.8:4975 , piece_id=MPDCSPMDGY4MEJHSXYOTT3VFFAZGRABQH7QR5VLSD2OWB25C2DBQ, seg_pos={0 0}, upload=12436]: Done uploading piece. err: <nil> Aug 7 21:37:43.732997881 (11h57m43.322342868s) [noise=true , address=104.36.82.14:4242 , piece_id=BOU64PUQXF4JOVE7GEVOYSXQKQPRAI75BRRHO577Q3BVXES5C7EQ, seg_pos={0 0}, upload=12436]: Done uploading piece. err: <nil> Aug 7 21:37:43.741488406 (11h57m43.330833383s) [noise=true , address=104.36.82.46:4508 , piece_id=LVE3T3E6XA5TR3W4XANXE44O54BMSBSAHLBRS2XROOEAXVF2K6DA, seg_pos={0 0}, upload=12436]: Done uploading piece. err: <nil> Aug 7 21:37:43.879657469 (11h57m43.469002456s) [noise=true , address=104.36.82.37:4671 , piece_id=GAEIZGDLAYYQ2ATH4PCFWG7ZDNDVLY3RWQNDKPZNVPBYZ4EPCCBQ, seg_pos={0 0}, upload=12436]: Done uploading piece. err: <nil> Aug 7 21:37:43.895681587 (11h57m43.485026574s) [noise=true , address=45.91.116.11:4150 , piece_id=CUAREN64B6BVTXHF3RIJPH6LRPR3QEPVHRKLZDVXVMZFWBEXVBSA, seg_pos={0 0}, upload=12436]: Done uploading piece. err: <nil> Aug 7 21:37:43.907596774 (11h57m43.496941751s) [noise=true , address=104.36.82.4:4830 , piece_id=MKXIFYHW4KNTLS4AIHCEI53CE7STNGL5ITSZA7IKWF36QMF2MSMQ, seg_pos={0 0}, upload=12436]: Done uploading piece. err: <nil> Aug 7 21:37:43.908572762 (11h57m43.497917749s) [noise=true , address=45.91.116.14:4205 , piece_id=PSGW5VTZ3JNFGHFGEO2IWQN4A6DUYEFOYGLOCJHCX625AZGSJHZA, seg_pos={0 0}, upload=12436]: Done uploading piece. err: <nil> Aug 7 21:37:43.912238615 (11h57m43.501583602s) [noise=true , address=104.36.82.60:4358 , piece_id=MQTJBJP572O6M7CC756QSNAU4LUWO67OD7FJDNHIXOVANN7GDPQQ, seg_pos={0 0}, upload=12436]: Done uploading piece. err: <nil> Aug 7 21:37:43.917180210 (11h57m43.506525197s) [noise=true , address=104.36.82.37:4683 , piece_id=ZYQOB47RWRIT4GC36XROSHCGKBKTYNE5PRDTECJJ4RIEWRXLNZ2Q, seg_pos={0 0}, upload=12436]: Done uploading piece. err: <nil> Aug 7 21:37:43.957535420 (11h57m43.546880407s) [noise=true , address=104.36.82.59:4314 , piece_id=RF5LCHJAO7EOIT4AVOUVKKYVVBST4F3S5BO63CPYVJBI4CAWLGJA, seg_pos={0 0}, upload=12436]: Done uploading piece. err: <nil> Aug 7 21:37:44.104572309 (11h57m43.693917296s) [noise=true , address=104.36.82.43:4861 , piece_id=AXOCVGDO4GD7PWM43RYNWUPGX2H7BK4X4QCWTAJ4C3CNFL7UICRA, seg_pos={0 0}, upload=12436]: Done uploading piece. err: <nil> Aug 7 21:37:44.104599301 (11h57m43.693944278s) [seg_pos={0 0}, upload=12436]: Segment reached optimal threshold of 14 pieces. Aug 7 21:37:44.104687199 (11h57m43.694032176s) [noise=true , address=104.36.82.23:4447 , piece_id=ZPOANDX46722LIXWQSDNXMFDB4ZHSS765MZQ3NOF4JXPUNZ4TFPQ, seg_pos={0 0}, upload=12436]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fqT9xA7kXTG8K9XwTUxL6FXibV35AkPHDK6grKGHg17fk91fP): protocol: expected piece hash; context canceled; EOF Aug 7 21:37:44.104922059 (11h57m43.694267046s) [noise=true , address=104.36.82.58:4271 , piece_id=VBXDX65YCLKD7WWXXCKLDKNOCUBMA4PTK53IMIJKCM5M5AOQRRYQ, seg_pos={0 0}, upload=12436]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QwLHnMQHafFPYL83gE6wffvzdkvCHwzzVMLBz4KBVdE7ynuf1): protocol: rpc: dial tcp 104.36.82.58:4271: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 21:37:44.104969008 (11h57m43.694313995s) [noise=true , address=104.36.82.58:4285 , piece_id=E7G4SBPGKX3OAR3J2Y2PG67ZWXLKAJWJ5L34KVWLDX44IAQHZFNA, seg_pos={0 0}, upload=12436]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rZhbymrDS6ahXLEqYRM64gcqRvG8CEnc8iGCTEhgXrpvRaQrp): protocol: write tcp 188.166.225.170:56894->104.36.82.58:4285: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 21:37:44.104982013 (11h57m43.69432699s) [noise=true , address=104.36.82.55:4193 , piece_id=YO4COWLA4546LGDVL7E3X7CGY2TLKVGEMVBI6A3D7LCI226ZQGMA, seg_pos={0 0}, upload=12436]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: write tcp 188.166.225.170:60782->104.36.82.55:4193: use of closed network connection; write tcp 188.166.225.170:60782->104.36.82.55:4193: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60782->104.36.82.55:4193: use of closed network connection Aug 7 21:37:44.104993535 (11h57m43.694338522s) [seg_pos={0 0}, upload=12436]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 21:37:44.105075582 (11h57m43.694420569s) [upload=12436]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 21:37:44.105079359 (11h57m43.694424336s) [upload=12436]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:37:44.447186820 (11h57m44.036531807s) [upload=12436]: Done uploading segments... Aug 7 21:37:44.447217970 (11h57m44.036562957s) [upload=12436]: Upload finished. err: <nil> Duration: 6.852761787s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 33 out of 60 is failed (threshold=3) [...........................XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.001159395s FAIL (2025-08-07 21:39:34 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.001169554s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 26 out of 60 is failed (threshold=3) [.................................X.XXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.000419825s FAIL (2025-08-07 21:39:33 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 485s) [upload=12293]: Uploading segments... Aug 7 21:29:18.794182454 (11h49m18.383527441s) [upload=12293]: Got next segment. Inline: true Aug 7 21:29:18.794187423 (11h49m18.383532411s) [upload=12293]: Waiting for error group managing segments... Aug 7 21:29:18.794239163 (11h49m18.38358415s) [upload=12293]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:29:18.794244643 (11h49m18.38358962s) [upload=12293]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:29:18.794247088 (11h49m18.383592076s) [upload=12293]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:30:18.146928865 (11h50m17.736273852s) [upload=12293]: Done uploading segments... Aug 7 21:30:18.146964162 (11h50m17.73630915s) [upload=12293]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:30:30.199348334 (11h50m29.788693321s) [upload=12315]: Starting upload Aug 7 21:30:30.199433327 (11h50m29.788778314s) [upload=12315]: Done starting upload Aug 7 21:30:30.199483272 (11h50m29.788828259s) [upload=12315]: Uploading segments... Aug 7 21:30:30.199515665 (11h50m29.788860652s) [upload=12315]: Got next segment. Inline: true Aug 7 21:30:30.199519302 (11h50m29.788864289s) [upload=12315]: Waiting for error group managing segments... Aug 7 21:30:30.199554779 (11h50m29.788899766s) [upload=12315]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:30:30.199559147 (11h50m29.788904134s) [upload=12315]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:30:30.199561592 (11h50m29.788906579s) [upload=12315]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:31:29.557680478 (11h51m29.147025465s) [upload=12315]: Done uploading segments... Aug 7 21:31:29.557706508 (11h51m29.147051485s) [upload=12315]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:31:30.468794234 (11h51m30.058139221s) [upload=12336]: Starting upload Aug 7 21:31:30.468935564 (11h51m30.058280551s) [upload=12336]: Done starting upload Aug 7 21:31:30.469048480 (11h51m30.058393467s) [upload=12336]: Uploading segments... Aug 7 21:31:30.469142200 (11h51m30.058487187s) [upload=12336]: Got next segment. Inline: true Aug 7 21:31:30.469147400 (11h51m30.058492387s) [upload=12336]: Waiting for error group managing segments... Aug 7 21:31:30.469203888 (11h51m30.058548875s) [upload=12336]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:31:30.469208757 (11h51m30.058553744s) [upload=12336]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:31:30.469211582 (11h51m30.058556569s) [upload=12336]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:32:29.815552389 (11h52m29.404897376s) [upload=12336]: Done uploading segments... Aug 7 21:32:29.815578508 (11h52m29.404923495s) [upload=12336]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:32:30.694545842 (11h52m30.283890829s) [upload=12349]: Starting upload Aug 7 21:32:30.694631607 (11h52m30.283976624s) [upload=12349]: Done starting upload Aug 7 21:32:30.694676312 (11h52m30.284021299s) [upload=12349]: Uploading segments... Aug 7 21:32:30.694717561 (11h52m30.284062538s) [upload=12349]: Got next segment. Inline: true Aug 7 21:32:30.694721909 (11h52m30.284066896s) [upload=12349]: Waiting for error group managing segments... Aug 7 21:32:30.694762737 (11h52m30.284107724s) [upload=12349]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:32:30.694766976 (11h52m30.284111953s) [upload=12349]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:32:30.694769561 (11h52m30.284114548s) [upload=12349]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:33:30.042542113 (11h53m29.6318871s) [upload=12349]: Done uploading segments... Aug 7 21:33:30.042561801 (11h53m29.631906788s) [upload=12349]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:33:30.916211822 (11h53m30.505556809s) [upload=12369]: Starting upload Aug 7 21:33:30.916281525 (11h53m30.505626512s) [upload=12369]: Done starting upload Aug 7 21:33:30.916351519 (11h53m30.505696567s) [upload=12369]: Uploading segments... Aug 7 21:33:30.916406474 (11h53m30.505751502s) [upload=12369]: Got next segment. Inline: true Aug 7 21:33:30.916413598 (11h53m30.505758605s) [upload=12369]: Waiting for error group managing segments... Aug 7 21:33:30.916454065 (11h53m30.505799052s) [upload=12369]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:33:30.916457992 (11h53m30.505802979s) [upload=12369]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:33:30.916460378 (11h53m30.505805355s) [upload=12369]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:34:30.262869110 (11h54m29.852214097s) [upload=12369]: Done uploading segments... Aug 7 21:34:30.262896352 (11h54m29.85224134s) [upload=12369]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:34:31.146607534 (11h54m30.735952521s) [upload=12385]: Starting upload Aug 7 21:34:31.146655144 (11h54m30.736000121s) [upload=12385]: Done starting upload Aug 7 21:34:31.146698838 (11h54m30.736043825s) [upload=12385]: Uploading segments... Aug 7 21:34:31.146741720 (11h54m30.736086697s) [upload=12385]: Got next segment. Inline: true Aug 7 21:34:31.146745548 (11h54m30.736090535s) [upload=12385]: Waiting for error group managing segments... Aug 7 21:34:31.146776246 (11h54m30.736121233s) [upload=12385]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:34:31.146779913 (11h54m30.73612489s) [upload=12385]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:34:31.146782388 (11h54m30.736127375s) [upload=12385]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:35:30.500521706 (11h55m30.089866693s) [upload=12385]: Done uploading segments... Aug 7 21:35:30.500542195 (11h55m30.089887182s) [upload=12385]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:35:31.371815861 (11h55m30.961160848s) [upload=12402]: Starting upload Aug 7 21:35:31.371969656 (11h55m30.961314643s) [upload=12402]: Done starting upload Aug 7 21:35:31.372062443 (11h55m30.96140743s) [upload=12402]: Uploading segments... Aug 7 21:35:31.372108090 (11h55m30.961453077s) [upload=12402]: Got next segment. Inline: true Aug 7 21:35:31.372114412 (11h55m30.961459399s) [upload=12402]: Waiting for error group managing segments... Aug 7 21:35:31.372166783 (11h55m30.96151177s) [upload=12402]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:35:31.372172002 (11h55m30.961516999s) [upload=12402]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:35:31.372174898 (11h55m30.961519885s) [upload=12402]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:36:30.721260917 (11h56m30.310605904s) [upload=12402]: Done uploading segments... Aug 7 21:36:30.721280625 (11h56m30.310625612s) [upload=12402]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:36:33.253873027 (11h56m32.843218014s) [upload=12416]: Starting upload Aug 7 21:36:33.253941188 (11h56m32.843286175s) [upload=12416]: Done starting upload Aug 7 21:36:33.254004108 (11h56m32.843349085s) [upload=12416]: Uploading segments... Aug 7 21:36:33.254039966 (11h56m32.843384953s) [upload=12416]: Got next segment. Inline: true Aug 7 21:36:33.254043462 (11h56m32.84338845s) [upload=12416]: Waiting for error group managing segments... Aug 7 21:36:33.254074321 (11h56m32.843419308s) [upload=12416]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:36:33.254078290 (11h56m32.843423277s) [upload=12416]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:36:33.254080714 (11h56m32.843425701s) [upload=12416]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:37:32.603532648 (11h57m32.192877635s) [upload=12416]: Done uploading segments... Aug 7 21:37:32.603556113 (11h57m32.1929011s) [upload=12416]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:37:33.470049892 (11h57m33.059394869s) [upload=12434]: Starting upload Aug 7 21:37:33.470104877 (11h57m33.059449865s) [upload=12434]: Done starting upload Aug 7 21:37:33.470145565 (11h57m33.059490552s) [upload=12434]: Uploading segments... Aug 7 21:37:33.470187375 (11h57m33.059532352s) [upload=12434]: Got next segment. Inline: true Aug 7 21:37:33.470190872 (11h57m33.059535859s) [upload=12434]: Waiting for error group managing segments... Aug 7 21:37:33.470223234 (11h57m33.059568221s) [upload=12434]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:37:33.470226710 (11h57m33.059571697s) [upload=12434]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:37:33.470228904 (11h57m33.059573881s) [upload=12434]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:38:32.827667037 (11h58m32.417012024s) [upload=12434]: Done uploading segments... Aug 7 21:38:32.827687857 (11h58m32.417032834s) [upload=12434]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:38:33.703666664 (11h58m33.293011651s) [upload=12450]: Starting upload Aug 7 21:38:33.703724295 (11h58m33.293069282s) [upload=12450]: Done starting upload Aug 7 21:38:33.703770663 (11h58m33.29311565s) [upload=12450]: Uploading segments... Aug 7 21:38:33.703817212 (11h58m33.293162199s) [upload=12450]: Got next segment. Inline: true Aug 7 21:38:33.703820899 (11h58m33.293165886s) [upload=12450]: Waiting for error group managing segments... Aug 7 21:38:33.703853411 (11h58m33.293198388s) [upload=12450]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:38:33.703857378 (11h58m33.293202365s) [upload=12450]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:38:33.703859744 (11h58m33.293204731s) [upload=12450]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:39:33.059235349 (11h59m32.648580326s) [upload=12450]: Done uploading segments... Aug 7 21:39:33.059253302 (11h59m32.648598289s) [upload=12450]: Upload finished. err: metaclient: context deadline exceeded Duration: 1m0.000435805s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 34 out of 60 is failed (threshold=3) [..........................XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.00144532s FAIL (2025-08-07 21:39:33 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.001462594s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 24 out of 60 is failed (threshold=3) [....................................XXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.00083569s FAIL (2025-08-07 21:39:27 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 30.000854476s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [...................................X...............X........] duration: 2.1530613s FAIL (2025-08-07 21:39:14 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/1K") ======LOGS====== 083457s) [upload=12258]: Done uploading segments... Aug 7 21:26:43.142756605 (11h46m42.732101593s) [upload=12258]: Upload finished. err: <nil> Aug 7 21:27:16.645746716 (11h47m16.235091703s) [upload=12264]: Starting upload Aug 7 21:27:16.645808424 (11h47m16.235153411s) [upload=12264]: Done starting upload Aug 7 21:27:16.645853019 (11h47m16.235198006s) [upload=12264]: Uploading segments... Aug 7 21:27:16.645881223 (11h47m16.23522621s) [upload=12264]: Got next segment. Inline: true Aug 7 21:27:16.645884680 (11h47m16.235229667s) [upload=12264]: Waiting for error group managing segments... Aug 7 21:27:16.645915299 (11h47m16.235260286s) [upload=12264]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:27:16.645919216 (11h47m16.235264203s) [upload=12264]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:27:16.645921510 (11h47m16.235266498s) [upload=12264]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:27:16.954194817 (11h47m16.543539804s) [upload=12264]: Done uploading segments... Aug 7 21:27:16.954212471 (11h47m16.543557459s) [upload=12264]: Upload finished. err: <nil> Aug 7 21:27:46.945635033 (11h47m46.53498005s) [upload=12274]: Starting upload Aug 7 21:27:46.945720126 (11h47m46.535065163s) [upload=12274]: Done starting upload Aug 7 21:27:46.945850435 (11h47m46.535195422s) [upload=12274]: Uploading segments... Aug 7 21:27:46.945918004 (11h47m46.535262991s) [upload=12274]: Got next segment. Inline: true Aug 7 21:27:46.945922443 (11h47m46.53526742s) [upload=12274]: Waiting for error group managing segments... Aug 7 21:27:46.945960726 (11h47m46.535305713s) [upload=12274]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:27:46.945965345 (11h47m46.535310332s) [upload=12274]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:27:46.945999029 (11h47m46.535344026s) [upload=12274]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:27:47.238086931 (11h47m46.827431908s) [upload=12274]: Done uploading segments... Aug 7 21:27:47.238104565 (11h47m46.827449552s) [upload=12274]: Upload finished. err: <nil> Aug 7 21:28:27.597717226 (11h48m27.187062213s) [upload=12282]: Starting upload Aug 7 21:28:27.597768604 (11h48m27.187113581s) [upload=12282]: Done starting upload Aug 7 21:28:27.597851232 (11h48m27.187196219s) [upload=12282]: Uploading segments... Aug 7 21:28:27.597881140 (11h48m27.187226127s) [upload=12282]: Got next segment. Inline: true Aug 7 21:28:27.597885238 (11h48m27.187230225s) [upload=12282]: Waiting for error group managing segments... Aug 7 21:28:27.597916337 (11h48m27.187261324s) [upload=12282]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:28:27.597920134 (11h48m27.187265111s) [upload=12282]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:28:27.597922509 (11h48m27.187267496s) [upload=12282]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:28:27.890743040 (11h48m27.480088057s) [upload=12282]: Done uploading segments... Aug 7 21:28:27.890764190 (11h48m27.480109177s) [upload=12282]: Upload finished. err: <nil> Aug 7 21:29:22.617930722 (11h49m22.207275709s) [upload=12295]: Starting upload Aug 7 21:29:22.617969135 (11h49m22.207314122s) [upload=12295]: Done starting upload Aug 7 21:29:22.618026967 (11h49m22.207371954s) [upload=12295]: Uploading segments... Aug 7 21:29:22.618059649 (11h49m22.207404636s) [upload=12295]: Got next segment. Inline: true Aug 7 21:29:22.618063226 (11h49m22.207408213s) [upload=12295]: Waiting for error group managing segments... Aug 7 21:29:22.618096049 (11h49m22.207441026s) [upload=12295]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:29:22.618100216 (11h49m22.207445193s) [upload=12295]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:29:22.618104334 (11h49m22.207449321s) [upload=12295]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:29:23.935417449 (11h49m23.524762486s) [upload=12295]: Done uploading segments... Aug 7 21:29:23.935440473 (11h49m23.52478548s) [upload=12295]: Upload finished. err: <nil> Aug 7 21:29:59.675674818 (11h49m59.265019805s) [upload=12308]: Starting upload Aug 7 21:29:59.675709946 (11h49m59.265054933s) [upload=12308]: Done starting upload Aug 7 21:29:59.675775310 (11h49m59.265120287s) [upload=12308]: Uploading segments... Aug 7 21:29:59.675809485 (11h49m59.265154462s) [upload=12308]: Got next segment. Inline: true Aug 7 21:29:59.675813022 (11h49m59.265158009s) [upload=12308]: Waiting for error group managing segments... Aug 7 21:29:59.675846747 (11h49m59.265191724s) [upload=12308]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:29:59.675850584 (11h49m59.265195572s) [upload=12308]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:29:59.675852909 (11h49m59.265197896s) [upload=12308]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:29:59.960382218 (11h49m59.549727205s) [upload=12308]: Done uploading segments... Aug 7 21:29:59.960402026 (11h49m59.549747013s) [upload=12308]: Upload finished. err: <nil> Aug 7 21:31:01.204540565 (11h51m0.793885552s) [upload=12324]: Starting upload Aug 7 21:31:01.204590701 (11h51m0.793935688s) [upload=12324]: Done starting upload Aug 7 21:31:01.204646679 (11h51m0.793991666s) [upload=12324]: Uploading segments... Aug 7 21:31:01.204679222 (11h51m0.794024209s) [upload=12324]: Got next segment. Inline: true Aug 7 21:31:01.204683269 (11h51m0.794028256s) [upload=12324]: Waiting for error group managing segments... Aug 7 21:31:01.204721322 (11h51m0.794066309s) [upload=12324]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:31:01.204725289 (11h51m0.794070276s) [upload=12324]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:31:01.204728024 (11h51m0.794073011s) [upload=12324]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:31:01.494645083 (11h51m1.08399007s) [upload=12324]: Done uploading segments... Aug 7 21:31:01.494665120 (11h51m1.084010107s) [upload=12324]: Upload finished. err: <nil> Aug 7 21:31:28.376153869 (11h51m27.965498856s) [upload=12334]: Starting upload Aug 7 21:31:28.376197382 (11h51m27.965542369s) [upload=12334]: Done starting upload Aug 7 21:31:28.376245204 (11h51m27.965590181s) [upload=12334]: Uploading segments... Aug 7 21:31:28.376273257 (11h51m27.965618245s) [upload=12334]: Got next segment. Inline: true Aug 7 21:31:28.376276624 (11h51m27.965621611s) [upload=12334]: Waiting for error group managing segments... Aug 7 21:31:28.376306892 (11h51m27.965651869s) [upload=12334]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:31:28.376312712 (11h51m27.9656577s) [upload=12334]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:31:28.376315419 (11h51m27.965660406s) [upload=12334]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:31:28.645925163 (11h51m28.23527015s) [upload=12334]: Done uploading segments... Aug 7 21:31:28.645942786 (11h51m28.235287773s) [upload=12334]: Upload finished. err: <nil> Aug 7 21:32:07.331024173 (11h52m6.92036916s) [upload=12342]: Starting upload Aug 7 21:32:07.331074259 (11h52m6.920419246s) [upload=12342]: Done starting upload Aug 7 21:32:07.331122251 (11h52m6.920467239s) [upload=12342]: Uploading segments... Aug 7 21:32:07.331154433 (11h52m6.92049942s) [upload=12342]: Got next segment. Inline: true Aug 7 21:32:07.331159583 (11h52m6.92050456s) [upload=12342]: Waiting for error group managing segments... Aug 7 21:32:07.331191624 (11h52m6.920536611s) [upload=12342]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:32:07.331195581 (11h52m6.920540558s) [upload=12342]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:32:07.331197946 (11h52m6.920542933s) [upload=12342]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:32:07.722093791 (11h52m7.311438768s) [upload=12342]: Done uploading segments... Aug 7 21:32:07.722110794 (11h52m7.311455781s) [upload=12342]: Upload finished. err: <nil> Aug 7 21:32:38.771578054 (11h52m38.360923031s) [upload=12353]: Starting upload Aug 7 21:32:38.771650211 (11h52m38.360995188s) [upload=12353]: Done starting upload Aug 7 21:32:38.771738161 (11h52m38.361083148s) [upload=12353]: Uploading segments... Aug 7 21:32:38.771779349 (11h52m38.361124336s) [upload=12353]: Got next segment. Inline: true Aug 7 21:32:38.771782916 (11h52m38.361127893s) [upload=12353]: Waiting for error group managing segments... Aug 7 21:32:38.771815357 (11h52m38.361160334s) [upload=12353]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:32:38.771819266 (11h52m38.361164253s) [upload=12353]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:32:38.771821730 (11h52m38.361166717s) [upload=12353]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:32:39.111523769 (11h52m38.700868796s) [upload=12353]: Done uploading segments... Aug 7 21:32:39.111547013 (11h52m38.700892031s) [upload=12353]: Upload finished. err: <nil> Aug 7 21:33:06.222382527 (11h53m5.811727514s) [upload=12362]: Starting upload Aug 7 21:33:06.222432713 (11h53m5.8117777s) [upload=12362]: Done starting upload Aug 7 21:33:06.222489962 (11h53m5.811834949s) [upload=12362]: Uploading segments... Aug 7 21:33:06.222536221 (11h53m5.811881208s) [upload=12362]: Got next segment. Inline: true Aug 7 21:33:06.222540199 (11h53m5.811885176s) [upload=12362]: Waiting for error group managing segments... Aug 7 21:33:06.222573302 (11h53m5.811918289s) [upload=12362]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:33:06.222576979 (11h53m5.811921966s) [upload=12362]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:33:06.222579153 (11h53m5.81192413s) [upload=12362]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:33:06.561908138 (11h53m6.151253125s) [upload=12362]: Done uploading segments... Aug 7 21:33:06.561986849 (11h53m6.151331836s) [upload=12362]: Upload finished. err: <nil> Duration: 2.153112098s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 29 out of 60 is failed (threshold=3) [...............................XXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.000914131s FAIL (2025-08-07 21:38:58 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 0567 (11h49m24.411655554s) [upload=12297]: Uploading segments... Aug 7 21:29:24.822336567 (11h49m24.411681544s) [upload=12297]: Got next segment. Inline: true Aug 7 21:29:24.822340274 (11h49m24.411685261s) [upload=12297]: Waiting for error group managing segments... Aug 7 21:29:24.822369891 (11h49m24.411714878s) [upload=12297]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:29:24.822374179 (11h49m24.411719166s) [upload=12297]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:29:24.822376544 (11h49m24.411721531s) [upload=12297]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:29:54.171228417 (11h49m53.760573394s) [upload=12297]: Done uploading segments... Aug 7 21:29:54.171248837 (11h49m53.760593824s) [upload=12297]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:30:04.137944213 (11h50m3.72728921s) [upload=12310]: Starting upload Aug 7 21:30:04.138024878 (11h50m3.727369875s) [upload=12310]: Done starting upload Aug 7 21:30:04.138088940 (11h50m3.727433937s) [upload=12310]: Uploading segments... Aug 7 21:30:04.138138645 (11h50m3.727483642s) [upload=12310]: Got next segment. Inline: true Aug 7 21:30:04.138144005 (11h50m3.727488992s) [upload=12310]: Waiting for error group managing segments... Aug 7 21:30:04.138191466 (11h50m3.727536453s) [upload=12310]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:30:04.138197808 (11h50m3.727542795s) [upload=12310]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:30:04.138201486 (11h50m3.727546483s) [upload=12310]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:30:33.459773387 (11h50m33.049118384s) [upload=12310]: Done uploading segments... Aug 7 21:30:33.459805027 (11h50m33.049150014s) [upload=12310]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:31:10.105493540 (11h51m9.694838527s) [upload=12327]: Starting upload Aug 7 21:31:10.105548566 (11h51m9.694893543s) [upload=12327]: Done starting upload Aug 7 21:31:10.105626315 (11h51m9.694971302s) [upload=12327]: Uploading segments... Aug 7 21:31:10.105718581 (11h51m9.695063568s) [upload=12327]: Got next segment. Inline: true Aug 7 21:31:10.105723311 (11h51m9.695068298s) [upload=12327]: Waiting for error group managing segments... Aug 7 21:31:10.105777513 (11h51m9.69512249s) [upload=12327]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:31:10.105782744 (11h51m9.695127731s) [upload=12327]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:31:10.105785079 (11h51m9.695130066s) [upload=12327]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:31:39.460569760 (11h51m39.049914747s) [upload=12327]: Done uploading segments... Aug 7 21:31:39.460597573 (11h51m39.04994255s) [upload=12327]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:32:24.324148762 (11h52m23.913493749s) [upload=12347]: Starting upload Aug 7 21:32:24.324206403 (11h52m23.91355139s) [upload=12347]: Done starting upload Aug 7 21:32:24.324252100 (11h52m23.913597087s) [upload=12347]: Uploading segments... Aug 7 21:32:24.324280415 (11h52m23.913625402s) [upload=12347]: Got next segment. Inline: true Aug 7 21:32:24.324284021 (11h52m23.913628998s) [upload=12347]: Waiting for error group managing segments... Aug 7 21:32:24.324329689 (11h52m23.913674686s) [upload=12347]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:32:24.324336372 (11h52m23.913681379s) [upload=12347]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:32:24.324341050 (11h52m23.913686057s) [upload=12347]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:32:53.678084170 (11h52m53.267429207s) [upload=12347]: Done uploading segments... Aug 7 21:32:53.678107405 (11h52m53.267452392s) [upload=12347]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:33:17.618427342 (11h53m17.207772389s) [upload=12364]: Starting upload Aug 7 21:33:17.618576718 (11h53m17.207921745s) [upload=12364]: Done starting upload Aug 7 21:33:17.618658725 (11h53m17.208003742s) [upload=12364]: Uploading segments... Aug 7 21:33:17.618713580 (11h53m17.208058597s) [upload=12364]: Got next segment. Inline: true Aug 7 21:33:17.618720533 (11h53m17.20806553s) [upload=12364]: Waiting for error group managing segments... Aug 7 21:33:17.618772433 (11h53m17.20811743s) [upload=12364]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:33:17.618779636 (11h53m17.208124643s) [upload=12364]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:33:17.618784776 (11h53m17.208129793s) [upload=12364]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:33:46.957323530 (11h53m46.546668577s) [upload=12364]: Done uploading segments... Aug 7 21:33:46.957345773 (11h53m46.54669075s) [upload=12364]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:34:19.197336641 (11h54m18.786681628s) [upload=12382]: Starting upload Aug 7 21:34:19.197461931 (11h54m18.786806918s) [upload=12382]: Done starting upload Aug 7 21:34:19.197512228 (11h54m18.786857225s) [upload=12382]: Uploading segments... Aug 7 21:34:19.197561282 (11h54m18.786906269s) [upload=12382]: Got next segment. Inline: true Aug 7 21:34:19.197565189 (11h54m18.786910166s) [upload=12382]: Waiting for error group managing segments... Aug 7 21:34:19.197610725 (11h54m18.786955723s) [upload=12382]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:34:19.197615966 (11h54m18.786960953s) [upload=12382]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:34:19.197619142 (11h54m18.786964139s) [upload=12382]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:34:48.535954936 (11h54m48.125299923s) [upload=12382]: Done uploading segments... Aug 7 21:34:48.535978650 (11h54m48.125323637s) [upload=12382]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:35:05.991757765 (11h55m5.581102752s) [upload=12393]: Starting upload Aug 7 21:35:05.991822780 (11h55m5.581167767s) [upload=12393]: Done starting upload Aug 7 21:35:05.991946456 (11h55m5.581291443s) [upload=12393]: Uploading segments... Aug 7 21:35:05.991988116 (11h55m5.581333104s) [upload=12393]: Got next segment. Inline: true Aug 7 21:35:05.991994929 (11h55m5.581339996s) [upload=12393]: Waiting for error group managing segments... Aug 7 21:35:05.992042741 (11h55m5.581387728s) [upload=12393]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:35:05.992048021 (11h55m5.581393018s) [upload=12393]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:35:05.992051768 (11h55m5.581396755s) [upload=12393]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:35:35.339368736 (11h55m34.928713723s) [upload=12393]: Done uploading segments... Aug 7 21:35:35.339503534 (11h55m34.928848522s) [upload=12393]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:36:12.952326182 (11h56m12.541671159s) [upload=12411]: Starting upload Aug 7 21:36:12.952385075 (11h56m12.541730052s) [upload=12411]: Done starting upload Aug 7 21:36:12.952428238 (11h56m12.541773225s) [upload=12411]: Uploading segments... Aug 7 21:36:12.952455510 (11h56m12.541800497s) [upload=12411]: Got next segment. Inline: true Aug 7 21:36:12.952459417 (11h56m12.541804404s) [upload=12411]: Waiting for error group managing segments... Aug 7 21:36:12.952507650 (11h56m12.541852637s) [upload=12411]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:36:12.952514352 (11h56m12.541859349s) [upload=12411]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:36:12.952518550 (11h56m12.541863537s) [upload=12411]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:36:42.299158316 (11h56m41.888503304s) [upload=12411]: Done uploading segments... Aug 7 21:36:42.299175039 (11h56m41.888520026s) [upload=12411]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:37:22.572699237 (11h57m22.162044224s) [upload=12432]: Starting upload Aug 7 21:37:22.572782937 (11h57m22.162127924s) [upload=12432]: Done starting upload Aug 7 21:37:22.572842260 (11h57m22.162187247s) [upload=12432]: Uploading segments... Aug 7 21:37:22.572885313 (11h57m22.1622303s) [upload=12432]: Got next segment. Inline: true Aug 7 21:37:22.572889591 (11h57m22.162234578s) [upload=12432]: Waiting for error group managing segments... Aug 7 21:37:22.572930138 (11h57m22.162275125s) [upload=12432]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:37:22.572934296 (11h57m22.162279283s) [upload=12432]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:37:22.572937121 (11h57m22.162282118s) [upload=12432]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:37:51.933954276 (11h57m51.523299253s) [upload=12432]: Done uploading segments... Aug 7 21:37:51.933970927 (11h57m51.523315914s) [upload=12432]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:38:28.745940745 (11h58m28.335285732s) [upload=12448]: Starting upload Aug 7 21:38:28.745997504 (11h58m28.335342491s) [upload=12448]: Done starting upload Aug 7 21:38:28.746040176 (11h58m28.335385163s) [upload=12448]: Uploading segments... Aug 7 21:38:28.746078458 (11h58m28.335423446s) [upload=12448]: Got next segment. Inline: true Aug 7 21:38:28.746081766 (11h58m28.335426743s) [upload=12448]: Waiting for error group managing segments... Aug 7 21:38:28.746111923 (11h58m28.33545691s) [upload=12448]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:38:28.746115540 (11h58m28.335460527s) [upload=12448]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:38:28.746118205 (11h58m28.335463192s) [upload=12448]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:38:58.105315693 (11h58m57.694660691s) [upload=12448]: Done uploading segments... Aug 7 21:38:58.105342465 (11h58m57.694687452s) [upload=12448]: Upload finished. err: metaclient: context deadline exceeded Duration: 30.000936974s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 25 out of 60 is failed (threshold=3) [...................................XXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.000825789s FAIL (2025-08-07 21:38:57 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.00083704s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 33 out of 60 is failed (threshold=3) [...........................XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.000366654s FAIL (2025-08-07 21:38:54 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.000389418s edge gateway us-select-1
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway) antiblip: 7 out of 60 is failed (threshold=3) [................................................XX.XX..X.XX.] duration: 230.539352ms remote: "tcp:207.211.208.129:443" target: https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222 FAIL (2025-08-07 21:38:53 UTC) orbiter warning: Get "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222": context deadline exceeded Duration: 230.550022ms edge gateway vhost bgp us1
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway) antiblip: 7 out of 60 is failed (threshold=3) [.................................................XX.XX..X.XX] duration: 2m0.000856508s remote: "tcp:207.211.208.129:443" target: https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222 FAIL (2025-08-07 21:38:52 UTC) orbiter warning: Get "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222": context deadline exceeded Duration: 2m0.000870496s edge gateway vhost bgp us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 29 out of 60 is failed (threshold=3) [.............................X..XXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.000616771s FAIL (2025-08-07 21:38:51 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 30.000630407s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 23 out of 60 is failed (threshold=3) [.....................................XXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.00061151s FAIL (2025-08-07 21:38:41 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== g 7 21:31:26.403163305 (11h51m25.992508292s) [noise=true , address=104.36.82.9:4036 , piece_id=473DGK6MQPSKHKLKCB3KMPELO5WHBGV7KTHEXKNPAKTUWKJRXABA, seg_pos={0 0}, upload=12321]: Done uploading piece. err: <nil> Aug 7 21:31:26.410169400 (11h51m25.999514377s) [noise=true , address=104.36.82.22:4524 , piece_id=UJUNX53EZDOT7NMYKILPPTRBJNOONOIGYULH5IOHQ73W5PMD7XJA, seg_pos={0 0}, upload=12321]: Done uploading piece. err: <nil> Aug 7 21:31:26.411736859 (11h51m26.001081846s) [noise=true , address=104.36.82.4:4825 , piece_id=TAAMMJMZEEO7UYZM6VVZUPSX53EGSOJTJ3VJY5HMEDUGLRRND3YA, seg_pos={0 0}, upload=12321]: Done uploading piece. err: <nil> Aug 7 21:31:26.420123765 (11h51m26.009468752s) [noise=true , address=104.36.82.56:4249 , piece_id=M5FBC6LH2A2ZCGSJJI6KUVYDO6UELARWIHMD22XUP6GQFVP2ZFXA, seg_pos={0 0}, upload=12321]: Done uploading piece. err: <nil> Aug 7 21:31:26.428567841 (11h51m26.017912818s) [noise=true , address=104.36.82.38:4714 , piece_id=7AYVJ5FRQUIZRQ2NTCUN43CWJWWNQZEHMD7UE3MCYXSRBFQOL56A, seg_pos={0 0}, upload=12321]: Done uploading piece. err: <nil> Aug 7 21:31:26.430706232 (11h51m26.020051219s) [noise=true , address=104.36.82.59:4313 , piece_id=BLGX433PKGAVB5WCNKXMA6P3EYPPWCPCB6WYHQ5RZ4I6WRUH4GHQ, seg_pos={0 0}, upload=12321]: Done uploading piece. err: <nil> Aug 7 21:31:26.612329184 (11h51m26.201674171s) [noise=true , address=104.36.82.55:4222 , piece_id=4W52URT2PUTVOIVFFYOFSE3RDOVBHMS56PG57MEEKCFU7QP3CZGQ, seg_pos={0 0}, upload=12321]: Done uploading piece. err: <nil> Aug 7 21:31:26.658377737 (11h51m26.247722724s) [noise=true , address=104.36.82.54:4162 , piece_id=6RDDMFE4ZG2YJG5GKRFLSANPJXNKADJXLY6H64A53RPJPSXEC4BA, seg_pos={0 0}, upload=12321]: Done uploading piece. err: <nil> Aug 7 21:31:26.674652975 (11h51m26.263997962s) [noise=true , address=104.36.82.45:4937 , piece_id=EBUAJKXBK6RZK7T46KXJT5P5LINBFVZCEPGVGG5OZ66SYNOA2RSQ, seg_pos={0 0}, upload=12321]: Done uploading piece. err: <nil> Aug 7 21:31:26.680556480 (11h51m26.269901467s) [noise=true , address=45.91.116.10:4110 , piece_id=F4ZMNLCCFQCBFB6FOPOJFTGPIL5BFYT6TPGL5C7SVLIHMTY4EJJA, seg_pos={0 0}, upload=12321]: Done uploading piece. err: <nil> Aug 7 21:31:26.680580406 (11h51m26.269925393s) [seg_pos={0 0}, upload=12321]: Segment reached optimal threshold of 14 pieces. Aug 7 21:31:26.680752626 (11h51m26.270097613s) [noise=true , address=45.91.116.6:4983 , piece_id=ZWH64KJWCVSTEGVN25WW35HRHTAO3BE3SPKKALNRT7NNLUKB4L4Q, seg_pos={0 0}, upload=12321]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122trJbDHuqeBMQHZT97HP2PgkeuLyUiDiuRm4nayu6ZTH7tat6): protocol: expected piece hash; context canceled; EOF Aug 7 21:31:26.680804515 (11h51m26.270149492s) [noise=true , address=45.91.116.7:4037 , piece_id=WY2F64PFXEGFVE4M6PUHHUYIMTI2KB4CB73DQADZK4QGIEZ2YCYQ, seg_pos={0 0}, upload=12321]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iNPwE2XSfXcaXAXcrH2jcyNGYeDiiWvzAcDQffrmjPwTVqVmK): protocol: expected piece hash; context canceled; EOF Aug 7 21:31:26.680954802 (11h51m26.270299789s) [noise=true , address=104.36.82.61:4375 , piece_id=5LQOQ3MR3EA36DIYLOVO6YDKF6EP2HPHMPG4P6MN76A4IA5BTBEQ, seg_pos={0 0}, upload=12321]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121fzX9v4rgEd78xnDAwcQtthyQHV3yUZUQYLXZ9byPDucGK4T5): protocol: write tcp 188.166.225.170:45604->104.36.82.61:4375: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 21:31:26.680994198 (11h51m26.270339185s) [noise=true , address=104.36.82.8:4977 , piece_id=VY47HC4FF6JU4QQTDVZ62NOIEGWSROTWYMGCB7XXNHJ3GZNJATMQ, seg_pos={0 0}, upload=12321]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WCAm12tXwimFSPqz7Seh6A4oxWqQVxqsF459xux4a5CqjpbAT): protocol: write tcp 188.166.225.170:36156->104.36.82.8:4977: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 21:31:26.681016300 (11h51m26.270361287s) [seg_pos={0 0}, upload=12321]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 21:31:26.681067097 (11h51m26.270412074s) [upload=12321]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 21:31:26.681072137 (11h51m26.270417114s) [upload=12321]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:32:39.816785277 (11h52m39.406130264s) [upload=12321]: Done uploading segments... Aug 7 21:32:39.816815415 (11h52m39.406160402s) [upload=12321]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:32:40.684664778 (11h52m40.274009755s) [upload=12355]: Starting upload Aug 7 21:32:40.684721386 (11h52m40.274066373s) [upload=12355]: Done starting upload Aug 7 21:32:40.684776000 (11h52m40.274121017s) [upload=12355]: Uploading segments... Aug 7 21:32:40.684893314 (11h52m40.274238292s) [upload=12355]: Got next segment. Inline: false Aug 7 21:32:40.684916699 (11h52m40.274261686s) [upload=12355]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:32:40.684920466 (11h52m40.274265443s) [upload=12355]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:33:40.036921304 (11h53m39.626266281s) [upload=12355]: Next segment err: context deadline exceeded Aug 7 21:33:40.037128099 (11h53m39.626473086s) [upload=12355]: Done uploading segments... Aug 7 21:33:40.037148919 (11h53m39.626493896s) [upload=12355]: Upload finished. err: context deadline exceeded Aug 7 21:33:40.899438535 (11h53m40.488783522s) [upload=12372]: Starting upload Aug 7 21:33:40.899490825 (11h53m40.488835832s) [upload=12372]: Done starting upload Aug 7 21:33:40.899590144 (11h53m40.488935131s) [upload=12372]: Uploading segments... Aug 7 21:33:40.899771832 (11h53m40.489116819s) [upload=12372]: Got next segment. Inline: false Aug 7 21:33:40.899795227 (11h53m40.489140214s) [upload=12372]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:33:40.899799275 (11h53m40.489144252s) [upload=12372]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:34:40.257761531 (11h54m39.847106518s) [upload=12372]: Next segment err: context deadline exceeded Aug 7 21:34:40.257885859 (11h54m39.847230856s) [upload=12372]: Done uploading segments... Aug 7 21:34:40.257904695 (11h54m39.847249672s) [upload=12372]: Upload finished. err: context deadline exceeded Aug 7 21:34:41.144124896 (11h54m40.733469873s) [upload=12386]: Starting upload Aug 7 21:34:41.144233103 (11h54m40.73357813s) [upload=12386]: Done starting upload Aug 7 21:34:41.144343113 (11h54m40.733688101s) [upload=12386]: Uploading segments... Aug 7 21:34:41.144700337 (11h54m40.734045364s) [upload=12386]: Got next segment. Inline: false Aug 7 21:34:41.144760612 (11h54m40.734105639s) [upload=12386]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:34:41.144768206 (11h54m40.734113223s) [upload=12386]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:35:40.491924814 (11h55m40.081269811s) [upload=12386]: Next segment err: context deadline exceeded Aug 7 21:35:40.492048891 (11h55m40.081393878s) [upload=12386]: Done uploading segments... Aug 7 21:35:40.492066364 (11h55m40.081411351s) [upload=12386]: Upload finished. err: context deadline exceeded Aug 7 21:35:41.353263670 (11h55m40.942608657s) [upload=12405]: Starting upload Aug 7 21:35:41.353369352 (11h55m40.942714339s) [upload=12405]: Done starting upload Aug 7 21:35:41.353452601 (11h55m40.942797588s) [upload=12405]: Uploading segments... Aug 7 21:35:41.353579473 (11h55m40.942924511s) [upload=12405]: Got next segment. Inline: false Aug 7 21:35:41.353621805 (11h55m40.942966792s) [upload=12405]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:35:41.353626053 (11h55m40.94297104s) [upload=12405]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:36:40.713868241 (11h56m40.303213228s) [upload=12405]: Next segment err: context deadline exceeded Aug 7 21:36:40.713958674 (11h56m40.303303651s) [upload=12405]: Done uploading segments... Aug 7 21:36:40.713972460 (11h56m40.303317447s) [upload=12405]: Upload finished. err: context deadline exceeded Aug 7 21:36:41.596698707 (11h56m41.186043744s) [upload=12418]: Starting upload Aug 7 21:36:41.596808026 (11h56m41.186153063s) [upload=12418]: Done starting upload Aug 7 21:36:41.596885475 (11h56m41.186230462s) [upload=12418]: Uploading segments... Aug 7 21:36:41.597098722 (11h56m41.186443709s) [upload=12418]: Got next segment. Inline: false Aug 7 21:36:41.597141925 (11h56m41.186486912s) [upload=12418]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:36:41.597146604 (11h56m41.186491591s) [upload=12418]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:37:40.947744552 (11h57m40.537089539s) [upload=12418]: Next segment err: context deadline exceeded Aug 7 21:37:40.947825697 (11h57m40.537170684s) [upload=12418]: Done uploading segments... Aug 7 21:37:40.947843592 (11h57m40.537188579s) [upload=12418]: Upload finished. err: context deadline exceeded Aug 7 21:37:41.823629920 (11h57m41.412974907s) [upload=12437]: Starting upload Aug 7 21:37:41.823680908 (11h57m41.413025895s) [upload=12437]: Done starting upload Aug 7 21:37:41.823826305 (11h57m41.413171293s) [upload=12437]: Uploading segments... Aug 7 21:37:41.823857055 (11h57m41.413202052s) [upload=12437]: Got next segment. Inline: false Aug 7 21:37:41.823914255 (11h57m41.413259242s) [upload=12437]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:37:41.823918503 (11h57m41.41326348s) [upload=12437]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:38:41.179029912 (11h58m40.768374899s) [upload=12437]: Next segment err: context deadline exceeded Aug 7 21:38:41.179128490 (11h58m40.768473477s) [upload=12437]: Done uploading segments... Aug 7 21:38:41.179142979 (11h58m40.768487966s) [upload=12437]: Upload finished. err: context deadline exceeded Duration: 1m0.000622852s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 32 out of 60 is failed (threshold=3) [............................XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.001273662s FAIL (2025-08-07 21:38:40 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.001283472s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 23 out of 60 is failed (threshold=3) [.....................................XXXXXXXXXXXXXXXXXXXXXXX] duration: 30.001268602s FAIL (2025-08-07 21:38:38 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 30.001287608s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 25 out of 60 is failed (threshold=3) [..................................X.XXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.000536689s FAIL (2025-08-07 21:38:32 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 107s) [upload=12277]: Uploading segments... Aug 7 21:28:18.552547911 (11h48m18.141892898s) [upload=12277]: Got next segment. Inline: true Aug 7 21:28:18.552552150 (11h48m18.141897137s) [upload=12277]: Waiting for error group managing segments... Aug 7 21:28:18.552600933 (11h48m18.14194592s) [upload=12277]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:28:18.552607205 (11h48m18.141952212s) [upload=12277]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:28:18.552612195 (11h48m18.141957192s) [upload=12277]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:29:17.904886609 (11h49m17.494231596s) [upload=12277]: Done uploading segments... Aug 7 21:29:17.904932337 (11h49m17.494277354s) [upload=12277]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:29:18.793973395 (11h49m18.383318382s) [upload=12293]: Starting upload Aug 7 21:29:18.794029602 (11h49m18.383374589s) [upload=12293]: Done starting upload Aug 7 21:29:18.794095498 (11h49m18.383440485s) [upload=12293]: Uploading segments... Aug 7 21:29:18.794182454 (11h49m18.383527441s) [upload=12293]: Got next segment. Inline: true Aug 7 21:29:18.794187423 (11h49m18.383532411s) [upload=12293]: Waiting for error group managing segments... Aug 7 21:29:18.794239163 (11h49m18.38358415s) [upload=12293]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:29:18.794244643 (11h49m18.38358962s) [upload=12293]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:29:18.794247088 (11h49m18.383592076s) [upload=12293]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:30:18.146928865 (11h50m17.736273852s) [upload=12293]: Done uploading segments... Aug 7 21:30:18.146964162 (11h50m17.73630915s) [upload=12293]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:30:30.199348334 (11h50m29.788693321s) [upload=12315]: Starting upload Aug 7 21:30:30.199433327 (11h50m29.788778314s) [upload=12315]: Done starting upload Aug 7 21:30:30.199483272 (11h50m29.788828259s) [upload=12315]: Uploading segments... Aug 7 21:30:30.199515665 (11h50m29.788860652s) [upload=12315]: Got next segment. Inline: true Aug 7 21:30:30.199519302 (11h50m29.788864289s) [upload=12315]: Waiting for error group managing segments... Aug 7 21:30:30.199554779 (11h50m29.788899766s) [upload=12315]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:30:30.199559147 (11h50m29.788904134s) [upload=12315]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:30:30.199561592 (11h50m29.788906579s) [upload=12315]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:31:29.557680478 (11h51m29.147025465s) [upload=12315]: Done uploading segments... Aug 7 21:31:29.557706508 (11h51m29.147051485s) [upload=12315]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:31:30.468794234 (11h51m30.058139221s) [upload=12336]: Starting upload Aug 7 21:31:30.468935564 (11h51m30.058280551s) [upload=12336]: Done starting upload Aug 7 21:31:30.469048480 (11h51m30.058393467s) [upload=12336]: Uploading segments... Aug 7 21:31:30.469142200 (11h51m30.058487187s) [upload=12336]: Got next segment. Inline: true Aug 7 21:31:30.469147400 (11h51m30.058492387s) [upload=12336]: Waiting for error group managing segments... Aug 7 21:31:30.469203888 (11h51m30.058548875s) [upload=12336]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:31:30.469208757 (11h51m30.058553744s) [upload=12336]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:31:30.469211582 (11h51m30.058556569s) [upload=12336]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:32:29.815552389 (11h52m29.404897376s) [upload=12336]: Done uploading segments... Aug 7 21:32:29.815578508 (11h52m29.404923495s) [upload=12336]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:32:30.694545842 (11h52m30.283890829s) [upload=12349]: Starting upload Aug 7 21:32:30.694631607 (11h52m30.283976624s) [upload=12349]: Done starting upload Aug 7 21:32:30.694676312 (11h52m30.284021299s) [upload=12349]: Uploading segments... Aug 7 21:32:30.694717561 (11h52m30.284062538s) [upload=12349]: Got next segment. Inline: true Aug 7 21:32:30.694721909 (11h52m30.284066896s) [upload=12349]: Waiting for error group managing segments... Aug 7 21:32:30.694762737 (11h52m30.284107724s) [upload=12349]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:32:30.694766976 (11h52m30.284111953s) [upload=12349]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:32:30.694769561 (11h52m30.284114548s) [upload=12349]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:33:30.042542113 (11h53m29.6318871s) [upload=12349]: Done uploading segments... Aug 7 21:33:30.042561801 (11h53m29.631906788s) [upload=12349]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:33:30.916211822 (11h53m30.505556809s) [upload=12369]: Starting upload Aug 7 21:33:30.916281525 (11h53m30.505626512s) [upload=12369]: Done starting upload Aug 7 21:33:30.916351519 (11h53m30.505696567s) [upload=12369]: Uploading segments... Aug 7 21:33:30.916406474 (11h53m30.505751502s) [upload=12369]: Got next segment. Inline: true Aug 7 21:33:30.916413598 (11h53m30.505758605s) [upload=12369]: Waiting for error group managing segments... Aug 7 21:33:30.916454065 (11h53m30.505799052s) [upload=12369]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:33:30.916457992 (11h53m30.505802979s) [upload=12369]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:33:30.916460378 (11h53m30.505805355s) [upload=12369]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:34:30.262869110 (11h54m29.852214097s) [upload=12369]: Done uploading segments... Aug 7 21:34:30.262896352 (11h54m29.85224134s) [upload=12369]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:34:31.146607534 (11h54m30.735952521s) [upload=12385]: Starting upload Aug 7 21:34:31.146655144 (11h54m30.736000121s) [upload=12385]: Done starting upload Aug 7 21:34:31.146698838 (11h54m30.736043825s) [upload=12385]: Uploading segments... Aug 7 21:34:31.146741720 (11h54m30.736086697s) [upload=12385]: Got next segment. Inline: true Aug 7 21:34:31.146745548 (11h54m30.736090535s) [upload=12385]: Waiting for error group managing segments... Aug 7 21:34:31.146776246 (11h54m30.736121233s) [upload=12385]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:34:31.146779913 (11h54m30.73612489s) [upload=12385]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:34:31.146782388 (11h54m30.736127375s) [upload=12385]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:35:30.500521706 (11h55m30.089866693s) [upload=12385]: Done uploading segments... Aug 7 21:35:30.500542195 (11h55m30.089887182s) [upload=12385]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:35:31.371815861 (11h55m30.961160848s) [upload=12402]: Starting upload Aug 7 21:35:31.371969656 (11h55m30.961314643s) [upload=12402]: Done starting upload Aug 7 21:35:31.372062443 (11h55m30.96140743s) [upload=12402]: Uploading segments... Aug 7 21:35:31.372108090 (11h55m30.961453077s) [upload=12402]: Got next segment. Inline: true Aug 7 21:35:31.372114412 (11h55m30.961459399s) [upload=12402]: Waiting for error group managing segments... Aug 7 21:35:31.372166783 (11h55m30.96151177s) [upload=12402]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:35:31.372172002 (11h55m30.961516999s) [upload=12402]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:35:31.372174898 (11h55m30.961519885s) [upload=12402]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:36:30.721260917 (11h56m30.310605904s) [upload=12402]: Done uploading segments... Aug 7 21:36:30.721280625 (11h56m30.310625612s) [upload=12402]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:36:33.253873027 (11h56m32.843218014s) [upload=12416]: Starting upload Aug 7 21:36:33.253941188 (11h56m32.843286175s) [upload=12416]: Done starting upload Aug 7 21:36:33.254004108 (11h56m32.843349085s) [upload=12416]: Uploading segments... Aug 7 21:36:33.254039966 (11h56m32.843384953s) [upload=12416]: Got next segment. Inline: true Aug 7 21:36:33.254043462 (11h56m32.84338845s) [upload=12416]: Waiting for error group managing segments... Aug 7 21:36:33.254074321 (11h56m32.843419308s) [upload=12416]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:36:33.254078290 (11h56m32.843423277s) [upload=12416]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:36:33.254080714 (11h56m32.843425701s) [upload=12416]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:37:32.603532648 (11h57m32.192877635s) [upload=12416]: Done uploading segments... Aug 7 21:37:32.603556113 (11h57m32.1929011s) [upload=12416]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:37:33.470049892 (11h57m33.059394869s) [upload=12434]: Starting upload Aug 7 21:37:33.470104877 (11h57m33.059449865s) [upload=12434]: Done starting upload Aug 7 21:37:33.470145565 (11h57m33.059490552s) [upload=12434]: Uploading segments... Aug 7 21:37:33.470187375 (11h57m33.059532352s) [upload=12434]: Got next segment. Inline: true Aug 7 21:37:33.470190872 (11h57m33.059535859s) [upload=12434]: Waiting for error group managing segments... Aug 7 21:37:33.470223234 (11h57m33.059568221s) [upload=12434]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:37:33.470226710 (11h57m33.059571697s) [upload=12434]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:37:33.470228904 (11h57m33.059573881s) [upload=12434]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:38:32.827667037 (11h58m32.417012024s) [upload=12434]: Done uploading segments... Aug 7 21:38:32.827687857 (11h58m32.417032834s) [upload=12434]: Upload finished. err: metaclient: context deadline exceeded Duration: 1m0.00054824s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway) antiblip: 4 out of 60 is failed (threshold=3) [.......................................X....X.X...X.........] duration: 698.155135ms remote: "tcp:45.38.24.2:443" target: https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410 FAIL (2025-08-07 21:38:32 UTC) orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": context deadline exceeded Duration: 698.202555ms edge gateway vhost us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 6 out of 60 is failed (threshold=3) [X.......................X.........X......X..X.............X.] duration: 6.219709682s FAIL (2025-08-07 21:38:26 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/4M):: after 3c7be29e56ef0a348f54a62643273e5baec016fc19e64e2fa674114dac8dddcf != before 2decbc1b39ad0a3120f73e14d303cc93862fc082b00150c3e706fb25a679be76 ======LOGS====== d=12436]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:37:41.241548466 (11h57m40.830893473s) [upload=12436]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:37:41.699604637 (11h57m41.288949624s) [seg_pos={0 0}, upload=12436]: Begin upload segment... Aug 7 21:37:41.699642941 (11h57m41.288987928s) [seg_pos={0 0}, upload=12436]: Done begin upload segment. Aug 7 21:37:41.699665653 (11h57m41.28901063s) [noise=true , address=64.49.13.137:4494 , piece_id=27I45T5P6HE5MB7KLKHIDYF3GKGB4X5EYUQEQCZAIPO56TWSUIXQ, seg_pos={0 0}, upload=12436]: Uploading piece... Aug 7 21:37:41.699677036 (11h57m41.289022023s) [noise=true , address=104.36.82.37:4671 , piece_id=GAEIZGDLAYYQ2ATH4PCFWG7ZDNDVLY3RWQNDKPZNVPBYZ4EPCCBQ, seg_pos={0 0}, upload=12436]: Uploading piece... Aug 7 21:37:41.700019711 (11h57m41.289364698s) [noise=true , address=104.36.82.58:4271 , piece_id=VBXDX65YCLKD7WWXXCKLDKNOCUBMA4PTK53IMIJKCM5M5AOQRRYQ, seg_pos={0 0}, upload=12436]: Uploading piece... Aug 7 21:37:41.700313864 (11h57m41.289658851s) [noise=true , address=104.36.82.55:4193 , piece_id=YO4COWLA4546LGDVL7E3X7CGY2TLKVGEMVBI6A3D7LCI226ZQGMA, seg_pos={0 0}, upload=12436]: Uploading piece... Aug 7 21:37:41.700321118 (11h57m41.289666105s) [noise=true , address=104.36.82.8:4975 , piece_id=MPDCSPMDGY4MEJHSXYOTT3VFFAZGRABQH7QR5VLSD2OWB25C2DBQ, seg_pos={0 0}, upload=12436]: Uploading piece... Aug 7 21:37:41.700555927 (11h57m41.289900914s) [noise=true , address=104.36.82.37:4683 , piece_id=ZYQOB47RWRIT4GC36XROSHCGKBKTYNE5PRDTECJJ4RIEWRXLNZ2Q, seg_pos={0 0}, upload=12436]: Uploading piece... Aug 7 21:37:41.700769827 (11h57m41.290114814s) [noise=true , address=79.127.201.215:4356 , piece_id=VA6MG6UXZ7VUH5LAMLMBPIK7C2GLTDXA33LCFDIZMBPXDDPLVXIA, seg_pos={0 0}, upload=12436]: Uploading piece... Aug 7 21:37:41.700792900 (11h57m41.290137887s) [noise=true , address=104.36.82.23:4447 , piece_id=ZPOANDX46722LIXWQSDNXMFDB4ZHSS765MZQ3NOF4JXPUNZ4TFPQ, seg_pos={0 0}, upload=12436]: Uploading piece... Aug 7 21:37:41.701033912 (11h57m41.290378889s) [noise=true , address=104.36.82.43:4861 , piece_id=AXOCVGDO4GD7PWM43RYNWUPGX2H7BK4X4QCWTAJ4C3CNFL7UICRA, seg_pos={0 0}, upload=12436]: Uploading piece... Aug 7 21:37:41.701082625 (11h57m41.290427642s) [noise=true , address=104.36.82.59:4314 , piece_id=RF5LCHJAO7EOIT4AVOUVKKYVVBST4F3S5BO63CPYVJBI4CAWLGJA, seg_pos={0 0}, upload=12436]: Uploading piece... Aug 7 21:37:41.701257619 (11h57m41.290602606s) [noise=true , address=104.36.82.60:4358 , piece_id=MQTJBJP572O6M7CC756QSNAU4LUWO67OD7FJDNHIXOVANN7GDPQQ, seg_pos={0 0}, upload=12436]: Uploading piece... Aug 7 21:37:41.701341520 (11h57m41.290686507s) [noise=true , address=104.36.82.58:4285 , piece_id=E7G4SBPGKX3OAR3J2Y2PG67ZWXLKAJWJ5L34KVWLDX44IAQHZFNA, seg_pos={0 0}, upload=12436]: Uploading piece... Aug 7 21:37:41.701453143 (11h57m41.29079813s) [noise=true , address=104.36.82.14:4242 , piece_id=BOU64PUQXF4JOVE7GEVOYSXQKQPRAI75BRRHO577Q3BVXES5C7EQ, seg_pos={0 0}, upload=12436]: Uploading piece... Aug 7 21:37:41.701589534 (11h57m41.290934521s) [noise=true , address=45.91.116.11:4150 , piece_id=CUAREN64B6BVTXHF3RIJPH6LRPR3QEPVHRKLZDVXVMZFWBEXVBSA, seg_pos={0 0}, upload=12436]: Uploading piece... Aug 7 21:37:41.701638548 (11h57m41.290983535s) [noise=true , address=104.36.82.4:4830 , piece_id=MKXIFYHW4KNTLS4AIHCEI53CE7STNGL5ITSZA7IKWF36QMF2MSMQ, seg_pos={0 0}, upload=12436]: Uploading piece... Aug 7 21:37:41.701846857 (11h57m41.291191844s) [noise=true , address=104.36.82.46:4508 , piece_id=LVE3T3E6XA5TR3W4XANXE44O54BMSBSAHLBRS2XROOEAXVF2K6DA, seg_pos={0 0}, upload=12436]: Uploading piece... Aug 7 21:37:41.701850394 (11h57m41.291195381s) [noise=true , address=45.91.116.14:4205 , piece_id=PSGW5VTZ3JNFGHFGEO2IWQN4A6DUYEFOYGLOCJHCX625AZGSJHZA, seg_pos={0 0}, upload=12436]: Uploading piece... Aug 7 21:37:41.702036911 (11h57m41.291381898s) [noise=true , address=45.91.116.7:4040 , piece_id=ILATCWOLO5UO5RPRF4IRL7Y4JNMYLUIBFNLZBUDUAJKQDVLTXCCQ, seg_pos={0 0}, upload=12436]: Uploading piece... Aug 7 21:37:42.901972883 (11h57m42.49131787s) [upload=12436]: Next returned nil segment Aug 7 21:37:42.901987461 (11h57m42.491332448s) [upload=12436]: Waiting for error group managing segments... Aug 7 21:37:43.221358460 (11h57m42.810703437s) [noise=true , address=64.49.13.137:4494 , piece_id=27I45T5P6HE5MB7KLKHIDYF3GKGB4X5EYUQEQCZAIPO56TWSUIXQ, seg_pos={0 0}, upload=12436]: Done uploading piece. err: <nil> Aug 7 21:37:43.635176843 (11h57m43.22452183s) [noise=true , address=45.91.116.7:4040 , piece_id=ILATCWOLO5UO5RPRF4IRL7Y4JNMYLUIBFNLZBUDUAJKQDVLTXCCQ, seg_pos={0 0}, upload=12436]: Done uploading piece. err: <nil> Aug 7 21:37:43.691989821 (11h57m43.281334828s) [noise=true , address=79.127.201.215:4356 , piece_id=VA6MG6UXZ7VUH5LAMLMBPIK7C2GLTDXA33LCFDIZMBPXDDPLVXIA, seg_pos={0 0}, upload=12436]: Done uploading piece. err: <nil> Aug 7 21:37:43.715706768 (11h57m43.305051755s) [noise=true , address=104.36.82.8:4975 , piece_id=MPDCSPMDGY4MEJHSXYOTT3VFFAZGRABQH7QR5VLSD2OWB25C2DBQ, seg_pos={0 0}, upload=12436]: Done uploading piece. err: <nil> Aug 7 21:37:43.732997881 (11h57m43.322342868s) [noise=true , address=104.36.82.14:4242 , piece_id=BOU64PUQXF4JOVE7GEVOYSXQKQPRAI75BRRHO577Q3BVXES5C7EQ, seg_pos={0 0}, upload=12436]: Done uploading piece. err: <nil> Aug 7 21:37:43.741488406 (11h57m43.330833383s) [noise=true , address=104.36.82.46:4508 , piece_id=LVE3T3E6XA5TR3W4XANXE44O54BMSBSAHLBRS2XROOEAXVF2K6DA, seg_pos={0 0}, upload=12436]: Done uploading piece. err: <nil> Aug 7 21:37:43.879657469 (11h57m43.469002456s) [noise=true , address=104.36.82.37:4671 , piece_id=GAEIZGDLAYYQ2ATH4PCFWG7ZDNDVLY3RWQNDKPZNVPBYZ4EPCCBQ, seg_pos={0 0}, upload=12436]: Done uploading piece. err: <nil> Aug 7 21:37:43.895681587 (11h57m43.485026574s) [noise=true , address=45.91.116.11:4150 , piece_id=CUAREN64B6BVTXHF3RIJPH6LRPR3QEPVHRKLZDVXVMZFWBEXVBSA, seg_pos={0 0}, upload=12436]: Done uploading piece. err: <nil> Aug 7 21:37:43.907596774 (11h57m43.496941751s) [noise=true , address=104.36.82.4:4830 , piece_id=MKXIFYHW4KNTLS4AIHCEI53CE7STNGL5ITSZA7IKWF36QMF2MSMQ, seg_pos={0 0}, upload=12436]: Done uploading piece. err: <nil> Aug 7 21:37:43.908572762 (11h57m43.497917749s) [noise=true , address=45.91.116.14:4205 , piece_id=PSGW5VTZ3JNFGHFGEO2IWQN4A6DUYEFOYGLOCJHCX625AZGSJHZA, seg_pos={0 0}, upload=12436]: Done uploading piece. err: <nil> Aug 7 21:37:43.912238615 (11h57m43.501583602s) [noise=true , address=104.36.82.60:4358 , piece_id=MQTJBJP572O6M7CC756QSNAU4LUWO67OD7FJDNHIXOVANN7GDPQQ, seg_pos={0 0}, upload=12436]: Done uploading piece. err: <nil> Aug 7 21:37:43.917180210 (11h57m43.506525197s) [noise=true , address=104.36.82.37:4683 , piece_id=ZYQOB47RWRIT4GC36XROSHCGKBKTYNE5PRDTECJJ4RIEWRXLNZ2Q, seg_pos={0 0}, upload=12436]: Done uploading piece. err: <nil> Aug 7 21:37:43.957535420 (11h57m43.546880407s) [noise=true , address=104.36.82.59:4314 , piece_id=RF5LCHJAO7EOIT4AVOUVKKYVVBST4F3S5BO63CPYVJBI4CAWLGJA, seg_pos={0 0}, upload=12436]: Done uploading piece. err: <nil> Aug 7 21:37:44.104572309 (11h57m43.693917296s) [noise=true , address=104.36.82.43:4861 , piece_id=AXOCVGDO4GD7PWM43RYNWUPGX2H7BK4X4QCWTAJ4C3CNFL7UICRA, seg_pos={0 0}, upload=12436]: Done uploading piece. err: <nil> Aug 7 21:37:44.104599301 (11h57m43.693944278s) [seg_pos={0 0}, upload=12436]: Segment reached optimal threshold of 14 pieces. Aug 7 21:37:44.104687199 (11h57m43.694032176s) [noise=true , address=104.36.82.23:4447 , piece_id=ZPOANDX46722LIXWQSDNXMFDB4ZHSS765MZQ3NOF4JXPUNZ4TFPQ, seg_pos={0 0}, upload=12436]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fqT9xA7kXTG8K9XwTUxL6FXibV35AkPHDK6grKGHg17fk91fP): protocol: expected piece hash; context canceled; EOF Aug 7 21:37:44.104922059 (11h57m43.694267046s) [noise=true , address=104.36.82.58:4271 , piece_id=VBXDX65YCLKD7WWXXCKLDKNOCUBMA4PTK53IMIJKCM5M5AOQRRYQ, seg_pos={0 0}, upload=12436]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QwLHnMQHafFPYL83gE6wffvzdkvCHwzzVMLBz4KBVdE7ynuf1): protocol: rpc: dial tcp 104.36.82.58:4271: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 21:37:44.104969008 (11h57m43.694313995s) [noise=true , address=104.36.82.58:4285 , piece_id=E7G4SBPGKX3OAR3J2Y2PG67ZWXLKAJWJ5L34KVWLDX44IAQHZFNA, seg_pos={0 0}, upload=12436]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rZhbymrDS6ahXLEqYRM64gcqRvG8CEnc8iGCTEhgXrpvRaQrp): protocol: write tcp 188.166.225.170:56894->104.36.82.58:4285: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 21:37:44.104982013 (11h57m43.69432699s) [noise=true , address=104.36.82.55:4193 , piece_id=YO4COWLA4546LGDVL7E3X7CGY2TLKVGEMVBI6A3D7LCI226ZQGMA, seg_pos={0 0}, upload=12436]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: write tcp 188.166.225.170:60782->104.36.82.55:4193: use of closed network connection; write tcp 188.166.225.170:60782->104.36.82.55:4193: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60782->104.36.82.55:4193: use of closed network connection Aug 7 21:37:44.104993535 (11h57m43.694338522s) [seg_pos={0 0}, upload=12436]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 21:37:44.105075582 (11h57m43.694420569s) [upload=12436]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 21:37:44.105079359 (11h57m43.694424336s) [upload=12436]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:37:44.447186820 (11h57m44.036531807s) [upload=12436]: Done uploading segments... Aug 7 21:37:44.447217970 (11h57m44.036562957s) [upload=12436]: Upload finished. err: <nil> Duration: 6.219735122s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 28 out of 60 is failed (threshold=3) [..............................X..XXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.001414623s FAIL (2025-08-07 21:38:21 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 30.00142888s core uplink us1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 32 out of 60 is failed (threshold=3) [............................XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.000820135s FAIL (2025-08-07 21:38:16 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.000830134s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 3 out of 60 is failed (threshold=3) [X...................................X...............X.......] duration: 1.826929815s FAIL (2025-08-07 21:38:09 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/1K") ======LOGS====== 083457s) [upload=12258]: Done uploading segments... Aug 7 21:26:43.142756605 (11h46m42.732101593s) [upload=12258]: Upload finished. err: <nil> Aug 7 21:27:16.645746716 (11h47m16.235091703s) [upload=12264]: Starting upload Aug 7 21:27:16.645808424 (11h47m16.235153411s) [upload=12264]: Done starting upload Aug 7 21:27:16.645853019 (11h47m16.235198006s) [upload=12264]: Uploading segments... Aug 7 21:27:16.645881223 (11h47m16.23522621s) [upload=12264]: Got next segment. Inline: true Aug 7 21:27:16.645884680 (11h47m16.235229667s) [upload=12264]: Waiting for error group managing segments... Aug 7 21:27:16.645915299 (11h47m16.235260286s) [upload=12264]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:27:16.645919216 (11h47m16.235264203s) [upload=12264]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:27:16.645921510 (11h47m16.235266498s) [upload=12264]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:27:16.954194817 (11h47m16.543539804s) [upload=12264]: Done uploading segments... Aug 7 21:27:16.954212471 (11h47m16.543557459s) [upload=12264]: Upload finished. err: <nil> Aug 7 21:27:46.945635033 (11h47m46.53498005s) [upload=12274]: Starting upload Aug 7 21:27:46.945720126 (11h47m46.535065163s) [upload=12274]: Done starting upload Aug 7 21:27:46.945850435 (11h47m46.535195422s) [upload=12274]: Uploading segments... Aug 7 21:27:46.945918004 (11h47m46.535262991s) [upload=12274]: Got next segment. Inline: true Aug 7 21:27:46.945922443 (11h47m46.53526742s) [upload=12274]: Waiting for error group managing segments... Aug 7 21:27:46.945960726 (11h47m46.535305713s) [upload=12274]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:27:46.945965345 (11h47m46.535310332s) [upload=12274]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:27:46.945999029 (11h47m46.535344026s) [upload=12274]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:27:47.238086931 (11h47m46.827431908s) [upload=12274]: Done uploading segments... Aug 7 21:27:47.238104565 (11h47m46.827449552s) [upload=12274]: Upload finished. err: <nil> Aug 7 21:28:27.597717226 (11h48m27.187062213s) [upload=12282]: Starting upload Aug 7 21:28:27.597768604 (11h48m27.187113581s) [upload=12282]: Done starting upload Aug 7 21:28:27.597851232 (11h48m27.187196219s) [upload=12282]: Uploading segments... Aug 7 21:28:27.597881140 (11h48m27.187226127s) [upload=12282]: Got next segment. Inline: true Aug 7 21:28:27.597885238 (11h48m27.187230225s) [upload=12282]: Waiting for error group managing segments... Aug 7 21:28:27.597916337 (11h48m27.187261324s) [upload=12282]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:28:27.597920134 (11h48m27.187265111s) [upload=12282]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:28:27.597922509 (11h48m27.187267496s) [upload=12282]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:28:27.890743040 (11h48m27.480088057s) [upload=12282]: Done uploading segments... Aug 7 21:28:27.890764190 (11h48m27.480109177s) [upload=12282]: Upload finished. err: <nil> Aug 7 21:29:22.617930722 (11h49m22.207275709s) [upload=12295]: Starting upload Aug 7 21:29:22.617969135 (11h49m22.207314122s) [upload=12295]: Done starting upload Aug 7 21:29:22.618026967 (11h49m22.207371954s) [upload=12295]: Uploading segments... Aug 7 21:29:22.618059649 (11h49m22.207404636s) [upload=12295]: Got next segment. Inline: true Aug 7 21:29:22.618063226 (11h49m22.207408213s) [upload=12295]: Waiting for error group managing segments... Aug 7 21:29:22.618096049 (11h49m22.207441026s) [upload=12295]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:29:22.618100216 (11h49m22.207445193s) [upload=12295]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:29:22.618104334 (11h49m22.207449321s) [upload=12295]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:29:23.935417449 (11h49m23.524762486s) [upload=12295]: Done uploading segments... Aug 7 21:29:23.935440473 (11h49m23.52478548s) [upload=12295]: Upload finished. err: <nil> Aug 7 21:29:59.675674818 (11h49m59.265019805s) [upload=12308]: Starting upload Aug 7 21:29:59.675709946 (11h49m59.265054933s) [upload=12308]: Done starting upload Aug 7 21:29:59.675775310 (11h49m59.265120287s) [upload=12308]: Uploading segments... Aug 7 21:29:59.675809485 (11h49m59.265154462s) [upload=12308]: Got next segment. Inline: true Aug 7 21:29:59.675813022 (11h49m59.265158009s) [upload=12308]: Waiting for error group managing segments... Aug 7 21:29:59.675846747 (11h49m59.265191724s) [upload=12308]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:29:59.675850584 (11h49m59.265195572s) [upload=12308]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:29:59.675852909 (11h49m59.265197896s) [upload=12308]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:29:59.960382218 (11h49m59.549727205s) [upload=12308]: Done uploading segments... Aug 7 21:29:59.960402026 (11h49m59.549747013s) [upload=12308]: Upload finished. err: <nil> Aug 7 21:31:01.204540565 (11h51m0.793885552s) [upload=12324]: Starting upload Aug 7 21:31:01.204590701 (11h51m0.793935688s) [upload=12324]: Done starting upload Aug 7 21:31:01.204646679 (11h51m0.793991666s) [upload=12324]: Uploading segments... Aug 7 21:31:01.204679222 (11h51m0.794024209s) [upload=12324]: Got next segment. Inline: true Aug 7 21:31:01.204683269 (11h51m0.794028256s) [upload=12324]: Waiting for error group managing segments... Aug 7 21:31:01.204721322 (11h51m0.794066309s) [upload=12324]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:31:01.204725289 (11h51m0.794070276s) [upload=12324]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:31:01.204728024 (11h51m0.794073011s) [upload=12324]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:31:01.494645083 (11h51m1.08399007s) [upload=12324]: Done uploading segments... Aug 7 21:31:01.494665120 (11h51m1.084010107s) [upload=12324]: Upload finished. err: <nil> Aug 7 21:31:28.376153869 (11h51m27.965498856s) [upload=12334]: Starting upload Aug 7 21:31:28.376197382 (11h51m27.965542369s) [upload=12334]: Done starting upload Aug 7 21:31:28.376245204 (11h51m27.965590181s) [upload=12334]: Uploading segments... Aug 7 21:31:28.376273257 (11h51m27.965618245s) [upload=12334]: Got next segment. Inline: true Aug 7 21:31:28.376276624 (11h51m27.965621611s) [upload=12334]: Waiting for error group managing segments... Aug 7 21:31:28.376306892 (11h51m27.965651869s) [upload=12334]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:31:28.376312712 (11h51m27.9656577s) [upload=12334]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:31:28.376315419 (11h51m27.965660406s) [upload=12334]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:31:28.645925163 (11h51m28.23527015s) [upload=12334]: Done uploading segments... Aug 7 21:31:28.645942786 (11h51m28.235287773s) [upload=12334]: Upload finished. err: <nil> Aug 7 21:32:07.331024173 (11h52m6.92036916s) [upload=12342]: Starting upload Aug 7 21:32:07.331074259 (11h52m6.920419246s) [upload=12342]: Done starting upload Aug 7 21:32:07.331122251 (11h52m6.920467239s) [upload=12342]: Uploading segments... Aug 7 21:32:07.331154433 (11h52m6.92049942s) [upload=12342]: Got next segment. Inline: true Aug 7 21:32:07.331159583 (11h52m6.92050456s) [upload=12342]: Waiting for error group managing segments... Aug 7 21:32:07.331191624 (11h52m6.920536611s) [upload=12342]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:32:07.331195581 (11h52m6.920540558s) [upload=12342]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:32:07.331197946 (11h52m6.920542933s) [upload=12342]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:32:07.722093791 (11h52m7.311438768s) [upload=12342]: Done uploading segments... Aug 7 21:32:07.722110794 (11h52m7.311455781s) [upload=12342]: Upload finished. err: <nil> Aug 7 21:32:38.771578054 (11h52m38.360923031s) [upload=12353]: Starting upload Aug 7 21:32:38.771650211 (11h52m38.360995188s) [upload=12353]: Done starting upload Aug 7 21:32:38.771738161 (11h52m38.361083148s) [upload=12353]: Uploading segments... Aug 7 21:32:38.771779349 (11h52m38.361124336s) [upload=12353]: Got next segment. Inline: true Aug 7 21:32:38.771782916 (11h52m38.361127893s) [upload=12353]: Waiting for error group managing segments... Aug 7 21:32:38.771815357 (11h52m38.361160334s) [upload=12353]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:32:38.771819266 (11h52m38.361164253s) [upload=12353]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:32:38.771821730 (11h52m38.361166717s) [upload=12353]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:32:39.111523769 (11h52m38.700868796s) [upload=12353]: Done uploading segments... Aug 7 21:32:39.111547013 (11h52m38.700892031s) [upload=12353]: Upload finished. err: <nil> Aug 7 21:33:06.222382527 (11h53m5.811727514s) [upload=12362]: Starting upload Aug 7 21:33:06.222432713 (11h53m5.8117777s) [upload=12362]: Done starting upload Aug 7 21:33:06.222489962 (11h53m5.811834949s) [upload=12362]: Uploading segments... Aug 7 21:33:06.222536221 (11h53m5.811881208s) [upload=12362]: Got next segment. Inline: true Aug 7 21:33:06.222540199 (11h53m5.811885176s) [upload=12362]: Waiting for error group managing segments... Aug 7 21:33:06.222573302 (11h53m5.811918289s) [upload=12362]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:33:06.222576979 (11h53m5.811921966s) [upload=12362]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:33:06.222579153 (11h53m5.81192413s) [upload=12362]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:33:06.561908138 (11h53m6.151253125s) [upload=12362]: Done uploading segments... Aug 7 21:33:06.561986849 (11h53m6.151331836s) [upload=12362]: Upload finished. err: <nil> Duration: 1.826955373s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 31 out of 60 is failed (threshold=3) [.............................XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.000524319s FAIL (2025-08-07 21:38:09 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.000534317s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 30 out of 60 is failed (threshold=3) [............................X..XXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.000825125s FAIL (2025-08-07 21:38:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== se=true , address=45.91.116.11:4118 , piece_id=NQVQKKFGHZ6TJHBIA6Y2CLRBUAEAR76J2UAF7LW5GZYD235CXUPA, seg_pos={0 0}, upload=12325]: Done uploading piece. err: <nil> Aug 7 21:31:12.868805076 (11h51m12.458150063s) [noise=true , address=104.36.82.7:4911 , piece_id=X6PW3FEO52KVJV65NA7AOK25JBLHU5NZKCUFROBWXUBJ67Z3X4SA, seg_pos={0 0}, upload=12325]: Done uploading piece. err: <nil> Aug 7 21:31:12.917061683 (11h51m12.50640667s) [noise=true , address=77.90.31.12:4589 , piece_id=XSFU2EZXJU5AS5ZAAXM2IEAX5OUK5RNWFJAR54RGBSTG3EHIRWWA, seg_pos={0 0}, upload=12325]: Done uploading piece. err: <nil> Aug 7 21:31:12.947560807 (11h51m12.536905794s) [noise=true , address=45.91.116.11:4119 , piece_id=WUA5BEGRS22XCSTO6BLG5WVLDDBSUHIL7YLZB34V2BQYDBHT2IUA, seg_pos={0 0}, upload=12325]: Done uploading piece. err: <nil> Aug 7 21:31:12.949625526 (11h51m12.538970513s) [noise=true , address=77.90.31.17:4772 , piece_id=DQQ7YCYCPEQTUXFLCFQVAKQWLKFNHUJLPEQJYJHDS3UD32GB3SZA, seg_pos={0 0}, upload=12325]: Done uploading piece. err: <nil> Aug 7 21:31:12.964088127 (11h51m12.553433114s) [noise=true , address=45.91.116.10:4081 , piece_id=K7DBSP2FYBF4R2VINZYDMFMOWC6CGFJU4424N4UH4K76GLPYUBGQ, seg_pos={0 0}, upload=12325]: Done uploading piece. err: <nil> Aug 7 21:31:13.098539068 (11h51m12.687884055s) [noise=true , address=77.90.31.14:4662 , piece_id=ZPA6BIPI4AQZYSYPBE5N6RDTV6JKAR3FFFE6RMD3QCGDZF3VCR6Q, seg_pos={0 0}, upload=12325]: Done uploading piece. err: <nil> Aug 7 21:31:13.113930835 (11h51m12.703275822s) [noise=true , address=45.91.116.12:4227 , piece_id=YLSUXR72VFQSBUMMKSJ7V5FUHFJJNI75BV4LNDDAU7GLRXC7QOKQ, seg_pos={0 0}, upload=12325]: Done uploading piece. err: <nil> Aug 7 21:31:13.193532534 (11h51m12.782877521s) [noise=true , address=45.91.116.13:4155 , piece_id=6WQ6W26YKE5G7HNHHSWYCEEJXZWKUQI5VP3ZIPRSW42VKNNEJSFA, seg_pos={0 0}, upload=12325]: Done uploading piece. err: <nil> Aug 7 21:31:13.193555358 (11h51m12.782900345s) [seg_pos={0 0}, upload=12325]: Segment reached optimal threshold of 13 pieces. Aug 7 21:31:13.193652643 (11h51m12.78299763s) [noise=true , address=77.90.31.15:4698 , piece_id=QYUSHWTBKQIQVL4UXFM7G6DGBEQFUD3PCCGR7VPK7ZEL5CPY25TA, seg_pos={0 0}, upload=12325]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj): protocol: expected piece hash; context canceled; EOF Aug 7 21:31:13.193731805 (11h51m12.783076792s) [noise=true , address=104.36.82.8:4969 , piece_id=O7QLRLAKDH4NA4WIVPGVIVSTKAQBHF3FHLKQKUO65NKSL3E5TTFA, seg_pos={0 0}, upload=12325]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX): protocol: context canceled; context canceled Aug 7 21:31:13.193782492 (11h51m12.783127469s) [noise=true , address=77.90.31.11:4555 , piece_id=IIN26RURP6UL2IHLXTLCYQN63SHDS54UOUDCJY6VQN37KOLKOAEQ, seg_pos={0 0}, upload=12325]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ): protocol: expected piece hash; context canceled; EOF Aug 7 21:31:13.193822459 (11h51m12.783167446s) [seg_pos={0 0}, upload=12325]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 7 21:31:13.193896029 (11h51m12.783241016s) [upload=12325]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 21:31:13.193900848 (11h51m12.783245835s) [upload=12325]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:32:57.622790374 (11h52m57.212135361s) [upload=12325]: Done uploading segments... Aug 7 21:32:57.622816474 (11h52m57.212161461s) [upload=12325]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:32:58.493067791 (11h52m58.082412768s) [upload=12358]: Starting upload Aug 7 21:32:58.493143516 (11h52m58.082488503s) [upload=12358]: Done starting upload Aug 7 21:32:58.493260219 (11h52m58.082605216s) [upload=12358]: Uploading segments... Aug 7 21:32:58.493408123 (11h52m58.08275312s) [upload=12358]: Got next segment. Inline: false Aug 7 21:32:58.493486823 (11h52m58.08283187s) [upload=12358]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:32:58.493495350 (11h52m58.082840397s) [upload=12358]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:33:27.834064010 (11h53m27.423408997s) [upload=12358]: Next segment err: context deadline exceeded Aug 7 21:33:27.834177297 (11h53m27.423522354s) [upload=12358]: Done uploading segments... Aug 7 21:33:27.834234777 (11h53m27.423579804s) [upload=12358]: Upload finished. err: context deadline exceeded Aug 7 21:34:06.788498448 (11h54m6.377843435s) [upload=12377]: Starting upload Aug 7 21:34:06.788552902 (11h54m6.377897879s) [upload=12377]: Done starting upload Aug 7 21:34:06.788640219 (11h54m6.377985206s) [upload=12377]: Uploading segments... Aug 7 21:34:06.788741333 (11h54m6.37808632s) [upload=12377]: Got next segment. Inline: false Aug 7 21:34:06.788771611 (11h54m6.378116588s) [upload=12377]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:34:06.788775478 (11h54m6.378120465s) [upload=12377]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:34:36.148498637 (11h54m35.737843624s) [upload=12377]: Next segment err: context deadline exceeded Aug 7 21:34:36.148571016 (11h54m35.737915993s) [upload=12377]: Done uploading segments... Aug 7 21:34:36.148592206 (11h54m35.737937193s) [upload=12377]: Upload finished. err: context deadline exceeded Aug 7 21:34:46.798231056 (11h54m46.387576043s) [upload=12390]: Starting upload Aug 7 21:34:46.798312272 (11h54m46.387657259s) [upload=12390]: Done starting upload Aug 7 21:34:46.798434505 (11h54m46.387779492s) [upload=12390]: Uploading segments... Aug 7 21:34:46.798669936 (11h54m46.388014923s) [upload=12390]: Got next segment. Inline: false Aug 7 21:34:46.798720914 (11h54m46.388065901s) [upload=12390]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:34:46.798728538 (11h54m46.388073525s) [upload=12390]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:35:16.156158424 (11h55m15.745503411s) [upload=12390]: Next segment err: context deadline exceeded Aug 7 21:35:16.156340202 (11h55m15.745685189s) [upload=12390]: Done uploading segments... Aug 7 21:35:16.156354228 (11h55m15.745699215s) [upload=12390]: Upload finished. err: context deadline exceeded Aug 7 21:35:17.073431127 (11h55m16.662776104s) [upload=12398]: Starting upload Aug 7 21:35:17.073512032 (11h55m16.662857019s) [upload=12398]: Done starting upload Aug 7 21:35:17.073649515 (11h55m16.662994502s) [upload=12398]: Uploading segments... Aug 7 21:35:17.073876329 (11h55m16.663221306s) [upload=12398]: Got next segment. Inline: false Aug 7 21:35:17.073946874 (11h55m16.663291861s) [upload=12398]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:35:17.073951052 (11h55m16.663296039s) [upload=12398]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:35:46.421300430 (11h55m46.010645417s) [upload=12398]: Next segment err: context deadline exceeded Aug 7 21:35:46.421379381 (11h55m46.010724368s) [upload=12398]: Done uploading segments... Aug 7 21:35:46.421403588 (11h55m46.010748575s) [upload=12398]: Upload finished. err: context deadline exceeded Aug 7 21:36:14.655988116 (11h56m14.245333103s) [upload=12412]: Starting upload Aug 7 21:36:14.656035766 (11h56m14.245380753s) [upload=12412]: Done starting upload Aug 7 21:36:14.656201223 (11h56m14.24554621s) [upload=12412]: Uploading segments... Aug 7 21:36:14.656239026 (11h56m14.245584013s) [upload=12412]: Got next segment. Inline: false Aug 7 21:36:14.656262921 (11h56m14.245607908s) [upload=12412]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:36:14.656266869 (11h56m14.245611856s) [upload=12412]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:36:44.014215517 (11h56m43.603560504s) [upload=12412]: Next segment err: context deadline exceeded Aug 7 21:36:44.014386335 (11h56m43.603731322s) [upload=12412]: Done uploading segments... Aug 7 21:36:44.014401784 (11h56m43.603746771s) [upload=12412]: Upload finished. err: context deadline exceeded Aug 7 21:36:44.888037208 (11h56m44.477382195s) [upload=12420]: Starting upload Aug 7 21:36:44.888098274 (11h56m44.477443261s) [upload=12420]: Done starting upload Aug 7 21:36:44.888214917 (11h56m44.477559964s) [upload=12420]: Uploading segments... Aug 7 21:36:44.888539819 (11h56m44.477884806s) [upload=12420]: Got next segment. Inline: false Aug 7 21:36:44.888567562 (11h56m44.477912549s) [upload=12420]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:36:44.888571570 (11h56m44.477916557s) [upload=12420]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:37:14.238615406 (11h57m13.827960393s) [upload=12420]: Next segment err: context deadline exceeded Aug 7 21:37:14.238762127 (11h57m13.828107114s) [upload=12420]: Done uploading segments... Aug 7 21:37:14.238799287 (11h57m13.828144274s) [upload=12420]: Upload finished. err: context deadline exceeded Aug 7 21:37:32.234403187 (11h57m31.823748174s) [upload=12433]: Starting upload Aug 7 21:37:32.234480776 (11h57m31.823825763s) [upload=12433]: Done starting upload Aug 7 21:37:32.234611054 (11h57m31.823956041s) [upload=12433]: Uploading segments... Aug 7 21:37:32.234741895 (11h57m31.824086882s) [upload=12433]: Got next segment. Inline: false Aug 7 21:37:32.234780930 (11h57m31.824125917s) [upload=12433]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:37:32.234785328 (11h57m31.824130305s) [upload=12433]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:38:01.579014502 (11h58m1.168359489s) [upload=12433]: Next segment err: context deadline exceeded Aug 7 21:38:01.579278507 (11h58m1.168623494s) [upload=12433]: Done uploading segments... Aug 7 21:38:01.579294227 (11h58m1.168639214s) [upload=12433]: Upload finished. err: context deadline exceeded Duration: 30.000845995s core uplink us-select-1
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway) antiblip: 4 out of 60 is failed (threshold=3) [........................................X....X.X...X........] duration: 729.205041ms remote: "tcp:45.38.24.2:443" target: https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410 FAIL (2025-08-07 21:37:56 UTC) orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": context deadline exceeded Duration: 729.227853ms edge gateway vhost us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 24 out of 60 is failed (threshold=3) [....................................XXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.001360907s FAIL (2025-08-07 21:37:56 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.001371086s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 28 out of 60 is failed (threshold=3) [................................XXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.000254652s FAIL (2025-08-07 21:37:51 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 6819 (11h48m45.131811806s) [upload=12290]: Uploading segments... Aug 7 21:28:45.542586358 (11h48m45.131931345s) [upload=12290]: Got next segment. Inline: true Aug 7 21:28:45.542591848 (11h48m45.131936835s) [upload=12290]: Waiting for error group managing segments... Aug 7 21:28:45.542637005 (11h48m45.131981982s) [upload=12290]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:28:45.542642495 (11h48m45.131987482s) [upload=12290]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:28:45.542645010 (11h48m45.131989987s) [upload=12290]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:29:14.895678169 (11h49m14.485023156s) [upload=12290]: Done uploading segments... Aug 7 21:29:14.895732473 (11h49m14.48507746s) [upload=12290]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:29:24.822220224 (11h49m24.411565212s) [upload=12297]: Starting upload Aug 7 21:29:24.822268116 (11h49m24.411613093s) [upload=12297]: Done starting upload Aug 7 21:29:24.822310567 (11h49m24.411655554s) [upload=12297]: Uploading segments... Aug 7 21:29:24.822336567 (11h49m24.411681544s) [upload=12297]: Got next segment. Inline: true Aug 7 21:29:24.822340274 (11h49m24.411685261s) [upload=12297]: Waiting for error group managing segments... Aug 7 21:29:24.822369891 (11h49m24.411714878s) [upload=12297]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:29:24.822374179 (11h49m24.411719166s) [upload=12297]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:29:24.822376544 (11h49m24.411721531s) [upload=12297]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:29:54.171228417 (11h49m53.760573394s) [upload=12297]: Done uploading segments... Aug 7 21:29:54.171248837 (11h49m53.760593824s) [upload=12297]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:30:04.137944213 (11h50m3.72728921s) [upload=12310]: Starting upload Aug 7 21:30:04.138024878 (11h50m3.727369875s) [upload=12310]: Done starting upload Aug 7 21:30:04.138088940 (11h50m3.727433937s) [upload=12310]: Uploading segments... Aug 7 21:30:04.138138645 (11h50m3.727483642s) [upload=12310]: Got next segment. Inline: true Aug 7 21:30:04.138144005 (11h50m3.727488992s) [upload=12310]: Waiting for error group managing segments... Aug 7 21:30:04.138191466 (11h50m3.727536453s) [upload=12310]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:30:04.138197808 (11h50m3.727542795s) [upload=12310]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:30:04.138201486 (11h50m3.727546483s) [upload=12310]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:30:33.459773387 (11h50m33.049118384s) [upload=12310]: Done uploading segments... Aug 7 21:30:33.459805027 (11h50m33.049150014s) [upload=12310]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:31:10.105493540 (11h51m9.694838527s) [upload=12327]: Starting upload Aug 7 21:31:10.105548566 (11h51m9.694893543s) [upload=12327]: Done starting upload Aug 7 21:31:10.105626315 (11h51m9.694971302s) [upload=12327]: Uploading segments... Aug 7 21:31:10.105718581 (11h51m9.695063568s) [upload=12327]: Got next segment. Inline: true Aug 7 21:31:10.105723311 (11h51m9.695068298s) [upload=12327]: Waiting for error group managing segments... Aug 7 21:31:10.105777513 (11h51m9.69512249s) [upload=12327]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:31:10.105782744 (11h51m9.695127731s) [upload=12327]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:31:10.105785079 (11h51m9.695130066s) [upload=12327]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:31:39.460569760 (11h51m39.049914747s) [upload=12327]: Done uploading segments... Aug 7 21:31:39.460597573 (11h51m39.04994255s) [upload=12327]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:32:24.324148762 (11h52m23.913493749s) [upload=12347]: Starting upload Aug 7 21:32:24.324206403 (11h52m23.91355139s) [upload=12347]: Done starting upload Aug 7 21:32:24.324252100 (11h52m23.913597087s) [upload=12347]: Uploading segments... Aug 7 21:32:24.324280415 (11h52m23.913625402s) [upload=12347]: Got next segment. Inline: true Aug 7 21:32:24.324284021 (11h52m23.913628998s) [upload=12347]: Waiting for error group managing segments... Aug 7 21:32:24.324329689 (11h52m23.913674686s) [upload=12347]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:32:24.324336372 (11h52m23.913681379s) [upload=12347]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:32:24.324341050 (11h52m23.913686057s) [upload=12347]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:32:53.678084170 (11h52m53.267429207s) [upload=12347]: Done uploading segments... Aug 7 21:32:53.678107405 (11h52m53.267452392s) [upload=12347]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:33:17.618427342 (11h53m17.207772389s) [upload=12364]: Starting upload Aug 7 21:33:17.618576718 (11h53m17.207921745s) [upload=12364]: Done starting upload Aug 7 21:33:17.618658725 (11h53m17.208003742s) [upload=12364]: Uploading segments... Aug 7 21:33:17.618713580 (11h53m17.208058597s) [upload=12364]: Got next segment. Inline: true Aug 7 21:33:17.618720533 (11h53m17.20806553s) [upload=12364]: Waiting for error group managing segments... Aug 7 21:33:17.618772433 (11h53m17.20811743s) [upload=12364]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:33:17.618779636 (11h53m17.208124643s) [upload=12364]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:33:17.618784776 (11h53m17.208129793s) [upload=12364]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:33:46.957323530 (11h53m46.546668577s) [upload=12364]: Done uploading segments... Aug 7 21:33:46.957345773 (11h53m46.54669075s) [upload=12364]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:34:19.197336641 (11h54m18.786681628s) [upload=12382]: Starting upload Aug 7 21:34:19.197461931 (11h54m18.786806918s) [upload=12382]: Done starting upload Aug 7 21:34:19.197512228 (11h54m18.786857225s) [upload=12382]: Uploading segments... Aug 7 21:34:19.197561282 (11h54m18.786906269s) [upload=12382]: Got next segment. Inline: true Aug 7 21:34:19.197565189 (11h54m18.786910166s) [upload=12382]: Waiting for error group managing segments... Aug 7 21:34:19.197610725 (11h54m18.786955723s) [upload=12382]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:34:19.197615966 (11h54m18.786960953s) [upload=12382]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:34:19.197619142 (11h54m18.786964139s) [upload=12382]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:34:48.535954936 (11h54m48.125299923s) [upload=12382]: Done uploading segments... Aug 7 21:34:48.535978650 (11h54m48.125323637s) [upload=12382]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:35:05.991757765 (11h55m5.581102752s) [upload=12393]: Starting upload Aug 7 21:35:05.991822780 (11h55m5.581167767s) [upload=12393]: Done starting upload Aug 7 21:35:05.991946456 (11h55m5.581291443s) [upload=12393]: Uploading segments... Aug 7 21:35:05.991988116 (11h55m5.581333104s) [upload=12393]: Got next segment. Inline: true Aug 7 21:35:05.991994929 (11h55m5.581339996s) [upload=12393]: Waiting for error group managing segments... Aug 7 21:35:05.992042741 (11h55m5.581387728s) [upload=12393]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:35:05.992048021 (11h55m5.581393018s) [upload=12393]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:35:05.992051768 (11h55m5.581396755s) [upload=12393]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:35:35.339368736 (11h55m34.928713723s) [upload=12393]: Done uploading segments... Aug 7 21:35:35.339503534 (11h55m34.928848522s) [upload=12393]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:36:12.952326182 (11h56m12.541671159s) [upload=12411]: Starting upload Aug 7 21:36:12.952385075 (11h56m12.541730052s) [upload=12411]: Done starting upload Aug 7 21:36:12.952428238 (11h56m12.541773225s) [upload=12411]: Uploading segments... Aug 7 21:36:12.952455510 (11h56m12.541800497s) [upload=12411]: Got next segment. Inline: true Aug 7 21:36:12.952459417 (11h56m12.541804404s) [upload=12411]: Waiting for error group managing segments... Aug 7 21:36:12.952507650 (11h56m12.541852637s) [upload=12411]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:36:12.952514352 (11h56m12.541859349s) [upload=12411]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:36:12.952518550 (11h56m12.541863537s) [upload=12411]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:36:42.299158316 (11h56m41.888503304s) [upload=12411]: Done uploading segments... Aug 7 21:36:42.299175039 (11h56m41.888520026s) [upload=12411]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:37:22.572699237 (11h57m22.162044224s) [upload=12432]: Starting upload Aug 7 21:37:22.572782937 (11h57m22.162127924s) [upload=12432]: Done starting upload Aug 7 21:37:22.572842260 (11h57m22.162187247s) [upload=12432]: Uploading segments... Aug 7 21:37:22.572885313 (11h57m22.1622303s) [upload=12432]: Got next segment. Inline: true Aug 7 21:37:22.572889591 (11h57m22.162234578s) [upload=12432]: Waiting for error group managing segments... Aug 7 21:37:22.572930138 (11h57m22.162275125s) [upload=12432]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:37:22.572934296 (11h57m22.162279283s) [upload=12432]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:37:22.572937121 (11h57m22.162282118s) [upload=12432]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:37:51.933954276 (11h57m51.523299253s) [upload=12432]: Done uploading segments... Aug 7 21:37:51.933970927 (11h57m51.523315914s) [upload=12432]: Upload finished. err: metaclient: context deadline exceeded Duration: 30.000299737s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 27 out of 60 is failed (threshold=3) [...............................X..XXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.000384219s FAIL (2025-08-07 21:37:51 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 30.000397093s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 6 out of 60 is failed (threshold=3) [.X.......................X.........X......X..X.............X] duration: 6.089453947s FAIL (2025-08-07 21:37:46 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/4M):: after 3c7be29e56ef0a348f54a62643273e5baec016fc19e64e2fa674114dac8dddcf != before 2decbc1b39ad0a3120f73e14d303cc93862fc082b00150c3e706fb25a679be76 ======LOGS====== d=12436]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:37:41.241548466 (11h57m40.830893473s) [upload=12436]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:37:41.699604637 (11h57m41.288949624s) [seg_pos={0 0}, upload=12436]: Begin upload segment... Aug 7 21:37:41.699642941 (11h57m41.288987928s) [seg_pos={0 0}, upload=12436]: Done begin upload segment. Aug 7 21:37:41.699665653 (11h57m41.28901063s) [noise=true , address=64.49.13.137:4494 , piece_id=27I45T5P6HE5MB7KLKHIDYF3GKGB4X5EYUQEQCZAIPO56TWSUIXQ, seg_pos={0 0}, upload=12436]: Uploading piece... Aug 7 21:37:41.699677036 (11h57m41.289022023s) [noise=true , address=104.36.82.37:4671 , piece_id=GAEIZGDLAYYQ2ATH4PCFWG7ZDNDVLY3RWQNDKPZNVPBYZ4EPCCBQ, seg_pos={0 0}, upload=12436]: Uploading piece... Aug 7 21:37:41.700019711 (11h57m41.289364698s) [noise=true , address=104.36.82.58:4271 , piece_id=VBXDX65YCLKD7WWXXCKLDKNOCUBMA4PTK53IMIJKCM5M5AOQRRYQ, seg_pos={0 0}, upload=12436]: Uploading piece... Aug 7 21:37:41.700313864 (11h57m41.289658851s) [noise=true , address=104.36.82.55:4193 , piece_id=YO4COWLA4546LGDVL7E3X7CGY2TLKVGEMVBI6A3D7LCI226ZQGMA, seg_pos={0 0}, upload=12436]: Uploading piece... Aug 7 21:37:41.700321118 (11h57m41.289666105s) [noise=true , address=104.36.82.8:4975 , piece_id=MPDCSPMDGY4MEJHSXYOTT3VFFAZGRABQH7QR5VLSD2OWB25C2DBQ, seg_pos={0 0}, upload=12436]: Uploading piece... Aug 7 21:37:41.700555927 (11h57m41.289900914s) [noise=true , address=104.36.82.37:4683 , piece_id=ZYQOB47RWRIT4GC36XROSHCGKBKTYNE5PRDTECJJ4RIEWRXLNZ2Q, seg_pos={0 0}, upload=12436]: Uploading piece... Aug 7 21:37:41.700769827 (11h57m41.290114814s) [noise=true , address=79.127.201.215:4356 , piece_id=VA6MG6UXZ7VUH5LAMLMBPIK7C2GLTDXA33LCFDIZMBPXDDPLVXIA, seg_pos={0 0}, upload=12436]: Uploading piece... Aug 7 21:37:41.700792900 (11h57m41.290137887s) [noise=true , address=104.36.82.23:4447 , piece_id=ZPOANDX46722LIXWQSDNXMFDB4ZHSS765MZQ3NOF4JXPUNZ4TFPQ, seg_pos={0 0}, upload=12436]: Uploading piece... Aug 7 21:37:41.701033912 (11h57m41.290378889s) [noise=true , address=104.36.82.43:4861 , piece_id=AXOCVGDO4GD7PWM43RYNWUPGX2H7BK4X4QCWTAJ4C3CNFL7UICRA, seg_pos={0 0}, upload=12436]: Uploading piece... Aug 7 21:37:41.701082625 (11h57m41.290427642s) [noise=true , address=104.36.82.59:4314 , piece_id=RF5LCHJAO7EOIT4AVOUVKKYVVBST4F3S5BO63CPYVJBI4CAWLGJA, seg_pos={0 0}, upload=12436]: Uploading piece... Aug 7 21:37:41.701257619 (11h57m41.290602606s) [noise=true , address=104.36.82.60:4358 , piece_id=MQTJBJP572O6M7CC756QSNAU4LUWO67OD7FJDNHIXOVANN7GDPQQ, seg_pos={0 0}, upload=12436]: Uploading piece... Aug 7 21:37:41.701341520 (11h57m41.290686507s) [noise=true , address=104.36.82.58:4285 , piece_id=E7G4SBPGKX3OAR3J2Y2PG67ZWXLKAJWJ5L34KVWLDX44IAQHZFNA, seg_pos={0 0}, upload=12436]: Uploading piece... Aug 7 21:37:41.701453143 (11h57m41.29079813s) [noise=true , address=104.36.82.14:4242 , piece_id=BOU64PUQXF4JOVE7GEVOYSXQKQPRAI75BRRHO577Q3BVXES5C7EQ, seg_pos={0 0}, upload=12436]: Uploading piece... Aug 7 21:37:41.701589534 (11h57m41.290934521s) [noise=true , address=45.91.116.11:4150 , piece_id=CUAREN64B6BVTXHF3RIJPH6LRPR3QEPVHRKLZDVXVMZFWBEXVBSA, seg_pos={0 0}, upload=12436]: Uploading piece... Aug 7 21:37:41.701638548 (11h57m41.290983535s) [noise=true , address=104.36.82.4:4830 , piece_id=MKXIFYHW4KNTLS4AIHCEI53CE7STNGL5ITSZA7IKWF36QMF2MSMQ, seg_pos={0 0}, upload=12436]: Uploading piece... Aug 7 21:37:41.701846857 (11h57m41.291191844s) [noise=true , address=104.36.82.46:4508 , piece_id=LVE3T3E6XA5TR3W4XANXE44O54BMSBSAHLBRS2XROOEAXVF2K6DA, seg_pos={0 0}, upload=12436]: Uploading piece... Aug 7 21:37:41.701850394 (11h57m41.291195381s) [noise=true , address=45.91.116.14:4205 , piece_id=PSGW5VTZ3JNFGHFGEO2IWQN4A6DUYEFOYGLOCJHCX625AZGSJHZA, seg_pos={0 0}, upload=12436]: Uploading piece... Aug 7 21:37:41.702036911 (11h57m41.291381898s) [noise=true , address=45.91.116.7:4040 , piece_id=ILATCWOLO5UO5RPRF4IRL7Y4JNMYLUIBFNLZBUDUAJKQDVLTXCCQ, seg_pos={0 0}, upload=12436]: Uploading piece... Aug 7 21:37:42.901972883 (11h57m42.49131787s) [upload=12436]: Next returned nil segment Aug 7 21:37:42.901987461 (11h57m42.491332448s) [upload=12436]: Waiting for error group managing segments... Aug 7 21:37:43.221358460 (11h57m42.810703437s) [noise=true , address=64.49.13.137:4494 , piece_id=27I45T5P6HE5MB7KLKHIDYF3GKGB4X5EYUQEQCZAIPO56TWSUIXQ, seg_pos={0 0}, upload=12436]: Done uploading piece. err: <nil> Aug 7 21:37:43.635176843 (11h57m43.22452183s) [noise=true , address=45.91.116.7:4040 , piece_id=ILATCWOLO5UO5RPRF4IRL7Y4JNMYLUIBFNLZBUDUAJKQDVLTXCCQ, seg_pos={0 0}, upload=12436]: Done uploading piece. err: <nil> Aug 7 21:37:43.691989821 (11h57m43.281334828s) [noise=true , address=79.127.201.215:4356 , piece_id=VA6MG6UXZ7VUH5LAMLMBPIK7C2GLTDXA33LCFDIZMBPXDDPLVXIA, seg_pos={0 0}, upload=12436]: Done uploading piece. err: <nil> Aug 7 21:37:43.715706768 (11h57m43.305051755s) [noise=true , address=104.36.82.8:4975 , piece_id=MPDCSPMDGY4MEJHSXYOTT3VFFAZGRABQH7QR5VLSD2OWB25C2DBQ, seg_pos={0 0}, upload=12436]: Done uploading piece. err: <nil> Aug 7 21:37:43.732997881 (11h57m43.322342868s) [noise=true , address=104.36.82.14:4242 , piece_id=BOU64PUQXF4JOVE7GEVOYSXQKQPRAI75BRRHO577Q3BVXES5C7EQ, seg_pos={0 0}, upload=12436]: Done uploading piece. err: <nil> Aug 7 21:37:43.741488406 (11h57m43.330833383s) [noise=true , address=104.36.82.46:4508 , piece_id=LVE3T3E6XA5TR3W4XANXE44O54BMSBSAHLBRS2XROOEAXVF2K6DA, seg_pos={0 0}, upload=12436]: Done uploading piece. err: <nil> Aug 7 21:37:43.879657469 (11h57m43.469002456s) [noise=true , address=104.36.82.37:4671 , piece_id=GAEIZGDLAYYQ2ATH4PCFWG7ZDNDVLY3RWQNDKPZNVPBYZ4EPCCBQ, seg_pos={0 0}, upload=12436]: Done uploading piece. err: <nil> Aug 7 21:37:43.895681587 (11h57m43.485026574s) [noise=true , address=45.91.116.11:4150 , piece_id=CUAREN64B6BVTXHF3RIJPH6LRPR3QEPVHRKLZDVXVMZFWBEXVBSA, seg_pos={0 0}, upload=12436]: Done uploading piece. err: <nil> Aug 7 21:37:43.907596774 (11h57m43.496941751s) [noise=true , address=104.36.82.4:4830 , piece_id=MKXIFYHW4KNTLS4AIHCEI53CE7STNGL5ITSZA7IKWF36QMF2MSMQ, seg_pos={0 0}, upload=12436]: Done uploading piece. err: <nil> Aug 7 21:37:43.908572762 (11h57m43.497917749s) [noise=true , address=45.91.116.14:4205 , piece_id=PSGW5VTZ3JNFGHFGEO2IWQN4A6DUYEFOYGLOCJHCX625AZGSJHZA, seg_pos={0 0}, upload=12436]: Done uploading piece. err: <nil> Aug 7 21:37:43.912238615 (11h57m43.501583602s) [noise=true , address=104.36.82.60:4358 , piece_id=MQTJBJP572O6M7CC756QSNAU4LUWO67OD7FJDNHIXOVANN7GDPQQ, seg_pos={0 0}, upload=12436]: Done uploading piece. err: <nil> Aug 7 21:37:43.917180210 (11h57m43.506525197s) [noise=true , address=104.36.82.37:4683 , piece_id=ZYQOB47RWRIT4GC36XROSHCGKBKTYNE5PRDTECJJ4RIEWRXLNZ2Q, seg_pos={0 0}, upload=12436]: Done uploading piece. err: <nil> Aug 7 21:37:43.957535420 (11h57m43.546880407s) [noise=true , address=104.36.82.59:4314 , piece_id=RF5LCHJAO7EOIT4AVOUVKKYVVBST4F3S5BO63CPYVJBI4CAWLGJA, seg_pos={0 0}, upload=12436]: Done uploading piece. err: <nil> Aug 7 21:37:44.104572309 (11h57m43.693917296s) [noise=true , address=104.36.82.43:4861 , piece_id=AXOCVGDO4GD7PWM43RYNWUPGX2H7BK4X4QCWTAJ4C3CNFL7UICRA, seg_pos={0 0}, upload=12436]: Done uploading piece. err: <nil> Aug 7 21:37:44.104599301 (11h57m43.693944278s) [seg_pos={0 0}, upload=12436]: Segment reached optimal threshold of 14 pieces. Aug 7 21:37:44.104687199 (11h57m43.694032176s) [noise=true , address=104.36.82.23:4447 , piece_id=ZPOANDX46722LIXWQSDNXMFDB4ZHSS765MZQ3NOF4JXPUNZ4TFPQ, seg_pos={0 0}, upload=12436]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fqT9xA7kXTG8K9XwTUxL6FXibV35AkPHDK6grKGHg17fk91fP): protocol: expected piece hash; context canceled; EOF Aug 7 21:37:44.104922059 (11h57m43.694267046s) [noise=true , address=104.36.82.58:4271 , piece_id=VBXDX65YCLKD7WWXXCKLDKNOCUBMA4PTK53IMIJKCM5M5AOQRRYQ, seg_pos={0 0}, upload=12436]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QwLHnMQHafFPYL83gE6wffvzdkvCHwzzVMLBz4KBVdE7ynuf1): protocol: rpc: dial tcp 104.36.82.58:4271: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 21:37:44.104969008 (11h57m43.694313995s) [noise=true , address=104.36.82.58:4285 , piece_id=E7G4SBPGKX3OAR3J2Y2PG67ZWXLKAJWJ5L34KVWLDX44IAQHZFNA, seg_pos={0 0}, upload=12436]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rZhbymrDS6ahXLEqYRM64gcqRvG8CEnc8iGCTEhgXrpvRaQrp): protocol: write tcp 188.166.225.170:56894->104.36.82.58:4285: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 21:37:44.104982013 (11h57m43.69432699s) [noise=true , address=104.36.82.55:4193 , piece_id=YO4COWLA4546LGDVL7E3X7CGY2TLKVGEMVBI6A3D7LCI226ZQGMA, seg_pos={0 0}, upload=12436]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: write tcp 188.166.225.170:60782->104.36.82.55:4193: use of closed network connection; write tcp 188.166.225.170:60782->104.36.82.55:4193: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60782->104.36.82.55:4193: use of closed network connection Aug 7 21:37:44.104993535 (11h57m43.694338522s) [seg_pos={0 0}, upload=12436]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 21:37:44.105075582 (11h57m43.694420569s) [upload=12436]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 21:37:44.105079359 (11h57m43.694424336s) [upload=12436]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:37:44.447186820 (11h57m44.036531807s) [upload=12436]: Done uploading segments... Aug 7 21:37:44.447217970 (11h57m44.036562957s) [upload=12436]: Upload finished. err: <nil> Duration: 6.089477122s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 31 out of 60 is failed (threshold=3) [.............................XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.001170343s FAIL (2025-08-07 21:37:46 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.00119045s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 22 out of 60 is failed (threshold=3) [......................................XXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.000874869s FAIL (2025-08-07 21:37:40 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== egments... Aug 7 21:31:25.733416063 (11h51m25.32276105s) [noise=true , address=79.127.205.233:4649 , piece_id=BFYTZVIKYNRUABFKYKCCSLWK5IABKDJJD6PNE75DNPUTWGPONVWA, seg_pos={0 0}, upload=12321]: Done uploading piece. err: <nil> Aug 7 21:31:26.261970307 (11h51m25.851315294s) [noise=true , address=104.36.82.46:4511 , piece_id=Y2HD26MH7KJI6D42TY7TSIG3GF24SOQFOZEZ5OMNFWDFWZ7QSWUQ, seg_pos={0 0}, upload=12321]: Done uploading piece. err: <nil> Aug 7 21:31:26.356060466 (11h51m25.945405443s) [noise=true , address=104.36.82.39:4745 , piece_id=D6TL4I4RWUMYICLCPGL755KSTAELDAW47IW5DN4TM2HANVGFDUSQ, seg_pos={0 0}, upload=12321]: Done uploading piece. err: <nil> Aug 7 21:31:26.390515658 (11h51m25.979860645s) [noise=true , address=104.36.82.62:4414 , piece_id=Z2UOXNG7PVOGVW4NH3N55TW7WMPCABT5YT6FBNXVBSISV3J3WJ3Q, seg_pos={0 0}, upload=12321]: Done uploading piece. err: <nil> Aug 7 21:31:26.403163305 (11h51m25.992508292s) [noise=true , address=104.36.82.9:4036 , piece_id=473DGK6MQPSKHKLKCB3KMPELO5WHBGV7KTHEXKNPAKTUWKJRXABA, seg_pos={0 0}, upload=12321]: Done uploading piece. err: <nil> Aug 7 21:31:26.410169400 (11h51m25.999514377s) [noise=true , address=104.36.82.22:4524 , piece_id=UJUNX53EZDOT7NMYKILPPTRBJNOONOIGYULH5IOHQ73W5PMD7XJA, seg_pos={0 0}, upload=12321]: Done uploading piece. err: <nil> Aug 7 21:31:26.411736859 (11h51m26.001081846s) [noise=true , address=104.36.82.4:4825 , piece_id=TAAMMJMZEEO7UYZM6VVZUPSX53EGSOJTJ3VJY5HMEDUGLRRND3YA, seg_pos={0 0}, upload=12321]: Done uploading piece. err: <nil> Aug 7 21:31:26.420123765 (11h51m26.009468752s) [noise=true , address=104.36.82.56:4249 , piece_id=M5FBC6LH2A2ZCGSJJI6KUVYDO6UELARWIHMD22XUP6GQFVP2ZFXA, seg_pos={0 0}, upload=12321]: Done uploading piece. err: <nil> Aug 7 21:31:26.428567841 (11h51m26.017912818s) [noise=true , address=104.36.82.38:4714 , piece_id=7AYVJ5FRQUIZRQ2NTCUN43CWJWWNQZEHMD7UE3MCYXSRBFQOL56A, seg_pos={0 0}, upload=12321]: Done uploading piece. err: <nil> Aug 7 21:31:26.430706232 (11h51m26.020051219s) [noise=true , address=104.36.82.59:4313 , piece_id=BLGX433PKGAVB5WCNKXMA6P3EYPPWCPCB6WYHQ5RZ4I6WRUH4GHQ, seg_pos={0 0}, upload=12321]: Done uploading piece. err: <nil> Aug 7 21:31:26.612329184 (11h51m26.201674171s) [noise=true , address=104.36.82.55:4222 , piece_id=4W52URT2PUTVOIVFFYOFSE3RDOVBHMS56PG57MEEKCFU7QP3CZGQ, seg_pos={0 0}, upload=12321]: Done uploading piece. err: <nil> Aug 7 21:31:26.658377737 (11h51m26.247722724s) [noise=true , address=104.36.82.54:4162 , piece_id=6RDDMFE4ZG2YJG5GKRFLSANPJXNKADJXLY6H64A53RPJPSXEC4BA, seg_pos={0 0}, upload=12321]: Done uploading piece. err: <nil> Aug 7 21:31:26.674652975 (11h51m26.263997962s) [noise=true , address=104.36.82.45:4937 , piece_id=EBUAJKXBK6RZK7T46KXJT5P5LINBFVZCEPGVGG5OZ66SYNOA2RSQ, seg_pos={0 0}, upload=12321]: Done uploading piece. err: <nil> Aug 7 21:31:26.680556480 (11h51m26.269901467s) [noise=true , address=45.91.116.10:4110 , piece_id=F4ZMNLCCFQCBFB6FOPOJFTGPIL5BFYT6TPGL5C7SVLIHMTY4EJJA, seg_pos={0 0}, upload=12321]: Done uploading piece. err: <nil> Aug 7 21:31:26.680580406 (11h51m26.269925393s) [seg_pos={0 0}, upload=12321]: Segment reached optimal threshold of 14 pieces. Aug 7 21:31:26.680752626 (11h51m26.270097613s) [noise=true , address=45.91.116.6:4983 , piece_id=ZWH64KJWCVSTEGVN25WW35HRHTAO3BE3SPKKALNRT7NNLUKB4L4Q, seg_pos={0 0}, upload=12321]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122trJbDHuqeBMQHZT97HP2PgkeuLyUiDiuRm4nayu6ZTH7tat6): protocol: expected piece hash; context canceled; EOF Aug 7 21:31:26.680804515 (11h51m26.270149492s) [noise=true , address=45.91.116.7:4037 , piece_id=WY2F64PFXEGFVE4M6PUHHUYIMTI2KB4CB73DQADZK4QGIEZ2YCYQ, seg_pos={0 0}, upload=12321]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iNPwE2XSfXcaXAXcrH2jcyNGYeDiiWvzAcDQffrmjPwTVqVmK): protocol: expected piece hash; context canceled; EOF Aug 7 21:31:26.680954802 (11h51m26.270299789s) [noise=true , address=104.36.82.61:4375 , piece_id=5LQOQ3MR3EA36DIYLOVO6YDKF6EP2HPHMPG4P6MN76A4IA5BTBEQ, seg_pos={0 0}, upload=12321]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121fzX9v4rgEd78xnDAwcQtthyQHV3yUZUQYLXZ9byPDucGK4T5): protocol: write tcp 188.166.225.170:45604->104.36.82.61:4375: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 21:31:26.680994198 (11h51m26.270339185s) [noise=true , address=104.36.82.8:4977 , piece_id=VY47HC4FF6JU4QQTDVZ62NOIEGWSROTWYMGCB7XXNHJ3GZNJATMQ, seg_pos={0 0}, upload=12321]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WCAm12tXwimFSPqz7Seh6A4oxWqQVxqsF459xux4a5CqjpbAT): protocol: write tcp 188.166.225.170:36156->104.36.82.8:4977: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 21:31:26.681016300 (11h51m26.270361287s) [seg_pos={0 0}, upload=12321]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 21:31:26.681067097 (11h51m26.270412074s) [upload=12321]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 21:31:26.681072137 (11h51m26.270417114s) [upload=12321]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:32:39.816785277 (11h52m39.406130264s) [upload=12321]: Done uploading segments... Aug 7 21:32:39.816815415 (11h52m39.406160402s) [upload=12321]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:32:40.684664778 (11h52m40.274009755s) [upload=12355]: Starting upload Aug 7 21:32:40.684721386 (11h52m40.274066373s) [upload=12355]: Done starting upload Aug 7 21:32:40.684776000 (11h52m40.274121017s) [upload=12355]: Uploading segments... Aug 7 21:32:40.684893314 (11h52m40.274238292s) [upload=12355]: Got next segment. Inline: false Aug 7 21:32:40.684916699 (11h52m40.274261686s) [upload=12355]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:32:40.684920466 (11h52m40.274265443s) [upload=12355]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:33:40.036921304 (11h53m39.626266281s) [upload=12355]: Next segment err: context deadline exceeded Aug 7 21:33:40.037128099 (11h53m39.626473086s) [upload=12355]: Done uploading segments... Aug 7 21:33:40.037148919 (11h53m39.626493896s) [upload=12355]: Upload finished. err: context deadline exceeded Aug 7 21:33:40.899438535 (11h53m40.488783522s) [upload=12372]: Starting upload Aug 7 21:33:40.899490825 (11h53m40.488835832s) [upload=12372]: Done starting upload Aug 7 21:33:40.899590144 (11h53m40.488935131s) [upload=12372]: Uploading segments... Aug 7 21:33:40.899771832 (11h53m40.489116819s) [upload=12372]: Got next segment. Inline: false Aug 7 21:33:40.899795227 (11h53m40.489140214s) [upload=12372]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:33:40.899799275 (11h53m40.489144252s) [upload=12372]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:34:40.257761531 (11h54m39.847106518s) [upload=12372]: Next segment err: context deadline exceeded Aug 7 21:34:40.257885859 (11h54m39.847230856s) [upload=12372]: Done uploading segments... Aug 7 21:34:40.257904695 (11h54m39.847249672s) [upload=12372]: Upload finished. err: context deadline exceeded Aug 7 21:34:41.144124896 (11h54m40.733469873s) [upload=12386]: Starting upload Aug 7 21:34:41.144233103 (11h54m40.73357813s) [upload=12386]: Done starting upload Aug 7 21:34:41.144343113 (11h54m40.733688101s) [upload=12386]: Uploading segments... Aug 7 21:34:41.144700337 (11h54m40.734045364s) [upload=12386]: Got next segment. Inline: false Aug 7 21:34:41.144760612 (11h54m40.734105639s) [upload=12386]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:34:41.144768206 (11h54m40.734113223s) [upload=12386]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:35:40.491924814 (11h55m40.081269811s) [upload=12386]: Next segment err: context deadline exceeded Aug 7 21:35:40.492048891 (11h55m40.081393878s) [upload=12386]: Done uploading segments... Aug 7 21:35:40.492066364 (11h55m40.081411351s) [upload=12386]: Upload finished. err: context deadline exceeded Aug 7 21:35:41.353263670 (11h55m40.942608657s) [upload=12405]: Starting upload Aug 7 21:35:41.353369352 (11h55m40.942714339s) [upload=12405]: Done starting upload Aug 7 21:35:41.353452601 (11h55m40.942797588s) [upload=12405]: Uploading segments... Aug 7 21:35:41.353579473 (11h55m40.942924511s) [upload=12405]: Got next segment. Inline: false Aug 7 21:35:41.353621805 (11h55m40.942966792s) [upload=12405]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:35:41.353626053 (11h55m40.94297104s) [upload=12405]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:36:40.713868241 (11h56m40.303213228s) [upload=12405]: Next segment err: context deadline exceeded Aug 7 21:36:40.713958674 (11h56m40.303303651s) [upload=12405]: Done uploading segments... Aug 7 21:36:40.713972460 (11h56m40.303317447s) [upload=12405]: Upload finished. err: context deadline exceeded Aug 7 21:36:41.596698707 (11h56m41.186043744s) [upload=12418]: Starting upload Aug 7 21:36:41.596808026 (11h56m41.186153063s) [upload=12418]: Done starting upload Aug 7 21:36:41.596885475 (11h56m41.186230462s) [upload=12418]: Uploading segments... Aug 7 21:36:41.597098722 (11h56m41.186443709s) [upload=12418]: Got next segment. Inline: false Aug 7 21:36:41.597141925 (11h56m41.186486912s) [upload=12418]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:36:41.597146604 (11h56m41.186491591s) [upload=12418]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:37:40.947744552 (11h57m40.537089539s) [upload=12418]: Next segment err: context deadline exceeded Aug 7 21:37:40.947825697 (11h57m40.537170684s) [upload=12418]: Done uploading segments... Aug 7 21:37:40.947843592 (11h57m40.537188579s) [upload=12418]: Upload finished. err: context deadline exceeded Duration: 1m0.000886862s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 30 out of 60 is failed (threshold=3) [..............................XXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.000677478s FAIL (2025-08-07 21:37:39 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.000687989s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 22 out of 60 is failed (threshold=3) [......................................XXXXXXXXXXXXXXXXXXXXXX] duration: 30.001088507s FAIL (2025-08-07 21:37:36 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 30.001110348s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 3 out of 60 is failed (threshold=3) [.X...................................X...............X......] duration: 1.891694507s FAIL (2025-08-07 21:37:35 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/1K") ======LOGS====== 083457s) [upload=12258]: Done uploading segments... Aug 7 21:26:43.142756605 (11h46m42.732101593s) [upload=12258]: Upload finished. err: <nil> Aug 7 21:27:16.645746716 (11h47m16.235091703s) [upload=12264]: Starting upload Aug 7 21:27:16.645808424 (11h47m16.235153411s) [upload=12264]: Done starting upload Aug 7 21:27:16.645853019 (11h47m16.235198006s) [upload=12264]: Uploading segments... Aug 7 21:27:16.645881223 (11h47m16.23522621s) [upload=12264]: Got next segment. Inline: true Aug 7 21:27:16.645884680 (11h47m16.235229667s) [upload=12264]: Waiting for error group managing segments... Aug 7 21:27:16.645915299 (11h47m16.235260286s) [upload=12264]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:27:16.645919216 (11h47m16.235264203s) [upload=12264]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:27:16.645921510 (11h47m16.235266498s) [upload=12264]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:27:16.954194817 (11h47m16.543539804s) [upload=12264]: Done uploading segments... Aug 7 21:27:16.954212471 (11h47m16.543557459s) [upload=12264]: Upload finished. err: <nil> Aug 7 21:27:46.945635033 (11h47m46.53498005s) [upload=12274]: Starting upload Aug 7 21:27:46.945720126 (11h47m46.535065163s) [upload=12274]: Done starting upload Aug 7 21:27:46.945850435 (11h47m46.535195422s) [upload=12274]: Uploading segments... Aug 7 21:27:46.945918004 (11h47m46.535262991s) [upload=12274]: Got next segment. Inline: true Aug 7 21:27:46.945922443 (11h47m46.53526742s) [upload=12274]: Waiting for error group managing segments... Aug 7 21:27:46.945960726 (11h47m46.535305713s) [upload=12274]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:27:46.945965345 (11h47m46.535310332s) [upload=12274]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:27:46.945999029 (11h47m46.535344026s) [upload=12274]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:27:47.238086931 (11h47m46.827431908s) [upload=12274]: Done uploading segments... Aug 7 21:27:47.238104565 (11h47m46.827449552s) [upload=12274]: Upload finished. err: <nil> Aug 7 21:28:27.597717226 (11h48m27.187062213s) [upload=12282]: Starting upload Aug 7 21:28:27.597768604 (11h48m27.187113581s) [upload=12282]: Done starting upload Aug 7 21:28:27.597851232 (11h48m27.187196219s) [upload=12282]: Uploading segments... Aug 7 21:28:27.597881140 (11h48m27.187226127s) [upload=12282]: Got next segment. Inline: true Aug 7 21:28:27.597885238 (11h48m27.187230225s) [upload=12282]: Waiting for error group managing segments... Aug 7 21:28:27.597916337 (11h48m27.187261324s) [upload=12282]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:28:27.597920134 (11h48m27.187265111s) [upload=12282]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:28:27.597922509 (11h48m27.187267496s) [upload=12282]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:28:27.890743040 (11h48m27.480088057s) [upload=12282]: Done uploading segments... Aug 7 21:28:27.890764190 (11h48m27.480109177s) [upload=12282]: Upload finished. err: <nil> Aug 7 21:29:22.617930722 (11h49m22.207275709s) [upload=12295]: Starting upload Aug 7 21:29:22.617969135 (11h49m22.207314122s) [upload=12295]: Done starting upload Aug 7 21:29:22.618026967 (11h49m22.207371954s) [upload=12295]: Uploading segments... Aug 7 21:29:22.618059649 (11h49m22.207404636s) [upload=12295]: Got next segment. Inline: true Aug 7 21:29:22.618063226 (11h49m22.207408213s) [upload=12295]: Waiting for error group managing segments... Aug 7 21:29:22.618096049 (11h49m22.207441026s) [upload=12295]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:29:22.618100216 (11h49m22.207445193s) [upload=12295]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:29:22.618104334 (11h49m22.207449321s) [upload=12295]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:29:23.935417449 (11h49m23.524762486s) [upload=12295]: Done uploading segments... Aug 7 21:29:23.935440473 (11h49m23.52478548s) [upload=12295]: Upload finished. err: <nil> Aug 7 21:29:59.675674818 (11h49m59.265019805s) [upload=12308]: Starting upload Aug 7 21:29:59.675709946 (11h49m59.265054933s) [upload=12308]: Done starting upload Aug 7 21:29:59.675775310 (11h49m59.265120287s) [upload=12308]: Uploading segments... Aug 7 21:29:59.675809485 (11h49m59.265154462s) [upload=12308]: Got next segment. Inline: true Aug 7 21:29:59.675813022 (11h49m59.265158009s) [upload=12308]: Waiting for error group managing segments... Aug 7 21:29:59.675846747 (11h49m59.265191724s) [upload=12308]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:29:59.675850584 (11h49m59.265195572s) [upload=12308]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:29:59.675852909 (11h49m59.265197896s) [upload=12308]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:29:59.960382218 (11h49m59.549727205s) [upload=12308]: Done uploading segments... Aug 7 21:29:59.960402026 (11h49m59.549747013s) [upload=12308]: Upload finished. err: <nil> Aug 7 21:31:01.204540565 (11h51m0.793885552s) [upload=12324]: Starting upload Aug 7 21:31:01.204590701 (11h51m0.793935688s) [upload=12324]: Done starting upload Aug 7 21:31:01.204646679 (11h51m0.793991666s) [upload=12324]: Uploading segments... Aug 7 21:31:01.204679222 (11h51m0.794024209s) [upload=12324]: Got next segment. Inline: true Aug 7 21:31:01.204683269 (11h51m0.794028256s) [upload=12324]: Waiting for error group managing segments... Aug 7 21:31:01.204721322 (11h51m0.794066309s) [upload=12324]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:31:01.204725289 (11h51m0.794070276s) [upload=12324]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:31:01.204728024 (11h51m0.794073011s) [upload=12324]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:31:01.494645083 (11h51m1.08399007s) [upload=12324]: Done uploading segments... Aug 7 21:31:01.494665120 (11h51m1.084010107s) [upload=12324]: Upload finished. err: <nil> Aug 7 21:31:28.376153869 (11h51m27.965498856s) [upload=12334]: Starting upload Aug 7 21:31:28.376197382 (11h51m27.965542369s) [upload=12334]: Done starting upload Aug 7 21:31:28.376245204 (11h51m27.965590181s) [upload=12334]: Uploading segments... Aug 7 21:31:28.376273257 (11h51m27.965618245s) [upload=12334]: Got next segment. Inline: true Aug 7 21:31:28.376276624 (11h51m27.965621611s) [upload=12334]: Waiting for error group managing segments... Aug 7 21:31:28.376306892 (11h51m27.965651869s) [upload=12334]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:31:28.376312712 (11h51m27.9656577s) [upload=12334]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:31:28.376315419 (11h51m27.965660406s) [upload=12334]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:31:28.645925163 (11h51m28.23527015s) [upload=12334]: Done uploading segments... Aug 7 21:31:28.645942786 (11h51m28.235287773s) [upload=12334]: Upload finished. err: <nil> Aug 7 21:32:07.331024173 (11h52m6.92036916s) [upload=12342]: Starting upload Aug 7 21:32:07.331074259 (11h52m6.920419246s) [upload=12342]: Done starting upload Aug 7 21:32:07.331122251 (11h52m6.920467239s) [upload=12342]: Uploading segments... Aug 7 21:32:07.331154433 (11h52m6.92049942s) [upload=12342]: Got next segment. Inline: true Aug 7 21:32:07.331159583 (11h52m6.92050456s) [upload=12342]: Waiting for error group managing segments... Aug 7 21:32:07.331191624 (11h52m6.920536611s) [upload=12342]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:32:07.331195581 (11h52m6.920540558s) [upload=12342]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:32:07.331197946 (11h52m6.920542933s) [upload=12342]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:32:07.722093791 (11h52m7.311438768s) [upload=12342]: Done uploading segments... Aug 7 21:32:07.722110794 (11h52m7.311455781s) [upload=12342]: Upload finished. err: <nil> Aug 7 21:32:38.771578054 (11h52m38.360923031s) [upload=12353]: Starting upload Aug 7 21:32:38.771650211 (11h52m38.360995188s) [upload=12353]: Done starting upload Aug 7 21:32:38.771738161 (11h52m38.361083148s) [upload=12353]: Uploading segments... Aug 7 21:32:38.771779349 (11h52m38.361124336s) [upload=12353]: Got next segment. Inline: true Aug 7 21:32:38.771782916 (11h52m38.361127893s) [upload=12353]: Waiting for error group managing segments... Aug 7 21:32:38.771815357 (11h52m38.361160334s) [upload=12353]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:32:38.771819266 (11h52m38.361164253s) [upload=12353]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:32:38.771821730 (11h52m38.361166717s) [upload=12353]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:32:39.111523769 (11h52m38.700868796s) [upload=12353]: Done uploading segments... Aug 7 21:32:39.111547013 (11h52m38.700892031s) [upload=12353]: Upload finished. err: <nil> Aug 7 21:33:06.222382527 (11h53m5.811727514s) [upload=12362]: Starting upload Aug 7 21:33:06.222432713 (11h53m5.8117777s) [upload=12362]: Done starting upload Aug 7 21:33:06.222489962 (11h53m5.811834949s) [upload=12362]: Uploading segments... Aug 7 21:33:06.222536221 (11h53m5.811881208s) [upload=12362]: Got next segment. Inline: true Aug 7 21:33:06.222540199 (11h53m5.811885176s) [upload=12362]: Waiting for error group managing segments... Aug 7 21:33:06.222573302 (11h53m5.811918289s) [upload=12362]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:33:06.222576979 (11h53m5.811921966s) [upload=12362]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:33:06.222579153 (11h53m5.81192413s) [upload=12362]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:33:06.561908138 (11h53m6.151253125s) [upload=12362]: Done uploading segments... Aug 7 21:33:06.561986849 (11h53m6.151331836s) [upload=12362]: Upload finished. err: <nil> Duration: 1.891724324s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 24 out of 60 is failed (threshold=3) [...................................X.XXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.000660506s FAIL (2025-08-07 21:37:32 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 4776s) [upload=12265]: Uploading segments... Aug 7 21:27:18.331346968 (11h47m17.920691945s) [upload=12265]: Got next segment. Inline: true Aug 7 21:27:18.331350675 (11h47m17.920695652s) [upload=12265]: Waiting for error group managing segments... Aug 7 21:27:18.331385723 (11h47m17.92073071s) [upload=12265]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:27:18.331389891 (11h47m17.920734878s) [upload=12265]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:27:18.331392486 (11h47m17.920737473s) [upload=12265]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:28:17.685855344 (11h48m17.275200351s) [upload=12265]: Done uploading segments... Aug 7 21:28:17.685882486 (11h48m17.275227473s) [upload=12265]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:28:18.552314505 (11h48m18.141659502s) [upload=12277]: Starting upload Aug 7 21:28:18.552431058 (11h48m18.141776075s) [upload=12277]: Done starting upload Aug 7 21:28:18.552500100 (11h48m18.141845107s) [upload=12277]: Uploading segments... Aug 7 21:28:18.552547911 (11h48m18.141892898s) [upload=12277]: Got next segment. Inline: true Aug 7 21:28:18.552552150 (11h48m18.141897137s) [upload=12277]: Waiting for error group managing segments... Aug 7 21:28:18.552600933 (11h48m18.14194592s) [upload=12277]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:28:18.552607205 (11h48m18.141952212s) [upload=12277]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:28:18.552612195 (11h48m18.141957192s) [upload=12277]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:29:17.904886609 (11h49m17.494231596s) [upload=12277]: Done uploading segments... Aug 7 21:29:17.904932337 (11h49m17.494277354s) [upload=12277]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:29:18.793973395 (11h49m18.383318382s) [upload=12293]: Starting upload Aug 7 21:29:18.794029602 (11h49m18.383374589s) [upload=12293]: Done starting upload Aug 7 21:29:18.794095498 (11h49m18.383440485s) [upload=12293]: Uploading segments... Aug 7 21:29:18.794182454 (11h49m18.383527441s) [upload=12293]: Got next segment. Inline: true Aug 7 21:29:18.794187423 (11h49m18.383532411s) [upload=12293]: Waiting for error group managing segments... Aug 7 21:29:18.794239163 (11h49m18.38358415s) [upload=12293]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:29:18.794244643 (11h49m18.38358962s) [upload=12293]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:29:18.794247088 (11h49m18.383592076s) [upload=12293]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:30:18.146928865 (11h50m17.736273852s) [upload=12293]: Done uploading segments... Aug 7 21:30:18.146964162 (11h50m17.73630915s) [upload=12293]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:30:30.199348334 (11h50m29.788693321s) [upload=12315]: Starting upload Aug 7 21:30:30.199433327 (11h50m29.788778314s) [upload=12315]: Done starting upload Aug 7 21:30:30.199483272 (11h50m29.788828259s) [upload=12315]: Uploading segments... Aug 7 21:30:30.199515665 (11h50m29.788860652s) [upload=12315]: Got next segment. Inline: true Aug 7 21:30:30.199519302 (11h50m29.788864289s) [upload=12315]: Waiting for error group managing segments... Aug 7 21:30:30.199554779 (11h50m29.788899766s) [upload=12315]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:30:30.199559147 (11h50m29.788904134s) [upload=12315]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:30:30.199561592 (11h50m29.788906579s) [upload=12315]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:31:29.557680478 (11h51m29.147025465s) [upload=12315]: Done uploading segments... Aug 7 21:31:29.557706508 (11h51m29.147051485s) [upload=12315]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:31:30.468794234 (11h51m30.058139221s) [upload=12336]: Starting upload Aug 7 21:31:30.468935564 (11h51m30.058280551s) [upload=12336]: Done starting upload Aug 7 21:31:30.469048480 (11h51m30.058393467s) [upload=12336]: Uploading segments... Aug 7 21:31:30.469142200 (11h51m30.058487187s) [upload=12336]: Got next segment. Inline: true Aug 7 21:31:30.469147400 (11h51m30.058492387s) [upload=12336]: Waiting for error group managing segments... Aug 7 21:31:30.469203888 (11h51m30.058548875s) [upload=12336]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:31:30.469208757 (11h51m30.058553744s) [upload=12336]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:31:30.469211582 (11h51m30.058556569s) [upload=12336]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:32:29.815552389 (11h52m29.404897376s) [upload=12336]: Done uploading segments... Aug 7 21:32:29.815578508 (11h52m29.404923495s) [upload=12336]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:32:30.694545842 (11h52m30.283890829s) [upload=12349]: Starting upload Aug 7 21:32:30.694631607 (11h52m30.283976624s) [upload=12349]: Done starting upload Aug 7 21:32:30.694676312 (11h52m30.284021299s) [upload=12349]: Uploading segments... Aug 7 21:32:30.694717561 (11h52m30.284062538s) [upload=12349]: Got next segment. Inline: true Aug 7 21:32:30.694721909 (11h52m30.284066896s) [upload=12349]: Waiting for error group managing segments... Aug 7 21:32:30.694762737 (11h52m30.284107724s) [upload=12349]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:32:30.694766976 (11h52m30.284111953s) [upload=12349]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:32:30.694769561 (11h52m30.284114548s) [upload=12349]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:33:30.042542113 (11h53m29.6318871s) [upload=12349]: Done uploading segments... Aug 7 21:33:30.042561801 (11h53m29.631906788s) [upload=12349]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:33:30.916211822 (11h53m30.505556809s) [upload=12369]: Starting upload Aug 7 21:33:30.916281525 (11h53m30.505626512s) [upload=12369]: Done starting upload Aug 7 21:33:30.916351519 (11h53m30.505696567s) [upload=12369]: Uploading segments... Aug 7 21:33:30.916406474 (11h53m30.505751502s) [upload=12369]: Got next segment. Inline: true Aug 7 21:33:30.916413598 (11h53m30.505758605s) [upload=12369]: Waiting for error group managing segments... Aug 7 21:33:30.916454065 (11h53m30.505799052s) [upload=12369]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:33:30.916457992 (11h53m30.505802979s) [upload=12369]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:33:30.916460378 (11h53m30.505805355s) [upload=12369]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:34:30.262869110 (11h54m29.852214097s) [upload=12369]: Done uploading segments... Aug 7 21:34:30.262896352 (11h54m29.85224134s) [upload=12369]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:34:31.146607534 (11h54m30.735952521s) [upload=12385]: Starting upload Aug 7 21:34:31.146655144 (11h54m30.736000121s) [upload=12385]: Done starting upload Aug 7 21:34:31.146698838 (11h54m30.736043825s) [upload=12385]: Uploading segments... Aug 7 21:34:31.146741720 (11h54m30.736086697s) [upload=12385]: Got next segment. Inline: true Aug 7 21:34:31.146745548 (11h54m30.736090535s) [upload=12385]: Waiting for error group managing segments... Aug 7 21:34:31.146776246 (11h54m30.736121233s) [upload=12385]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:34:31.146779913 (11h54m30.73612489s) [upload=12385]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:34:31.146782388 (11h54m30.736127375s) [upload=12385]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:35:30.500521706 (11h55m30.089866693s) [upload=12385]: Done uploading segments... Aug 7 21:35:30.500542195 (11h55m30.089887182s) [upload=12385]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:35:31.371815861 (11h55m30.961160848s) [upload=12402]: Starting upload Aug 7 21:35:31.371969656 (11h55m30.961314643s) [upload=12402]: Done starting upload Aug 7 21:35:31.372062443 (11h55m30.96140743s) [upload=12402]: Uploading segments... Aug 7 21:35:31.372108090 (11h55m30.961453077s) [upload=12402]: Got next segment. Inline: true Aug 7 21:35:31.372114412 (11h55m30.961459399s) [upload=12402]: Waiting for error group managing segments... Aug 7 21:35:31.372166783 (11h55m30.96151177s) [upload=12402]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:35:31.372172002 (11h55m30.961516999s) [upload=12402]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:35:31.372174898 (11h55m30.961519885s) [upload=12402]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:36:30.721260917 (11h56m30.310605904s) [upload=12402]: Done uploading segments... Aug 7 21:36:30.721280625 (11h56m30.310625612s) [upload=12402]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:36:33.253873027 (11h56m32.843218014s) [upload=12416]: Starting upload Aug 7 21:36:33.253941188 (11h56m32.843286175s) [upload=12416]: Done starting upload Aug 7 21:36:33.254004108 (11h56m32.843349085s) [upload=12416]: Uploading segments... Aug 7 21:36:33.254039966 (11h56m32.843384953s) [upload=12416]: Got next segment. Inline: true Aug 7 21:36:33.254043462 (11h56m32.84338845s) [upload=12416]: Waiting for error group managing segments... Aug 7 21:36:33.254074321 (11h56m32.843419308s) [upload=12416]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:36:33.254078290 (11h56m32.843423277s) [upload=12416]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:36:33.254080714 (11h56m32.843425701s) [upload=12416]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:37:32.603532648 (11h57m32.192877635s) [upload=12416]: Done uploading segments... Aug 7 21:37:32.603556113 (11h57m32.1929011s) [upload=12416]: Upload finished. err: metaclient: context deadline exceeded Duration: 1m0.000681075s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 5 out of 60 is failed (threshold=3) [..X.......................X.........X......X..X.............] duration: 6.733999659s FAIL (2025-08-07 21:37:25 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/4M") ======LOGS====== item: pb.BatchRequestItem_SegmentBegin Aug 7 21:29:25.857659319 (11h49m25.447004296s) [seg_pos={0 0}, upload=12298]: Begin upload segment... Aug 7 21:29:25.857698454 (11h49m25.447043441s) [seg_pos={0 0}, upload=12298]: Done begin upload segment. Aug 7 21:29:25.857736887 (11h49m25.447081864s) [noise=true , address=104.36.82.35:4612 , piece_id=U5WPP5IL75AVFHDGGDB7RLFKNDVVQURKFY23AFG4C52V4YV2ABLQ, seg_pos={0 0}, upload=12298]: Uploading piece... Aug 7 21:29:25.858116173 (11h49m25.44746116s) [noise=true , address=64.49.13.139:4469 , piece_id=M3PL6MGQMOJYJHHRUSDCCHIYP7XOZLYW3IY6O4UIJFYYPER5FTYA, seg_pos={0 0}, upload=12298]: Uploading piece... Aug 7 21:29:25.858376170 (11h49m25.447721147s) [noise=true , address=104.36.82.61:4403 , piece_id=CHFTEE4XZ47N4P4CSLWE6ZPXZ56YYU4SMBSLEQHNH6TZZCVRTFQA, seg_pos={0 0}, upload=12298]: Uploading piece... Aug 7 21:29:25.858676685 (11h49m25.448021662s) [noise=true , address=45.91.116.6:4005 , piece_id=RCN2IDUW4WZA6QNPYHM2E5XM6O75NIC5RUX6KXMOGMWNV2QVAJIQ, seg_pos={0 0}, upload=12298]: Uploading piece... Aug 7 21:29:25.858915152 (11h49m25.448260139s) [noise=true , address=79.127.205.233:4625 , piece_id=KLCUPCY7XM3W3P5RY44CW4QXWMVSTUAZJMIVRF2VWZTIOBBQ6VFQ, seg_pos={0 0}, upload=12298]: Uploading piece... Aug 7 21:29:25.859163256 (11h49m25.448508243s) [noise=true , address=104.36.82.45:4932 , piece_id=25R2X3COEKXS3B3KGM3V33CSIB7XDJYUY5NI3EQ6OVKMPFKZPWHA, seg_pos={0 0}, upload=12298]: Uploading piece... Aug 7 21:29:25.859388758 (11h49m25.448733735s) [noise=true , address=104.36.82.9:4032 , piece_id=UNNJDD7KMLRSVQQOJTV2ORP25CQK4KEWKSAP2ETUHYB4C2QWCZAA, seg_pos={0 0}, upload=12298]: Uploading piece... Aug 7 21:29:25.859588490 (11h49m25.448933477s) [noise=true , address=104.36.82.59:4324 , piece_id=P32XRROGC5HCWJHQJ6U3C2AYPK4M2QVTURJLAR3MIX2BLYOQQBOA, seg_pos={0 0}, upload=12298]: Uploading piece... Aug 7 21:29:25.859768514 (11h49m25.449113491s) [noise=true , address=104.36.82.60:4344 , piece_id=J4HBJAD6EBHQTDYMPCHOCR2QXYCW757LRCSB37FZ4EPFYLRG3BKQ, seg_pos={0 0}, upload=12298]: Uploading piece... Aug 7 21:29:25.861899992 (11h49m25.451244979s) [noise=true , address=104.36.82.16:4300 , piece_id=7MYLVBQJW3UIDKHB3FEE5BYU6XL4Q6BNM2D4ODNKT6IZFC23IAGA, seg_pos={0 0}, upload=12298]: Uploading piece... Aug 7 21:29:25.862982253 (11h49m25.45232724s) [noise=true , address=104.36.82.18:4373 , piece_id=OE3PSGXX5XLV4WUHS3B3IJ3SRAEWM7XNGSJUOE3ZUGR2DZPIT2VA, seg_pos={0 0}, upload=12298]: Uploading piece... Aug 7 21:29:25.863485917 (11h49m25.452830914s) [noise=true , address=64.49.13.140:4510 , piece_id=ZWUGZSDGNXK7PMG7L4CSIJOZ2Y2DGFBQJOMJXWT7FWVHUCMGFHBA, seg_pos={0 0}, upload=12298]: Uploading piece... Aug 7 21:29:25.863967227 (11h49m25.453312214s) [noise=true , address=104.36.82.44:4871 , piece_id=2KTOQN3PLZYK3II27DMUOMJ7XGD3EWZUPUFLMPEYGBNQN37OF6IA, seg_pos={0 0}, upload=12298]: Uploading piece... Aug 7 21:29:25.864375699 (11h49m25.453720686s) [noise=true , address=156.146.43.225:4449 , piece_id=NUURTOFKDGY2QCTR7UFE35CBOBVI2KPFR33RK4MWPURVS4WEBFAQ, seg_pos={0 0}, upload=12298]: Uploading piece... Aug 7 21:29:25.864746759 (11h49m25.454091746s) [noise=true , address=104.36.82.39:4727 , piece_id=2SVEE2SU56DR5U436F7DAYIW6ORGTZLYOUHFK6RPOJ7H3PKSIDPA, seg_pos={0 0}, upload=12298]: Uploading piece... Aug 7 21:29:25.865111086 (11h49m25.454456073s) [noise=true , address=104.36.82.3:4773 , piece_id=NTSJYDBRHNAJVH7MY2LUPLRGMGOHMFIILWR7OW47WUWHKSEDE54Q, seg_pos={0 0}, upload=12298]: Uploading piece... Aug 7 21:29:25.865481785 (11h49m25.454826772s) [noise=true , address=79.127.205.234:4585 , piece_id=2757TS4CR7SO3UT77TOTMBXTJ2GZIG7HJLHQYLX5UUQR5F624OSA, seg_pos={0 0}, upload=12298]: Uploading piece... Aug 7 21:29:25.865876501 (11h49m25.455221488s) [noise=true , address=104.36.82.46:4504 , piece_id=II3WOQ42IDFMPOYSSXYXJ5OCLY6ASVSXJ5VBQ7IM5SLYOCFXFKGQ, seg_pos={0 0}, upload=12298]: Uploading piece... Aug 7 21:29:26.957286373 (11h49m26.54663135s) [upload=12298]: Next returned nil segment Aug 7 21:29:26.957302725 (11h49m26.546647712s) [upload=12298]: Waiting for error group managing segments... Aug 7 21:29:27.242773775 (11h49m26.832118762s) [noise=true , address=64.49.13.140:4510 , piece_id=ZWUGZSDGNXK7PMG7L4CSIJOZ2Y2DGFBQJOMJXWT7FWVHUCMGFHBA, seg_pos={0 0}, upload=12298]: Done uploading piece. err: <nil> Aug 7 21:29:27.254906699 (11h49m26.844251686s) [noise=true , address=79.127.205.234:4585 , piece_id=2757TS4CR7SO3UT77TOTMBXTJ2GZIG7HJLHQYLX5UUQR5F624OSA, seg_pos={0 0}, upload=12298]: Done uploading piece. err: <nil> Aug 7 21:29:27.297381664 (11h49m26.886726651s) [noise=true , address=79.127.205.233:4625 , piece_id=KLCUPCY7XM3W3P5RY44CW4QXWMVSTUAZJMIVRF2VWZTIOBBQ6VFQ, seg_pos={0 0}, upload=12298]: Done uploading piece. err: <nil> Aug 7 21:29:27.304377109 (11h49m26.893722096s) [noise=true , address=64.49.13.139:4469 , piece_id=M3PL6MGQMOJYJHHRUSDCCHIYP7XOZLYW3IY6O4UIJFYYPER5FTYA, seg_pos={0 0}, upload=12298]: Done uploading piece. err: <nil> Aug 7 21:29:27.751965933 (11h49m27.34131091s) [noise=true , address=156.146.43.225:4449 , piece_id=NUURTOFKDGY2QCTR7UFE35CBOBVI2KPFR33RK4MWPURVS4WEBFAQ, seg_pos={0 0}, upload=12298]: Done uploading piece. err: <nil> Aug 7 21:29:27.811346684 (11h49m27.400691671s) [noise=true , address=104.36.82.35:4612 , piece_id=U5WPP5IL75AVFHDGGDB7RLFKNDVVQURKFY23AFG4C52V4YV2ABLQ, seg_pos={0 0}, upload=12298]: Done uploading piece. err: <nil> Aug 7 21:29:27.871486808 (11h49m27.460831795s) [noise=true , address=104.36.82.46:4504 , piece_id=II3WOQ42IDFMPOYSSXYXJ5OCLY6ASVSXJ5VBQ7IM5SLYOCFXFKGQ, seg_pos={0 0}, upload=12298]: Done uploading piece. err: <nil> Aug 7 21:29:27.875300604 (11h49m27.464645591s) [noise=true , address=104.36.82.45:4932 , piece_id=25R2X3COEKXS3B3KGM3V33CSIB7XDJYUY5NI3EQ6OVKMPFKZPWHA, seg_pos={0 0}, upload=12298]: Done uploading piece. err: <nil> Aug 7 21:29:27.932817469 (11h49m27.522162456s) [noise=true , address=45.91.116.6:4005 , piece_id=RCN2IDUW4WZA6QNPYHM2E5XM6O75NIC5RUX6KXMOGMWNV2QVAJIQ, seg_pos={0 0}, upload=12298]: Done uploading piece. err: <nil> Aug 7 21:29:27.938530751 (11h49m27.527875768s) [noise=true , address=104.36.82.18:4373 , piece_id=OE3PSGXX5XLV4WUHS3B3IJ3SRAEWM7XNGSJUOE3ZUGR2DZPIT2VA, seg_pos={0 0}, upload=12298]: Done uploading piece. err: <nil> Aug 7 21:29:27.940085755 (11h49m27.529430742s) [noise=true , address=104.36.82.44:4871 , piece_id=2KTOQN3PLZYK3II27DMUOMJ7XGD3EWZUPUFLMPEYGBNQN37OF6IA, seg_pos={0 0}, upload=12298]: Done uploading piece. err: <nil> Aug 7 21:29:27.961921222 (11h49m27.551266209s) [noise=true , address=104.36.82.3:4773 , piece_id=NTSJYDBRHNAJVH7MY2LUPLRGMGOHMFIILWR7OW47WUWHKSEDE54Q, seg_pos={0 0}, upload=12298]: Done uploading piece. err: <nil> Aug 7 21:29:27.984789475 (11h49m27.574134462s) [noise=true , address=104.36.82.16:4300 , piece_id=7MYLVBQJW3UIDKHB3FEE5BYU6XL4Q6BNM2D4ODNKT6IZFC23IAGA, seg_pos={0 0}, upload=12298]: Done uploading piece. err: <nil> Aug 7 21:29:28.004349759 (11h49m27.593694746s) [noise=true , address=104.36.82.59:4324 , piece_id=P32XRROGC5HCWJHQJ6U3C2AYPK4M2QVTURJLAR3MIX2BLYOQQBOA, seg_pos={0 0}, upload=12298]: Done uploading piece. err: <nil> Aug 7 21:29:28.004368014 (11h49m27.593713001s) [seg_pos={0 0}, upload=12298]: Segment reached optimal threshold of 14 pieces. Aug 7 21:29:28.004510657 (11h49m27.593855654s) [noise=true , address=104.36.82.60:4344 , piece_id=J4HBJAD6EBHQTDYMPCHOCR2QXYCW757LRCSB37FZ4EPFYLRG3BKQ, seg_pos={0 0}, upload=12298]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127WWD5hr1ZBkRLCHz3jb7kpqqnxPS6VuZ1MyQDmitFbhTSbJNK): protocol: expected piece hash; context canceled; EOF Aug 7 21:29:28.004949036 (11h49m27.594294063s) [noise=true , address=104.36.82.61:4403 , piece_id=CHFTEE4XZ47N4P4CSLWE6ZPXZ56YYU4SMBSLEQHNH6TZZCVRTFQA, seg_pos={0 0}, upload=12298]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Lhmif97DEkfhkR7CoWuz54524kk9LWpsvaVLoMRRtKSHMe1pg): protocol: write tcp 188.166.225.170:54274->104.36.82.61:4403: use of closed network connection; write tcp 188.166.225.170:54274->104.36.82.61:4403: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54274->104.36.82.61:4403: use of closed network connection Aug 7 21:29:28.005023909 (11h49m27.594368896s) [noise=true , address=104.36.82.9:4032 , piece_id=UNNJDD7KMLRSVQQOJTV2ORP25CQK4KEWKSAP2ETUHYB4C2QWCZAA, seg_pos={0 0}, upload=12298]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ferXUYkCNRALvKCfTAfqHrNuDJZcqAx7Qx3hr7TaaMdVGzxQQ): protocol: rpc: dial tcp 104.36.82.9:4032: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 21:29:28.005067853 (11h49m27.59441284s) [noise=true , address=104.36.82.39:4727 , piece_id=2SVEE2SU56DR5U436F7DAYIW6ORGTZLYOUHFK6RPOJ7H3PKSIDPA, seg_pos={0 0}, upload=12298]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3): protocol: write tcp 188.166.225.170:49838->104.36.82.39:4727: use of closed network connection; write tcp 188.166.225.170:49838->104.36.82.39:4727: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49838->104.36.82.39:4727: use of closed network connection Aug 7 21:29:28.005079395 (11h49m27.594424382s) [seg_pos={0 0}, upload=12298]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 21:29:28.005142837 (11h49m27.594487824s) [upload=12298]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 21:29:28.005146774 (11h49m27.594491761s) [upload=12298]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:29:28.333526479 (11h49m27.922871497s) [upload=12298]: Done uploading segments... Aug 7 21:29:28.333548311 (11h49m27.922893298s) [upload=12298]: Upload finished. err: <nil> Duration:* 6.734024145s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 26 out of 60 is failed (threshold=3) [................................X..XXXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.000876776s FAIL (2025-08-07 21:37:19 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 30.000896525s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 29 out of 60 is failed (threshold=3) [.............................X..XXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.001792612s FAIL (2025-08-07 21:37:14 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== 548s) [noise=true , address=45.91.116.8:4048 , piece_id=UPSUMKRW74KD65MWWWWPMDGNI2AI7YTLQCYLJDQLW5ZMFQ5L6JTQ, seg_pos={0 0}, upload=12325]: Done uploading piece. err: <nil> Aug 7 21:31:12.754316149 (11h51m12.343661126s) [noise=true , address=45.91.116.4:4938 , piece_id=UQSYGM7V5W4D3TRBMN3Y23GCBCRBOO7PKL2JGYQMGHRILDLMWFWA, seg_pos={0 0}, upload=12325]: Done uploading piece. err: <nil> Aug 7 21:31:12.770614482 (11h51m12.359959469s) [noise=true , address=77.90.31.10:4518 , piece_id=R637TWCQH2CWNFHPKYWPJSCN6VTMVA5ZKOPOLYUXNX32YO7BRWXA, seg_pos={0 0}, upload=12325]: Done uploading piece. err: <nil> Aug 7 21:31:12.821512202 (11h51m12.410857189s) [noise=true , address=104.36.82.11:4113 , piece_id=ZR6MD3XT36EY6X2L6VPTG2FER6KSGR7WCAUXX2VSGTCIKX463I2Q, seg_pos={0 0}, upload=12325]: Done uploading piece. err: <nil> Aug 7 21:31:12.836582997 (11h51m12.425927984s) [noise=true , address=45.91.116.11:4118 , piece_id=NQVQKKFGHZ6TJHBIA6Y2CLRBUAEAR76J2UAF7LW5GZYD235CXUPA, seg_pos={0 0}, upload=12325]: Done uploading piece. err: <nil> Aug 7 21:31:12.868805076 (11h51m12.458150063s) [noise=true , address=104.36.82.7:4911 , piece_id=X6PW3FEO52KVJV65NA7AOK25JBLHU5NZKCUFROBWXUBJ67Z3X4SA, seg_pos={0 0}, upload=12325]: Done uploading piece. err: <nil> Aug 7 21:31:12.917061683 (11h51m12.50640667s) [noise=true , address=77.90.31.12:4589 , piece_id=XSFU2EZXJU5AS5ZAAXM2IEAX5OUK5RNWFJAR54RGBSTG3EHIRWWA, seg_pos={0 0}, upload=12325]: Done uploading piece. err: <nil> Aug 7 21:31:12.947560807 (11h51m12.536905794s) [noise=true , address=45.91.116.11:4119 , piece_id=WUA5BEGRS22XCSTO6BLG5WVLDDBSUHIL7YLZB34V2BQYDBHT2IUA, seg_pos={0 0}, upload=12325]: Done uploading piece. err: <nil> Aug 7 21:31:12.949625526 (11h51m12.538970513s) [noise=true , address=77.90.31.17:4772 , piece_id=DQQ7YCYCPEQTUXFLCFQVAKQWLKFNHUJLPEQJYJHDS3UD32GB3SZA, seg_pos={0 0}, upload=12325]: Done uploading piece. err: <nil> Aug 7 21:31:12.964088127 (11h51m12.553433114s) [noise=true , address=45.91.116.10:4081 , piece_id=K7DBSP2FYBF4R2VINZYDMFMOWC6CGFJU4424N4UH4K76GLPYUBGQ, seg_pos={0 0}, upload=12325]: Done uploading piece. err: <nil> Aug 7 21:31:13.098539068 (11h51m12.687884055s) [noise=true , address=77.90.31.14:4662 , piece_id=ZPA6BIPI4AQZYSYPBE5N6RDTV6JKAR3FFFE6RMD3QCGDZF3VCR6Q, seg_pos={0 0}, upload=12325]: Done uploading piece. err: <nil> Aug 7 21:31:13.113930835 (11h51m12.703275822s) [noise=true , address=45.91.116.12:4227 , piece_id=YLSUXR72VFQSBUMMKSJ7V5FUHFJJNI75BV4LNDDAU7GLRXC7QOKQ, seg_pos={0 0}, upload=12325]: Done uploading piece. err: <nil> Aug 7 21:31:13.193532534 (11h51m12.782877521s) [noise=true , address=45.91.116.13:4155 , piece_id=6WQ6W26YKE5G7HNHHSWYCEEJXZWKUQI5VP3ZIPRSW42VKNNEJSFA, seg_pos={0 0}, upload=12325]: Done uploading piece. err: <nil> Aug 7 21:31:13.193555358 (11h51m12.782900345s) [seg_pos={0 0}, upload=12325]: Segment reached optimal threshold of 13 pieces. Aug 7 21:31:13.193652643 (11h51m12.78299763s) [noise=true , address=77.90.31.15:4698 , piece_id=QYUSHWTBKQIQVL4UXFM7G6DGBEQFUD3PCCGR7VPK7ZEL5CPY25TA, seg_pos={0 0}, upload=12325]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj): protocol: expected piece hash; context canceled; EOF Aug 7 21:31:13.193731805 (11h51m12.783076792s) [noise=true , address=104.36.82.8:4969 , piece_id=O7QLRLAKDH4NA4WIVPGVIVSTKAQBHF3FHLKQKUO65NKSL3E5TTFA, seg_pos={0 0}, upload=12325]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX): protocol: context canceled; context canceled Aug 7 21:31:13.193782492 (11h51m12.783127469s) [noise=true , address=77.90.31.11:4555 , piece_id=IIN26RURP6UL2IHLXTLCYQN63SHDS54UOUDCJY6VQN37KOLKOAEQ, seg_pos={0 0}, upload=12325]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ): protocol: expected piece hash; context canceled; EOF Aug 7 21:31:13.193822459 (11h51m12.783167446s) [seg_pos={0 0}, upload=12325]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 7 21:31:13.193896029 (11h51m12.783241016s) [upload=12325]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 21:31:13.193900848 (11h51m12.783245835s) [upload=12325]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:32:57.622790374 (11h52m57.212135361s) [upload=12325]: Done uploading segments... Aug 7 21:32:57.622816474 (11h52m57.212161461s) [upload=12325]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:32:58.493067791 (11h52m58.082412768s) [upload=12358]: Starting upload Aug 7 21:32:58.493143516 (11h52m58.082488503s) [upload=12358]: Done starting upload Aug 7 21:32:58.493260219 (11h52m58.082605216s) [upload=12358]: Uploading segments... Aug 7 21:32:58.493408123 (11h52m58.08275312s) [upload=12358]: Got next segment. Inline: false Aug 7 21:32:58.493486823 (11h52m58.08283187s) [upload=12358]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:32:58.493495350 (11h52m58.082840397s) [upload=12358]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:33:27.834064010 (11h53m27.423408997s) [upload=12358]: Next segment err: context deadline exceeded Aug 7 21:33:27.834177297 (11h53m27.423522354s) [upload=12358]: Done uploading segments... Aug 7 21:33:27.834234777 (11h53m27.423579804s) [upload=12358]: Upload finished. err: context deadline exceeded Aug 7 21:34:06.788498448 (11h54m6.377843435s) [upload=12377]: Starting upload Aug 7 21:34:06.788552902 (11h54m6.377897879s) [upload=12377]: Done starting upload Aug 7 21:34:06.788640219 (11h54m6.377985206s) [upload=12377]: Uploading segments... Aug 7 21:34:06.788741333 (11h54m6.37808632s) [upload=12377]: Got next segment. Inline: false Aug 7 21:34:06.788771611 (11h54m6.378116588s) [upload=12377]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:34:06.788775478 (11h54m6.378120465s) [upload=12377]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:34:36.148498637 (11h54m35.737843624s) [upload=12377]: Next segment err: context deadline exceeded Aug 7 21:34:36.148571016 (11h54m35.737915993s) [upload=12377]: Done uploading segments... Aug 7 21:34:36.148592206 (11h54m35.737937193s) [upload=12377]: Upload finished. err: context deadline exceeded Aug 7 21:34:46.798231056 (11h54m46.387576043s) [upload=12390]: Starting upload Aug 7 21:34:46.798312272 (11h54m46.387657259s) [upload=12390]: Done starting upload Aug 7 21:34:46.798434505 (11h54m46.387779492s) [upload=12390]: Uploading segments... Aug 7 21:34:46.798669936 (11h54m46.388014923s) [upload=12390]: Got next segment. Inline: false Aug 7 21:34:46.798720914 (11h54m46.388065901s) [upload=12390]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:34:46.798728538 (11h54m46.388073525s) [upload=12390]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:35:16.156158424 (11h55m15.745503411s) [upload=12390]: Next segment err: context deadline exceeded Aug 7 21:35:16.156340202 (11h55m15.745685189s) [upload=12390]: Done uploading segments... Aug 7 21:35:16.156354228 (11h55m15.745699215s) [upload=12390]: Upload finished. err: context deadline exceeded Aug 7 21:35:17.073431127 (11h55m16.662776104s) [upload=12398]: Starting upload Aug 7 21:35:17.073512032 (11h55m16.662857019s) [upload=12398]: Done starting upload Aug 7 21:35:17.073649515 (11h55m16.662994502s) [upload=12398]: Uploading segments... Aug 7 21:35:17.073876329 (11h55m16.663221306s) [upload=12398]: Got next segment. Inline: false Aug 7 21:35:17.073946874 (11h55m16.663291861s) [upload=12398]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:35:17.073951052 (11h55m16.663296039s) [upload=12398]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:35:46.421300430 (11h55m46.010645417s) [upload=12398]: Next segment err: context deadline exceeded Aug 7 21:35:46.421379381 (11h55m46.010724368s) [upload=12398]: Done uploading segments... Aug 7 21:35:46.421403588 (11h55m46.010748575s) [upload=12398]: Upload finished. err: context deadline exceeded Aug 7 21:36:14.655988116 (11h56m14.245333103s) [upload=12412]: Starting upload Aug 7 21:36:14.656035766 (11h56m14.245380753s) [upload=12412]: Done starting upload Aug 7 21:36:14.656201223 (11h56m14.24554621s) [upload=12412]: Uploading segments... Aug 7 21:36:14.656239026 (11h56m14.245584013s) [upload=12412]: Got next segment. Inline: false Aug 7 21:36:14.656262921 (11h56m14.245607908s) [upload=12412]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:36:14.656266869 (11h56m14.245611856s) [upload=12412]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:36:44.014215517 (11h56m43.603560504s) [upload=12412]: Next segment err: context deadline exceeded Aug 7 21:36:44.014386335 (11h56m43.603731322s) [upload=12412]: Done uploading segments... Aug 7 21:36:44.014401784 (11h56m43.603746771s) [upload=12412]: Upload finished. err: context deadline exceeded Aug 7 21:36:44.888037208 (11h56m44.477382195s) [upload=12420]: Starting upload Aug 7 21:36:44.888098274 (11h56m44.477443261s) [upload=12420]: Done starting upload Aug 7 21:36:44.888214917 (11h56m44.477559964s) [upload=12420]: Uploading segments... Aug 7 21:36:44.888539819 (11h56m44.477884806s) [upload=12420]: Got next segment. Inline: false Aug 7 21:36:44.888567562 (11h56m44.477912549s) [upload=12420]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:36:44.888571570 (11h56m44.477916557s) [upload=12420]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:37:14.238615406 (11h57m13.827960393s) [upload=12420]: Next segment err: context deadline exceeded Aug 7 21:37:14.238762127 (11h57m13.828107114s) [upload=12420]: Done uploading segments... Aug 7 21:37:14.238799287 (11h57m13.828144274s) [upload=12420]: Upload finished. err: context deadline exceeded Duration: 30.001805436s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 29 out of 60 is failed (threshold=3) [...............................XXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.000416479s FAIL (2025-08-07 21:37:09 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.000432119s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 30 out of 60 is failed (threshold=3) [..............................XXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.001192426s FAIL (2025-08-07 21:36:58 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.001211001s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 23 out of 60 is failed (threshold=3) [.....................................XXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.000819021s FAIL (2025-08-07 21:36:56 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.000829221s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 21 out of 60 is failed (threshold=3) [.......................................XXXXXXXXXXXXXXXXXXXXX] duration: 30.000508206s FAIL (2025-08-07 21:36:55 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 30.000527743s core uplink us1
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway) antiblip: 4 out of 60 is failed (threshold=3) [.........................................X....X.X...X.......] duration: 691.446379ms remote: "tcp:45.38.24.2:443" target: https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410 FAIL (2025-08-07 21:36:52 UTC) orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": context deadline exceeded Duration: 691.474753ms edge gateway vhost us-select-1
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway) antiblip: 6 out of 60 is failed (threshold=3) [..................................................XX.XX..X.X] duration: 2m0.000768592s remote: "tcp:207.211.208.129:443" target: https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222 FAIL (2025-08-07 21:36:52 UTC) orbiter warning: Get "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222": context deadline exceeded Duration: 2m0.000792548s edge gateway vhost bgp us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 3 out of 60 is failed (threshold=3) [..X...................................X...............X.....] duration: 1.863306116s FAIL (2025-08-07 21:36:50 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/1K") ======LOGS====== 083457s) [upload=12258]: Done uploading segments... Aug 7 21:26:43.142756605 (11h46m42.732101593s) [upload=12258]: Upload finished. err: <nil> Aug 7 21:27:16.645746716 (11h47m16.235091703s) [upload=12264]: Starting upload Aug 7 21:27:16.645808424 (11h47m16.235153411s) [upload=12264]: Done starting upload Aug 7 21:27:16.645853019 (11h47m16.235198006s) [upload=12264]: Uploading segments... Aug 7 21:27:16.645881223 (11h47m16.23522621s) [upload=12264]: Got next segment. Inline: true Aug 7 21:27:16.645884680 (11h47m16.235229667s) [upload=12264]: Waiting for error group managing segments... Aug 7 21:27:16.645915299 (11h47m16.235260286s) [upload=12264]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:27:16.645919216 (11h47m16.235264203s) [upload=12264]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:27:16.645921510 (11h47m16.235266498s) [upload=12264]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:27:16.954194817 (11h47m16.543539804s) [upload=12264]: Done uploading segments... Aug 7 21:27:16.954212471 (11h47m16.543557459s) [upload=12264]: Upload finished. err: <nil> Aug 7 21:27:46.945635033 (11h47m46.53498005s) [upload=12274]: Starting upload Aug 7 21:27:46.945720126 (11h47m46.535065163s) [upload=12274]: Done starting upload Aug 7 21:27:46.945850435 (11h47m46.535195422s) [upload=12274]: Uploading segments... Aug 7 21:27:46.945918004 (11h47m46.535262991s) [upload=12274]: Got next segment. Inline: true Aug 7 21:27:46.945922443 (11h47m46.53526742s) [upload=12274]: Waiting for error group managing segments... Aug 7 21:27:46.945960726 (11h47m46.535305713s) [upload=12274]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:27:46.945965345 (11h47m46.535310332s) [upload=12274]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:27:46.945999029 (11h47m46.535344026s) [upload=12274]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:27:47.238086931 (11h47m46.827431908s) [upload=12274]: Done uploading segments... Aug 7 21:27:47.238104565 (11h47m46.827449552s) [upload=12274]: Upload finished. err: <nil> Aug 7 21:28:27.597717226 (11h48m27.187062213s) [upload=12282]: Starting upload Aug 7 21:28:27.597768604 (11h48m27.187113581s) [upload=12282]: Done starting upload Aug 7 21:28:27.597851232 (11h48m27.187196219s) [upload=12282]: Uploading segments... Aug 7 21:28:27.597881140 (11h48m27.187226127s) [upload=12282]: Got next segment. Inline: true Aug 7 21:28:27.597885238 (11h48m27.187230225s) [upload=12282]: Waiting for error group managing segments... Aug 7 21:28:27.597916337 (11h48m27.187261324s) [upload=12282]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:28:27.597920134 (11h48m27.187265111s) [upload=12282]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:28:27.597922509 (11h48m27.187267496s) [upload=12282]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:28:27.890743040 (11h48m27.480088057s) [upload=12282]: Done uploading segments... Aug 7 21:28:27.890764190 (11h48m27.480109177s) [upload=12282]: Upload finished. err: <nil> Aug 7 21:29:22.617930722 (11h49m22.207275709s) [upload=12295]: Starting upload Aug 7 21:29:22.617969135 (11h49m22.207314122s) [upload=12295]: Done starting upload Aug 7 21:29:22.618026967 (11h49m22.207371954s) [upload=12295]: Uploading segments... Aug 7 21:29:22.618059649 (11h49m22.207404636s) [upload=12295]: Got next segment. Inline: true Aug 7 21:29:22.618063226 (11h49m22.207408213s) [upload=12295]: Waiting for error group managing segments... Aug 7 21:29:22.618096049 (11h49m22.207441026s) [upload=12295]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:29:22.618100216 (11h49m22.207445193s) [upload=12295]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:29:22.618104334 (11h49m22.207449321s) [upload=12295]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:29:23.935417449 (11h49m23.524762486s) [upload=12295]: Done uploading segments... Aug 7 21:29:23.935440473 (11h49m23.52478548s) [upload=12295]: Upload finished. err: <nil> Aug 7 21:29:59.675674818 (11h49m59.265019805s) [upload=12308]: Starting upload Aug 7 21:29:59.675709946 (11h49m59.265054933s) [upload=12308]: Done starting upload Aug 7 21:29:59.675775310 (11h49m59.265120287s) [upload=12308]: Uploading segments... Aug 7 21:29:59.675809485 (11h49m59.265154462s) [upload=12308]: Got next segment. Inline: true Aug 7 21:29:59.675813022 (11h49m59.265158009s) [upload=12308]: Waiting for error group managing segments... Aug 7 21:29:59.675846747 (11h49m59.265191724s) [upload=12308]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:29:59.675850584 (11h49m59.265195572s) [upload=12308]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:29:59.675852909 (11h49m59.265197896s) [upload=12308]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:29:59.960382218 (11h49m59.549727205s) [upload=12308]: Done uploading segments... Aug 7 21:29:59.960402026 (11h49m59.549747013s) [upload=12308]: Upload finished. err: <nil> Aug 7 21:31:01.204540565 (11h51m0.793885552s) [upload=12324]: Starting upload Aug 7 21:31:01.204590701 (11h51m0.793935688s) [upload=12324]: Done starting upload Aug 7 21:31:01.204646679 (11h51m0.793991666s) [upload=12324]: Uploading segments... Aug 7 21:31:01.204679222 (11h51m0.794024209s) [upload=12324]: Got next segment. Inline: true Aug 7 21:31:01.204683269 (11h51m0.794028256s) [upload=12324]: Waiting for error group managing segments... Aug 7 21:31:01.204721322 (11h51m0.794066309s) [upload=12324]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:31:01.204725289 (11h51m0.794070276s) [upload=12324]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:31:01.204728024 (11h51m0.794073011s) [upload=12324]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:31:01.494645083 (11h51m1.08399007s) [upload=12324]: Done uploading segments... Aug 7 21:31:01.494665120 (11h51m1.084010107s) [upload=12324]: Upload finished. err: <nil> Aug 7 21:31:28.376153869 (11h51m27.965498856s) [upload=12334]: Starting upload Aug 7 21:31:28.376197382 (11h51m27.965542369s) [upload=12334]: Done starting upload Aug 7 21:31:28.376245204 (11h51m27.965590181s) [upload=12334]: Uploading segments... Aug 7 21:31:28.376273257 (11h51m27.965618245s) [upload=12334]: Got next segment. Inline: true Aug 7 21:31:28.376276624 (11h51m27.965621611s) [upload=12334]: Waiting for error group managing segments... Aug 7 21:31:28.376306892 (11h51m27.965651869s) [upload=12334]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:31:28.376312712 (11h51m27.9656577s) [upload=12334]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:31:28.376315419 (11h51m27.965660406s) [upload=12334]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:31:28.645925163 (11h51m28.23527015s) [upload=12334]: Done uploading segments... Aug 7 21:31:28.645942786 (11h51m28.235287773s) [upload=12334]: Upload finished. err: <nil> Aug 7 21:32:07.331024173 (11h52m6.92036916s) [upload=12342]: Starting upload Aug 7 21:32:07.331074259 (11h52m6.920419246s) [upload=12342]: Done starting upload Aug 7 21:32:07.331122251 (11h52m6.920467239s) [upload=12342]: Uploading segments... Aug 7 21:32:07.331154433 (11h52m6.92049942s) [upload=12342]: Got next segment. Inline: true Aug 7 21:32:07.331159583 (11h52m6.92050456s) [upload=12342]: Waiting for error group managing segments... Aug 7 21:32:07.331191624 (11h52m6.920536611s) [upload=12342]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:32:07.331195581 (11h52m6.920540558s) [upload=12342]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:32:07.331197946 (11h52m6.920542933s) [upload=12342]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:32:07.722093791 (11h52m7.311438768s) [upload=12342]: Done uploading segments... Aug 7 21:32:07.722110794 (11h52m7.311455781s) [upload=12342]: Upload finished. err: <nil> Aug 7 21:32:38.771578054 (11h52m38.360923031s) [upload=12353]: Starting upload Aug 7 21:32:38.771650211 (11h52m38.360995188s) [upload=12353]: Done starting upload Aug 7 21:32:38.771738161 (11h52m38.361083148s) [upload=12353]: Uploading segments... Aug 7 21:32:38.771779349 (11h52m38.361124336s) [upload=12353]: Got next segment. Inline: true Aug 7 21:32:38.771782916 (11h52m38.361127893s) [upload=12353]: Waiting for error group managing segments... Aug 7 21:32:38.771815357 (11h52m38.361160334s) [upload=12353]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:32:38.771819266 (11h52m38.361164253s) [upload=12353]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:32:38.771821730 (11h52m38.361166717s) [upload=12353]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:32:39.111523769 (11h52m38.700868796s) [upload=12353]: Done uploading segments... Aug 7 21:32:39.111547013 (11h52m38.700892031s) [upload=12353]: Upload finished. err: <nil> Aug 7 21:33:06.222382527 (11h53m5.811727514s) [upload=12362]: Starting upload Aug 7 21:33:06.222432713 (11h53m5.8117777s) [upload=12362]: Done starting upload Aug 7 21:33:06.222489962 (11h53m5.811834949s) [upload=12362]: Uploading segments... Aug 7 21:33:06.222536221 (11h53m5.811881208s) [upload=12362]: Got next segment. Inline: true Aug 7 21:33:06.222540199 (11h53m5.811885176s) [upload=12362]: Waiting for error group managing segments... Aug 7 21:33:06.222573302 (11h53m5.811918289s) [upload=12362]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:33:06.222576979 (11h53m5.811921966s) [upload=12362]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:33:06.222579153 (11h53m5.81192413s) [upload=12362]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:33:06.561908138 (11h53m6.151253125s) [upload=12362]: Done uploading segments... Aug 7 21:33:06.561986849 (11h53m6.151331836s) [upload=12362]: Upload finished. err: <nil> Duration: 1.863328447s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 28 out of 60 is failed (threshold=3) [..............................X..XXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.000518103s FAIL (2025-08-07 21:36:44 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== pload=12325]: Uploading piece... Aug 7 21:31:10.545694760 (11h51m10.135039747s) [noise=true , address=45.91.116.11:4118 , piece_id=NQVQKKFGHZ6TJHBIA6Y2CLRBUAEAR76J2UAF7LW5GZYD235CXUPA, seg_pos={0 0}, upload=12325]: Uploading piece... Aug 7 21:31:10.545946482 (11h51m10.135291469s) [noise=true , address=104.36.82.11:4113 , piece_id=ZR6MD3XT36EY6X2L6VPTG2FER6KSGR7WCAUXX2VSGTCIKX463I2Q, seg_pos={0 0}, upload=12325]: Uploading piece... Aug 7 21:31:10.546188054 (11h51m10.135533041s) [noise=true , address=77.90.31.15:4698 , piece_id=QYUSHWTBKQIQVL4UXFM7G6DGBEQFUD3PCCGR7VPK7ZEL5CPY25TA, seg_pos={0 0}, upload=12325]: Uploading piece... Aug 7 21:31:12.296546206 (11h51m11.885891193s) [upload=12325]: Next returned nil segment Aug 7 21:31:12.296574871 (11h51m11.885919858s) [upload=12325]: Waiting for error group managing segments... Aug 7 21:31:12.684002571 (11h51m12.273347548s) [noise=true , address=45.91.116.8:4048 , piece_id=UPSUMKRW74KD65MWWWWPMDGNI2AI7YTLQCYLJDQLW5ZMFQ5L6JTQ, seg_pos={0 0}, upload=12325]: Done uploading piece. err: <nil> Aug 7 21:31:12.754316149 (11h51m12.343661126s) [noise=true , address=45.91.116.4:4938 , piece_id=UQSYGM7V5W4D3TRBMN3Y23GCBCRBOO7PKL2JGYQMGHRILDLMWFWA, seg_pos={0 0}, upload=12325]: Done uploading piece. err: <nil> Aug 7 21:31:12.770614482 (11h51m12.359959469s) [noise=true , address=77.90.31.10:4518 , piece_id=R637TWCQH2CWNFHPKYWPJSCN6VTMVA5ZKOPOLYUXNX32YO7BRWXA, seg_pos={0 0}, upload=12325]: Done uploading piece. err: <nil> Aug 7 21:31:12.821512202 (11h51m12.410857189s) [noise=true , address=104.36.82.11:4113 , piece_id=ZR6MD3XT36EY6X2L6VPTG2FER6KSGR7WCAUXX2VSGTCIKX463I2Q, seg_pos={0 0}, upload=12325]: Done uploading piece. err: <nil> Aug 7 21:31:12.836582997 (11h51m12.425927984s) [noise=true , address=45.91.116.11:4118 , piece_id=NQVQKKFGHZ6TJHBIA6Y2CLRBUAEAR76J2UAF7LW5GZYD235CXUPA, seg_pos={0 0}, upload=12325]: Done uploading piece. err: <nil> Aug 7 21:31:12.868805076 (11h51m12.458150063s) [noise=true , address=104.36.82.7:4911 , piece_id=X6PW3FEO52KVJV65NA7AOK25JBLHU5NZKCUFROBWXUBJ67Z3X4SA, seg_pos={0 0}, upload=12325]: Done uploading piece. err: <nil> Aug 7 21:31:12.917061683 (11h51m12.50640667s) [noise=true , address=77.90.31.12:4589 , piece_id=XSFU2EZXJU5AS5ZAAXM2IEAX5OUK5RNWFJAR54RGBSTG3EHIRWWA, seg_pos={0 0}, upload=12325]: Done uploading piece. err: <nil> Aug 7 21:31:12.947560807 (11h51m12.536905794s) [noise=true , address=45.91.116.11:4119 , piece_id=WUA5BEGRS22XCSTO6BLG5WVLDDBSUHIL7YLZB34V2BQYDBHT2IUA, seg_pos={0 0}, upload=12325]: Done uploading piece. err: <nil> Aug 7 21:31:12.949625526 (11h51m12.538970513s) [noise=true , address=77.90.31.17:4772 , piece_id=DQQ7YCYCPEQTUXFLCFQVAKQWLKFNHUJLPEQJYJHDS3UD32GB3SZA, seg_pos={0 0}, upload=12325]: Done uploading piece. err: <nil> Aug 7 21:31:12.964088127 (11h51m12.553433114s) [noise=true , address=45.91.116.10:4081 , piece_id=K7DBSP2FYBF4R2VINZYDMFMOWC6CGFJU4424N4UH4K76GLPYUBGQ, seg_pos={0 0}, upload=12325]: Done uploading piece. err: <nil> Aug 7 21:31:13.098539068 (11h51m12.687884055s) [noise=true , address=77.90.31.14:4662 , piece_id=ZPA6BIPI4AQZYSYPBE5N6RDTV6JKAR3FFFE6RMD3QCGDZF3VCR6Q, seg_pos={0 0}, upload=12325]: Done uploading piece. err: <nil> Aug 7 21:31:13.113930835 (11h51m12.703275822s) [noise=true , address=45.91.116.12:4227 , piece_id=YLSUXR72VFQSBUMMKSJ7V5FUHFJJNI75BV4LNDDAU7GLRXC7QOKQ, seg_pos={0 0}, upload=12325]: Done uploading piece. err: <nil> Aug 7 21:31:13.193532534 (11h51m12.782877521s) [noise=true , address=45.91.116.13:4155 , piece_id=6WQ6W26YKE5G7HNHHSWYCEEJXZWKUQI5VP3ZIPRSW42VKNNEJSFA, seg_pos={0 0}, upload=12325]: Done uploading piece. err: <nil> Aug 7 21:31:13.193555358 (11h51m12.782900345s) [seg_pos={0 0}, upload=12325]: Segment reached optimal threshold of 13 pieces. Aug 7 21:31:13.193652643 (11h51m12.78299763s) [noise=true , address=77.90.31.15:4698 , piece_id=QYUSHWTBKQIQVL4UXFM7G6DGBEQFUD3PCCGR7VPK7ZEL5CPY25TA, seg_pos={0 0}, upload=12325]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj): protocol: expected piece hash; context canceled; EOF Aug 7 21:31:13.193731805 (11h51m12.783076792s) [noise=true , address=104.36.82.8:4969 , piece_id=O7QLRLAKDH4NA4WIVPGVIVSTKAQBHF3FHLKQKUO65NKSL3E5TTFA, seg_pos={0 0}, upload=12325]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX): protocol: context canceled; context canceled Aug 7 21:31:13.193782492 (11h51m12.783127469s) [noise=true , address=77.90.31.11:4555 , piece_id=IIN26RURP6UL2IHLXTLCYQN63SHDS54UOUDCJY6VQN37KOLKOAEQ, seg_pos={0 0}, upload=12325]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ): protocol: expected piece hash; context canceled; EOF Aug 7 21:31:13.193822459 (11h51m12.783167446s) [seg_pos={0 0}, upload=12325]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 7 21:31:13.193896029 (11h51m12.783241016s) [upload=12325]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 21:31:13.193900848 (11h51m12.783245835s) [upload=12325]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:32:57.622790374 (11h52m57.212135361s) [upload=12325]: Done uploading segments... Aug 7 21:32:57.622816474 (11h52m57.212161461s) [upload=12325]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:32:58.493067791 (11h52m58.082412768s) [upload=12358]: Starting upload Aug 7 21:32:58.493143516 (11h52m58.082488503s) [upload=12358]: Done starting upload Aug 7 21:32:58.493260219 (11h52m58.082605216s) [upload=12358]: Uploading segments... Aug 7 21:32:58.493408123 (11h52m58.08275312s) [upload=12358]: Got next segment. Inline: false Aug 7 21:32:58.493486823 (11h52m58.08283187s) [upload=12358]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:32:58.493495350 (11h52m58.082840397s) [upload=12358]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:33:27.834064010 (11h53m27.423408997s) [upload=12358]: Next segment err: context deadline exceeded Aug 7 21:33:27.834177297 (11h53m27.423522354s) [upload=12358]: Done uploading segments... Aug 7 21:33:27.834234777 (11h53m27.423579804s) [upload=12358]: Upload finished. err: context deadline exceeded Aug 7 21:34:06.788498448 (11h54m6.377843435s) [upload=12377]: Starting upload Aug 7 21:34:06.788552902 (11h54m6.377897879s) [upload=12377]: Done starting upload Aug 7 21:34:06.788640219 (11h54m6.377985206s) [upload=12377]: Uploading segments... Aug 7 21:34:06.788741333 (11h54m6.37808632s) [upload=12377]: Got next segment. Inline: false Aug 7 21:34:06.788771611 (11h54m6.378116588s) [upload=12377]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:34:06.788775478 (11h54m6.378120465s) [upload=12377]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:34:36.148498637 (11h54m35.737843624s) [upload=12377]: Next segment err: context deadline exceeded Aug 7 21:34:36.148571016 (11h54m35.737915993s) [upload=12377]: Done uploading segments... Aug 7 21:34:36.148592206 (11h54m35.737937193s) [upload=12377]: Upload finished. err: context deadline exceeded Aug 7 21:34:46.798231056 (11h54m46.387576043s) [upload=12390]: Starting upload Aug 7 21:34:46.798312272 (11h54m46.387657259s) [upload=12390]: Done starting upload Aug 7 21:34:46.798434505 (11h54m46.387779492s) [upload=12390]: Uploading segments... Aug 7 21:34:46.798669936 (11h54m46.388014923s) [upload=12390]: Got next segment. Inline: false Aug 7 21:34:46.798720914 (11h54m46.388065901s) [upload=12390]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:34:46.798728538 (11h54m46.388073525s) [upload=12390]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:35:16.156158424 (11h55m15.745503411s) [upload=12390]: Next segment err: context deadline exceeded Aug 7 21:35:16.156340202 (11h55m15.745685189s) [upload=12390]: Done uploading segments... Aug 7 21:35:16.156354228 (11h55m15.745699215s) [upload=12390]: Upload finished. err: context deadline exceeded Aug 7 21:35:17.073431127 (11h55m16.662776104s) [upload=12398]: Starting upload Aug 7 21:35:17.073512032 (11h55m16.662857019s) [upload=12398]: Done starting upload Aug 7 21:35:17.073649515 (11h55m16.662994502s) [upload=12398]: Uploading segments... Aug 7 21:35:17.073876329 (11h55m16.663221306s) [upload=12398]: Got next segment. Inline: false Aug 7 21:35:17.073946874 (11h55m16.663291861s) [upload=12398]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:35:17.073951052 (11h55m16.663296039s) [upload=12398]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:35:46.421300430 (11h55m46.010645417s) [upload=12398]: Next segment err: context deadline exceeded Aug 7 21:35:46.421379381 (11h55m46.010724368s) [upload=12398]: Done uploading segments... Aug 7 21:35:46.421403588 (11h55m46.010748575s) [upload=12398]: Upload finished. err: context deadline exceeded Aug 7 21:36:14.655988116 (11h56m14.245333103s) [upload=12412]: Starting upload Aug 7 21:36:14.656035766 (11h56m14.245380753s) [upload=12412]: Done starting upload Aug 7 21:36:14.656201223 (11h56m14.24554621s) [upload=12412]: Uploading segments... Aug 7 21:36:14.656239026 (11h56m14.245584013s) [upload=12412]: Got next segment. Inline: false Aug 7 21:36:14.656262921 (11h56m14.245607908s) [upload=12412]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:36:14.656266869 (11h56m14.245611856s) [upload=12412]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:36:44.014215517 (11h56m43.603560504s) [upload=12412]: Next segment err: context deadline exceeded Aug 7 21:36:44.014386335 (11h56m43.603731322s) [upload=12412]: Done uploading segments... Aug 7 21:36:44.014401784 (11h56m43.603746771s) [upload=12412]: Upload finished. err: context deadline exceeded Duration: 30.000537028s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway)

checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway)

checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)