Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 29 out of 60 is failed (threshold=3) [.............................X.XXXXXXXXXXXXXXXXXXXXXXXXXXXX.] duration: 6.317338748s FAIL (2025-08-07 21:42:51 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 94676312 (11h52m30.284021299s) [upload=12349]: Uploading segments... Aug 7 21:32:30.694717561 (11h52m30.284062538s) [upload=12349]: Got next segment. Inline: true Aug 7 21:32:30.694721909 (11h52m30.284066896s) [upload=12349]: Waiting for error group managing segments... Aug 7 21:32:30.694762737 (11h52m30.284107724s) [upload=12349]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:32:30.694766976 (11h52m30.284111953s) [upload=12349]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:32:30.694769561 (11h52m30.284114548s) [upload=12349]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:33:30.042542113 (11h53m29.6318871s) [upload=12349]: Done uploading segments... Aug 7 21:33:30.042561801 (11h53m29.631906788s) [upload=12349]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:33:30.916211822 (11h53m30.505556809s) [upload=12369]: Starting upload Aug 7 21:33:30.916281525 (11h53m30.505626512s) [upload=12369]: Done starting upload Aug 7 21:33:30.916351519 (11h53m30.505696567s) [upload=12369]: Uploading segments... Aug 7 21:33:30.916406474 (11h53m30.505751502s) [upload=12369]: Got next segment. Inline: true Aug 7 21:33:30.916413598 (11h53m30.505758605s) [upload=12369]: Waiting for error group managing segments... Aug 7 21:33:30.916454065 (11h53m30.505799052s) [upload=12369]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:33:30.916457992 (11h53m30.505802979s) [upload=12369]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:33:30.916460378 (11h53m30.505805355s) [upload=12369]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:34:30.262869110 (11h54m29.852214097s) [upload=12369]: Done uploading segments... Aug 7 21:34:30.262896352 (11h54m29.85224134s) [upload=12369]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:34:31.146607534 (11h54m30.735952521s) [upload=12385]: Starting upload Aug 7 21:34:31.146655144 (11h54m30.736000121s) [upload=12385]: Done starting upload Aug 7 21:34:31.146698838 (11h54m30.736043825s) [upload=12385]: Uploading segments... Aug 7 21:34:31.146741720 (11h54m30.736086697s) [upload=12385]: Got next segment. Inline: true Aug 7 21:34:31.146745548 (11h54m30.736090535s) [upload=12385]: Waiting for error group managing segments... Aug 7 21:34:31.146776246 (11h54m30.736121233s) [upload=12385]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:34:31.146779913 (11h54m30.73612489s) [upload=12385]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:34:31.146782388 (11h54m30.736127375s) [upload=12385]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:35:30.500521706 (11h55m30.089866693s) [upload=12385]: Done uploading segments... Aug 7 21:35:30.500542195 (11h55m30.089887182s) [upload=12385]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:35:31.371815861 (11h55m30.961160848s) [upload=12402]: Starting upload Aug 7 21:35:31.371969656 (11h55m30.961314643s) [upload=12402]: Done starting upload Aug 7 21:35:31.372062443 (11h55m30.96140743s) [upload=12402]: Uploading segments... Aug 7 21:35:31.372108090 (11h55m30.961453077s) [upload=12402]: Got next segment. Inline: true Aug 7 21:35:31.372114412 (11h55m30.961459399s) [upload=12402]: Waiting for error group managing segments... Aug 7 21:35:31.372166783 (11h55m30.96151177s) [upload=12402]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:35:31.372172002 (11h55m30.961516999s) [upload=12402]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:35:31.372174898 (11h55m30.961519885s) [upload=12402]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:36:30.721260917 (11h56m30.310605904s) [upload=12402]: Done uploading segments... Aug 7 21:36:30.721280625 (11h56m30.310625612s) [upload=12402]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:36:33.253873027 (11h56m32.843218014s) [upload=12416]: Starting upload Aug 7 21:36:33.253941188 (11h56m32.843286175s) [upload=12416]: Done starting upload Aug 7 21:36:33.254004108 (11h56m32.843349085s) [upload=12416]: Uploading segments... Aug 7 21:36:33.254039966 (11h56m32.843384953s) [upload=12416]: Got next segment. Inline: true Aug 7 21:36:33.254043462 (11h56m32.84338845s) [upload=12416]: Waiting for error group managing segments... Aug 7 21:36:33.254074321 (11h56m32.843419308s) [upload=12416]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:36:33.254078290 (11h56m32.843423277s) [upload=12416]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:36:33.254080714 (11h56m32.843425701s) [upload=12416]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:37:32.603532648 (11h57m32.192877635s) [upload=12416]: Done uploading segments... Aug 7 21:37:32.603556113 (11h57m32.1929011s) [upload=12416]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:37:33.470049892 (11h57m33.059394869s) [upload=12434]: Starting upload Aug 7 21:37:33.470104877 (11h57m33.059449865s) [upload=12434]: Done starting upload Aug 7 21:37:33.470145565 (11h57m33.059490552s) [upload=12434]: Uploading segments... Aug 7 21:37:33.470187375 (11h57m33.059532352s) [upload=12434]: Got next segment. Inline: true Aug 7 21:37:33.470190872 (11h57m33.059535859s) [upload=12434]: Waiting for error group managing segments... Aug 7 21:37:33.470223234 (11h57m33.059568221s) [upload=12434]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:37:33.470226710 (11h57m33.059571697s) [upload=12434]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:37:33.470228904 (11h57m33.059573881s) [upload=12434]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:38:32.827667037 (11h58m32.417012024s) [upload=12434]: Done uploading segments... Aug 7 21:38:32.827687857 (11h58m32.417032834s) [upload=12434]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:38:33.703666664 (11h58m33.293011651s) [upload=12450]: Starting upload Aug 7 21:38:33.703724295 (11h58m33.293069282s) [upload=12450]: Done starting upload Aug 7 21:38:33.703770663 (11h58m33.29311565s) [upload=12450]: Uploading segments... Aug 7 21:38:33.703817212 (11h58m33.293162199s) [upload=12450]: Got next segment. Inline: true Aug 7 21:38:33.703820899 (11h58m33.293165886s) [upload=12450]: Waiting for error group managing segments... Aug 7 21:38:33.703853411 (11h58m33.293198388s) [upload=12450]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:38:33.703857378 (11h58m33.293202365s) [upload=12450]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:38:33.703859744 (11h58m33.293204731s) [upload=12450]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:39:33.059235349 (11h59m32.648580326s) [upload=12450]: Done uploading segments... Aug 7 21:39:33.059253302 (11h59m32.648598289s) [upload=12450]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:39:33.956833499 (11h59m33.546178486s) [upload=12464]: Starting upload Aug 7 21:39:33.956907240 (11h59m33.546252227s) [upload=12464]: Done starting upload Aug 7 21:39:33.956967195 (11h59m33.546312182s) [upload=12464]: Uploading segments... Aug 7 21:39:33.957011540 (11h59m33.546356527s) [upload=12464]: Got next segment. Inline: true Aug 7 21:39:33.957016229 (11h59m33.546361226s) [upload=12464]: Waiting for error group managing segments... Aug 7 21:39:33.957053820 (11h59m33.546398807s) [upload=12464]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:39:33.957058250 (11h59m33.546403237s) [upload=12464]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:39:33.957061736 (11h59m33.546406723s) [upload=12464]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:40:33.302006486 (12h0m32.891351473s) [upload=12464]: Done uploading segments... Aug 7 21:40:33.302025733 (12h0m32.89137071s) [upload=12464]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:40:34.211725785 (12h0m33.801070772s) [upload=12482]: Starting upload Aug 7 21:40:34.211806970 (12h0m33.801151957s) [upload=12482]: Done starting upload Aug 7 21:40:34.211965303 (12h0m33.8013103s) [upload=12482]: Uploading segments... Aug 7 21:40:34.212066617 (12h0m33.801411614s) [upload=12482]: Got next segment. Inline: true Aug 7 21:40:34.212072819 (12h0m33.801417806s) [upload=12482]: Waiting for error group managing segments... Aug 7 21:40:34.212126081 (12h0m33.801471068s) [upload=12482]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:40:34.212132573 (12h0m33.80147756s) [upload=12482]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:40:34.212136120 (12h0m33.801481107s) [upload=12482]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:41:33.557664583 (12h1m33.14700957s) [upload=12482]: Done uploading segments... Aug 7 21:41:33.557694410 (12h1m33.147039387s) [upload=12482]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:41:34.468736652 (12h1m34.058081639s) [upload=12501]: Starting upload Aug 7 21:41:34.468803340 (12h1m34.058148327s) [upload=12501]: Done starting upload Aug 7 21:41:34.468862643 (12h1m34.05820763s) [upload=12501]: Uploading segments... Aug 7 21:41:34.468901758 (12h1m34.058246735s) [upload=12501]: Got next segment. Inline: true Aug 7 21:41:34.468905907 (12h1m34.058250884s) [upload=12501]: Waiting for error group managing segments... Aug 7 21:41:34.468943147 (12h1m34.058288134s) [upload=12501]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:41:34.468947105 (12h1m34.058292092s) [upload=12501]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:41:34.468949690 (12h1m34.058294677s) [upload=12501]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:42:33.814395640 (12h2m33.403740627s) [upload=12501]: Done uploading segments... Aug 7 21:42:33.814423744 (12h2m33.403768731s) [upload=12501]: Upload finished. err: metaclient: context deadline exceeded Duration: 6.317360911s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 26 out of 60 is failed (threshold=3) [.................................XXXXXXXXXXXXXXXXXXXXXXXXXX.] duration: 8.950729944s FAIL (2025-08-07 21:42:49 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 8.95074325s core uplink us1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 36 out of 60 is failed (threshold=3) [......................XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.X.] duration: 12.275130902s FAIL (2025-08-07 21:42:49 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 12.275152062s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 32 out of 60 is failed (threshold=3) [..........................XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.X.] duration: 18.372115345s FAIL (2025-08-07 21:42:48 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 745s) [upload=12364]: Done starting upload Aug 7 21:33:17.618658725 (11h53m17.208003742s) [upload=12364]: Uploading segments... Aug 7 21:33:17.618713580 (11h53m17.208058597s) [upload=12364]: Got next segment. Inline: true Aug 7 21:33:17.618720533 (11h53m17.20806553s) [upload=12364]: Waiting for error group managing segments... Aug 7 21:33:17.618772433 (11h53m17.20811743s) [upload=12364]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:33:17.618779636 (11h53m17.208124643s) [upload=12364]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:33:17.618784776 (11h53m17.208129793s) [upload=12364]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:33:46.957323530 (11h53m46.546668577s) [upload=12364]: Done uploading segments... Aug 7 21:33:46.957345773 (11h53m46.54669075s) [upload=12364]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:34:19.197336641 (11h54m18.786681628s) [upload=12382]: Starting upload Aug 7 21:34:19.197461931 (11h54m18.786806918s) [upload=12382]: Done starting upload Aug 7 21:34:19.197512228 (11h54m18.786857225s) [upload=12382]: Uploading segments... Aug 7 21:34:19.197561282 (11h54m18.786906269s) [upload=12382]: Got next segment. Inline: true Aug 7 21:34:19.197565189 (11h54m18.786910166s) [upload=12382]: Waiting for error group managing segments... Aug 7 21:34:19.197610725 (11h54m18.786955723s) [upload=12382]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:34:19.197615966 (11h54m18.786960953s) [upload=12382]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:34:19.197619142 (11h54m18.786964139s) [upload=12382]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:34:48.535954936 (11h54m48.125299923s) [upload=12382]: Done uploading segments... Aug 7 21:34:48.535978650 (11h54m48.125323637s) [upload=12382]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:35:05.991757765 (11h55m5.581102752s) [upload=12393]: Starting upload Aug 7 21:35:05.991822780 (11h55m5.581167767s) [upload=12393]: Done starting upload Aug 7 21:35:05.991946456 (11h55m5.581291443s) [upload=12393]: Uploading segments... Aug 7 21:35:05.991988116 (11h55m5.581333104s) [upload=12393]: Got next segment. Inline: true Aug 7 21:35:05.991994929 (11h55m5.581339996s) [upload=12393]: Waiting for error group managing segments... Aug 7 21:35:05.992042741 (11h55m5.581387728s) [upload=12393]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:35:05.992048021 (11h55m5.581393018s) [upload=12393]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:35:05.992051768 (11h55m5.581396755s) [upload=12393]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:35:35.339368736 (11h55m34.928713723s) [upload=12393]: Done uploading segments... Aug 7 21:35:35.339503534 (11h55m34.928848522s) [upload=12393]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:36:12.952326182 (11h56m12.541671159s) [upload=12411]: Starting upload Aug 7 21:36:12.952385075 (11h56m12.541730052s) [upload=12411]: Done starting upload Aug 7 21:36:12.952428238 (11h56m12.541773225s) [upload=12411]: Uploading segments... Aug 7 21:36:12.952455510 (11h56m12.541800497s) [upload=12411]: Got next segment. Inline: true Aug 7 21:36:12.952459417 (11h56m12.541804404s) [upload=12411]: Waiting for error group managing segments... Aug 7 21:36:12.952507650 (11h56m12.541852637s) [upload=12411]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:36:12.952514352 (11h56m12.541859349s) [upload=12411]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:36:12.952518550 (11h56m12.541863537s) [upload=12411]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:36:42.299158316 (11h56m41.888503304s) [upload=12411]: Done uploading segments... Aug 7 21:36:42.299175039 (11h56m41.888520026s) [upload=12411]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:37:22.572699237 (11h57m22.162044224s) [upload=12432]: Starting upload Aug 7 21:37:22.572782937 (11h57m22.162127924s) [upload=12432]: Done starting upload Aug 7 21:37:22.572842260 (11h57m22.162187247s) [upload=12432]: Uploading segments... Aug 7 21:37:22.572885313 (11h57m22.1622303s) [upload=12432]: Got next segment. Inline: true Aug 7 21:37:22.572889591 (11h57m22.162234578s) [upload=12432]: Waiting for error group managing segments... Aug 7 21:37:22.572930138 (11h57m22.162275125s) [upload=12432]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:37:22.572934296 (11h57m22.162279283s) [upload=12432]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:37:22.572937121 (11h57m22.162282118s) [upload=12432]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:37:51.933954276 (11h57m51.523299253s) [upload=12432]: Done uploading segments... Aug 7 21:37:51.933970927 (11h57m51.523315914s) [upload=12432]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:38:28.745940745 (11h58m28.335285732s) [upload=12448]: Starting upload Aug 7 21:38:28.745997504 (11h58m28.335342491s) [upload=12448]: Done starting upload Aug 7 21:38:28.746040176 (11h58m28.335385163s) [upload=12448]: Uploading segments... Aug 7 21:38:28.746078458 (11h58m28.335423446s) [upload=12448]: Got next segment. Inline: true Aug 7 21:38:28.746081766 (11h58m28.335426743s) [upload=12448]: Waiting for error group managing segments... Aug 7 21:38:28.746111923 (11h58m28.33545691s) [upload=12448]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:38:28.746115540 (11h58m28.335460527s) [upload=12448]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:38:28.746118205 (11h58m28.335463192s) [upload=12448]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:38:58.105315693 (11h58m57.694660691s) [upload=12448]: Done uploading segments... Aug 7 21:38:58.105342465 (11h58m57.694687452s) [upload=12448]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:39:27.137238449 (11h59m26.726583436s) [upload=12462]: Starting upload Aug 7 21:39:27.137294907 (11h59m26.726639894s) [upload=12462]: Done starting upload Aug 7 21:39:27.137337368 (11h59m26.726682355s) [upload=12462]: Uploading segments... Aug 7 21:39:27.137366043 (11h59m26.72671103s) [upload=12462]: Got next segment. Inline: true Aug 7 21:39:27.137370061 (11h59m26.726715048s) [upload=12462]: Waiting for error group managing segments... Aug 7 21:39:27.137401901 (11h59m26.726746879s) [upload=12462]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:39:27.137405518 (11h59m26.726750495s) [upload=12462]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:39:27.137407813 (11h59m26.72675279s) [upload=12462]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:39:56.495186447 (11h59m56.084531434s) [upload=12462]: Done uploading segments... Aug 7 21:39:56.495206415 (11h59m56.084551402s) [upload=12462]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:40:01.914615953 (12h0m1.50396094s) [upload=12475]: Starting upload Aug 7 21:40:01.914684295 (12h0m1.504029282s) [upload=12475]: Done starting upload Aug 7 21:40:01.914741163 (12h0m1.50408616s) [upload=12475]: Uploading segments... Aug 7 21:40:01.914783103 (12h0m1.50412809s) [upload=12475]: Got next segment. Inline: true Aug 7 21:40:01.914788143 (12h0m1.504133131s) [upload=12475]: Waiting for error group managing segments... Aug 7 21:40:01.914833950 (12h0m1.504178937s) [upload=12475]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:40:01.914840073 (12h0m1.50418507s) [upload=12475]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:40:01.914846014 (12h0m1.504191011s) [upload=12475]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:40:31.268196986 (12h0m30.857542023s) [upload=12475]: Done uploading segments... Aug 7 21:40:31.268223536 (12h0m30.857568533s) [upload=12475]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:40:46.248438399 (12h0m45.837783386s) [upload=12487]: Starting upload Aug 7 21:40:46.248491410 (12h0m45.837836387s) [upload=12487]: Done starting upload Aug 7 21:40:46.248540414 (12h0m45.837885401s) [upload=12487]: Uploading segments... Aug 7 21:40:46.248572345 (12h0m45.837917332s) [upload=12487]: Got next segment. Inline: true Aug 7 21:40:46.248576133 (12h0m45.83792112s) [upload=12487]: Waiting for error group managing segments... Aug 7 21:40:46.248613013 (12h0m45.837958s) [upload=12487]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:40:46.248616850 (12h0m45.837961837s) [upload=12487]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:40:46.248619255 (12h0m45.837964242s) [upload=12487]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:40:55.261300379 (12h0m54.850645366s) [upload=12487]: Done uploading segments... Aug 7 21:40:55.261318514 (12h0m54.850663501s) [upload=12487]: Upload finished. err: <nil> Aug 7 21:42:00.373248983 (12h1m59.962594s) [upload=12509]: Starting upload Aug 7 21:42:00.373363813 (12h1m59.96270884s) [upload=12509]: Done starting upload Aug 7 21:42:00.373438045 (12h1m59.962783062s) [upload=12509]: Uploading segments... Aug 7 21:42:00.373501075 (12h1m59.962846083s) [upload=12509]: Got next segment. Inline: true Aug 7 21:42:00.373507809 (12h1m59.962852816s) [upload=12509]: Waiting for error group managing segments... Aug 7 21:42:00.373577642 (12h1m59.962922669s) [upload=12509]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:42:00.373584325 (12h1m59.962929342s) [upload=12509]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:42:00.373590016 (12h1m59.962935053s) [upload=12509]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:42:29.724044552 (12h2m29.313389539s) [upload=12509]: Done uploading segments... Aug 7 21:42:29.724072415 (12h2m29.313417392s) [upload=12509]: Upload finished. err: metaclient: context deadline exceeded Duration: 18.372129984s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 35 out of 60 is failed (threshold=3) [......................X..XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.] duration: 14.063487447s FAIL (2025-08-07 21:42:47 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== s) [upload=12494]: Done starting upload Aug 7 21:41:12.973492313 (12h1m12.5628373s) [upload=12494]: Uploading segments... Aug 7 21:41:12.973522961 (12h1m12.562867939s) [upload=12494]: Got next segment. Inline: false Aug 7 21:41:12.973563950 (12h1m12.562908988s) [upload=12494]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:41:12.973573599 (12h1m12.562918626s) [upload=12494]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:41:42.322571975 (12h1m41.911916962s) [upload=12494]: Next segment err: context deadline exceeded Aug 7 21:41:42.322630547 (12h1m41.911975534s) [upload=12494]: Done uploading segments... Aug 7 21:41:42.322649854 (12h1m41.911994841s) [upload=12494]: Upload finished. err: context deadline exceeded Aug 7 21:41:49.603002482 (12h1m49.19234746s) [upload=12504]: Starting upload Aug 7 21:41:49.603077727 (12h1m49.192422714s) [upload=12504]: Done starting upload Aug 7 21:41:49.603138152 (12h1m49.192483139s) [upload=12504]: Uploading segments... Aug 7 21:41:49.603256618 (12h1m49.192601606s) [upload=12504]: Got next segment. Inline: false Aug 7 21:41:49.603282889 (12h1m49.192627876s) [upload=12504]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:41:49.603287447 (12h1m49.192632434s) [upload=12504]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:41:50.630672074 (12h1m50.220017061s) [seg_pos={0 0}, upload=12504]: Begin upload segment... Aug 7 21:41:50.630711870 (12h1m50.220056857s) [seg_pos={0 0}, upload=12504]: Done begin upload segment. Aug 7 21:41:50.630733421 (12h1m50.220078408s) [noise=true , address=104.36.82.8:4970 , piece_id=6GY27L2TPOD2PW2LKDT45RWFF7JQE26FOOLGD2WKQJ6HXS4JY65A, seg_pos={0 0}, upload=12504]: Uploading piece... Aug 7 21:41:50.630806139 (12h1m50.220151126s) [noise=true , address=104.36.82.2:4731 , piece_id=HRNMAQO6IBTYFPQAZM5CM4ULWGEU5QOE3N6UJVXMV7IA7JULAVVQ, seg_pos={0 0}, upload=12504]: Uploading piece... Aug 7 21:41:50.631087859 (12h1m50.220432846s) [noise=true , address=45.91.116.5:4298 , piece_id=G2FJAO5RNJB7RL7SPUT7MH7GDGJRIWNHH4V2DO7JA4R57FJ4QSTA, seg_pos={0 0}, upload=12504]: Uploading piece... Aug 7 21:41:50.631294434 (12h1m50.220639421s) [noise=true , address=45.91.116.8:4048 , piece_id=BGU6N35D7JRGMREZ6MMSWYH73DS5XYGV3C2JWCXERW7GOJRAWPLA, seg_pos={0 0}, upload=12504]: Uploading piece... Aug 7 21:41:50.631404244 (12h1m50.220749231s) [noise=true , address=45.91.116.11:4117 , piece_id=6XLHAB34M3ODXHZTOH42IKLK3LH5YPJYM2COEOZU4NON6JVBL2PA, seg_pos={0 0}, upload=12504]: Uploading piece... Aug 7 21:41:50.631632832 (12h1m50.220977819s) [noise=true , address=104.36.82.6:4873 , piece_id=ZGGFHDPLWVCJ2FVBKV2O57VWTHFLVAQVNCWVC4ADBN7YUJTXBOOQ, seg_pos={0 0}, upload=12504]: Uploading piece... Aug 7 21:41:50.631676455 (12h1m50.221021432s) [noise=true , address=77.90.31.16:4735 , piece_id=7XJ7ZILIPMTSG6W3WFMUCP2LQW76VCXGYJK46XLDGAIRM6A3WM5Q, seg_pos={0 0}, upload=12504]: Uploading piece... Aug 7 21:41:50.632110275 (12h1m50.221455262s) [noise=true , address=77.90.31.15:4698 , piece_id=5SRHWJG6K2ARPHZQA3VTA7G2CUHOILFQNNRY73DN47DT2JWFOQ5Q, seg_pos={0 0}, upload=12504]: Uploading piece... Aug 7 21:41:50.632117268 (12h1m50.221462245s) [noise=true , address=77.90.31.13:4628 , piece_id=37RNUEEWVPLRQRGGBWFLGT4EQWTSAOXDWCUAMVNIKUXVTGSQRTZA, seg_pos={0 0}, upload=12504]: Uploading piece... Aug 7 21:41:50.632377196 (12h1m50.221722183s) [noise=true , address=45.91.116.10:4083 , piece_id=OMZCRWRNRFR6PJCGJWADFCFFHR6ZF5IH6Y6VB3LBA6JQMAZP2OAQ, seg_pos={0 0}, upload=12504]: Uploading piece... Aug 7 21:41:50.632446719 (12h1m50.221791706s) [noise=true , address=104.36.82.11:4115 , piece_id=SY2P7GDDHPSXKK6KSYGKPJD4LA2Y52I53OCDEOQPP4ESOPORHNGQ, seg_pos={0 0}, upload=12504]: Uploading piece... Aug 7 21:41:50.632589261 (12h1m50.221934238s) [noise=true , address=77.90.31.19:4841 , piece_id=WPDBTVTA3D5LPPUVIPU2462TBSWADAKHCJK5EDPAZQL6EQ6DKM5Q, seg_pos={0 0}, upload=12504]: Uploading piece... Aug 7 21:41:50.632732737 (12h1m50.222077724s) [noise=true , address=77.90.31.18:4807 , piece_id=QMPHP3YXYA2DKNTCZ3OLKLMX5ZYACGRNU5LBD7LIMBUHSKJDQSUA, seg_pos={0 0}, upload=12504]: Uploading piece... Aug 7 21:41:50.632801227 (12h1m50.222146214s) [noise=true , address=104.36.82.4:4802 , piece_id=5VAFZVXLCNGU5H6IGBM7V5OEXGDOH2BK3GVNTXSOXMEZLVUA4ALA, seg_pos={0 0}, upload=12504]: Uploading piece... Aug 7 21:41:50.633001149 (12h1m50.222346136s) [noise=true , address=104.36.82.7:4909 , piece_id=BOVTCJYYV72MYZFYWTDLUEBSDT456KODYJ7R5NCEU56AX6VPXCSQ, seg_pos={0 0}, upload=12504]: Uploading piece... Aug 7 21:41:50.633031818 (12h1m50.222376795s) [noise=true , address=45.91.116.4:4939 , piece_id=4MD7JBYAZ5VRTMJEQ7YGYM6765QLRHVHYTIEF6LDIY33WBKOL2GA, seg_pos={0 0}, upload=12504]: Uploading piece... Aug 7 21:41:52.379566135 (12h1m51.968911122s) [upload=12504]: Next returned nil segment Aug 7 21:41:52.379583499 (12h1m51.968928486s) [upload=12504]: Waiting for error group managing segments... Aug 7 21:41:52.728315944 (12h1m52.317660921s) [noise=true , address=104.36.82.8:4970 , piece_id=6GY27L2TPOD2PW2LKDT45RWFF7JQE26FOOLGD2WKQJ6HXS4JY65A, seg_pos={0 0}, upload=12504]: Done uploading piece. err: <nil> Aug 7 21:41:52.758521465 (12h1m52.347866452s) [noise=true , address=104.36.82.2:4731 , piece_id=HRNMAQO6IBTYFPQAZM5CM4ULWGEU5QOE3N6UJVXMV7IA7JULAVVQ, seg_pos={0 0}, upload=12504]: Done uploading piece. err: <nil> Aug 7 21:41:52.837840504 (12h1m52.427185481s) [noise=true , address=104.36.82.11:4115 , piece_id=SY2P7GDDHPSXKK6KSYGKPJD4LA2Y52I53OCDEOQPP4ESOPORHNGQ, seg_pos={0 0}, upload=12504]: Done uploading piece. err: <nil> Aug 7 21:41:52.839003488 (12h1m52.428348475s) [noise=true , address=45.91.116.5:4298 , piece_id=G2FJAO5RNJB7RL7SPUT7MH7GDGJRIWNHH4V2DO7JA4R57FJ4QSTA, seg_pos={0 0}, upload=12504]: Done uploading piece. err: <nil> Aug 7 21:41:52.841339678 (12h1m52.430684665s) [noise=true , address=77.90.31.13:4628 , piece_id=37RNUEEWVPLRQRGGBWFLGT4EQWTSAOXDWCUAMVNIKUXVTGSQRTZA, seg_pos={0 0}, upload=12504]: Done uploading piece. err: <nil> Aug 7 21:41:52.842190817 (12h1m52.431535844s) [noise=true , address=104.36.82.6:4873 , piece_id=ZGGFHDPLWVCJ2FVBKV2O57VWTHFLVAQVNCWVC4ADBN7YUJTXBOOQ, seg_pos={0 0}, upload=12504]: Done uploading piece. err: <nil> Aug 7 21:41:52.870129371 (12h1m52.459474358s) [noise=true , address=77.90.31.19:4841 , piece_id=WPDBTVTA3D5LPPUVIPU2462TBSWADAKHCJK5EDPAZQL6EQ6DKM5Q, seg_pos={0 0}, upload=12504]: Done uploading piece. err: <nil> Aug 7 21:41:52.922791747 (12h1m52.512136764s) [noise=true , address=45.91.116.11:4117 , piece_id=6XLHAB34M3ODXHZTOH42IKLK3LH5YPJYM2COEOZU4NON6JVBL2PA, seg_pos={0 0}, upload=12504]: Done uploading piece. err: <nil> Aug 7 21:41:52.940185306 (12h1m52.529530333s) [noise=true , address=45.91.116.8:4048 , piece_id=BGU6N35D7JRGMREZ6MMSWYH73DS5XYGV3C2JWCXERW7GOJRAWPLA, seg_pos={0 0}, upload=12504]: Done uploading piece. err: <nil> Aug 7 21:41:53.087159767 (12h1m52.676504754s) [noise=true , address=77.90.31.16:4735 , piece_id=7XJ7ZILIPMTSG6W3WFMUCP2LQW76VCXGYJK46XLDGAIRM6A3WM5Q, seg_pos={0 0}, upload=12504]: Done uploading piece. err: <nil> Aug 7 21:41:53.100486544 (12h1m52.689831531s) [noise=true , address=77.90.31.15:4698 , piece_id=5SRHWJG6K2ARPHZQA3VTA7G2CUHOILFQNNRY73DN47DT2JWFOQ5Q, seg_pos={0 0}, upload=12504]: Done uploading piece. err: <nil> Aug 7 21:41:53.379418073 (12h1m52.968763051s) [noise=true , address=45.91.116.10:4083 , piece_id=OMZCRWRNRFR6PJCGJWADFCFFHR6ZF5IH6Y6VB3LBA6JQMAZP2OAQ, seg_pos={0 0}, upload=12504]: Done uploading piece. err: <nil> Aug 7 21:41:53.462051994 (12h1m53.051396971s) [noise=true , address=45.91.116.4:4939 , piece_id=4MD7JBYAZ5VRTMJEQ7YGYM6765QLRHVHYTIEF6LDIY33WBKOL2GA, seg_pos={0 0}, upload=12504]: Done uploading piece. err: <nil> Aug 7 21:41:53.462067704 (12h1m53.051412691s) [seg_pos={0 0}, upload=12504]: Segment reached optimal threshold of 13 pieces. Aug 7 21:41:53.462212441 (12h1m53.051557428s) [noise=true , address=104.36.82.7:4909 , piece_id=BOVTCJYYV72MYZFYWTDLUEBSDT456KODYJ7R5NCEU56AX6VPXCSQ, seg_pos={0 0}, upload=12504]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x): protocol: expected piece hash; context canceled; EOF Aug 7 21:41:53.462341318 (12h1m53.051686295s) [noise=true , address=104.36.82.4:4802 , piece_id=5VAFZVXLCNGU5H6IGBM7V5OEXGDOH2BK3GVNTXSOXMEZLVUA4ALA, seg_pos={0 0}, upload=12504]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4): protocol: write tcp 188.166.225.170:48226->104.36.82.4:4802: use of closed network connection; write tcp 188.166.225.170:48226->104.36.82.4:4802: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48226->104.36.82.4:4802: use of closed network connection Aug 7 21:41:53.462400330 (12h1m53.051745308s) [noise=true , address=77.90.31.18:4807 , piece_id=QMPHP3YXYA2DKNTCZ3OLKLMX5ZYACGRNU5LBD7LIMBUHSKJDQSUA, seg_pos={0 0}, upload=12504]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS): protocol: write tcp 188.166.225.170:57834->77.90.31.18:4807: use of closed network connection; context canceled Aug 7 21:41:53.462410060 (12h1m53.051755047s) [seg_pos={0 0}, upload=12504]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 7 21:41:53.462470285 (12h1m53.051815272s) [upload=12504]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 21:41:53.462476076 (12h1m53.051821063s) [upload=12504]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:42:19.994804113 (12h2m19.58414909s) [upload=12504]: Done uploading segments... Aug 7 21:42:19.994827829 (12h2m19.584172816s) [upload=12504]: Upload finished. err: metaclient: context deadline exceeded Duration: 14.063519077s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 25 out of 60 is failed (threshold=3) [................................XXXXXXXXXXXXXXXXXXXXXXXXX...] duration: 50.997102073s FAIL (2025-08-07 21:42:41 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== 2975 (11h51m26.263997962s) [noise=true , address=104.36.82.45:4937 , piece_id=EBUAJKXBK6RZK7T46KXJT5P5LINBFVZCEPGVGG5OZ66SYNOA2RSQ, seg_pos={0 0}, upload=12321]: Done uploading piece. err: <nil> Aug 7 21:31:26.680556480 (11h51m26.269901467s) [noise=true , address=45.91.116.10:4110 , piece_id=F4ZMNLCCFQCBFB6FOPOJFTGPIL5BFYT6TPGL5C7SVLIHMTY4EJJA, seg_pos={0 0}, upload=12321]: Done uploading piece. err: <nil> Aug 7 21:31:26.680580406 (11h51m26.269925393s) [seg_pos={0 0}, upload=12321]: Segment reached optimal threshold of 14 pieces. Aug 7 21:31:26.680752626 (11h51m26.270097613s) [noise=true , address=45.91.116.6:4983 , piece_id=ZWH64KJWCVSTEGVN25WW35HRHTAO3BE3SPKKALNRT7NNLUKB4L4Q, seg_pos={0 0}, upload=12321]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122trJbDHuqeBMQHZT97HP2PgkeuLyUiDiuRm4nayu6ZTH7tat6): protocol: expected piece hash; context canceled; EOF Aug 7 21:31:26.680804515 (11h51m26.270149492s) [noise=true , address=45.91.116.7:4037 , piece_id=WY2F64PFXEGFVE4M6PUHHUYIMTI2KB4CB73DQADZK4QGIEZ2YCYQ, seg_pos={0 0}, upload=12321]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iNPwE2XSfXcaXAXcrH2jcyNGYeDiiWvzAcDQffrmjPwTVqVmK): protocol: expected piece hash; context canceled; EOF Aug 7 21:31:26.680954802 (11h51m26.270299789s) [noise=true , address=104.36.82.61:4375 , piece_id=5LQOQ3MR3EA36DIYLOVO6YDKF6EP2HPHMPG4P6MN76A4IA5BTBEQ, seg_pos={0 0}, upload=12321]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121fzX9v4rgEd78xnDAwcQtthyQHV3yUZUQYLXZ9byPDucGK4T5): protocol: write tcp 188.166.225.170:45604->104.36.82.61:4375: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 21:31:26.680994198 (11h51m26.270339185s) [noise=true , address=104.36.82.8:4977 , piece_id=VY47HC4FF6JU4QQTDVZ62NOIEGWSROTWYMGCB7XXNHJ3GZNJATMQ, seg_pos={0 0}, upload=12321]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WCAm12tXwimFSPqz7Seh6A4oxWqQVxqsF459xux4a5CqjpbAT): protocol: write tcp 188.166.225.170:36156->104.36.82.8:4977: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 21:31:26.681016300 (11h51m26.270361287s) [seg_pos={0 0}, upload=12321]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 21:31:26.681067097 (11h51m26.270412074s) [upload=12321]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 21:31:26.681072137 (11h51m26.270417114s) [upload=12321]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:32:39.816785277 (11h52m39.406130264s) [upload=12321]: Done uploading segments... Aug 7 21:32:39.816815415 (11h52m39.406160402s) [upload=12321]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:32:40.684664778 (11h52m40.274009755s) [upload=12355]: Starting upload Aug 7 21:32:40.684721386 (11h52m40.274066373s) [upload=12355]: Done starting upload Aug 7 21:32:40.684776000 (11h52m40.274121017s) [upload=12355]: Uploading segments... Aug 7 21:32:40.684893314 (11h52m40.274238292s) [upload=12355]: Got next segment. Inline: false Aug 7 21:32:40.684916699 (11h52m40.274261686s) [upload=12355]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:32:40.684920466 (11h52m40.274265443s) [upload=12355]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:33:40.036921304 (11h53m39.626266281s) [upload=12355]: Next segment err: context deadline exceeded Aug 7 21:33:40.037128099 (11h53m39.626473086s) [upload=12355]: Done uploading segments... Aug 7 21:33:40.037148919 (11h53m39.626493896s) [upload=12355]: Upload finished. err: context deadline exceeded Aug 7 21:33:40.899438535 (11h53m40.488783522s) [upload=12372]: Starting upload Aug 7 21:33:40.899490825 (11h53m40.488835832s) [upload=12372]: Done starting upload Aug 7 21:33:40.899590144 (11h53m40.488935131s) [upload=12372]: Uploading segments... Aug 7 21:33:40.899771832 (11h53m40.489116819s) [upload=12372]: Got next segment. Inline: false Aug 7 21:33:40.899795227 (11h53m40.489140214s) [upload=12372]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:33:40.899799275 (11h53m40.489144252s) [upload=12372]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:34:40.257761531 (11h54m39.847106518s) [upload=12372]: Next segment err: context deadline exceeded Aug 7 21:34:40.257885859 (11h54m39.847230856s) [upload=12372]: Done uploading segments... Aug 7 21:34:40.257904695 (11h54m39.847249672s) [upload=12372]: Upload finished. err: context deadline exceeded Aug 7 21:34:41.144124896 (11h54m40.733469873s) [upload=12386]: Starting upload Aug 7 21:34:41.144233103 (11h54m40.73357813s) [upload=12386]: Done starting upload Aug 7 21:34:41.144343113 (11h54m40.733688101s) [upload=12386]: Uploading segments... Aug 7 21:34:41.144700337 (11h54m40.734045364s) [upload=12386]: Got next segment. Inline: false Aug 7 21:34:41.144760612 (11h54m40.734105639s) [upload=12386]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:34:41.144768206 (11h54m40.734113223s) [upload=12386]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:35:40.491924814 (11h55m40.081269811s) [upload=12386]: Next segment err: context deadline exceeded Aug 7 21:35:40.492048891 (11h55m40.081393878s) [upload=12386]: Done uploading segments... Aug 7 21:35:40.492066364 (11h55m40.081411351s) [upload=12386]: Upload finished. err: context deadline exceeded Aug 7 21:35:41.353263670 (11h55m40.942608657s) [upload=12405]: Starting upload Aug 7 21:35:41.353369352 (11h55m40.942714339s) [upload=12405]: Done starting upload Aug 7 21:35:41.353452601 (11h55m40.942797588s) [upload=12405]: Uploading segments... Aug 7 21:35:41.353579473 (11h55m40.942924511s) [upload=12405]: Got next segment. Inline: false Aug 7 21:35:41.353621805 (11h55m40.942966792s) [upload=12405]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:35:41.353626053 (11h55m40.94297104s) [upload=12405]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:36:40.713868241 (11h56m40.303213228s) [upload=12405]: Next segment err: context deadline exceeded Aug 7 21:36:40.713958674 (11h56m40.303303651s) [upload=12405]: Done uploading segments... Aug 7 21:36:40.713972460 (11h56m40.303317447s) [upload=12405]: Upload finished. err: context deadline exceeded Aug 7 21:36:41.596698707 (11h56m41.186043744s) [upload=12418]: Starting upload Aug 7 21:36:41.596808026 (11h56m41.186153063s) [upload=12418]: Done starting upload Aug 7 21:36:41.596885475 (11h56m41.186230462s) [upload=12418]: Uploading segments... Aug 7 21:36:41.597098722 (11h56m41.186443709s) [upload=12418]: Got next segment. Inline: false Aug 7 21:36:41.597141925 (11h56m41.186486912s) [upload=12418]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:36:41.597146604 (11h56m41.186491591s) [upload=12418]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:37:40.947744552 (11h57m40.537089539s) [upload=12418]: Next segment err: context deadline exceeded Aug 7 21:37:40.947825697 (11h57m40.537170684s) [upload=12418]: Done uploading segments... Aug 7 21:37:40.947843592 (11h57m40.537188579s) [upload=12418]: Upload finished. err: context deadline exceeded Aug 7 21:37:41.823629920 (11h57m41.412974907s) [upload=12437]: Starting upload Aug 7 21:37:41.823680908 (11h57m41.413025895s) [upload=12437]: Done starting upload Aug 7 21:37:41.823826305 (11h57m41.413171293s) [upload=12437]: Uploading segments... Aug 7 21:37:41.823857055 (11h57m41.413202052s) [upload=12437]: Got next segment. Inline: false Aug 7 21:37:41.823914255 (11h57m41.413259242s) [upload=12437]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:37:41.823918503 (11h57m41.41326348s) [upload=12437]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:38:41.179029912 (11h58m40.768374899s) [upload=12437]: Next segment err: context deadline exceeded Aug 7 21:38:41.179128490 (11h58m40.768473477s) [upload=12437]: Done uploading segments... Aug 7 21:38:41.179142979 (11h58m40.768487966s) [upload=12437]: Upload finished. err: context deadline exceeded Aug 7 21:38:42.074438453 (11h58m41.66378344s) [upload=12451]: Starting upload Aug 7 21:38:42.074516963 (11h58m41.66386195s) [upload=12451]: Done starting upload Aug 7 21:38:42.074605202 (11h58m41.663950189s) [upload=12451]: Uploading segments... Aug 7 21:38:42.074689222 (11h58m41.664034209s) [upload=12451]: Got next segment. Inline: false Aug 7 21:38:42.074715102 (11h58m41.664060089s) [upload=12451]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:38:42.074718829 (11h58m41.664063816s) [upload=12451]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:39:41.428394360 (11h59m41.017739347s) [upload=12451]: Next segment err: context deadline exceeded Aug 7 21:39:41.428496375 (11h59m41.017841372s) [upload=12451]: Done uploading segments... Aug 7 21:39:41.428519450 (11h59m41.017864437s) [upload=12451]: Upload finished. err: context deadline exceeded Aug 7 21:39:42.313134109 (11h59m41.902479096s) [upload=12468]: Starting upload Aug 7 21:39:42.313191258 (11h59m41.902536245s) [upload=12468]: Done starting upload Aug 7 21:39:42.313256903 (11h59m41.90260189s) [upload=12468]: Uploading segments... Aug 7 21:39:42.313381803 (11h59m41.90272684s) [upload=12468]: Got next segment. Inline: false Aug 7 21:39:42.313472736 (11h59m41.902817763s) [upload=12468]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:39:42.313480300 (11h59m41.902825308s) [upload=12468]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:40:41.669512466 (12h0m41.258857454s) [upload=12468]: Next segment err: context deadline exceeded Aug 7 21:40:41.669582440 (12h0m41.258927427s) [upload=12468]: Done uploading segments... Aug 7 21:40:41.669600465 (12h0m41.258945452s) [upload=12468]: Upload finished. err: context deadline exceeded Duration: 50.997128684s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 6 out of 60 is failed (threshold=3) [..................X.........X......X..X.............X..X....] duration: 7.319919419s FAIL (2025-08-07 21:42:41 UTC) orbiter warning: operation: ListObjects ======ERROR====== unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/4M ======LOGS====== ad=12478]: Done starting upload Aug 7 21:40:13.160345841 (12h0m12.749690828s) [upload=12478]: Uploading segments... Aug 7 21:40:13.160465419 (12h0m12.749810406s) [upload=12478]: Got next segment. Inline: false Aug 7 21:40:13.160488423 (12h0m12.74983341s) [upload=12478]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:40:13.160492521 (12h0m12.749837508s) [upload=12478]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:40:14.024449389 (12h0m13.613794386s) [seg_pos={0 0}, upload=12478]: Begin upload segment... Aug 7 21:40:14.024497139 (12h0m13.613842126s) [seg_pos={0 0}, upload=12478]: Done begin upload segment. Aug 7 21:40:14.024520314 (12h0m13.613865301s) [noise=true , address=104.36.82.12:4163 , piece_id=HCC7QXBZMB3P33M366DZRH2IWVQQVOT56LAWYLY5MW5RARFSISVQ, seg_pos={0 0}, upload=12478]: Uploading piece... Aug 7 21:40:14.024527568 (12h0m13.613872555s) [noise=true , address=104.36.82.12:4177 , piece_id=IPZF77LYNKRXAEDUDGMCUXT4EG5JSIFP3I7E3VPEIN3WNV7XCAKA, seg_pos={0 0}, upload=12478]: Uploading piece... Aug 7 21:40:14.024915139 (12h0m13.614260126s) [noise=true , address=104.36.82.46:4496 , piece_id=FAAS2ISJ3FABRTBOOUGW77IT6A2325ZKX4CSGMHSBRZWSQ4XNOGA, seg_pos={0 0}, upload=12478]: Uploading piece... Aug 7 21:40:14.024946630 (12h0m13.614291617s) [noise=true , address=104.36.82.52:4109 , piece_id=XMAEWF5LMHRJMQAA6FSX6TCYB4OUJ2GVDZQUFDPVFWGXGRQZCZIA, seg_pos={0 0}, upload=12478]: Uploading piece... Aug 7 21:40:14.025230222 (12h0m13.614575199s) [noise=true , address=104.36.82.23:4449 , piece_id=Y52RDGW6QODSWLLQSKBJPSNPUTHB5OLEJHN3Y4GUQ6OZK5XARCRA, seg_pos={0 0}, upload=12478]: Uploading piece... Aug 7 21:40:14.025320585 (12h0m13.614665572s) [noise=true , address=104.36.82.58:4291 , piece_id=246EQHXDOHPPFCKBCQNBTJ46UDD5ZRRPHA7YUUGU22FPJJLKNQ2Q, seg_pos={0 0}, upload=12478]: Uploading piece... Aug 7 21:40:14.025541578 (12h0m13.614886565s) [noise=true , address=64.49.13.170:4721 , piece_id=N7YWI7FRIBKFLUF7TQZLWTT3YTVRDGHUPQJQ2GUP3ENNCZO6DXHQ, seg_pos={0 0}, upload=12478]: Uploading piece... Aug 7 21:40:14.025702406 (12h0m13.615047393s) [noise=true , address=104.36.82.61:4394 , piece_id=2XK3J33NNGFMQU7YDFG35IZ3JFAHUFQ7C3OIUFSNPOSFXB6KEKPQ, seg_pos={0 0}, upload=12478]: Uploading piece... Aug 7 21:40:14.025850619 (12h0m13.615195606s) [noise=true , address=104.36.82.5:4859 , piece_id=3KS2DGR4OLOW4ZUVHSMTVE6XIXZQ3VWOBUXVGTCJTLHNIWGVZOEA, seg_pos={0 0}, upload=12478]: Uploading piece... Aug 7 21:40:14.026035403 (12h0m13.61538039s) [noise=true , address=45.91.116.5:4324 , piece_id=F55DGRECWWVUHFXVMHYRYUHSTO262DXLEIAB2UABJRTIDWG4MEMA, seg_pos={0 0}, upload=12478]: Uploading piece... Aug 7 21:40:14.026287314 (12h0m13.615632301s) [noise=true , address=104.36.82.36:4643 , piece_id=VU7DYF2KI6TB74ERJLYE6BDDQQPMDBSMLZD4H6DKNRKW442ICVGA, seg_pos={0 0}, upload=12478]: Uploading piece... Aug 7 21:40:14.026331439 (12h0m13.615676426s) [noise=true , address=45.91.116.6:4999 , piece_id=CHC3CKW5P6UDSGH2HRU7TMLEEK7A4Q6USGZXSOWVH7SWDBQ6ASDA, seg_pos={0 0}, upload=12478]: Uploading piece... Aug 7 21:40:14.026644838 (12h0m13.615989826s) [noise=true , address=45.91.116.7:4012 , piece_id=FCVPPAJSDQDEMSIP3CJPXF5FQXQ6W3G3CAMR66IOILUY4OQK7P4A, seg_pos={0 0}, upload=12478]: Uploading piece... Aug 7 21:40:14.026679736 (12h0m13.616024713s) [noise=true , address=104.36.82.54:4160 , piece_id=ODKE4XCXDIENJAZZJR7FUE35EM3XNJL3MITY2CFDLZRBBSHQIKXQ, seg_pos={0 0}, upload=12478]: Uploading piece... Aug 7 21:40:14.026890909 (12h0m13.616235896s) [noise=true , address=77.90.31.18:4828 , piece_id=GPXCXC2RW6MRDMLAZJXDML4JO62QAZAZLYZP3JV7J6DQR7QEMTBA, seg_pos={0 0}, upload=12478]: Uploading piece... Aug 7 21:40:14.027132201 (12h0m13.616477178s) [noise=true , address=104.36.82.15:4283 , piece_id=AXCQRAQ6EYNFFHRDCWO6DFLB76ASUYOXWMPMQZYZHHIN6NF3QS5A, seg_pos={0 0}, upload=12478]: Uploading piece... Aug 7 21:40:14.027330621 (12h0m13.616675608s) [noise=true , address=104.36.82.45:4939 , piece_id=BECNPJEALTYQXO4IBONPU27XA2MOFWIP7ATHDJFMYTIG5OVAOHLA, seg_pos={0 0}, upload=12478]: Uploading piece... Aug 7 21:40:14.027577493 (12h0m13.61692248s) [noise=true , address=64.49.13.140:4510 , piece_id=VJJD4RVIWXNIDCZXKEEMBDQD6ZQXPOWS3WKWBAFOOKHBUTJH6CLQ, seg_pos={0 0}, upload=12478]: Uploading piece... Aug 7 21:40:15.144921916 (12h0m14.734266903s) [upload=12478]: Next returned nil segment Aug 7 21:40:15.144943788 (12h0m14.734288775s) [upload=12478]: Waiting for error group managing segments... Aug 7 21:40:15.425370688 (12h0m15.014715665s) [noise=true , address=64.49.13.140:4510 , piece_id=VJJD4RVIWXNIDCZXKEEMBDQD6ZQXPOWS3WKWBAFOOKHBUTJH6CLQ, seg_pos={0 0}, upload=12478]: Done uploading piece. err: <nil> Aug 7 21:40:15.528965431 (12h0m15.118310418s) [noise=true , address=64.49.13.170:4721 , piece_id=N7YWI7FRIBKFLUF7TQZLWTT3YTVRDGHUPQJQ2GUP3ENNCZO6DXHQ, seg_pos={0 0}, upload=12478]: Done uploading piece. err: <nil> Aug 7 21:40:15.945504159 (12h0m15.534849146s) [noise=true , address=104.36.82.46:4496 , piece_id=FAAS2ISJ3FABRTBOOUGW77IT6A2325ZKX4CSGMHSBRZWSQ4XNOGA, seg_pos={0 0}, upload=12478]: Done uploading piece. err: <nil> Aug 7 21:40:15.973785008 (12h0m15.563129995s) [noise=true , address=104.36.82.12:4163 , piece_id=HCC7QXBZMB3P33M366DZRH2IWVQQVOT56LAWYLY5MW5RARFSISVQ, seg_pos={0 0}, upload=12478]: Done uploading piece. err: <nil> Aug 7 21:40:16.002098709 (12h0m15.591443696s) [noise=true , address=104.36.82.52:4109 , piece_id=XMAEWF5LMHRJMQAA6FSX6TCYB4OUJ2GVDZQUFDPVFWGXGRQZCZIA, seg_pos={0 0}, upload=12478]: Done uploading piece. err: <nil> Aug 7 21:40:16.030274577 (12h0m15.619619604s) [noise=true , address=104.36.82.23:4449 , piece_id=Y52RDGW6QODSWLLQSKBJPSNPUTHB5OLEJHN3Y4GUQ6OZK5XARCRA, seg_pos={0 0}, upload=12478]: Done uploading piece. err: <nil> Aug 7 21:40:16.044682273 (12h0m15.63402725s) [noise=true , address=104.36.82.54:4160 , piece_id=ODKE4XCXDIENJAZZJR7FUE35EM3XNJL3MITY2CFDLZRBBSHQIKXQ, seg_pos={0 0}, upload=12478]: Done uploading piece. err: <nil> Aug 7 21:40:16.053282688 (12h0m15.642627665s) [noise=true , address=45.91.116.5:4324 , piece_id=F55DGRECWWVUHFXVMHYRYUHSTO262DXLEIAB2UABJRTIDWG4MEMA, seg_pos={0 0}, upload=12478]: Done uploading piece. err: <nil> Aug 7 21:40:16.058737404 (12h0m15.648082391s) [noise=true , address=104.36.82.58:4291 , piece_id=246EQHXDOHPPFCKBCQNBTJ46UDD5ZRRPHA7YUUGU22FPJJLKNQ2Q, seg_pos={0 0}, upload=12478]: Done uploading piece. err: <nil> Aug 7 21:40:16.093091522 (12h0m15.682436509s) [noise=true , address=104.36.82.12:4177 , piece_id=IPZF77LYNKRXAEDUDGMCUXT4EG5JSIFP3I7E3VPEIN3WNV7XCAKA, seg_pos={0 0}, upload=12478]: Done uploading piece. err: <nil> Aug 7 21:40:16.176548758 (12h0m15.765893735s) [noise=true , address=104.36.82.45:4939 , piece_id=BECNPJEALTYQXO4IBONPU27XA2MOFWIP7ATHDJFMYTIG5OVAOHLA, seg_pos={0 0}, upload=12478]: Done uploading piece. err: <nil> Aug 7 21:40:16.186654263 (12h0m15.77599925s) [noise=true , address=104.36.82.5:4859 , piece_id=3KS2DGR4OLOW4ZUVHSMTVE6XIXZQ3VWOBUXVGTCJTLHNIWGVZOEA, seg_pos={0 0}, upload=12478]: Done uploading piece. err: <nil> Aug 7 21:40:16.255094056 (12h0m15.844439053s) [noise=true , address=45.91.116.7:4012 , piece_id=FCVPPAJSDQDEMSIP3CJPXF5FQXQ6W3G3CAMR66IOILUY4OQK7P4A, seg_pos={0 0}, upload=12478]: Done uploading piece. err: <nil> Aug 7 21:40:16.268089419 (12h0m15.857434406s) [noise=true , address=77.90.31.18:4828 , piece_id=GPXCXC2RW6MRDMLAZJXDML4JO62QAZAZLYZP3JV7J6DQR7QEMTBA, seg_pos={0 0}, upload=12478]: Done uploading piece. err: <nil> Aug 7 21:40:16.268110299 (12h0m15.857455286s) [seg_pos={0 0}, upload=12478]: Segment reached optimal threshold of 14 pieces. Aug 7 21:40:16.268265978 (12h0m15.857610965s) [noise=true , address=104.36.82.61:4394 , piece_id=2XK3J33NNGFMQU7YDFG35IZ3JFAHUFQ7C3OIUFSNPOSFXB6KEKPQ, seg_pos={0 0}, upload=12478]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1dSpTqCxeAcL9PVYbP7BhmuKE7papFPc9swB1Fy83WcFDgQbcF): protocol: expected piece hash; context canceled; EOF Aug 7 21:40:16.268327325 (12h0m15.857672302s) [noise=true , address=45.91.116.6:4999 , piece_id=CHC3CKW5P6UDSGH2HRU7TMLEEK7A4Q6USGZXSOWVH7SWDBQ6ASDA, seg_pos={0 0}, upload=12478]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12evc8vF5f1Z7cPrrC7G7BsGuDWjzttAE2FsQciV7nztBDmrCZB): protocol: expected piece hash; context canceled; EOF Aug 7 21:40:16.268492010 (12h0m15.857836997s) [noise=true , address=104.36.82.36:4643 , piece_id=VU7DYF2KI6TB74ERJLYE6BDDQQPMDBSMLZD4H6DKNRKW442ICVGA, seg_pos={0 0}, upload=12478]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zQsZ4qrfBa95hJYTzjWJJzSea3noCnZ7W22V8ViohjrB5opEw): protocol: write tcp 188.166.225.170:42342->104.36.82.36:4643: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 21:40:16.268605697 (12h0m15.857950684s) [noise=true , address=104.36.82.15:4283 , piece_id=AXCQRAQ6EYNFFHRDCWO6DFLB76ASUYOXWMPMQZYZHHIN6NF3QS5A, seg_pos={0 0}, upload=12478]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ytKjZnmERiZtgJuMJhXG5KFwUQwie27odve4LPE7Pw5hR5ZaU): protocol: write tcp 188.166.225.170:45054->104.36.82.15:4283: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 21:40:16.268677886 (12h0m15.858022873s) [seg_pos={0 0}, upload=12478]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 21:40:16.268787846 (12h0m15.858132843s) [upload=12478]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 21:40:16.268796623 (12h0m15.85814161s) [upload=12478]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:40:17.124513483 (12h0m16.71385847s) [upload=12478]: Done uploading segments... Aug 7 21:40:17.124534484 (12h0m16.713879471s) [upload=12478]: Upload finished. err: <nil> Duration: 7.319948254s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 26 out of 60 is failed (threshold=3) [..................................XXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m45.26757233s FAIL (2025-08-07 21:42:40 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 1m45.267595696s core uplink us1
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway) antiblip: 7 out of 60 is failed (threshold=3) [............................................XX.XX..X.XX.....] duration: 2.356833939s remote: "tcp:207.211.208.129:443" target: https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222 FAIL (2025-08-07 21:42:39 UTC) orbiter warning: Get "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222": context deadline exceeded Duration: 2.35686006s edge gateway vhost bgp us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 29 out of 60 is failed (threshold=3) [..............................X.XXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.00118426s FAIL (2025-08-07 21:42:33 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 94676312 (11h52m30.284021299s) [upload=12349]: Uploading segments... Aug 7 21:32:30.694717561 (11h52m30.284062538s) [upload=12349]: Got next segment. Inline: true Aug 7 21:32:30.694721909 (11h52m30.284066896s) [upload=12349]: Waiting for error group managing segments... Aug 7 21:32:30.694762737 (11h52m30.284107724s) [upload=12349]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:32:30.694766976 (11h52m30.284111953s) [upload=12349]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:32:30.694769561 (11h52m30.284114548s) [upload=12349]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:33:30.042542113 (11h53m29.6318871s) [upload=12349]: Done uploading segments... Aug 7 21:33:30.042561801 (11h53m29.631906788s) [upload=12349]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:33:30.916211822 (11h53m30.505556809s) [upload=12369]: Starting upload Aug 7 21:33:30.916281525 (11h53m30.505626512s) [upload=12369]: Done starting upload Aug 7 21:33:30.916351519 (11h53m30.505696567s) [upload=12369]: Uploading segments... Aug 7 21:33:30.916406474 (11h53m30.505751502s) [upload=12369]: Got next segment. Inline: true Aug 7 21:33:30.916413598 (11h53m30.505758605s) [upload=12369]: Waiting for error group managing segments... Aug 7 21:33:30.916454065 (11h53m30.505799052s) [upload=12369]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:33:30.916457992 (11h53m30.505802979s) [upload=12369]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:33:30.916460378 (11h53m30.505805355s) [upload=12369]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:34:30.262869110 (11h54m29.852214097s) [upload=12369]: Done uploading segments... Aug 7 21:34:30.262896352 (11h54m29.85224134s) [upload=12369]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:34:31.146607534 (11h54m30.735952521s) [upload=12385]: Starting upload Aug 7 21:34:31.146655144 (11h54m30.736000121s) [upload=12385]: Done starting upload Aug 7 21:34:31.146698838 (11h54m30.736043825s) [upload=12385]: Uploading segments... Aug 7 21:34:31.146741720 (11h54m30.736086697s) [upload=12385]: Got next segment. Inline: true Aug 7 21:34:31.146745548 (11h54m30.736090535s) [upload=12385]: Waiting for error group managing segments... Aug 7 21:34:31.146776246 (11h54m30.736121233s) [upload=12385]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:34:31.146779913 (11h54m30.73612489s) [upload=12385]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:34:31.146782388 (11h54m30.736127375s) [upload=12385]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:35:30.500521706 (11h55m30.089866693s) [upload=12385]: Done uploading segments... Aug 7 21:35:30.500542195 (11h55m30.089887182s) [upload=12385]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:35:31.371815861 (11h55m30.961160848s) [upload=12402]: Starting upload Aug 7 21:35:31.371969656 (11h55m30.961314643s) [upload=12402]: Done starting upload Aug 7 21:35:31.372062443 (11h55m30.96140743s) [upload=12402]: Uploading segments... Aug 7 21:35:31.372108090 (11h55m30.961453077s) [upload=12402]: Got next segment. Inline: true Aug 7 21:35:31.372114412 (11h55m30.961459399s) [upload=12402]: Waiting for error group managing segments... Aug 7 21:35:31.372166783 (11h55m30.96151177s) [upload=12402]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:35:31.372172002 (11h55m30.961516999s) [upload=12402]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:35:31.372174898 (11h55m30.961519885s) [upload=12402]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:36:30.721260917 (11h56m30.310605904s) [upload=12402]: Done uploading segments... Aug 7 21:36:30.721280625 (11h56m30.310625612s) [upload=12402]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:36:33.253873027 (11h56m32.843218014s) [upload=12416]: Starting upload Aug 7 21:36:33.253941188 (11h56m32.843286175s) [upload=12416]: Done starting upload Aug 7 21:36:33.254004108 (11h56m32.843349085s) [upload=12416]: Uploading segments... Aug 7 21:36:33.254039966 (11h56m32.843384953s) [upload=12416]: Got next segment. Inline: true Aug 7 21:36:33.254043462 (11h56m32.84338845s) [upload=12416]: Waiting for error group managing segments... Aug 7 21:36:33.254074321 (11h56m32.843419308s) [upload=12416]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:36:33.254078290 (11h56m32.843423277s) [upload=12416]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:36:33.254080714 (11h56m32.843425701s) [upload=12416]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:37:32.603532648 (11h57m32.192877635s) [upload=12416]: Done uploading segments... Aug 7 21:37:32.603556113 (11h57m32.1929011s) [upload=12416]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:37:33.470049892 (11h57m33.059394869s) [upload=12434]: Starting upload Aug 7 21:37:33.470104877 (11h57m33.059449865s) [upload=12434]: Done starting upload Aug 7 21:37:33.470145565 (11h57m33.059490552s) [upload=12434]: Uploading segments... Aug 7 21:37:33.470187375 (11h57m33.059532352s) [upload=12434]: Got next segment. Inline: true Aug 7 21:37:33.470190872 (11h57m33.059535859s) [upload=12434]: Waiting for error group managing segments... Aug 7 21:37:33.470223234 (11h57m33.059568221s) [upload=12434]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:37:33.470226710 (11h57m33.059571697s) [upload=12434]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:37:33.470228904 (11h57m33.059573881s) [upload=12434]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:38:32.827667037 (11h58m32.417012024s) [upload=12434]: Done uploading segments... Aug 7 21:38:32.827687857 (11h58m32.417032834s) [upload=12434]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:38:33.703666664 (11h58m33.293011651s) [upload=12450]: Starting upload Aug 7 21:38:33.703724295 (11h58m33.293069282s) [upload=12450]: Done starting upload Aug 7 21:38:33.703770663 (11h58m33.29311565s) [upload=12450]: Uploading segments... Aug 7 21:38:33.703817212 (11h58m33.293162199s) [upload=12450]: Got next segment. Inline: true Aug 7 21:38:33.703820899 (11h58m33.293165886s) [upload=12450]: Waiting for error group managing segments... Aug 7 21:38:33.703853411 (11h58m33.293198388s) [upload=12450]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:38:33.703857378 (11h58m33.293202365s) [upload=12450]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:38:33.703859744 (11h58m33.293204731s) [upload=12450]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:39:33.059235349 (11h59m32.648580326s) [upload=12450]: Done uploading segments... Aug 7 21:39:33.059253302 (11h59m32.648598289s) [upload=12450]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:39:33.956833499 (11h59m33.546178486s) [upload=12464]: Starting upload Aug 7 21:39:33.956907240 (11h59m33.546252227s) [upload=12464]: Done starting upload Aug 7 21:39:33.956967195 (11h59m33.546312182s) [upload=12464]: Uploading segments... Aug 7 21:39:33.957011540 (11h59m33.546356527s) [upload=12464]: Got next segment. Inline: true Aug 7 21:39:33.957016229 (11h59m33.546361226s) [upload=12464]: Waiting for error group managing segments... Aug 7 21:39:33.957053820 (11h59m33.546398807s) [upload=12464]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:39:33.957058250 (11h59m33.546403237s) [upload=12464]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:39:33.957061736 (11h59m33.546406723s) [upload=12464]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:40:33.302006486 (12h0m32.891351473s) [upload=12464]: Done uploading segments... Aug 7 21:40:33.302025733 (12h0m32.89137071s) [upload=12464]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:40:34.211725785 (12h0m33.801070772s) [upload=12482]: Starting upload Aug 7 21:40:34.211806970 (12h0m33.801151957s) [upload=12482]: Done starting upload Aug 7 21:40:34.211965303 (12h0m33.8013103s) [upload=12482]: Uploading segments... Aug 7 21:40:34.212066617 (12h0m33.801411614s) [upload=12482]: Got next segment. Inline: true Aug 7 21:40:34.212072819 (12h0m33.801417806s) [upload=12482]: Waiting for error group managing segments... Aug 7 21:40:34.212126081 (12h0m33.801471068s) [upload=12482]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:40:34.212132573 (12h0m33.80147756s) [upload=12482]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:40:34.212136120 (12h0m33.801481107s) [upload=12482]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:41:33.557664583 (12h1m33.14700957s) [upload=12482]: Done uploading segments... Aug 7 21:41:33.557694410 (12h1m33.147039387s) [upload=12482]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:41:34.468736652 (12h1m34.058081639s) [upload=12501]: Starting upload Aug 7 21:41:34.468803340 (12h1m34.058148327s) [upload=12501]: Done starting upload Aug 7 21:41:34.468862643 (12h1m34.05820763s) [upload=12501]: Uploading segments... Aug 7 21:41:34.468901758 (12h1m34.058246735s) [upload=12501]: Got next segment. Inline: true Aug 7 21:41:34.468905907 (12h1m34.058250884s) [upload=12501]: Waiting for error group managing segments... Aug 7 21:41:34.468943147 (12h1m34.058288134s) [upload=12501]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:41:34.468947105 (12h1m34.058292092s) [upload=12501]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:41:34.468949690 (12h1m34.058294677s) [upload=12501]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:42:33.814395640 (12h2m33.403740627s) [upload=12501]: Done uploading segments... Aug 7 21:42:33.814423744 (12h2m33.403768731s) [upload=12501]: Upload finished. err: metaclient: context deadline exceeded Duration: 1m0.001194198s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 32 out of 60 is failed (threshold=3) [...........................XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.X] duration: 30.000938762s FAIL (2025-08-07 21:42:29 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 745s) [upload=12364]: Done starting upload Aug 7 21:33:17.618658725 (11h53m17.208003742s) [upload=12364]: Uploading segments... Aug 7 21:33:17.618713580 (11h53m17.208058597s) [upload=12364]: Got next segment. Inline: true Aug 7 21:33:17.618720533 (11h53m17.20806553s) [upload=12364]: Waiting for error group managing segments... Aug 7 21:33:17.618772433 (11h53m17.20811743s) [upload=12364]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:33:17.618779636 (11h53m17.208124643s) [upload=12364]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:33:17.618784776 (11h53m17.208129793s) [upload=12364]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:33:46.957323530 (11h53m46.546668577s) [upload=12364]: Done uploading segments... Aug 7 21:33:46.957345773 (11h53m46.54669075s) [upload=12364]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:34:19.197336641 (11h54m18.786681628s) [upload=12382]: Starting upload Aug 7 21:34:19.197461931 (11h54m18.786806918s) [upload=12382]: Done starting upload Aug 7 21:34:19.197512228 (11h54m18.786857225s) [upload=12382]: Uploading segments... Aug 7 21:34:19.197561282 (11h54m18.786906269s) [upload=12382]: Got next segment. Inline: true Aug 7 21:34:19.197565189 (11h54m18.786910166s) [upload=12382]: Waiting for error group managing segments... Aug 7 21:34:19.197610725 (11h54m18.786955723s) [upload=12382]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:34:19.197615966 (11h54m18.786960953s) [upload=12382]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:34:19.197619142 (11h54m18.786964139s) [upload=12382]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:34:48.535954936 (11h54m48.125299923s) [upload=12382]: Done uploading segments... Aug 7 21:34:48.535978650 (11h54m48.125323637s) [upload=12382]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:35:05.991757765 (11h55m5.581102752s) [upload=12393]: Starting upload Aug 7 21:35:05.991822780 (11h55m5.581167767s) [upload=12393]: Done starting upload Aug 7 21:35:05.991946456 (11h55m5.581291443s) [upload=12393]: Uploading segments... Aug 7 21:35:05.991988116 (11h55m5.581333104s) [upload=12393]: Got next segment. Inline: true Aug 7 21:35:05.991994929 (11h55m5.581339996s) [upload=12393]: Waiting for error group managing segments... Aug 7 21:35:05.992042741 (11h55m5.581387728s) [upload=12393]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:35:05.992048021 (11h55m5.581393018s) [upload=12393]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:35:05.992051768 (11h55m5.581396755s) [upload=12393]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:35:35.339368736 (11h55m34.928713723s) [upload=12393]: Done uploading segments... Aug 7 21:35:35.339503534 (11h55m34.928848522s) [upload=12393]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:36:12.952326182 (11h56m12.541671159s) [upload=12411]: Starting upload Aug 7 21:36:12.952385075 (11h56m12.541730052s) [upload=12411]: Done starting upload Aug 7 21:36:12.952428238 (11h56m12.541773225s) [upload=12411]: Uploading segments... Aug 7 21:36:12.952455510 (11h56m12.541800497s) [upload=12411]: Got next segment. Inline: true Aug 7 21:36:12.952459417 (11h56m12.541804404s) [upload=12411]: Waiting for error group managing segments... Aug 7 21:36:12.952507650 (11h56m12.541852637s) [upload=12411]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:36:12.952514352 (11h56m12.541859349s) [upload=12411]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:36:12.952518550 (11h56m12.541863537s) [upload=12411]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:36:42.299158316 (11h56m41.888503304s) [upload=12411]: Done uploading segments... Aug 7 21:36:42.299175039 (11h56m41.888520026s) [upload=12411]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:37:22.572699237 (11h57m22.162044224s) [upload=12432]: Starting upload Aug 7 21:37:22.572782937 (11h57m22.162127924s) [upload=12432]: Done starting upload Aug 7 21:37:22.572842260 (11h57m22.162187247s) [upload=12432]: Uploading segments... Aug 7 21:37:22.572885313 (11h57m22.1622303s) [upload=12432]: Got next segment. Inline: true Aug 7 21:37:22.572889591 (11h57m22.162234578s) [upload=12432]: Waiting for error group managing segments... Aug 7 21:37:22.572930138 (11h57m22.162275125s) [upload=12432]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:37:22.572934296 (11h57m22.162279283s) [upload=12432]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:37:22.572937121 (11h57m22.162282118s) [upload=12432]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:37:51.933954276 (11h57m51.523299253s) [upload=12432]: Done uploading segments... Aug 7 21:37:51.933970927 (11h57m51.523315914s) [upload=12432]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:38:28.745940745 (11h58m28.335285732s) [upload=12448]: Starting upload Aug 7 21:38:28.745997504 (11h58m28.335342491s) [upload=12448]: Done starting upload Aug 7 21:38:28.746040176 (11h58m28.335385163s) [upload=12448]: Uploading segments... Aug 7 21:38:28.746078458 (11h58m28.335423446s) [upload=12448]: Got next segment. Inline: true Aug 7 21:38:28.746081766 (11h58m28.335426743s) [upload=12448]: Waiting for error group managing segments... Aug 7 21:38:28.746111923 (11h58m28.33545691s) [upload=12448]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:38:28.746115540 (11h58m28.335460527s) [upload=12448]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:38:28.746118205 (11h58m28.335463192s) [upload=12448]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:38:58.105315693 (11h58m57.694660691s) [upload=12448]: Done uploading segments... Aug 7 21:38:58.105342465 (11h58m57.694687452s) [upload=12448]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:39:27.137238449 (11h59m26.726583436s) [upload=12462]: Starting upload Aug 7 21:39:27.137294907 (11h59m26.726639894s) [upload=12462]: Done starting upload Aug 7 21:39:27.137337368 (11h59m26.726682355s) [upload=12462]: Uploading segments... Aug 7 21:39:27.137366043 (11h59m26.72671103s) [upload=12462]: Got next segment. Inline: true Aug 7 21:39:27.137370061 (11h59m26.726715048s) [upload=12462]: Waiting for error group managing segments... Aug 7 21:39:27.137401901 (11h59m26.726746879s) [upload=12462]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:39:27.137405518 (11h59m26.726750495s) [upload=12462]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:39:27.137407813 (11h59m26.72675279s) [upload=12462]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:39:56.495186447 (11h59m56.084531434s) [upload=12462]: Done uploading segments... Aug 7 21:39:56.495206415 (11h59m56.084551402s) [upload=12462]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:40:01.914615953 (12h0m1.50396094s) [upload=12475]: Starting upload Aug 7 21:40:01.914684295 (12h0m1.504029282s) [upload=12475]: Done starting upload Aug 7 21:40:01.914741163 (12h0m1.50408616s) [upload=12475]: Uploading segments... Aug 7 21:40:01.914783103 (12h0m1.50412809s) [upload=12475]: Got next segment. Inline: true Aug 7 21:40:01.914788143 (12h0m1.504133131s) [upload=12475]: Waiting for error group managing segments... Aug 7 21:40:01.914833950 (12h0m1.504178937s) [upload=12475]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:40:01.914840073 (12h0m1.50418507s) [upload=12475]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:40:01.914846014 (12h0m1.504191011s) [upload=12475]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:40:31.268196986 (12h0m30.857542023s) [upload=12475]: Done uploading segments... Aug 7 21:40:31.268223536 (12h0m30.857568533s) [upload=12475]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:40:46.248438399 (12h0m45.837783386s) [upload=12487]: Starting upload Aug 7 21:40:46.248491410 (12h0m45.837836387s) [upload=12487]: Done starting upload Aug 7 21:40:46.248540414 (12h0m45.837885401s) [upload=12487]: Uploading segments... Aug 7 21:40:46.248572345 (12h0m45.837917332s) [upload=12487]: Got next segment. Inline: true Aug 7 21:40:46.248576133 (12h0m45.83792112s) [upload=12487]: Waiting for error group managing segments... Aug 7 21:40:46.248613013 (12h0m45.837958s) [upload=12487]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:40:46.248616850 (12h0m45.837961837s) [upload=12487]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:40:46.248619255 (12h0m45.837964242s) [upload=12487]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:40:55.261300379 (12h0m54.850645366s) [upload=12487]: Done uploading segments... Aug 7 21:40:55.261318514 (12h0m54.850663501s) [upload=12487]: Upload finished. err: <nil> Aug 7 21:42:00.373248983 (12h1m59.962594s) [upload=12509]: Starting upload Aug 7 21:42:00.373363813 (12h1m59.96270884s) [upload=12509]: Done starting upload Aug 7 21:42:00.373438045 (12h1m59.962783062s) [upload=12509]: Uploading segments... Aug 7 21:42:00.373501075 (12h1m59.962846083s) [upload=12509]: Got next segment. Inline: true Aug 7 21:42:00.373507809 (12h1m59.962852816s) [upload=12509]: Waiting for error group managing segments... Aug 7 21:42:00.373577642 (12h1m59.962922669s) [upload=12509]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:42:00.373584325 (12h1m59.962929342s) [upload=12509]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:42:00.373590016 (12h1m59.962935053s) [upload=12509]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:42:29.724044552 (12h2m29.313389539s) [upload=12509]: Done uploading segments... Aug 7 21:42:29.724072415 (12h2m29.313417392s) [upload=12509]: Upload finished. err: metaclient: context deadline exceeded Duration: 30.000957648s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 3 out of 60 is failed (threshold=3) [..............................X...............X..........X..] duration: 2.380812209s FAIL (2025-08-07 21:42:29 UTC) orbiter warning: operation: ListObjects ======ERROR====== unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/1K ======LOGS====== 735125827s) [upload=12376]: Done uploading segments... Aug 7 21:34:02.145826196 (11h54m1.735171183s) [upload=12376]: Upload finished. err: <nil> Aug 7 21:34:47.873603965 (11h54m47.462948952s) [upload=12391]: Starting upload Aug 7 21:34:47.873650053 (11h54m47.46299503s) [upload=12391]: Done starting upload Aug 7 21:34:47.873700940 (11h54m47.463045928s) [upload=12391]: Uploading segments... Aug 7 21:34:47.873733743 (11h54m47.46307872s) [upload=12391]: Got next segment. Inline: true Aug 7 21:34:47.873739335 (11h54m47.463084322s) [upload=12391]: Waiting for error group managing segments... Aug 7 21:34:47.873774331 (11h54m47.463119318s) [upload=12391]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:34:47.873777988 (11h54m47.463122975s) [upload=12391]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:34:47.873780623 (11h54m47.46312561s) [upload=12391]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:34:48.144972113 (11h54m47.7343171s) [upload=12391]: Done uploading segments... Aug 7 21:34:48.144991410 (11h54m47.734336397s) [upload=12391]: Upload finished. err: <nil> Aug 7 21:35:20.363321680 (11h55m19.952666667s) [upload=12399]: Starting upload Aug 7 21:35:20.363369802 (11h55m19.952714789s) [upload=12399]: Done starting upload Aug 7 21:35:20.363434526 (11h55m19.952779513s) [upload=12399]: Uploading segments... Aug 7 21:35:20.363500943 (11h55m19.95284597s) [upload=12399]: Got next segment. Inline: true Aug 7 21:35:20.363506363 (11h55m19.95285135s) [upload=12399]: Waiting for error group managing segments... Aug 7 21:35:20.363545789 (11h55m19.952890776s) [upload=12399]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:35:20.363550498 (11h55m19.952895485s) [upload=12399]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:35:20.363552953 (11h55m19.95289793s) [upload=12399]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:35:20.648472420 (11h55m20.237817407s) [upload=12399]: Done uploading segments... Aug 7 21:35:20.648492398 (11h55m20.237837386s) [upload=12399]: Upload finished. err: <nil> Aug 7 21:35:35.916328511 (11h55m35.505673488s) [upload=12404]: Starting upload Aug 7 21:35:35.916377014 (11h55m35.505722001s) [upload=12404]: Done starting upload Aug 7 21:35:35.916425818 (11h55m35.505770805s) [upload=12404]: Uploading segments... Aug 7 21:35:35.916457979 (11h55m35.505802956s) [upload=12404]: Got next segment. Inline: true Aug 7 21:35:35.916461466 (11h55m35.505806443s) [upload=12404]: Waiting for error group managing segments... Aug 7 21:35:35.916496753 (11h55m35.50584173s) [upload=12404]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:35:35.916501372 (11h55m35.505846359s) [upload=12404]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:35:35.916503666 (11h55m35.505848653s) [upload=12404]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:35:36.531148061 (11h55m36.120493049s) [upload=12404]: Done uploading segments... Aug 7 21:35:36.531167528 (11h55m36.120512505s) [upload=12404]: Upload finished. err: <nil> Aug 7 21:36:48.991080864 (11h56m48.580425851s) [upload=12421]: Starting upload Aug 7 21:36:48.991131251 (11h56m48.580476238s) [upload=12421]: Done starting upload Aug 7 21:36:48.991177239 (11h56m48.580522226s) [upload=12421]: Uploading segments... Aug 7 21:36:48.991205413 (11h56m48.58055039s) [upload=12421]: Got next segment. Inline: true Aug 7 21:36:48.991209060 (11h56m48.580554037s) [upload=12421]: Waiting for error group managing segments... Aug 7 21:36:48.991242914 (11h56m48.580587901s) [upload=12421]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:36:48.991246341 (11h56m48.580591328s) [upload=12421]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:36:48.991248505 (11h56m48.580593492s) [upload=12421]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:36:49.270270137 (11h56m48.859615134s) [upload=12421]: Done uploading segments... Aug 7 21:36:49.270288603 (11h56m48.85963359s) [upload=12421]: Upload finished. err: <nil> Aug 7 21:37:34.116423494 (11h57m33.705768481s) [upload=12435]: Starting upload Aug 7 21:37:34.116478008 (11h57m33.705822996s) [upload=12435]: Done starting upload Aug 7 21:37:34.116546550 (11h57m33.705891537s) [upload=12435]: Uploading segments... Aug 7 21:37:34.116585414 (11h57m33.705930401s) [upload=12435]: Got next segment. Inline: true Aug 7 21:37:34.116589281 (11h57m33.705934268s) [upload=12435]: Waiting for error group managing segments... Aug 7 21:37:34.116631473 (11h57m33.70597646s) [upload=12435]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:37:34.116635470 (11h57m33.705980457s) [upload=12435]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:37:34.116638065 (11h57m33.705983042s) [upload=12435]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:37:34.430160454 (11h57m34.019505441s) [upload=12435]: Done uploading segments... Aug 7 21:37:34.430179780 (11h57m34.019524767s) [upload=12435]: Upload finished. err: <nil> Aug 7 21:38:07.899363525 (11h58m7.488708512s) [upload=12444]: Starting upload Aug 7 21:38:07.899432257 (11h58m7.488777254s) [upload=12444]: Done starting upload Aug 7 21:38:07.899478635 (11h58m7.488823622s) [upload=12444]: Uploading segments... Aug 7 21:38:07.899513011 (11h58m7.488857998s) [upload=12444]: Got next segment. Inline: true Aug 7 21:38:07.899516348 (11h58m7.488861325s) [upload=12444]: Waiting for error group managing segments... Aug 7 21:38:07.899544021 (11h58m7.488888998s) [upload=12444]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:38:07.899547738 (11h58m7.488892725s) [upload=12444]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:38:07.899551905 (11h58m7.488896892s) [upload=12444]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:38:08.175487142 (11h58m7.764832129s) [upload=12444]: Done uploading segments... Aug 7 21:38:08.175506650 (11h58m7.764851637s) [upload=12444]: Upload finished. err: <nil> Aug 7 21:39:12.808926452 (11h59m12.398271439s) [upload=12459]: Starting upload Aug 7 21:39:12.808989873 (11h59m12.39833486s) [upload=12459]: Done starting upload Aug 7 21:39:12.809038286 (11h59m12.398383273s) [upload=12459]: Uploading segments... Aug 7 21:39:12.809068093 (11h59m12.39841308s) [upload=12459]: Got next segment. Inline: true Aug 7 21:39:12.809072080 (11h59m12.398417068s) [upload=12459]: Waiting for error group managing segments... Aug 7 21:39:12.809118720 (11h59m12.398463757s) [upload=12459]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:39:12.809125252 (11h59m12.398470239s) [upload=12459]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:39:12.809127897 (11h59m12.398472884s) [upload=12459]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:39:13.234618442 (11h59m12.823963429s) [upload=12459]: Done uploading segments... Aug 7 21:39:13.234636597 (11h59m12.823981584s) [upload=12459]: Upload finished. err: <nil> Aug 7 21:39:55.593720326 (11h59m55.183065313s) [upload=12472]: Starting upload Aug 7 21:39:55.593767186 (11h59m55.183112163s) [upload=12472]: Done starting upload Aug 7 21:39:55.593842129 (11h59m55.183187117s) [upload=12472]: Uploading segments... Aug 7 21:39:55.593879050 (11h59m55.183224037s) [upload=12472]: Got next segment. Inline: true Aug 7 21:39:55.593882546 (11h59m55.183227533s) [upload=12472]: Waiting for error group managing segments... Aug 7 21:39:55.593915990 (11h59m55.183260978s) [upload=12472]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:39:55.593919928 (11h59m55.183264916s) [upload=12472]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:39:55.593922323 (11h59m55.18326731s) [upload=12472]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:39:55.991096020 (11h59m55.580441007s) [upload=12472]: Done uploading segments... Aug 7 21:39:55.991111470 (11h59m55.580456457s) [upload=12472]: Upload finished. err: <nil> Aug 7 21:40:19.545259354 (12h0m19.134604341s) [upload=12480]: Starting upload Aug 7 21:40:19.545304791 (12h0m19.134649778s) [upload=12480]: Done starting upload Aug 7 21:40:19.545361891 (12h0m19.134706878s) [upload=12480]: Uploading segments... Aug 7 21:40:19.545393461 (12h0m19.134738448s) [upload=12480]: Got next segment. Inline: true Aug 7 21:40:19.545397098 (12h0m19.134742085s) [upload=12480]: Waiting for error group managing segments... Aug 7 21:40:19.545429900 (12h0m19.134774887s) [upload=12480]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:40:19.545433667 (12h0m19.134778655s) [upload=12480]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:40:19.545435852 (12h0m19.134780839s) [upload=12480]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:40:19.891464235 (12h0m19.480809212s) [upload=12480]: Done uploading segments... Aug 7 21:40:19.891480988 (12h0m19.480825965s) [upload=12480]: Upload finished. err: <nil> Aug 7 21:41:23.105250182 (12h1m22.694595169s) [upload=12497]: Starting upload Aug 7 21:41:23.105299878 (12h1m22.694644865s) [upload=12497]: Done starting upload Aug 7 21:41:23.105354182 (12h1m22.694699169s) [upload=12497]: Uploading segments... Aug 7 21:41:23.105392174 (12h1m22.694737161s) [upload=12497]: Got next segment. Inline: true Aug 7 21:41:23.105395991 (12h1m22.694740978s) [upload=12497]: Waiting for error group managing segments... Aug 7 21:41:23.105430738 (12h1m22.694775715s) [upload=12497]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:41:23.105434485 (12h1m22.694779472s) [upload=12497]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:41:23.105438433 (12h1m22.69478342s) [upload=12497]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:41:23.395695219 (12h1m22.985040206s) [upload=12497]: Done uploading segments... Aug 7 21:41:23.395713514 (12h1m22.985058501s) [upload=12497]: Upload finished. err: <nil> Duration: 2.38083831s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 36 out of 60 is failed (threshold=3) [.......................XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.X] duration: 30.001115398s FAIL (2025-08-07 21:42:22 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.001138321s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 35 out of 60 is failed (threshold=3) [.......................X..XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 31.038507206s FAIL (2025-08-07 21:42:19 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== s) [upload=12494]: Done starting upload Aug 7 21:41:12.973492313 (12h1m12.5628373s) [upload=12494]: Uploading segments... Aug 7 21:41:12.973522961 (12h1m12.562867939s) [upload=12494]: Got next segment. Inline: false Aug 7 21:41:12.973563950 (12h1m12.562908988s) [upload=12494]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:41:12.973573599 (12h1m12.562918626s) [upload=12494]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:41:42.322571975 (12h1m41.911916962s) [upload=12494]: Next segment err: context deadline exceeded Aug 7 21:41:42.322630547 (12h1m41.911975534s) [upload=12494]: Done uploading segments... Aug 7 21:41:42.322649854 (12h1m41.911994841s) [upload=12494]: Upload finished. err: context deadline exceeded Aug 7 21:41:49.603002482 (12h1m49.19234746s) [upload=12504]: Starting upload Aug 7 21:41:49.603077727 (12h1m49.192422714s) [upload=12504]: Done starting upload Aug 7 21:41:49.603138152 (12h1m49.192483139s) [upload=12504]: Uploading segments... Aug 7 21:41:49.603256618 (12h1m49.192601606s) [upload=12504]: Got next segment. Inline: false Aug 7 21:41:49.603282889 (12h1m49.192627876s) [upload=12504]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:41:49.603287447 (12h1m49.192632434s) [upload=12504]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:41:50.630672074 (12h1m50.220017061s) [seg_pos={0 0}, upload=12504]: Begin upload segment... Aug 7 21:41:50.630711870 (12h1m50.220056857s) [seg_pos={0 0}, upload=12504]: Done begin upload segment. Aug 7 21:41:50.630733421 (12h1m50.220078408s) [noise=true , address=104.36.82.8:4970 , piece_id=6GY27L2TPOD2PW2LKDT45RWFF7JQE26FOOLGD2WKQJ6HXS4JY65A, seg_pos={0 0}, upload=12504]: Uploading piece... Aug 7 21:41:50.630806139 (12h1m50.220151126s) [noise=true , address=104.36.82.2:4731 , piece_id=HRNMAQO6IBTYFPQAZM5CM4ULWGEU5QOE3N6UJVXMV7IA7JULAVVQ, seg_pos={0 0}, upload=12504]: Uploading piece... Aug 7 21:41:50.631087859 (12h1m50.220432846s) [noise=true , address=45.91.116.5:4298 , piece_id=G2FJAO5RNJB7RL7SPUT7MH7GDGJRIWNHH4V2DO7JA4R57FJ4QSTA, seg_pos={0 0}, upload=12504]: Uploading piece... Aug 7 21:41:50.631294434 (12h1m50.220639421s) [noise=true , address=45.91.116.8:4048 , piece_id=BGU6N35D7JRGMREZ6MMSWYH73DS5XYGV3C2JWCXERW7GOJRAWPLA, seg_pos={0 0}, upload=12504]: Uploading piece... Aug 7 21:41:50.631404244 (12h1m50.220749231s) [noise=true , address=45.91.116.11:4117 , piece_id=6XLHAB34M3ODXHZTOH42IKLK3LH5YPJYM2COEOZU4NON6JVBL2PA, seg_pos={0 0}, upload=12504]: Uploading piece... Aug 7 21:41:50.631632832 (12h1m50.220977819s) [noise=true , address=104.36.82.6:4873 , piece_id=ZGGFHDPLWVCJ2FVBKV2O57VWTHFLVAQVNCWVC4ADBN7YUJTXBOOQ, seg_pos={0 0}, upload=12504]: Uploading piece... Aug 7 21:41:50.631676455 (12h1m50.221021432s) [noise=true , address=77.90.31.16:4735 , piece_id=7XJ7ZILIPMTSG6W3WFMUCP2LQW76VCXGYJK46XLDGAIRM6A3WM5Q, seg_pos={0 0}, upload=12504]: Uploading piece... Aug 7 21:41:50.632110275 (12h1m50.221455262s) [noise=true , address=77.90.31.15:4698 , piece_id=5SRHWJG6K2ARPHZQA3VTA7G2CUHOILFQNNRY73DN47DT2JWFOQ5Q, seg_pos={0 0}, upload=12504]: Uploading piece... Aug 7 21:41:50.632117268 (12h1m50.221462245s) [noise=true , address=77.90.31.13:4628 , piece_id=37RNUEEWVPLRQRGGBWFLGT4EQWTSAOXDWCUAMVNIKUXVTGSQRTZA, seg_pos={0 0}, upload=12504]: Uploading piece... Aug 7 21:41:50.632377196 (12h1m50.221722183s) [noise=true , address=45.91.116.10:4083 , piece_id=OMZCRWRNRFR6PJCGJWADFCFFHR6ZF5IH6Y6VB3LBA6JQMAZP2OAQ, seg_pos={0 0}, upload=12504]: Uploading piece... Aug 7 21:41:50.632446719 (12h1m50.221791706s) [noise=true , address=104.36.82.11:4115 , piece_id=SY2P7GDDHPSXKK6KSYGKPJD4LA2Y52I53OCDEOQPP4ESOPORHNGQ, seg_pos={0 0}, upload=12504]: Uploading piece... Aug 7 21:41:50.632589261 (12h1m50.221934238s) [noise=true , address=77.90.31.19:4841 , piece_id=WPDBTVTA3D5LPPUVIPU2462TBSWADAKHCJK5EDPAZQL6EQ6DKM5Q, seg_pos={0 0}, upload=12504]: Uploading piece... Aug 7 21:41:50.632732737 (12h1m50.222077724s) [noise=true , address=77.90.31.18:4807 , piece_id=QMPHP3YXYA2DKNTCZ3OLKLMX5ZYACGRNU5LBD7LIMBUHSKJDQSUA, seg_pos={0 0}, upload=12504]: Uploading piece... Aug 7 21:41:50.632801227 (12h1m50.222146214s) [noise=true , address=104.36.82.4:4802 , piece_id=5VAFZVXLCNGU5H6IGBM7V5OEXGDOH2BK3GVNTXSOXMEZLVUA4ALA, seg_pos={0 0}, upload=12504]: Uploading piece... Aug 7 21:41:50.633001149 (12h1m50.222346136s) [noise=true , address=104.36.82.7:4909 , piece_id=BOVTCJYYV72MYZFYWTDLUEBSDT456KODYJ7R5NCEU56AX6VPXCSQ, seg_pos={0 0}, upload=12504]: Uploading piece... Aug 7 21:41:50.633031818 (12h1m50.222376795s) [noise=true , address=45.91.116.4:4939 , piece_id=4MD7JBYAZ5VRTMJEQ7YGYM6765QLRHVHYTIEF6LDIY33WBKOL2GA, seg_pos={0 0}, upload=12504]: Uploading piece... Aug 7 21:41:52.379566135 (12h1m51.968911122s) [upload=12504]: Next returned nil segment Aug 7 21:41:52.379583499 (12h1m51.968928486s) [upload=12504]: Waiting for error group managing segments... Aug 7 21:41:52.728315944 (12h1m52.317660921s) [noise=true , address=104.36.82.8:4970 , piece_id=6GY27L2TPOD2PW2LKDT45RWFF7JQE26FOOLGD2WKQJ6HXS4JY65A, seg_pos={0 0}, upload=12504]: Done uploading piece. err: <nil> Aug 7 21:41:52.758521465 (12h1m52.347866452s) [noise=true , address=104.36.82.2:4731 , piece_id=HRNMAQO6IBTYFPQAZM5CM4ULWGEU5QOE3N6UJVXMV7IA7JULAVVQ, seg_pos={0 0}, upload=12504]: Done uploading piece. err: <nil> Aug 7 21:41:52.837840504 (12h1m52.427185481s) [noise=true , address=104.36.82.11:4115 , piece_id=SY2P7GDDHPSXKK6KSYGKPJD4LA2Y52I53OCDEOQPP4ESOPORHNGQ, seg_pos={0 0}, upload=12504]: Done uploading piece. err: <nil> Aug 7 21:41:52.839003488 (12h1m52.428348475s) [noise=true , address=45.91.116.5:4298 , piece_id=G2FJAO5RNJB7RL7SPUT7MH7GDGJRIWNHH4V2DO7JA4R57FJ4QSTA, seg_pos={0 0}, upload=12504]: Done uploading piece. err: <nil> Aug 7 21:41:52.841339678 (12h1m52.430684665s) [noise=true , address=77.90.31.13:4628 , piece_id=37RNUEEWVPLRQRGGBWFLGT4EQWTSAOXDWCUAMVNIKUXVTGSQRTZA, seg_pos={0 0}, upload=12504]: Done uploading piece. err: <nil> Aug 7 21:41:52.842190817 (12h1m52.431535844s) [noise=true , address=104.36.82.6:4873 , piece_id=ZGGFHDPLWVCJ2FVBKV2O57VWTHFLVAQVNCWVC4ADBN7YUJTXBOOQ, seg_pos={0 0}, upload=12504]: Done uploading piece. err: <nil> Aug 7 21:41:52.870129371 (12h1m52.459474358s) [noise=true , address=77.90.31.19:4841 , piece_id=WPDBTVTA3D5LPPUVIPU2462TBSWADAKHCJK5EDPAZQL6EQ6DKM5Q, seg_pos={0 0}, upload=12504]: Done uploading piece. err: <nil> Aug 7 21:41:52.922791747 (12h1m52.512136764s) [noise=true , address=45.91.116.11:4117 , piece_id=6XLHAB34M3ODXHZTOH42IKLK3LH5YPJYM2COEOZU4NON6JVBL2PA, seg_pos={0 0}, upload=12504]: Done uploading piece. err: <nil> Aug 7 21:41:52.940185306 (12h1m52.529530333s) [noise=true , address=45.91.116.8:4048 , piece_id=BGU6N35D7JRGMREZ6MMSWYH73DS5XYGV3C2JWCXERW7GOJRAWPLA, seg_pos={0 0}, upload=12504]: Done uploading piece. err: <nil> Aug 7 21:41:53.087159767 (12h1m52.676504754s) [noise=true , address=77.90.31.16:4735 , piece_id=7XJ7ZILIPMTSG6W3WFMUCP2LQW76VCXGYJK46XLDGAIRM6A3WM5Q, seg_pos={0 0}, upload=12504]: Done uploading piece. err: <nil> Aug 7 21:41:53.100486544 (12h1m52.689831531s) [noise=true , address=77.90.31.15:4698 , piece_id=5SRHWJG6K2ARPHZQA3VTA7G2CUHOILFQNNRY73DN47DT2JWFOQ5Q, seg_pos={0 0}, upload=12504]: Done uploading piece. err: <nil> Aug 7 21:41:53.379418073 (12h1m52.968763051s) [noise=true , address=45.91.116.10:4083 , piece_id=OMZCRWRNRFR6PJCGJWADFCFFHR6ZF5IH6Y6VB3LBA6JQMAZP2OAQ, seg_pos={0 0}, upload=12504]: Done uploading piece. err: <nil> Aug 7 21:41:53.462051994 (12h1m53.051396971s) [noise=true , address=45.91.116.4:4939 , piece_id=4MD7JBYAZ5VRTMJEQ7YGYM6765QLRHVHYTIEF6LDIY33WBKOL2GA, seg_pos={0 0}, upload=12504]: Done uploading piece. err: <nil> Aug 7 21:41:53.462067704 (12h1m53.051412691s) [seg_pos={0 0}, upload=12504]: Segment reached optimal threshold of 13 pieces. Aug 7 21:41:53.462212441 (12h1m53.051557428s) [noise=true , address=104.36.82.7:4909 , piece_id=BOVTCJYYV72MYZFYWTDLUEBSDT456KODYJ7R5NCEU56AX6VPXCSQ, seg_pos={0 0}, upload=12504]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x): protocol: expected piece hash; context canceled; EOF Aug 7 21:41:53.462341318 (12h1m53.051686295s) [noise=true , address=104.36.82.4:4802 , piece_id=5VAFZVXLCNGU5H6IGBM7V5OEXGDOH2BK3GVNTXSOXMEZLVUA4ALA, seg_pos={0 0}, upload=12504]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4): protocol: write tcp 188.166.225.170:48226->104.36.82.4:4802: use of closed network connection; write tcp 188.166.225.170:48226->104.36.82.4:4802: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48226->104.36.82.4:4802: use of closed network connection Aug 7 21:41:53.462400330 (12h1m53.051745308s) [noise=true , address=77.90.31.18:4807 , piece_id=QMPHP3YXYA2DKNTCZ3OLKLMX5ZYACGRNU5LBD7LIMBUHSKJDQSUA, seg_pos={0 0}, upload=12504]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS): protocol: write tcp 188.166.225.170:57834->77.90.31.18:4807: use of closed network connection; context canceled Aug 7 21:41:53.462410060 (12h1m53.051755047s) [seg_pos={0 0}, upload=12504]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 7 21:41:53.462470285 (12h1m53.051815272s) [upload=12504]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 21:41:53.462476076 (12h1m53.051821063s) [upload=12504]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:42:19.994804113 (12h2m19.58414909s) [upload=12504]: Done uploading segments... Aug 7 21:42:19.994827829 (12h2m19.584172816s) [upload=12504]: Upload finished. err: metaclient: context deadline exceeded Duration: 31.038528988s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 26 out of 60 is failed (threshold=3) [................................XXXXXXXXXXXXXXXXXXXXXXXXXX..] duration: 1m0.000482998s FAIL (2025-08-07 21:42:13 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.000501783s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 6 out of 60 is failed (threshold=3) [...................X.........X......X..X.............X..X...] duration: 9.02098173s FAIL (2025-08-07 21:42:04 UTC) orbiter warning: operation: ListObjects ======ERROR====== unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/4M ======LOGS====== ad=12478]: Done starting upload Aug 7 21:40:13.160345841 (12h0m12.749690828s) [upload=12478]: Uploading segments... Aug 7 21:40:13.160465419 (12h0m12.749810406s) [upload=12478]: Got next segment. Inline: false Aug 7 21:40:13.160488423 (12h0m12.74983341s) [upload=12478]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:40:13.160492521 (12h0m12.749837508s) [upload=12478]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:40:14.024449389 (12h0m13.613794386s) [seg_pos={0 0}, upload=12478]: Begin upload segment... Aug 7 21:40:14.024497139 (12h0m13.613842126s) [seg_pos={0 0}, upload=12478]: Done begin upload segment. Aug 7 21:40:14.024520314 (12h0m13.613865301s) [noise=true , address=104.36.82.12:4163 , piece_id=HCC7QXBZMB3P33M366DZRH2IWVQQVOT56LAWYLY5MW5RARFSISVQ, seg_pos={0 0}, upload=12478]: Uploading piece... Aug 7 21:40:14.024527568 (12h0m13.613872555s) [noise=true , address=104.36.82.12:4177 , piece_id=IPZF77LYNKRXAEDUDGMCUXT4EG5JSIFP3I7E3VPEIN3WNV7XCAKA, seg_pos={0 0}, upload=12478]: Uploading piece... Aug 7 21:40:14.024915139 (12h0m13.614260126s) [noise=true , address=104.36.82.46:4496 , piece_id=FAAS2ISJ3FABRTBOOUGW77IT6A2325ZKX4CSGMHSBRZWSQ4XNOGA, seg_pos={0 0}, upload=12478]: Uploading piece... Aug 7 21:40:14.024946630 (12h0m13.614291617s) [noise=true , address=104.36.82.52:4109 , piece_id=XMAEWF5LMHRJMQAA6FSX6TCYB4OUJ2GVDZQUFDPVFWGXGRQZCZIA, seg_pos={0 0}, upload=12478]: Uploading piece... Aug 7 21:40:14.025230222 (12h0m13.614575199s) [noise=true , address=104.36.82.23:4449 , piece_id=Y52RDGW6QODSWLLQSKBJPSNPUTHB5OLEJHN3Y4GUQ6OZK5XARCRA, seg_pos={0 0}, upload=12478]: Uploading piece... Aug 7 21:40:14.025320585 (12h0m13.614665572s) [noise=true , address=104.36.82.58:4291 , piece_id=246EQHXDOHPPFCKBCQNBTJ46UDD5ZRRPHA7YUUGU22FPJJLKNQ2Q, seg_pos={0 0}, upload=12478]: Uploading piece... Aug 7 21:40:14.025541578 (12h0m13.614886565s) [noise=true , address=64.49.13.170:4721 , piece_id=N7YWI7FRIBKFLUF7TQZLWTT3YTVRDGHUPQJQ2GUP3ENNCZO6DXHQ, seg_pos={0 0}, upload=12478]: Uploading piece... Aug 7 21:40:14.025702406 (12h0m13.615047393s) [noise=true , address=104.36.82.61:4394 , piece_id=2XK3J33NNGFMQU7YDFG35IZ3JFAHUFQ7C3OIUFSNPOSFXB6KEKPQ, seg_pos={0 0}, upload=12478]: Uploading piece... Aug 7 21:40:14.025850619 (12h0m13.615195606s) [noise=true , address=104.36.82.5:4859 , piece_id=3KS2DGR4OLOW4ZUVHSMTVE6XIXZQ3VWOBUXVGTCJTLHNIWGVZOEA, seg_pos={0 0}, upload=12478]: Uploading piece... Aug 7 21:40:14.026035403 (12h0m13.61538039s) [noise=true , address=45.91.116.5:4324 , piece_id=F55DGRECWWVUHFXVMHYRYUHSTO262DXLEIAB2UABJRTIDWG4MEMA, seg_pos={0 0}, upload=12478]: Uploading piece... Aug 7 21:40:14.026287314 (12h0m13.615632301s) [noise=true , address=104.36.82.36:4643 , piece_id=VU7DYF2KI6TB74ERJLYE6BDDQQPMDBSMLZD4H6DKNRKW442ICVGA, seg_pos={0 0}, upload=12478]: Uploading piece... Aug 7 21:40:14.026331439 (12h0m13.615676426s) [noise=true , address=45.91.116.6:4999 , piece_id=CHC3CKW5P6UDSGH2HRU7TMLEEK7A4Q6USGZXSOWVH7SWDBQ6ASDA, seg_pos={0 0}, upload=12478]: Uploading piece... Aug 7 21:40:14.026644838 (12h0m13.615989826s) [noise=true , address=45.91.116.7:4012 , piece_id=FCVPPAJSDQDEMSIP3CJPXF5FQXQ6W3G3CAMR66IOILUY4OQK7P4A, seg_pos={0 0}, upload=12478]: Uploading piece... Aug 7 21:40:14.026679736 (12h0m13.616024713s) [noise=true , address=104.36.82.54:4160 , piece_id=ODKE4XCXDIENJAZZJR7FUE35EM3XNJL3MITY2CFDLZRBBSHQIKXQ, seg_pos={0 0}, upload=12478]: Uploading piece... Aug 7 21:40:14.026890909 (12h0m13.616235896s) [noise=true , address=77.90.31.18:4828 , piece_id=GPXCXC2RW6MRDMLAZJXDML4JO62QAZAZLYZP3JV7J6DQR7QEMTBA, seg_pos={0 0}, upload=12478]: Uploading piece... Aug 7 21:40:14.027132201 (12h0m13.616477178s) [noise=true , address=104.36.82.15:4283 , piece_id=AXCQRAQ6EYNFFHRDCWO6DFLB76ASUYOXWMPMQZYZHHIN6NF3QS5A, seg_pos={0 0}, upload=12478]: Uploading piece... Aug 7 21:40:14.027330621 (12h0m13.616675608s) [noise=true , address=104.36.82.45:4939 , piece_id=BECNPJEALTYQXO4IBONPU27XA2MOFWIP7ATHDJFMYTIG5OVAOHLA, seg_pos={0 0}, upload=12478]: Uploading piece... Aug 7 21:40:14.027577493 (12h0m13.61692248s) [noise=true , address=64.49.13.140:4510 , piece_id=VJJD4RVIWXNIDCZXKEEMBDQD6ZQXPOWS3WKWBAFOOKHBUTJH6CLQ, seg_pos={0 0}, upload=12478]: Uploading piece... Aug 7 21:40:15.144921916 (12h0m14.734266903s) [upload=12478]: Next returned nil segment Aug 7 21:40:15.144943788 (12h0m14.734288775s) [upload=12478]: Waiting for error group managing segments... Aug 7 21:40:15.425370688 (12h0m15.014715665s) [noise=true , address=64.49.13.140:4510 , piece_id=VJJD4RVIWXNIDCZXKEEMBDQD6ZQXPOWS3WKWBAFOOKHBUTJH6CLQ, seg_pos={0 0}, upload=12478]: Done uploading piece. err: <nil> Aug 7 21:40:15.528965431 (12h0m15.118310418s) [noise=true , address=64.49.13.170:4721 , piece_id=N7YWI7FRIBKFLUF7TQZLWTT3YTVRDGHUPQJQ2GUP3ENNCZO6DXHQ, seg_pos={0 0}, upload=12478]: Done uploading piece. err: <nil> Aug 7 21:40:15.945504159 (12h0m15.534849146s) [noise=true , address=104.36.82.46:4496 , piece_id=FAAS2ISJ3FABRTBOOUGW77IT6A2325ZKX4CSGMHSBRZWSQ4XNOGA, seg_pos={0 0}, upload=12478]: Done uploading piece. err: <nil> Aug 7 21:40:15.973785008 (12h0m15.563129995s) [noise=true , address=104.36.82.12:4163 , piece_id=HCC7QXBZMB3P33M366DZRH2IWVQQVOT56LAWYLY5MW5RARFSISVQ, seg_pos={0 0}, upload=12478]: Done uploading piece. err: <nil> Aug 7 21:40:16.002098709 (12h0m15.591443696s) [noise=true , address=104.36.82.52:4109 , piece_id=XMAEWF5LMHRJMQAA6FSX6TCYB4OUJ2GVDZQUFDPVFWGXGRQZCZIA, seg_pos={0 0}, upload=12478]: Done uploading piece. err: <nil> Aug 7 21:40:16.030274577 (12h0m15.619619604s) [noise=true , address=104.36.82.23:4449 , piece_id=Y52RDGW6QODSWLLQSKBJPSNPUTHB5OLEJHN3Y4GUQ6OZK5XARCRA, seg_pos={0 0}, upload=12478]: Done uploading piece. err: <nil> Aug 7 21:40:16.044682273 (12h0m15.63402725s) [noise=true , address=104.36.82.54:4160 , piece_id=ODKE4XCXDIENJAZZJR7FUE35EM3XNJL3MITY2CFDLZRBBSHQIKXQ, seg_pos={0 0}, upload=12478]: Done uploading piece. err: <nil> Aug 7 21:40:16.053282688 (12h0m15.642627665s) [noise=true , address=45.91.116.5:4324 , piece_id=F55DGRECWWVUHFXVMHYRYUHSTO262DXLEIAB2UABJRTIDWG4MEMA, seg_pos={0 0}, upload=12478]: Done uploading piece. err: <nil> Aug 7 21:40:16.058737404 (12h0m15.648082391s) [noise=true , address=104.36.82.58:4291 , piece_id=246EQHXDOHPPFCKBCQNBTJ46UDD5ZRRPHA7YUUGU22FPJJLKNQ2Q, seg_pos={0 0}, upload=12478]: Done uploading piece. err: <nil> Aug 7 21:40:16.093091522 (12h0m15.682436509s) [noise=true , address=104.36.82.12:4177 , piece_id=IPZF77LYNKRXAEDUDGMCUXT4EG5JSIFP3I7E3VPEIN3WNV7XCAKA, seg_pos={0 0}, upload=12478]: Done uploading piece. err: <nil> Aug 7 21:40:16.176548758 (12h0m15.765893735s) [noise=true , address=104.36.82.45:4939 , piece_id=BECNPJEALTYQXO4IBONPU27XA2MOFWIP7ATHDJFMYTIG5OVAOHLA, seg_pos={0 0}, upload=12478]: Done uploading piece. err: <nil> Aug 7 21:40:16.186654263 (12h0m15.77599925s) [noise=true , address=104.36.82.5:4859 , piece_id=3KS2DGR4OLOW4ZUVHSMTVE6XIXZQ3VWOBUXVGTCJTLHNIWGVZOEA, seg_pos={0 0}, upload=12478]: Done uploading piece. err: <nil> Aug 7 21:40:16.255094056 (12h0m15.844439053s) [noise=true , address=45.91.116.7:4012 , piece_id=FCVPPAJSDQDEMSIP3CJPXF5FQXQ6W3G3CAMR66IOILUY4OQK7P4A, seg_pos={0 0}, upload=12478]: Done uploading piece. err: <nil> Aug 7 21:40:16.268089419 (12h0m15.857434406s) [noise=true , address=77.90.31.18:4828 , piece_id=GPXCXC2RW6MRDMLAZJXDML4JO62QAZAZLYZP3JV7J6DQR7QEMTBA, seg_pos={0 0}, upload=12478]: Done uploading piece. err: <nil> Aug 7 21:40:16.268110299 (12h0m15.857455286s) [seg_pos={0 0}, upload=12478]: Segment reached optimal threshold of 14 pieces. Aug 7 21:40:16.268265978 (12h0m15.857610965s) [noise=true , address=104.36.82.61:4394 , piece_id=2XK3J33NNGFMQU7YDFG35IZ3JFAHUFQ7C3OIUFSNPOSFXB6KEKPQ, seg_pos={0 0}, upload=12478]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1dSpTqCxeAcL9PVYbP7BhmuKE7papFPc9swB1Fy83WcFDgQbcF): protocol: expected piece hash; context canceled; EOF Aug 7 21:40:16.268327325 (12h0m15.857672302s) [noise=true , address=45.91.116.6:4999 , piece_id=CHC3CKW5P6UDSGH2HRU7TMLEEK7A4Q6USGZXSOWVH7SWDBQ6ASDA, seg_pos={0 0}, upload=12478]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12evc8vF5f1Z7cPrrC7G7BsGuDWjzttAE2FsQciV7nztBDmrCZB): protocol: expected piece hash; context canceled; EOF Aug 7 21:40:16.268492010 (12h0m15.857836997s) [noise=true , address=104.36.82.36:4643 , piece_id=VU7DYF2KI6TB74ERJLYE6BDDQQPMDBSMLZD4H6DKNRKW442ICVGA, seg_pos={0 0}, upload=12478]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zQsZ4qrfBa95hJYTzjWJJzSea3noCnZ7W22V8ViohjrB5opEw): protocol: write tcp 188.166.225.170:42342->104.36.82.36:4643: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 21:40:16.268605697 (12h0m15.857950684s) [noise=true , address=104.36.82.15:4283 , piece_id=AXCQRAQ6EYNFFHRDCWO6DFLB76ASUYOXWMPMQZYZHHIN6NF3QS5A, seg_pos={0 0}, upload=12478]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ytKjZnmERiZtgJuMJhXG5KFwUQwie27odve4LPE7Pw5hR5ZaU): protocol: write tcp 188.166.225.170:45054->104.36.82.15:4283: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 21:40:16.268677886 (12h0m15.858022873s) [seg_pos={0 0}, upload=12478]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 21:40:16.268787846 (12h0m15.858132843s) [upload=12478]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 21:40:16.268796623 (12h0m15.85814161s) [upload=12478]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:40:17.124513483 (12h0m16.71385847s) [upload=12478]: Done uploading segments... Aug 7 21:40:17.124534484 (12h0m16.713879471s) [upload=12478]: Upload finished. err: <nil> Duration: 9.021019191s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 3 out of 60 is failed (threshold=3) [...............................X...............X..........X.] duration: 4.93084625s FAIL (2025-08-07 21:42:04 UTC) orbiter warning: operation: ListObjects ======ERROR====== unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/1K ======LOGS====== 735125827s) [upload=12376]: Done uploading segments... Aug 7 21:34:02.145826196 (11h54m1.735171183s) [upload=12376]: Upload finished. err: <nil> Aug 7 21:34:47.873603965 (11h54m47.462948952s) [upload=12391]: Starting upload Aug 7 21:34:47.873650053 (11h54m47.46299503s) [upload=12391]: Done starting upload Aug 7 21:34:47.873700940 (11h54m47.463045928s) [upload=12391]: Uploading segments... Aug 7 21:34:47.873733743 (11h54m47.46307872s) [upload=12391]: Got next segment. Inline: true Aug 7 21:34:47.873739335 (11h54m47.463084322s) [upload=12391]: Waiting for error group managing segments... Aug 7 21:34:47.873774331 (11h54m47.463119318s) [upload=12391]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:34:47.873777988 (11h54m47.463122975s) [upload=12391]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:34:47.873780623 (11h54m47.46312561s) [upload=12391]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:34:48.144972113 (11h54m47.7343171s) [upload=12391]: Done uploading segments... Aug 7 21:34:48.144991410 (11h54m47.734336397s) [upload=12391]: Upload finished. err: <nil> Aug 7 21:35:20.363321680 (11h55m19.952666667s) [upload=12399]: Starting upload Aug 7 21:35:20.363369802 (11h55m19.952714789s) [upload=12399]: Done starting upload Aug 7 21:35:20.363434526 (11h55m19.952779513s) [upload=12399]: Uploading segments... Aug 7 21:35:20.363500943 (11h55m19.95284597s) [upload=12399]: Got next segment. Inline: true Aug 7 21:35:20.363506363 (11h55m19.95285135s) [upload=12399]: Waiting for error group managing segments... Aug 7 21:35:20.363545789 (11h55m19.952890776s) [upload=12399]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:35:20.363550498 (11h55m19.952895485s) [upload=12399]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:35:20.363552953 (11h55m19.95289793s) [upload=12399]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:35:20.648472420 (11h55m20.237817407s) [upload=12399]: Done uploading segments... Aug 7 21:35:20.648492398 (11h55m20.237837386s) [upload=12399]: Upload finished. err: <nil> Aug 7 21:35:35.916328511 (11h55m35.505673488s) [upload=12404]: Starting upload Aug 7 21:35:35.916377014 (11h55m35.505722001s) [upload=12404]: Done starting upload Aug 7 21:35:35.916425818 (11h55m35.505770805s) [upload=12404]: Uploading segments... Aug 7 21:35:35.916457979 (11h55m35.505802956s) [upload=12404]: Got next segment. Inline: true Aug 7 21:35:35.916461466 (11h55m35.505806443s) [upload=12404]: Waiting for error group managing segments... Aug 7 21:35:35.916496753 (11h55m35.50584173s) [upload=12404]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:35:35.916501372 (11h55m35.505846359s) [upload=12404]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:35:35.916503666 (11h55m35.505848653s) [upload=12404]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:35:36.531148061 (11h55m36.120493049s) [upload=12404]: Done uploading segments... Aug 7 21:35:36.531167528 (11h55m36.120512505s) [upload=12404]: Upload finished. err: <nil> Aug 7 21:36:48.991080864 (11h56m48.580425851s) [upload=12421]: Starting upload Aug 7 21:36:48.991131251 (11h56m48.580476238s) [upload=12421]: Done starting upload Aug 7 21:36:48.991177239 (11h56m48.580522226s) [upload=12421]: Uploading segments... Aug 7 21:36:48.991205413 (11h56m48.58055039s) [upload=12421]: Got next segment. Inline: true Aug 7 21:36:48.991209060 (11h56m48.580554037s) [upload=12421]: Waiting for error group managing segments... Aug 7 21:36:48.991242914 (11h56m48.580587901s) [upload=12421]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:36:48.991246341 (11h56m48.580591328s) [upload=12421]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:36:48.991248505 (11h56m48.580593492s) [upload=12421]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:36:49.270270137 (11h56m48.859615134s) [upload=12421]: Done uploading segments... Aug 7 21:36:49.270288603 (11h56m48.85963359s) [upload=12421]: Upload finished. err: <nil> Aug 7 21:37:34.116423494 (11h57m33.705768481s) [upload=12435]: Starting upload Aug 7 21:37:34.116478008 (11h57m33.705822996s) [upload=12435]: Done starting upload Aug 7 21:37:34.116546550 (11h57m33.705891537s) [upload=12435]: Uploading segments... Aug 7 21:37:34.116585414 (11h57m33.705930401s) [upload=12435]: Got next segment. Inline: true Aug 7 21:37:34.116589281 (11h57m33.705934268s) [upload=12435]: Waiting for error group managing segments... Aug 7 21:37:34.116631473 (11h57m33.70597646s) [upload=12435]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:37:34.116635470 (11h57m33.705980457s) [upload=12435]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:37:34.116638065 (11h57m33.705983042s) [upload=12435]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:37:34.430160454 (11h57m34.019505441s) [upload=12435]: Done uploading segments... Aug 7 21:37:34.430179780 (11h57m34.019524767s) [upload=12435]: Upload finished. err: <nil> Aug 7 21:38:07.899363525 (11h58m7.488708512s) [upload=12444]: Starting upload Aug 7 21:38:07.899432257 (11h58m7.488777254s) [upload=12444]: Done starting upload Aug 7 21:38:07.899478635 (11h58m7.488823622s) [upload=12444]: Uploading segments... Aug 7 21:38:07.899513011 (11h58m7.488857998s) [upload=12444]: Got next segment. Inline: true Aug 7 21:38:07.899516348 (11h58m7.488861325s) [upload=12444]: Waiting for error group managing segments... Aug 7 21:38:07.899544021 (11h58m7.488888998s) [upload=12444]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:38:07.899547738 (11h58m7.488892725s) [upload=12444]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:38:07.899551905 (11h58m7.488896892s) [upload=12444]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:38:08.175487142 (11h58m7.764832129s) [upload=12444]: Done uploading segments... Aug 7 21:38:08.175506650 (11h58m7.764851637s) [upload=12444]: Upload finished. err: <nil> Aug 7 21:39:12.808926452 (11h59m12.398271439s) [upload=12459]: Starting upload Aug 7 21:39:12.808989873 (11h59m12.39833486s) [upload=12459]: Done starting upload Aug 7 21:39:12.809038286 (11h59m12.398383273s) [upload=12459]: Uploading segments... Aug 7 21:39:12.809068093 (11h59m12.39841308s) [upload=12459]: Got next segment. Inline: true Aug 7 21:39:12.809072080 (11h59m12.398417068s) [upload=12459]: Waiting for error group managing segments... Aug 7 21:39:12.809118720 (11h59m12.398463757s) [upload=12459]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:39:12.809125252 (11h59m12.398470239s) [upload=12459]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:39:12.809127897 (11h59m12.398472884s) [upload=12459]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:39:13.234618442 (11h59m12.823963429s) [upload=12459]: Done uploading segments... Aug 7 21:39:13.234636597 (11h59m12.823981584s) [upload=12459]: Upload finished. err: <nil> Aug 7 21:39:55.593720326 (11h59m55.183065313s) [upload=12472]: Starting upload Aug 7 21:39:55.593767186 (11h59m55.183112163s) [upload=12472]: Done starting upload Aug 7 21:39:55.593842129 (11h59m55.183187117s) [upload=12472]: Uploading segments... Aug 7 21:39:55.593879050 (11h59m55.183224037s) [upload=12472]: Got next segment. Inline: true Aug 7 21:39:55.593882546 (11h59m55.183227533s) [upload=12472]: Waiting for error group managing segments... Aug 7 21:39:55.593915990 (11h59m55.183260978s) [upload=12472]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:39:55.593919928 (11h59m55.183264916s) [upload=12472]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:39:55.593922323 (11h59m55.18326731s) [upload=12472]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:39:55.991096020 (11h59m55.580441007s) [upload=12472]: Done uploading segments... Aug 7 21:39:55.991111470 (11h59m55.580456457s) [upload=12472]: Upload finished. err: <nil> Aug 7 21:40:19.545259354 (12h0m19.134604341s) [upload=12480]: Starting upload Aug 7 21:40:19.545304791 (12h0m19.134649778s) [upload=12480]: Done starting upload Aug 7 21:40:19.545361891 (12h0m19.134706878s) [upload=12480]: Uploading segments... Aug 7 21:40:19.545393461 (12h0m19.134738448s) [upload=12480]: Got next segment. Inline: true Aug 7 21:40:19.545397098 (12h0m19.134742085s) [upload=12480]: Waiting for error group managing segments... Aug 7 21:40:19.545429900 (12h0m19.134774887s) [upload=12480]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:40:19.545433667 (12h0m19.134778655s) [upload=12480]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:40:19.545435852 (12h0m19.134780839s) [upload=12480]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:40:19.891464235 (12h0m19.480809212s) [upload=12480]: Done uploading segments... Aug 7 21:40:19.891480988 (12h0m19.480825965s) [upload=12480]: Upload finished. err: <nil> Aug 7 21:41:23.105250182 (12h1m22.694595169s) [upload=12497]: Starting upload Aug 7 21:41:23.105299878 (12h1m22.694644865s) [upload=12497]: Done starting upload Aug 7 21:41:23.105354182 (12h1m22.694699169s) [upload=12497]: Uploading segments... Aug 7 21:41:23.105392174 (12h1m22.694737161s) [upload=12497]: Got next segment. Inline: true Aug 7 21:41:23.105395991 (12h1m22.694740978s) [upload=12497]: Waiting for error group managing segments... Aug 7 21:41:23.105430738 (12h1m22.694775715s) [upload=12497]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:41:23.105434485 (12h1m22.694779472s) [upload=12497]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:41:23.105438433 (12h1m22.69478342s) [upload=12497]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:41:23.395695219 (12h1m22.985040206s) [upload=12497]: Done uploading segments... Aug 7 21:41:23.395713514 (12h1m22.985058501s) [upload=12497]: Upload finished. err: <nil> Duration: 4.930880245s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 37 out of 60 is failed (threshold=3) [.......................XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.00059985s FAIL (2025-08-07 21:42:03 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.000618585s edge gateway us-select-1
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway) antiblip: 4 out of 60 is failed (threshold=3) [....................................X....X.X...X............] duration: 728.6664ms remote: "tcp:45.38.24.2:443" target: https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410 FAIL (2025-08-07 21:41:55 UTC) orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": context deadline exceeded Duration: 728.6925ms edge gateway vhost us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 32 out of 60 is failed (threshold=3) [.........................X..XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.] duration: 3.901060741s FAIL (2025-08-07 21:41:49 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 3.90108181s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 25 out of 60 is failed (threshold=3) [.................................XXXXXXXXXXXXXXXXXXXXXXXXX..] duration: 17.442782456s FAIL (2025-08-07 21:41:46 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== 2975 (11h51m26.263997962s) [noise=true , address=104.36.82.45:4937 , piece_id=EBUAJKXBK6RZK7T46KXJT5P5LINBFVZCEPGVGG5OZ66SYNOA2RSQ, seg_pos={0 0}, upload=12321]: Done uploading piece. err: <nil> Aug 7 21:31:26.680556480 (11h51m26.269901467s) [noise=true , address=45.91.116.10:4110 , piece_id=F4ZMNLCCFQCBFB6FOPOJFTGPIL5BFYT6TPGL5C7SVLIHMTY4EJJA, seg_pos={0 0}, upload=12321]: Done uploading piece. err: <nil> Aug 7 21:31:26.680580406 (11h51m26.269925393s) [seg_pos={0 0}, upload=12321]: Segment reached optimal threshold of 14 pieces. Aug 7 21:31:26.680752626 (11h51m26.270097613s) [noise=true , address=45.91.116.6:4983 , piece_id=ZWH64KJWCVSTEGVN25WW35HRHTAO3BE3SPKKALNRT7NNLUKB4L4Q, seg_pos={0 0}, upload=12321]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122trJbDHuqeBMQHZT97HP2PgkeuLyUiDiuRm4nayu6ZTH7tat6): protocol: expected piece hash; context canceled; EOF Aug 7 21:31:26.680804515 (11h51m26.270149492s) [noise=true , address=45.91.116.7:4037 , piece_id=WY2F64PFXEGFVE4M6PUHHUYIMTI2KB4CB73DQADZK4QGIEZ2YCYQ, seg_pos={0 0}, upload=12321]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iNPwE2XSfXcaXAXcrH2jcyNGYeDiiWvzAcDQffrmjPwTVqVmK): protocol: expected piece hash; context canceled; EOF Aug 7 21:31:26.680954802 (11h51m26.270299789s) [noise=true , address=104.36.82.61:4375 , piece_id=5LQOQ3MR3EA36DIYLOVO6YDKF6EP2HPHMPG4P6MN76A4IA5BTBEQ, seg_pos={0 0}, upload=12321]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121fzX9v4rgEd78xnDAwcQtthyQHV3yUZUQYLXZ9byPDucGK4T5): protocol: write tcp 188.166.225.170:45604->104.36.82.61:4375: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 21:31:26.680994198 (11h51m26.270339185s) [noise=true , address=104.36.82.8:4977 , piece_id=VY47HC4FF6JU4QQTDVZ62NOIEGWSROTWYMGCB7XXNHJ3GZNJATMQ, seg_pos={0 0}, upload=12321]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WCAm12tXwimFSPqz7Seh6A4oxWqQVxqsF459xux4a5CqjpbAT): protocol: write tcp 188.166.225.170:36156->104.36.82.8:4977: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 21:31:26.681016300 (11h51m26.270361287s) [seg_pos={0 0}, upload=12321]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 21:31:26.681067097 (11h51m26.270412074s) [upload=12321]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 21:31:26.681072137 (11h51m26.270417114s) [upload=12321]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:32:39.816785277 (11h52m39.406130264s) [upload=12321]: Done uploading segments... Aug 7 21:32:39.816815415 (11h52m39.406160402s) [upload=12321]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:32:40.684664778 (11h52m40.274009755s) [upload=12355]: Starting upload Aug 7 21:32:40.684721386 (11h52m40.274066373s) [upload=12355]: Done starting upload Aug 7 21:32:40.684776000 (11h52m40.274121017s) [upload=12355]: Uploading segments... Aug 7 21:32:40.684893314 (11h52m40.274238292s) [upload=12355]: Got next segment. Inline: false Aug 7 21:32:40.684916699 (11h52m40.274261686s) [upload=12355]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:32:40.684920466 (11h52m40.274265443s) [upload=12355]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:33:40.036921304 (11h53m39.626266281s) [upload=12355]: Next segment err: context deadline exceeded Aug 7 21:33:40.037128099 (11h53m39.626473086s) [upload=12355]: Done uploading segments... Aug 7 21:33:40.037148919 (11h53m39.626493896s) [upload=12355]: Upload finished. err: context deadline exceeded Aug 7 21:33:40.899438535 (11h53m40.488783522s) [upload=12372]: Starting upload Aug 7 21:33:40.899490825 (11h53m40.488835832s) [upload=12372]: Done starting upload Aug 7 21:33:40.899590144 (11h53m40.488935131s) [upload=12372]: Uploading segments... Aug 7 21:33:40.899771832 (11h53m40.489116819s) [upload=12372]: Got next segment. Inline: false Aug 7 21:33:40.899795227 (11h53m40.489140214s) [upload=12372]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:33:40.899799275 (11h53m40.489144252s) [upload=12372]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:34:40.257761531 (11h54m39.847106518s) [upload=12372]: Next segment err: context deadline exceeded Aug 7 21:34:40.257885859 (11h54m39.847230856s) [upload=12372]: Done uploading segments... Aug 7 21:34:40.257904695 (11h54m39.847249672s) [upload=12372]: Upload finished. err: context deadline exceeded Aug 7 21:34:41.144124896 (11h54m40.733469873s) [upload=12386]: Starting upload Aug 7 21:34:41.144233103 (11h54m40.73357813s) [upload=12386]: Done starting upload Aug 7 21:34:41.144343113 (11h54m40.733688101s) [upload=12386]: Uploading segments... Aug 7 21:34:41.144700337 (11h54m40.734045364s) [upload=12386]: Got next segment. Inline: false Aug 7 21:34:41.144760612 (11h54m40.734105639s) [upload=12386]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:34:41.144768206 (11h54m40.734113223s) [upload=12386]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:35:40.491924814 (11h55m40.081269811s) [upload=12386]: Next segment err: context deadline exceeded Aug 7 21:35:40.492048891 (11h55m40.081393878s) [upload=12386]: Done uploading segments... Aug 7 21:35:40.492066364 (11h55m40.081411351s) [upload=12386]: Upload finished. err: context deadline exceeded Aug 7 21:35:41.353263670 (11h55m40.942608657s) [upload=12405]: Starting upload Aug 7 21:35:41.353369352 (11h55m40.942714339s) [upload=12405]: Done starting upload Aug 7 21:35:41.353452601 (11h55m40.942797588s) [upload=12405]: Uploading segments... Aug 7 21:35:41.353579473 (11h55m40.942924511s) [upload=12405]: Got next segment. Inline: false Aug 7 21:35:41.353621805 (11h55m40.942966792s) [upload=12405]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:35:41.353626053 (11h55m40.94297104s) [upload=12405]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:36:40.713868241 (11h56m40.303213228s) [upload=12405]: Next segment err: context deadline exceeded Aug 7 21:36:40.713958674 (11h56m40.303303651s) [upload=12405]: Done uploading segments... Aug 7 21:36:40.713972460 (11h56m40.303317447s) [upload=12405]: Upload finished. err: context deadline exceeded Aug 7 21:36:41.596698707 (11h56m41.186043744s) [upload=12418]: Starting upload Aug 7 21:36:41.596808026 (11h56m41.186153063s) [upload=12418]: Done starting upload Aug 7 21:36:41.596885475 (11h56m41.186230462s) [upload=12418]: Uploading segments... Aug 7 21:36:41.597098722 (11h56m41.186443709s) [upload=12418]: Got next segment. Inline: false Aug 7 21:36:41.597141925 (11h56m41.186486912s) [upload=12418]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:36:41.597146604 (11h56m41.186491591s) [upload=12418]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:37:40.947744552 (11h57m40.537089539s) [upload=12418]: Next segment err: context deadline exceeded Aug 7 21:37:40.947825697 (11h57m40.537170684s) [upload=12418]: Done uploading segments... Aug 7 21:37:40.947843592 (11h57m40.537188579s) [upload=12418]: Upload finished. err: context deadline exceeded Aug 7 21:37:41.823629920 (11h57m41.412974907s) [upload=12437]: Starting upload Aug 7 21:37:41.823680908 (11h57m41.413025895s) [upload=12437]: Done starting upload Aug 7 21:37:41.823826305 (11h57m41.413171293s) [upload=12437]: Uploading segments... Aug 7 21:37:41.823857055 (11h57m41.413202052s) [upload=12437]: Got next segment. Inline: false Aug 7 21:37:41.823914255 (11h57m41.413259242s) [upload=12437]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:37:41.823918503 (11h57m41.41326348s) [upload=12437]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:38:41.179029912 (11h58m40.768374899s) [upload=12437]: Next segment err: context deadline exceeded Aug 7 21:38:41.179128490 (11h58m40.768473477s) [upload=12437]: Done uploading segments... Aug 7 21:38:41.179142979 (11h58m40.768487966s) [upload=12437]: Upload finished. err: context deadline exceeded Aug 7 21:38:42.074438453 (11h58m41.66378344s) [upload=12451]: Starting upload Aug 7 21:38:42.074516963 (11h58m41.66386195s) [upload=12451]: Done starting upload Aug 7 21:38:42.074605202 (11h58m41.663950189s) [upload=12451]: Uploading segments... Aug 7 21:38:42.074689222 (11h58m41.664034209s) [upload=12451]: Got next segment. Inline: false Aug 7 21:38:42.074715102 (11h58m41.664060089s) [upload=12451]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:38:42.074718829 (11h58m41.664063816s) [upload=12451]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:39:41.428394360 (11h59m41.017739347s) [upload=12451]: Next segment err: context deadline exceeded Aug 7 21:39:41.428496375 (11h59m41.017841372s) [upload=12451]: Done uploading segments... Aug 7 21:39:41.428519450 (11h59m41.017864437s) [upload=12451]: Upload finished. err: context deadline exceeded Aug 7 21:39:42.313134109 (11h59m41.902479096s) [upload=12468]: Starting upload Aug 7 21:39:42.313191258 (11h59m41.902536245s) [upload=12468]: Done starting upload Aug 7 21:39:42.313256903 (11h59m41.90260189s) [upload=12468]: Uploading segments... Aug 7 21:39:42.313381803 (11h59m41.90272684s) [upload=12468]: Got next segment. Inline: false Aug 7 21:39:42.313472736 (11h59m41.902817763s) [upload=12468]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:39:42.313480300 (11h59m41.902825308s) [upload=12468]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:40:41.669512466 (12h0m41.258857454s) [upload=12468]: Next segment err: context deadline exceeded Aug 7 21:40:41.669582440 (12h0m41.258927427s) [upload=12468]: Done uploading segments... Aug 7 21:40:41.669600465 (12h0m41.258945452s) [upload=12468]: Upload finished. err: context deadline exceeded Duration: 17.442796333s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 32 out of 60 is failed (threshold=3) [..........................X..XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.003760251s FAIL (2025-08-07 21:41:45 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 30.003783534s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 34 out of 60 is failed (threshold=3) [........................X..XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.00128447s FAIL (2025-08-07 21:41:42 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== :38:48.688674213 (11h58m48.278019201s) [noise=true , address=104.36.82.15:4257 , piece_id=U7SFDCW5WBXVOCSJLGFOMXVNIMLQB7CY6BVPA2YUV63LBEC4IWAQ, seg_pos={0 0}, upload=12449]: Uploading piece... Aug 7 21:38:48.689020405 (11h58m48.278365392s) [noise=true , address=104.36.82.13:4186 , piece_id=XY22B7MTHZJ4MJFMJCTVDS4JAYQ3TWQDOPVMBUDZE3EF5VUAQTWQ, seg_pos={0 0}, upload=12449]: Uploading piece... Aug 7 21:38:48.689281786 (11h58m48.278626773s) [noise=true , address=104.36.82.10:4078 , piece_id=XYCM4QHQ7UU2S34Q5DUF7G5YGD2Z7KIDHHUMBRQNBEZUMD4GKNZQ, seg_pos={0 0}, upload=12449]: Uploading piece... Aug 7 21:38:48.689581169 (11h58m48.278926156s) [noise=true , address=45.91.116.5:4299 , piece_id=ED5YULTZVTYLCHQDFVZJIBUSEAYQUJMXGHHB7TTO7GQ5CSBBHYQA, seg_pos={0 0}, upload=12449]: Uploading piece... Aug 7 21:38:48.689893847 (11h58m48.279238824s) [noise=true , address=45.91.116.12:4225 , piece_id=INA2TNZ6KIYF45J4UFGRSLNRELTKISWF7IYIGPNXFDLLGZ7QVJWQ, seg_pos={0 0}, upload=12449]: Uploading piece... Aug 7 21:38:48.690256691 (11h58m48.279601678s) [noise=true , address=104.36.82.11:4114 , piece_id=QVVBKLBIFQC6RRFOJW7MM6AMHL2ZGLQAMH5ZE73XLBGY3VERYEVA, seg_pos={0 0}, upload=12449]: Uploading piece... Aug 7 21:38:48.690626529 (11h58m48.279971516s) [noise=true , address=45.91.116.10:4082 , piece_id=QX3QMQSXP5XBAWE4QY4P5A5QK2QXFBSV67YFPNIQV2BFIH2RVEKQ, seg_pos={0 0}, upload=12449]: Uploading piece... Aug 7 21:38:48.690901335 (11h58m48.280246322s) [noise=true , address=104.36.82.11:4113 , piece_id=2EGFZD54ZFDWU5SP6RQCC4YPP3USNQYBZNFSEI3V7CPMYLLB2A5Q, seg_pos={0 0}, upload=12449]: Uploading piece... Aug 7 21:38:48.691203833 (11h58m48.280548841s) [noise=true , address=77.90.31.12:4592 , piece_id=WW3UFNYQSC4MDLIZ2AJPTLTQN33HWQ2423TRFV5KEUNJ2R6XWKSQ, seg_pos={0 0}, upload=12449]: Uploading piece... Aug 7 21:38:48.691489149 (11h58m48.280834126s) [noise=true , address=77.90.31.21:4879 , piece_id=VB2RZ4HV4VI6WNNSHXZZRHZEAINPP5KU4LO6HVR2FLSGI5H3HP5A, seg_pos={0 0}, upload=12449]: Uploading piece... Aug 7 21:38:48.691733436 (11h58m48.281078424s) [noise=true , address=45.91.116.9:4263 , piece_id=EEBTJKDOQWVNQSWOUI47IY3RHKY5FDAMYQL4LJSUOS6GULR52TGA, seg_pos={0 0}, upload=12449]: Uploading piece... Aug 7 21:38:48.692068177 (11h58m48.281413164s) [noise=true , address=104.36.82.14:4221 , piece_id=W24SNKHE75CCO7XJFHUIH53M26MCEM7ILYZEGH672SQVK6ZCRBLQ, seg_pos={0 0}, upload=12449]: Uploading piece... Aug 7 21:38:50.513228864 (11h58m50.102573851s) [upload=12449]: Next returned nil segment Aug 7 21:38:50.513249563 (11h58m50.10259454s) [upload=12449]: Waiting for error group managing segments... Aug 7 21:38:50.810863968 (11h58m50.400208945s) [noise=true , address=104.36.82.2:4731 , piece_id=EA5DJOHUKB4JMDX56BNKNC3Z3SP5MLHLYN5PTJHOL4PC3DC2YZQA, seg_pos={0 0}, upload=12449]: Done uploading piece. err: <nil> Aug 7 21:38:50.873747961 (11h58m50.463092948s) [noise=true , address=104.36.82.11:4113 , piece_id=2EGFZD54ZFDWU5SP6RQCC4YPP3USNQYBZNFSEI3V7CPMYLLB2A5Q, seg_pos={0 0}, upload=12449]: Done uploading piece. err: <nil> Aug 7 21:38:50.944657320 (11h58m50.534002307s) [noise=true , address=45.91.116.8:4045 , piece_id=JUV7DENMC32B4EJWTW6BJM4TEQRQ2U4FR7IFQAEJMCHRSV5JBXVQ, seg_pos={0 0}, upload=12449]: Done uploading piece. err: <nil> Aug 7 21:38:50.972481134 (11h58m50.561826121s) [noise=true , address=104.36.82.14:4221 , piece_id=W24SNKHE75CCO7XJFHUIH53M26MCEM7ILYZEGH672SQVK6ZCRBLQ, seg_pos={0 0}, upload=12449]: Done uploading piece. err: <nil> Aug 7 21:38:50.980065425 (11h58m50.569410412s) [noise=true , address=104.36.82.13:4186 , piece_id=XY22B7MTHZJ4MJFMJCTVDS4JAYQ3TWQDOPVMBUDZE3EF5VUAQTWQ, seg_pos={0 0}, upload=12449]: Done uploading piece. err: <nil> Aug 7 21:38:50.981707937 (11h58m50.571052925s) [noise=true , address=104.36.82.11:4114 , piece_id=QVVBKLBIFQC6RRFOJW7MM6AMHL2ZGLQAMH5ZE73XLBGY3VERYEVA, seg_pos={0 0}, upload=12449]: Done uploading piece. err: <nil> Aug 7 21:38:51.023162491 (11h58m50.612507508s) [noise=true , address=104.36.82.13:4185 , piece_id=GCSEQS5RYA7DIYXXEYRSVTCZWM4GII3CCOWDZXCWGK73LSPH6VDQ, seg_pos={0 0}, upload=12449]: Done uploading piece. err: <nil> Aug 7 21:38:51.075895351 (11h58m50.665240338s) [noise=true , address=77.90.31.11:4554 , piece_id=Y5X5HMEMMOAD5QMICFFJFWSBHXSVZKTD54AP2NB4I2DJT42KJGVQ, seg_pos={0 0}, upload=12449]: Done uploading piece. err: <nil> Aug 7 21:38:51.095722626 (11h58m50.685067613s) [noise=true , address=45.91.116.5:4299 , piece_id=ED5YULTZVTYLCHQDFVZJIBUSEAYQUJMXGHHB7TTO7GQ5CSBBHYQA, seg_pos={0 0}, upload=12449]: Done uploading piece. err: <nil> Aug 7 21:38:51.098735141 (11h58m50.688080128s) [noise=true , address=104.36.82.15:4257 , piece_id=U7SFDCW5WBXVOCSJLGFOMXVNIMLQB7CY6BVPA2YUV63LBEC4IWAQ, seg_pos={0 0}, upload=12449]: Done uploading piece. err: <nil> Aug 7 21:38:51.107956113 (11h58m50.6973011s) [noise=true , address=77.90.31.21:4879 , piece_id=VB2RZ4HV4VI6WNNSHXZZRHZEAINPP5KU4LO6HVR2FLSGI5H3HP5A, seg_pos={0 0}, upload=12449]: Done uploading piece. err: <nil> Aug 7 21:38:51.133243133 (11h58m50.72258812s) [noise=true , address=77.90.31.12:4592 , piece_id=WW3UFNYQSC4MDLIZ2AJPTLTQN33HWQ2423TRFV5KEUNJ2R6XWKSQ, seg_pos={0 0}, upload=12449]: Done uploading piece. err: <nil> Aug 7 21:38:51.249092953 (11h58m50.83843794s) [noise=true , address=45.91.116.9:4263 , piece_id=EEBTJKDOQWVNQSWOUI47IY3RHKY5FDAMYQL4LJSUOS6GULR52TGA, seg_pos={0 0}, upload=12449]: Done uploading piece. err: <nil> Aug 7 21:38:51.249118061 (11h58m50.838463048s) [seg_pos={0 0}, upload=12449]: Segment reached optimal threshold of 13 pieces. Aug 7 21:38:51.249306762 (11h58m50.838651749s) [noise=true , address=45.91.116.12:4225 , piece_id=INA2TNZ6KIYF45J4UFGRSLNRELTKISWF7IYIGPNXFDLLGZ7QVJWQ, seg_pos={0 0}, upload=12449]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: expected piece hash; context canceled; EOF Aug 7 21:38:51.249323974 (11h58m50.838668961s) [noise=true , address=45.91.116.10:4082 , piece_id=QX3QMQSXP5XBAWE4QY4P5A5QK2QXFBSV67YFPNIQV2BFIH2RVEKQ, seg_pos={0 0}, upload=12449]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Aug 7 21:38:51.249493108 (11h58m50.838838095s) [noise=true , address=104.36.82.10:4078 , piece_id=XYCM4QHQ7UU2S34Q5DUF7G5YGD2Z7KIDHHUMBRQNBEZUMD4GKNZQ, seg_pos={0 0}, upload=12449]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr): protocol: context canceled; context canceled Aug 7 21:38:51.249505242 (11h58m50.838850229s) [seg_pos={0 0}, upload=12449]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 7 21:38:51.249581758 (11h58m50.838926735s) [upload=12449]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 21:38:51.249586157 (11h58m50.838931144s) [upload=12449]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:39:38.936884982 (11h59m38.526229969s) [upload=12449]: Done uploading segments... Aug 7 21:39:38.936908165 (11h59m38.526253152s) [upload=12449]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:39:39.835057740 (11h59m39.424402717s) [upload=12467]: Starting upload Aug 7 21:39:39.835133405 (11h59m39.424478382s) [upload=12467]: Done starting upload Aug 7 21:39:39.835203008 (11h59m39.424547995s) [upload=12467]: Uploading segments... Aug 7 21:39:39.835296918 (11h59m39.424641895s) [upload=12467]: Got next segment. Inline: false Aug 7 21:39:39.835322778 (11h59m39.424667765s) [upload=12467]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:39:39.835327146 (11h59m39.424672133s) [upload=12467]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:40:09.178127344 (12h0m8.767472331s) [upload=12467]: Next segment err: context deadline exceeded Aug 7 21:40:09.178178722 (12h0m8.767523709s) [upload=12467]: Done uploading segments... Aug 7 21:40:09.178192309 (12h0m8.767537296s) [upload=12467]: Upload finished. err: context deadline exceeded Aug 7 21:40:10.047916739 (12h0m9.637261726s) [upload=12477]: Starting upload Aug 7 21:40:10.047969550 (12h0m9.637314537s) [upload=12477]: Done starting upload Aug 7 21:40:10.048040987 (12h0m9.637385984s) [upload=12477]: Uploading segments... Aug 7 21:40:10.048146028 (12h0m9.637491015s) [upload=12477]: Got next segment. Inline: false Aug 7 21:40:10.048205722 (12h0m9.637550709s) [upload=12477]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:40:10.048214258 (12h0m9.637559275s) [upload=12477]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:40:39.404316574 (12h0m38.993661551s) [upload=12477]: Next segment err: context deadline exceeded Aug 7 21:40:39.404376929 (12h0m38.993721907s) [upload=12477]: Done uploading segments... Aug 7 21:40:39.404394894 (12h0m38.993739881s) [upload=12477]: Upload finished. err: context deadline exceeded Aug 7 21:41:12.973177742 (12h1m12.562522779s) [upload=12494]: Starting upload Aug 7 21:41:12.973277582 (12h1m12.562622579s) [upload=12494]: Done starting upload Aug 7 21:41:12.973492313 (12h1m12.5628373s) [upload=12494]: Uploading segments... Aug 7 21:41:12.973522961 (12h1m12.562867939s) [upload=12494]: Got next segment. Inline: false Aug 7 21:41:12.973563950 (12h1m12.562908988s) [upload=12494]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:41:12.973573599 (12h1m12.562918626s) [upload=12494]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:41:42.322571975 (12h1m41.911916962s) [upload=12494]: Next segment err: context deadline exceeded Aug 7 21:41:42.322630547 (12h1m41.911975534s) [upload=12494]: Done uploading segments... Aug 7 21:41:42.322649854 (12h1m41.911994841s) [upload=12494]: Upload finished. err: context deadline exceeded Duration: 30.001303576s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 6 out of 60 is failed (threshold=3) [....................X.........X......X..X.............X..X..] duration: 7.149613783s FAIL (2025-08-07 21:41:41 UTC) orbiter warning: operation: ListObjects ======ERROR====== unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/4M ======LOGS====== ad=12478]: Done starting upload Aug 7 21:40:13.160345841 (12h0m12.749690828s) [upload=12478]: Uploading segments... Aug 7 21:40:13.160465419 (12h0m12.749810406s) [upload=12478]: Got next segment. Inline: false Aug 7 21:40:13.160488423 (12h0m12.74983341s) [upload=12478]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:40:13.160492521 (12h0m12.749837508s) [upload=12478]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:40:14.024449389 (12h0m13.613794386s) [seg_pos={0 0}, upload=12478]: Begin upload segment... Aug 7 21:40:14.024497139 (12h0m13.613842126s) [seg_pos={0 0}, upload=12478]: Done begin upload segment. Aug 7 21:40:14.024520314 (12h0m13.613865301s) [noise=true , address=104.36.82.12:4163 , piece_id=HCC7QXBZMB3P33M366DZRH2IWVQQVOT56LAWYLY5MW5RARFSISVQ, seg_pos={0 0}, upload=12478]: Uploading piece... Aug 7 21:40:14.024527568 (12h0m13.613872555s) [noise=true , address=104.36.82.12:4177 , piece_id=IPZF77LYNKRXAEDUDGMCUXT4EG5JSIFP3I7E3VPEIN3WNV7XCAKA, seg_pos={0 0}, upload=12478]: Uploading piece... Aug 7 21:40:14.024915139 (12h0m13.614260126s) [noise=true , address=104.36.82.46:4496 , piece_id=FAAS2ISJ3FABRTBOOUGW77IT6A2325ZKX4CSGMHSBRZWSQ4XNOGA, seg_pos={0 0}, upload=12478]: Uploading piece... Aug 7 21:40:14.024946630 (12h0m13.614291617s) [noise=true , address=104.36.82.52:4109 , piece_id=XMAEWF5LMHRJMQAA6FSX6TCYB4OUJ2GVDZQUFDPVFWGXGRQZCZIA, seg_pos={0 0}, upload=12478]: Uploading piece... Aug 7 21:40:14.025230222 (12h0m13.614575199s) [noise=true , address=104.36.82.23:4449 , piece_id=Y52RDGW6QODSWLLQSKBJPSNPUTHB5OLEJHN3Y4GUQ6OZK5XARCRA, seg_pos={0 0}, upload=12478]: Uploading piece... Aug 7 21:40:14.025320585 (12h0m13.614665572s) [noise=true , address=104.36.82.58:4291 , piece_id=246EQHXDOHPPFCKBCQNBTJ46UDD5ZRRPHA7YUUGU22FPJJLKNQ2Q, seg_pos={0 0}, upload=12478]: Uploading piece... Aug 7 21:40:14.025541578 (12h0m13.614886565s) [noise=true , address=64.49.13.170:4721 , piece_id=N7YWI7FRIBKFLUF7TQZLWTT3YTVRDGHUPQJQ2GUP3ENNCZO6DXHQ, seg_pos={0 0}, upload=12478]: Uploading piece... Aug 7 21:40:14.025702406 (12h0m13.615047393s) [noise=true , address=104.36.82.61:4394 , piece_id=2XK3J33NNGFMQU7YDFG35IZ3JFAHUFQ7C3OIUFSNPOSFXB6KEKPQ, seg_pos={0 0}, upload=12478]: Uploading piece... Aug 7 21:40:14.025850619 (12h0m13.615195606s) [noise=true , address=104.36.82.5:4859 , piece_id=3KS2DGR4OLOW4ZUVHSMTVE6XIXZQ3VWOBUXVGTCJTLHNIWGVZOEA, seg_pos={0 0}, upload=12478]: Uploading piece... Aug 7 21:40:14.026035403 (12h0m13.61538039s) [noise=true , address=45.91.116.5:4324 , piece_id=F55DGRECWWVUHFXVMHYRYUHSTO262DXLEIAB2UABJRTIDWG4MEMA, seg_pos={0 0}, upload=12478]: Uploading piece... Aug 7 21:40:14.026287314 (12h0m13.615632301s) [noise=true , address=104.36.82.36:4643 , piece_id=VU7DYF2KI6TB74ERJLYE6BDDQQPMDBSMLZD4H6DKNRKW442ICVGA, seg_pos={0 0}, upload=12478]: Uploading piece... Aug 7 21:40:14.026331439 (12h0m13.615676426s) [noise=true , address=45.91.116.6:4999 , piece_id=CHC3CKW5P6UDSGH2HRU7TMLEEK7A4Q6USGZXSOWVH7SWDBQ6ASDA, seg_pos={0 0}, upload=12478]: Uploading piece... Aug 7 21:40:14.026644838 (12h0m13.615989826s) [noise=true , address=45.91.116.7:4012 , piece_id=FCVPPAJSDQDEMSIP3CJPXF5FQXQ6W3G3CAMR66IOILUY4OQK7P4A, seg_pos={0 0}, upload=12478]: Uploading piece... Aug 7 21:40:14.026679736 (12h0m13.616024713s) [noise=true , address=104.36.82.54:4160 , piece_id=ODKE4XCXDIENJAZZJR7FUE35EM3XNJL3MITY2CFDLZRBBSHQIKXQ, seg_pos={0 0}, upload=12478]: Uploading piece... Aug 7 21:40:14.026890909 (12h0m13.616235896s) [noise=true , address=77.90.31.18:4828 , piece_id=GPXCXC2RW6MRDMLAZJXDML4JO62QAZAZLYZP3JV7J6DQR7QEMTBA, seg_pos={0 0}, upload=12478]: Uploading piece... Aug 7 21:40:14.027132201 (12h0m13.616477178s) [noise=true , address=104.36.82.15:4283 , piece_id=AXCQRAQ6EYNFFHRDCWO6DFLB76ASUYOXWMPMQZYZHHIN6NF3QS5A, seg_pos={0 0}, upload=12478]: Uploading piece... Aug 7 21:40:14.027330621 (12h0m13.616675608s) [noise=true , address=104.36.82.45:4939 , piece_id=BECNPJEALTYQXO4IBONPU27XA2MOFWIP7ATHDJFMYTIG5OVAOHLA, seg_pos={0 0}, upload=12478]: Uploading piece... Aug 7 21:40:14.027577493 (12h0m13.61692248s) [noise=true , address=64.49.13.140:4510 , piece_id=VJJD4RVIWXNIDCZXKEEMBDQD6ZQXPOWS3WKWBAFOOKHBUTJH6CLQ, seg_pos={0 0}, upload=12478]: Uploading piece... Aug 7 21:40:15.144921916 (12h0m14.734266903s) [upload=12478]: Next returned nil segment Aug 7 21:40:15.144943788 (12h0m14.734288775s) [upload=12478]: Waiting for error group managing segments... Aug 7 21:40:15.425370688 (12h0m15.014715665s) [noise=true , address=64.49.13.140:4510 , piece_id=VJJD4RVIWXNIDCZXKEEMBDQD6ZQXPOWS3WKWBAFOOKHBUTJH6CLQ, seg_pos={0 0}, upload=12478]: Done uploading piece. err: <nil> Aug 7 21:40:15.528965431 (12h0m15.118310418s) [noise=true , address=64.49.13.170:4721 , piece_id=N7YWI7FRIBKFLUF7TQZLWTT3YTVRDGHUPQJQ2GUP3ENNCZO6DXHQ, seg_pos={0 0}, upload=12478]: Done uploading piece. err: <nil> Aug 7 21:40:15.945504159 (12h0m15.534849146s) [noise=true , address=104.36.82.46:4496 , piece_id=FAAS2ISJ3FABRTBOOUGW77IT6A2325ZKX4CSGMHSBRZWSQ4XNOGA, seg_pos={0 0}, upload=12478]: Done uploading piece. err: <nil> Aug 7 21:40:15.973785008 (12h0m15.563129995s) [noise=true , address=104.36.82.12:4163 , piece_id=HCC7QXBZMB3P33M366DZRH2IWVQQVOT56LAWYLY5MW5RARFSISVQ, seg_pos={0 0}, upload=12478]: Done uploading piece. err: <nil> Aug 7 21:40:16.002098709 (12h0m15.591443696s) [noise=true , address=104.36.82.52:4109 , piece_id=XMAEWF5LMHRJMQAA6FSX6TCYB4OUJ2GVDZQUFDPVFWGXGRQZCZIA, seg_pos={0 0}, upload=12478]: Done uploading piece. err: <nil> Aug 7 21:40:16.030274577 (12h0m15.619619604s) [noise=true , address=104.36.82.23:4449 , piece_id=Y52RDGW6QODSWLLQSKBJPSNPUTHB5OLEJHN3Y4GUQ6OZK5XARCRA, seg_pos={0 0}, upload=12478]: Done uploading piece. err: <nil> Aug 7 21:40:16.044682273 (12h0m15.63402725s) [noise=true , address=104.36.82.54:4160 , piece_id=ODKE4XCXDIENJAZZJR7FUE35EM3XNJL3MITY2CFDLZRBBSHQIKXQ, seg_pos={0 0}, upload=12478]: Done uploading piece. err: <nil> Aug 7 21:40:16.053282688 (12h0m15.642627665s) [noise=true , address=45.91.116.5:4324 , piece_id=F55DGRECWWVUHFXVMHYRYUHSTO262DXLEIAB2UABJRTIDWG4MEMA, seg_pos={0 0}, upload=12478]: Done uploading piece. err: <nil> Aug 7 21:40:16.058737404 (12h0m15.648082391s) [noise=true , address=104.36.82.58:4291 , piece_id=246EQHXDOHPPFCKBCQNBTJ46UDD5ZRRPHA7YUUGU22FPJJLKNQ2Q, seg_pos={0 0}, upload=12478]: Done uploading piece. err: <nil> Aug 7 21:40:16.093091522 (12h0m15.682436509s) [noise=true , address=104.36.82.12:4177 , piece_id=IPZF77LYNKRXAEDUDGMCUXT4EG5JSIFP3I7E3VPEIN3WNV7XCAKA, seg_pos={0 0}, upload=12478]: Done uploading piece. err: <nil> Aug 7 21:40:16.176548758 (12h0m15.765893735s) [noise=true , address=104.36.82.45:4939 , piece_id=BECNPJEALTYQXO4IBONPU27XA2MOFWIP7ATHDJFMYTIG5OVAOHLA, seg_pos={0 0}, upload=12478]: Done uploading piece. err: <nil> Aug 7 21:40:16.186654263 (12h0m15.77599925s) [noise=true , address=104.36.82.5:4859 , piece_id=3KS2DGR4OLOW4ZUVHSMTVE6XIXZQ3VWOBUXVGTCJTLHNIWGVZOEA, seg_pos={0 0}, upload=12478]: Done uploading piece. err: <nil> Aug 7 21:40:16.255094056 (12h0m15.844439053s) [noise=true , address=45.91.116.7:4012 , piece_id=FCVPPAJSDQDEMSIP3CJPXF5FQXQ6W3G3CAMR66IOILUY4OQK7P4A, seg_pos={0 0}, upload=12478]: Done uploading piece. err: <nil> Aug 7 21:40:16.268089419 (12h0m15.857434406s) [noise=true , address=77.90.31.18:4828 , piece_id=GPXCXC2RW6MRDMLAZJXDML4JO62QAZAZLYZP3JV7J6DQR7QEMTBA, seg_pos={0 0}, upload=12478]: Done uploading piece. err: <nil> Aug 7 21:40:16.268110299 (12h0m15.857455286s) [seg_pos={0 0}, upload=12478]: Segment reached optimal threshold of 14 pieces. Aug 7 21:40:16.268265978 (12h0m15.857610965s) [noise=true , address=104.36.82.61:4394 , piece_id=2XK3J33NNGFMQU7YDFG35IZ3JFAHUFQ7C3OIUFSNPOSFXB6KEKPQ, seg_pos={0 0}, upload=12478]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1dSpTqCxeAcL9PVYbP7BhmuKE7papFPc9swB1Fy83WcFDgQbcF): protocol: expected piece hash; context canceled; EOF Aug 7 21:40:16.268327325 (12h0m15.857672302s) [noise=true , address=45.91.116.6:4999 , piece_id=CHC3CKW5P6UDSGH2HRU7TMLEEK7A4Q6USGZXSOWVH7SWDBQ6ASDA, seg_pos={0 0}, upload=12478]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12evc8vF5f1Z7cPrrC7G7BsGuDWjzttAE2FsQciV7nztBDmrCZB): protocol: expected piece hash; context canceled; EOF Aug 7 21:40:16.268492010 (12h0m15.857836997s) [noise=true , address=104.36.82.36:4643 , piece_id=VU7DYF2KI6TB74ERJLYE6BDDQQPMDBSMLZD4H6DKNRKW442ICVGA, seg_pos={0 0}, upload=12478]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zQsZ4qrfBa95hJYTzjWJJzSea3noCnZ7W22V8ViohjrB5opEw): protocol: write tcp 188.166.225.170:42342->104.36.82.36:4643: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 21:40:16.268605697 (12h0m15.857950684s) [noise=true , address=104.36.82.15:4283 , piece_id=AXCQRAQ6EYNFFHRDCWO6DFLB76ASUYOXWMPMQZYZHHIN6NF3QS5A, seg_pos={0 0}, upload=12478]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ytKjZnmERiZtgJuMJhXG5KFwUQwie27odve4LPE7Pw5hR5ZaU): protocol: write tcp 188.166.225.170:45054->104.36.82.15:4283: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 21:40:16.268677886 (12h0m15.858022873s) [seg_pos={0 0}, upload=12478]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 21:40:16.268787846 (12h0m15.858132843s) [upload=12478]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 21:40:16.268796623 (12h0m15.85814161s) [upload=12478]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:40:17.124513483 (12h0m16.71385847s) [upload=12478]: Done uploading segments... Aug 7 21:40:17.124534484 (12h0m16.713879471s) [upload=12478]: Upload finished. err: <nil> Duration: 7.149644059s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway) antiblip: 7 out of 60 is failed (threshold=3) [.............................................XX.XX..X.XX....] duration: 233.213008ms remote: "tcp:207.211.208.129:443" target: https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222 FAIL (2025-08-07 21:41:36 UTC) orbiter warning: Get "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222": context deadline exceeded Duration: 233.236583ms edge gateway vhost bgp us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 28 out of 60 is failed (threshold=3) [...............................X.XXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.00095848s FAIL (2025-08-07 21:41:33 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 1h51m30.058393467s) [upload=12336]: Uploading segments... Aug 7 21:31:30.469142200 (11h51m30.058487187s) [upload=12336]: Got next segment. Inline: true Aug 7 21:31:30.469147400 (11h51m30.058492387s) [upload=12336]: Waiting for error group managing segments... Aug 7 21:31:30.469203888 (11h51m30.058548875s) [upload=12336]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:31:30.469208757 (11h51m30.058553744s) [upload=12336]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:31:30.469211582 (11h51m30.058556569s) [upload=12336]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:32:29.815552389 (11h52m29.404897376s) [upload=12336]: Done uploading segments... Aug 7 21:32:29.815578508 (11h52m29.404923495s) [upload=12336]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:32:30.694545842 (11h52m30.283890829s) [upload=12349]: Starting upload Aug 7 21:32:30.694631607 (11h52m30.283976624s) [upload=12349]: Done starting upload Aug 7 21:32:30.694676312 (11h52m30.284021299s) [upload=12349]: Uploading segments... Aug 7 21:32:30.694717561 (11h52m30.284062538s) [upload=12349]: Got next segment. Inline: true Aug 7 21:32:30.694721909 (11h52m30.284066896s) [upload=12349]: Waiting for error group managing segments... Aug 7 21:32:30.694762737 (11h52m30.284107724s) [upload=12349]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:32:30.694766976 (11h52m30.284111953s) [upload=12349]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:32:30.694769561 (11h52m30.284114548s) [upload=12349]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:33:30.042542113 (11h53m29.6318871s) [upload=12349]: Done uploading segments... Aug 7 21:33:30.042561801 (11h53m29.631906788s) [upload=12349]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:33:30.916211822 (11h53m30.505556809s) [upload=12369]: Starting upload Aug 7 21:33:30.916281525 (11h53m30.505626512s) [upload=12369]: Done starting upload Aug 7 21:33:30.916351519 (11h53m30.505696567s) [upload=12369]: Uploading segments... Aug 7 21:33:30.916406474 (11h53m30.505751502s) [upload=12369]: Got next segment. Inline: true Aug 7 21:33:30.916413598 (11h53m30.505758605s) [upload=12369]: Waiting for error group managing segments... Aug 7 21:33:30.916454065 (11h53m30.505799052s) [upload=12369]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:33:30.916457992 (11h53m30.505802979s) [upload=12369]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:33:30.916460378 (11h53m30.505805355s) [upload=12369]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:34:30.262869110 (11h54m29.852214097s) [upload=12369]: Done uploading segments... Aug 7 21:34:30.262896352 (11h54m29.85224134s) [upload=12369]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:34:31.146607534 (11h54m30.735952521s) [upload=12385]: Starting upload Aug 7 21:34:31.146655144 (11h54m30.736000121s) [upload=12385]: Done starting upload Aug 7 21:34:31.146698838 (11h54m30.736043825s) [upload=12385]: Uploading segments... Aug 7 21:34:31.146741720 (11h54m30.736086697s) [upload=12385]: Got next segment. Inline: true Aug 7 21:34:31.146745548 (11h54m30.736090535s) [upload=12385]: Waiting for error group managing segments... Aug 7 21:34:31.146776246 (11h54m30.736121233s) [upload=12385]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:34:31.146779913 (11h54m30.73612489s) [upload=12385]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:34:31.146782388 (11h54m30.736127375s) [upload=12385]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:35:30.500521706 (11h55m30.089866693s) [upload=12385]: Done uploading segments... Aug 7 21:35:30.500542195 (11h55m30.089887182s) [upload=12385]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:35:31.371815861 (11h55m30.961160848s) [upload=12402]: Starting upload Aug 7 21:35:31.371969656 (11h55m30.961314643s) [upload=12402]: Done starting upload Aug 7 21:35:31.372062443 (11h55m30.96140743s) [upload=12402]: Uploading segments... Aug 7 21:35:31.372108090 (11h55m30.961453077s) [upload=12402]: Got next segment. Inline: true Aug 7 21:35:31.372114412 (11h55m30.961459399s) [upload=12402]: Waiting for error group managing segments... Aug 7 21:35:31.372166783 (11h55m30.96151177s) [upload=12402]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:35:31.372172002 (11h55m30.961516999s) [upload=12402]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:35:31.372174898 (11h55m30.961519885s) [upload=12402]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:36:30.721260917 (11h56m30.310605904s) [upload=12402]: Done uploading segments... Aug 7 21:36:30.721280625 (11h56m30.310625612s) [upload=12402]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:36:33.253873027 (11h56m32.843218014s) [upload=12416]: Starting upload Aug 7 21:36:33.253941188 (11h56m32.843286175s) [upload=12416]: Done starting upload Aug 7 21:36:33.254004108 (11h56m32.843349085s) [upload=12416]: Uploading segments... Aug 7 21:36:33.254039966 (11h56m32.843384953s) [upload=12416]: Got next segment. Inline: true Aug 7 21:36:33.254043462 (11h56m32.84338845s) [upload=12416]: Waiting for error group managing segments... Aug 7 21:36:33.254074321 (11h56m32.843419308s) [upload=12416]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:36:33.254078290 (11h56m32.843423277s) [upload=12416]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:36:33.254080714 (11h56m32.843425701s) [upload=12416]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:37:32.603532648 (11h57m32.192877635s) [upload=12416]: Done uploading segments... Aug 7 21:37:32.603556113 (11h57m32.1929011s) [upload=12416]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:37:33.470049892 (11h57m33.059394869s) [upload=12434]: Starting upload Aug 7 21:37:33.470104877 (11h57m33.059449865s) [upload=12434]: Done starting upload Aug 7 21:37:33.470145565 (11h57m33.059490552s) [upload=12434]: Uploading segments... Aug 7 21:37:33.470187375 (11h57m33.059532352s) [upload=12434]: Got next segment. Inline: true Aug 7 21:37:33.470190872 (11h57m33.059535859s) [upload=12434]: Waiting for error group managing segments... Aug 7 21:37:33.470223234 (11h57m33.059568221s) [upload=12434]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:37:33.470226710 (11h57m33.059571697s) [upload=12434]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:37:33.470228904 (11h57m33.059573881s) [upload=12434]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:38:32.827667037 (11h58m32.417012024s) [upload=12434]: Done uploading segments... Aug 7 21:38:32.827687857 (11h58m32.417032834s) [upload=12434]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:38:33.703666664 (11h58m33.293011651s) [upload=12450]: Starting upload Aug 7 21:38:33.703724295 (11h58m33.293069282s) [upload=12450]: Done starting upload Aug 7 21:38:33.703770663 (11h58m33.29311565s) [upload=12450]: Uploading segments... Aug 7 21:38:33.703817212 (11h58m33.293162199s) [upload=12450]: Got next segment. Inline: true Aug 7 21:38:33.703820899 (11h58m33.293165886s) [upload=12450]: Waiting for error group managing segments... Aug 7 21:38:33.703853411 (11h58m33.293198388s) [upload=12450]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:38:33.703857378 (11h58m33.293202365s) [upload=12450]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:38:33.703859744 (11h58m33.293204731s) [upload=12450]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:39:33.059235349 (11h59m32.648580326s) [upload=12450]: Done uploading segments... Aug 7 21:39:33.059253302 (11h59m32.648598289s) [upload=12450]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:39:33.956833499 (11h59m33.546178486s) [upload=12464]: Starting upload Aug 7 21:39:33.956907240 (11h59m33.546252227s) [upload=12464]: Done starting upload Aug 7 21:39:33.956967195 (11h59m33.546312182s) [upload=12464]: Uploading segments... Aug 7 21:39:33.957011540 (11h59m33.546356527s) [upload=12464]: Got next segment. Inline: true Aug 7 21:39:33.957016229 (11h59m33.546361226s) [upload=12464]: Waiting for error group managing segments... Aug 7 21:39:33.957053820 (11h59m33.546398807s) [upload=12464]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:39:33.957058250 (11h59m33.546403237s) [upload=12464]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:39:33.957061736 (11h59m33.546406723s) [upload=12464]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:40:33.302006486 (12h0m32.891351473s) [upload=12464]: Done uploading segments... Aug 7 21:40:33.302025733 (12h0m32.89137071s) [upload=12464]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:40:34.211725785 (12h0m33.801070772s) [upload=12482]: Starting upload Aug 7 21:40:34.211806970 (12h0m33.801151957s) [upload=12482]: Done starting upload Aug 7 21:40:34.211965303 (12h0m33.8013103s) [upload=12482]: Uploading segments... Aug 7 21:40:34.212066617 (12h0m33.801411614s) [upload=12482]: Got next segment. Inline: true Aug 7 21:40:34.212072819 (12h0m33.801417806s) [upload=12482]: Waiting for error group managing segments... Aug 7 21:40:34.212126081 (12h0m33.801471068s) [upload=12482]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:40:34.212132573 (12h0m33.80147756s) [upload=12482]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:40:34.212136120 (12h0m33.801481107s) [upload=12482]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:41:33.557664583 (12h1m33.14700957s) [upload=12482]: Done uploading segments... Aug 7 21:41:33.557694410 (12h1m33.147039387s) [upload=12482]: Upload finished. err: metaclient: context deadline exceeded Duration: 1m0.000970784s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 35 out of 60 is failed (threshold=3) [........................XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.] duration: 15.720942763s FAIL (2025-08-07 21:41:33 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 15.7209557s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 25 out of 60 is failed (threshold=3) [..................................XXXXXXXXXXXXXXXXXXXXXXXXX.] duration: 46.981870165s FAIL (2025-08-07 21:41:28 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== 2975 (11h51m26.263997962s) [noise=true , address=104.36.82.45:4937 , piece_id=EBUAJKXBK6RZK7T46KXJT5P5LINBFVZCEPGVGG5OZ66SYNOA2RSQ, seg_pos={0 0}, upload=12321]: Done uploading piece. err: <nil> Aug 7 21:31:26.680556480 (11h51m26.269901467s) [noise=true , address=45.91.116.10:4110 , piece_id=F4ZMNLCCFQCBFB6FOPOJFTGPIL5BFYT6TPGL5C7SVLIHMTY4EJJA, seg_pos={0 0}, upload=12321]: Done uploading piece. err: <nil> Aug 7 21:31:26.680580406 (11h51m26.269925393s) [seg_pos={0 0}, upload=12321]: Segment reached optimal threshold of 14 pieces. Aug 7 21:31:26.680752626 (11h51m26.270097613s) [noise=true , address=45.91.116.6:4983 , piece_id=ZWH64KJWCVSTEGVN25WW35HRHTAO3BE3SPKKALNRT7NNLUKB4L4Q, seg_pos={0 0}, upload=12321]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122trJbDHuqeBMQHZT97HP2PgkeuLyUiDiuRm4nayu6ZTH7tat6): protocol: expected piece hash; context canceled; EOF Aug 7 21:31:26.680804515 (11h51m26.270149492s) [noise=true , address=45.91.116.7:4037 , piece_id=WY2F64PFXEGFVE4M6PUHHUYIMTI2KB4CB73DQADZK4QGIEZ2YCYQ, seg_pos={0 0}, upload=12321]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iNPwE2XSfXcaXAXcrH2jcyNGYeDiiWvzAcDQffrmjPwTVqVmK): protocol: expected piece hash; context canceled; EOF Aug 7 21:31:26.680954802 (11h51m26.270299789s) [noise=true , address=104.36.82.61:4375 , piece_id=5LQOQ3MR3EA36DIYLOVO6YDKF6EP2HPHMPG4P6MN76A4IA5BTBEQ, seg_pos={0 0}, upload=12321]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121fzX9v4rgEd78xnDAwcQtthyQHV3yUZUQYLXZ9byPDucGK4T5): protocol: write tcp 188.166.225.170:45604->104.36.82.61:4375: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 21:31:26.680994198 (11h51m26.270339185s) [noise=true , address=104.36.82.8:4977 , piece_id=VY47HC4FF6JU4QQTDVZ62NOIEGWSROTWYMGCB7XXNHJ3GZNJATMQ, seg_pos={0 0}, upload=12321]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WCAm12tXwimFSPqz7Seh6A4oxWqQVxqsF459xux4a5CqjpbAT): protocol: write tcp 188.166.225.170:36156->104.36.82.8:4977: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 21:31:26.681016300 (11h51m26.270361287s) [seg_pos={0 0}, upload=12321]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 21:31:26.681067097 (11h51m26.270412074s) [upload=12321]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 21:31:26.681072137 (11h51m26.270417114s) [upload=12321]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:32:39.816785277 (11h52m39.406130264s) [upload=12321]: Done uploading segments... Aug 7 21:32:39.816815415 (11h52m39.406160402s) [upload=12321]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:32:40.684664778 (11h52m40.274009755s) [upload=12355]: Starting upload Aug 7 21:32:40.684721386 (11h52m40.274066373s) [upload=12355]: Done starting upload Aug 7 21:32:40.684776000 (11h52m40.274121017s) [upload=12355]: Uploading segments... Aug 7 21:32:40.684893314 (11h52m40.274238292s) [upload=12355]: Got next segment. Inline: false Aug 7 21:32:40.684916699 (11h52m40.274261686s) [upload=12355]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:32:40.684920466 (11h52m40.274265443s) [upload=12355]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:33:40.036921304 (11h53m39.626266281s) [upload=12355]: Next segment err: context deadline exceeded Aug 7 21:33:40.037128099 (11h53m39.626473086s) [upload=12355]: Done uploading segments... Aug 7 21:33:40.037148919 (11h53m39.626493896s) [upload=12355]: Upload finished. err: context deadline exceeded Aug 7 21:33:40.899438535 (11h53m40.488783522s) [upload=12372]: Starting upload Aug 7 21:33:40.899490825 (11h53m40.488835832s) [upload=12372]: Done starting upload Aug 7 21:33:40.899590144 (11h53m40.488935131s) [upload=12372]: Uploading segments... Aug 7 21:33:40.899771832 (11h53m40.489116819s) [upload=12372]: Got next segment. Inline: false Aug 7 21:33:40.899795227 (11h53m40.489140214s) [upload=12372]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:33:40.899799275 (11h53m40.489144252s) [upload=12372]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:34:40.257761531 (11h54m39.847106518s) [upload=12372]: Next segment err: context deadline exceeded Aug 7 21:34:40.257885859 (11h54m39.847230856s) [upload=12372]: Done uploading segments... Aug 7 21:34:40.257904695 (11h54m39.847249672s) [upload=12372]: Upload finished. err: context deadline exceeded Aug 7 21:34:41.144124896 (11h54m40.733469873s) [upload=12386]: Starting upload Aug 7 21:34:41.144233103 (11h54m40.73357813s) [upload=12386]: Done starting upload Aug 7 21:34:41.144343113 (11h54m40.733688101s) [upload=12386]: Uploading segments... Aug 7 21:34:41.144700337 (11h54m40.734045364s) [upload=12386]: Got next segment. Inline: false Aug 7 21:34:41.144760612 (11h54m40.734105639s) [upload=12386]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:34:41.144768206 (11h54m40.734113223s) [upload=12386]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:35:40.491924814 (11h55m40.081269811s) [upload=12386]: Next segment err: context deadline exceeded Aug 7 21:35:40.492048891 (11h55m40.081393878s) [upload=12386]: Done uploading segments... Aug 7 21:35:40.492066364 (11h55m40.081411351s) [upload=12386]: Upload finished. err: context deadline exceeded Aug 7 21:35:41.353263670 (11h55m40.942608657s) [upload=12405]: Starting upload Aug 7 21:35:41.353369352 (11h55m40.942714339s) [upload=12405]: Done starting upload Aug 7 21:35:41.353452601 (11h55m40.942797588s) [upload=12405]: Uploading segments... Aug 7 21:35:41.353579473 (11h55m40.942924511s) [upload=12405]: Got next segment. Inline: false Aug 7 21:35:41.353621805 (11h55m40.942966792s) [upload=12405]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:35:41.353626053 (11h55m40.94297104s) [upload=12405]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:36:40.713868241 (11h56m40.303213228s) [upload=12405]: Next segment err: context deadline exceeded Aug 7 21:36:40.713958674 (11h56m40.303303651s) [upload=12405]: Done uploading segments... Aug 7 21:36:40.713972460 (11h56m40.303317447s) [upload=12405]: Upload finished. err: context deadline exceeded Aug 7 21:36:41.596698707 (11h56m41.186043744s) [upload=12418]: Starting upload Aug 7 21:36:41.596808026 (11h56m41.186153063s) [upload=12418]: Done starting upload Aug 7 21:36:41.596885475 (11h56m41.186230462s) [upload=12418]: Uploading segments... Aug 7 21:36:41.597098722 (11h56m41.186443709s) [upload=12418]: Got next segment. Inline: false Aug 7 21:36:41.597141925 (11h56m41.186486912s) [upload=12418]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:36:41.597146604 (11h56m41.186491591s) [upload=12418]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:37:40.947744552 (11h57m40.537089539s) [upload=12418]: Next segment err: context deadline exceeded Aug 7 21:37:40.947825697 (11h57m40.537170684s) [upload=12418]: Done uploading segments... Aug 7 21:37:40.947843592 (11h57m40.537188579s) [upload=12418]: Upload finished. err: context deadline exceeded Aug 7 21:37:41.823629920 (11h57m41.412974907s) [upload=12437]: Starting upload Aug 7 21:37:41.823680908 (11h57m41.413025895s) [upload=12437]: Done starting upload Aug 7 21:37:41.823826305 (11h57m41.413171293s) [upload=12437]: Uploading segments... Aug 7 21:37:41.823857055 (11h57m41.413202052s) [upload=12437]: Got next segment. Inline: false Aug 7 21:37:41.823914255 (11h57m41.413259242s) [upload=12437]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:37:41.823918503 (11h57m41.41326348s) [upload=12437]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:38:41.179029912 (11h58m40.768374899s) [upload=12437]: Next segment err: context deadline exceeded Aug 7 21:38:41.179128490 (11h58m40.768473477s) [upload=12437]: Done uploading segments... Aug 7 21:38:41.179142979 (11h58m40.768487966s) [upload=12437]: Upload finished. err: context deadline exceeded Aug 7 21:38:42.074438453 (11h58m41.66378344s) [upload=12451]: Starting upload Aug 7 21:38:42.074516963 (11h58m41.66386195s) [upload=12451]: Done starting upload Aug 7 21:38:42.074605202 (11h58m41.663950189s) [upload=12451]: Uploading segments... Aug 7 21:38:42.074689222 (11h58m41.664034209s) [upload=12451]: Got next segment. Inline: false Aug 7 21:38:42.074715102 (11h58m41.664060089s) [upload=12451]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:38:42.074718829 (11h58m41.664063816s) [upload=12451]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:39:41.428394360 (11h59m41.017739347s) [upload=12451]: Next segment err: context deadline exceeded Aug 7 21:39:41.428496375 (11h59m41.017841372s) [upload=12451]: Done uploading segments... Aug 7 21:39:41.428519450 (11h59m41.017864437s) [upload=12451]: Upload finished. err: context deadline exceeded Aug 7 21:39:42.313134109 (11h59m41.902479096s) [upload=12468]: Starting upload Aug 7 21:39:42.313191258 (11h59m41.902536245s) [upload=12468]: Done starting upload Aug 7 21:39:42.313256903 (11h59m41.90260189s) [upload=12468]: Uploading segments... Aug 7 21:39:42.313381803 (11h59m41.90272684s) [upload=12468]: Got next segment. Inline: false Aug 7 21:39:42.313472736 (11h59m41.902817763s) [upload=12468]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:39:42.313480300 (11h59m41.902825308s) [upload=12468]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:40:41.669512466 (12h0m41.258857454s) [upload=12468]: Next segment err: context deadline exceeded Aug 7 21:40:41.669582440 (12h0m41.258927427s) [upload=12468]: Done uploading segments... Aug 7 21:40:41.669600465 (12h0m41.258945452s) [upload=12468]: Upload finished. err: context deadline exceeded Duration: 46.981889804s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 3 out of 60 is failed (threshold=3) [................................X...............X..........X] duration: 1.413486929s FAIL (2025-08-07 21:41:23 UTC) orbiter warning: operation: ListObjects ======ERROR====== unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/1K ======LOGS====== 735125827s) [upload=12376]: Done uploading segments... Aug 7 21:34:02.145826196 (11h54m1.735171183s) [upload=12376]: Upload finished. err: <nil> Aug 7 21:34:47.873603965 (11h54m47.462948952s) [upload=12391]: Starting upload Aug 7 21:34:47.873650053 (11h54m47.46299503s) [upload=12391]: Done starting upload Aug 7 21:34:47.873700940 (11h54m47.463045928s) [upload=12391]: Uploading segments... Aug 7 21:34:47.873733743 (11h54m47.46307872s) [upload=12391]: Got next segment. Inline: true Aug 7 21:34:47.873739335 (11h54m47.463084322s) [upload=12391]: Waiting for error group managing segments... Aug 7 21:34:47.873774331 (11h54m47.463119318s) [upload=12391]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:34:47.873777988 (11h54m47.463122975s) [upload=12391]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:34:47.873780623 (11h54m47.46312561s) [upload=12391]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:34:48.144972113 (11h54m47.7343171s) [upload=12391]: Done uploading segments... Aug 7 21:34:48.144991410 (11h54m47.734336397s) [upload=12391]: Upload finished. err: <nil> Aug 7 21:35:20.363321680 (11h55m19.952666667s) [upload=12399]: Starting upload Aug 7 21:35:20.363369802 (11h55m19.952714789s) [upload=12399]: Done starting upload Aug 7 21:35:20.363434526 (11h55m19.952779513s) [upload=12399]: Uploading segments... Aug 7 21:35:20.363500943 (11h55m19.95284597s) [upload=12399]: Got next segment. Inline: true Aug 7 21:35:20.363506363 (11h55m19.95285135s) [upload=12399]: Waiting for error group managing segments... Aug 7 21:35:20.363545789 (11h55m19.952890776s) [upload=12399]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:35:20.363550498 (11h55m19.952895485s) [upload=12399]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:35:20.363552953 (11h55m19.95289793s) [upload=12399]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:35:20.648472420 (11h55m20.237817407s) [upload=12399]: Done uploading segments... Aug 7 21:35:20.648492398 (11h55m20.237837386s) [upload=12399]: Upload finished. err: <nil> Aug 7 21:35:35.916328511 (11h55m35.505673488s) [upload=12404]: Starting upload Aug 7 21:35:35.916377014 (11h55m35.505722001s) [upload=12404]: Done starting upload Aug 7 21:35:35.916425818 (11h55m35.505770805s) [upload=12404]: Uploading segments... Aug 7 21:35:35.916457979 (11h55m35.505802956s) [upload=12404]: Got next segment. Inline: true Aug 7 21:35:35.916461466 (11h55m35.505806443s) [upload=12404]: Waiting for error group managing segments... Aug 7 21:35:35.916496753 (11h55m35.50584173s) [upload=12404]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:35:35.916501372 (11h55m35.505846359s) [upload=12404]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:35:35.916503666 (11h55m35.505848653s) [upload=12404]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:35:36.531148061 (11h55m36.120493049s) [upload=12404]: Done uploading segments... Aug 7 21:35:36.531167528 (11h55m36.120512505s) [upload=12404]: Upload finished. err: <nil> Aug 7 21:36:48.991080864 (11h56m48.580425851s) [upload=12421]: Starting upload Aug 7 21:36:48.991131251 (11h56m48.580476238s) [upload=12421]: Done starting upload Aug 7 21:36:48.991177239 (11h56m48.580522226s) [upload=12421]: Uploading segments... Aug 7 21:36:48.991205413 (11h56m48.58055039s) [upload=12421]: Got next segment. Inline: true Aug 7 21:36:48.991209060 (11h56m48.580554037s) [upload=12421]: Waiting for error group managing segments... Aug 7 21:36:48.991242914 (11h56m48.580587901s) [upload=12421]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:36:48.991246341 (11h56m48.580591328s) [upload=12421]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:36:48.991248505 (11h56m48.580593492s) [upload=12421]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:36:49.270270137 (11h56m48.859615134s) [upload=12421]: Done uploading segments... Aug 7 21:36:49.270288603 (11h56m48.85963359s) [upload=12421]: Upload finished. err: <nil> Aug 7 21:37:34.116423494 (11h57m33.705768481s) [upload=12435]: Starting upload Aug 7 21:37:34.116478008 (11h57m33.705822996s) [upload=12435]: Done starting upload Aug 7 21:37:34.116546550 (11h57m33.705891537s) [upload=12435]: Uploading segments... Aug 7 21:37:34.116585414 (11h57m33.705930401s) [upload=12435]: Got next segment. Inline: true Aug 7 21:37:34.116589281 (11h57m33.705934268s) [upload=12435]: Waiting for error group managing segments... Aug 7 21:37:34.116631473 (11h57m33.70597646s) [upload=12435]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:37:34.116635470 (11h57m33.705980457s) [upload=12435]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:37:34.116638065 (11h57m33.705983042s) [upload=12435]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:37:34.430160454 (11h57m34.019505441s) [upload=12435]: Done uploading segments... Aug 7 21:37:34.430179780 (11h57m34.019524767s) [upload=12435]: Upload finished. err: <nil> Aug 7 21:38:07.899363525 (11h58m7.488708512s) [upload=12444]: Starting upload Aug 7 21:38:07.899432257 (11h58m7.488777254s) [upload=12444]: Done starting upload Aug 7 21:38:07.899478635 (11h58m7.488823622s) [upload=12444]: Uploading segments... Aug 7 21:38:07.899513011 (11h58m7.488857998s) [upload=12444]: Got next segment. Inline: true Aug 7 21:38:07.899516348 (11h58m7.488861325s) [upload=12444]: Waiting for error group managing segments... Aug 7 21:38:07.899544021 (11h58m7.488888998s) [upload=12444]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:38:07.899547738 (11h58m7.488892725s) [upload=12444]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:38:07.899551905 (11h58m7.488896892s) [upload=12444]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:38:08.175487142 (11h58m7.764832129s) [upload=12444]: Done uploading segments... Aug 7 21:38:08.175506650 (11h58m7.764851637s) [upload=12444]: Upload finished. err: <nil> Aug 7 21:39:12.808926452 (11h59m12.398271439s) [upload=12459]: Starting upload Aug 7 21:39:12.808989873 (11h59m12.39833486s) [upload=12459]: Done starting upload Aug 7 21:39:12.809038286 (11h59m12.398383273s) [upload=12459]: Uploading segments... Aug 7 21:39:12.809068093 (11h59m12.39841308s) [upload=12459]: Got next segment. Inline: true Aug 7 21:39:12.809072080 (11h59m12.398417068s) [upload=12459]: Waiting for error group managing segments... Aug 7 21:39:12.809118720 (11h59m12.398463757s) [upload=12459]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:39:12.809125252 (11h59m12.398470239s) [upload=12459]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:39:12.809127897 (11h59m12.398472884s) [upload=12459]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:39:13.234618442 (11h59m12.823963429s) [upload=12459]: Done uploading segments... Aug 7 21:39:13.234636597 (11h59m12.823981584s) [upload=12459]: Upload finished. err: <nil> Aug 7 21:39:55.593720326 (11h59m55.183065313s) [upload=12472]: Starting upload Aug 7 21:39:55.593767186 (11h59m55.183112163s) [upload=12472]: Done starting upload Aug 7 21:39:55.593842129 (11h59m55.183187117s) [upload=12472]: Uploading segments... Aug 7 21:39:55.593879050 (11h59m55.183224037s) [upload=12472]: Got next segment. Inline: true Aug 7 21:39:55.593882546 (11h59m55.183227533s) [upload=12472]: Waiting for error group managing segments... Aug 7 21:39:55.593915990 (11h59m55.183260978s) [upload=12472]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:39:55.593919928 (11h59m55.183264916s) [upload=12472]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:39:55.593922323 (11h59m55.18326731s) [upload=12472]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:39:55.991096020 (11h59m55.580441007s) [upload=12472]: Done uploading segments... Aug 7 21:39:55.991111470 (11h59m55.580456457s) [upload=12472]: Upload finished. err: <nil> Aug 7 21:40:19.545259354 (12h0m19.134604341s) [upload=12480]: Starting upload Aug 7 21:40:19.545304791 (12h0m19.134649778s) [upload=12480]: Done starting upload Aug 7 21:40:19.545361891 (12h0m19.134706878s) [upload=12480]: Uploading segments... Aug 7 21:40:19.545393461 (12h0m19.134738448s) [upload=12480]: Got next segment. Inline: true Aug 7 21:40:19.545397098 (12h0m19.134742085s) [upload=12480]: Waiting for error group managing segments... Aug 7 21:40:19.545429900 (12h0m19.134774887s) [upload=12480]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:40:19.545433667 (12h0m19.134778655s) [upload=12480]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:40:19.545435852 (12h0m19.134780839s) [upload=12480]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:40:19.891464235 (12h0m19.480809212s) [upload=12480]: Done uploading segments... Aug 7 21:40:19.891480988 (12h0m19.480825965s) [upload=12480]: Upload finished. err: <nil> Aug 7 21:41:23.105250182 (12h1m22.694595169s) [upload=12497]: Starting upload Aug 7 21:41:23.105299878 (12h1m22.694644865s) [upload=12497]: Done starting upload Aug 7 21:41:23.105354182 (12h1m22.694699169s) [upload=12497]: Uploading segments... Aug 7 21:41:23.105392174 (12h1m22.694737161s) [upload=12497]: Got next segment. Inline: true Aug 7 21:41:23.105395991 (12h1m22.694740978s) [upload=12497]: Waiting for error group managing segments... Aug 7 21:41:23.105430738 (12h1m22.694775715s) [upload=12497]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:41:23.105434485 (12h1m22.694779472s) [upload=12497]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:41:23.105438433 (12h1m22.69478342s) [upload=12497]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:41:23.395695219 (12h1m22.985040206s) [upload=12497]: Done uploading segments... Aug 7 21:41:23.395713514 (12h1m22.985058501s) [upload=12497]: Upload finished. err: <nil> Duration: 1.413519523s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 35 out of 60 is failed (threshold=3) [.........................XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.001336594s FAIL (2025-08-07 21:41:17 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.00135487s edge gateway us-select-1
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway) antiblip: 4 out of 60 is failed (threshold=3) [.....................................X....X.X...X...........] duration: 9.588086033s remote: "tcp:45.38.24.2:443" target: https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410 FAIL (2025-08-07 21:41:13 UTC) orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": context deadline exceeded Duration: 9.588105251s edge gateway vhost us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 31 out of 60 is failed (threshold=3) [...........................X..XXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.000866522s FAIL (2025-08-07 21:41:10 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 30.000888625s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 6 out of 60 is failed (threshold=3) [.....................X.........X......X..X.............X..X.] duration: 6.954115203s FAIL (2025-08-07 21:41:05 UTC) orbiter warning: operation: ListObjects ======ERROR====== unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/4M ======LOGS====== ad=12478]: Done starting upload Aug 7 21:40:13.160345841 (12h0m12.749690828s) [upload=12478]: Uploading segments... Aug 7 21:40:13.160465419 (12h0m12.749810406s) [upload=12478]: Got next segment. Inline: false Aug 7 21:40:13.160488423 (12h0m12.74983341s) [upload=12478]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:40:13.160492521 (12h0m12.749837508s) [upload=12478]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:40:14.024449389 (12h0m13.613794386s) [seg_pos={0 0}, upload=12478]: Begin upload segment... Aug 7 21:40:14.024497139 (12h0m13.613842126s) [seg_pos={0 0}, upload=12478]: Done begin upload segment. Aug 7 21:40:14.024520314 (12h0m13.613865301s) [noise=true , address=104.36.82.12:4163 , piece_id=HCC7QXBZMB3P33M366DZRH2IWVQQVOT56LAWYLY5MW5RARFSISVQ, seg_pos={0 0}, upload=12478]: Uploading piece... Aug 7 21:40:14.024527568 (12h0m13.613872555s) [noise=true , address=104.36.82.12:4177 , piece_id=IPZF77LYNKRXAEDUDGMCUXT4EG5JSIFP3I7E3VPEIN3WNV7XCAKA, seg_pos={0 0}, upload=12478]: Uploading piece... Aug 7 21:40:14.024915139 (12h0m13.614260126s) [noise=true , address=104.36.82.46:4496 , piece_id=FAAS2ISJ3FABRTBOOUGW77IT6A2325ZKX4CSGMHSBRZWSQ4XNOGA, seg_pos={0 0}, upload=12478]: Uploading piece... Aug 7 21:40:14.024946630 (12h0m13.614291617s) [noise=true , address=104.36.82.52:4109 , piece_id=XMAEWF5LMHRJMQAA6FSX6TCYB4OUJ2GVDZQUFDPVFWGXGRQZCZIA, seg_pos={0 0}, upload=12478]: Uploading piece... Aug 7 21:40:14.025230222 (12h0m13.614575199s) [noise=true , address=104.36.82.23:4449 , piece_id=Y52RDGW6QODSWLLQSKBJPSNPUTHB5OLEJHN3Y4GUQ6OZK5XARCRA, seg_pos={0 0}, upload=12478]: Uploading piece... Aug 7 21:40:14.025320585 (12h0m13.614665572s) [noise=true , address=104.36.82.58:4291 , piece_id=246EQHXDOHPPFCKBCQNBTJ46UDD5ZRRPHA7YUUGU22FPJJLKNQ2Q, seg_pos={0 0}, upload=12478]: Uploading piece... Aug 7 21:40:14.025541578 (12h0m13.614886565s) [noise=true , address=64.49.13.170:4721 , piece_id=N7YWI7FRIBKFLUF7TQZLWTT3YTVRDGHUPQJQ2GUP3ENNCZO6DXHQ, seg_pos={0 0}, upload=12478]: Uploading piece... Aug 7 21:40:14.025702406 (12h0m13.615047393s) [noise=true , address=104.36.82.61:4394 , piece_id=2XK3J33NNGFMQU7YDFG35IZ3JFAHUFQ7C3OIUFSNPOSFXB6KEKPQ, seg_pos={0 0}, upload=12478]: Uploading piece... Aug 7 21:40:14.025850619 (12h0m13.615195606s) [noise=true , address=104.36.82.5:4859 , piece_id=3KS2DGR4OLOW4ZUVHSMTVE6XIXZQ3VWOBUXVGTCJTLHNIWGVZOEA, seg_pos={0 0}, upload=12478]: Uploading piece... Aug 7 21:40:14.026035403 (12h0m13.61538039s) [noise=true , address=45.91.116.5:4324 , piece_id=F55DGRECWWVUHFXVMHYRYUHSTO262DXLEIAB2UABJRTIDWG4MEMA, seg_pos={0 0}, upload=12478]: Uploading piece... Aug 7 21:40:14.026287314 (12h0m13.615632301s) [noise=true , address=104.36.82.36:4643 , piece_id=VU7DYF2KI6TB74ERJLYE6BDDQQPMDBSMLZD4H6DKNRKW442ICVGA, seg_pos={0 0}, upload=12478]: Uploading piece... Aug 7 21:40:14.026331439 (12h0m13.615676426s) [noise=true , address=45.91.116.6:4999 , piece_id=CHC3CKW5P6UDSGH2HRU7TMLEEK7A4Q6USGZXSOWVH7SWDBQ6ASDA, seg_pos={0 0}, upload=12478]: Uploading piece... Aug 7 21:40:14.026644838 (12h0m13.615989826s) [noise=true , address=45.91.116.7:4012 , piece_id=FCVPPAJSDQDEMSIP3CJPXF5FQXQ6W3G3CAMR66IOILUY4OQK7P4A, seg_pos={0 0}, upload=12478]: Uploading piece... Aug 7 21:40:14.026679736 (12h0m13.616024713s) [noise=true , address=104.36.82.54:4160 , piece_id=ODKE4XCXDIENJAZZJR7FUE35EM3XNJL3MITY2CFDLZRBBSHQIKXQ, seg_pos={0 0}, upload=12478]: Uploading piece... Aug 7 21:40:14.026890909 (12h0m13.616235896s) [noise=true , address=77.90.31.18:4828 , piece_id=GPXCXC2RW6MRDMLAZJXDML4JO62QAZAZLYZP3JV7J6DQR7QEMTBA, seg_pos={0 0}, upload=12478]: Uploading piece... Aug 7 21:40:14.027132201 (12h0m13.616477178s) [noise=true , address=104.36.82.15:4283 , piece_id=AXCQRAQ6EYNFFHRDCWO6DFLB76ASUYOXWMPMQZYZHHIN6NF3QS5A, seg_pos={0 0}, upload=12478]: Uploading piece... Aug 7 21:40:14.027330621 (12h0m13.616675608s) [noise=true , address=104.36.82.45:4939 , piece_id=BECNPJEALTYQXO4IBONPU27XA2MOFWIP7ATHDJFMYTIG5OVAOHLA, seg_pos={0 0}, upload=12478]: Uploading piece... Aug 7 21:40:14.027577493 (12h0m13.61692248s) [noise=true , address=64.49.13.140:4510 , piece_id=VJJD4RVIWXNIDCZXKEEMBDQD6ZQXPOWS3WKWBAFOOKHBUTJH6CLQ, seg_pos={0 0}, upload=12478]: Uploading piece... Aug 7 21:40:15.144921916 (12h0m14.734266903s) [upload=12478]: Next returned nil segment Aug 7 21:40:15.144943788 (12h0m14.734288775s) [upload=12478]: Waiting for error group managing segments... Aug 7 21:40:15.425370688 (12h0m15.014715665s) [noise=true , address=64.49.13.140:4510 , piece_id=VJJD4RVIWXNIDCZXKEEMBDQD6ZQXPOWS3WKWBAFOOKHBUTJH6CLQ, seg_pos={0 0}, upload=12478]: Done uploading piece. err: <nil> Aug 7 21:40:15.528965431 (12h0m15.118310418s) [noise=true , address=64.49.13.170:4721 , piece_id=N7YWI7FRIBKFLUF7TQZLWTT3YTVRDGHUPQJQ2GUP3ENNCZO6DXHQ, seg_pos={0 0}, upload=12478]: Done uploading piece. err: <nil> Aug 7 21:40:15.945504159 (12h0m15.534849146s) [noise=true , address=104.36.82.46:4496 , piece_id=FAAS2ISJ3FABRTBOOUGW77IT6A2325ZKX4CSGMHSBRZWSQ4XNOGA, seg_pos={0 0}, upload=12478]: Done uploading piece. err: <nil> Aug 7 21:40:15.973785008 (12h0m15.563129995s) [noise=true , address=104.36.82.12:4163 , piece_id=HCC7QXBZMB3P33M366DZRH2IWVQQVOT56LAWYLY5MW5RARFSISVQ, seg_pos={0 0}, upload=12478]: Done uploading piece. err: <nil> Aug 7 21:40:16.002098709 (12h0m15.591443696s) [noise=true , address=104.36.82.52:4109 , piece_id=XMAEWF5LMHRJMQAA6FSX6TCYB4OUJ2GVDZQUFDPVFWGXGRQZCZIA, seg_pos={0 0}, upload=12478]: Done uploading piece. err: <nil> Aug 7 21:40:16.030274577 (12h0m15.619619604s) [noise=true , address=104.36.82.23:4449 , piece_id=Y52RDGW6QODSWLLQSKBJPSNPUTHB5OLEJHN3Y4GUQ6OZK5XARCRA, seg_pos={0 0}, upload=12478]: Done uploading piece. err: <nil> Aug 7 21:40:16.044682273 (12h0m15.63402725s) [noise=true , address=104.36.82.54:4160 , piece_id=ODKE4XCXDIENJAZZJR7FUE35EM3XNJL3MITY2CFDLZRBBSHQIKXQ, seg_pos={0 0}, upload=12478]: Done uploading piece. err: <nil> Aug 7 21:40:16.053282688 (12h0m15.642627665s) [noise=true , address=45.91.116.5:4324 , piece_id=F55DGRECWWVUHFXVMHYRYUHSTO262DXLEIAB2UABJRTIDWG4MEMA, seg_pos={0 0}, upload=12478]: Done uploading piece. err: <nil> Aug 7 21:40:16.058737404 (12h0m15.648082391s) [noise=true , address=104.36.82.58:4291 , piece_id=246EQHXDOHPPFCKBCQNBTJ46UDD5ZRRPHA7YUUGU22FPJJLKNQ2Q, seg_pos={0 0}, upload=12478]: Done uploading piece. err: <nil> Aug 7 21:40:16.093091522 (12h0m15.682436509s) [noise=true , address=104.36.82.12:4177 , piece_id=IPZF77LYNKRXAEDUDGMCUXT4EG5JSIFP3I7E3VPEIN3WNV7XCAKA, seg_pos={0 0}, upload=12478]: Done uploading piece. err: <nil> Aug 7 21:40:16.176548758 (12h0m15.765893735s) [noise=true , address=104.36.82.45:4939 , piece_id=BECNPJEALTYQXO4IBONPU27XA2MOFWIP7ATHDJFMYTIG5OVAOHLA, seg_pos={0 0}, upload=12478]: Done uploading piece. err: <nil> Aug 7 21:40:16.186654263 (12h0m15.77599925s) [noise=true , address=104.36.82.5:4859 , piece_id=3KS2DGR4OLOW4ZUVHSMTVE6XIXZQ3VWOBUXVGTCJTLHNIWGVZOEA, seg_pos={0 0}, upload=12478]: Done uploading piece. err: <nil> Aug 7 21:40:16.255094056 (12h0m15.844439053s) [noise=true , address=45.91.116.7:4012 , piece_id=FCVPPAJSDQDEMSIP3CJPXF5FQXQ6W3G3CAMR66IOILUY4OQK7P4A, seg_pos={0 0}, upload=12478]: Done uploading piece. err: <nil> Aug 7 21:40:16.268089419 (12h0m15.857434406s) [noise=true , address=77.90.31.18:4828 , piece_id=GPXCXC2RW6MRDMLAZJXDML4JO62QAZAZLYZP3JV7J6DQR7QEMTBA, seg_pos={0 0}, upload=12478]: Done uploading piece. err: <nil> Aug 7 21:40:16.268110299 (12h0m15.857455286s) [seg_pos={0 0}, upload=12478]: Segment reached optimal threshold of 14 pieces. Aug 7 21:40:16.268265978 (12h0m15.857610965s) [noise=true , address=104.36.82.61:4394 , piece_id=2XK3J33NNGFMQU7YDFG35IZ3JFAHUFQ7C3OIUFSNPOSFXB6KEKPQ, seg_pos={0 0}, upload=12478]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1dSpTqCxeAcL9PVYbP7BhmuKE7papFPc9swB1Fy83WcFDgQbcF): protocol: expected piece hash; context canceled; EOF Aug 7 21:40:16.268327325 (12h0m15.857672302s) [noise=true , address=45.91.116.6:4999 , piece_id=CHC3CKW5P6UDSGH2HRU7TMLEEK7A4Q6USGZXSOWVH7SWDBQ6ASDA, seg_pos={0 0}, upload=12478]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12evc8vF5f1Z7cPrrC7G7BsGuDWjzttAE2FsQciV7nztBDmrCZB): protocol: expected piece hash; context canceled; EOF Aug 7 21:40:16.268492010 (12h0m15.857836997s) [noise=true , address=104.36.82.36:4643 , piece_id=VU7DYF2KI6TB74ERJLYE6BDDQQPMDBSMLZD4H6DKNRKW442ICVGA, seg_pos={0 0}, upload=12478]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zQsZ4qrfBa95hJYTzjWJJzSea3noCnZ7W22V8ViohjrB5opEw): protocol: write tcp 188.166.225.170:42342->104.36.82.36:4643: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 21:40:16.268605697 (12h0m15.857950684s) [noise=true , address=104.36.82.15:4283 , piece_id=AXCQRAQ6EYNFFHRDCWO6DFLB76ASUYOXWMPMQZYZHHIN6NF3QS5A, seg_pos={0 0}, upload=12478]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ytKjZnmERiZtgJuMJhXG5KFwUQwie27odve4LPE7Pw5hR5ZaU): protocol: write tcp 188.166.225.170:45054->104.36.82.15:4283: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 21:40:16.268677886 (12h0m15.858022873s) [seg_pos={0 0}, upload=12478]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 21:40:16.268787846 (12h0m15.858132843s) [upload=12478]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 21:40:16.268796623 (12h0m15.85814161s) [upload=12478]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:40:17.124513483 (12h0m16.71385847s) [upload=12478]: Done uploading segments... Aug 7 21:40:17.124534484 (12h0m16.713879471s) [upload=12478]: Upload finished. err: <nil> Duration: 6.95414478s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 26 out of 60 is failed (threshold=3) [.................................XXXXXXXXXXXXXXXXXXXXXXXXXX.] duration: 53.310199853s FAIL (2025-08-07 21:41:04 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 53.310215032s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway) antiblip: 4 out of 60 is failed (threshold=3) [......................................X....X.X...X..........] duration: 1m28.64132709s remote: "tcp:45.38.24.2:443" target: https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410 FAIL (2025-08-07 21:41:03 UTC) orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": context deadline exceeded Duration: 1m28.641355554s edge gateway vhost us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 31 out of 60 is failed (threshold=3) [............................XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.] duration: 16.599666552s FAIL (2025-08-07 21:41:02 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 1:31:10.105626315 (11h51m9.694971302s) [upload=12327]: Uploading segments... Aug 7 21:31:10.105718581 (11h51m9.695063568s) [upload=12327]: Got next segment. Inline: true Aug 7 21:31:10.105723311 (11h51m9.695068298s) [upload=12327]: Waiting for error group managing segments... Aug 7 21:31:10.105777513 (11h51m9.69512249s) [upload=12327]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:31:10.105782744 (11h51m9.695127731s) [upload=12327]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:31:10.105785079 (11h51m9.695130066s) [upload=12327]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:31:39.460569760 (11h51m39.049914747s) [upload=12327]: Done uploading segments... Aug 7 21:31:39.460597573 (11h51m39.04994255s) [upload=12327]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:32:24.324148762 (11h52m23.913493749s) [upload=12347]: Starting upload Aug 7 21:32:24.324206403 (11h52m23.91355139s) [upload=12347]: Done starting upload Aug 7 21:32:24.324252100 (11h52m23.913597087s) [upload=12347]: Uploading segments... Aug 7 21:32:24.324280415 (11h52m23.913625402s) [upload=12347]: Got next segment. Inline: true Aug 7 21:32:24.324284021 (11h52m23.913628998s) [upload=12347]: Waiting for error group managing segments... Aug 7 21:32:24.324329689 (11h52m23.913674686s) [upload=12347]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:32:24.324336372 (11h52m23.913681379s) [upload=12347]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:32:24.324341050 (11h52m23.913686057s) [upload=12347]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:32:53.678084170 (11h52m53.267429207s) [upload=12347]: Done uploading segments... Aug 7 21:32:53.678107405 (11h52m53.267452392s) [upload=12347]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:33:17.618427342 (11h53m17.207772389s) [upload=12364]: Starting upload Aug 7 21:33:17.618576718 (11h53m17.207921745s) [upload=12364]: Done starting upload Aug 7 21:33:17.618658725 (11h53m17.208003742s) [upload=12364]: Uploading segments... Aug 7 21:33:17.618713580 (11h53m17.208058597s) [upload=12364]: Got next segment. Inline: true Aug 7 21:33:17.618720533 (11h53m17.20806553s) [upload=12364]: Waiting for error group managing segments... Aug 7 21:33:17.618772433 (11h53m17.20811743s) [upload=12364]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:33:17.618779636 (11h53m17.208124643s) [upload=12364]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:33:17.618784776 (11h53m17.208129793s) [upload=12364]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:33:46.957323530 (11h53m46.546668577s) [upload=12364]: Done uploading segments... Aug 7 21:33:46.957345773 (11h53m46.54669075s) [upload=12364]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:34:19.197336641 (11h54m18.786681628s) [upload=12382]: Starting upload Aug 7 21:34:19.197461931 (11h54m18.786806918s) [upload=12382]: Done starting upload Aug 7 21:34:19.197512228 (11h54m18.786857225s) [upload=12382]: Uploading segments... Aug 7 21:34:19.197561282 (11h54m18.786906269s) [upload=12382]: Got next segment. Inline: true Aug 7 21:34:19.197565189 (11h54m18.786910166s) [upload=12382]: Waiting for error group managing segments... Aug 7 21:34:19.197610725 (11h54m18.786955723s) [upload=12382]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:34:19.197615966 (11h54m18.786960953s) [upload=12382]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:34:19.197619142 (11h54m18.786964139s) [upload=12382]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:34:48.535954936 (11h54m48.125299923s) [upload=12382]: Done uploading segments... Aug 7 21:34:48.535978650 (11h54m48.125323637s) [upload=12382]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:35:05.991757765 (11h55m5.581102752s) [upload=12393]: Starting upload Aug 7 21:35:05.991822780 (11h55m5.581167767s) [upload=12393]: Done starting upload Aug 7 21:35:05.991946456 (11h55m5.581291443s) [upload=12393]: Uploading segments... Aug 7 21:35:05.991988116 (11h55m5.581333104s) [upload=12393]: Got next segment. Inline: true Aug 7 21:35:05.991994929 (11h55m5.581339996s) [upload=12393]: Waiting for error group managing segments... Aug 7 21:35:05.992042741 (11h55m5.581387728s) [upload=12393]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:35:05.992048021 (11h55m5.581393018s) [upload=12393]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:35:05.992051768 (11h55m5.581396755s) [upload=12393]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:35:35.339368736 (11h55m34.928713723s) [upload=12393]: Done uploading segments... Aug 7 21:35:35.339503534 (11h55m34.928848522s) [upload=12393]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:36:12.952326182 (11h56m12.541671159s) [upload=12411]: Starting upload Aug 7 21:36:12.952385075 (11h56m12.541730052s) [upload=12411]: Done starting upload Aug 7 21:36:12.952428238 (11h56m12.541773225s) [upload=12411]: Uploading segments... Aug 7 21:36:12.952455510 (11h56m12.541800497s) [upload=12411]: Got next segment. Inline: true Aug 7 21:36:12.952459417 (11h56m12.541804404s) [upload=12411]: Waiting for error group managing segments... Aug 7 21:36:12.952507650 (11h56m12.541852637s) [upload=12411]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:36:12.952514352 (11h56m12.541859349s) [upload=12411]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:36:12.952518550 (11h56m12.541863537s) [upload=12411]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:36:42.299158316 (11h56m41.888503304s) [upload=12411]: Done uploading segments... Aug 7 21:36:42.299175039 (11h56m41.888520026s) [upload=12411]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:37:22.572699237 (11h57m22.162044224s) [upload=12432]: Starting upload Aug 7 21:37:22.572782937 (11h57m22.162127924s) [upload=12432]: Done starting upload Aug 7 21:37:22.572842260 (11h57m22.162187247s) [upload=12432]: Uploading segments... Aug 7 21:37:22.572885313 (11h57m22.1622303s) [upload=12432]: Got next segment. Inline: true Aug 7 21:37:22.572889591 (11h57m22.162234578s) [upload=12432]: Waiting for error group managing segments... Aug 7 21:37:22.572930138 (11h57m22.162275125s) [upload=12432]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:37:22.572934296 (11h57m22.162279283s) [upload=12432]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:37:22.572937121 (11h57m22.162282118s) [upload=12432]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:37:51.933954276 (11h57m51.523299253s) [upload=12432]: Done uploading segments... Aug 7 21:37:51.933970927 (11h57m51.523315914s) [upload=12432]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:38:28.745940745 (11h58m28.335285732s) [upload=12448]: Starting upload Aug 7 21:38:28.745997504 (11h58m28.335342491s) [upload=12448]: Done starting upload Aug 7 21:38:28.746040176 (11h58m28.335385163s) [upload=12448]: Uploading segments... Aug 7 21:38:28.746078458 (11h58m28.335423446s) [upload=12448]: Got next segment. Inline: true Aug 7 21:38:28.746081766 (11h58m28.335426743s) [upload=12448]: Waiting for error group managing segments... Aug 7 21:38:28.746111923 (11h58m28.33545691s) [upload=12448]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:38:28.746115540 (11h58m28.335460527s) [upload=12448]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:38:28.746118205 (11h58m28.335463192s) [upload=12448]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:38:58.105315693 (11h58m57.694660691s) [upload=12448]: Done uploading segments... Aug 7 21:38:58.105342465 (11h58m57.694687452s) [upload=12448]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:39:27.137238449 (11h59m26.726583436s) [upload=12462]: Starting upload Aug 7 21:39:27.137294907 (11h59m26.726639894s) [upload=12462]: Done starting upload Aug 7 21:39:27.137337368 (11h59m26.726682355s) [upload=12462]: Uploading segments... Aug 7 21:39:27.137366043 (11h59m26.72671103s) [upload=12462]: Got next segment. Inline: true Aug 7 21:39:27.137370061 (11h59m26.726715048s) [upload=12462]: Waiting for error group managing segments... Aug 7 21:39:27.137401901 (11h59m26.726746879s) [upload=12462]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:39:27.137405518 (11h59m26.726750495s) [upload=12462]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:39:27.137407813 (11h59m26.72675279s) [upload=12462]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:39:56.495186447 (11h59m56.084531434s) [upload=12462]: Done uploading segments... Aug 7 21:39:56.495206415 (11h59m56.084551402s) [upload=12462]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:40:01.914615953 (12h0m1.50396094s) [upload=12475]: Starting upload Aug 7 21:40:01.914684295 (12h0m1.504029282s) [upload=12475]: Done starting upload Aug 7 21:40:01.914741163 (12h0m1.50408616s) [upload=12475]: Uploading segments... Aug 7 21:40:01.914783103 (12h0m1.50412809s) [upload=12475]: Got next segment. Inline: true Aug 7 21:40:01.914788143 (12h0m1.504133131s) [upload=12475]: Waiting for error group managing segments... Aug 7 21:40:01.914833950 (12h0m1.504178937s) [upload=12475]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:40:01.914840073 (12h0m1.50418507s) [upload=12475]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:40:01.914846014 (12h0m1.504191011s) [upload=12475]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:40:31.268196986 (12h0m30.857542023s) [upload=12475]: Done uploading segments... Aug 7 21:40:31.268223536 (12h0m30.857568533s) [upload=12475]: Upload finished. err: metaclient: context deadline exceeded Duration: 16.599699827s core uplink us-select-1
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway) antiblip: 7 out of 60 is failed (threshold=3) [..............................................XX.XX..X.XX...] duration: 7.954793048s remote: "tcp:207.211.208.129:443" target: https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222 FAIL (2025-08-07 21:41:02 UTC) orbiter warning: Get "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222": context deadline exceeded Duration: 7.954814027s edge gateway vhost bgp us1
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway) antiblip: 7 out of 60 is failed (threshold=3) [...............................................XX.XX..X.XX..] duration: 1m46.418899478s remote: "tcp:207.211.208.129:443" target: https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222 FAIL (2025-08-07 21:40:54 UTC) orbiter warning: Get "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222": context deadline exceeded Duration: 1m46.418924317s edge gateway vhost bgp us1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 36 out of 60 is failed (threshold=3) [........................XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.000668223s FAIL (2025-08-07 21:40:54 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.000697991s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 25 out of 60 is failed (threshold=3) [...................................XXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.001172137s FAIL (2025-08-07 21:40:41 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== 2975 (11h51m26.263997962s) [noise=true , address=104.36.82.45:4937 , piece_id=EBUAJKXBK6RZK7T46KXJT5P5LINBFVZCEPGVGG5OZ66SYNOA2RSQ, seg_pos={0 0}, upload=12321]: Done uploading piece. err: <nil> Aug 7 21:31:26.680556480 (11h51m26.269901467s) [noise=true , address=45.91.116.10:4110 , piece_id=F4ZMNLCCFQCBFB6FOPOJFTGPIL5BFYT6TPGL5C7SVLIHMTY4EJJA, seg_pos={0 0}, upload=12321]: Done uploading piece. err: <nil> Aug 7 21:31:26.680580406 (11h51m26.269925393s) [seg_pos={0 0}, upload=12321]: Segment reached optimal threshold of 14 pieces. Aug 7 21:31:26.680752626 (11h51m26.270097613s) [noise=true , address=45.91.116.6:4983 , piece_id=ZWH64KJWCVSTEGVN25WW35HRHTAO3BE3SPKKALNRT7NNLUKB4L4Q, seg_pos={0 0}, upload=12321]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122trJbDHuqeBMQHZT97HP2PgkeuLyUiDiuRm4nayu6ZTH7tat6): protocol: expected piece hash; context canceled; EOF Aug 7 21:31:26.680804515 (11h51m26.270149492s) [noise=true , address=45.91.116.7:4037 , piece_id=WY2F64PFXEGFVE4M6PUHHUYIMTI2KB4CB73DQADZK4QGIEZ2YCYQ, seg_pos={0 0}, upload=12321]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iNPwE2XSfXcaXAXcrH2jcyNGYeDiiWvzAcDQffrmjPwTVqVmK): protocol: expected piece hash; context canceled; EOF Aug 7 21:31:26.680954802 (11h51m26.270299789s) [noise=true , address=104.36.82.61:4375 , piece_id=5LQOQ3MR3EA36DIYLOVO6YDKF6EP2HPHMPG4P6MN76A4IA5BTBEQ, seg_pos={0 0}, upload=12321]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121fzX9v4rgEd78xnDAwcQtthyQHV3yUZUQYLXZ9byPDucGK4T5): protocol: write tcp 188.166.225.170:45604->104.36.82.61:4375: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 21:31:26.680994198 (11h51m26.270339185s) [noise=true , address=104.36.82.8:4977 , piece_id=VY47HC4FF6JU4QQTDVZ62NOIEGWSROTWYMGCB7XXNHJ3GZNJATMQ, seg_pos={0 0}, upload=12321]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WCAm12tXwimFSPqz7Seh6A4oxWqQVxqsF459xux4a5CqjpbAT): protocol: write tcp 188.166.225.170:36156->104.36.82.8:4977: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 21:31:26.681016300 (11h51m26.270361287s) [seg_pos={0 0}, upload=12321]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 21:31:26.681067097 (11h51m26.270412074s) [upload=12321]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 21:31:26.681072137 (11h51m26.270417114s) [upload=12321]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:32:39.816785277 (11h52m39.406130264s) [upload=12321]: Done uploading segments... Aug 7 21:32:39.816815415 (11h52m39.406160402s) [upload=12321]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:32:40.684664778 (11h52m40.274009755s) [upload=12355]: Starting upload Aug 7 21:32:40.684721386 (11h52m40.274066373s) [upload=12355]: Done starting upload Aug 7 21:32:40.684776000 (11h52m40.274121017s) [upload=12355]: Uploading segments... Aug 7 21:32:40.684893314 (11h52m40.274238292s) [upload=12355]: Got next segment. Inline: false Aug 7 21:32:40.684916699 (11h52m40.274261686s) [upload=12355]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:32:40.684920466 (11h52m40.274265443s) [upload=12355]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:33:40.036921304 (11h53m39.626266281s) [upload=12355]: Next segment err: context deadline exceeded Aug 7 21:33:40.037128099 (11h53m39.626473086s) [upload=12355]: Done uploading segments... Aug 7 21:33:40.037148919 (11h53m39.626493896s) [upload=12355]: Upload finished. err: context deadline exceeded Aug 7 21:33:40.899438535 (11h53m40.488783522s) [upload=12372]: Starting upload Aug 7 21:33:40.899490825 (11h53m40.488835832s) [upload=12372]: Done starting upload Aug 7 21:33:40.899590144 (11h53m40.488935131s) [upload=12372]: Uploading segments... Aug 7 21:33:40.899771832 (11h53m40.489116819s) [upload=12372]: Got next segment. Inline: false Aug 7 21:33:40.899795227 (11h53m40.489140214s) [upload=12372]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:33:40.899799275 (11h53m40.489144252s) [upload=12372]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:34:40.257761531 (11h54m39.847106518s) [upload=12372]: Next segment err: context deadline exceeded Aug 7 21:34:40.257885859 (11h54m39.847230856s) [upload=12372]: Done uploading segments... Aug 7 21:34:40.257904695 (11h54m39.847249672s) [upload=12372]: Upload finished. err: context deadline exceeded Aug 7 21:34:41.144124896 (11h54m40.733469873s) [upload=12386]: Starting upload Aug 7 21:34:41.144233103 (11h54m40.73357813s) [upload=12386]: Done starting upload Aug 7 21:34:41.144343113 (11h54m40.733688101s) [upload=12386]: Uploading segments... Aug 7 21:34:41.144700337 (11h54m40.734045364s) [upload=12386]: Got next segment. Inline: false Aug 7 21:34:41.144760612 (11h54m40.734105639s) [upload=12386]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:34:41.144768206 (11h54m40.734113223s) [upload=12386]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:35:40.491924814 (11h55m40.081269811s) [upload=12386]: Next segment err: context deadline exceeded Aug 7 21:35:40.492048891 (11h55m40.081393878s) [upload=12386]: Done uploading segments... Aug 7 21:35:40.492066364 (11h55m40.081411351s) [upload=12386]: Upload finished. err: context deadline exceeded Aug 7 21:35:41.353263670 (11h55m40.942608657s) [upload=12405]: Starting upload Aug 7 21:35:41.353369352 (11h55m40.942714339s) [upload=12405]: Done starting upload Aug 7 21:35:41.353452601 (11h55m40.942797588s) [upload=12405]: Uploading segments... Aug 7 21:35:41.353579473 (11h55m40.942924511s) [upload=12405]: Got next segment. Inline: false Aug 7 21:35:41.353621805 (11h55m40.942966792s) [upload=12405]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:35:41.353626053 (11h55m40.94297104s) [upload=12405]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:36:40.713868241 (11h56m40.303213228s) [upload=12405]: Next segment err: context deadline exceeded Aug 7 21:36:40.713958674 (11h56m40.303303651s) [upload=12405]: Done uploading segments... Aug 7 21:36:40.713972460 (11h56m40.303317447s) [upload=12405]: Upload finished. err: context deadline exceeded Aug 7 21:36:41.596698707 (11h56m41.186043744s) [upload=12418]: Starting upload Aug 7 21:36:41.596808026 (11h56m41.186153063s) [upload=12418]: Done starting upload Aug 7 21:36:41.596885475 (11h56m41.186230462s) [upload=12418]: Uploading segments... Aug 7 21:36:41.597098722 (11h56m41.186443709s) [upload=12418]: Got next segment. Inline: false Aug 7 21:36:41.597141925 (11h56m41.186486912s) [upload=12418]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:36:41.597146604 (11h56m41.186491591s) [upload=12418]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:37:40.947744552 (11h57m40.537089539s) [upload=12418]: Next segment err: context deadline exceeded Aug 7 21:37:40.947825697 (11h57m40.537170684s) [upload=12418]: Done uploading segments... Aug 7 21:37:40.947843592 (11h57m40.537188579s) [upload=12418]: Upload finished. err: context deadline exceeded Aug 7 21:37:41.823629920 (11h57m41.412974907s) [upload=12437]: Starting upload Aug 7 21:37:41.823680908 (11h57m41.413025895s) [upload=12437]: Done starting upload Aug 7 21:37:41.823826305 (11h57m41.413171293s) [upload=12437]: Uploading segments... Aug 7 21:37:41.823857055 (11h57m41.413202052s) [upload=12437]: Got next segment. Inline: false Aug 7 21:37:41.823914255 (11h57m41.413259242s) [upload=12437]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:37:41.823918503 (11h57m41.41326348s) [upload=12437]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:38:41.179029912 (11h58m40.768374899s) [upload=12437]: Next segment err: context deadline exceeded Aug 7 21:38:41.179128490 (11h58m40.768473477s) [upload=12437]: Done uploading segments... Aug 7 21:38:41.179142979 (11h58m40.768487966s) [upload=12437]: Upload finished. err: context deadline exceeded Aug 7 21:38:42.074438453 (11h58m41.66378344s) [upload=12451]: Starting upload Aug 7 21:38:42.074516963 (11h58m41.66386195s) [upload=12451]: Done starting upload Aug 7 21:38:42.074605202 (11h58m41.663950189s) [upload=12451]: Uploading segments... Aug 7 21:38:42.074689222 (11h58m41.664034209s) [upload=12451]: Got next segment. Inline: false Aug 7 21:38:42.074715102 (11h58m41.664060089s) [upload=12451]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:38:42.074718829 (11h58m41.664063816s) [upload=12451]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:39:41.428394360 (11h59m41.017739347s) [upload=12451]: Next segment err: context deadline exceeded Aug 7 21:39:41.428496375 (11h59m41.017841372s) [upload=12451]: Done uploading segments... Aug 7 21:39:41.428519450 (11h59m41.017864437s) [upload=12451]: Upload finished. err: context deadline exceeded Aug 7 21:39:42.313134109 (11h59m41.902479096s) [upload=12468]: Starting upload Aug 7 21:39:42.313191258 (11h59m41.902536245s) [upload=12468]: Done starting upload Aug 7 21:39:42.313256903 (11h59m41.90260189s) [upload=12468]: Uploading segments... Aug 7 21:39:42.313381803 (11h59m41.90272684s) [upload=12468]: Got next segment. Inline: false Aug 7 21:39:42.313472736 (11h59m41.902817763s) [upload=12468]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:39:42.313480300 (11h59m41.902825308s) [upload=12468]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:40:41.669512466 (12h0m41.258857454s) [upload=12468]: Next segment err: context deadline exceeded Aug 7 21:40:41.669582440 (12h0m41.258927427s) [upload=12468]: Done uploading segments... Aug 7 21:40:41.669600465 (12h0m41.258945452s) [upload=12468]: Upload finished. err: context deadline exceeded Duration: 1m0.001185112s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 33 out of 60 is failed (threshold=3) [.........................X..XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.0003864s FAIL (2025-08-07 21:40:39 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== {0 0}, upload=12449]: Done begin upload segment. Aug 7 21:38:48.686907212 (11h58m48.276252239s) [noise=true , address=104.36.82.2:4731 , piece_id=EA5DJOHUKB4JMDX56BNKNC3Z3SP5MLHLYN5PTJHOL4PC3DC2YZQA, seg_pos={0 0}, upload=12449]: Uploading piece... Aug 7 21:38:48.687437808 (11h58m48.276782795s) [noise=true , address=104.36.82.13:4185 , piece_id=GCSEQS5RYA7DIYXXEYRSVTCZWM4GII3CCOWDZXCWGK73LSPH6VDQ, seg_pos={0 0}, upload=12449]: Uploading piece... Aug 7 21:38:48.687923266 (11h58m48.277268253s) [noise=true , address=77.90.31.11:4554 , piece_id=Y5X5HMEMMOAD5QMICFFJFWSBHXSVZKTD54AP2NB4I2DJT42KJGVQ, seg_pos={0 0}, upload=12449]: Uploading piece... Aug 7 21:38:48.688304686 (11h58m48.277649673s) [noise=true , address=45.91.116.8:4045 , piece_id=JUV7DENMC32B4EJWTW6BJM4TEQRQ2U4FR7IFQAEJMCHRSV5JBXVQ, seg_pos={0 0}, upload=12449]: Uploading piece... Aug 7 21:38:48.688674213 (11h58m48.278019201s) [noise=true , address=104.36.82.15:4257 , piece_id=U7SFDCW5WBXVOCSJLGFOMXVNIMLQB7CY6BVPA2YUV63LBEC4IWAQ, seg_pos={0 0}, upload=12449]: Uploading piece... Aug 7 21:38:48.689020405 (11h58m48.278365392s) [noise=true , address=104.36.82.13:4186 , piece_id=XY22B7MTHZJ4MJFMJCTVDS4JAYQ3TWQDOPVMBUDZE3EF5VUAQTWQ, seg_pos={0 0}, upload=12449]: Uploading piece... Aug 7 21:38:48.689281786 (11h58m48.278626773s) [noise=true , address=104.36.82.10:4078 , piece_id=XYCM4QHQ7UU2S34Q5DUF7G5YGD2Z7KIDHHUMBRQNBEZUMD4GKNZQ, seg_pos={0 0}, upload=12449]: Uploading piece... Aug 7 21:38:48.689581169 (11h58m48.278926156s) [noise=true , address=45.91.116.5:4299 , piece_id=ED5YULTZVTYLCHQDFVZJIBUSEAYQUJMXGHHB7TTO7GQ5CSBBHYQA, seg_pos={0 0}, upload=12449]: Uploading piece... Aug 7 21:38:48.689893847 (11h58m48.279238824s) [noise=true , address=45.91.116.12:4225 , piece_id=INA2TNZ6KIYF45J4UFGRSLNRELTKISWF7IYIGPNXFDLLGZ7QVJWQ, seg_pos={0 0}, upload=12449]: Uploading piece... Aug 7 21:38:48.690256691 (11h58m48.279601678s) [noise=true , address=104.36.82.11:4114 , piece_id=QVVBKLBIFQC6RRFOJW7MM6AMHL2ZGLQAMH5ZE73XLBGY3VERYEVA, seg_pos={0 0}, upload=12449]: Uploading piece... Aug 7 21:38:48.690626529 (11h58m48.279971516s) [noise=true , address=45.91.116.10:4082 , piece_id=QX3QMQSXP5XBAWE4QY4P5A5QK2QXFBSV67YFPNIQV2BFIH2RVEKQ, seg_pos={0 0}, upload=12449]: Uploading piece... Aug 7 21:38:48.690901335 (11h58m48.280246322s) [noise=true , address=104.36.82.11:4113 , piece_id=2EGFZD54ZFDWU5SP6RQCC4YPP3USNQYBZNFSEI3V7CPMYLLB2A5Q, seg_pos={0 0}, upload=12449]: Uploading piece... Aug 7 21:38:48.691203833 (11h58m48.280548841s) [noise=true , address=77.90.31.12:4592 , piece_id=WW3UFNYQSC4MDLIZ2AJPTLTQN33HWQ2423TRFV5KEUNJ2R6XWKSQ, seg_pos={0 0}, upload=12449]: Uploading piece... Aug 7 21:38:48.691489149 (11h58m48.280834126s) [noise=true , address=77.90.31.21:4879 , piece_id=VB2RZ4HV4VI6WNNSHXZZRHZEAINPP5KU4LO6HVR2FLSGI5H3HP5A, seg_pos={0 0}, upload=12449]: Uploading piece... Aug 7 21:38:48.691733436 (11h58m48.281078424s) [noise=true , address=45.91.116.9:4263 , piece_id=EEBTJKDOQWVNQSWOUI47IY3RHKY5FDAMYQL4LJSUOS6GULR52TGA, seg_pos={0 0}, upload=12449]: Uploading piece... Aug 7 21:38:48.692068177 (11h58m48.281413164s) [noise=true , address=104.36.82.14:4221 , piece_id=W24SNKHE75CCO7XJFHUIH53M26MCEM7ILYZEGH672SQVK6ZCRBLQ, seg_pos={0 0}, upload=12449]: Uploading piece... Aug 7 21:38:50.513228864 (11h58m50.102573851s) [upload=12449]: Next returned nil segment Aug 7 21:38:50.513249563 (11h58m50.10259454s) [upload=12449]: Waiting for error group managing segments... Aug 7 21:38:50.810863968 (11h58m50.400208945s) [noise=true , address=104.36.82.2:4731 , piece_id=EA5DJOHUKB4JMDX56BNKNC3Z3SP5MLHLYN5PTJHOL4PC3DC2YZQA, seg_pos={0 0}, upload=12449]: Done uploading piece. err: <nil> Aug 7 21:38:50.873747961 (11h58m50.463092948s) [noise=true , address=104.36.82.11:4113 , piece_id=2EGFZD54ZFDWU5SP6RQCC4YPP3USNQYBZNFSEI3V7CPMYLLB2A5Q, seg_pos={0 0}, upload=12449]: Done uploading piece. err: <nil> Aug 7 21:38:50.944657320 (11h58m50.534002307s) [noise=true , address=45.91.116.8:4045 , piece_id=JUV7DENMC32B4EJWTW6BJM4TEQRQ2U4FR7IFQAEJMCHRSV5JBXVQ, seg_pos={0 0}, upload=12449]: Done uploading piece. err: <nil> Aug 7 21:38:50.972481134 (11h58m50.561826121s) [noise=true , address=104.36.82.14:4221 , piece_id=W24SNKHE75CCO7XJFHUIH53M26MCEM7ILYZEGH672SQVK6ZCRBLQ, seg_pos={0 0}, upload=12449]: Done uploading piece. err: <nil> Aug 7 21:38:50.980065425 (11h58m50.569410412s) [noise=true , address=104.36.82.13:4186 , piece_id=XY22B7MTHZJ4MJFMJCTVDS4JAYQ3TWQDOPVMBUDZE3EF5VUAQTWQ, seg_pos={0 0}, upload=12449]: Done uploading piece. err: <nil> Aug 7 21:38:50.981707937 (11h58m50.571052925s) [noise=true , address=104.36.82.11:4114 , piece_id=QVVBKLBIFQC6RRFOJW7MM6AMHL2ZGLQAMH5ZE73XLBGY3VERYEVA, seg_pos={0 0}, upload=12449]: Done uploading piece. err: <nil> Aug 7 21:38:51.023162491 (11h58m50.612507508s) [noise=true , address=104.36.82.13:4185 , piece_id=GCSEQS5RYA7DIYXXEYRSVTCZWM4GII3CCOWDZXCWGK73LSPH6VDQ, seg_pos={0 0}, upload=12449]: Done uploading piece. err: <nil> Aug 7 21:38:51.075895351 (11h58m50.665240338s) [noise=true , address=77.90.31.11:4554 , piece_id=Y5X5HMEMMOAD5QMICFFJFWSBHXSVZKTD54AP2NB4I2DJT42KJGVQ, seg_pos={0 0}, upload=12449]: Done uploading piece. err: <nil> Aug 7 21:38:51.095722626 (11h58m50.685067613s) [noise=true , address=45.91.116.5:4299 , piece_id=ED5YULTZVTYLCHQDFVZJIBUSEAYQUJMXGHHB7TTO7GQ5CSBBHYQA, seg_pos={0 0}, upload=12449]: Done uploading piece. err: <nil> Aug 7 21:38:51.098735141 (11h58m50.688080128s) [noise=true , address=104.36.82.15:4257 , piece_id=U7SFDCW5WBXVOCSJLGFOMXVNIMLQB7CY6BVPA2YUV63LBEC4IWAQ, seg_pos={0 0}, upload=12449]: Done uploading piece. err: <nil> Aug 7 21:38:51.107956113 (11h58m50.6973011s) [noise=true , address=77.90.31.21:4879 , piece_id=VB2RZ4HV4VI6WNNSHXZZRHZEAINPP5KU4LO6HVR2FLSGI5H3HP5A, seg_pos={0 0}, upload=12449]: Done uploading piece. err: <nil> Aug 7 21:38:51.133243133 (11h58m50.72258812s) [noise=true , address=77.90.31.12:4592 , piece_id=WW3UFNYQSC4MDLIZ2AJPTLTQN33HWQ2423TRFV5KEUNJ2R6XWKSQ, seg_pos={0 0}, upload=12449]: Done uploading piece. err: <nil> Aug 7 21:38:51.249092953 (11h58m50.83843794s) [noise=true , address=45.91.116.9:4263 , piece_id=EEBTJKDOQWVNQSWOUI47IY3RHKY5FDAMYQL4LJSUOS6GULR52TGA, seg_pos={0 0}, upload=12449]: Done uploading piece. err: <nil> Aug 7 21:38:51.249118061 (11h58m50.838463048s) [seg_pos={0 0}, upload=12449]: Segment reached optimal threshold of 13 pieces. Aug 7 21:38:51.249306762 (11h58m50.838651749s) [noise=true , address=45.91.116.12:4225 , piece_id=INA2TNZ6KIYF45J4UFGRSLNRELTKISWF7IYIGPNXFDLLGZ7QVJWQ, seg_pos={0 0}, upload=12449]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: expected piece hash; context canceled; EOF Aug 7 21:38:51.249323974 (11h58m50.838668961s) [noise=true , address=45.91.116.10:4082 , piece_id=QX3QMQSXP5XBAWE4QY4P5A5QK2QXFBSV67YFPNIQV2BFIH2RVEKQ, seg_pos={0 0}, upload=12449]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Aug 7 21:38:51.249493108 (11h58m50.838838095s) [noise=true , address=104.36.82.10:4078 , piece_id=XYCM4QHQ7UU2S34Q5DUF7G5YGD2Z7KIDHHUMBRQNBEZUMD4GKNZQ, seg_pos={0 0}, upload=12449]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr): protocol: context canceled; context canceled Aug 7 21:38:51.249505242 (11h58m50.838850229s) [seg_pos={0 0}, upload=12449]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 7 21:38:51.249581758 (11h58m50.838926735s) [upload=12449]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 21:38:51.249586157 (11h58m50.838931144s) [upload=12449]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:39:38.936884982 (11h59m38.526229969s) [upload=12449]: Done uploading segments... Aug 7 21:39:38.936908165 (11h59m38.526253152s) [upload=12449]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:39:39.835057740 (11h59m39.424402717s) [upload=12467]: Starting upload Aug 7 21:39:39.835133405 (11h59m39.424478382s) [upload=12467]: Done starting upload Aug 7 21:39:39.835203008 (11h59m39.424547995s) [upload=12467]: Uploading segments... Aug 7 21:39:39.835296918 (11h59m39.424641895s) [upload=12467]: Got next segment. Inline: false Aug 7 21:39:39.835322778 (11h59m39.424667765s) [upload=12467]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:39:39.835327146 (11h59m39.424672133s) [upload=12467]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:40:09.178127344 (12h0m8.767472331s) [upload=12467]: Next segment err: context deadline exceeded Aug 7 21:40:09.178178722 (12h0m8.767523709s) [upload=12467]: Done uploading segments... Aug 7 21:40:09.178192309 (12h0m8.767537296s) [upload=12467]: Upload finished. err: context deadline exceeded Aug 7 21:40:10.047916739 (12h0m9.637261726s) [upload=12477]: Starting upload Aug 7 21:40:10.047969550 (12h0m9.637314537s) [upload=12477]: Done starting upload Aug 7 21:40:10.048040987 (12h0m9.637385984s) [upload=12477]: Uploading segments... Aug 7 21:40:10.048146028 (12h0m9.637491015s) [upload=12477]: Got next segment. Inline: false Aug 7 21:40:10.048205722 (12h0m9.637550709s) [upload=12477]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:40:10.048214258 (12h0m9.637559275s) [upload=12477]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:40:39.404316574 (12h0m38.993661551s) [upload=12477]: Next segment err: context deadline exceeded Aug 7 21:40:39.404376929 (12h0m38.993721907s) [upload=12477]: Done uploading segments... Aug 7 21:40:39.404394894 (12h0m38.993739881s) [upload=12477]: Upload finished. err: context deadline exceeded Duration: 30.000399084s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 34 out of 60 is failed (threshold=3) [..........................XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.000886978s FAIL (2025-08-07 21:40:38 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.000904251s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 27 out of 60 is failed (threshold=3) [................................X.XXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.001059341s FAIL (2025-08-07 21:40:33 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 259s) [upload=12315]: Uploading segments... Aug 7 21:30:30.199515665 (11h50m29.788860652s) [upload=12315]: Got next segment. Inline: true Aug 7 21:30:30.199519302 (11h50m29.788864289s) [upload=12315]: Waiting for error group managing segments... Aug 7 21:30:30.199554779 (11h50m29.788899766s) [upload=12315]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:30:30.199559147 (11h50m29.788904134s) [upload=12315]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:30:30.199561592 (11h50m29.788906579s) [upload=12315]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:31:29.557680478 (11h51m29.147025465s) [upload=12315]: Done uploading segments... Aug 7 21:31:29.557706508 (11h51m29.147051485s) [upload=12315]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:31:30.468794234 (11h51m30.058139221s) [upload=12336]: Starting upload Aug 7 21:31:30.468935564 (11h51m30.058280551s) [upload=12336]: Done starting upload Aug 7 21:31:30.469048480 (11h51m30.058393467s) [upload=12336]: Uploading segments... Aug 7 21:31:30.469142200 (11h51m30.058487187s) [upload=12336]: Got next segment. Inline: true Aug 7 21:31:30.469147400 (11h51m30.058492387s) [upload=12336]: Waiting for error group managing segments... Aug 7 21:31:30.469203888 (11h51m30.058548875s) [upload=12336]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:31:30.469208757 (11h51m30.058553744s) [upload=12336]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:31:30.469211582 (11h51m30.058556569s) [upload=12336]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:32:29.815552389 (11h52m29.404897376s) [upload=12336]: Done uploading segments... Aug 7 21:32:29.815578508 (11h52m29.404923495s) [upload=12336]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:32:30.694545842 (11h52m30.283890829s) [upload=12349]: Starting upload Aug 7 21:32:30.694631607 (11h52m30.283976624s) [upload=12349]: Done starting upload Aug 7 21:32:30.694676312 (11h52m30.284021299s) [upload=12349]: Uploading segments... Aug 7 21:32:30.694717561 (11h52m30.284062538s) [upload=12349]: Got next segment. Inline: true Aug 7 21:32:30.694721909 (11h52m30.284066896s) [upload=12349]: Waiting for error group managing segments... Aug 7 21:32:30.694762737 (11h52m30.284107724s) [upload=12349]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:32:30.694766976 (11h52m30.284111953s) [upload=12349]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:32:30.694769561 (11h52m30.284114548s) [upload=12349]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:33:30.042542113 (11h53m29.6318871s) [upload=12349]: Done uploading segments... Aug 7 21:33:30.042561801 (11h53m29.631906788s) [upload=12349]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:33:30.916211822 (11h53m30.505556809s) [upload=12369]: Starting upload Aug 7 21:33:30.916281525 (11h53m30.505626512s) [upload=12369]: Done starting upload Aug 7 21:33:30.916351519 (11h53m30.505696567s) [upload=12369]: Uploading segments... Aug 7 21:33:30.916406474 (11h53m30.505751502s) [upload=12369]: Got next segment. Inline: true Aug 7 21:33:30.916413598 (11h53m30.505758605s) [upload=12369]: Waiting for error group managing segments... Aug 7 21:33:30.916454065 (11h53m30.505799052s) [upload=12369]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:33:30.916457992 (11h53m30.505802979s) [upload=12369]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:33:30.916460378 (11h53m30.505805355s) [upload=12369]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:34:30.262869110 (11h54m29.852214097s) [upload=12369]: Done uploading segments... Aug 7 21:34:30.262896352 (11h54m29.85224134s) [upload=12369]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:34:31.146607534 (11h54m30.735952521s) [upload=12385]: Starting upload Aug 7 21:34:31.146655144 (11h54m30.736000121s) [upload=12385]: Done starting upload Aug 7 21:34:31.146698838 (11h54m30.736043825s) [upload=12385]: Uploading segments... Aug 7 21:34:31.146741720 (11h54m30.736086697s) [upload=12385]: Got next segment. Inline: true Aug 7 21:34:31.146745548 (11h54m30.736090535s) [upload=12385]: Waiting for error group managing segments... Aug 7 21:34:31.146776246 (11h54m30.736121233s) [upload=12385]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:34:31.146779913 (11h54m30.73612489s) [upload=12385]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:34:31.146782388 (11h54m30.736127375s) [upload=12385]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:35:30.500521706 (11h55m30.089866693s) [upload=12385]: Done uploading segments... Aug 7 21:35:30.500542195 (11h55m30.089887182s) [upload=12385]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:35:31.371815861 (11h55m30.961160848s) [upload=12402]: Starting upload Aug 7 21:35:31.371969656 (11h55m30.961314643s) [upload=12402]: Done starting upload Aug 7 21:35:31.372062443 (11h55m30.96140743s) [upload=12402]: Uploading segments... Aug 7 21:35:31.372108090 (11h55m30.961453077s) [upload=12402]: Got next segment. Inline: true Aug 7 21:35:31.372114412 (11h55m30.961459399s) [upload=12402]: Waiting for error group managing segments... Aug 7 21:35:31.372166783 (11h55m30.96151177s) [upload=12402]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:35:31.372172002 (11h55m30.961516999s) [upload=12402]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:35:31.372174898 (11h55m30.961519885s) [upload=12402]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:36:30.721260917 (11h56m30.310605904s) [upload=12402]: Done uploading segments... Aug 7 21:36:30.721280625 (11h56m30.310625612s) [upload=12402]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:36:33.253873027 (11h56m32.843218014s) [upload=12416]: Starting upload Aug 7 21:36:33.253941188 (11h56m32.843286175s) [upload=12416]: Done starting upload Aug 7 21:36:33.254004108 (11h56m32.843349085s) [upload=12416]: Uploading segments... Aug 7 21:36:33.254039966 (11h56m32.843384953s) [upload=12416]: Got next segment. Inline: true Aug 7 21:36:33.254043462 (11h56m32.84338845s) [upload=12416]: Waiting for error group managing segments... Aug 7 21:36:33.254074321 (11h56m32.843419308s) [upload=12416]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:36:33.254078290 (11h56m32.843423277s) [upload=12416]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:36:33.254080714 (11h56m32.843425701s) [upload=12416]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:37:32.603532648 (11h57m32.192877635s) [upload=12416]: Done uploading segments... Aug 7 21:37:32.603556113 (11h57m32.1929011s) [upload=12416]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:37:33.470049892 (11h57m33.059394869s) [upload=12434]: Starting upload Aug 7 21:37:33.470104877 (11h57m33.059449865s) [upload=12434]: Done starting upload Aug 7 21:37:33.470145565 (11h57m33.059490552s) [upload=12434]: Uploading segments... Aug 7 21:37:33.470187375 (11h57m33.059532352s) [upload=12434]: Got next segment. Inline: true Aug 7 21:37:33.470190872 (11h57m33.059535859s) [upload=12434]: Waiting for error group managing segments... Aug 7 21:37:33.470223234 (11h57m33.059568221s) [upload=12434]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:37:33.470226710 (11h57m33.059571697s) [upload=12434]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:37:33.470228904 (11h57m33.059573881s) [upload=12434]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:38:32.827667037 (11h58m32.417012024s) [upload=12434]: Done uploading segments... Aug 7 21:38:32.827687857 (11h58m32.417032834s) [upload=12434]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:38:33.703666664 (11h58m33.293011651s) [upload=12450]: Starting upload Aug 7 21:38:33.703724295 (11h58m33.293069282s) [upload=12450]: Done starting upload Aug 7 21:38:33.703770663 (11h58m33.29311565s) [upload=12450]: Uploading segments... Aug 7 21:38:33.703817212 (11h58m33.293162199s) [upload=12450]: Got next segment. Inline: true Aug 7 21:38:33.703820899 (11h58m33.293165886s) [upload=12450]: Waiting for error group managing segments... Aug 7 21:38:33.703853411 (11h58m33.293198388s) [upload=12450]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:38:33.703857378 (11h58m33.293202365s) [upload=12450]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:38:33.703859744 (11h58m33.293204731s) [upload=12450]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:39:33.059235349 (11h59m32.648580326s) [upload=12450]: Done uploading segments... Aug 7 21:39:33.059253302 (11h59m32.648598289s) [upload=12450]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:39:33.956833499 (11h59m33.546178486s) [upload=12464]: Starting upload Aug 7 21:39:33.956907240 (11h59m33.546252227s) [upload=12464]: Done starting upload Aug 7 21:39:33.956967195 (11h59m33.546312182s) [upload=12464]: Uploading segments... Aug 7 21:39:33.957011540 (11h59m33.546356527s) [upload=12464]: Got next segment. Inline: true Aug 7 21:39:33.957016229 (11h59m33.546361226s) [upload=12464]: Waiting for error group managing segments... Aug 7 21:39:33.957053820 (11h59m33.546398807s) [upload=12464]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:39:33.957058250 (11h59m33.546403237s) [upload=12464]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:39:33.957061736 (11h59m33.546406723s) [upload=12464]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:40:33.302006486 (12h0m32.891351473s) [upload=12464]: Done uploading segments... Aug 7 21:40:33.302025733 (12h0m32.89137071s) [upload=12464]: Upload finished. err: metaclient: context deadline exceeded Duration: 1m0.001070141s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 31 out of 60 is failed (threshold=3) [.............................XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.001685696s FAIL (2025-08-07 21:40:31 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 1:31:10.105626315 (11h51m9.694971302s) [upload=12327]: Uploading segments... Aug 7 21:31:10.105718581 (11h51m9.695063568s) [upload=12327]: Got next segment. Inline: true Aug 7 21:31:10.105723311 (11h51m9.695068298s) [upload=12327]: Waiting for error group managing segments... Aug 7 21:31:10.105777513 (11h51m9.69512249s) [upload=12327]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:31:10.105782744 (11h51m9.695127731s) [upload=12327]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:31:10.105785079 (11h51m9.695130066s) [upload=12327]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:31:39.460569760 (11h51m39.049914747s) [upload=12327]: Done uploading segments... Aug 7 21:31:39.460597573 (11h51m39.04994255s) [upload=12327]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:32:24.324148762 (11h52m23.913493749s) [upload=12347]: Starting upload Aug 7 21:32:24.324206403 (11h52m23.91355139s) [upload=12347]: Done starting upload Aug 7 21:32:24.324252100 (11h52m23.913597087s) [upload=12347]: Uploading segments... Aug 7 21:32:24.324280415 (11h52m23.913625402s) [upload=12347]: Got next segment. Inline: true Aug 7 21:32:24.324284021 (11h52m23.913628998s) [upload=12347]: Waiting for error group managing segments... Aug 7 21:32:24.324329689 (11h52m23.913674686s) [upload=12347]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:32:24.324336372 (11h52m23.913681379s) [upload=12347]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:32:24.324341050 (11h52m23.913686057s) [upload=12347]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:32:53.678084170 (11h52m53.267429207s) [upload=12347]: Done uploading segments... Aug 7 21:32:53.678107405 (11h52m53.267452392s) [upload=12347]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:33:17.618427342 (11h53m17.207772389s) [upload=12364]: Starting upload Aug 7 21:33:17.618576718 (11h53m17.207921745s) [upload=12364]: Done starting upload Aug 7 21:33:17.618658725 (11h53m17.208003742s) [upload=12364]: Uploading segments... Aug 7 21:33:17.618713580 (11h53m17.208058597s) [upload=12364]: Got next segment. Inline: true Aug 7 21:33:17.618720533 (11h53m17.20806553s) [upload=12364]: Waiting for error group managing segments... Aug 7 21:33:17.618772433 (11h53m17.20811743s) [upload=12364]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:33:17.618779636 (11h53m17.208124643s) [upload=12364]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:33:17.618784776 (11h53m17.208129793s) [upload=12364]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:33:46.957323530 (11h53m46.546668577s) [upload=12364]: Done uploading segments... Aug 7 21:33:46.957345773 (11h53m46.54669075s) [upload=12364]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:34:19.197336641 (11h54m18.786681628s) [upload=12382]: Starting upload Aug 7 21:34:19.197461931 (11h54m18.786806918s) [upload=12382]: Done starting upload Aug 7 21:34:19.197512228 (11h54m18.786857225s) [upload=12382]: Uploading segments... Aug 7 21:34:19.197561282 (11h54m18.786906269s) [upload=12382]: Got next segment. Inline: true Aug 7 21:34:19.197565189 (11h54m18.786910166s) [upload=12382]: Waiting for error group managing segments... Aug 7 21:34:19.197610725 (11h54m18.786955723s) [upload=12382]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:34:19.197615966 (11h54m18.786960953s) [upload=12382]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:34:19.197619142 (11h54m18.786964139s) [upload=12382]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:34:48.535954936 (11h54m48.125299923s) [upload=12382]: Done uploading segments... Aug 7 21:34:48.535978650 (11h54m48.125323637s) [upload=12382]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:35:05.991757765 (11h55m5.581102752s) [upload=12393]: Starting upload Aug 7 21:35:05.991822780 (11h55m5.581167767s) [upload=12393]: Done starting upload Aug 7 21:35:05.991946456 (11h55m5.581291443s) [upload=12393]: Uploading segments... Aug 7 21:35:05.991988116 (11h55m5.581333104s) [upload=12393]: Got next segment. Inline: true Aug 7 21:35:05.991994929 (11h55m5.581339996s) [upload=12393]: Waiting for error group managing segments... Aug 7 21:35:05.992042741 (11h55m5.581387728s) [upload=12393]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:35:05.992048021 (11h55m5.581393018s) [upload=12393]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:35:05.992051768 (11h55m5.581396755s) [upload=12393]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:35:35.339368736 (11h55m34.928713723s) [upload=12393]: Done uploading segments... Aug 7 21:35:35.339503534 (11h55m34.928848522s) [upload=12393]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:36:12.952326182 (11h56m12.541671159s) [upload=12411]: Starting upload Aug 7 21:36:12.952385075 (11h56m12.541730052s) [upload=12411]: Done starting upload Aug 7 21:36:12.952428238 (11h56m12.541773225s) [upload=12411]: Uploading segments... Aug 7 21:36:12.952455510 (11h56m12.541800497s) [upload=12411]: Got next segment. Inline: true Aug 7 21:36:12.952459417 (11h56m12.541804404s) [upload=12411]: Waiting for error group managing segments... Aug 7 21:36:12.952507650 (11h56m12.541852637s) [upload=12411]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:36:12.952514352 (11h56m12.541859349s) [upload=12411]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:36:12.952518550 (11h56m12.541863537s) [upload=12411]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:36:42.299158316 (11h56m41.888503304s) [upload=12411]: Done uploading segments... Aug 7 21:36:42.299175039 (11h56m41.888520026s) [upload=12411]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:37:22.572699237 (11h57m22.162044224s) [upload=12432]: Starting upload Aug 7 21:37:22.572782937 (11h57m22.162127924s) [upload=12432]: Done starting upload Aug 7 21:37:22.572842260 (11h57m22.162187247s) [upload=12432]: Uploading segments... Aug 7 21:37:22.572885313 (11h57m22.1622303s) [upload=12432]: Got next segment. Inline: true Aug 7 21:37:22.572889591 (11h57m22.162234578s) [upload=12432]: Waiting for error group managing segments... Aug 7 21:37:22.572930138 (11h57m22.162275125s) [upload=12432]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:37:22.572934296 (11h57m22.162279283s) [upload=12432]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:37:22.572937121 (11h57m22.162282118s) [upload=12432]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:37:51.933954276 (11h57m51.523299253s) [upload=12432]: Done uploading segments... Aug 7 21:37:51.933970927 (11h57m51.523315914s) [upload=12432]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:38:28.745940745 (11h58m28.335285732s) [upload=12448]: Starting upload Aug 7 21:38:28.745997504 (11h58m28.335342491s) [upload=12448]: Done starting upload Aug 7 21:38:28.746040176 (11h58m28.335385163s) [upload=12448]: Uploading segments... Aug 7 21:38:28.746078458 (11h58m28.335423446s) [upload=12448]: Got next segment. Inline: true Aug 7 21:38:28.746081766 (11h58m28.335426743s) [upload=12448]: Waiting for error group managing segments... Aug 7 21:38:28.746111923 (11h58m28.33545691s) [upload=12448]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:38:28.746115540 (11h58m28.335460527s) [upload=12448]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:38:28.746118205 (11h58m28.335463192s) [upload=12448]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:38:58.105315693 (11h58m57.694660691s) [upload=12448]: Done uploading segments... Aug 7 21:38:58.105342465 (11h58m57.694687452s) [upload=12448]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:39:27.137238449 (11h59m26.726583436s) [upload=12462]: Starting upload Aug 7 21:39:27.137294907 (11h59m26.726639894s) [upload=12462]: Done starting upload Aug 7 21:39:27.137337368 (11h59m26.726682355s) [upload=12462]: Uploading segments... Aug 7 21:39:27.137366043 (11h59m26.72671103s) [upload=12462]: Got next segment. Inline: true Aug 7 21:39:27.137370061 (11h59m26.726715048s) [upload=12462]: Waiting for error group managing segments... Aug 7 21:39:27.137401901 (11h59m26.726746879s) [upload=12462]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:39:27.137405518 (11h59m26.726750495s) [upload=12462]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:39:27.137407813 (11h59m26.72675279s) [upload=12462]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:39:56.495186447 (11h59m56.084531434s) [upload=12462]: Done uploading segments... Aug 7 21:39:56.495206415 (11h59m56.084551402s) [upload=12462]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:40:01.914615953 (12h0m1.50396094s) [upload=12475]: Starting upload Aug 7 21:40:01.914684295 (12h0m1.504029282s) [upload=12475]: Done starting upload Aug 7 21:40:01.914741163 (12h0m1.50408616s) [upload=12475]: Uploading segments... Aug 7 21:40:01.914783103 (12h0m1.50412809s) [upload=12475]: Got next segment. Inline: true Aug 7 21:40:01.914788143 (12h0m1.504133131s) [upload=12475]: Waiting for error group managing segments... Aug 7 21:40:01.914833950 (12h0m1.504178937s) [upload=12475]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:40:01.914840073 (12h0m1.50418507s) [upload=12475]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:40:01.914846014 (12h0m1.504191011s) [upload=12475]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:40:31.268196986 (12h0m30.857542023s) [upload=12475]: Done uploading segments... Aug 7 21:40:31.268223536 (12h0m30.857568533s) [upload=12475]: Upload finished. err: metaclient: context deadline exceeded Duration: 30.001717387s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 35 out of 60 is failed (threshold=3) [.........................XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.001466331s FAIL (2025-08-07 21:40:24 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.001488603s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [.................................X...............X..........] duration: 2.119546786s FAIL (2025-08-07 21:40:21 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/1K") ======LOGS====== 083457s) [upload=12258]: Done uploading segments... Aug 7 21:26:43.142756605 (11h46m42.732101593s) [upload=12258]: Upload finished. err: <nil> Aug 7 21:27:16.645746716 (11h47m16.235091703s) [upload=12264]: Starting upload Aug 7 21:27:16.645808424 (11h47m16.235153411s) [upload=12264]: Done starting upload Aug 7 21:27:16.645853019 (11h47m16.235198006s) [upload=12264]: Uploading segments... Aug 7 21:27:16.645881223 (11h47m16.23522621s) [upload=12264]: Got next segment. Inline: true Aug 7 21:27:16.645884680 (11h47m16.235229667s) [upload=12264]: Waiting for error group managing segments... Aug 7 21:27:16.645915299 (11h47m16.235260286s) [upload=12264]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:27:16.645919216 (11h47m16.235264203s) [upload=12264]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:27:16.645921510 (11h47m16.235266498s) [upload=12264]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:27:16.954194817 (11h47m16.543539804s) [upload=12264]: Done uploading segments... Aug 7 21:27:16.954212471 (11h47m16.543557459s) [upload=12264]: Upload finished. err: <nil> Aug 7 21:27:46.945635033 (11h47m46.53498005s) [upload=12274]: Starting upload Aug 7 21:27:46.945720126 (11h47m46.535065163s) [upload=12274]: Done starting upload Aug 7 21:27:46.945850435 (11h47m46.535195422s) [upload=12274]: Uploading segments... Aug 7 21:27:46.945918004 (11h47m46.535262991s) [upload=12274]: Got next segment. Inline: true Aug 7 21:27:46.945922443 (11h47m46.53526742s) [upload=12274]: Waiting for error group managing segments... Aug 7 21:27:46.945960726 (11h47m46.535305713s) [upload=12274]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:27:46.945965345 (11h47m46.535310332s) [upload=12274]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:27:46.945999029 (11h47m46.535344026s) [upload=12274]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:27:47.238086931 (11h47m46.827431908s) [upload=12274]: Done uploading segments... Aug 7 21:27:47.238104565 (11h47m46.827449552s) [upload=12274]: Upload finished. err: <nil> Aug 7 21:28:27.597717226 (11h48m27.187062213s) [upload=12282]: Starting upload Aug 7 21:28:27.597768604 (11h48m27.187113581s) [upload=12282]: Done starting upload Aug 7 21:28:27.597851232 (11h48m27.187196219s) [upload=12282]: Uploading segments... Aug 7 21:28:27.597881140 (11h48m27.187226127s) [upload=12282]: Got next segment. Inline: true Aug 7 21:28:27.597885238 (11h48m27.187230225s) [upload=12282]: Waiting for error group managing segments... Aug 7 21:28:27.597916337 (11h48m27.187261324s) [upload=12282]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:28:27.597920134 (11h48m27.187265111s) [upload=12282]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:28:27.597922509 (11h48m27.187267496s) [upload=12282]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:28:27.890743040 (11h48m27.480088057s) [upload=12282]: Done uploading segments... Aug 7 21:28:27.890764190 (11h48m27.480109177s) [upload=12282]: Upload finished. err: <nil> Aug 7 21:29:22.617930722 (11h49m22.207275709s) [upload=12295]: Starting upload Aug 7 21:29:22.617969135 (11h49m22.207314122s) [upload=12295]: Done starting upload Aug 7 21:29:22.618026967 (11h49m22.207371954s) [upload=12295]: Uploading segments... Aug 7 21:29:22.618059649 (11h49m22.207404636s) [upload=12295]: Got next segment. Inline: true Aug 7 21:29:22.618063226 (11h49m22.207408213s) [upload=12295]: Waiting for error group managing segments... Aug 7 21:29:22.618096049 (11h49m22.207441026s) [upload=12295]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:29:22.618100216 (11h49m22.207445193s) [upload=12295]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:29:22.618104334 (11h49m22.207449321s) [upload=12295]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:29:23.935417449 (11h49m23.524762486s) [upload=12295]: Done uploading segments... Aug 7 21:29:23.935440473 (11h49m23.52478548s) [upload=12295]: Upload finished. err: <nil> Aug 7 21:29:59.675674818 (11h49m59.265019805s) [upload=12308]: Starting upload Aug 7 21:29:59.675709946 (11h49m59.265054933s) [upload=12308]: Done starting upload Aug 7 21:29:59.675775310 (11h49m59.265120287s) [upload=12308]: Uploading segments... Aug 7 21:29:59.675809485 (11h49m59.265154462s) [upload=12308]: Got next segment. Inline: true Aug 7 21:29:59.675813022 (11h49m59.265158009s) [upload=12308]: Waiting for error group managing segments... Aug 7 21:29:59.675846747 (11h49m59.265191724s) [upload=12308]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:29:59.675850584 (11h49m59.265195572s) [upload=12308]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:29:59.675852909 (11h49m59.265197896s) [upload=12308]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:29:59.960382218 (11h49m59.549727205s) [upload=12308]: Done uploading segments... Aug 7 21:29:59.960402026 (11h49m59.549747013s) [upload=12308]: Upload finished. err: <nil> Aug 7 21:31:01.204540565 (11h51m0.793885552s) [upload=12324]: Starting upload Aug 7 21:31:01.204590701 (11h51m0.793935688s) [upload=12324]: Done starting upload Aug 7 21:31:01.204646679 (11h51m0.793991666s) [upload=12324]: Uploading segments... Aug 7 21:31:01.204679222 (11h51m0.794024209s) [upload=12324]: Got next segment. Inline: true Aug 7 21:31:01.204683269 (11h51m0.794028256s) [upload=12324]: Waiting for error group managing segments... Aug 7 21:31:01.204721322 (11h51m0.794066309s) [upload=12324]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:31:01.204725289 (11h51m0.794070276s) [upload=12324]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:31:01.204728024 (11h51m0.794073011s) [upload=12324]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:31:01.494645083 (11h51m1.08399007s) [upload=12324]: Done uploading segments... Aug 7 21:31:01.494665120 (11h51m1.084010107s) [upload=12324]: Upload finished. err: <nil> Aug 7 21:31:28.376153869 (11h51m27.965498856s) [upload=12334]: Starting upload Aug 7 21:31:28.376197382 (11h51m27.965542369s) [upload=12334]: Done starting upload Aug 7 21:31:28.376245204 (11h51m27.965590181s) [upload=12334]: Uploading segments... Aug 7 21:31:28.376273257 (11h51m27.965618245s) [upload=12334]: Got next segment. Inline: true Aug 7 21:31:28.376276624 (11h51m27.965621611s) [upload=12334]: Waiting for error group managing segments... Aug 7 21:31:28.376306892 (11h51m27.965651869s) [upload=12334]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:31:28.376312712 (11h51m27.9656577s) [upload=12334]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:31:28.376315419 (11h51m27.965660406s) [upload=12334]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:31:28.645925163 (11h51m28.23527015s) [upload=12334]: Done uploading segments... Aug 7 21:31:28.645942786 (11h51m28.235287773s) [upload=12334]: Upload finished. err: <nil> Aug 7 21:32:07.331024173 (11h52m6.92036916s) [upload=12342]: Starting upload Aug 7 21:32:07.331074259 (11h52m6.920419246s) [upload=12342]: Done starting upload Aug 7 21:32:07.331122251 (11h52m6.920467239s) [upload=12342]: Uploading segments... Aug 7 21:32:07.331154433 (11h52m6.92049942s) [upload=12342]: Got next segment. Inline: true Aug 7 21:32:07.331159583 (11h52m6.92050456s) [upload=12342]: Waiting for error group managing segments... Aug 7 21:32:07.331191624 (11h52m6.920536611s) [upload=12342]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:32:07.331195581 (11h52m6.920540558s) [upload=12342]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:32:07.331197946 (11h52m6.920542933s) [upload=12342]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:32:07.722093791 (11h52m7.311438768s) [upload=12342]: Done uploading segments... Aug 7 21:32:07.722110794 (11h52m7.311455781s) [upload=12342]: Upload finished. err: <nil> Aug 7 21:32:38.771578054 (11h52m38.360923031s) [upload=12353]: Starting upload Aug 7 21:32:38.771650211 (11h52m38.360995188s) [upload=12353]: Done starting upload Aug 7 21:32:38.771738161 (11h52m38.361083148s) [upload=12353]: Uploading segments... Aug 7 21:32:38.771779349 (11h52m38.361124336s) [upload=12353]: Got next segment. Inline: true Aug 7 21:32:38.771782916 (11h52m38.361127893s) [upload=12353]: Waiting for error group managing segments... Aug 7 21:32:38.771815357 (11h52m38.361160334s) [upload=12353]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:32:38.771819266 (11h52m38.361164253s) [upload=12353]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:32:38.771821730 (11h52m38.361166717s) [upload=12353]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:32:39.111523769 (11h52m38.700868796s) [upload=12353]: Done uploading segments... Aug 7 21:32:39.111547013 (11h52m38.700892031s) [upload=12353]: Upload finished. err: <nil> Aug 7 21:33:06.222382527 (11h53m5.811727514s) [upload=12362]: Starting upload Aug 7 21:33:06.222432713 (11h53m5.8117777s) [upload=12362]: Done starting upload Aug 7 21:33:06.222489962 (11h53m5.811834949s) [upload=12362]: Uploading segments... Aug 7 21:33:06.222536221 (11h53m5.811881208s) [upload=12362]: Got next segment. Inline: true Aug 7 21:33:06.222540199 (11h53m5.811885176s) [upload=12362]: Waiting for error group managing segments... Aug 7 21:33:06.222573302 (11h53m5.811918289s) [upload=12362]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:33:06.222576979 (11h53m5.811921966s) [upload=12362]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:33:06.222579153 (11h53m5.81192413s) [upload=12362]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:33:06.561908138 (11h53m6.151253125s) [upload=12362]: Done uploading segments... Aug 7 21:33:06.561986849 (11h53m6.151331836s) [upload=12362]: Upload finished. err: <nil> Duration: 2.119572696s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 6 out of 60 is failed (threshold=3) [......................X.........X......X..X.............X..X] duration: 5.784969383s FAIL (2025-08-07 21:40:18 UTC) orbiter warning: operation: ListObjects ======ERROR====== unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/4M ======LOGS====== ad=12478]: Done starting upload Aug 7 21:40:13.160345841 (12h0m12.749690828s) [upload=12478]: Uploading segments... Aug 7 21:40:13.160465419 (12h0m12.749810406s) [upload=12478]: Got next segment. Inline: false Aug 7 21:40:13.160488423 (12h0m12.74983341s) [upload=12478]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:40:13.160492521 (12h0m12.749837508s) [upload=12478]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:40:14.024449389 (12h0m13.613794386s) [seg_pos={0 0}, upload=12478]: Begin upload segment... Aug 7 21:40:14.024497139 (12h0m13.613842126s) [seg_pos={0 0}, upload=12478]: Done begin upload segment. Aug 7 21:40:14.024520314 (12h0m13.613865301s) [noise=true , address=104.36.82.12:4163 , piece_id=HCC7QXBZMB3P33M366DZRH2IWVQQVOT56LAWYLY5MW5RARFSISVQ, seg_pos={0 0}, upload=12478]: Uploading piece... Aug 7 21:40:14.024527568 (12h0m13.613872555s) [noise=true , address=104.36.82.12:4177 , piece_id=IPZF77LYNKRXAEDUDGMCUXT4EG5JSIFP3I7E3VPEIN3WNV7XCAKA, seg_pos={0 0}, upload=12478]: Uploading piece... Aug 7 21:40:14.024915139 (12h0m13.614260126s) [noise=true , address=104.36.82.46:4496 , piece_id=FAAS2ISJ3FABRTBOOUGW77IT6A2325ZKX4CSGMHSBRZWSQ4XNOGA, seg_pos={0 0}, upload=12478]: Uploading piece... Aug 7 21:40:14.024946630 (12h0m13.614291617s) [noise=true , address=104.36.82.52:4109 , piece_id=XMAEWF5LMHRJMQAA6FSX6TCYB4OUJ2GVDZQUFDPVFWGXGRQZCZIA, seg_pos={0 0}, upload=12478]: Uploading piece... Aug 7 21:40:14.025230222 (12h0m13.614575199s) [noise=true , address=104.36.82.23:4449 , piece_id=Y52RDGW6QODSWLLQSKBJPSNPUTHB5OLEJHN3Y4GUQ6OZK5XARCRA, seg_pos={0 0}, upload=12478]: Uploading piece... Aug 7 21:40:14.025320585 (12h0m13.614665572s) [noise=true , address=104.36.82.58:4291 , piece_id=246EQHXDOHPPFCKBCQNBTJ46UDD5ZRRPHA7YUUGU22FPJJLKNQ2Q, seg_pos={0 0}, upload=12478]: Uploading piece... Aug 7 21:40:14.025541578 (12h0m13.614886565s) [noise=true , address=64.49.13.170:4721 , piece_id=N7YWI7FRIBKFLUF7TQZLWTT3YTVRDGHUPQJQ2GUP3ENNCZO6DXHQ, seg_pos={0 0}, upload=12478]: Uploading piece... Aug 7 21:40:14.025702406 (12h0m13.615047393s) [noise=true , address=104.36.82.61:4394 , piece_id=2XK3J33NNGFMQU7YDFG35IZ3JFAHUFQ7C3OIUFSNPOSFXB6KEKPQ, seg_pos={0 0}, upload=12478]: Uploading piece... Aug 7 21:40:14.025850619 (12h0m13.615195606s) [noise=true , address=104.36.82.5:4859 , piece_id=3KS2DGR4OLOW4ZUVHSMTVE6XIXZQ3VWOBUXVGTCJTLHNIWGVZOEA, seg_pos={0 0}, upload=12478]: Uploading piece... Aug 7 21:40:14.026035403 (12h0m13.61538039s) [noise=true , address=45.91.116.5:4324 , piece_id=F55DGRECWWVUHFXVMHYRYUHSTO262DXLEIAB2UABJRTIDWG4MEMA, seg_pos={0 0}, upload=12478]: Uploading piece... Aug 7 21:40:14.026287314 (12h0m13.615632301s) [noise=true , address=104.36.82.36:4643 , piece_id=VU7DYF2KI6TB74ERJLYE6BDDQQPMDBSMLZD4H6DKNRKW442ICVGA, seg_pos={0 0}, upload=12478]: Uploading piece... Aug 7 21:40:14.026331439 (12h0m13.615676426s) [noise=true , address=45.91.116.6:4999 , piece_id=CHC3CKW5P6UDSGH2HRU7TMLEEK7A4Q6USGZXSOWVH7SWDBQ6ASDA, seg_pos={0 0}, upload=12478]: Uploading piece... Aug 7 21:40:14.026644838 (12h0m13.615989826s) [noise=true , address=45.91.116.7:4012 , piece_id=FCVPPAJSDQDEMSIP3CJPXF5FQXQ6W3G3CAMR66IOILUY4OQK7P4A, seg_pos={0 0}, upload=12478]: Uploading piece... Aug 7 21:40:14.026679736 (12h0m13.616024713s) [noise=true , address=104.36.82.54:4160 , piece_id=ODKE4XCXDIENJAZZJR7FUE35EM3XNJL3MITY2CFDLZRBBSHQIKXQ, seg_pos={0 0}, upload=12478]: Uploading piece... Aug 7 21:40:14.026890909 (12h0m13.616235896s) [noise=true , address=77.90.31.18:4828 , piece_id=GPXCXC2RW6MRDMLAZJXDML4JO62QAZAZLYZP3JV7J6DQR7QEMTBA, seg_pos={0 0}, upload=12478]: Uploading piece... Aug 7 21:40:14.027132201 (12h0m13.616477178s) [noise=true , address=104.36.82.15:4283 , piece_id=AXCQRAQ6EYNFFHRDCWO6DFLB76ASUYOXWMPMQZYZHHIN6NF3QS5A, seg_pos={0 0}, upload=12478]: Uploading piece... Aug 7 21:40:14.027330621 (12h0m13.616675608s) [noise=true , address=104.36.82.45:4939 , piece_id=BECNPJEALTYQXO4IBONPU27XA2MOFWIP7ATHDJFMYTIG5OVAOHLA, seg_pos={0 0}, upload=12478]: Uploading piece... Aug 7 21:40:14.027577493 (12h0m13.61692248s) [noise=true , address=64.49.13.140:4510 , piece_id=VJJD4RVIWXNIDCZXKEEMBDQD6ZQXPOWS3WKWBAFOOKHBUTJH6CLQ, seg_pos={0 0}, upload=12478]: Uploading piece... Aug 7 21:40:15.144921916 (12h0m14.734266903s) [upload=12478]: Next returned nil segment Aug 7 21:40:15.144943788 (12h0m14.734288775s) [upload=12478]: Waiting for error group managing segments... Aug 7 21:40:15.425370688 (12h0m15.014715665s) [noise=true , address=64.49.13.140:4510 , piece_id=VJJD4RVIWXNIDCZXKEEMBDQD6ZQXPOWS3WKWBAFOOKHBUTJH6CLQ, seg_pos={0 0}, upload=12478]: Done uploading piece. err: <nil> Aug 7 21:40:15.528965431 (12h0m15.118310418s) [noise=true , address=64.49.13.170:4721 , piece_id=N7YWI7FRIBKFLUF7TQZLWTT3YTVRDGHUPQJQ2GUP3ENNCZO6DXHQ, seg_pos={0 0}, upload=12478]: Done uploading piece. err: <nil> Aug 7 21:40:15.945504159 (12h0m15.534849146s) [noise=true , address=104.36.82.46:4496 , piece_id=FAAS2ISJ3FABRTBOOUGW77IT6A2325ZKX4CSGMHSBRZWSQ4XNOGA, seg_pos={0 0}, upload=12478]: Done uploading piece. err: <nil> Aug 7 21:40:15.973785008 (12h0m15.563129995s) [noise=true , address=104.36.82.12:4163 , piece_id=HCC7QXBZMB3P33M366DZRH2IWVQQVOT56LAWYLY5MW5RARFSISVQ, seg_pos={0 0}, upload=12478]: Done uploading piece. err: <nil> Aug 7 21:40:16.002098709 (12h0m15.591443696s) [noise=true , address=104.36.82.52:4109 , piece_id=XMAEWF5LMHRJMQAA6FSX6TCYB4OUJ2GVDZQUFDPVFWGXGRQZCZIA, seg_pos={0 0}, upload=12478]: Done uploading piece. err: <nil> Aug 7 21:40:16.030274577 (12h0m15.619619604s) [noise=true , address=104.36.82.23:4449 , piece_id=Y52RDGW6QODSWLLQSKBJPSNPUTHB5OLEJHN3Y4GUQ6OZK5XARCRA, seg_pos={0 0}, upload=12478]: Done uploading piece. err: <nil> Aug 7 21:40:16.044682273 (12h0m15.63402725s) [noise=true , address=104.36.82.54:4160 , piece_id=ODKE4XCXDIENJAZZJR7FUE35EM3XNJL3MITY2CFDLZRBBSHQIKXQ, seg_pos={0 0}, upload=12478]: Done uploading piece. err: <nil> Aug 7 21:40:16.053282688 (12h0m15.642627665s) [noise=true , address=45.91.116.5:4324 , piece_id=F55DGRECWWVUHFXVMHYRYUHSTO262DXLEIAB2UABJRTIDWG4MEMA, seg_pos={0 0}, upload=12478]: Done uploading piece. err: <nil> Aug 7 21:40:16.058737404 (12h0m15.648082391s) [noise=true , address=104.36.82.58:4291 , piece_id=246EQHXDOHPPFCKBCQNBTJ46UDD5ZRRPHA7YUUGU22FPJJLKNQ2Q, seg_pos={0 0}, upload=12478]: Done uploading piece. err: <nil> Aug 7 21:40:16.093091522 (12h0m15.682436509s) [noise=true , address=104.36.82.12:4177 , piece_id=IPZF77LYNKRXAEDUDGMCUXT4EG5JSIFP3I7E3VPEIN3WNV7XCAKA, seg_pos={0 0}, upload=12478]: Done uploading piece. err: <nil> Aug 7 21:40:16.176548758 (12h0m15.765893735s) [noise=true , address=104.36.82.45:4939 , piece_id=BECNPJEALTYQXO4IBONPU27XA2MOFWIP7ATHDJFMYTIG5OVAOHLA, seg_pos={0 0}, upload=12478]: Done uploading piece. err: <nil> Aug 7 21:40:16.186654263 (12h0m15.77599925s) [noise=true , address=104.36.82.5:4859 , piece_id=3KS2DGR4OLOW4ZUVHSMTVE6XIXZQ3VWOBUXVGTCJTLHNIWGVZOEA, seg_pos={0 0}, upload=12478]: Done uploading piece. err: <nil> Aug 7 21:40:16.255094056 (12h0m15.844439053s) [noise=true , address=45.91.116.7:4012 , piece_id=FCVPPAJSDQDEMSIP3CJPXF5FQXQ6W3G3CAMR66IOILUY4OQK7P4A, seg_pos={0 0}, upload=12478]: Done uploading piece. err: <nil> Aug 7 21:40:16.268089419 (12h0m15.857434406s) [noise=true , address=77.90.31.18:4828 , piece_id=GPXCXC2RW6MRDMLAZJXDML4JO62QAZAZLYZP3JV7J6DQR7QEMTBA, seg_pos={0 0}, upload=12478]: Done uploading piece. err: <nil> Aug 7 21:40:16.268110299 (12h0m15.857455286s) [seg_pos={0 0}, upload=12478]: Segment reached optimal threshold of 14 pieces. Aug 7 21:40:16.268265978 (12h0m15.857610965s) [noise=true , address=104.36.82.61:4394 , piece_id=2XK3J33NNGFMQU7YDFG35IZ3JFAHUFQ7C3OIUFSNPOSFXB6KEKPQ, seg_pos={0 0}, upload=12478]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1dSpTqCxeAcL9PVYbP7BhmuKE7papFPc9swB1Fy83WcFDgQbcF): protocol: expected piece hash; context canceled; EOF Aug 7 21:40:16.268327325 (12h0m15.857672302s) [noise=true , address=45.91.116.6:4999 , piece_id=CHC3CKW5P6UDSGH2HRU7TMLEEK7A4Q6USGZXSOWVH7SWDBQ6ASDA, seg_pos={0 0}, upload=12478]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12evc8vF5f1Z7cPrrC7G7BsGuDWjzttAE2FsQciV7nztBDmrCZB): protocol: expected piece hash; context canceled; EOF Aug 7 21:40:16.268492010 (12h0m15.857836997s) [noise=true , address=104.36.82.36:4643 , piece_id=VU7DYF2KI6TB74ERJLYE6BDDQQPMDBSMLZD4H6DKNRKW442ICVGA, seg_pos={0 0}, upload=12478]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zQsZ4qrfBa95hJYTzjWJJzSea3noCnZ7W22V8ViohjrB5opEw): protocol: write tcp 188.166.225.170:42342->104.36.82.36:4643: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 21:40:16.268605697 (12h0m15.857950684s) [noise=true , address=104.36.82.15:4283 , piece_id=AXCQRAQ6EYNFFHRDCWO6DFLB76ASUYOXWMPMQZYZHHIN6NF3QS5A, seg_pos={0 0}, upload=12478]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ytKjZnmERiZtgJuMJhXG5KFwUQwie27odve4LPE7Pw5hR5ZaU): protocol: write tcp 188.166.225.170:45054->104.36.82.15:4283: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 21:40:16.268677886 (12h0m15.858022873s) [seg_pos={0 0}, upload=12478]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 21:40:16.268787846 (12h0m15.858132843s) [upload=12478]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 21:40:16.268796623 (12h0m15.85814161s) [upload=12478]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:40:17.124513483 (12h0m16.71385847s) [upload=12478]: Done uploading segments... Aug 7 21:40:17.124534484 (12h0m16.713879471s) [upload=12478]: Upload finished. err: <nil> Duration: 5.78499424s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 25 out of 60 is failed (threshold=3) [...................................XXXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.00126782s FAIL (2025-08-07 21:40:13 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 30.001287698s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 26 out of 60 is failed (threshold=3) [..................................XXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.001319888s FAIL (2025-08-07 21:40:11 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.001340998s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 32 out of 60 is failed (threshold=3) [..........................X..XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.001251258s FAIL (2025-08-07 21:40:09 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== one uploading segments... Aug 7 21:38:01.579294227 (11h58m1.168639214s) [upload=12433]: Upload finished. err: context deadline exceeded Aug 7 21:38:32.979307013 (11h58m32.56865199s) [upload=12449]: Starting upload Aug 7 21:38:32.979360956 (11h58m32.568705943s) [upload=12449]: Done starting upload Aug 7 21:38:32.979501265 (11h58m32.568846252s) [upload=12449]: Uploading segments... Aug 7 21:38:32.979631284 (11h58m32.568976271s) [upload=12449]: Got next segment. Inline: false Aug 7 21:38:32.979691148 (11h58m32.569036135s) [upload=12449]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:38:32.979696198 (11h58m32.569041185s) [upload=12449]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:38:48.686817210 (11h58m48.276162267s) [seg_pos={0 0}, upload=12449]: Begin upload segment... Aug 7 21:38:48.686871564 (11h58m48.276216581s) [seg_pos={0 0}, upload=12449]: Done begin upload segment. Aug 7 21:38:48.686907212 (11h58m48.276252239s) [noise=true , address=104.36.82.2:4731 , piece_id=EA5DJOHUKB4JMDX56BNKNC3Z3SP5MLHLYN5PTJHOL4PC3DC2YZQA, seg_pos={0 0}, upload=12449]: Uploading piece... Aug 7 21:38:48.687437808 (11h58m48.276782795s) [noise=true , address=104.36.82.13:4185 , piece_id=GCSEQS5RYA7DIYXXEYRSVTCZWM4GII3CCOWDZXCWGK73LSPH6VDQ, seg_pos={0 0}, upload=12449]: Uploading piece... Aug 7 21:38:48.687923266 (11h58m48.277268253s) [noise=true , address=77.90.31.11:4554 , piece_id=Y5X5HMEMMOAD5QMICFFJFWSBHXSVZKTD54AP2NB4I2DJT42KJGVQ, seg_pos={0 0}, upload=12449]: Uploading piece... Aug 7 21:38:48.688304686 (11h58m48.277649673s) [noise=true , address=45.91.116.8:4045 , piece_id=JUV7DENMC32B4EJWTW6BJM4TEQRQ2U4FR7IFQAEJMCHRSV5JBXVQ, seg_pos={0 0}, upload=12449]: Uploading piece... Aug 7 21:38:48.688674213 (11h58m48.278019201s) [noise=true , address=104.36.82.15:4257 , piece_id=U7SFDCW5WBXVOCSJLGFOMXVNIMLQB7CY6BVPA2YUV63LBEC4IWAQ, seg_pos={0 0}, upload=12449]: Uploading piece... Aug 7 21:38:48.689020405 (11h58m48.278365392s) [noise=true , address=104.36.82.13:4186 , piece_id=XY22B7MTHZJ4MJFMJCTVDS4JAYQ3TWQDOPVMBUDZE3EF5VUAQTWQ, seg_pos={0 0}, upload=12449]: Uploading piece... Aug 7 21:38:48.689281786 (11h58m48.278626773s) [noise=true , address=104.36.82.10:4078 , piece_id=XYCM4QHQ7UU2S34Q5DUF7G5YGD2Z7KIDHHUMBRQNBEZUMD4GKNZQ, seg_pos={0 0}, upload=12449]: Uploading piece... Aug 7 21:38:48.689581169 (11h58m48.278926156s) [noise=true , address=45.91.116.5:4299 , piece_id=ED5YULTZVTYLCHQDFVZJIBUSEAYQUJMXGHHB7TTO7GQ5CSBBHYQA, seg_pos={0 0}, upload=12449]: Uploading piece... Aug 7 21:38:48.689893847 (11h58m48.279238824s) [noise=true , address=45.91.116.12:4225 , piece_id=INA2TNZ6KIYF45J4UFGRSLNRELTKISWF7IYIGPNXFDLLGZ7QVJWQ, seg_pos={0 0}, upload=12449]: Uploading piece... Aug 7 21:38:48.690256691 (11h58m48.279601678s) [noise=true , address=104.36.82.11:4114 , piece_id=QVVBKLBIFQC6RRFOJW7MM6AMHL2ZGLQAMH5ZE73XLBGY3VERYEVA, seg_pos={0 0}, upload=12449]: Uploading piece... Aug 7 21:38:48.690626529 (11h58m48.279971516s) [noise=true , address=45.91.116.10:4082 , piece_id=QX3QMQSXP5XBAWE4QY4P5A5QK2QXFBSV67YFPNIQV2BFIH2RVEKQ, seg_pos={0 0}, upload=12449]: Uploading piece... Aug 7 21:38:48.690901335 (11h58m48.280246322s) [noise=true , address=104.36.82.11:4113 , piece_id=2EGFZD54ZFDWU5SP6RQCC4YPP3USNQYBZNFSEI3V7CPMYLLB2A5Q, seg_pos={0 0}, upload=12449]: Uploading piece... Aug 7 21:38:48.691203833 (11h58m48.280548841s) [noise=true , address=77.90.31.12:4592 , piece_id=WW3UFNYQSC4MDLIZ2AJPTLTQN33HWQ2423TRFV5KEUNJ2R6XWKSQ, seg_pos={0 0}, upload=12449]: Uploading piece... Aug 7 21:38:48.691489149 (11h58m48.280834126s) [noise=true , address=77.90.31.21:4879 , piece_id=VB2RZ4HV4VI6WNNSHXZZRHZEAINPP5KU4LO6HVR2FLSGI5H3HP5A, seg_pos={0 0}, upload=12449]: Uploading piece... Aug 7 21:38:48.691733436 (11h58m48.281078424s) [noise=true , address=45.91.116.9:4263 , piece_id=EEBTJKDOQWVNQSWOUI47IY3RHKY5FDAMYQL4LJSUOS6GULR52TGA, seg_pos={0 0}, upload=12449]: Uploading piece... Aug 7 21:38:48.692068177 (11h58m48.281413164s) [noise=true , address=104.36.82.14:4221 , piece_id=W24SNKHE75CCO7XJFHUIH53M26MCEM7ILYZEGH672SQVK6ZCRBLQ, seg_pos={0 0}, upload=12449]: Uploading piece... Aug 7 21:38:50.513228864 (11h58m50.102573851s) [upload=12449]: Next returned nil segment Aug 7 21:38:50.513249563 (11h58m50.10259454s) [upload=12449]: Waiting for error group managing segments... Aug 7 21:38:50.810863968 (11h58m50.400208945s) [noise=true , address=104.36.82.2:4731 , piece_id=EA5DJOHUKB4JMDX56BNKNC3Z3SP5MLHLYN5PTJHOL4PC3DC2YZQA, seg_pos={0 0}, upload=12449]: Done uploading piece. err: <nil> Aug 7 21:38:50.873747961 (11h58m50.463092948s) [noise=true , address=104.36.82.11:4113 , piece_id=2EGFZD54ZFDWU5SP6RQCC4YPP3USNQYBZNFSEI3V7CPMYLLB2A5Q, seg_pos={0 0}, upload=12449]: Done uploading piece. err: <nil> Aug 7 21:38:50.944657320 (11h58m50.534002307s) [noise=true , address=45.91.116.8:4045 , piece_id=JUV7DENMC32B4EJWTW6BJM4TEQRQ2U4FR7IFQAEJMCHRSV5JBXVQ, seg_pos={0 0}, upload=12449]: Done uploading piece. err: <nil> Aug 7 21:38:50.972481134 (11h58m50.561826121s) [noise=true , address=104.36.82.14:4221 , piece_id=W24SNKHE75CCO7XJFHUIH53M26MCEM7ILYZEGH672SQVK6ZCRBLQ, seg_pos={0 0}, upload=12449]: Done uploading piece. err: <nil> Aug 7 21:38:50.980065425 (11h58m50.569410412s) [noise=true , address=104.36.82.13:4186 , piece_id=XY22B7MTHZJ4MJFMJCTVDS4JAYQ3TWQDOPVMBUDZE3EF5VUAQTWQ, seg_pos={0 0}, upload=12449]: Done uploading piece. err: <nil> Aug 7 21:38:50.981707937 (11h58m50.571052925s) [noise=true , address=104.36.82.11:4114 , piece_id=QVVBKLBIFQC6RRFOJW7MM6AMHL2ZGLQAMH5ZE73XLBGY3VERYEVA, seg_pos={0 0}, upload=12449]: Done uploading piece. err: <nil> Aug 7 21:38:51.023162491 (11h58m50.612507508s) [noise=true , address=104.36.82.13:4185 , piece_id=GCSEQS5RYA7DIYXXEYRSVTCZWM4GII3CCOWDZXCWGK73LSPH6VDQ, seg_pos={0 0}, upload=12449]: Done uploading piece. err: <nil> Aug 7 21:38:51.075895351 (11h58m50.665240338s) [noise=true , address=77.90.31.11:4554 , piece_id=Y5X5HMEMMOAD5QMICFFJFWSBHXSVZKTD54AP2NB4I2DJT42KJGVQ, seg_pos={0 0}, upload=12449]: Done uploading piece. err: <nil> Aug 7 21:38:51.095722626 (11h58m50.685067613s) [noise=true , address=45.91.116.5:4299 , piece_id=ED5YULTZVTYLCHQDFVZJIBUSEAYQUJMXGHHB7TTO7GQ5CSBBHYQA, seg_pos={0 0}, upload=12449]: Done uploading piece. err: <nil> Aug 7 21:38:51.098735141 (11h58m50.688080128s) [noise=true , address=104.36.82.15:4257 , piece_id=U7SFDCW5WBXVOCSJLGFOMXVNIMLQB7CY6BVPA2YUV63LBEC4IWAQ, seg_pos={0 0}, upload=12449]: Done uploading piece. err: <nil> Aug 7 21:38:51.107956113 (11h58m50.6973011s) [noise=true , address=77.90.31.21:4879 , piece_id=VB2RZ4HV4VI6WNNSHXZZRHZEAINPP5KU4LO6HVR2FLSGI5H3HP5A, seg_pos={0 0}, upload=12449]: Done uploading piece. err: <nil> Aug 7 21:38:51.133243133 (11h58m50.72258812s) [noise=true , address=77.90.31.12:4592 , piece_id=WW3UFNYQSC4MDLIZ2AJPTLTQN33HWQ2423TRFV5KEUNJ2R6XWKSQ, seg_pos={0 0}, upload=12449]: Done uploading piece. err: <nil> Aug 7 21:38:51.249092953 (11h58m50.83843794s) [noise=true , address=45.91.116.9:4263 , piece_id=EEBTJKDOQWVNQSWOUI47IY3RHKY5FDAMYQL4LJSUOS6GULR52TGA, seg_pos={0 0}, upload=12449]: Done uploading piece. err: <nil> Aug 7 21:38:51.249118061 (11h58m50.838463048s) [seg_pos={0 0}, upload=12449]: Segment reached optimal threshold of 13 pieces. Aug 7 21:38:51.249306762 (11h58m50.838651749s) [noise=true , address=45.91.116.12:4225 , piece_id=INA2TNZ6KIYF45J4UFGRSLNRELTKISWF7IYIGPNXFDLLGZ7QVJWQ, seg_pos={0 0}, upload=12449]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: expected piece hash; context canceled; EOF Aug 7 21:38:51.249323974 (11h58m50.838668961s) [noise=true , address=45.91.116.10:4082 , piece_id=QX3QMQSXP5XBAWE4QY4P5A5QK2QXFBSV67YFPNIQV2BFIH2RVEKQ, seg_pos={0 0}, upload=12449]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Aug 7 21:38:51.249493108 (11h58m50.838838095s) [noise=true , address=104.36.82.10:4078 , piece_id=XYCM4QHQ7UU2S34Q5DUF7G5YGD2Z7KIDHHUMBRQNBEZUMD4GKNZQ, seg_pos={0 0}, upload=12449]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr): protocol: context canceled; context canceled Aug 7 21:38:51.249505242 (11h58m50.838850229s) [seg_pos={0 0}, upload=12449]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 7 21:38:51.249581758 (11h58m50.838926735s) [upload=12449]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 21:38:51.249586157 (11h58m50.838931144s) [upload=12449]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:39:38.936884982 (11h59m38.526229969s) [upload=12449]: Done uploading segments... Aug 7 21:39:38.936908165 (11h59m38.526253152s) [upload=12449]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:39:39.835057740 (11h59m39.424402717s) [upload=12467]: Starting upload Aug 7 21:39:39.835133405 (11h59m39.424478382s) [upload=12467]: Done starting upload Aug 7 21:39:39.835203008 (11h59m39.424547995s) [upload=12467]: Uploading segments... Aug 7 21:39:39.835296918 (11h59m39.424641895s) [upload=12467]: Got next segment. Inline: false Aug 7 21:39:39.835322778 (11h59m39.424667765s) [upload=12467]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:39:39.835327146 (11h59m39.424672133s) [upload=12467]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:40:09.178127344 (12h0m8.767472331s) [upload=12467]: Next segment err: context deadline exceeded Aug 7 21:40:09.178178722 (12h0m8.767523709s) [upload=12467]: Done uploading segments... Aug 7 21:40:09.178192309 (12h0m8.767537296s) [upload=12467]: Upload finished. err: context deadline exceeded Duration: 30.001261997s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway)

checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)