Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 20 out of 60 is failed (threshold=3) [........................................XXXXXXXXXXXXXXXXXXXX] duration: 30.000787999s FAIL (2025-08-07 21:29:54 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 556593194s) [upload=12182]: Uploading segments... Aug 7 21:21:26.967275329 (11h41m26.556620316s) [upload=12182]: Got next segment. Inline: true Aug 7 21:21:26.967278815 (11h41m26.556623802s) [upload=12182]: Waiting for error group managing segments... Aug 7 21:21:26.967311758 (11h41m26.556656745s) [upload=12182]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:21:26.967315786 (11h41m26.556660773s) [upload=12182]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:21:26.967318120 (11h41m26.556663107s) [upload=12182]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:21:56.309770442 (11h41m55.899115429s) [upload=12182]: Done uploading segments... Aug 7 21:21:56.309792865 (11h41m55.899137852s) [upload=12182]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:22:21.925019495 (11h42m21.514364572s) [upload=12195]: Starting upload Aug 7 21:22:21.925098336 (11h42m21.514443323s) [upload=12195]: Done starting upload Aug 7 21:22:21.925142180 (11h42m21.514487167s) [upload=12195]: Uploading segments... Aug 7 21:22:21.925175574 (11h42m21.514520561s) [upload=12195]: Got next segment. Inline: true Aug 7 21:22:21.925179491 (11h42m21.514524478s) [upload=12195]: Waiting for error group managing segments... Aug 7 21:22:21.925210671 (11h42m21.514555648s) [upload=12195]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:22:21.925214048 (11h42m21.514559035s) [upload=12195]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:22:21.925216302 (11h42m21.51456129s) [upload=12195]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:22:51.283998863 (11h42m50.87334385s) [upload=12195]: Done uploading segments... Aug 7 21:22:51.284018922 (11h42m50.873363899s) [upload=12195]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:23:36.916300477 (11h43m36.505645464s) [upload=12212]: Starting upload Aug 7 21:23:36.916350923 (11h43m36.50569591s) [upload=12212]: Done starting upload Aug 7 21:23:36.916395318 (11h43m36.505740295s) [upload=12212]: Uploading segments... Aug 7 21:23:36.916436557 (11h43m36.505781544s) [upload=12212]: Got next segment. Inline: true Aug 7 21:23:36.916439863 (11h43m36.50578484s) [upload=12212]: Waiting for error group managing segments... Aug 7 21:23:36.916471534 (11h43m36.505816511s) [upload=12212]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:23:36.916475351 (11h43m36.505820338s) [upload=12212]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:23:36.916477665 (11h43m36.505822652s) [upload=12212]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:24:06.267030467 (11h44m5.856375454s) [upload=12212]: Done uploading segments... Aug 7 21:24:06.267052209 (11h44m5.856397196s) [upload=12212]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:24:29.996857636 (11h44m29.586202623s) [upload=12224]: Starting upload Aug 7 21:24:29.996913823 (11h44m29.5862588s) [upload=12224]: Done starting upload Aug 7 21:24:29.996961495 (11h44m29.586306482s) [upload=12224]: Uploading segments... Aug 7 21:24:29.996996181 (11h44m29.586341158s) [upload=12224]: Got next segment. Inline: true Aug 7 21:24:29.996999788 (11h44m29.586344765s) [upload=12224]: Waiting for error group managing segments... Aug 7 21:24:29.997033352 (11h44m29.586378329s) [upload=12224]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:24:29.997037590 (11h44m29.586382577s) [upload=12224]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:24:29.997040075 (11h44m29.586385062s) [upload=12224]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:24:59.358108170 (11h44m58.947453157s) [upload=12224]: Done uploading segments... Aug 7 21:24:59.358131014 (11h44m58.947476001s) [upload=12224]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:25:00.224254754 (11h44m59.813599741s) [upload=12230]: Starting upload Aug 7 21:25:00.224330690 (11h44m59.813675677s) [upload=12230]: Done starting upload Aug 7 21:25:00.224406304 (11h44m59.813751311s) [upload=12230]: Uploading segments... Aug 7 21:25:00.224533207 (11h44m59.813878204s) [upload=12230]: Got next segment. Inline: true Aug 7 21:25:00.224541352 (11h44m59.813886339s) [upload=12230]: Waiting for error group managing segments... Aug 7 21:25:00.224626114 (11h44m59.813971121s) [upload=12230]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:25:00.224634080 (11h44m59.813979107s) [upload=12230]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:25:00.224638929 (11h44m59.813983937s) [upload=12230]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:25:29.580431600 (11h45m29.169776597s) [upload=12230]: Done uploading segments... Aug 7 21:25:29.580456288 (11h45m29.169801275s) [upload=12230]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:25:38.048208784 (11h45m37.637553761s) [upload=12241]: Starting upload Aug 7 21:25:38.048265823 (11h45m37.63761081s) [upload=12241]: Done starting upload Aug 7 21:25:38.048311521 (11h45m37.637656508s) [upload=12241]: Uploading segments... Aug 7 21:25:38.048347439 (11h45m37.637692426s) [upload=12241]: Got next segment. Inline: true Aug 7 21:25:38.048350867 (11h45m37.637695854s) [upload=12241]: Waiting for error group managing segments... Aug 7 21:25:38.048383188 (11h45m37.637728186s) [upload=12241]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:25:38.048387396 (11h45m37.637732383s) [upload=12241]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:25:38.048389881 (11h45m37.637734868s) [upload=12241]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:26:07.388744771 (11h46m6.978089758s) [upload=12241]: Done uploading segments... Aug 7 21:26:07.388753889 (11h46m6.978098876s) [upload=12241]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:26:51.633559976 (11h46m51.222904953s) [upload=12259]: Starting upload Aug 7 21:26:51.633607808 (11h46m51.222952795s) [upload=12259]: Done starting upload Aug 7 21:26:51.633657383 (11h46m51.22300237s) [upload=12259]: Uploading segments... Aug 7 21:26:51.633689063 (11h46m51.22303405s) [upload=12259]: Got next segment. Inline: true Aug 7 21:26:51.633692660 (11h46m51.223037647s) [upload=12259]: Waiting for error group managing segments... Aug 7 21:26:51.633727035 (11h46m51.223072012s) [upload=12259]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:26:51.633730782 (11h46m51.22307577s) [upload=12259]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:26:51.633733327 (11h46m51.223078314s) [upload=12259]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:27:20.991495967 (11h47m20.580840944s) [upload=12259]: Done uploading segments... Aug 7 21:27:20.991524641 (11h47m20.580869628s) [upload=12259]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:27:32.297072953 (11h47m31.88641795s) [upload=12271]: Starting upload Aug 7 21:27:32.297139892 (11h47m31.886484889s) [upload=12271]: Done starting upload Aug 7 21:27:32.297212079 (11h47m31.886557066s) [upload=12271]: Uploading segments... Aug 7 21:27:32.297260662 (11h47m31.886605649s) [upload=12271]: Got next segment. Inline: true Aug 7 21:27:32.297264640 (11h47m31.886609627s) [upload=12271]: Waiting for error group managing segments... Aug 7 21:27:32.297341267 (11h47m31.886686254s) [upload=12271]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:27:32.297345635 (11h47m31.886690622s) [upload=12271]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:27:32.297348029 (11h47m31.886693016s) [upload=12271]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:28:01.648543031 (11h48m1.237888018s) [upload=12271]: Done uploading segments... Aug 7 21:28:01.648565273 (11h48m1.23791026s) [upload=12271]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:28:45.542334506 (11h48m45.131679483s) [upload=12290]: Starting upload Aug 7 21:28:45.542390844 (11h48m45.131735831s) [upload=12290]: Done starting upload Aug 7 21:28:45.542466819 (11h48m45.131811806s) [upload=12290]: Uploading segments... Aug 7 21:28:45.542586358 (11h48m45.131931345s) [upload=12290]: Got next segment. Inline: true Aug 7 21:28:45.542591848 (11h48m45.131936835s) [upload=12290]: Waiting for error group managing segments... Aug 7 21:28:45.542637005 (11h48m45.131981982s) [upload=12290]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:28:45.542642495 (11h48m45.131987482s) [upload=12290]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:28:45.542645010 (11h48m45.131989987s) [upload=12290]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:29:14.895678169 (11h49m14.485023156s) [upload=12290]: Done uploading segments... Aug 7 21:29:14.895732473 (11h49m14.48507746s) [upload=12290]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:29:24.822220224 (11h49m24.411565212s) [upload=12297]: Starting upload Aug 7 21:29:24.822268116 (11h49m24.411613093s) [upload=12297]: Done starting upload Aug 7 21:29:24.822310567 (11h49m24.411655554s) [upload=12297]: Uploading segments... Aug 7 21:29:24.822336567 (11h49m24.411681544s) [upload=12297]: Got next segment. Inline: true Aug 7 21:29:24.822340274 (11h49m24.411685261s) [upload=12297]: Waiting for error group managing segments... Aug 7 21:29:24.822369891 (11h49m24.411714878s) [upload=12297]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:29:24.822374179 (11h49m24.411719166s) [upload=12297]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:29:24.822376544 (11h49m24.411721531s) [upload=12297]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:29:54.171228417 (11h49m53.760573394s) [upload=12297]: Done uploading segments... Aug 7 21:29:54.171248837 (11h49m53.760593824s) [upload=12297]: Upload finished. err: metaclient: context deadline exceeded Duration: 30.000809429s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 12 out of 60 is failed (threshold=3) [................................................XXXXXXXXXXXX] duration: 30.001597346s FAIL (2025-08-07 21:29:53 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 30.001609921s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 17 out of 60 is failed (threshold=3) [.........................................X..XXXXXXXXXXXXXXXX] duration: 30.000989945s FAIL (2025-08-07 21:29:50 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 30.001010295s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 16 out of 60 is failed (threshold=3) [............................................XXXXXXXXXXXXXXXX] duration: 1m0.000484384s FAIL (2025-08-07 21:29:44 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.000495336s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 15 out of 60 is failed (threshold=3) [.............................................XXXXXXXXXXXXXXX] duration: 1m0.005939227s FAIL (2025-08-07 21:29:41 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== pload=12106]: Next segment err: context deadline exceeded Aug 7 21:17:45.371471144 (11h37m44.960816142s) [upload=12106]: Done uploading segments... Aug 7 21:17:45.371492135 (11h37m44.960837122s) [upload=12106]: Upload finished. err: context deadline exceeded Aug 7 21:17:49.811942678 (11h37m49.401287665s) [upload=12122]: Starting upload Aug 7 21:17:49.812064030 (11h37m49.401409007s) [upload=12122]: Done starting upload Aug 7 21:17:49.812137160 (11h37m49.401482147s) [upload=12122]: Uploading segments... Aug 7 21:17:49.812396918 (11h37m49.401741895s) [upload=12122]: Got next segment. Inline: false Aug 7 21:17:49.812426735 (11h37m49.401771722s) [upload=12122]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:17:49.812430862 (11h37m49.401775849s) [upload=12122]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:18:49.169738786 (11h38m48.759083813s) [upload=12122]: Next segment err: context deadline exceeded Aug 7 21:18:49.169939078 (11h38m48.759284086s) [upload=12122]: Done uploading segments... Aug 7 21:18:49.169962594 (11h38m48.759307591s) [upload=12122]: Upload finished. err: context deadline exceeded Aug 7 21:18:50.060561583 (11h38m49.64990661s) [upload=12134]: Starting upload Aug 7 21:18:50.060675751 (11h38m49.650020808s) [upload=12134]: Done starting upload Aug 7 21:18:50.060772135 (11h38m49.650117122s) [upload=12134]: Uploading segments... Aug 7 21:18:50.061042412 (11h38m49.650387409s) [upload=12134]: Got next segment. Inline: false Aug 7 21:18:50.061084393 (11h38m49.65042944s) [upload=12134]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:18:50.061091987 (11h38m49.650437004s) [upload=12134]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:19:49.417978828 (11h39m49.007323815s) [upload=12134]: Next segment err: context deadline exceeded Aug 7 21:19:49.418015799 (11h39m49.007360786s) [upload=12134]: Done uploading segments... Aug 7 21:19:49.418037982 (11h39m49.007382959s) [upload=12134]: Upload finished. err: context deadline exceeded Aug 7 21:20:02.821763948 (11h40m2.411108945s) [upload=12155]: Starting upload Aug 7 21:20:02.821848960 (11h40m2.411193957s) [upload=12155]: Done starting upload Aug 7 21:20:02.821939103 (11h40m2.41128409s) [upload=12155]: Uploading segments... Aug 7 21:20:02.822080344 (11h40m2.411425331s) [upload=12155]: Got next segment. Inline: false Aug 7 21:20:02.822124998 (11h40m2.411469985s) [upload=12155]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:20:02.822132784 (11h40m2.411477771s) [upload=12155]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:21:02.173704394 (11h41m1.763049381s) [upload=12155]: Next segment err: context deadline exceeded Aug 7 21:21:02.173855945 (11h41m1.763200932s) [upload=12155]: Done uploading segments... Aug 7 21:21:02.173881643 (11h41m1.76322663s) [upload=12155]: Upload finished. err: context deadline exceeded Aug 7 21:21:03.041754970 (11h41m2.631099958s) [upload=12171]: Starting upload Aug 7 21:21:03.041871343 (11h41m2.63121633s) [upload=12171]: Done starting upload Aug 7 21:21:03.041935847 (11h41m2.631280834s) [upload=12171]: Uploading segments... Aug 7 21:21:03.042038162 (11h41m2.631383149s) [upload=12171]: Got next segment. Inline: false Aug 7 21:21:03.042070074 (11h41m2.631415061s) [upload=12171]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:21:03.042074442 (11h41m2.631419429s) [upload=12171]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:22:02.399093104 (11h42m1.988438091s) [upload=12171]: Next segment err: context deadline exceeded Aug 7 21:22:02.399277657 (11h42m1.988622644s) [upload=12171]: Done uploading segments... Aug 7 21:22:02.399305961 (11h42m1.988650948s) [upload=12171]: Upload finished. err: context deadline exceeded Aug 7 21:22:03.257202341 (11h42m2.846547328s) [upload=12189]: Starting upload Aug 7 21:22:03.257267245 (11h42m2.846612232s) [upload=12189]: Done starting upload Aug 7 21:22:03.257356857 (11h42m2.846701844s) [upload=12189]: Uploading segments... Aug 7 21:22:03.257444765 (11h42m2.846789742s) [upload=12189]: Got next segment. Inline: false Aug 7 21:22:03.257470033 (11h42m2.84681501s) [upload=12189]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:22:03.257474171 (11h42m2.846819158s) [upload=12189]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:23:02.611478992 (11h43m2.20082398s) [upload=12189]: Next segment err: context deadline exceeded Aug 7 21:23:02.611504952 (11h43m2.200849939s) [upload=12189]: Done uploading segments... Aug 7 21:23:02.611523989 (11h43m2.200868976s) [upload=12189]: Upload finished. err: context deadline exceeded Aug 7 21:23:11.364660428 (11h43m10.954005435s) [upload=12204]: Starting upload Aug 7 21:23:11.364761051 (11h43m10.954106038s) [upload=12204]: Done starting upload Aug 7 21:23:11.364863036 (11h43m10.954208023s) [upload=12204]: Uploading segments... Aug 7 21:23:11.364973286 (11h43m10.954318264s) [upload=12204]: Got next segment. Inline: false Aug 7 21:23:11.365036508 (11h43m10.954381495s) [upload=12204]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:23:11.365041447 (11h43m10.954386434s) [upload=12204]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:24:10.722815877 (11h44m10.312160864s) [upload=12204]: Next segment err: context deadline exceeded Aug 7 21:24:10.722857978 (11h44m10.312202965s) [upload=12204]: Done uploading segments... Aug 7 21:24:10.722871453 (11h44m10.31221644s) [upload=12204]: Upload finished. err: context deadline exceeded Aug 7 21:24:23.410577309 (11h44m22.999922296s) [upload=12220]: Starting upload Aug 7 21:24:23.410633838 (11h44m22.999978825s) [upload=12220]: Done starting upload Aug 7 21:24:23.410696377 (11h44m23.000041364s) [upload=12220]: Uploading segments... Aug 7 21:24:23.410808262 (11h44m23.000153239s) [upload=12220]: Got next segment. Inline: false Aug 7 21:24:23.410833469 (11h44m23.000178456s) [upload=12220]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:24:23.410837017 (11h44m23.000182004s) [upload=12220]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:25:22.762515918 (11h45m22.351860905s) [upload=12220]: Next segment err: context deadline exceeded Aug 7 21:25:22.762663662 (11h45m22.352008639s) [upload=12220]: Done uploading segments... Aug 7 21:25:22.762679401 (11h45m22.352024378s) [upload=12220]: Upload finished. err: context deadline exceeded Aug 7 21:25:23.709743833 (11h45m23.29908882s) [upload=12237]: Starting upload Aug 7 21:25:23.709835399 (11h45m23.299180386s) [upload=12237]: Done starting upload Aug 7 21:25:23.709934688 (11h45m23.299279675s) [upload=12237]: Uploading segments... Aug 7 21:25:23.710036063 (11h45m23.29938105s) [upload=12237]: Got next segment. Inline: false Aug 7 21:25:23.710068875 (11h45m23.299413862s) [upload=12237]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:25:23.710072732 (11h45m23.299417719s) [upload=12237]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:26:23.059097010 (11h46m22.648441997s) [upload=12237]: Next segment err: context deadline exceeded Aug 7 21:26:23.059154440 (11h46m22.648499427s) [upload=12237]: Done uploading segments... Aug 7 21:26:23.059167756 (11h46m22.648512743s) [upload=12237]: Upload finished. err: context deadline exceeded Aug 7 21:26:32.555948975 (11h46m32.145293952s) [upload=12256]: Starting upload Aug 7 21:26:32.555997727 (11h46m32.145342704s) [upload=12256]: Done starting upload Aug 7 21:26:32.556084444 (11h46m32.145429421s) [upload=12256]: Uploading segments... Aug 7 21:26:32.556166260 (11h46m32.145511247s) [upload=12256]: Got next segment. Inline: false Aug 7 21:26:32.556194715 (11h46m32.145539702s) [upload=12256]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:26:32.556198572 (11h46m32.145543559s) [upload=12256]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:27:31.908976804 (11h47m31.498321791s) [upload=12256]: Next segment err: context deadline exceeded Aug 7 21:27:31.909049884 (11h47m31.498394861s) [upload=12256]: Done uploading segments... Aug 7 21:27:31.909064964 (11h47m31.498409951s) [upload=12256]: Upload finished. err: context deadline exceeded Aug 7 21:27:32.799954317 (11h47m32.389299304s) [upload=12272]: Starting upload Aug 7 21:27:32.800028559 (11h47m32.389373546s) [upload=12272]: Done starting upload Aug 7 21:27:32.800168577 (11h47m32.389513564s) [upload=12272]: Uploading segments... Aug 7 21:27:32.800449224 (11h47m32.389794211s) [upload=12272]: Got next segment. Inline: false Aug 7 21:27:32.800502656 (11h47m32.389847643s) [upload=12272]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:27:32.800507295 (11h47m32.389852272s) [upload=12272]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:28:32.145562627 (11h48m31.734907614s) [upload=12272]: Next segment err: context deadline exceeded Aug 7 21:28:32.145781085 (11h48m31.735126132s) [upload=12272]: Done uploading segments... Aug 7 21:28:32.145837483 (11h48m31.73518251s) [upload=12272]: Upload finished. err: context deadline exceeded Aug 7 21:28:42.274062299 (11h48m41.863407286s) [upload=12287]: Starting upload Aug 7 21:28:42.274122003 (11h48m41.86346699s) [upload=12287]: Done starting upload Aug 7 21:28:42.274224129 (11h48m41.863569116s) [upload=12287]: Uploading segments... Aug 7 21:28:42.274354589 (11h48m41.863699576s) [upload=12287]: Got next segment. Inline: false Aug 7 21:28:42.274393773 (11h48m41.86373876s) [upload=12287]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:28:42.274397991 (11h48m41.863742978s) [upload=12287]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:29:41.624421841 (11h49m41.213766828s) [upload=12287]: Next segment err: context deadline exceeded Aug 7 21:29:41.624663694 (11h49m41.214008681s) [upload=12287]: Done uploading segments... Aug 7 21:29:41.624678522 (11h49m41.214023509s) [upload=12287]: Upload finished. err: context deadline exceeded Duration: 1m0.005958223s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway) antiblip: 3 out of 60 is failed (threshold=3) [.................................................X....X.X...] duration: 769.891592ms remote: "tcp:45.38.24.2:443" target: https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410 FAIL (2025-08-07 21:29:41 UTC) orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": context deadline exceeded Duration: 769.90657ms edge gateway vhost us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [.........................................XXXXXXXXXXXXXXXXXXX] duration: 30.00051258s FAIL (2025-08-07 21:29:40 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.000532868s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 7 out of 60 is failed (threshold=3) [..XX...........X.......................X.........X......X..X] duration: 4.239574884s FAIL (2025-08-07 21:29:29 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/4M") ======LOGS====== item: pb.BatchRequestItem_SegmentBegin Aug 7 21:29:25.857659319 (11h49m25.447004296s) [seg_pos={0 0}, upload=12298]: Begin upload segment... Aug 7 21:29:25.857698454 (11h49m25.447043441s) [seg_pos={0 0}, upload=12298]: Done begin upload segment. Aug 7 21:29:25.857736887 (11h49m25.447081864s) [noise=true , address=104.36.82.35:4612 , piece_id=U5WPP5IL75AVFHDGGDB7RLFKNDVVQURKFY23AFG4C52V4YV2ABLQ, seg_pos={0 0}, upload=12298]: Uploading piece... Aug 7 21:29:25.858116173 (11h49m25.44746116s) [noise=true , address=64.49.13.139:4469 , piece_id=M3PL6MGQMOJYJHHRUSDCCHIYP7XOZLYW3IY6O4UIJFYYPER5FTYA, seg_pos={0 0}, upload=12298]: Uploading piece... Aug 7 21:29:25.858376170 (11h49m25.447721147s) [noise=true , address=104.36.82.61:4403 , piece_id=CHFTEE4XZ47N4P4CSLWE6ZPXZ56YYU4SMBSLEQHNH6TZZCVRTFQA, seg_pos={0 0}, upload=12298]: Uploading piece... Aug 7 21:29:25.858676685 (11h49m25.448021662s) [noise=true , address=45.91.116.6:4005 , piece_id=RCN2IDUW4WZA6QNPYHM2E5XM6O75NIC5RUX6KXMOGMWNV2QVAJIQ, seg_pos={0 0}, upload=12298]: Uploading piece... Aug 7 21:29:25.858915152 (11h49m25.448260139s) [noise=true , address=79.127.205.233:4625 , piece_id=KLCUPCY7XM3W3P5RY44CW4QXWMVSTUAZJMIVRF2VWZTIOBBQ6VFQ, seg_pos={0 0}, upload=12298]: Uploading piece... Aug 7 21:29:25.859163256 (11h49m25.448508243s) [noise=true , address=104.36.82.45:4932 , piece_id=25R2X3COEKXS3B3KGM3V33CSIB7XDJYUY5NI3EQ6OVKMPFKZPWHA, seg_pos={0 0}, upload=12298]: Uploading piece... Aug 7 21:29:25.859388758 (11h49m25.448733735s) [noise=true , address=104.36.82.9:4032 , piece_id=UNNJDD7KMLRSVQQOJTV2ORP25CQK4KEWKSAP2ETUHYB4C2QWCZAA, seg_pos={0 0}, upload=12298]: Uploading piece... Aug 7 21:29:25.859588490 (11h49m25.448933477s) [noise=true , address=104.36.82.59:4324 , piece_id=P32XRROGC5HCWJHQJ6U3C2AYPK4M2QVTURJLAR3MIX2BLYOQQBOA, seg_pos={0 0}, upload=12298]: Uploading piece... Aug 7 21:29:25.859768514 (11h49m25.449113491s) [noise=true , address=104.36.82.60:4344 , piece_id=J4HBJAD6EBHQTDYMPCHOCR2QXYCW757LRCSB37FZ4EPFYLRG3BKQ, seg_pos={0 0}, upload=12298]: Uploading piece... Aug 7 21:29:25.861899992 (11h49m25.451244979s) [noise=true , address=104.36.82.16:4300 , piece_id=7MYLVBQJW3UIDKHB3FEE5BYU6XL4Q6BNM2D4ODNKT6IZFC23IAGA, seg_pos={0 0}, upload=12298]: Uploading piece... Aug 7 21:29:25.862982253 (11h49m25.45232724s) [noise=true , address=104.36.82.18:4373 , piece_id=OE3PSGXX5XLV4WUHS3B3IJ3SRAEWM7XNGSJUOE3ZUGR2DZPIT2VA, seg_pos={0 0}, upload=12298]: Uploading piece... Aug 7 21:29:25.863485917 (11h49m25.452830914s) [noise=true , address=64.49.13.140:4510 , piece_id=ZWUGZSDGNXK7PMG7L4CSIJOZ2Y2DGFBQJOMJXWT7FWVHUCMGFHBA, seg_pos={0 0}, upload=12298]: Uploading piece... Aug 7 21:29:25.863967227 (11h49m25.453312214s) [noise=true , address=104.36.82.44:4871 , piece_id=2KTOQN3PLZYK3II27DMUOMJ7XGD3EWZUPUFLMPEYGBNQN37OF6IA, seg_pos={0 0}, upload=12298]: Uploading piece... Aug 7 21:29:25.864375699 (11h49m25.453720686s) [noise=true , address=156.146.43.225:4449 , piece_id=NUURTOFKDGY2QCTR7UFE35CBOBVI2KPFR33RK4MWPURVS4WEBFAQ, seg_pos={0 0}, upload=12298]: Uploading piece... Aug 7 21:29:25.864746759 (11h49m25.454091746s) [noise=true , address=104.36.82.39:4727 , piece_id=2SVEE2SU56DR5U436F7DAYIW6ORGTZLYOUHFK6RPOJ7H3PKSIDPA, seg_pos={0 0}, upload=12298]: Uploading piece... Aug 7 21:29:25.865111086 (11h49m25.454456073s) [noise=true , address=104.36.82.3:4773 , piece_id=NTSJYDBRHNAJVH7MY2LUPLRGMGOHMFIILWR7OW47WUWHKSEDE54Q, seg_pos={0 0}, upload=12298]: Uploading piece... Aug 7 21:29:25.865481785 (11h49m25.454826772s) [noise=true , address=79.127.205.234:4585 , piece_id=2757TS4CR7SO3UT77TOTMBXTJ2GZIG7HJLHQYLX5UUQR5F624OSA, seg_pos={0 0}, upload=12298]: Uploading piece... Aug 7 21:29:25.865876501 (11h49m25.455221488s) [noise=true , address=104.36.82.46:4504 , piece_id=II3WOQ42IDFMPOYSSXYXJ5OCLY6ASVSXJ5VBQ7IM5SLYOCFXFKGQ, seg_pos={0 0}, upload=12298]: Uploading piece... Aug 7 21:29:26.957286373 (11h49m26.54663135s) [upload=12298]: Next returned nil segment Aug 7 21:29:26.957302725 (11h49m26.546647712s) [upload=12298]: Waiting for error group managing segments... Aug 7 21:29:27.242773775 (11h49m26.832118762s) [noise=true , address=64.49.13.140:4510 , piece_id=ZWUGZSDGNXK7PMG7L4CSIJOZ2Y2DGFBQJOMJXWT7FWVHUCMGFHBA, seg_pos={0 0}, upload=12298]: Done uploading piece. err: <nil> Aug 7 21:29:27.254906699 (11h49m26.844251686s) [noise=true , address=79.127.205.234:4585 , piece_id=2757TS4CR7SO3UT77TOTMBXTJ2GZIG7HJLHQYLX5UUQR5F624OSA, seg_pos={0 0}, upload=12298]: Done uploading piece. err: <nil> Aug 7 21:29:27.297381664 (11h49m26.886726651s) [noise=true , address=79.127.205.233:4625 , piece_id=KLCUPCY7XM3W3P5RY44CW4QXWMVSTUAZJMIVRF2VWZTIOBBQ6VFQ, seg_pos={0 0}, upload=12298]: Done uploading piece. err: <nil> Aug 7 21:29:27.304377109 (11h49m26.893722096s) [noise=true , address=64.49.13.139:4469 , piece_id=M3PL6MGQMOJYJHHRUSDCCHIYP7XOZLYW3IY6O4UIJFYYPER5FTYA, seg_pos={0 0}, upload=12298]: Done uploading piece. err: <nil> Aug 7 21:29:27.751965933 (11h49m27.34131091s) [noise=true , address=156.146.43.225:4449 , piece_id=NUURTOFKDGY2QCTR7UFE35CBOBVI2KPFR33RK4MWPURVS4WEBFAQ, seg_pos={0 0}, upload=12298]: Done uploading piece. err: <nil> Aug 7 21:29:27.811346684 (11h49m27.400691671s) [noise=true , address=104.36.82.35:4612 , piece_id=U5WPP5IL75AVFHDGGDB7RLFKNDVVQURKFY23AFG4C52V4YV2ABLQ, seg_pos={0 0}, upload=12298]: Done uploading piece. err: <nil> Aug 7 21:29:27.871486808 (11h49m27.460831795s) [noise=true , address=104.36.82.46:4504 , piece_id=II3WOQ42IDFMPOYSSXYXJ5OCLY6ASVSXJ5VBQ7IM5SLYOCFXFKGQ, seg_pos={0 0}, upload=12298]: Done uploading piece. err: <nil> Aug 7 21:29:27.875300604 (11h49m27.464645591s) [noise=true , address=104.36.82.45:4932 , piece_id=25R2X3COEKXS3B3KGM3V33CSIB7XDJYUY5NI3EQ6OVKMPFKZPWHA, seg_pos={0 0}, upload=12298]: Done uploading piece. err: <nil> Aug 7 21:29:27.932817469 (11h49m27.522162456s) [noise=true , address=45.91.116.6:4005 , piece_id=RCN2IDUW4WZA6QNPYHM2E5XM6O75NIC5RUX6KXMOGMWNV2QVAJIQ, seg_pos={0 0}, upload=12298]: Done uploading piece. err: <nil> Aug 7 21:29:27.938530751 (11h49m27.527875768s) [noise=true , address=104.36.82.18:4373 , piece_id=OE3PSGXX5XLV4WUHS3B3IJ3SRAEWM7XNGSJUOE3ZUGR2DZPIT2VA, seg_pos={0 0}, upload=12298]: Done uploading piece. err: <nil> Aug 7 21:29:27.940085755 (11h49m27.529430742s) [noise=true , address=104.36.82.44:4871 , piece_id=2KTOQN3PLZYK3II27DMUOMJ7XGD3EWZUPUFLMPEYGBNQN37OF6IA, seg_pos={0 0}, upload=12298]: Done uploading piece. err: <nil> Aug 7 21:29:27.961921222 (11h49m27.551266209s) [noise=true , address=104.36.82.3:4773 , piece_id=NTSJYDBRHNAJVH7MY2LUPLRGMGOHMFIILWR7OW47WUWHKSEDE54Q, seg_pos={0 0}, upload=12298]: Done uploading piece. err: <nil> Aug 7 21:29:27.984789475 (11h49m27.574134462s) [noise=true , address=104.36.82.16:4300 , piece_id=7MYLVBQJW3UIDKHB3FEE5BYU6XL4Q6BNM2D4ODNKT6IZFC23IAGA, seg_pos={0 0}, upload=12298]: Done uploading piece. err: <nil> Aug 7 21:29:28.004349759 (11h49m27.593694746s) [noise=true , address=104.36.82.59:4324 , piece_id=P32XRROGC5HCWJHQJ6U3C2AYPK4M2QVTURJLAR3MIX2BLYOQQBOA, seg_pos={0 0}, upload=12298]: Done uploading piece. err: <nil> Aug 7 21:29:28.004368014 (11h49m27.593713001s) [seg_pos={0 0}, upload=12298]: Segment reached optimal threshold of 14 pieces. Aug 7 21:29:28.004510657 (11h49m27.593855654s) [noise=true , address=104.36.82.60:4344 , piece_id=J4HBJAD6EBHQTDYMPCHOCR2QXYCW757LRCSB37FZ4EPFYLRG3BKQ, seg_pos={0 0}, upload=12298]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127WWD5hr1ZBkRLCHz3jb7kpqqnxPS6VuZ1MyQDmitFbhTSbJNK): protocol: expected piece hash; context canceled; EOF Aug 7 21:29:28.004949036 (11h49m27.594294063s) [noise=true , address=104.36.82.61:4403 , piece_id=CHFTEE4XZ47N4P4CSLWE6ZPXZ56YYU4SMBSLEQHNH6TZZCVRTFQA, seg_pos={0 0}, upload=12298]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Lhmif97DEkfhkR7CoWuz54524kk9LWpsvaVLoMRRtKSHMe1pg): protocol: write tcp 188.166.225.170:54274->104.36.82.61:4403: use of closed network connection; write tcp 188.166.225.170:54274->104.36.82.61:4403: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54274->104.36.82.61:4403: use of closed network connection Aug 7 21:29:28.005023909 (11h49m27.594368896s) [noise=true , address=104.36.82.9:4032 , piece_id=UNNJDD7KMLRSVQQOJTV2ORP25CQK4KEWKSAP2ETUHYB4C2QWCZAA, seg_pos={0 0}, upload=12298]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ferXUYkCNRALvKCfTAfqHrNuDJZcqAx7Qx3hr7TaaMdVGzxQQ): protocol: rpc: dial tcp 104.36.82.9:4032: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 21:29:28.005067853 (11h49m27.59441284s) [noise=true , address=104.36.82.39:4727 , piece_id=2SVEE2SU56DR5U436F7DAYIW6ORGTZLYOUHFK6RPOJ7H3PKSIDPA, seg_pos={0 0}, upload=12298]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3): protocol: write tcp 188.166.225.170:49838->104.36.82.39:4727: use of closed network connection; write tcp 188.166.225.170:49838->104.36.82.39:4727: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49838->104.36.82.39:4727: use of closed network connection Aug 7 21:29:28.005079395 (11h49m27.594424382s) [seg_pos={0 0}, upload=12298]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 21:29:28.005142837 (11h49m27.594487824s) [upload=12298]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 21:29:28.005146774 (11h49m27.594491761s) [upload=12298]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:29:28.333526479 (11h49m27.922871497s) [upload=12298]: Done uploading segments... Aug 7 21:29:28.333548311 (11h49m27.922893298s) [upload=12298]: Upload finished. err: <nil> Duration:* 4.239601515s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [.............X...................................X..........] duration: 3.552946825s FAIL (2025-08-07 21:29:25 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/1K") ======LOGS====== ]: Done uploading segments... Aug 7 21:13:55.370929874 (11h33m54.960274861s) [upload=12060]: Upload finished. err: <nil> Aug 7 21:14:45.614661829 (11h34m45.204006816s) [upload=12073]: Starting upload Aug 7 21:14:45.614700943 (11h34m45.20404593s) [upload=12073]: Done starting upload Aug 7 21:14:45.614745929 (11h34m45.204090917s) [upload=12073]: Uploading segments... Aug 7 21:14:45.614774204 (11h34m45.204119181s) [upload=12073]: Got next segment. Inline: true Aug 7 21:14:45.614777830 (11h34m45.204122817s) [upload=12073]: Waiting for error group managing segments... Aug 7 21:14:45.614809601 (11h34m45.204154578s) [upload=12073]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:14:45.614813057 (11h34m45.204158045s) [upload=12073]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:14:45.614815242 (11h34m45.204160219s) [upload=12073]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:14:45.911976434 (11h34m45.501321421s) [upload=12073]: Done uploading segments... Aug 7 21:14:45.911995080 (11h34m45.501340067s) [upload=12073]: Upload finished. err: <nil> Aug 7 21:15:14.898649385 (11h35m14.487994372s) [upload=12079]: Starting upload Aug 7 21:15:14.898694180 (11h35m14.488039167s) [upload=12079]: Done starting upload Aug 7 21:15:14.898741050 (11h35m14.488086027s) [upload=12079]: Uploading segments... Aug 7 21:15:14.898781948 (11h35m14.488126965s) [upload=12079]: Got next segment. Inline: true Aug 7 21:15:14.898789092 (11h35m14.488134079s) [upload=12079]: Waiting for error group managing segments... Aug 7 21:15:14.898847644 (11h35m14.488192641s) [upload=12079]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:15:14.898856401 (11h35m14.488201408s) [upload=12079]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:15:14.898860989 (11h35m14.488206026s) [upload=12079]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:15:15.228046082 (11h35m14.817391079s) [upload=12079]: Done uploading segments... Aug 7 21:15:15.228066881 (11h35m14.817411878s) [upload=12079]: Upload finished. err: <nil> Aug 7 21:16:25.941982337 (11h36m25.531327324s) [upload=12099]: Starting upload Aug 7 21:16:25.942029607 (11h36m25.531374594s) [upload=12099]: Done starting upload Aug 7 21:16:25.942075234 (11h36m25.531420221s) [upload=12099]: Uploading segments... Aug 7 21:16:25.942105222 (11h36m25.531450199s) [upload=12099]: Got next segment. Inline: true Aug 7 21:16:25.942108588 (11h36m25.531453575s) [upload=12099]: Waiting for error group managing segments... Aug 7 21:16:25.942142683 (11h36m25.53148767s) [upload=12099]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:16:25.942146340 (11h36m25.531491317s) [upload=12099]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:16:25.942148595 (11h36m25.531493582s) [upload=12099]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:16:26.215688283 (11h36m25.80503327s) [upload=12099]: Done uploading segments... Aug 7 21:16:26.215705086 (11h36m25.805050073s) [upload=12099]: Upload finished. err: <nil> Aug 7 21:17:38.285379669 (11h37m37.874724656s) [upload=12117]: Starting upload Aug 7 21:17:38.285441276 (11h37m37.874786323s) [upload=12117]: Done starting upload Aug 7 21:17:38.285507584 (11h37m37.874852611s) [upload=12117]: Uploading segments... Aug 7 21:17:38.285571005 (11h37m37.874915992s) [upload=12117]: Got next segment. Inline: true Aug 7 21:17:38.285575654 (11h37m37.874920641s) [upload=12117]: Waiting for error group managing segments... Aug 7 21:17:38.285618215 (11h37m37.874963243s) [upload=12117]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:17:38.285624958 (11h37m37.874969945s) [upload=12117]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:17:38.285629477 (11h37m37.874974474s) [upload=12117]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:17:38.561705487 (11h37m38.151050484s) [upload=12117]: Done uploading segments... Aug 7 21:17:38.561727969 (11h37m38.151072957s) [upload=12117]: Upload finished. err: <nil> Aug 7 21:18:46.880586061 (11h38m46.469931048s) [upload=12133]: Starting upload Aug 7 21:18:46.880627500 (11h38m46.469972487s) [upload=12133]: Done starting upload Aug 7 21:18:46.880681523 (11h38m46.47002651s) [upload=12133]: Uploading segments... Aug 7 21:18:46.880714476 (11h38m46.470059463s) [upload=12133]: Got next segment. Inline: true Aug 7 21:18:46.880718043 (11h38m46.47006302s) [upload=12133]: Waiting for error group managing segments... Aug 7 21:18:46.880752089 (11h38m46.470097076s) [upload=12133]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:18:46.880755715 (11h38m46.470100702s) [upload=12133]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:18:46.880758120 (11h38m46.470103107s) [upload=12133]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:18:47.149461450 (11h38m46.738806437s) [upload=12133]: Done uploading segments... Aug 7 21:18:47.149478763 (11h38m46.73882375s) [upload=12133]: Upload finished. err: <nil> Aug 7 21:19:44.345616276 (11h39m43.934961263s) [upload=12152]: Starting upload Aug 7 21:19:44.345660120 (11h39m43.935005098s) [upload=12152]: Done starting upload Aug 7 21:19:44.345731898 (11h39m43.935076885s) [upload=12152]: Uploading segments... Aug 7 21:19:44.345839653 (11h39m43.93518463s) [upload=12152]: Got next segment. Inline: true Aug 7 21:19:44.345844203 (11h39m43.93518919s) [upload=12152]: Waiting for error group managing segments... Aug 7 21:19:44.345886133 (11h39m43.93523111s) [upload=12152]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:19:44.345890932 (11h39m43.935235919s) [upload=12152]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:19:44.345893316 (11h39m43.935238303s) [upload=12152]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:19:44.666041542 (11h39m44.255386529s) [upload=12152]: Done uploading segments... Aug 7 21:19:44.666059567 (11h39m44.255404554s) [upload=12152]: Upload finished. err: <nil> Aug 7 21:20:10.176957983 (11h40m9.76630296s) [upload=12157]: Starting upload Aug 7 21:20:10.177009311 (11h40m9.766354298s) [upload=12157]: Done starting upload Aug 7 21:20:10.177070559 (11h40m9.766415546s) [upload=12157]: Uploading segments... Aug 7 21:20:10.177121496 (11h40m9.766466483s) [upload=12157]: Got next segment. Inline: true Aug 7 21:20:10.177125895 (11h40m9.766470882s) [upload=12157]: Waiting for error group managing segments... Aug 7 21:20:10.177160220 (11h40m9.766505207s) [upload=12157]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:20:10.177164087 (11h40m9.766509074s) [upload=12157]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:20:10.177166371 (11h40m9.766511359s) [upload=12157]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:20:10.463794245 (11h40m10.053139232s) [upload=12157]: Done uploading segments... Aug 7 21:20:10.463812280 (11h40m10.053157257s) [upload=12157]: Upload finished. err: <nil> Aug 7 21:20:47.295095407 (11h40m46.884440394s) [upload=12169]: Starting upload Aug 7 21:20:47.295162806 (11h40m46.884507793s) [upload=12169]: Done starting upload Aug 7 21:20:47.295214385 (11h40m46.884559372s) [upload=12169]: Uploading segments... Aug 7 21:20:47.295251876 (11h40m46.884596864s) [upload=12169]: Got next segment. Inline: true Aug 7 21:20:47.295255583 (11h40m46.88460057s) [upload=12169]: Waiting for error group managing segments... Aug 7 21:20:47.295291642 (11h40m46.884636629s) [upload=12169]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:20:47.295295450 (11h40m46.884640437s) [upload=12169]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:20:47.295297755 (11h40m46.884642742s) [upload=12169]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:20:47.581180532 (11h40m47.170525529s) [upload=12169]: Done uploading segments... Aug 7 21:20:47.581201993 (11h40m47.17054698s) [upload=12169]: Upload finished. err: <nil> Aug 7 21:21:39.503326248 (11h41m39.092671235s) [upload=12184]: Starting upload Aug 7 21:21:39.503368609 (11h41m39.092713586s) [upload=12184]: Done starting upload Aug 7 21:21:39.503416451 (11h41m39.092761438s) [upload=12184]: Uploading segments... Aug 7 21:21:39.503447480 (11h41m39.092792467s) [upload=12184]: Got next segment. Inline: true Aug 7 21:21:39.503451247 (11h41m39.092796234s) [upload=12184]: Waiting for error group managing segments... Aug 7 21:21:39.503485393 (11h41m39.09283038s) [upload=12184]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:21:39.503489370 (11h41m39.092834357s) [upload=12184]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:21:39.503491825 (11h41m39.092836802s) [upload=12184]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:21:39.826158170 (11h41m39.415503147s) [upload=12184]: Done uploading segments... Aug 7 21:21:39.826174040 (11h41m39.415519027s) [upload=12184]: Upload finished. err: <nil> Aug 7 21:22:46.427091203 (11h42m46.01643619s) [upload=12199]: Starting upload Aug 7 21:22:46.427135929 (11h42m46.016480916s) [upload=12199]: Done starting upload Aug 7 21:22:46.427188519 (11h42m46.016533496s) [upload=12199]: Uploading segments... Aug 7 21:22:46.427215882 (11h42m46.016560869s) [upload=12199]: Got next segment. Inline: true Aug 7 21:22:46.427219829 (11h42m46.016564806s) [upload=12199]: Waiting for error group managing segments... Aug 7 21:22:46.427252391 (11h42m46.016597378s) [upload=12199]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:22:46.427256048 (11h42m46.016601025s) [upload=12199]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:22:46.427268202 (11h42m46.016613189s) [upload=12199]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:22:46.705229475 (11h42m46.294574462s) [upload=12199]: Done uploading segments... Aug 7 21:22:46.705247530 (11h42m46.294592507s) [upload=12199]: Upload finished. err: <nil> Duration: 3.552971963s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 11 out of 60 is failed (threshold=3) [.................................................XXXXXXXXXXX] duration: 7m20.104805995s FAIL (2025-08-07 21:29:23 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 7m20.104828108s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 16 out of 60 is failed (threshold=3) [...........................................X.XXXXXXXXXXXXXXX] duration: 1m0.001136155s FAIL (2025-08-07 21:29:17 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 45 (11h38m57.549589842s) [upload=12139]: Uploading segments... Aug 7 21:18:57.960294299 (11h38m57.549639286s) [upload=12139]: Got next segment. Inline: true Aug 7 21:18:57.960298928 (11h38m57.549643915s) [upload=12139]: Waiting for error group managing segments... Aug 7 21:18:57.960349855 (11h38m57.549694853s) [upload=12139]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:18:57.960354916 (11h38m57.549699903s) [upload=12139]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:18:57.960358112 (11h38m57.549703109s) [upload=12139]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:19:57.313173678 (11h39m56.902518665s) [upload=12139]: Done uploading segments... Aug 7 21:19:57.313193636 (11h39m56.902538623s) [upload=12139]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:20:04.133582941 (11h40m3.722927928s) [upload=12156]: Starting upload Aug 7 21:20:04.133643998 (11h40m3.722988985s) [upload=12156]: Done starting upload Aug 7 21:20:04.133712729 (11h40m3.723057726s) [upload=12156]: Uploading segments... Aug 7 21:20:04.133764127 (11h40m3.723109114s) [upload=12156]: Got next segment. Inline: true Aug 7 21:20:04.133769046 (11h40m3.723114033s) [upload=12156]: Waiting for error group managing segments... Aug 7 21:20:04.133838079 (11h40m3.723183066s) [upload=12156]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:20:04.133842868 (11h40m3.723187855s) [upload=12156]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:20:04.133845723 (11h40m3.72319071s) [upload=12156]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:21:03.481385970 (11h41m3.070730947s) [upload=12156]: Done uploading segments... Aug 7 21:21:03.481407009 (11h41m3.070751986s) [upload=12156]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:21:04.366237551 (11h41m3.955582538s) [upload=12172]: Starting upload Aug 7 21:21:04.366309369 (11h41m3.955654356s) [upload=12172]: Done starting upload Aug 7 21:21:04.366360317 (11h41m3.955705304s) [upload=12172]: Uploading segments... Aug 7 21:21:04.366406304 (11h41m3.955751291s) [upload=12172]: Got next segment. Inline: true Aug 7 21:21:04.366410022 (11h41m3.955755009s) [upload=12172]: Waiting for error group managing segments... Aug 7 21:21:04.366441823 (11h41m3.95578681s) [upload=12172]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:21:04.366445840 (11h41m3.955790827s) [upload=12172]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:21:04.366448265 (11h41m3.955793252s) [upload=12172]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:22:03.717958127 (11h42m3.307303114s) [upload=12172]: Done uploading segments... Aug 7 21:22:03.717976892 (11h42m3.307321879s) [upload=12172]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:22:11.311019027 (11h42m10.900364014s) [upload=12193]: Starting upload Aug 7 21:22:11.311074243 (11h42m10.90041923s) [upload=12193]: Done starting upload Aug 7 21:22:11.311123727 (11h42m10.900468714s) [upload=12193]: Uploading segments... Aug 7 21:22:11.311167090 (11h42m10.900512077s) [upload=12193]: Got next segment. Inline: true Aug 7 21:22:11.311183532 (11h42m10.900528519s) [upload=12193]: Waiting for error group managing segments... Aug 7 21:22:11.311220152 (11h42m10.900565139s) [upload=12193]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:22:11.311224931 (11h42m10.900569918s) [upload=12193]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:22:11.311227586 (11h42m10.900572573s) [upload=12193]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:23:10.658643233 (11h43m10.24798821s) [upload=12193]: Done uploading segments... Aug 7 21:23:10.658662571 (11h43m10.248007558s) [upload=12193]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:23:11.571475028 (11h43m11.160820015s) [upload=12205]: Starting upload Aug 7 21:23:11.571532048 (11h43m11.160877035s) [upload=12205]: Done starting upload Aug 7 21:23:11.571582173 (11h43m11.16092716s) [upload=12205]: Uploading segments... Aug 7 21:23:11.571617942 (11h43m11.160962929s) [upload=12205]: Got next segment. Inline: true Aug 7 21:23:11.571621489 (11h43m11.160966486s) [upload=12205]: Waiting for error group managing segments... Aug 7 21:23:11.571655484 (11h43m11.161000471s) [upload=12205]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:23:11.571659531 (11h43m11.161004528s) [upload=12205]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:23:11.571662447 (11h43m11.161007444s) [upload=12205]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:24:10.926729728 (11h44m10.516074715s) [upload=12205]: Done uploading segments... Aug 7 21:24:10.926747603 (11h44m10.51609259s) [upload=12205]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:24:11.802089650 (11h44m11.391434637s) [upload=12218]: Starting upload Aug 7 21:24:11.802151558 (11h44m11.391496545s) [upload=12218]: Done starting upload Aug 7 21:24:11.802205020 (11h44m11.391549997s) [upload=12218]: Uploading segments... Aug 7 21:24:11.802240418 (11h44m11.391585406s) [upload=12218]: Got next segment. Inline: true Aug 7 21:24:11.802244236 (11h44m11.391589223s) [upload=12218]: Waiting for error group managing segments... Aug 7 21:24:11.802278040 (11h44m11.391623027s) [upload=12218]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:24:11.802282458 (11h44m11.391627445s) [upload=12218]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:24:11.802284602 (11h44m11.391629579s) [upload=12218]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:25:11.151652598 (11h45m10.740997575s) [upload=12218]: Done uploading segments... Aug 7 21:25:11.151674050 (11h45m10.741019037s) [upload=12218]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:25:17.859162022 (11h45m17.44850703s) [upload=12235]: Starting upload Aug 7 21:25:17.859247908 (11h45m17.448592915s) [upload=12235]: Done starting upload Aug 7 21:25:17.859314314 (11h45m17.448659301s) [upload=12235]: Uploading segments... Aug 7 21:25:17.859360473 (11h45m17.44870546s) [upload=12235]: Got next segment. Inline: true Aug 7 21:25:17.859365562 (11h45m17.44871056s) [upload=12235]: Waiting for error group managing segments... Aug 7 21:25:17.859407051 (11h45m17.448752058s) [upload=12235]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:25:17.859412402 (11h45m17.448757399s) [upload=12235]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:25:17.859415648 (11h45m17.448760646s) [upload=12235]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:26:17.209779892 (11h46m16.799124879s) [upload=12235]: Done uploading segments... Aug 7 21:26:17.209836961 (11h46m16.799181949s) [upload=12235]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:26:18.091653415 (11h46m17.680998403s) [upload=12251]: Starting upload Aug 7 21:26:18.091708220 (11h46m17.681053207s) [upload=12251]: Done starting upload Aug 7 21:26:18.091749139 (11h46m17.681094126s) [upload=12251]: Uploading segments... Aug 7 21:26:18.091784216 (11h46m17.681129203s) [upload=12251]: Got next segment. Inline: true Aug 7 21:26:18.091787672 (11h46m17.681132649s) [upload=12251]: Waiting for error group managing segments... Aug 7 21:26:18.091819754 (11h46m17.681164741s) [upload=12251]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:26:18.091823481 (11h46m17.681168468s) [upload=12251]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:26:18.091826015 (11h46m17.681170992s) [upload=12251]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:27:17.447643584 (11h47m17.036988571s) [upload=12251]: Done uploading segments... Aug 7 21:27:17.447669333 (11h47m17.03701433s) [upload=12251]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:27:18.331191031 (11h47m17.920536018s) [upload=12265]: Starting upload Aug 7 21:27:18.331246827 (11h47m17.920591815s) [upload=12265]: Done starting upload Aug 7 21:27:18.331299789 (11h47m17.920644776s) [upload=12265]: Uploading segments... Aug 7 21:27:18.331346968 (11h47m17.920691945s) [upload=12265]: Got next segment. Inline: true Aug 7 21:27:18.331350675 (11h47m17.920695652s) [upload=12265]: Waiting for error group managing segments... Aug 7 21:27:18.331385723 (11h47m17.92073071s) [upload=12265]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:27:18.331389891 (11h47m17.920734878s) [upload=12265]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:27:18.331392486 (11h47m17.920737473s) [upload=12265]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:28:17.685855344 (11h48m17.275200351s) [upload=12265]: Done uploading segments... Aug 7 21:28:17.685882486 (11h48m17.275227473s) [upload=12265]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:28:18.552314505 (11h48m18.141659502s) [upload=12277]: Starting upload Aug 7 21:28:18.552431058 (11h48m18.141776075s) [upload=12277]: Done starting upload Aug 7 21:28:18.552500100 (11h48m18.141845107s) [upload=12277]: Uploading segments... Aug 7 21:28:18.552547911 (11h48m18.141892898s) [upload=12277]: Got next segment. Inline: true Aug 7 21:28:18.552552150 (11h48m18.141897137s) [upload=12277]: Waiting for error group managing segments... Aug 7 21:28:18.552600933 (11h48m18.14194592s) [upload=12277]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:28:18.552607205 (11h48m18.141952212s) [upload=12277]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:28:18.552612195 (11h48m18.141957192s) [upload=12277]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:29:17.904886609 (11h49m17.494231596s) [upload=12277]: Done uploading segments... Aug 7 21:29:17.904932337 (11h49m17.494277354s) [upload=12277]: Upload finished. err: metaclient: context deadline exceeded Duration: 1m0.00114911s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 19 out of 60 is failed (threshold=3) [.........................................XXXXXXXXXXXXXXXXXXX] duration: 30.001370218s FAIL (2025-08-07 21:29:14 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== .508006025s) [upload=12161]: Uploading segments... Aug 7 21:20:14.918718718 (11h40m14.508063705s) [upload=12161]: Got next segment. Inline: true Aug 7 21:20:14.918725691 (11h40m14.508070688s) [upload=12161]: Waiting for error group managing segments... Aug 7 21:20:14.918778933 (11h40m14.50812394s) [upload=12161]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:20:14.918785126 (11h40m14.508130133s) [upload=12161]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:20:14.918790756 (11h40m14.508135773s) [upload=12161]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:20:44.279825769 (11h40m43.869170756s) [upload=12161]: Done uploading segments... Aug 7 21:20:44.279847039 (11h40m43.869192027s) [upload=12161]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:21:26.967144759 (11h41m26.556489736s) [upload=12182]: Starting upload Aug 7 21:21:26.967204002 (11h41m26.556549009s) [upload=12182]: Done starting upload Aug 7 21:21:26.967248207 (11h41m26.556593194s) [upload=12182]: Uploading segments... Aug 7 21:21:26.967275329 (11h41m26.556620316s) [upload=12182]: Got next segment. Inline: true Aug 7 21:21:26.967278815 (11h41m26.556623802s) [upload=12182]: Waiting for error group managing segments... Aug 7 21:21:26.967311758 (11h41m26.556656745s) [upload=12182]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:21:26.967315786 (11h41m26.556660773s) [upload=12182]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:21:26.967318120 (11h41m26.556663107s) [upload=12182]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:21:56.309770442 (11h41m55.899115429s) [upload=12182]: Done uploading segments... Aug 7 21:21:56.309792865 (11h41m55.899137852s) [upload=12182]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:22:21.925019495 (11h42m21.514364572s) [upload=12195]: Starting upload Aug 7 21:22:21.925098336 (11h42m21.514443323s) [upload=12195]: Done starting upload Aug 7 21:22:21.925142180 (11h42m21.514487167s) [upload=12195]: Uploading segments... Aug 7 21:22:21.925175574 (11h42m21.514520561s) [upload=12195]: Got next segment. Inline: true Aug 7 21:22:21.925179491 (11h42m21.514524478s) [upload=12195]: Waiting for error group managing segments... Aug 7 21:22:21.925210671 (11h42m21.514555648s) [upload=12195]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:22:21.925214048 (11h42m21.514559035s) [upload=12195]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:22:21.925216302 (11h42m21.51456129s) [upload=12195]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:22:51.283998863 (11h42m50.87334385s) [upload=12195]: Done uploading segments... Aug 7 21:22:51.284018922 (11h42m50.873363899s) [upload=12195]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:23:36.916300477 (11h43m36.505645464s) [upload=12212]: Starting upload Aug 7 21:23:36.916350923 (11h43m36.50569591s) [upload=12212]: Done starting upload Aug 7 21:23:36.916395318 (11h43m36.505740295s) [upload=12212]: Uploading segments... Aug 7 21:23:36.916436557 (11h43m36.505781544s) [upload=12212]: Got next segment. Inline: true Aug 7 21:23:36.916439863 (11h43m36.50578484s) [upload=12212]: Waiting for error group managing segments... Aug 7 21:23:36.916471534 (11h43m36.505816511s) [upload=12212]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:23:36.916475351 (11h43m36.505820338s) [upload=12212]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:23:36.916477665 (11h43m36.505822652s) [upload=12212]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:24:06.267030467 (11h44m5.856375454s) [upload=12212]: Done uploading segments... Aug 7 21:24:06.267052209 (11h44m5.856397196s) [upload=12212]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:24:29.996857636 (11h44m29.586202623s) [upload=12224]: Starting upload Aug 7 21:24:29.996913823 (11h44m29.5862588s) [upload=12224]: Done starting upload Aug 7 21:24:29.996961495 (11h44m29.586306482s) [upload=12224]: Uploading segments... Aug 7 21:24:29.996996181 (11h44m29.586341158s) [upload=12224]: Got next segment. Inline: true Aug 7 21:24:29.996999788 (11h44m29.586344765s) [upload=12224]: Waiting for error group managing segments... Aug 7 21:24:29.997033352 (11h44m29.586378329s) [upload=12224]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:24:29.997037590 (11h44m29.586382577s) [upload=12224]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:24:29.997040075 (11h44m29.586385062s) [upload=12224]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:24:59.358108170 (11h44m58.947453157s) [upload=12224]: Done uploading segments... Aug 7 21:24:59.358131014 (11h44m58.947476001s) [upload=12224]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:25:00.224254754 (11h44m59.813599741s) [upload=12230]: Starting upload Aug 7 21:25:00.224330690 (11h44m59.813675677s) [upload=12230]: Done starting upload Aug 7 21:25:00.224406304 (11h44m59.813751311s) [upload=12230]: Uploading segments... Aug 7 21:25:00.224533207 (11h44m59.813878204s) [upload=12230]: Got next segment. Inline: true Aug 7 21:25:00.224541352 (11h44m59.813886339s) [upload=12230]: Waiting for error group managing segments... Aug 7 21:25:00.224626114 (11h44m59.813971121s) [upload=12230]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:25:00.224634080 (11h44m59.813979107s) [upload=12230]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:25:00.224638929 (11h44m59.813983937s) [upload=12230]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:25:29.580431600 (11h45m29.169776597s) [upload=12230]: Done uploading segments... Aug 7 21:25:29.580456288 (11h45m29.169801275s) [upload=12230]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:25:38.048208784 (11h45m37.637553761s) [upload=12241]: Starting upload Aug 7 21:25:38.048265823 (11h45m37.63761081s) [upload=12241]: Done starting upload Aug 7 21:25:38.048311521 (11h45m37.637656508s) [upload=12241]: Uploading segments... Aug 7 21:25:38.048347439 (11h45m37.637692426s) [upload=12241]: Got next segment. Inline: true Aug 7 21:25:38.048350867 (11h45m37.637695854s) [upload=12241]: Waiting for error group managing segments... Aug 7 21:25:38.048383188 (11h45m37.637728186s) [upload=12241]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:25:38.048387396 (11h45m37.637732383s) [upload=12241]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:25:38.048389881 (11h45m37.637734868s) [upload=12241]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:26:07.388744771 (11h46m6.978089758s) [upload=12241]: Done uploading segments... Aug 7 21:26:07.388753889 (11h46m6.978098876s) [upload=12241]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:26:51.633559976 (11h46m51.222904953s) [upload=12259]: Starting upload Aug 7 21:26:51.633607808 (11h46m51.222952795s) [upload=12259]: Done starting upload Aug 7 21:26:51.633657383 (11h46m51.22300237s) [upload=12259]: Uploading segments... Aug 7 21:26:51.633689063 (11h46m51.22303405s) [upload=12259]: Got next segment. Inline: true Aug 7 21:26:51.633692660 (11h46m51.223037647s) [upload=12259]: Waiting for error group managing segments... Aug 7 21:26:51.633727035 (11h46m51.223072012s) [upload=12259]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:26:51.633730782 (11h46m51.22307577s) [upload=12259]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:26:51.633733327 (11h46m51.223078314s) [upload=12259]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:27:20.991495967 (11h47m20.580840944s) [upload=12259]: Done uploading segments... Aug 7 21:27:20.991524641 (11h47m20.580869628s) [upload=12259]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:27:32.297072953 (11h47m31.88641795s) [upload=12271]: Starting upload Aug 7 21:27:32.297139892 (11h47m31.886484889s) [upload=12271]: Done starting upload Aug 7 21:27:32.297212079 (11h47m31.886557066s) [upload=12271]: Uploading segments... Aug 7 21:27:32.297260662 (11h47m31.886605649s) [upload=12271]: Got next segment. Inline: true Aug 7 21:27:32.297264640 (11h47m31.886609627s) [upload=12271]: Waiting for error group managing segments... Aug 7 21:27:32.297341267 (11h47m31.886686254s) [upload=12271]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:27:32.297345635 (11h47m31.886690622s) [upload=12271]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:27:32.297348029 (11h47m31.886693016s) [upload=12271]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:28:01.648543031 (11h48m1.237888018s) [upload=12271]: Done uploading segments... Aug 7 21:28:01.648565273 (11h48m1.23791026s) [upload=12271]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:28:45.542334506 (11h48m45.131679483s) [upload=12290]: Starting upload Aug 7 21:28:45.542390844 (11h48m45.131735831s) [upload=12290]: Done starting upload Aug 7 21:28:45.542466819 (11h48m45.131811806s) [upload=12290]: Uploading segments... Aug 7 21:28:45.542586358 (11h48m45.131931345s) [upload=12290]: Got next segment. Inline: true Aug 7 21:28:45.542591848 (11h48m45.131936835s) [upload=12290]: Waiting for error group managing segments... Aug 7 21:28:45.542637005 (11h48m45.131981982s) [upload=12290]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:28:45.542642495 (11h48m45.131987482s) [upload=12290]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:28:45.542645010 (11h48m45.131989987s) [upload=12290]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:29:14.895678169 (11h49m14.485023156s) [upload=12290]: Done uploading segments... Aug 7 21:29:14.895732473 (11h49m14.48507746s) [upload=12290]: Upload finished. err: metaclient: context deadline exceeded Duration: 30.001384004s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 20 out of 60 is failed (threshold=3) [........................................XXXXXXXXXXXXXXXXXXXX] duration: 30.00140341s FAIL (2025-08-07 21:29:11 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.001416946s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 16 out of 60 is failed (threshold=3) [..........................................X..XXXXXXXXXXXXXXX] duration: 30.000850415s FAIL (2025-08-07 21:29:06 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 30.000872518s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [......................................X..XXXXXXXXXXXXXXXXXXX] duration: 30.000664285s FAIL (2025-08-07 21:28:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== Next segment err: context deadline exceeded Aug 7 21:20:09.053707913 (11h40m8.6430529s) [upload=12149]: Done uploading segments... Aug 7 21:20:09.053723724 (11h40m8.643068711s) [upload=12149]: Upload finished. err: context deadline exceeded Aug 7 21:20:24.058132378 (11h40m23.647477365s) [upload=12163]: Starting upload Aug 7 21:20:24.058185559 (11h40m23.647530546s) [upload=12163]: Done starting upload Aug 7 21:20:24.058265552 (11h40m23.647610529s) [upload=12163]: Uploading segments... Aug 7 21:20:24.058369542 (11h40m23.647714539s) [upload=12163]: Got next segment. Inline: false Aug 7 21:20:24.058412013 (11h40m23.647757s) [upload=12163]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:20:24.058418646 (11h40m23.647763663s) [upload=12163]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:20:53.409396630 (11h40m52.998741617s) [upload=12163]: Next segment err: context deadline exceeded Aug 7 21:20:53.409563509 (11h40m52.998908496s) [upload=12163]: Done uploading segments... Aug 7 21:20:53.409604537 (11h40m52.998949514s) [upload=12163]: Upload finished. err: context deadline exceeded Aug 7 21:21:23.928736473 (11h41m23.51808146s) [upload=12180]: Starting upload Aug 7 21:21:23.928792831 (11h41m23.518137818s) [upload=12180]: Done starting upload Aug 7 21:21:23.928856973 (11h41m23.51820195s) [upload=12180]: Uploading segments... Aug 7 21:21:23.929051695 (11h41m23.518396682s) [upload=12180]: Got next segment. Inline: false Aug 7 21:21:23.929103304 (11h41m23.518448291s) [upload=12180]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:21:23.929107933 (11h41m23.51845292s) [upload=12180]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:21:53.279642106 (11h41m52.868987093s) [upload=12180]: Next segment err: context deadline exceeded Aug 7 21:21:53.279858500 (11h41m52.869203477s) [upload=12180]: Done uploading segments... Aug 7 21:21:53.279873498 (11h41m52.869218485s) [upload=12180]: Upload finished. err: context deadline exceeded Aug 7 21:21:54.191662378 (11h41m53.781007366s) [upload=12188]: Starting upload Aug 7 21:21:54.191712274 (11h41m53.781057251s) [upload=12188]: Done starting upload Aug 7 21:21:54.191790404 (11h41m53.781135391s) [upload=12188]: Uploading segments... Aug 7 21:21:54.192007459 (11h41m53.781352446s) [upload=12188]: Got next segment. Inline: false Aug 7 21:21:54.192034671 (11h41m53.781379658s) [upload=12188]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:21:54.192038719 (11h41m53.781383706s) [upload=12188]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:22:23.533395958 (11h42m23.122740945s) [upload=12188]: Next segment err: context deadline exceeded Aug 7 21:22:23.533474659 (11h42m23.122819646s) [upload=12188]: Done uploading segments... Aug 7 21:22:23.533496701 (11h42m23.122841688s) [upload=12188]: Upload finished. err: context deadline exceeded Aug 7 21:23:08.442262622 (11h43m8.031607609s) [upload=12203]: Starting upload Aug 7 21:23:08.442325042 (11h43m8.031670019s) [upload=12203]: Done starting upload Aug 7 21:23:08.442385668 (11h43m8.031730655s) [upload=12203]: Uploading segments... Aug 7 21:23:08.442501099 (11h43m8.031846076s) [upload=12203]: Got next segment. Inline: false Aug 7 21:23:08.442542839 (11h43m8.031887826s) [upload=12203]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:23:08.442546706 (11h43m8.031891683s) [upload=12203]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:23:37.798210470 (11h43m37.387555457s) [upload=12203]: Next segment err: context deadline exceeded Aug 7 21:23:37.798367780 (11h43m37.387712767s) [upload=12203]: Done uploading segments... Aug 7 21:23:37.798381718 (11h43m37.387726705s) [upload=12203]: Upload finished. err: context deadline exceeded Aug 7 21:23:54.967168247 (11h43m54.556513234s) [upload=12215]: Starting upload Aug 7 21:23:54.967268208 (11h43m54.556613185s) [upload=12215]: Done starting upload Aug 7 21:23:54.967374321 (11h43m54.556719308s) [upload=12215]: Uploading segments... Aug 7 21:23:54.967600634 (11h43m54.556945621s) [upload=12215]: Got next segment. Inline: false Aug 7 21:23:54.967649888 (11h43m54.556994875s) [upload=12215]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:23:54.967657021 (11h43m54.557002029s) [upload=12215]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:24:24.312833601 (11h44m23.902178598s) [upload=12215]: Next segment err: context deadline exceeded Aug 7 21:24:24.313044925 (11h44m23.902389912s) [upload=12215]: Done uploading segments... Aug 7 21:24:24.313087757 (11h44m23.902432764s) [upload=12215]: Upload finished. err: context deadline exceeded Aug 7 21:24:25.196484464 (11h44m24.785829451s) [upload=12221]: Starting upload Aug 7 21:24:25.196569166 (11h44m24.785914153s) [upload=12221]: Done starting upload Aug 7 21:24:25.196697702 (11h44m24.786042719s) [upload=12221]: Uploading segments... Aug 7 21:24:25.196786912 (11h44m24.786131899s) [upload=12221]: Got next segment. Inline: false Aug 7 21:24:25.196818883 (11h44m24.78616387s) [upload=12221]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:24:25.196823522 (11h44m24.786168509s) [upload=12221]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:24:54.542928213 (11h44m54.1322732s) [upload=12221]: Next segment err: context deadline exceeded Aug 7 21:24:54.543143736 (11h44m54.132488713s) [upload=12221]: Done uploading segments... Aug 7 21:24:54.543169415 (11h44m54.132514402s) [upload=12221]: Upload finished. err: context deadline exceeded Aug 7 21:25:30.578713383 (11h45m30.16805837s) [upload=12239]: Starting upload Aug 7 21:25:30.578779109 (11h45m30.168124096s) [upload=12239]: Done starting upload Aug 7 21:25:30.578836729 (11h45m30.168181716s) [upload=12239]: Uploading segments... Aug 7 21:25:30.579017655 (11h45m30.168362642s) [upload=12239]: Got next segment. Inline: false Aug 7 21:25:30.579083341 (11h45m30.168428328s) [upload=12239]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:25:30.579091086 (11h45m30.168436073s) [upload=12239]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:25:59.907330605 (11h45m59.496675592s) [upload=12239]: Next segment err: context deadline exceeded Aug 7 21:25:59.907404857 (11h45m59.496749845s) [upload=12239]: Done uploading segments... Aug 7 21:25:59.907422762 (11h45m59.496767749s) [upload=12239]: Upload finished. err: context deadline exceeded Aug 7 21:26:16.857419024 (11h46m16.446764011s) [upload=12250]: Starting upload Aug 7 21:26:16.857477205 (11h46m16.446822182s) [upload=12250]: Done starting upload Aug 7 21:26:16.857572879 (11h46m16.446917856s) [upload=12250]: Uploading segments... Aug 7 21:26:16.857644405 (11h46m16.446989392s) [upload=12250]: Got next segment. Inline: false Aug 7 21:26:16.857669784 (11h46m16.447014771s) [upload=12250]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:26:16.857673381 (11h46m16.447018368s) [upload=12250]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:26:46.211380241 (11h46m45.800725218s) [upload=12250]: Next segment err: context deadline exceeded Aug 7 21:26:46.211514648 (11h46m45.800859625s) [upload=12250]: Done uploading segments... Aug 7 21:26:46.211531050 (11h46m45.800876037s) [upload=12250]: Upload finished. err: context deadline exceeded Aug 7 21:27:19.636960898 (11h47m19.226305885s) [upload=12266]: Starting upload Aug 7 21:27:19.637068003 (11h47m19.22641299s) [upload=12266]: Done starting upload Aug 7 21:27:19.637205767 (11h47m19.226550744s) [upload=12266]: Uploading segments... Aug 7 21:27:19.637342969 (11h47m19.226687956s) [upload=12266]: Got next segment. Inline: false Aug 7 21:27:19.637391532 (11h47m19.22673655s) [upload=12266]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:27:19.637399387 (11h47m19.226744374s) [upload=12266]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:27:48.990525160 (11h47m48.579870148s) [upload=12266]: Next segment err: context deadline exceeded Aug 7 21:27:48.990618549 (11h47m48.579963526s) [upload=12266]: Done uploading segments... Aug 7 21:27:48.990636003 (11h47m48.57998099s) [upload=12266]: Upload finished. err: context deadline exceeded Aug 7 21:27:56.650673215 (11h47m56.240018202s) [upload=12276]: Starting upload Aug 7 21:27:56.650729763 (11h47m56.24007474s) [upload=12276]: Done starting upload Aug 7 21:27:56.650797482 (11h47m56.240142469s) [upload=12276]: Uploading segments... Aug 7 21:27:56.650928032 (11h47m56.240273019s) [upload=12276]: Got next segment. Inline: false Aug 7 21:27:56.650969893 (11h47m56.24031487s) [upload=12276]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:27:56.650973740 (11h47m56.240318727s) [upload=12276]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:28:25.997700487 (11h48m25.587045464s) [upload=12276]: Next segment err: context deadline exceeded Aug 7 21:28:25.997775731 (11h48m25.587120708s) [upload=12276]: Done uploading segments... Aug 7 21:28:25.997832740 (11h48m25.587177727s) [upload=12276]: Upload finished. err: context deadline exceeded Aug 7 21:28:26.889850473 (11h48m26.47919546s) [upload=12281]: Starting upload Aug 7 21:28:26.889910888 (11h48m26.479255875s) [upload=12281]: Done starting upload Aug 7 21:28:26.890174312 (11h48m26.479519299s) [upload=12281]: Uploading segments... Aug 7 21:28:26.890228235 (11h48m26.479573242s) [upload=12281]: Got next segment. Inline: false Aug 7 21:28:26.890266779 (11h48m26.479611796s) [upload=12281]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:28:26.890274093 (11h48m26.47961909s) [upload=12281]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:28:56.237411920 (11h48m55.826756907s) [upload=12281]: Next segment err: context deadline exceeded Aug 7 21:28:56.237578128 (11h48m55.826923105s) [upload=12281]: Done uploading segments... Aug 7 21:28:56.237592366 (11h48m55.826937343s) [upload=12281]: Upload finished. err: context deadline exceeded Duration: 30.000674836s core uplink us-select-1
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway) antiblip: 3 out of 60 is failed (threshold=3) [..................................................X....X.X..] duration: 719.493049ms remote: "tcp:45.38.24.2:443" target: https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410 FAIL (2025-08-07 21:28:56 UTC) orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": context deadline exceeded Duration: 719.526282ms edge gateway vhost us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 6 out of 60 is failed (threshold=3) [...XX...........X.......................X.........X......X..] duration: 6.434079943s FAIL (2025-08-07 21:28:47 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/4M):: after 0c7bb902d1fc786d53d64877c5bf041e2c416fa3a23853fbffdc1a20ffb50890 != before bf5169665124c7207457a5823fc93758035c94fbdbbd7bf4beebc1b976aa7211 ======LOGS====== Aug 7 21:27:25.141891015 (11h47m24.731236002s) [upload=12268]: Got next segment. Inline: false Aug 7 21:27:25.141916354 (11h47m24.731261341s) [upload=12268]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:27:25.141920291 (11h47m24.731265278s) [upload=12268]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:27:25.654570615 (11h47m25.243915602s) [seg_pos={0 0}, upload=12268]: Begin upload segment... Aug 7 21:27:25.654611634 (11h47m25.243956621s) [seg_pos={0 0}, upload=12268]: Done begin upload segment. Aug 7 21:27:25.654634086 (11h47m25.243979073s) [noise=true , address=104.36.82.20:4448 , piece_id=X2KIKPSWI3SCJEEYILD4BVVHRIZZFDH4KGE5OHDIXEIM2HVBZ5NA, seg_pos={0 0}, upload=12268]: Uploading piece... Aug 7 21:27:25.654721364 (11h47m25.244066351s) [noise=true , address=104.36.82.22:4527 , piece_id=YCZOVT6QJT3VE2KFBC4DU6M34Y6DUBKTH3YTHDTO7ZQ7UFIKRAFQ, seg_pos={0 0}, upload=12268]: Uploading piece... Aug 7 21:27:25.655016387 (11h47m25.244361365s) [noise=true , address=104.36.82.46:4512 , piece_id=AWY2BPUJPBECXNJXDXYHMQWOZB3DGLTGP5776L677C64XIVW6ZCQ, seg_pos={0 0}, upload=12268]: Uploading piece... Aug 7 21:27:25.655133612 (11h47m25.2444786s) [noise=true , address=104.36.82.59:4329 , piece_id=CB2FO7D6IW7C2KSBJPR4J7XZQAM6NTHLMAW7XOB6JH6SHRGLQV7A, seg_pos={0 0}, upload=12268]: Uploading piece... Aug 7 21:27:25.655336751 (11h47m25.244681728s) [noise=true , address=104.36.82.55:4211 , piece_id=7IRAZ4EU7BMJNOGPAU7G6VIZ4TSTNCZK4PVFNLFPQM6E2234ME3Q, seg_pos={0 0}, upload=12268]: Uploading piece... Aug 7 21:27:25.655423657 (11h47m25.244768634s) [noise=true , address=104.36.82.37:4679 , piece_id=DOECPGUUPVOBOWQRAX5GY6BD5LA2VNYMLI34VEWZRC7KLT3AA2XA, seg_pos={0 0}, upload=12268]: Uploading piece... Aug 7 21:27:25.655662334 (11h47m25.245007311s) [noise=true , address=104.36.82.58:4279 , piece_id=OPZ353KP2L23TZGNZBO52DLTVSX66WL4D7GYM7TMTJOAT2RPOGHQ, seg_pos={0 0}, upload=12268]: Uploading piece... Aug 7 21:27:25.655733840 (11h47m25.245078827s) [noise=true , address=104.36.82.9:4019 , piece_id=ZDF7MVKTAV4AQD5XGAGHBDDB4TQQMBJGXWPKOBD2ONQI4BVLBFGQ, seg_pos={0 0}, upload=12268]: Uploading piece... Aug 7 21:27:25.656037241 (11h47m25.245382228s) [noise=true , address=104.36.82.2:4758 , piece_id=WFBYODP72YMTCPMKQQKVLNCWKMAETBHH6WOQ5UCEXYCBZSY5WWRA, seg_pos={0 0}, upload=12268]: Uploading piece... Aug 7 21:27:25.656051578 (11h47m25.245396566s) [noise=true , address=104.36.82.54:4157 , piece_id=PGULUJFV37PY3FOW2OD3PKK2UGB7RWDNSWPYZJKVQRWGONRSKEXQ, seg_pos={0 0}, upload=12268]: Uploading piece... Aug 7 21:27:25.656335652 (11h47m25.245680639s) [noise=true , address=104.36.82.36:4633 , piece_id=NXYS3XJMQZAJNSPGBFLXTVR4ZDB5RYHB5K4TUTJ4DANWVSQYEAIQ, seg_pos={0 0}, upload=12268]: Uploading piece... Aug 7 21:27:25.656354137 (11h47m25.245699124s) [noise=true , address=45.91.116.8:4054 , piece_id=PHJGQGBYXAZCULVJX5KSSLMUPOGMD6S7I67QVJBQE35ZFT5U6HFA, seg_pos={0 0}, upload=12268]: Uploading piece... Aug 7 21:27:25.656550413 (11h47m25.24589539s) [noise=true , address=104.36.82.60:4345 , piece_id=L5CI5AMDNOXC2TNERZEBAC2SW3PU4NXI5ILTSTAPA2TSHNEQNHHQ, seg_pos={0 0}, upload=12268]: Uploading piece... Aug 7 21:27:25.656587022 (11h47m25.24593201s) [noise=true , address=79.127.205.234:4586 , piece_id=GI36ODJVM2GYWC2WHFASOQUKJMSJ3R4JANFQ5B45OAFNTBGUEBMA, seg_pos={0 0}, upload=12268]: Uploading piece... Aug 7 21:27:25.656785703 (11h47m25.24613069s) [noise=true , address=104.36.82.43:4837 , piece_id=AQ2NEBR5MYDUQYSKIYKLQJOGRIHKGD3F5QM36GXTECXX37R6H5SQ, seg_pos={0 0}, upload=12268]: Uploading piece... Aug 7 21:27:25.656804429 (11h47m25.246149406s) [noise=true , address=104.36.82.20:4439 , piece_id=CJ7R576SA66NFSO7OHECHKCHULKZN7EZDUTQ3YGRWE436DKO4MWA, seg_pos={0 0}, upload=12268]: Uploading piece... Aug 7 21:27:25.657025892 (11h47m25.246370879s) [noise=true , address=79.127.205.233:4632 , piece_id=T4VQ3H6ZHBIXZN7X47ZWK4WYQBD5KTGSJWKH6R3PM7UCFEGHKOFQ, seg_pos={0 0}, upload=12268]: Uploading piece... Aug 7 21:27:25.657252346 (11h47m25.246597333s) [noise=true , address=45.91.116.14:4218 , piece_id=X7RNM4ZIW35EBZJNTVT5NYPQKS7K4JNT2RQW2X5D7JVGQQZKHV3Q, seg_pos={0 0}, upload=12268]: Uploading piece... Aug 7 21:27:26.765000728 (11h47m26.354345715s) [upload=12268]: Next returned nil segment Aug 7 21:27:26.765014103 (11h47m26.35435909s) [upload=12268]: Waiting for error group managing segments... Aug 7 21:27:27.057992337 (11h47m26.647337324s) [noise=true , address=79.127.205.234:4586 , piece_id=GI36ODJVM2GYWC2WHFASOQUKJMSJ3R4JANFQ5B45OAFNTBGUEBMA, seg_pos={0 0}, upload=12268]: Done uploading piece. err: <nil> Aug 7 21:27:27.086935163 (11h47m26.67628014s) [noise=true , address=79.127.205.233:4632 , piece_id=T4VQ3H6ZHBIXZN7X47ZWK4WYQBD5KTGSJWKH6R3PM7UCFEGHKOFQ, seg_pos={0 0}, upload=12268]: Done uploading piece. err: <nil> Aug 7 21:27:27.601292823 (11h47m27.1906378s) [noise=true , address=104.36.82.2:4758 , piece_id=WFBYODP72YMTCPMKQQKVLNCWKMAETBHH6WOQ5UCEXYCBZSY5WWRA, seg_pos={0 0}, upload=12268]: Done uploading piece. err: <nil> Aug 7 21:27:27.699555195 (11h47m27.288900192s) [noise=true , address=104.36.82.9:4019 , piece_id=ZDF7MVKTAV4AQD5XGAGHBDDB4TQQMBJGXWPKOBD2ONQI4BVLBFGQ, seg_pos={0 0}, upload=12268]: Done uploading piece. err: <nil> Aug 7 21:27:27.710943893 (11h47m27.30028888s) [noise=true , address=104.36.82.43:4837 , piece_id=AQ2NEBR5MYDUQYSKIYKLQJOGRIHKGD3F5QM36GXTECXX37R6H5SQ, seg_pos={0 0}, upload=12268]: Done uploading piece. err: <nil> Aug 7 21:27:27.719413398 (11h47m27.308758385s) [noise=true , address=104.36.82.46:4512 , piece_id=AWY2BPUJPBECXNJXDXYHMQWOZB3DGLTGP5776L677C64XIVW6ZCQ, seg_pos={0 0}, upload=12268]: Done uploading piece. err: <nil> Aug 7 21:27:27.751471103 (11h47m27.34081608s) [noise=true , address=104.36.82.58:4279 , piece_id=OPZ353KP2L23TZGNZBO52DLTVSX66WL4D7GYM7TMTJOAT2RPOGHQ, seg_pos={0 0}, upload=12268]: Done uploading piece. err: <nil> Aug 7 21:27:27.752665698 (11h47m27.342010685s) [noise=true , address=104.36.82.36:4633 , piece_id=NXYS3XJMQZAJNSPGBFLXTVR4ZDB5RYHB5K4TUTJ4DANWVSQYEAIQ, seg_pos={0 0}, upload=12268]: Done uploading piece. err: <nil> Aug 7 21:27:27.803219381 (11h47m27.392564368s) [noise=true , address=104.36.82.55:4211 , piece_id=7IRAZ4EU7BMJNOGPAU7G6VIZ4TSTNCZK4PVFNLFPQM6E2234ME3Q, seg_pos={0 0}, upload=12268]: Done uploading piece. err: <nil> Aug 7 21:27:27.855671476 (11h47m27.445016463s) [noise=true , address=104.36.82.22:4527 , piece_id=YCZOVT6QJT3VE2KFBC4DU6M34Y6DUBKTH3YTHDTO7ZQ7UFIKRAFQ, seg_pos={0 0}, upload=12268]: Done uploading piece. err: <nil> Aug 7 21:27:27.879451272 (11h47m27.468796259s) [noise=true , address=104.36.82.20:4439 , piece_id=CJ7R576SA66NFSO7OHECHKCHULKZN7EZDUTQ3YGRWE436DKO4MWA, seg_pos={0 0}, upload=12268]: Done uploading piece. err: <nil> Aug 7 21:27:27.941304094 (11h47m27.530649081s) [noise=true , address=45.91.116.8:4054 , piece_id=PHJGQGBYXAZCULVJX5KSSLMUPOGMD6S7I67QVJBQE35ZFT5U6HFA, seg_pos={0 0}, upload=12268]: Done uploading piece. err: <nil> Aug 7 21:27:28.313392665 (11h47m27.902737652s) [noise=true , address=45.91.116.14:4218 , piece_id=X7RNM4ZIW35EBZJNTVT5NYPQKS7K4JNT2RQW2X5D7JVGQQZKHV3Q, seg_pos={0 0}, upload=12268]: Done uploading piece. err: <nil> Aug 7 21:27:28.385132253 (11h47m27.97447724s) [noise=true , address=104.36.82.59:4329 , piece_id=CB2FO7D6IW7C2KSBJPR4J7XZQAM6NTHLMAW7XOB6JH6SHRGLQV7A, seg_pos={0 0}, upload=12268]: Done uploading piece. err: <nil> Aug 7 21:27:28.385181175 (11h47m27.974526153s) [seg_pos={0 0}, upload=12268]: Segment reached optimal threshold of 14 pieces. Aug 7 21:27:28.385286848 (11h47m27.974631835s) [noise=true , address=104.36.82.54:4157 , piece_id=PGULUJFV37PY3FOW2OD3PKK2UGB7RWDNSWPYZJKVQRWGONRSKEXQ, seg_pos={0 0}, upload=12268]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZrzADk2mKG4R9tjxUKjQeJvb4WmfjLksGFyvstHRfYbBC7XNz): protocol: expected piece hash; context canceled; EOF Aug 7 21:27:28.385451353 (11h47m27.97479633s) [noise=true , address=104.36.82.20:4448 , piece_id=X2KIKPSWI3SCJEEYILD4BVVHRIZZFDH4KGE5OHDIXEIM2HVBZ5NA, seg_pos={0 0}, upload=12268]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12vbYQVrjLwiHTMGsAGAcjze8KeVQUqemqGiVTWF63Xq6F7ABUp): protocol: read tcp 188.166.225.170:56322->104.36.82.20:4448: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 21:27:28.385487432 (11h47m27.974832419s) [noise=true , address=104.36.82.37:4679 , piece_id=DOECPGUUPVOBOWQRAX5GY6BD5LA2VNYMLI34VEWZRC7KLT3AA2XA, seg_pos={0 0}, upload=12268]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VHQihNGcx6bTvwitH9PCSqyuKWuchdYhYzZXwSuoruQFByGoj): protocol: rpc: dial tcp 104.36.82.37:4679: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 21:27:28.385527619 (11h47m27.974872606s) [noise=true , address=104.36.82.60:4345 , piece_id=L5CI5AMDNOXC2TNERZEBAC2SW3PU4NXI5ILTSTAPA2TSHNEQNHHQ, seg_pos={0 0}, upload=12268]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KSTyRJ62wcMHx8K7uwqp5e8L8yhxH7V3YtBHnYu6QJ3K3bXpQ): protocol: read tcp 188.166.225.170:45768->104.36.82.60:4345: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 21:27:28.385544672 (11h47m27.974889659s) [seg_pos={0 0}, upload=12268]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 21:27:28.385597523 (11h47m27.97494251s) [upload=12268]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 21:27:28.385601019 (11h47m27.974946007s) [upload=12268]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:27:28.699165600 (11h47m28.288510587s) [upload=12268]: Done uploading segments... Aug 7 21:27:28.699200797 (11h47m28.288545784s) [upload=12268]: Upload finished. err: <nil> Duration: 6.434105212s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 15 out of 60 is failed (threshold=3) [.............................................XXXXXXXXXXXXXXX] duration: 1m0.001037017s FAIL (2025-08-07 21:28:44 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.001045293s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway) antiblip: 3 out of 60 is failed (threshold=3) [...................................................X....X.X.] duration: 826.9074ms remote: "tcp:45.38.24.2:443" target: https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410 FAIL (2025-08-07 21:28:34 UTC) orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": context deadline exceeded Duration: 826.950863ms edge gateway vhost us-select-1
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway) antiblip: 3 out of 60 is failed (threshold=3) [....................................................X....X.X] duration: 2m0.001326728s remote: "tcp:45.38.24.2:443" target: https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410 FAIL (2025-08-07 21:28:33 UTC) orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": context deadline exceeded Duration: 2m0.001347068s edge gateway vhost us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 14 out of 60 is failed (threshold=3) [..............................................XXXXXXXXXXXXXX] duration: 1m0.001042347s FAIL (2025-08-07 21:28:32 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== pload=12088]: Next segment err: context deadline exceeded Aug 7 21:16:45.135160053 (11h36m44.72450504s) [upload=12088]: Done uploading segments... Aug 7 21:16:45.135188378 (11h36m44.724533375s) [upload=12088]: Upload finished. err: context deadline exceeded Aug 7 21:16:46.020697803 (11h36m45.61004279s) [upload=12106]: Starting upload Aug 7 21:16:46.020754431 (11h36m45.610099448s) [upload=12106]: Done starting upload Aug 7 21:16:46.020843461 (11h36m45.610188448s) [upload=12106]: Uploading segments... Aug 7 21:16:46.020994882 (11h36m45.610339869s) [upload=12106]: Got next segment. Inline: false Aug 7 21:16:46.021030459 (11h36m45.610375446s) [upload=12106]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:16:46.021035288 (11h36m45.610380276s) [upload=12106]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:17:45.371432070 (11h37m44.960777057s) [upload=12106]: Next segment err: context deadline exceeded Aug 7 21:17:45.371471144 (11h37m44.960816142s) [upload=12106]: Done uploading segments... Aug 7 21:17:45.371492135 (11h37m44.960837122s) [upload=12106]: Upload finished. err: context deadline exceeded Aug 7 21:17:49.811942678 (11h37m49.401287665s) [upload=12122]: Starting upload Aug 7 21:17:49.812064030 (11h37m49.401409007s) [upload=12122]: Done starting upload Aug 7 21:17:49.812137160 (11h37m49.401482147s) [upload=12122]: Uploading segments... Aug 7 21:17:49.812396918 (11h37m49.401741895s) [upload=12122]: Got next segment. Inline: false Aug 7 21:17:49.812426735 (11h37m49.401771722s) [upload=12122]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:17:49.812430862 (11h37m49.401775849s) [upload=12122]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:18:49.169738786 (11h38m48.759083813s) [upload=12122]: Next segment err: context deadline exceeded Aug 7 21:18:49.169939078 (11h38m48.759284086s) [upload=12122]: Done uploading segments... Aug 7 21:18:49.169962594 (11h38m48.759307591s) [upload=12122]: Upload finished. err: context deadline exceeded Aug 7 21:18:50.060561583 (11h38m49.64990661s) [upload=12134]: Starting upload Aug 7 21:18:50.060675751 (11h38m49.650020808s) [upload=12134]: Done starting upload Aug 7 21:18:50.060772135 (11h38m49.650117122s) [upload=12134]: Uploading segments... Aug 7 21:18:50.061042412 (11h38m49.650387409s) [upload=12134]: Got next segment. Inline: false Aug 7 21:18:50.061084393 (11h38m49.65042944s) [upload=12134]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:18:50.061091987 (11h38m49.650437004s) [upload=12134]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:19:49.417978828 (11h39m49.007323815s) [upload=12134]: Next segment err: context deadline exceeded Aug 7 21:19:49.418015799 (11h39m49.007360786s) [upload=12134]: Done uploading segments... Aug 7 21:19:49.418037982 (11h39m49.007382959s) [upload=12134]: Upload finished. err: context deadline exceeded Aug 7 21:20:02.821763948 (11h40m2.411108945s) [upload=12155]: Starting upload Aug 7 21:20:02.821848960 (11h40m2.411193957s) [upload=12155]: Done starting upload Aug 7 21:20:02.821939103 (11h40m2.41128409s) [upload=12155]: Uploading segments... Aug 7 21:20:02.822080344 (11h40m2.411425331s) [upload=12155]: Got next segment. Inline: false Aug 7 21:20:02.822124998 (11h40m2.411469985s) [upload=12155]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:20:02.822132784 (11h40m2.411477771s) [upload=12155]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:21:02.173704394 (11h41m1.763049381s) [upload=12155]: Next segment err: context deadline exceeded Aug 7 21:21:02.173855945 (11h41m1.763200932s) [upload=12155]: Done uploading segments... Aug 7 21:21:02.173881643 (11h41m1.76322663s) [upload=12155]: Upload finished. err: context deadline exceeded Aug 7 21:21:03.041754970 (11h41m2.631099958s) [upload=12171]: Starting upload Aug 7 21:21:03.041871343 (11h41m2.63121633s) [upload=12171]: Done starting upload Aug 7 21:21:03.041935847 (11h41m2.631280834s) [upload=12171]: Uploading segments... Aug 7 21:21:03.042038162 (11h41m2.631383149s) [upload=12171]: Got next segment. Inline: false Aug 7 21:21:03.042070074 (11h41m2.631415061s) [upload=12171]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:21:03.042074442 (11h41m2.631419429s) [upload=12171]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:22:02.399093104 (11h42m1.988438091s) [upload=12171]: Next segment err: context deadline exceeded Aug 7 21:22:02.399277657 (11h42m1.988622644s) [upload=12171]: Done uploading segments... Aug 7 21:22:02.399305961 (11h42m1.988650948s) [upload=12171]: Upload finished. err: context deadline exceeded Aug 7 21:22:03.257202341 (11h42m2.846547328s) [upload=12189]: Starting upload Aug 7 21:22:03.257267245 (11h42m2.846612232s) [upload=12189]: Done starting upload Aug 7 21:22:03.257356857 (11h42m2.846701844s) [upload=12189]: Uploading segments... Aug 7 21:22:03.257444765 (11h42m2.846789742s) [upload=12189]: Got next segment. Inline: false Aug 7 21:22:03.257470033 (11h42m2.84681501s) [upload=12189]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:22:03.257474171 (11h42m2.846819158s) [upload=12189]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:23:02.611478992 (11h43m2.20082398s) [upload=12189]: Next segment err: context deadline exceeded Aug 7 21:23:02.611504952 (11h43m2.200849939s) [upload=12189]: Done uploading segments... Aug 7 21:23:02.611523989 (11h43m2.200868976s) [upload=12189]: Upload finished. err: context deadline exceeded Aug 7 21:23:11.364660428 (11h43m10.954005435s) [upload=12204]: Starting upload Aug 7 21:23:11.364761051 (11h43m10.954106038s) [upload=12204]: Done starting upload Aug 7 21:23:11.364863036 (11h43m10.954208023s) [upload=12204]: Uploading segments... Aug 7 21:23:11.364973286 (11h43m10.954318264s) [upload=12204]: Got next segment. Inline: false Aug 7 21:23:11.365036508 (11h43m10.954381495s) [upload=12204]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:23:11.365041447 (11h43m10.954386434s) [upload=12204]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:24:10.722815877 (11h44m10.312160864s) [upload=12204]: Next segment err: context deadline exceeded Aug 7 21:24:10.722857978 (11h44m10.312202965s) [upload=12204]: Done uploading segments... Aug 7 21:24:10.722871453 (11h44m10.31221644s) [upload=12204]: Upload finished. err: context deadline exceeded Aug 7 21:24:23.410577309 (11h44m22.999922296s) [upload=12220]: Starting upload Aug 7 21:24:23.410633838 (11h44m22.999978825s) [upload=12220]: Done starting upload Aug 7 21:24:23.410696377 (11h44m23.000041364s) [upload=12220]: Uploading segments... Aug 7 21:24:23.410808262 (11h44m23.000153239s) [upload=12220]: Got next segment. Inline: false Aug 7 21:24:23.410833469 (11h44m23.000178456s) [upload=12220]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:24:23.410837017 (11h44m23.000182004s) [upload=12220]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:25:22.762515918 (11h45m22.351860905s) [upload=12220]: Next segment err: context deadline exceeded Aug 7 21:25:22.762663662 (11h45m22.352008639s) [upload=12220]: Done uploading segments... Aug 7 21:25:22.762679401 (11h45m22.352024378s) [upload=12220]: Upload finished. err: context deadline exceeded Aug 7 21:25:23.709743833 (11h45m23.29908882s) [upload=12237]: Starting upload Aug 7 21:25:23.709835399 (11h45m23.299180386s) [upload=12237]: Done starting upload Aug 7 21:25:23.709934688 (11h45m23.299279675s) [upload=12237]: Uploading segments... Aug 7 21:25:23.710036063 (11h45m23.29938105s) [upload=12237]: Got next segment. Inline: false Aug 7 21:25:23.710068875 (11h45m23.299413862s) [upload=12237]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:25:23.710072732 (11h45m23.299417719s) [upload=12237]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:26:23.059097010 (11h46m22.648441997s) [upload=12237]: Next segment err: context deadline exceeded Aug 7 21:26:23.059154440 (11h46m22.648499427s) [upload=12237]: Done uploading segments... Aug 7 21:26:23.059167756 (11h46m22.648512743s) [upload=12237]: Upload finished. err: context deadline exceeded Aug 7 21:26:32.555948975 (11h46m32.145293952s) [upload=12256]: Starting upload Aug 7 21:26:32.555997727 (11h46m32.145342704s) [upload=12256]: Done starting upload Aug 7 21:26:32.556084444 (11h46m32.145429421s) [upload=12256]: Uploading segments... Aug 7 21:26:32.556166260 (11h46m32.145511247s) [upload=12256]: Got next segment. Inline: false Aug 7 21:26:32.556194715 (11h46m32.145539702s) [upload=12256]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:26:32.556198572 (11h46m32.145543559s) [upload=12256]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:27:31.908976804 (11h47m31.498321791s) [upload=12256]: Next segment err: context deadline exceeded Aug 7 21:27:31.909049884 (11h47m31.498394861s) [upload=12256]: Done uploading segments... Aug 7 21:27:31.909064964 (11h47m31.498409951s) [upload=12256]: Upload finished. err: context deadline exceeded Aug 7 21:27:32.799954317 (11h47m32.389299304s) [upload=12272]: Starting upload Aug 7 21:27:32.800028559 (11h47m32.389373546s) [upload=12272]: Done starting upload Aug 7 21:27:32.800168577 (11h47m32.389513564s) [upload=12272]: Uploading segments... Aug 7 21:27:32.800449224 (11h47m32.389794211s) [upload=12272]: Got next segment. Inline: false Aug 7 21:27:32.800502656 (11h47m32.389847643s) [upload=12272]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:27:32.800507295 (11h47m32.389852272s) [upload=12272]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:28:32.145562627 (11h48m31.734907614s) [upload=12272]: Next segment err: context deadline exceeded Aug 7 21:28:32.145781085 (11h48m31.735126132s) [upload=12272]: Done uploading segments... Aug 7 21:28:32.145837483 (11h48m31.73518251s) [upload=12272]: Upload finished. err: context deadline exceeded Duration: 1m0.001080129s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [..........................................XXXXXXXXXXXXXXXXXX] duration: 30.000425201s FAIL (2025-08-07 21:28:31 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.000510744s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 6 out of 60 is failed (threshold=3) [....XX...........X.......................X.........X......X.] duration: 6.543056478s FAIL (2025-08-07 21:28:30 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/4M):: after 0c7bb902d1fc786d53d64877c5bf041e2c416fa3a23853fbffdc1a20ffb50890 != before bf5169665124c7207457a5823fc93758035c94fbdbbd7bf4beebc1b976aa7211 ======LOGS====== Aug 7 21:27:25.141891015 (11h47m24.731236002s) [upload=12268]: Got next segment. Inline: false Aug 7 21:27:25.141916354 (11h47m24.731261341s) [upload=12268]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:27:25.141920291 (11h47m24.731265278s) [upload=12268]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:27:25.654570615 (11h47m25.243915602s) [seg_pos={0 0}, upload=12268]: Begin upload segment... Aug 7 21:27:25.654611634 (11h47m25.243956621s) [seg_pos={0 0}, upload=12268]: Done begin upload segment. Aug 7 21:27:25.654634086 (11h47m25.243979073s) [noise=true , address=104.36.82.20:4448 , piece_id=X2KIKPSWI3SCJEEYILD4BVVHRIZZFDH4KGE5OHDIXEIM2HVBZ5NA, seg_pos={0 0}, upload=12268]: Uploading piece... Aug 7 21:27:25.654721364 (11h47m25.244066351s) [noise=true , address=104.36.82.22:4527 , piece_id=YCZOVT6QJT3VE2KFBC4DU6M34Y6DUBKTH3YTHDTO7ZQ7UFIKRAFQ, seg_pos={0 0}, upload=12268]: Uploading piece... Aug 7 21:27:25.655016387 (11h47m25.244361365s) [noise=true , address=104.36.82.46:4512 , piece_id=AWY2BPUJPBECXNJXDXYHMQWOZB3DGLTGP5776L677C64XIVW6ZCQ, seg_pos={0 0}, upload=12268]: Uploading piece... Aug 7 21:27:25.655133612 (11h47m25.2444786s) [noise=true , address=104.36.82.59:4329 , piece_id=CB2FO7D6IW7C2KSBJPR4J7XZQAM6NTHLMAW7XOB6JH6SHRGLQV7A, seg_pos={0 0}, upload=12268]: Uploading piece... Aug 7 21:27:25.655336751 (11h47m25.244681728s) [noise=true , address=104.36.82.55:4211 , piece_id=7IRAZ4EU7BMJNOGPAU7G6VIZ4TSTNCZK4PVFNLFPQM6E2234ME3Q, seg_pos={0 0}, upload=12268]: Uploading piece... Aug 7 21:27:25.655423657 (11h47m25.244768634s) [noise=true , address=104.36.82.37:4679 , piece_id=DOECPGUUPVOBOWQRAX5GY6BD5LA2VNYMLI34VEWZRC7KLT3AA2XA, seg_pos={0 0}, upload=12268]: Uploading piece... Aug 7 21:27:25.655662334 (11h47m25.245007311s) [noise=true , address=104.36.82.58:4279 , piece_id=OPZ353KP2L23TZGNZBO52DLTVSX66WL4D7GYM7TMTJOAT2RPOGHQ, seg_pos={0 0}, upload=12268]: Uploading piece... Aug 7 21:27:25.655733840 (11h47m25.245078827s) [noise=true , address=104.36.82.9:4019 , piece_id=ZDF7MVKTAV4AQD5XGAGHBDDB4TQQMBJGXWPKOBD2ONQI4BVLBFGQ, seg_pos={0 0}, upload=12268]: Uploading piece... Aug 7 21:27:25.656037241 (11h47m25.245382228s) [noise=true , address=104.36.82.2:4758 , piece_id=WFBYODP72YMTCPMKQQKVLNCWKMAETBHH6WOQ5UCEXYCBZSY5WWRA, seg_pos={0 0}, upload=12268]: Uploading piece... Aug 7 21:27:25.656051578 (11h47m25.245396566s) [noise=true , address=104.36.82.54:4157 , piece_id=PGULUJFV37PY3FOW2OD3PKK2UGB7RWDNSWPYZJKVQRWGONRSKEXQ, seg_pos={0 0}, upload=12268]: Uploading piece... Aug 7 21:27:25.656335652 (11h47m25.245680639s) [noise=true , address=104.36.82.36:4633 , piece_id=NXYS3XJMQZAJNSPGBFLXTVR4ZDB5RYHB5K4TUTJ4DANWVSQYEAIQ, seg_pos={0 0}, upload=12268]: Uploading piece... Aug 7 21:27:25.656354137 (11h47m25.245699124s) [noise=true , address=45.91.116.8:4054 , piece_id=PHJGQGBYXAZCULVJX5KSSLMUPOGMD6S7I67QVJBQE35ZFT5U6HFA, seg_pos={0 0}, upload=12268]: Uploading piece... Aug 7 21:27:25.656550413 (11h47m25.24589539s) [noise=true , address=104.36.82.60:4345 , piece_id=L5CI5AMDNOXC2TNERZEBAC2SW3PU4NXI5ILTSTAPA2TSHNEQNHHQ, seg_pos={0 0}, upload=12268]: Uploading piece... Aug 7 21:27:25.656587022 (11h47m25.24593201s) [noise=true , address=79.127.205.234:4586 , piece_id=GI36ODJVM2GYWC2WHFASOQUKJMSJ3R4JANFQ5B45OAFNTBGUEBMA, seg_pos={0 0}, upload=12268]: Uploading piece... Aug 7 21:27:25.656785703 (11h47m25.24613069s) [noise=true , address=104.36.82.43:4837 , piece_id=AQ2NEBR5MYDUQYSKIYKLQJOGRIHKGD3F5QM36GXTECXX37R6H5SQ, seg_pos={0 0}, upload=12268]: Uploading piece... Aug 7 21:27:25.656804429 (11h47m25.246149406s) [noise=true , address=104.36.82.20:4439 , piece_id=CJ7R576SA66NFSO7OHECHKCHULKZN7EZDUTQ3YGRWE436DKO4MWA, seg_pos={0 0}, upload=12268]: Uploading piece... Aug 7 21:27:25.657025892 (11h47m25.246370879s) [noise=true , address=79.127.205.233:4632 , piece_id=T4VQ3H6ZHBIXZN7X47ZWK4WYQBD5KTGSJWKH6R3PM7UCFEGHKOFQ, seg_pos={0 0}, upload=12268]: Uploading piece... Aug 7 21:27:25.657252346 (11h47m25.246597333s) [noise=true , address=45.91.116.14:4218 , piece_id=X7RNM4ZIW35EBZJNTVT5NYPQKS7K4JNT2RQW2X5D7JVGQQZKHV3Q, seg_pos={0 0}, upload=12268]: Uploading piece... Aug 7 21:27:26.765000728 (11h47m26.354345715s) [upload=12268]: Next returned nil segment Aug 7 21:27:26.765014103 (11h47m26.35435909s) [upload=12268]: Waiting for error group managing segments... Aug 7 21:27:27.057992337 (11h47m26.647337324s) [noise=true , address=79.127.205.234:4586 , piece_id=GI36ODJVM2GYWC2WHFASOQUKJMSJ3R4JANFQ5B45OAFNTBGUEBMA, seg_pos={0 0}, upload=12268]: Done uploading piece. err: <nil> Aug 7 21:27:27.086935163 (11h47m26.67628014s) [noise=true , address=79.127.205.233:4632 , piece_id=T4VQ3H6ZHBIXZN7X47ZWK4WYQBD5KTGSJWKH6R3PM7UCFEGHKOFQ, seg_pos={0 0}, upload=12268]: Done uploading piece. err: <nil> Aug 7 21:27:27.601292823 (11h47m27.1906378s) [noise=true , address=104.36.82.2:4758 , piece_id=WFBYODP72YMTCPMKQQKVLNCWKMAETBHH6WOQ5UCEXYCBZSY5WWRA, seg_pos={0 0}, upload=12268]: Done uploading piece. err: <nil> Aug 7 21:27:27.699555195 (11h47m27.288900192s) [noise=true , address=104.36.82.9:4019 , piece_id=ZDF7MVKTAV4AQD5XGAGHBDDB4TQQMBJGXWPKOBD2ONQI4BVLBFGQ, seg_pos={0 0}, upload=12268]: Done uploading piece. err: <nil> Aug 7 21:27:27.710943893 (11h47m27.30028888s) [noise=true , address=104.36.82.43:4837 , piece_id=AQ2NEBR5MYDUQYSKIYKLQJOGRIHKGD3F5QM36GXTECXX37R6H5SQ, seg_pos={0 0}, upload=12268]: Done uploading piece. err: <nil> Aug 7 21:27:27.719413398 (11h47m27.308758385s) [noise=true , address=104.36.82.46:4512 , piece_id=AWY2BPUJPBECXNJXDXYHMQWOZB3DGLTGP5776L677C64XIVW6ZCQ, seg_pos={0 0}, upload=12268]: Done uploading piece. err: <nil> Aug 7 21:27:27.751471103 (11h47m27.34081608s) [noise=true , address=104.36.82.58:4279 , piece_id=OPZ353KP2L23TZGNZBO52DLTVSX66WL4D7GYM7TMTJOAT2RPOGHQ, seg_pos={0 0}, upload=12268]: Done uploading piece. err: <nil> Aug 7 21:27:27.752665698 (11h47m27.342010685s) [noise=true , address=104.36.82.36:4633 , piece_id=NXYS3XJMQZAJNSPGBFLXTVR4ZDB5RYHB5K4TUTJ4DANWVSQYEAIQ, seg_pos={0 0}, upload=12268]: Done uploading piece. err: <nil> Aug 7 21:27:27.803219381 (11h47m27.392564368s) [noise=true , address=104.36.82.55:4211 , piece_id=7IRAZ4EU7BMJNOGPAU7G6VIZ4TSTNCZK4PVFNLFPQM6E2234ME3Q, seg_pos={0 0}, upload=12268]: Done uploading piece. err: <nil> Aug 7 21:27:27.855671476 (11h47m27.445016463s) [noise=true , address=104.36.82.22:4527 , piece_id=YCZOVT6QJT3VE2KFBC4DU6M34Y6DUBKTH3YTHDTO7ZQ7UFIKRAFQ, seg_pos={0 0}, upload=12268]: Done uploading piece. err: <nil> Aug 7 21:27:27.879451272 (11h47m27.468796259s) [noise=true , address=104.36.82.20:4439 , piece_id=CJ7R576SA66NFSO7OHECHKCHULKZN7EZDUTQ3YGRWE436DKO4MWA, seg_pos={0 0}, upload=12268]: Done uploading piece. err: <nil> Aug 7 21:27:27.941304094 (11h47m27.530649081s) [noise=true , address=45.91.116.8:4054 , piece_id=PHJGQGBYXAZCULVJX5KSSLMUPOGMD6S7I67QVJBQE35ZFT5U6HFA, seg_pos={0 0}, upload=12268]: Done uploading piece. err: <nil> Aug 7 21:27:28.313392665 (11h47m27.902737652s) [noise=true , address=45.91.116.14:4218 , piece_id=X7RNM4ZIW35EBZJNTVT5NYPQKS7K4JNT2RQW2X5D7JVGQQZKHV3Q, seg_pos={0 0}, upload=12268]: Done uploading piece. err: <nil> Aug 7 21:27:28.385132253 (11h47m27.97447724s) [noise=true , address=104.36.82.59:4329 , piece_id=CB2FO7D6IW7C2KSBJPR4J7XZQAM6NTHLMAW7XOB6JH6SHRGLQV7A, seg_pos={0 0}, upload=12268]: Done uploading piece. err: <nil> Aug 7 21:27:28.385181175 (11h47m27.974526153s) [seg_pos={0 0}, upload=12268]: Segment reached optimal threshold of 14 pieces. Aug 7 21:27:28.385286848 (11h47m27.974631835s) [noise=true , address=104.36.82.54:4157 , piece_id=PGULUJFV37PY3FOW2OD3PKK2UGB7RWDNSWPYZJKVQRWGONRSKEXQ, seg_pos={0 0}, upload=12268]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZrzADk2mKG4R9tjxUKjQeJvb4WmfjLksGFyvstHRfYbBC7XNz): protocol: expected piece hash; context canceled; EOF Aug 7 21:27:28.385451353 (11h47m27.97479633s) [noise=true , address=104.36.82.20:4448 , piece_id=X2KIKPSWI3SCJEEYILD4BVVHRIZZFDH4KGE5OHDIXEIM2HVBZ5NA, seg_pos={0 0}, upload=12268]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12vbYQVrjLwiHTMGsAGAcjze8KeVQUqemqGiVTWF63Xq6F7ABUp): protocol: read tcp 188.166.225.170:56322->104.36.82.20:4448: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 21:27:28.385487432 (11h47m27.974832419s) [noise=true , address=104.36.82.37:4679 , piece_id=DOECPGUUPVOBOWQRAX5GY6BD5LA2VNYMLI34VEWZRC7KLT3AA2XA, seg_pos={0 0}, upload=12268]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VHQihNGcx6bTvwitH9PCSqyuKWuchdYhYzZXwSuoruQFByGoj): protocol: rpc: dial tcp 104.36.82.37:4679: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 21:27:28.385527619 (11h47m27.974872606s) [noise=true , address=104.36.82.60:4345 , piece_id=L5CI5AMDNOXC2TNERZEBAC2SW3PU4NXI5ILTSTAPA2TSHNEQNHHQ, seg_pos={0 0}, upload=12268]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KSTyRJ62wcMHx8K7uwqp5e8L8yhxH7V3YtBHnYu6QJ3K3bXpQ): protocol: read tcp 188.166.225.170:45768->104.36.82.60:4345: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 21:27:28.385544672 (11h47m27.974889659s) [seg_pos={0 0}, upload=12268]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 21:27:28.385597523 (11h47m27.97494251s) [upload=12268]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 21:27:28.385601019 (11h47m27.974946007s) [upload=12268]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:27:28.699165600 (11h47m28.288510587s) [upload=12268]: Done uploading segments... Aug 7 21:27:28.699200797 (11h47m28.288545784s) [upload=12268]: Upload finished. err: <nil> Duration: 6.54308295s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway) antiblip: 3 out of 60 is failed (threshold=3) [........................................................XX.X] duration: 2m0.001236527s remote: "tcp:207.211.208.129:443" target: https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222 FAIL (2025-08-07 21:28:29 UTC) orbiter warning: Get "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222": context deadline exceeded Duration: 2m0.001259491s edge gateway vhost bgp us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [..............X...................................X.........] duration: 1.886310903s FAIL (2025-08-07 21:28:28 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/1K") ======LOGS====== ]: Done uploading segments... Aug 7 21:13:55.370929874 (11h33m54.960274861s) [upload=12060]: Upload finished. err: <nil> Aug 7 21:14:45.614661829 (11h34m45.204006816s) [upload=12073]: Starting upload Aug 7 21:14:45.614700943 (11h34m45.20404593s) [upload=12073]: Done starting upload Aug 7 21:14:45.614745929 (11h34m45.204090917s) [upload=12073]: Uploading segments... Aug 7 21:14:45.614774204 (11h34m45.204119181s) [upload=12073]: Got next segment. Inline: true Aug 7 21:14:45.614777830 (11h34m45.204122817s) [upload=12073]: Waiting for error group managing segments... Aug 7 21:14:45.614809601 (11h34m45.204154578s) [upload=12073]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:14:45.614813057 (11h34m45.204158045s) [upload=12073]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:14:45.614815242 (11h34m45.204160219s) [upload=12073]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:14:45.911976434 (11h34m45.501321421s) [upload=12073]: Done uploading segments... Aug 7 21:14:45.911995080 (11h34m45.501340067s) [upload=12073]: Upload finished. err: <nil> Aug 7 21:15:14.898649385 (11h35m14.487994372s) [upload=12079]: Starting upload Aug 7 21:15:14.898694180 (11h35m14.488039167s) [upload=12079]: Done starting upload Aug 7 21:15:14.898741050 (11h35m14.488086027s) [upload=12079]: Uploading segments... Aug 7 21:15:14.898781948 (11h35m14.488126965s) [upload=12079]: Got next segment. Inline: true Aug 7 21:15:14.898789092 (11h35m14.488134079s) [upload=12079]: Waiting for error group managing segments... Aug 7 21:15:14.898847644 (11h35m14.488192641s) [upload=12079]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:15:14.898856401 (11h35m14.488201408s) [upload=12079]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:15:14.898860989 (11h35m14.488206026s) [upload=12079]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:15:15.228046082 (11h35m14.817391079s) [upload=12079]: Done uploading segments... Aug 7 21:15:15.228066881 (11h35m14.817411878s) [upload=12079]: Upload finished. err: <nil> Aug 7 21:16:25.941982337 (11h36m25.531327324s) [upload=12099]: Starting upload Aug 7 21:16:25.942029607 (11h36m25.531374594s) [upload=12099]: Done starting upload Aug 7 21:16:25.942075234 (11h36m25.531420221s) [upload=12099]: Uploading segments... Aug 7 21:16:25.942105222 (11h36m25.531450199s) [upload=12099]: Got next segment. Inline: true Aug 7 21:16:25.942108588 (11h36m25.531453575s) [upload=12099]: Waiting for error group managing segments... Aug 7 21:16:25.942142683 (11h36m25.53148767s) [upload=12099]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:16:25.942146340 (11h36m25.531491317s) [upload=12099]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:16:25.942148595 (11h36m25.531493582s) [upload=12099]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:16:26.215688283 (11h36m25.80503327s) [upload=12099]: Done uploading segments... Aug 7 21:16:26.215705086 (11h36m25.805050073s) [upload=12099]: Upload finished. err: <nil> Aug 7 21:17:38.285379669 (11h37m37.874724656s) [upload=12117]: Starting upload Aug 7 21:17:38.285441276 (11h37m37.874786323s) [upload=12117]: Done starting upload Aug 7 21:17:38.285507584 (11h37m37.874852611s) [upload=12117]: Uploading segments... Aug 7 21:17:38.285571005 (11h37m37.874915992s) [upload=12117]: Got next segment. Inline: true Aug 7 21:17:38.285575654 (11h37m37.874920641s) [upload=12117]: Waiting for error group managing segments... Aug 7 21:17:38.285618215 (11h37m37.874963243s) [upload=12117]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:17:38.285624958 (11h37m37.874969945s) [upload=12117]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:17:38.285629477 (11h37m37.874974474s) [upload=12117]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:17:38.561705487 (11h37m38.151050484s) [upload=12117]: Done uploading segments... Aug 7 21:17:38.561727969 (11h37m38.151072957s) [upload=12117]: Upload finished. err: <nil> Aug 7 21:18:46.880586061 (11h38m46.469931048s) [upload=12133]: Starting upload Aug 7 21:18:46.880627500 (11h38m46.469972487s) [upload=12133]: Done starting upload Aug 7 21:18:46.880681523 (11h38m46.47002651s) [upload=12133]: Uploading segments... Aug 7 21:18:46.880714476 (11h38m46.470059463s) [upload=12133]: Got next segment. Inline: true Aug 7 21:18:46.880718043 (11h38m46.47006302s) [upload=12133]: Waiting for error group managing segments... Aug 7 21:18:46.880752089 (11h38m46.470097076s) [upload=12133]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:18:46.880755715 (11h38m46.470100702s) [upload=12133]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:18:46.880758120 (11h38m46.470103107s) [upload=12133]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:18:47.149461450 (11h38m46.738806437s) [upload=12133]: Done uploading segments... Aug 7 21:18:47.149478763 (11h38m46.73882375s) [upload=12133]: Upload finished. err: <nil> Aug 7 21:19:44.345616276 (11h39m43.934961263s) [upload=12152]: Starting upload Aug 7 21:19:44.345660120 (11h39m43.935005098s) [upload=12152]: Done starting upload Aug 7 21:19:44.345731898 (11h39m43.935076885s) [upload=12152]: Uploading segments... Aug 7 21:19:44.345839653 (11h39m43.93518463s) [upload=12152]: Got next segment. Inline: true Aug 7 21:19:44.345844203 (11h39m43.93518919s) [upload=12152]: Waiting for error group managing segments... Aug 7 21:19:44.345886133 (11h39m43.93523111s) [upload=12152]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:19:44.345890932 (11h39m43.935235919s) [upload=12152]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:19:44.345893316 (11h39m43.935238303s) [upload=12152]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:19:44.666041542 (11h39m44.255386529s) [upload=12152]: Done uploading segments... Aug 7 21:19:44.666059567 (11h39m44.255404554s) [upload=12152]: Upload finished. err: <nil> Aug 7 21:20:10.176957983 (11h40m9.76630296s) [upload=12157]: Starting upload Aug 7 21:20:10.177009311 (11h40m9.766354298s) [upload=12157]: Done starting upload Aug 7 21:20:10.177070559 (11h40m9.766415546s) [upload=12157]: Uploading segments... Aug 7 21:20:10.177121496 (11h40m9.766466483s) [upload=12157]: Got next segment. Inline: true Aug 7 21:20:10.177125895 (11h40m9.766470882s) [upload=12157]: Waiting for error group managing segments... Aug 7 21:20:10.177160220 (11h40m9.766505207s) [upload=12157]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:20:10.177164087 (11h40m9.766509074s) [upload=12157]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:20:10.177166371 (11h40m9.766511359s) [upload=12157]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:20:10.463794245 (11h40m10.053139232s) [upload=12157]: Done uploading segments... Aug 7 21:20:10.463812280 (11h40m10.053157257s) [upload=12157]: Upload finished. err: <nil> Aug 7 21:20:47.295095407 (11h40m46.884440394s) [upload=12169]: Starting upload Aug 7 21:20:47.295162806 (11h40m46.884507793s) [upload=12169]: Done starting upload Aug 7 21:20:47.295214385 (11h40m46.884559372s) [upload=12169]: Uploading segments... Aug 7 21:20:47.295251876 (11h40m46.884596864s) [upload=12169]: Got next segment. Inline: true Aug 7 21:20:47.295255583 (11h40m46.88460057s) [upload=12169]: Waiting for error group managing segments... Aug 7 21:20:47.295291642 (11h40m46.884636629s) [upload=12169]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:20:47.295295450 (11h40m46.884640437s) [upload=12169]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:20:47.295297755 (11h40m46.884642742s) [upload=12169]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:20:47.581180532 (11h40m47.170525529s) [upload=12169]: Done uploading segments... Aug 7 21:20:47.581201993 (11h40m47.17054698s) [upload=12169]: Upload finished. err: <nil> Aug 7 21:21:39.503326248 (11h41m39.092671235s) [upload=12184]: Starting upload Aug 7 21:21:39.503368609 (11h41m39.092713586s) [upload=12184]: Done starting upload Aug 7 21:21:39.503416451 (11h41m39.092761438s) [upload=12184]: Uploading segments... Aug 7 21:21:39.503447480 (11h41m39.092792467s) [upload=12184]: Got next segment. Inline: true Aug 7 21:21:39.503451247 (11h41m39.092796234s) [upload=12184]: Waiting for error group managing segments... Aug 7 21:21:39.503485393 (11h41m39.09283038s) [upload=12184]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:21:39.503489370 (11h41m39.092834357s) [upload=12184]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:21:39.503491825 (11h41m39.092836802s) [upload=12184]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:21:39.826158170 (11h41m39.415503147s) [upload=12184]: Done uploading segments... Aug 7 21:21:39.826174040 (11h41m39.415519027s) [upload=12184]: Upload finished. err: <nil> Aug 7 21:22:46.427091203 (11h42m46.01643619s) [upload=12199]: Starting upload Aug 7 21:22:46.427135929 (11h42m46.016480916s) [upload=12199]: Done starting upload Aug 7 21:22:46.427188519 (11h42m46.016533496s) [upload=12199]: Uploading segments... Aug 7 21:22:46.427215882 (11h42m46.016560869s) [upload=12199]: Got next segment. Inline: true Aug 7 21:22:46.427219829 (11h42m46.016564806s) [upload=12199]: Waiting for error group managing segments... Aug 7 21:22:46.427252391 (11h42m46.016597378s) [upload=12199]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:22:46.427256048 (11h42m46.016601025s) [upload=12199]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:22:46.427268202 (11h42m46.016613189s) [upload=12199]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:22:46.705229475 (11h42m46.294574462s) [upload=12199]: Done uploading segments... Aug 7 21:22:46.705247530 (11h42m46.294592507s) [upload=12199]: Upload finished. err: <nil> Duration: 1.886336873s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [.......................................X..XXXXXXXXXXXXXXXXXX] duration: 30.001004069s FAIL (2025-08-07 21:28:25 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== Next segment err: context deadline exceeded Aug 7 21:19:21.302254121 (11h39m20.891599108s) [upload=12135]: Done uploading segments... Aug 7 21:19:21.302274270 (11h39m20.891619267s) [upload=12135]: Upload finished. err: context deadline exceeded Aug 7 21:19:39.710866482 (11h39m39.300211459s) [upload=12149]: Starting upload Aug 7 21:19:39.710923851 (11h39m39.300268829s) [upload=12149]: Done starting upload Aug 7 21:19:39.710996290 (11h39m39.300341277s) [upload=12149]: Uploading segments... Aug 7 21:19:39.711114146 (11h39m39.300459133s) [upload=12149]: Got next segment. Inline: false Aug 7 21:19:39.711140085 (11h39m39.300485102s) [upload=12149]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:19:39.711144824 (11h39m39.300489811s) [upload=12149]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:20:09.053604686 (11h40m8.642949683s) [upload=12149]: Next segment err: context deadline exceeded Aug 7 21:20:09.053707913 (11h40m8.6430529s) [upload=12149]: Done uploading segments... Aug 7 21:20:09.053723724 (11h40m8.643068711s) [upload=12149]: Upload finished. err: context deadline exceeded Aug 7 21:20:24.058132378 (11h40m23.647477365s) [upload=12163]: Starting upload Aug 7 21:20:24.058185559 (11h40m23.647530546s) [upload=12163]: Done starting upload Aug 7 21:20:24.058265552 (11h40m23.647610529s) [upload=12163]: Uploading segments... Aug 7 21:20:24.058369542 (11h40m23.647714539s) [upload=12163]: Got next segment. Inline: false Aug 7 21:20:24.058412013 (11h40m23.647757s) [upload=12163]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:20:24.058418646 (11h40m23.647763663s) [upload=12163]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:20:53.409396630 (11h40m52.998741617s) [upload=12163]: Next segment err: context deadline exceeded Aug 7 21:20:53.409563509 (11h40m52.998908496s) [upload=12163]: Done uploading segments... Aug 7 21:20:53.409604537 (11h40m52.998949514s) [upload=12163]: Upload finished. err: context deadline exceeded Aug 7 21:21:23.928736473 (11h41m23.51808146s) [upload=12180]: Starting upload Aug 7 21:21:23.928792831 (11h41m23.518137818s) [upload=12180]: Done starting upload Aug 7 21:21:23.928856973 (11h41m23.51820195s) [upload=12180]: Uploading segments... Aug 7 21:21:23.929051695 (11h41m23.518396682s) [upload=12180]: Got next segment. Inline: false Aug 7 21:21:23.929103304 (11h41m23.518448291s) [upload=12180]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:21:23.929107933 (11h41m23.51845292s) [upload=12180]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:21:53.279642106 (11h41m52.868987093s) [upload=12180]: Next segment err: context deadline exceeded Aug 7 21:21:53.279858500 (11h41m52.869203477s) [upload=12180]: Done uploading segments... Aug 7 21:21:53.279873498 (11h41m52.869218485s) [upload=12180]: Upload finished. err: context deadline exceeded Aug 7 21:21:54.191662378 (11h41m53.781007366s) [upload=12188]: Starting upload Aug 7 21:21:54.191712274 (11h41m53.781057251s) [upload=12188]: Done starting upload Aug 7 21:21:54.191790404 (11h41m53.781135391s) [upload=12188]: Uploading segments... Aug 7 21:21:54.192007459 (11h41m53.781352446s) [upload=12188]: Got next segment. Inline: false Aug 7 21:21:54.192034671 (11h41m53.781379658s) [upload=12188]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:21:54.192038719 (11h41m53.781383706s) [upload=12188]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:22:23.533395958 (11h42m23.122740945s) [upload=12188]: Next segment err: context deadline exceeded Aug 7 21:22:23.533474659 (11h42m23.122819646s) [upload=12188]: Done uploading segments... Aug 7 21:22:23.533496701 (11h42m23.122841688s) [upload=12188]: Upload finished. err: context deadline exceeded Aug 7 21:23:08.442262622 (11h43m8.031607609s) [upload=12203]: Starting upload Aug 7 21:23:08.442325042 (11h43m8.031670019s) [upload=12203]: Done starting upload Aug 7 21:23:08.442385668 (11h43m8.031730655s) [upload=12203]: Uploading segments... Aug 7 21:23:08.442501099 (11h43m8.031846076s) [upload=12203]: Got next segment. Inline: false Aug 7 21:23:08.442542839 (11h43m8.031887826s) [upload=12203]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:23:08.442546706 (11h43m8.031891683s) [upload=12203]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:23:37.798210470 (11h43m37.387555457s) [upload=12203]: Next segment err: context deadline exceeded Aug 7 21:23:37.798367780 (11h43m37.387712767s) [upload=12203]: Done uploading segments... Aug 7 21:23:37.798381718 (11h43m37.387726705s) [upload=12203]: Upload finished. err: context deadline exceeded Aug 7 21:23:54.967168247 (11h43m54.556513234s) [upload=12215]: Starting upload Aug 7 21:23:54.967268208 (11h43m54.556613185s) [upload=12215]: Done starting upload Aug 7 21:23:54.967374321 (11h43m54.556719308s) [upload=12215]: Uploading segments... Aug 7 21:23:54.967600634 (11h43m54.556945621s) [upload=12215]: Got next segment. Inline: false Aug 7 21:23:54.967649888 (11h43m54.556994875s) [upload=12215]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:23:54.967657021 (11h43m54.557002029s) [upload=12215]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:24:24.312833601 (11h44m23.902178598s) [upload=12215]: Next segment err: context deadline exceeded Aug 7 21:24:24.313044925 (11h44m23.902389912s) [upload=12215]: Done uploading segments... Aug 7 21:24:24.313087757 (11h44m23.902432764s) [upload=12215]: Upload finished. err: context deadline exceeded Aug 7 21:24:25.196484464 (11h44m24.785829451s) [upload=12221]: Starting upload Aug 7 21:24:25.196569166 (11h44m24.785914153s) [upload=12221]: Done starting upload Aug 7 21:24:25.196697702 (11h44m24.786042719s) [upload=12221]: Uploading segments... Aug 7 21:24:25.196786912 (11h44m24.786131899s) [upload=12221]: Got next segment. Inline: false Aug 7 21:24:25.196818883 (11h44m24.78616387s) [upload=12221]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:24:25.196823522 (11h44m24.786168509s) [upload=12221]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:24:54.542928213 (11h44m54.1322732s) [upload=12221]: Next segment err: context deadline exceeded Aug 7 21:24:54.543143736 (11h44m54.132488713s) [upload=12221]: Done uploading segments... Aug 7 21:24:54.543169415 (11h44m54.132514402s) [upload=12221]: Upload finished. err: context deadline exceeded Aug 7 21:25:30.578713383 (11h45m30.16805837s) [upload=12239]: Starting upload Aug 7 21:25:30.578779109 (11h45m30.168124096s) [upload=12239]: Done starting upload Aug 7 21:25:30.578836729 (11h45m30.168181716s) [upload=12239]: Uploading segments... Aug 7 21:25:30.579017655 (11h45m30.168362642s) [upload=12239]: Got next segment. Inline: false Aug 7 21:25:30.579083341 (11h45m30.168428328s) [upload=12239]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:25:30.579091086 (11h45m30.168436073s) [upload=12239]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:25:59.907330605 (11h45m59.496675592s) [upload=12239]: Next segment err: context deadline exceeded Aug 7 21:25:59.907404857 (11h45m59.496749845s) [upload=12239]: Done uploading segments... Aug 7 21:25:59.907422762 (11h45m59.496767749s) [upload=12239]: Upload finished. err: context deadline exceeded Aug 7 21:26:16.857419024 (11h46m16.446764011s) [upload=12250]: Starting upload Aug 7 21:26:16.857477205 (11h46m16.446822182s) [upload=12250]: Done starting upload Aug 7 21:26:16.857572879 (11h46m16.446917856s) [upload=12250]: Uploading segments... Aug 7 21:26:16.857644405 (11h46m16.446989392s) [upload=12250]: Got next segment. Inline: false Aug 7 21:26:16.857669784 (11h46m16.447014771s) [upload=12250]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:26:16.857673381 (11h46m16.447018368s) [upload=12250]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:26:46.211380241 (11h46m45.800725218s) [upload=12250]: Next segment err: context deadline exceeded Aug 7 21:26:46.211514648 (11h46m45.800859625s) [upload=12250]: Done uploading segments... Aug 7 21:26:46.211531050 (11h46m45.800876037s) [upload=12250]: Upload finished. err: context deadline exceeded Aug 7 21:27:19.636960898 (11h47m19.226305885s) [upload=12266]: Starting upload Aug 7 21:27:19.637068003 (11h47m19.22641299s) [upload=12266]: Done starting upload Aug 7 21:27:19.637205767 (11h47m19.226550744s) [upload=12266]: Uploading segments... Aug 7 21:27:19.637342969 (11h47m19.226687956s) [upload=12266]: Got next segment. Inline: false Aug 7 21:27:19.637391532 (11h47m19.22673655s) [upload=12266]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:27:19.637399387 (11h47m19.226744374s) [upload=12266]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:27:48.990525160 (11h47m48.579870148s) [upload=12266]: Next segment err: context deadline exceeded Aug 7 21:27:48.990618549 (11h47m48.579963526s) [upload=12266]: Done uploading segments... Aug 7 21:27:48.990636003 (11h47m48.57998099s) [upload=12266]: Upload finished. err: context deadline exceeded Aug 7 21:27:56.650673215 (11h47m56.240018202s) [upload=12276]: Starting upload Aug 7 21:27:56.650729763 (11h47m56.24007474s) [upload=12276]: Done starting upload Aug 7 21:27:56.650797482 (11h47m56.240142469s) [upload=12276]: Uploading segments... Aug 7 21:27:56.650928032 (11h47m56.240273019s) [upload=12276]: Got next segment. Inline: false Aug 7 21:27:56.650969893 (11h47m56.24031487s) [upload=12276]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:27:56.650973740 (11h47m56.240318727s) [upload=12276]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:28:25.997700487 (11h48m25.587045464s) [upload=12276]: Next segment err: context deadline exceeded Aug 7 21:28:25.997775731 (11h48m25.587120708s) [upload=12276]: Done uploading segments... Aug 7 21:28:25.997832740 (11h48m25.587177727s) [upload=12276]: Upload finished. err: context deadline exceeded Duration: 30.001022156s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 19 out of 60 is failed (threshold=3) [.........................................XXXXXXXXXXXXXXXXXXX] duration: 30.001409583s FAIL (2025-08-07 21:28:24 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.001419543s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 15 out of 60 is failed (threshold=3) [............................................X.XXXXXXXXXXXXXX] duration: 1m0.002899696s FAIL (2025-08-07 21:28:17 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 12481 (11h37m46.031957468s) [upload=12121]: Uploading segments... Aug 7 21:17:46.442661796 (11h37m46.032006783s) [upload=12121]: Got next segment. Inline: true Aug 7 21:17:46.442665573 (11h37m46.03201056s) [upload=12121]: Waiting for error group managing segments... Aug 7 21:17:46.442697183 (11h37m46.03204217s) [upload=12121]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:17:46.442700580 (11h37m46.032045557s) [upload=12121]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:17:46.442704637 (11h37m46.032049624s) [upload=12121]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:18:45.798465603 (11h38m45.38781059s) [upload=12121]: Done uploading segments... Aug 7 21:18:45.798489779 (11h38m45.387834766s) [upload=12121]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:18:57.960074990 (11h38m57.549419977s) [upload=12139]: Starting upload Aug 7 21:18:57.960165033 (11h38m57.54951003s) [upload=12139]: Done starting upload Aug 7 21:18:57.960244845 (11h38m57.549589842s) [upload=12139]: Uploading segments... Aug 7 21:18:57.960294299 (11h38m57.549639286s) [upload=12139]: Got next segment. Inline: true Aug 7 21:18:57.960298928 (11h38m57.549643915s) [upload=12139]: Waiting for error group managing segments... Aug 7 21:18:57.960349855 (11h38m57.549694853s) [upload=12139]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:18:57.960354916 (11h38m57.549699903s) [upload=12139]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:18:57.960358112 (11h38m57.549703109s) [upload=12139]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:19:57.313173678 (11h39m56.902518665s) [upload=12139]: Done uploading segments... Aug 7 21:19:57.313193636 (11h39m56.902538623s) [upload=12139]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:20:04.133582941 (11h40m3.722927928s) [upload=12156]: Starting upload Aug 7 21:20:04.133643998 (11h40m3.722988985s) [upload=12156]: Done starting upload Aug 7 21:20:04.133712729 (11h40m3.723057726s) [upload=12156]: Uploading segments... Aug 7 21:20:04.133764127 (11h40m3.723109114s) [upload=12156]: Got next segment. Inline: true Aug 7 21:20:04.133769046 (11h40m3.723114033s) [upload=12156]: Waiting for error group managing segments... Aug 7 21:20:04.133838079 (11h40m3.723183066s) [upload=12156]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:20:04.133842868 (11h40m3.723187855s) [upload=12156]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:20:04.133845723 (11h40m3.72319071s) [upload=12156]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:21:03.481385970 (11h41m3.070730947s) [upload=12156]: Done uploading segments... Aug 7 21:21:03.481407009 (11h41m3.070751986s) [upload=12156]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:21:04.366237551 (11h41m3.955582538s) [upload=12172]: Starting upload Aug 7 21:21:04.366309369 (11h41m3.955654356s) [upload=12172]: Done starting upload Aug 7 21:21:04.366360317 (11h41m3.955705304s) [upload=12172]: Uploading segments... Aug 7 21:21:04.366406304 (11h41m3.955751291s) [upload=12172]: Got next segment. Inline: true Aug 7 21:21:04.366410022 (11h41m3.955755009s) [upload=12172]: Waiting for error group managing segments... Aug 7 21:21:04.366441823 (11h41m3.95578681s) [upload=12172]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:21:04.366445840 (11h41m3.955790827s) [upload=12172]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:21:04.366448265 (11h41m3.955793252s) [upload=12172]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:22:03.717958127 (11h42m3.307303114s) [upload=12172]: Done uploading segments... Aug 7 21:22:03.717976892 (11h42m3.307321879s) [upload=12172]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:22:11.311019027 (11h42m10.900364014s) [upload=12193]: Starting upload Aug 7 21:22:11.311074243 (11h42m10.90041923s) [upload=12193]: Done starting upload Aug 7 21:22:11.311123727 (11h42m10.900468714s) [upload=12193]: Uploading segments... Aug 7 21:22:11.311167090 (11h42m10.900512077s) [upload=12193]: Got next segment. Inline: true Aug 7 21:22:11.311183532 (11h42m10.900528519s) [upload=12193]: Waiting for error group managing segments... Aug 7 21:22:11.311220152 (11h42m10.900565139s) [upload=12193]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:22:11.311224931 (11h42m10.900569918s) [upload=12193]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:22:11.311227586 (11h42m10.900572573s) [upload=12193]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:23:10.658643233 (11h43m10.24798821s) [upload=12193]: Done uploading segments... Aug 7 21:23:10.658662571 (11h43m10.248007558s) [upload=12193]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:23:11.571475028 (11h43m11.160820015s) [upload=12205]: Starting upload Aug 7 21:23:11.571532048 (11h43m11.160877035s) [upload=12205]: Done starting upload Aug 7 21:23:11.571582173 (11h43m11.16092716s) [upload=12205]: Uploading segments... Aug 7 21:23:11.571617942 (11h43m11.160962929s) [upload=12205]: Got next segment. Inline: true Aug 7 21:23:11.571621489 (11h43m11.160966486s) [upload=12205]: Waiting for error group managing segments... Aug 7 21:23:11.571655484 (11h43m11.161000471s) [upload=12205]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:23:11.571659531 (11h43m11.161004528s) [upload=12205]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:23:11.571662447 (11h43m11.161007444s) [upload=12205]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:24:10.926729728 (11h44m10.516074715s) [upload=12205]: Done uploading segments... Aug 7 21:24:10.926747603 (11h44m10.51609259s) [upload=12205]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:24:11.802089650 (11h44m11.391434637s) [upload=12218]: Starting upload Aug 7 21:24:11.802151558 (11h44m11.391496545s) [upload=12218]: Done starting upload Aug 7 21:24:11.802205020 (11h44m11.391549997s) [upload=12218]: Uploading segments... Aug 7 21:24:11.802240418 (11h44m11.391585406s) [upload=12218]: Got next segment. Inline: true Aug 7 21:24:11.802244236 (11h44m11.391589223s) [upload=12218]: Waiting for error group managing segments... Aug 7 21:24:11.802278040 (11h44m11.391623027s) [upload=12218]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:24:11.802282458 (11h44m11.391627445s) [upload=12218]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:24:11.802284602 (11h44m11.391629579s) [upload=12218]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:25:11.151652598 (11h45m10.740997575s) [upload=12218]: Done uploading segments... Aug 7 21:25:11.151674050 (11h45m10.741019037s) [upload=12218]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:25:17.859162022 (11h45m17.44850703s) [upload=12235]: Starting upload Aug 7 21:25:17.859247908 (11h45m17.448592915s) [upload=12235]: Done starting upload Aug 7 21:25:17.859314314 (11h45m17.448659301s) [upload=12235]: Uploading segments... Aug 7 21:25:17.859360473 (11h45m17.44870546s) [upload=12235]: Got next segment. Inline: true Aug 7 21:25:17.859365562 (11h45m17.44871056s) [upload=12235]: Waiting for error group managing segments... Aug 7 21:25:17.859407051 (11h45m17.448752058s) [upload=12235]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:25:17.859412402 (11h45m17.448757399s) [upload=12235]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:25:17.859415648 (11h45m17.448760646s) [upload=12235]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:26:17.209779892 (11h46m16.799124879s) [upload=12235]: Done uploading segments... Aug 7 21:26:17.209836961 (11h46m16.799181949s) [upload=12235]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:26:18.091653415 (11h46m17.680998403s) [upload=12251]: Starting upload Aug 7 21:26:18.091708220 (11h46m17.681053207s) [upload=12251]: Done starting upload Aug 7 21:26:18.091749139 (11h46m17.681094126s) [upload=12251]: Uploading segments... Aug 7 21:26:18.091784216 (11h46m17.681129203s) [upload=12251]: Got next segment. Inline: true Aug 7 21:26:18.091787672 (11h46m17.681132649s) [upload=12251]: Waiting for error group managing segments... Aug 7 21:26:18.091819754 (11h46m17.681164741s) [upload=12251]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:26:18.091823481 (11h46m17.681168468s) [upload=12251]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:26:18.091826015 (11h46m17.681170992s) [upload=12251]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:27:17.447643584 (11h47m17.036988571s) [upload=12251]: Done uploading segments... Aug 7 21:27:17.447669333 (11h47m17.03701433s) [upload=12251]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:27:18.331191031 (11h47m17.920536018s) [upload=12265]: Starting upload Aug 7 21:27:18.331246827 (11h47m17.920591815s) [upload=12265]: Done starting upload Aug 7 21:27:18.331299789 (11h47m17.920644776s) [upload=12265]: Uploading segments... Aug 7 21:27:18.331346968 (11h47m17.920691945s) [upload=12265]: Got next segment. Inline: true Aug 7 21:27:18.331350675 (11h47m17.920695652s) [upload=12265]: Waiting for error group managing segments... Aug 7 21:27:18.331385723 (11h47m17.92073071s) [upload=12265]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:27:18.331389891 (11h47m17.920734878s) [upload=12265]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:27:18.331392486 (11h47m17.920737473s) [upload=12265]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:28:17.685855344 (11h48m17.275200351s) [upload=12265]: Done uploading segments... Aug 7 21:28:17.685882486 (11h48m17.275227473s) [upload=12265]: Upload finished. err: metaclient: context deadline exceeded Duration: 1m0.00291195s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 18 out of 60 is failed (threshold=3) [..........................................XXXXXXXXXXXXXXXXXX] duration: 30.000590834s FAIL (2025-08-07 21:28:01 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 31.440497986s) [upload=12148]: Uploading segments... Aug 7 21:19:31.851212192 (11h39m31.440557179s) [upload=12148]: Got next segment. Inline: true Aug 7 21:19:31.851215829 (11h39m31.440560816s) [upload=12148]: Waiting for error group managing segments... Aug 7 21:19:31.851259963 (11h39m31.44060495s) [upload=12148]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:19:31.851264352 (11h39m31.440609329s) [upload=12148]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:19:31.851266806 (11h39m31.440611793s) [upload=12148]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:20:01.198301602 (11h40m0.787646589s) [upload=12148]: Done uploading segments... Aug 7 21:20:01.198330468 (11h40m0.787675455s) [upload=12148]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:20:14.918521871 (11h40m14.507866859s) [upload=12161]: Starting upload Aug 7 21:20:14.918603488 (11h40m14.507948495s) [upload=12161]: Done starting upload Aug 7 21:20:14.918661038 (11h40m14.508006025s) [upload=12161]: Uploading segments... Aug 7 21:20:14.918718718 (11h40m14.508063705s) [upload=12161]: Got next segment. Inline: true Aug 7 21:20:14.918725691 (11h40m14.508070688s) [upload=12161]: Waiting for error group managing segments... Aug 7 21:20:14.918778933 (11h40m14.50812394s) [upload=12161]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:20:14.918785126 (11h40m14.508130133s) [upload=12161]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:20:14.918790756 (11h40m14.508135773s) [upload=12161]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:20:44.279825769 (11h40m43.869170756s) [upload=12161]: Done uploading segments... Aug 7 21:20:44.279847039 (11h40m43.869192027s) [upload=12161]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:21:26.967144759 (11h41m26.556489736s) [upload=12182]: Starting upload Aug 7 21:21:26.967204002 (11h41m26.556549009s) [upload=12182]: Done starting upload Aug 7 21:21:26.967248207 (11h41m26.556593194s) [upload=12182]: Uploading segments... Aug 7 21:21:26.967275329 (11h41m26.556620316s) [upload=12182]: Got next segment. Inline: true Aug 7 21:21:26.967278815 (11h41m26.556623802s) [upload=12182]: Waiting for error group managing segments... Aug 7 21:21:26.967311758 (11h41m26.556656745s) [upload=12182]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:21:26.967315786 (11h41m26.556660773s) [upload=12182]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:21:26.967318120 (11h41m26.556663107s) [upload=12182]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:21:56.309770442 (11h41m55.899115429s) [upload=12182]: Done uploading segments... Aug 7 21:21:56.309792865 (11h41m55.899137852s) [upload=12182]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:22:21.925019495 (11h42m21.514364572s) [upload=12195]: Starting upload Aug 7 21:22:21.925098336 (11h42m21.514443323s) [upload=12195]: Done starting upload Aug 7 21:22:21.925142180 (11h42m21.514487167s) [upload=12195]: Uploading segments... Aug 7 21:22:21.925175574 (11h42m21.514520561s) [upload=12195]: Got next segment. Inline: true Aug 7 21:22:21.925179491 (11h42m21.514524478s) [upload=12195]: Waiting for error group managing segments... Aug 7 21:22:21.925210671 (11h42m21.514555648s) [upload=12195]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:22:21.925214048 (11h42m21.514559035s) [upload=12195]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:22:21.925216302 (11h42m21.51456129s) [upload=12195]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:22:51.283998863 (11h42m50.87334385s) [upload=12195]: Done uploading segments... Aug 7 21:22:51.284018922 (11h42m50.873363899s) [upload=12195]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:23:36.916300477 (11h43m36.505645464s) [upload=12212]: Starting upload Aug 7 21:23:36.916350923 (11h43m36.50569591s) [upload=12212]: Done starting upload Aug 7 21:23:36.916395318 (11h43m36.505740295s) [upload=12212]: Uploading segments... Aug 7 21:23:36.916436557 (11h43m36.505781544s) [upload=12212]: Got next segment. Inline: true Aug 7 21:23:36.916439863 (11h43m36.50578484s) [upload=12212]: Waiting for error group managing segments... Aug 7 21:23:36.916471534 (11h43m36.505816511s) [upload=12212]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:23:36.916475351 (11h43m36.505820338s) [upload=12212]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:23:36.916477665 (11h43m36.505822652s) [upload=12212]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:24:06.267030467 (11h44m5.856375454s) [upload=12212]: Done uploading segments... Aug 7 21:24:06.267052209 (11h44m5.856397196s) [upload=12212]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:24:29.996857636 (11h44m29.586202623s) [upload=12224]: Starting upload Aug 7 21:24:29.996913823 (11h44m29.5862588s) [upload=12224]: Done starting upload Aug 7 21:24:29.996961495 (11h44m29.586306482s) [upload=12224]: Uploading segments... Aug 7 21:24:29.996996181 (11h44m29.586341158s) [upload=12224]: Got next segment. Inline: true Aug 7 21:24:29.996999788 (11h44m29.586344765s) [upload=12224]: Waiting for error group managing segments... Aug 7 21:24:29.997033352 (11h44m29.586378329s) [upload=12224]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:24:29.997037590 (11h44m29.586382577s) [upload=12224]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:24:29.997040075 (11h44m29.586385062s) [upload=12224]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:24:59.358108170 (11h44m58.947453157s) [upload=12224]: Done uploading segments... Aug 7 21:24:59.358131014 (11h44m58.947476001s) [upload=12224]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:25:00.224254754 (11h44m59.813599741s) [upload=12230]: Starting upload Aug 7 21:25:00.224330690 (11h44m59.813675677s) [upload=12230]: Done starting upload Aug 7 21:25:00.224406304 (11h44m59.813751311s) [upload=12230]: Uploading segments... Aug 7 21:25:00.224533207 (11h44m59.813878204s) [upload=12230]: Got next segment. Inline: true Aug 7 21:25:00.224541352 (11h44m59.813886339s) [upload=12230]: Waiting for error group managing segments... Aug 7 21:25:00.224626114 (11h44m59.813971121s) [upload=12230]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:25:00.224634080 (11h44m59.813979107s) [upload=12230]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:25:00.224638929 (11h44m59.813983937s) [upload=12230]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:25:29.580431600 (11h45m29.169776597s) [upload=12230]: Done uploading segments... Aug 7 21:25:29.580456288 (11h45m29.169801275s) [upload=12230]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:25:38.048208784 (11h45m37.637553761s) [upload=12241]: Starting upload Aug 7 21:25:38.048265823 (11h45m37.63761081s) [upload=12241]: Done starting upload Aug 7 21:25:38.048311521 (11h45m37.637656508s) [upload=12241]: Uploading segments... Aug 7 21:25:38.048347439 (11h45m37.637692426s) [upload=12241]: Got next segment. Inline: true Aug 7 21:25:38.048350867 (11h45m37.637695854s) [upload=12241]: Waiting for error group managing segments... Aug 7 21:25:38.048383188 (11h45m37.637728186s) [upload=12241]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:25:38.048387396 (11h45m37.637732383s) [upload=12241]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:25:38.048389881 (11h45m37.637734868s) [upload=12241]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:26:07.388744771 (11h46m6.978089758s) [upload=12241]: Done uploading segments... Aug 7 21:26:07.388753889 (11h46m6.978098876s) [upload=12241]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:26:51.633559976 (11h46m51.222904953s) [upload=12259]: Starting upload Aug 7 21:26:51.633607808 (11h46m51.222952795s) [upload=12259]: Done starting upload Aug 7 21:26:51.633657383 (11h46m51.22300237s) [upload=12259]: Uploading segments... Aug 7 21:26:51.633689063 (11h46m51.22303405s) [upload=12259]: Got next segment. Inline: true Aug 7 21:26:51.633692660 (11h46m51.223037647s) [upload=12259]: Waiting for error group managing segments... Aug 7 21:26:51.633727035 (11h46m51.223072012s) [upload=12259]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:26:51.633730782 (11h46m51.22307577s) [upload=12259]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:26:51.633733327 (11h46m51.223078314s) [upload=12259]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:27:20.991495967 (11h47m20.580840944s) [upload=12259]: Done uploading segments... Aug 7 21:27:20.991524641 (11h47m20.580869628s) [upload=12259]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:27:32.297072953 (11h47m31.88641795s) [upload=12271]: Starting upload Aug 7 21:27:32.297139892 (11h47m31.886484889s) [upload=12271]: Done starting upload Aug 7 21:27:32.297212079 (11h47m31.886557066s) [upload=12271]: Uploading segments... Aug 7 21:27:32.297260662 (11h47m31.886605649s) [upload=12271]: Got next segment. Inline: true Aug 7 21:27:32.297264640 (11h47m31.886609627s) [upload=12271]: Waiting for error group managing segments... Aug 7 21:27:32.297341267 (11h47m31.886686254s) [upload=12271]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:27:32.297345635 (11h47m31.886690622s) [upload=12271]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:27:32.297348029 (11h47m31.886693016s) [upload=12271]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:28:01.648543031 (11h48m1.237888018s) [upload=12271]: Done uploading segments... Aug 7 21:28:01.648565273 (11h48m1.23791026s) [upload=12271]: Upload finished. err: metaclient: context deadline exceeded Duration: 30.000606113s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 15 out of 60 is failed (threshold=3) [...........................................X..XXXXXXXXXXXXXX] duration: 30.000428862s FAIL (2025-08-07 21:27:59 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 30.000446636s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [........................................X..XXXXXXXXXXXXXXXXX] duration: 30.000571043s FAIL (2025-08-07 21:27:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== Next segment err: context deadline exceeded Aug 7 21:18:09.894385774 (11h38m9.483730782s) [upload=12118]: Done uploading segments... Aug 7 21:18:09.894407416 (11h38m9.483752433s) [upload=12118]: Upload finished. err: context deadline exceeded Aug 7 21:18:51.953663291 (11h38m51.543008278s) [upload=12135]: Starting upload Aug 7 21:18:51.953718226 (11h38m51.543063213s) [upload=12135]: Done starting upload Aug 7 21:18:51.953825141 (11h38m51.543170128s) [upload=12135]: Uploading segments... Aug 7 21:18:51.953923359 (11h38m51.543268346s) [upload=12135]: Got next segment. Inline: false Aug 7 21:18:51.953958877 (11h38m51.543303864s) [upload=12135]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:18:51.953963656 (11h38m51.543308643s) [upload=12135]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:19:21.302189358 (11h39m20.891534335s) [upload=12135]: Next segment err: context deadline exceeded Aug 7 21:19:21.302254121 (11h39m20.891599108s) [upload=12135]: Done uploading segments... Aug 7 21:19:21.302274270 (11h39m20.891619267s) [upload=12135]: Upload finished. err: context deadline exceeded Aug 7 21:19:39.710866482 (11h39m39.300211459s) [upload=12149]: Starting upload Aug 7 21:19:39.710923851 (11h39m39.300268829s) [upload=12149]: Done starting upload Aug 7 21:19:39.710996290 (11h39m39.300341277s) [upload=12149]: Uploading segments... Aug 7 21:19:39.711114146 (11h39m39.300459133s) [upload=12149]: Got next segment. Inline: false Aug 7 21:19:39.711140085 (11h39m39.300485102s) [upload=12149]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:19:39.711144824 (11h39m39.300489811s) [upload=12149]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:20:09.053604686 (11h40m8.642949683s) [upload=12149]: Next segment err: context deadline exceeded Aug 7 21:20:09.053707913 (11h40m8.6430529s) [upload=12149]: Done uploading segments... Aug 7 21:20:09.053723724 (11h40m8.643068711s) [upload=12149]: Upload finished. err: context deadline exceeded Aug 7 21:20:24.058132378 (11h40m23.647477365s) [upload=12163]: Starting upload Aug 7 21:20:24.058185559 (11h40m23.647530546s) [upload=12163]: Done starting upload Aug 7 21:20:24.058265552 (11h40m23.647610529s) [upload=12163]: Uploading segments... Aug 7 21:20:24.058369542 (11h40m23.647714539s) [upload=12163]: Got next segment. Inline: false Aug 7 21:20:24.058412013 (11h40m23.647757s) [upload=12163]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:20:24.058418646 (11h40m23.647763663s) [upload=12163]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:20:53.409396630 (11h40m52.998741617s) [upload=12163]: Next segment err: context deadline exceeded Aug 7 21:20:53.409563509 (11h40m52.998908496s) [upload=12163]: Done uploading segments... Aug 7 21:20:53.409604537 (11h40m52.998949514s) [upload=12163]: Upload finished. err: context deadline exceeded Aug 7 21:21:23.928736473 (11h41m23.51808146s) [upload=12180]: Starting upload Aug 7 21:21:23.928792831 (11h41m23.518137818s) [upload=12180]: Done starting upload Aug 7 21:21:23.928856973 (11h41m23.51820195s) [upload=12180]: Uploading segments... Aug 7 21:21:23.929051695 (11h41m23.518396682s) [upload=12180]: Got next segment. Inline: false Aug 7 21:21:23.929103304 (11h41m23.518448291s) [upload=12180]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:21:23.929107933 (11h41m23.51845292s) [upload=12180]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:21:53.279642106 (11h41m52.868987093s) [upload=12180]: Next segment err: context deadline exceeded Aug 7 21:21:53.279858500 (11h41m52.869203477s) [upload=12180]: Done uploading segments... Aug 7 21:21:53.279873498 (11h41m52.869218485s) [upload=12180]: Upload finished. err: context deadline exceeded Aug 7 21:21:54.191662378 (11h41m53.781007366s) [upload=12188]: Starting upload Aug 7 21:21:54.191712274 (11h41m53.781057251s) [upload=12188]: Done starting upload Aug 7 21:21:54.191790404 (11h41m53.781135391s) [upload=12188]: Uploading segments... Aug 7 21:21:54.192007459 (11h41m53.781352446s) [upload=12188]: Got next segment. Inline: false Aug 7 21:21:54.192034671 (11h41m53.781379658s) [upload=12188]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:21:54.192038719 (11h41m53.781383706s) [upload=12188]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:22:23.533395958 (11h42m23.122740945s) [upload=12188]: Next segment err: context deadline exceeded Aug 7 21:22:23.533474659 (11h42m23.122819646s) [upload=12188]: Done uploading segments... Aug 7 21:22:23.533496701 (11h42m23.122841688s) [upload=12188]: Upload finished. err: context deadline exceeded Aug 7 21:23:08.442262622 (11h43m8.031607609s) [upload=12203]: Starting upload Aug 7 21:23:08.442325042 (11h43m8.031670019s) [upload=12203]: Done starting upload Aug 7 21:23:08.442385668 (11h43m8.031730655s) [upload=12203]: Uploading segments... Aug 7 21:23:08.442501099 (11h43m8.031846076s) [upload=12203]: Got next segment. Inline: false Aug 7 21:23:08.442542839 (11h43m8.031887826s) [upload=12203]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:23:08.442546706 (11h43m8.031891683s) [upload=12203]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:23:37.798210470 (11h43m37.387555457s) [upload=12203]: Next segment err: context deadline exceeded Aug 7 21:23:37.798367780 (11h43m37.387712767s) [upload=12203]: Done uploading segments... Aug 7 21:23:37.798381718 (11h43m37.387726705s) [upload=12203]: Upload finished. err: context deadline exceeded Aug 7 21:23:54.967168247 (11h43m54.556513234s) [upload=12215]: Starting upload Aug 7 21:23:54.967268208 (11h43m54.556613185s) [upload=12215]: Done starting upload Aug 7 21:23:54.967374321 (11h43m54.556719308s) [upload=12215]: Uploading segments... Aug 7 21:23:54.967600634 (11h43m54.556945621s) [upload=12215]: Got next segment. Inline: false Aug 7 21:23:54.967649888 (11h43m54.556994875s) [upload=12215]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:23:54.967657021 (11h43m54.557002029s) [upload=12215]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:24:24.312833601 (11h44m23.902178598s) [upload=12215]: Next segment err: context deadline exceeded Aug 7 21:24:24.313044925 (11h44m23.902389912s) [upload=12215]: Done uploading segments... Aug 7 21:24:24.313087757 (11h44m23.902432764s) [upload=12215]: Upload finished. err: context deadline exceeded Aug 7 21:24:25.196484464 (11h44m24.785829451s) [upload=12221]: Starting upload Aug 7 21:24:25.196569166 (11h44m24.785914153s) [upload=12221]: Done starting upload Aug 7 21:24:25.196697702 (11h44m24.786042719s) [upload=12221]: Uploading segments... Aug 7 21:24:25.196786912 (11h44m24.786131899s) [upload=12221]: Got next segment. Inline: false Aug 7 21:24:25.196818883 (11h44m24.78616387s) [upload=12221]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:24:25.196823522 (11h44m24.786168509s) [upload=12221]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:24:54.542928213 (11h44m54.1322732s) [upload=12221]: Next segment err: context deadline exceeded Aug 7 21:24:54.543143736 (11h44m54.132488713s) [upload=12221]: Done uploading segments... Aug 7 21:24:54.543169415 (11h44m54.132514402s) [upload=12221]: Upload finished. err: context deadline exceeded Aug 7 21:25:30.578713383 (11h45m30.16805837s) [upload=12239]: Starting upload Aug 7 21:25:30.578779109 (11h45m30.168124096s) [upload=12239]: Done starting upload Aug 7 21:25:30.578836729 (11h45m30.168181716s) [upload=12239]: Uploading segments... Aug 7 21:25:30.579017655 (11h45m30.168362642s) [upload=12239]: Got next segment. Inline: false Aug 7 21:25:30.579083341 (11h45m30.168428328s) [upload=12239]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:25:30.579091086 (11h45m30.168436073s) [upload=12239]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:25:59.907330605 (11h45m59.496675592s) [upload=12239]: Next segment err: context deadline exceeded Aug 7 21:25:59.907404857 (11h45m59.496749845s) [upload=12239]: Done uploading segments... Aug 7 21:25:59.907422762 (11h45m59.496767749s) [upload=12239]: Upload finished. err: context deadline exceeded Aug 7 21:26:16.857419024 (11h46m16.446764011s) [upload=12250]: Starting upload Aug 7 21:26:16.857477205 (11h46m16.446822182s) [upload=12250]: Done starting upload Aug 7 21:26:16.857572879 (11h46m16.446917856s) [upload=12250]: Uploading segments... Aug 7 21:26:16.857644405 (11h46m16.446989392s) [upload=12250]: Got next segment. Inline: false Aug 7 21:26:16.857669784 (11h46m16.447014771s) [upload=12250]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:26:16.857673381 (11h46m16.447018368s) [upload=12250]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:26:46.211380241 (11h46m45.800725218s) [upload=12250]: Next segment err: context deadline exceeded Aug 7 21:26:46.211514648 (11h46m45.800859625s) [upload=12250]: Done uploading segments... Aug 7 21:26:46.211531050 (11h46m45.800876037s) [upload=12250]: Upload finished. err: context deadline exceeded Aug 7 21:27:19.636960898 (11h47m19.226305885s) [upload=12266]: Starting upload Aug 7 21:27:19.637068003 (11h47m19.22641299s) [upload=12266]: Done starting upload Aug 7 21:27:19.637205767 (11h47m19.226550744s) [upload=12266]: Uploading segments... Aug 7 21:27:19.637342969 (11h47m19.226687956s) [upload=12266]: Got next segment. Inline: false Aug 7 21:27:19.637391532 (11h47m19.22673655s) [upload=12266]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:27:19.637399387 (11h47m19.226744374s) [upload=12266]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:27:48.990525160 (11h47m48.579870148s) [upload=12266]: Next segment err: context deadline exceeded Aug 7 21:27:48.990618549 (11h47m48.579963526s) [upload=12266]: Done uploading segments... Aug 7 21:27:48.990636003 (11h47m48.57998099s) [upload=12266]: Upload finished. err: context deadline exceeded Duration: 30.000595602s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [...............X...................................X........] duration: 1.863188315s FAIL (2025-08-07 21:27:48 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/1K") ======LOGS====== ]: Done uploading segments... Aug 7 21:13:55.370929874 (11h33m54.960274861s) [upload=12060]: Upload finished. err: <nil> Aug 7 21:14:45.614661829 (11h34m45.204006816s) [upload=12073]: Starting upload Aug 7 21:14:45.614700943 (11h34m45.20404593s) [upload=12073]: Done starting upload Aug 7 21:14:45.614745929 (11h34m45.204090917s) [upload=12073]: Uploading segments... Aug 7 21:14:45.614774204 (11h34m45.204119181s) [upload=12073]: Got next segment. Inline: true Aug 7 21:14:45.614777830 (11h34m45.204122817s) [upload=12073]: Waiting for error group managing segments... Aug 7 21:14:45.614809601 (11h34m45.204154578s) [upload=12073]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:14:45.614813057 (11h34m45.204158045s) [upload=12073]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:14:45.614815242 (11h34m45.204160219s) [upload=12073]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:14:45.911976434 (11h34m45.501321421s) [upload=12073]: Done uploading segments... Aug 7 21:14:45.911995080 (11h34m45.501340067s) [upload=12073]: Upload finished. err: <nil> Aug 7 21:15:14.898649385 (11h35m14.487994372s) [upload=12079]: Starting upload Aug 7 21:15:14.898694180 (11h35m14.488039167s) [upload=12079]: Done starting upload Aug 7 21:15:14.898741050 (11h35m14.488086027s) [upload=12079]: Uploading segments... Aug 7 21:15:14.898781948 (11h35m14.488126965s) [upload=12079]: Got next segment. Inline: true Aug 7 21:15:14.898789092 (11h35m14.488134079s) [upload=12079]: Waiting for error group managing segments... Aug 7 21:15:14.898847644 (11h35m14.488192641s) [upload=12079]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:15:14.898856401 (11h35m14.488201408s) [upload=12079]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:15:14.898860989 (11h35m14.488206026s) [upload=12079]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:15:15.228046082 (11h35m14.817391079s) [upload=12079]: Done uploading segments... Aug 7 21:15:15.228066881 (11h35m14.817411878s) [upload=12079]: Upload finished. err: <nil> Aug 7 21:16:25.941982337 (11h36m25.531327324s) [upload=12099]: Starting upload Aug 7 21:16:25.942029607 (11h36m25.531374594s) [upload=12099]: Done starting upload Aug 7 21:16:25.942075234 (11h36m25.531420221s) [upload=12099]: Uploading segments... Aug 7 21:16:25.942105222 (11h36m25.531450199s) [upload=12099]: Got next segment. Inline: true Aug 7 21:16:25.942108588 (11h36m25.531453575s) [upload=12099]: Waiting for error group managing segments... Aug 7 21:16:25.942142683 (11h36m25.53148767s) [upload=12099]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:16:25.942146340 (11h36m25.531491317s) [upload=12099]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:16:25.942148595 (11h36m25.531493582s) [upload=12099]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:16:26.215688283 (11h36m25.80503327s) [upload=12099]: Done uploading segments... Aug 7 21:16:26.215705086 (11h36m25.805050073s) [upload=12099]: Upload finished. err: <nil> Aug 7 21:17:38.285379669 (11h37m37.874724656s) [upload=12117]: Starting upload Aug 7 21:17:38.285441276 (11h37m37.874786323s) [upload=12117]: Done starting upload Aug 7 21:17:38.285507584 (11h37m37.874852611s) [upload=12117]: Uploading segments... Aug 7 21:17:38.285571005 (11h37m37.874915992s) [upload=12117]: Got next segment. Inline: true Aug 7 21:17:38.285575654 (11h37m37.874920641s) [upload=12117]: Waiting for error group managing segments... Aug 7 21:17:38.285618215 (11h37m37.874963243s) [upload=12117]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:17:38.285624958 (11h37m37.874969945s) [upload=12117]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:17:38.285629477 (11h37m37.874974474s) [upload=12117]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:17:38.561705487 (11h37m38.151050484s) [upload=12117]: Done uploading segments... Aug 7 21:17:38.561727969 (11h37m38.151072957s) [upload=12117]: Upload finished. err: <nil> Aug 7 21:18:46.880586061 (11h38m46.469931048s) [upload=12133]: Starting upload Aug 7 21:18:46.880627500 (11h38m46.469972487s) [upload=12133]: Done starting upload Aug 7 21:18:46.880681523 (11h38m46.47002651s) [upload=12133]: Uploading segments... Aug 7 21:18:46.880714476 (11h38m46.470059463s) [upload=12133]: Got next segment. Inline: true Aug 7 21:18:46.880718043 (11h38m46.47006302s) [upload=12133]: Waiting for error group managing segments... Aug 7 21:18:46.880752089 (11h38m46.470097076s) [upload=12133]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:18:46.880755715 (11h38m46.470100702s) [upload=12133]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:18:46.880758120 (11h38m46.470103107s) [upload=12133]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:18:47.149461450 (11h38m46.738806437s) [upload=12133]: Done uploading segments... Aug 7 21:18:47.149478763 (11h38m46.73882375s) [upload=12133]: Upload finished. err: <nil> Aug 7 21:19:44.345616276 (11h39m43.934961263s) [upload=12152]: Starting upload Aug 7 21:19:44.345660120 (11h39m43.935005098s) [upload=12152]: Done starting upload Aug 7 21:19:44.345731898 (11h39m43.935076885s) [upload=12152]: Uploading segments... Aug 7 21:19:44.345839653 (11h39m43.93518463s) [upload=12152]: Got next segment. Inline: true Aug 7 21:19:44.345844203 (11h39m43.93518919s) [upload=12152]: Waiting for error group managing segments... Aug 7 21:19:44.345886133 (11h39m43.93523111s) [upload=12152]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:19:44.345890932 (11h39m43.935235919s) [upload=12152]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:19:44.345893316 (11h39m43.935238303s) [upload=12152]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:19:44.666041542 (11h39m44.255386529s) [upload=12152]: Done uploading segments... Aug 7 21:19:44.666059567 (11h39m44.255404554s) [upload=12152]: Upload finished. err: <nil> Aug 7 21:20:10.176957983 (11h40m9.76630296s) [upload=12157]: Starting upload Aug 7 21:20:10.177009311 (11h40m9.766354298s) [upload=12157]: Done starting upload Aug 7 21:20:10.177070559 (11h40m9.766415546s) [upload=12157]: Uploading segments... Aug 7 21:20:10.177121496 (11h40m9.766466483s) [upload=12157]: Got next segment. Inline: true Aug 7 21:20:10.177125895 (11h40m9.766470882s) [upload=12157]: Waiting for error group managing segments... Aug 7 21:20:10.177160220 (11h40m9.766505207s) [upload=12157]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:20:10.177164087 (11h40m9.766509074s) [upload=12157]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:20:10.177166371 (11h40m9.766511359s) [upload=12157]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:20:10.463794245 (11h40m10.053139232s) [upload=12157]: Done uploading segments... Aug 7 21:20:10.463812280 (11h40m10.053157257s) [upload=12157]: Upload finished. err: <nil> Aug 7 21:20:47.295095407 (11h40m46.884440394s) [upload=12169]: Starting upload Aug 7 21:20:47.295162806 (11h40m46.884507793s) [upload=12169]: Done starting upload Aug 7 21:20:47.295214385 (11h40m46.884559372s) [upload=12169]: Uploading segments... Aug 7 21:20:47.295251876 (11h40m46.884596864s) [upload=12169]: Got next segment. Inline: true Aug 7 21:20:47.295255583 (11h40m46.88460057s) [upload=12169]: Waiting for error group managing segments... Aug 7 21:20:47.295291642 (11h40m46.884636629s) [upload=12169]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:20:47.295295450 (11h40m46.884640437s) [upload=12169]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:20:47.295297755 (11h40m46.884642742s) [upload=12169]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:20:47.581180532 (11h40m47.170525529s) [upload=12169]: Done uploading segments... Aug 7 21:20:47.581201993 (11h40m47.17054698s) [upload=12169]: Upload finished. err: <nil> Aug 7 21:21:39.503326248 (11h41m39.092671235s) [upload=12184]: Starting upload Aug 7 21:21:39.503368609 (11h41m39.092713586s) [upload=12184]: Done starting upload Aug 7 21:21:39.503416451 (11h41m39.092761438s) [upload=12184]: Uploading segments... Aug 7 21:21:39.503447480 (11h41m39.092792467s) [upload=12184]: Got next segment. Inline: true Aug 7 21:21:39.503451247 (11h41m39.092796234s) [upload=12184]: Waiting for error group managing segments... Aug 7 21:21:39.503485393 (11h41m39.09283038s) [upload=12184]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:21:39.503489370 (11h41m39.092834357s) [upload=12184]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:21:39.503491825 (11h41m39.092836802s) [upload=12184]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:21:39.826158170 (11h41m39.415503147s) [upload=12184]: Done uploading segments... Aug 7 21:21:39.826174040 (11h41m39.415519027s) [upload=12184]: Upload finished. err: <nil> Aug 7 21:22:46.427091203 (11h42m46.01643619s) [upload=12199]: Starting upload Aug 7 21:22:46.427135929 (11h42m46.016480916s) [upload=12199]: Done starting upload Aug 7 21:22:46.427188519 (11h42m46.016533496s) [upload=12199]: Uploading segments... Aug 7 21:22:46.427215882 (11h42m46.016560869s) [upload=12199]: Got next segment. Inline: true Aug 7 21:22:46.427219829 (11h42m46.016564806s) [upload=12199]: Waiting for error group managing segments... Aug 7 21:22:46.427252391 (11h42m46.016597378s) [upload=12199]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:22:46.427256048 (11h42m46.016601025s) [upload=12199]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:22:46.427268202 (11h42m46.016613189s) [upload=12199]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:22:46.705229475 (11h42m46.294574462s) [upload=12199]: Done uploading segments... Aug 7 21:22:46.705247530 (11h42m46.294592507s) [upload=12199]: Upload finished. err: <nil> Duration: 1.863235736s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 18 out of 60 is failed (threshold=3) [..........................................XXXXXXXXXXXXXXXXXX] duration: 30.000891266s FAIL (2025-08-07 21:27:48 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.000932757s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 14 out of 60 is failed (threshold=3) [..............................................XXXXXXXXXXXXXX] duration: 1m0.000526187s FAIL (2025-08-07 21:27:44 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.000537378s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 6 out of 60 is failed (threshold=3) [.....XX...........X.......................X.........X......X] duration: 8.377813992s FAIL (2025-08-07 21:27:32 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/4M):: after 0c7bb902d1fc786d53d64877c5bf041e2c416fa3a23853fbffdc1a20ffb50890 != before bf5169665124c7207457a5823fc93758035c94fbdbbd7bf4beebc1b976aa7211 ======LOGS====== Aug 7 21:27:25.141891015 (11h47m24.731236002s) [upload=12268]: Got next segment. Inline: false Aug 7 21:27:25.141916354 (11h47m24.731261341s) [upload=12268]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:27:25.141920291 (11h47m24.731265278s) [upload=12268]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:27:25.654570615 (11h47m25.243915602s) [seg_pos={0 0}, upload=12268]: Begin upload segment... Aug 7 21:27:25.654611634 (11h47m25.243956621s) [seg_pos={0 0}, upload=12268]: Done begin upload segment. Aug 7 21:27:25.654634086 (11h47m25.243979073s) [noise=true , address=104.36.82.20:4448 , piece_id=X2KIKPSWI3SCJEEYILD4BVVHRIZZFDH4KGE5OHDIXEIM2HVBZ5NA, seg_pos={0 0}, upload=12268]: Uploading piece... Aug 7 21:27:25.654721364 (11h47m25.244066351s) [noise=true , address=104.36.82.22:4527 , piece_id=YCZOVT6QJT3VE2KFBC4DU6M34Y6DUBKTH3YTHDTO7ZQ7UFIKRAFQ, seg_pos={0 0}, upload=12268]: Uploading piece... Aug 7 21:27:25.655016387 (11h47m25.244361365s) [noise=true , address=104.36.82.46:4512 , piece_id=AWY2BPUJPBECXNJXDXYHMQWOZB3DGLTGP5776L677C64XIVW6ZCQ, seg_pos={0 0}, upload=12268]: Uploading piece... Aug 7 21:27:25.655133612 (11h47m25.2444786s) [noise=true , address=104.36.82.59:4329 , piece_id=CB2FO7D6IW7C2KSBJPR4J7XZQAM6NTHLMAW7XOB6JH6SHRGLQV7A, seg_pos={0 0}, upload=12268]: Uploading piece... Aug 7 21:27:25.655336751 (11h47m25.244681728s) [noise=true , address=104.36.82.55:4211 , piece_id=7IRAZ4EU7BMJNOGPAU7G6VIZ4TSTNCZK4PVFNLFPQM6E2234ME3Q, seg_pos={0 0}, upload=12268]: Uploading piece... Aug 7 21:27:25.655423657 (11h47m25.244768634s) [noise=true , address=104.36.82.37:4679 , piece_id=DOECPGUUPVOBOWQRAX5GY6BD5LA2VNYMLI34VEWZRC7KLT3AA2XA, seg_pos={0 0}, upload=12268]: Uploading piece... Aug 7 21:27:25.655662334 (11h47m25.245007311s) [noise=true , address=104.36.82.58:4279 , piece_id=OPZ353KP2L23TZGNZBO52DLTVSX66WL4D7GYM7TMTJOAT2RPOGHQ, seg_pos={0 0}, upload=12268]: Uploading piece... Aug 7 21:27:25.655733840 (11h47m25.245078827s) [noise=true , address=104.36.82.9:4019 , piece_id=ZDF7MVKTAV4AQD5XGAGHBDDB4TQQMBJGXWPKOBD2ONQI4BVLBFGQ, seg_pos={0 0}, upload=12268]: Uploading piece... Aug 7 21:27:25.656037241 (11h47m25.245382228s) [noise=true , address=104.36.82.2:4758 , piece_id=WFBYODP72YMTCPMKQQKVLNCWKMAETBHH6WOQ5UCEXYCBZSY5WWRA, seg_pos={0 0}, upload=12268]: Uploading piece... Aug 7 21:27:25.656051578 (11h47m25.245396566s) [noise=true , address=104.36.82.54:4157 , piece_id=PGULUJFV37PY3FOW2OD3PKK2UGB7RWDNSWPYZJKVQRWGONRSKEXQ, seg_pos={0 0}, upload=12268]: Uploading piece... Aug 7 21:27:25.656335652 (11h47m25.245680639s) [noise=true , address=104.36.82.36:4633 , piece_id=NXYS3XJMQZAJNSPGBFLXTVR4ZDB5RYHB5K4TUTJ4DANWVSQYEAIQ, seg_pos={0 0}, upload=12268]: Uploading piece... Aug 7 21:27:25.656354137 (11h47m25.245699124s) [noise=true , address=45.91.116.8:4054 , piece_id=PHJGQGBYXAZCULVJX5KSSLMUPOGMD6S7I67QVJBQE35ZFT5U6HFA, seg_pos={0 0}, upload=12268]: Uploading piece... Aug 7 21:27:25.656550413 (11h47m25.24589539s) [noise=true , address=104.36.82.60:4345 , piece_id=L5CI5AMDNOXC2TNERZEBAC2SW3PU4NXI5ILTSTAPA2TSHNEQNHHQ, seg_pos={0 0}, upload=12268]: Uploading piece... Aug 7 21:27:25.656587022 (11h47m25.24593201s) [noise=true , address=79.127.205.234:4586 , piece_id=GI36ODJVM2GYWC2WHFASOQUKJMSJ3R4JANFQ5B45OAFNTBGUEBMA, seg_pos={0 0}, upload=12268]: Uploading piece... Aug 7 21:27:25.656785703 (11h47m25.24613069s) [noise=true , address=104.36.82.43:4837 , piece_id=AQ2NEBR5MYDUQYSKIYKLQJOGRIHKGD3F5QM36GXTECXX37R6H5SQ, seg_pos={0 0}, upload=12268]: Uploading piece... Aug 7 21:27:25.656804429 (11h47m25.246149406s) [noise=true , address=104.36.82.20:4439 , piece_id=CJ7R576SA66NFSO7OHECHKCHULKZN7EZDUTQ3YGRWE436DKO4MWA, seg_pos={0 0}, upload=12268]: Uploading piece... Aug 7 21:27:25.657025892 (11h47m25.246370879s) [noise=true , address=79.127.205.233:4632 , piece_id=T4VQ3H6ZHBIXZN7X47ZWK4WYQBD5KTGSJWKH6R3PM7UCFEGHKOFQ, seg_pos={0 0}, upload=12268]: Uploading piece... Aug 7 21:27:25.657252346 (11h47m25.246597333s) [noise=true , address=45.91.116.14:4218 , piece_id=X7RNM4ZIW35EBZJNTVT5NYPQKS7K4JNT2RQW2X5D7JVGQQZKHV3Q, seg_pos={0 0}, upload=12268]: Uploading piece... Aug 7 21:27:26.765000728 (11h47m26.354345715s) [upload=12268]: Next returned nil segment Aug 7 21:27:26.765014103 (11h47m26.35435909s) [upload=12268]: Waiting for error group managing segments... Aug 7 21:27:27.057992337 (11h47m26.647337324s) [noise=true , address=79.127.205.234:4586 , piece_id=GI36ODJVM2GYWC2WHFASOQUKJMSJ3R4JANFQ5B45OAFNTBGUEBMA, seg_pos={0 0}, upload=12268]: Done uploading piece. err: <nil> Aug 7 21:27:27.086935163 (11h47m26.67628014s) [noise=true , address=79.127.205.233:4632 , piece_id=T4VQ3H6ZHBIXZN7X47ZWK4WYQBD5KTGSJWKH6R3PM7UCFEGHKOFQ, seg_pos={0 0}, upload=12268]: Done uploading piece. err: <nil> Aug 7 21:27:27.601292823 (11h47m27.1906378s) [noise=true , address=104.36.82.2:4758 , piece_id=WFBYODP72YMTCPMKQQKVLNCWKMAETBHH6WOQ5UCEXYCBZSY5WWRA, seg_pos={0 0}, upload=12268]: Done uploading piece. err: <nil> Aug 7 21:27:27.699555195 (11h47m27.288900192s) [noise=true , address=104.36.82.9:4019 , piece_id=ZDF7MVKTAV4AQD5XGAGHBDDB4TQQMBJGXWPKOBD2ONQI4BVLBFGQ, seg_pos={0 0}, upload=12268]: Done uploading piece. err: <nil> Aug 7 21:27:27.710943893 (11h47m27.30028888s) [noise=true , address=104.36.82.43:4837 , piece_id=AQ2NEBR5MYDUQYSKIYKLQJOGRIHKGD3F5QM36GXTECXX37R6H5SQ, seg_pos={0 0}, upload=12268]: Done uploading piece. err: <nil> Aug 7 21:27:27.719413398 (11h47m27.308758385s) [noise=true , address=104.36.82.46:4512 , piece_id=AWY2BPUJPBECXNJXDXYHMQWOZB3DGLTGP5776L677C64XIVW6ZCQ, seg_pos={0 0}, upload=12268]: Done uploading piece. err: <nil> Aug 7 21:27:27.751471103 (11h47m27.34081608s) [noise=true , address=104.36.82.58:4279 , piece_id=OPZ353KP2L23TZGNZBO52DLTVSX66WL4D7GYM7TMTJOAT2RPOGHQ, seg_pos={0 0}, upload=12268]: Done uploading piece. err: <nil> Aug 7 21:27:27.752665698 (11h47m27.342010685s) [noise=true , address=104.36.82.36:4633 , piece_id=NXYS3XJMQZAJNSPGBFLXTVR4ZDB5RYHB5K4TUTJ4DANWVSQYEAIQ, seg_pos={0 0}, upload=12268]: Done uploading piece. err: <nil> Aug 7 21:27:27.803219381 (11h47m27.392564368s) [noise=true , address=104.36.82.55:4211 , piece_id=7IRAZ4EU7BMJNOGPAU7G6VIZ4TSTNCZK4PVFNLFPQM6E2234ME3Q, seg_pos={0 0}, upload=12268]: Done uploading piece. err: <nil> Aug 7 21:27:27.855671476 (11h47m27.445016463s) [noise=true , address=104.36.82.22:4527 , piece_id=YCZOVT6QJT3VE2KFBC4DU6M34Y6DUBKTH3YTHDTO7ZQ7UFIKRAFQ, seg_pos={0 0}, upload=12268]: Done uploading piece. err: <nil> Aug 7 21:27:27.879451272 (11h47m27.468796259s) [noise=true , address=104.36.82.20:4439 , piece_id=CJ7R576SA66NFSO7OHECHKCHULKZN7EZDUTQ3YGRWE436DKO4MWA, seg_pos={0 0}, upload=12268]: Done uploading piece. err: <nil> Aug 7 21:27:27.941304094 (11h47m27.530649081s) [noise=true , address=45.91.116.8:4054 , piece_id=PHJGQGBYXAZCULVJX5KSSLMUPOGMD6S7I67QVJBQE35ZFT5U6HFA, seg_pos={0 0}, upload=12268]: Done uploading piece. err: <nil> Aug 7 21:27:28.313392665 (11h47m27.902737652s) [noise=true , address=45.91.116.14:4218 , piece_id=X7RNM4ZIW35EBZJNTVT5NYPQKS7K4JNT2RQW2X5D7JVGQQZKHV3Q, seg_pos={0 0}, upload=12268]: Done uploading piece. err: <nil> Aug 7 21:27:28.385132253 (11h47m27.97447724s) [noise=true , address=104.36.82.59:4329 , piece_id=CB2FO7D6IW7C2KSBJPR4J7XZQAM6NTHLMAW7XOB6JH6SHRGLQV7A, seg_pos={0 0}, upload=12268]: Done uploading piece. err: <nil> Aug 7 21:27:28.385181175 (11h47m27.974526153s) [seg_pos={0 0}, upload=12268]: Segment reached optimal threshold of 14 pieces. Aug 7 21:27:28.385286848 (11h47m27.974631835s) [noise=true , address=104.36.82.54:4157 , piece_id=PGULUJFV37PY3FOW2OD3PKK2UGB7RWDNSWPYZJKVQRWGONRSKEXQ, seg_pos={0 0}, upload=12268]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZrzADk2mKG4R9tjxUKjQeJvb4WmfjLksGFyvstHRfYbBC7XNz): protocol: expected piece hash; context canceled; EOF Aug 7 21:27:28.385451353 (11h47m27.97479633s) [noise=true , address=104.36.82.20:4448 , piece_id=X2KIKPSWI3SCJEEYILD4BVVHRIZZFDH4KGE5OHDIXEIM2HVBZ5NA, seg_pos={0 0}, upload=12268]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12vbYQVrjLwiHTMGsAGAcjze8KeVQUqemqGiVTWF63Xq6F7ABUp): protocol: read tcp 188.166.225.170:56322->104.36.82.20:4448: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 21:27:28.385487432 (11h47m27.974832419s) [noise=true , address=104.36.82.37:4679 , piece_id=DOECPGUUPVOBOWQRAX5GY6BD5LA2VNYMLI34VEWZRC7KLT3AA2XA, seg_pos={0 0}, upload=12268]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VHQihNGcx6bTvwitH9PCSqyuKWuchdYhYzZXwSuoruQFByGoj): protocol: rpc: dial tcp 104.36.82.37:4679: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 21:27:28.385527619 (11h47m27.974872606s) [noise=true , address=104.36.82.60:4345 , piece_id=L5CI5AMDNOXC2TNERZEBAC2SW3PU4NXI5ILTSTAPA2TSHNEQNHHQ, seg_pos={0 0}, upload=12268]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KSTyRJ62wcMHx8K7uwqp5e8L8yhxH7V3YtBHnYu6QJ3K3bXpQ): protocol: read tcp 188.166.225.170:45768->104.36.82.60:4345: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 21:27:28.385544672 (11h47m27.974889659s) [seg_pos={0 0}, upload=12268]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 21:27:28.385597523 (11h47m27.97494251s) [upload=12268]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 21:27:28.385601019 (11h47m27.974946007s) [upload=12268]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:27:28.699165600 (11h47m28.288510587s) [upload=12268]: Done uploading segments... Aug 7 21:27:28.699200797 (11h47m28.288545784s) [upload=12268]: Upload finished. err: <nil> Duration: 8.377839331s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 13 out of 60 is failed (threshold=3) [...............................................XXXXXXXXXXXXX] duration: 1m0.001025999s FAIL (2025-08-07 21:27:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== pload=12072]: Next segment err: context deadline exceeded Aug 7 21:15:44.909693113 (11h35m44.49903809s) [upload=12072]: Done uploading segments... Aug 7 21:15:44.909705847 (11h35m44.499050834s) [upload=12072]: Upload finished. err: context deadline exceeded Aug 7 21:15:45.780163446 (11h35m45.369508433s) [upload=12088]: Starting upload Aug 7 21:15:45.780230324 (11h35m45.369575311s) [upload=12088]: Done starting upload Aug 7 21:15:45.780343460 (11h35m45.369688497s) [upload=12088]: Uploading segments... Aug 7 21:15:45.780437410 (11h35m45.369782397s) [upload=12088]: Got next segment. Inline: false Aug 7 21:15:45.780481765 (11h35m45.369826772s) [upload=12088]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:15:45.780488508 (11h35m45.369833535s) [upload=12088]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:16:45.135036416 (11h36m44.724381403s) [upload=12088]: Next segment err: context deadline exceeded Aug 7 21:16:45.135160053 (11h36m44.72450504s) [upload=12088]: Done uploading segments... Aug 7 21:16:45.135188378 (11h36m44.724533375s) [upload=12088]: Upload finished. err: context deadline exceeded Aug 7 21:16:46.020697803 (11h36m45.61004279s) [upload=12106]: Starting upload Aug 7 21:16:46.020754431 (11h36m45.610099448s) [upload=12106]: Done starting upload Aug 7 21:16:46.020843461 (11h36m45.610188448s) [upload=12106]: Uploading segments... Aug 7 21:16:46.020994882 (11h36m45.610339869s) [upload=12106]: Got next segment. Inline: false Aug 7 21:16:46.021030459 (11h36m45.610375446s) [upload=12106]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:16:46.021035288 (11h36m45.610380276s) [upload=12106]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:17:45.371432070 (11h37m44.960777057s) [upload=12106]: Next segment err: context deadline exceeded Aug 7 21:17:45.371471144 (11h37m44.960816142s) [upload=12106]: Done uploading segments... Aug 7 21:17:45.371492135 (11h37m44.960837122s) [upload=12106]: Upload finished. err: context deadline exceeded Aug 7 21:17:49.811942678 (11h37m49.401287665s) [upload=12122]: Starting upload Aug 7 21:17:49.812064030 (11h37m49.401409007s) [upload=12122]: Done starting upload Aug 7 21:17:49.812137160 (11h37m49.401482147s) [upload=12122]: Uploading segments... Aug 7 21:17:49.812396918 (11h37m49.401741895s) [upload=12122]: Got next segment. Inline: false Aug 7 21:17:49.812426735 (11h37m49.401771722s) [upload=12122]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:17:49.812430862 (11h37m49.401775849s) [upload=12122]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:18:49.169738786 (11h38m48.759083813s) [upload=12122]: Next segment err: context deadline exceeded Aug 7 21:18:49.169939078 (11h38m48.759284086s) [upload=12122]: Done uploading segments... Aug 7 21:18:49.169962594 (11h38m48.759307591s) [upload=12122]: Upload finished. err: context deadline exceeded Aug 7 21:18:50.060561583 (11h38m49.64990661s) [upload=12134]: Starting upload Aug 7 21:18:50.060675751 (11h38m49.650020808s) [upload=12134]: Done starting upload Aug 7 21:18:50.060772135 (11h38m49.650117122s) [upload=12134]: Uploading segments... Aug 7 21:18:50.061042412 (11h38m49.650387409s) [upload=12134]: Got next segment. Inline: false Aug 7 21:18:50.061084393 (11h38m49.65042944s) [upload=12134]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:18:50.061091987 (11h38m49.650437004s) [upload=12134]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:19:49.417978828 (11h39m49.007323815s) [upload=12134]: Next segment err: context deadline exceeded Aug 7 21:19:49.418015799 (11h39m49.007360786s) [upload=12134]: Done uploading segments... Aug 7 21:19:49.418037982 (11h39m49.007382959s) [upload=12134]: Upload finished. err: context deadline exceeded Aug 7 21:20:02.821763948 (11h40m2.411108945s) [upload=12155]: Starting upload Aug 7 21:20:02.821848960 (11h40m2.411193957s) [upload=12155]: Done starting upload Aug 7 21:20:02.821939103 (11h40m2.41128409s) [upload=12155]: Uploading segments... Aug 7 21:20:02.822080344 (11h40m2.411425331s) [upload=12155]: Got next segment. Inline: false Aug 7 21:20:02.822124998 (11h40m2.411469985s) [upload=12155]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:20:02.822132784 (11h40m2.411477771s) [upload=12155]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:21:02.173704394 (11h41m1.763049381s) [upload=12155]: Next segment err: context deadline exceeded Aug 7 21:21:02.173855945 (11h41m1.763200932s) [upload=12155]: Done uploading segments... Aug 7 21:21:02.173881643 (11h41m1.76322663s) [upload=12155]: Upload finished. err: context deadline exceeded Aug 7 21:21:03.041754970 (11h41m2.631099958s) [upload=12171]: Starting upload Aug 7 21:21:03.041871343 (11h41m2.63121633s) [upload=12171]: Done starting upload Aug 7 21:21:03.041935847 (11h41m2.631280834s) [upload=12171]: Uploading segments... Aug 7 21:21:03.042038162 (11h41m2.631383149s) [upload=12171]: Got next segment. Inline: false Aug 7 21:21:03.042070074 (11h41m2.631415061s) [upload=12171]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:21:03.042074442 (11h41m2.631419429s) [upload=12171]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:22:02.399093104 (11h42m1.988438091s) [upload=12171]: Next segment err: context deadline exceeded Aug 7 21:22:02.399277657 (11h42m1.988622644s) [upload=12171]: Done uploading segments... Aug 7 21:22:02.399305961 (11h42m1.988650948s) [upload=12171]: Upload finished. err: context deadline exceeded Aug 7 21:22:03.257202341 (11h42m2.846547328s) [upload=12189]: Starting upload Aug 7 21:22:03.257267245 (11h42m2.846612232s) [upload=12189]: Done starting upload Aug 7 21:22:03.257356857 (11h42m2.846701844s) [upload=12189]: Uploading segments... Aug 7 21:22:03.257444765 (11h42m2.846789742s) [upload=12189]: Got next segment. Inline: false Aug 7 21:22:03.257470033 (11h42m2.84681501s) [upload=12189]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:22:03.257474171 (11h42m2.846819158s) [upload=12189]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:23:02.611478992 (11h43m2.20082398s) [upload=12189]: Next segment err: context deadline exceeded Aug 7 21:23:02.611504952 (11h43m2.200849939s) [upload=12189]: Done uploading segments... Aug 7 21:23:02.611523989 (11h43m2.200868976s) [upload=12189]: Upload finished. err: context deadline exceeded Aug 7 21:23:11.364660428 (11h43m10.954005435s) [upload=12204]: Starting upload Aug 7 21:23:11.364761051 (11h43m10.954106038s) [upload=12204]: Done starting upload Aug 7 21:23:11.364863036 (11h43m10.954208023s) [upload=12204]: Uploading segments... Aug 7 21:23:11.364973286 (11h43m10.954318264s) [upload=12204]: Got next segment. Inline: false Aug 7 21:23:11.365036508 (11h43m10.954381495s) [upload=12204]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:23:11.365041447 (11h43m10.954386434s) [upload=12204]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:24:10.722815877 (11h44m10.312160864s) [upload=12204]: Next segment err: context deadline exceeded Aug 7 21:24:10.722857978 (11h44m10.312202965s) [upload=12204]: Done uploading segments... Aug 7 21:24:10.722871453 (11h44m10.31221644s) [upload=12204]: Upload finished. err: context deadline exceeded Aug 7 21:24:23.410577309 (11h44m22.999922296s) [upload=12220]: Starting upload Aug 7 21:24:23.410633838 (11h44m22.999978825s) [upload=12220]: Done starting upload Aug 7 21:24:23.410696377 (11h44m23.000041364s) [upload=12220]: Uploading segments... Aug 7 21:24:23.410808262 (11h44m23.000153239s) [upload=12220]: Got next segment. Inline: false Aug 7 21:24:23.410833469 (11h44m23.000178456s) [upload=12220]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:24:23.410837017 (11h44m23.000182004s) [upload=12220]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:25:22.762515918 (11h45m22.351860905s) [upload=12220]: Next segment err: context deadline exceeded Aug 7 21:25:22.762663662 (11h45m22.352008639s) [upload=12220]: Done uploading segments... Aug 7 21:25:22.762679401 (11h45m22.352024378s) [upload=12220]: Upload finished. err: context deadline exceeded Aug 7 21:25:23.709743833 (11h45m23.29908882s) [upload=12237]: Starting upload Aug 7 21:25:23.709835399 (11h45m23.299180386s) [upload=12237]: Done starting upload Aug 7 21:25:23.709934688 (11h45m23.299279675s) [upload=12237]: Uploading segments... Aug 7 21:25:23.710036063 (11h45m23.29938105s) [upload=12237]: Got next segment. Inline: false Aug 7 21:25:23.710068875 (11h45m23.299413862s) [upload=12237]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:25:23.710072732 (11h45m23.299417719s) [upload=12237]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:26:23.059097010 (11h46m22.648441997s) [upload=12237]: Next segment err: context deadline exceeded Aug 7 21:26:23.059154440 (11h46m22.648499427s) [upload=12237]: Done uploading segments... Aug 7 21:26:23.059167756 (11h46m22.648512743s) [upload=12237]: Upload finished. err: context deadline exceeded Aug 7 21:26:32.555948975 (11h46m32.145293952s) [upload=12256]: Starting upload Aug 7 21:26:32.555997727 (11h46m32.145342704s) [upload=12256]: Done starting upload Aug 7 21:26:32.556084444 (11h46m32.145429421s) [upload=12256]: Uploading segments... Aug 7 21:26:32.556166260 (11h46m32.145511247s) [upload=12256]: Got next segment. Inline: false Aug 7 21:26:32.556194715 (11h46m32.145539702s) [upload=12256]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:26:32.556198572 (11h46m32.145543559s) [upload=12256]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:27:31.908976804 (11h47m31.498321791s) [upload=12256]: Next segment err: context deadline exceeded Aug 7 21:27:31.909049884 (11h47m31.498394861s) [upload=12256]: Done uploading segments... Aug 7 21:27:31.909064964 (11h47m31.498409951s) [upload=12256]: Upload finished. err: context deadline exceeded Duration: 1m0.001044595s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [...........................................XXXXXXXXXXXXXXXXX] duration: 30.001141761s FAIL (2025-08-07 21:27:28 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.001160868s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 17 out of 60 is failed (threshold=3) [...........................................XXXXXXXXXXXXXXXXX] duration: 30.001057432s FAIL (2025-08-07 21:27:20 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== .913729104s) [upload=12138]: Uploading segments... Aug 7 21:18:57.324553602 (11h38m56.91389866s) [upload=12138]: Got next segment. Inline: true Aug 7 21:18:57.324563731 (11h38m56.913908739s) [upload=12138]: Waiting for error group managing segments... Aug 7 21:18:57.324671017 (11h38m56.914016054s) [upload=12138]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:18:57.324679312 (11h38m56.91402431s) [upload=12138]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:18:57.324726984 (11h38m56.914071981s) [upload=12138]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:19:26.673751120 (11h39m26.263096147s) [upload=12138]: Done uploading segments... Aug 7 21:19:26.673774865 (11h39m26.263119882s) [upload=12138]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:19:31.851042818 (11h39m31.440387805s) [upload=12148]: Starting upload Aug 7 21:19:31.851107531 (11h39m31.440452519s) [upload=12148]: Done starting upload Aug 7 21:19:31.851152999 (11h39m31.440497986s) [upload=12148]: Uploading segments... Aug 7 21:19:31.851212192 (11h39m31.440557179s) [upload=12148]: Got next segment. Inline: true Aug 7 21:19:31.851215829 (11h39m31.440560816s) [upload=12148]: Waiting for error group managing segments... Aug 7 21:19:31.851259963 (11h39m31.44060495s) [upload=12148]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:19:31.851264352 (11h39m31.440609329s) [upload=12148]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:19:31.851266806 (11h39m31.440611793s) [upload=12148]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:20:01.198301602 (11h40m0.787646589s) [upload=12148]: Done uploading segments... Aug 7 21:20:01.198330468 (11h40m0.787675455s) [upload=12148]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:20:14.918521871 (11h40m14.507866859s) [upload=12161]: Starting upload Aug 7 21:20:14.918603488 (11h40m14.507948495s) [upload=12161]: Done starting upload Aug 7 21:20:14.918661038 (11h40m14.508006025s) [upload=12161]: Uploading segments... Aug 7 21:20:14.918718718 (11h40m14.508063705s) [upload=12161]: Got next segment. Inline: true Aug 7 21:20:14.918725691 (11h40m14.508070688s) [upload=12161]: Waiting for error group managing segments... Aug 7 21:20:14.918778933 (11h40m14.50812394s) [upload=12161]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:20:14.918785126 (11h40m14.508130133s) [upload=12161]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:20:14.918790756 (11h40m14.508135773s) [upload=12161]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:20:44.279825769 (11h40m43.869170756s) [upload=12161]: Done uploading segments... Aug 7 21:20:44.279847039 (11h40m43.869192027s) [upload=12161]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:21:26.967144759 (11h41m26.556489736s) [upload=12182]: Starting upload Aug 7 21:21:26.967204002 (11h41m26.556549009s) [upload=12182]: Done starting upload Aug 7 21:21:26.967248207 (11h41m26.556593194s) [upload=12182]: Uploading segments... Aug 7 21:21:26.967275329 (11h41m26.556620316s) [upload=12182]: Got next segment. Inline: true Aug 7 21:21:26.967278815 (11h41m26.556623802s) [upload=12182]: Waiting for error group managing segments... Aug 7 21:21:26.967311758 (11h41m26.556656745s) [upload=12182]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:21:26.967315786 (11h41m26.556660773s) [upload=12182]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:21:26.967318120 (11h41m26.556663107s) [upload=12182]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:21:56.309770442 (11h41m55.899115429s) [upload=12182]: Done uploading segments... Aug 7 21:21:56.309792865 (11h41m55.899137852s) [upload=12182]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:22:21.925019495 (11h42m21.514364572s) [upload=12195]: Starting upload Aug 7 21:22:21.925098336 (11h42m21.514443323s) [upload=12195]: Done starting upload Aug 7 21:22:21.925142180 (11h42m21.514487167s) [upload=12195]: Uploading segments... Aug 7 21:22:21.925175574 (11h42m21.514520561s) [upload=12195]: Got next segment. Inline: true Aug 7 21:22:21.925179491 (11h42m21.514524478s) [upload=12195]: Waiting for error group managing segments... Aug 7 21:22:21.925210671 (11h42m21.514555648s) [upload=12195]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:22:21.925214048 (11h42m21.514559035s) [upload=12195]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:22:21.925216302 (11h42m21.51456129s) [upload=12195]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:22:51.283998863 (11h42m50.87334385s) [upload=12195]: Done uploading segments... Aug 7 21:22:51.284018922 (11h42m50.873363899s) [upload=12195]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:23:36.916300477 (11h43m36.505645464s) [upload=12212]: Starting upload Aug 7 21:23:36.916350923 (11h43m36.50569591s) [upload=12212]: Done starting upload Aug 7 21:23:36.916395318 (11h43m36.505740295s) [upload=12212]: Uploading segments... Aug 7 21:23:36.916436557 (11h43m36.505781544s) [upload=12212]: Got next segment. Inline: true Aug 7 21:23:36.916439863 (11h43m36.50578484s) [upload=12212]: Waiting for error group managing segments... Aug 7 21:23:36.916471534 (11h43m36.505816511s) [upload=12212]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:23:36.916475351 (11h43m36.505820338s) [upload=12212]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:23:36.916477665 (11h43m36.505822652s) [upload=12212]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:24:06.267030467 (11h44m5.856375454s) [upload=12212]: Done uploading segments... Aug 7 21:24:06.267052209 (11h44m5.856397196s) [upload=12212]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:24:29.996857636 (11h44m29.586202623s) [upload=12224]: Starting upload Aug 7 21:24:29.996913823 (11h44m29.5862588s) [upload=12224]: Done starting upload Aug 7 21:24:29.996961495 (11h44m29.586306482s) [upload=12224]: Uploading segments... Aug 7 21:24:29.996996181 (11h44m29.586341158s) [upload=12224]: Got next segment. Inline: true Aug 7 21:24:29.996999788 (11h44m29.586344765s) [upload=12224]: Waiting for error group managing segments... Aug 7 21:24:29.997033352 (11h44m29.586378329s) [upload=12224]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:24:29.997037590 (11h44m29.586382577s) [upload=12224]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:24:29.997040075 (11h44m29.586385062s) [upload=12224]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:24:59.358108170 (11h44m58.947453157s) [upload=12224]: Done uploading segments... Aug 7 21:24:59.358131014 (11h44m58.947476001s) [upload=12224]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:25:00.224254754 (11h44m59.813599741s) [upload=12230]: Starting upload Aug 7 21:25:00.224330690 (11h44m59.813675677s) [upload=12230]: Done starting upload Aug 7 21:25:00.224406304 (11h44m59.813751311s) [upload=12230]: Uploading segments... Aug 7 21:25:00.224533207 (11h44m59.813878204s) [upload=12230]: Got next segment. Inline: true Aug 7 21:25:00.224541352 (11h44m59.813886339s) [upload=12230]: Waiting for error group managing segments... Aug 7 21:25:00.224626114 (11h44m59.813971121s) [upload=12230]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:25:00.224634080 (11h44m59.813979107s) [upload=12230]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:25:00.224638929 (11h44m59.813983937s) [upload=12230]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:25:29.580431600 (11h45m29.169776597s) [upload=12230]: Done uploading segments... Aug 7 21:25:29.580456288 (11h45m29.169801275s) [upload=12230]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:25:38.048208784 (11h45m37.637553761s) [upload=12241]: Starting upload Aug 7 21:25:38.048265823 (11h45m37.63761081s) [upload=12241]: Done starting upload Aug 7 21:25:38.048311521 (11h45m37.637656508s) [upload=12241]: Uploading segments... Aug 7 21:25:38.048347439 (11h45m37.637692426s) [upload=12241]: Got next segment. Inline: true Aug 7 21:25:38.048350867 (11h45m37.637695854s) [upload=12241]: Waiting for error group managing segments... Aug 7 21:25:38.048383188 (11h45m37.637728186s) [upload=12241]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:25:38.048387396 (11h45m37.637732383s) [upload=12241]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:25:38.048389881 (11h45m37.637734868s) [upload=12241]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:26:07.388744771 (11h46m6.978089758s) [upload=12241]: Done uploading segments... Aug 7 21:26:07.388753889 (11h46m6.978098876s) [upload=12241]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:26:51.633559976 (11h46m51.222904953s) [upload=12259]: Starting upload Aug 7 21:26:51.633607808 (11h46m51.222952795s) [upload=12259]: Done starting upload Aug 7 21:26:51.633657383 (11h46m51.22300237s) [upload=12259]: Uploading segments... Aug 7 21:26:51.633689063 (11h46m51.22303405s) [upload=12259]: Got next segment. Inline: true Aug 7 21:26:51.633692660 (11h46m51.223037647s) [upload=12259]: Waiting for error group managing segments... Aug 7 21:26:51.633727035 (11h46m51.223072012s) [upload=12259]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:26:51.633730782 (11h46m51.22307577s) [upload=12259]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:26:51.633733327 (11h46m51.223078314s) [upload=12259]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:27:20.991495967 (11h47m20.580840944s) [upload=12259]: Done uploading segments... Aug 7 21:27:20.991524641 (11h47m20.580869628s) [upload=12259]: Upload finished. err: metaclient: context deadline exceeded Duration: 30.00107753s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 14 out of 60 is failed (threshold=3) [.............................................X.XXXXXXXXXXXXX] duration: 1m0.001163192s FAIL (2025-08-07 21:27:17 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 19029552 (11h36m45.808374539s) [upload=12107]: Uploading segments... Aug 7 21:16:46.219157317 (11h36m45.808502334s) [upload=12107]: Got next segment. Inline: true Aug 7 21:16:46.219163238 (11h36m45.808508225s) [upload=12107]: Waiting for error group managing segments... Aug 7 21:16:46.219206872 (11h36m45.808551859s) [upload=12107]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:16:46.219212492 (11h36m45.80855747s) [upload=12107]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:16:46.219215178 (11h36m45.808560155s) [upload=12107]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:17:45.569887093 (11h37m45.15923208s) [upload=12107]: Done uploading segments... Aug 7 21:17:45.569902753 (11h37m45.15924774s) [upload=12107]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:17:46.442498934 (11h37m46.031843911s) [upload=12121]: Starting upload Aug 7 21:17:46.442561293 (11h37m46.03190628s) [upload=12121]: Done starting upload Aug 7 21:17:46.442612481 (11h37m46.031957468s) [upload=12121]: Uploading segments... Aug 7 21:17:46.442661796 (11h37m46.032006783s) [upload=12121]: Got next segment. Inline: true Aug 7 21:17:46.442665573 (11h37m46.03201056s) [upload=12121]: Waiting for error group managing segments... Aug 7 21:17:46.442697183 (11h37m46.03204217s) [upload=12121]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:17:46.442700580 (11h37m46.032045557s) [upload=12121]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:17:46.442704637 (11h37m46.032049624s) [upload=12121]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:18:45.798465603 (11h38m45.38781059s) [upload=12121]: Done uploading segments... Aug 7 21:18:45.798489779 (11h38m45.387834766s) [upload=12121]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:18:57.960074990 (11h38m57.549419977s) [upload=12139]: Starting upload Aug 7 21:18:57.960165033 (11h38m57.54951003s) [upload=12139]: Done starting upload Aug 7 21:18:57.960244845 (11h38m57.549589842s) [upload=12139]: Uploading segments... Aug 7 21:18:57.960294299 (11h38m57.549639286s) [upload=12139]: Got next segment. Inline: true Aug 7 21:18:57.960298928 (11h38m57.549643915s) [upload=12139]: Waiting for error group managing segments... Aug 7 21:18:57.960349855 (11h38m57.549694853s) [upload=12139]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:18:57.960354916 (11h38m57.549699903s) [upload=12139]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:18:57.960358112 (11h38m57.549703109s) [upload=12139]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:19:57.313173678 (11h39m56.902518665s) [upload=12139]: Done uploading segments... Aug 7 21:19:57.313193636 (11h39m56.902538623s) [upload=12139]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:20:04.133582941 (11h40m3.722927928s) [upload=12156]: Starting upload Aug 7 21:20:04.133643998 (11h40m3.722988985s) [upload=12156]: Done starting upload Aug 7 21:20:04.133712729 (11h40m3.723057726s) [upload=12156]: Uploading segments... Aug 7 21:20:04.133764127 (11h40m3.723109114s) [upload=12156]: Got next segment. Inline: true Aug 7 21:20:04.133769046 (11h40m3.723114033s) [upload=12156]: Waiting for error group managing segments... Aug 7 21:20:04.133838079 (11h40m3.723183066s) [upload=12156]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:20:04.133842868 (11h40m3.723187855s) [upload=12156]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:20:04.133845723 (11h40m3.72319071s) [upload=12156]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:21:03.481385970 (11h41m3.070730947s) [upload=12156]: Done uploading segments... Aug 7 21:21:03.481407009 (11h41m3.070751986s) [upload=12156]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:21:04.366237551 (11h41m3.955582538s) [upload=12172]: Starting upload Aug 7 21:21:04.366309369 (11h41m3.955654356s) [upload=12172]: Done starting upload Aug 7 21:21:04.366360317 (11h41m3.955705304s) [upload=12172]: Uploading segments... Aug 7 21:21:04.366406304 (11h41m3.955751291s) [upload=12172]: Got next segment. Inline: true Aug 7 21:21:04.366410022 (11h41m3.955755009s) [upload=12172]: Waiting for error group managing segments... Aug 7 21:21:04.366441823 (11h41m3.95578681s) [upload=12172]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:21:04.366445840 (11h41m3.955790827s) [upload=12172]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:21:04.366448265 (11h41m3.955793252s) [upload=12172]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:22:03.717958127 (11h42m3.307303114s) [upload=12172]: Done uploading segments... Aug 7 21:22:03.717976892 (11h42m3.307321879s) [upload=12172]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:22:11.311019027 (11h42m10.900364014s) [upload=12193]: Starting upload Aug 7 21:22:11.311074243 (11h42m10.90041923s) [upload=12193]: Done starting upload Aug 7 21:22:11.311123727 (11h42m10.900468714s) [upload=12193]: Uploading segments... Aug 7 21:22:11.311167090 (11h42m10.900512077s) [upload=12193]: Got next segment. Inline: true Aug 7 21:22:11.311183532 (11h42m10.900528519s) [upload=12193]: Waiting for error group managing segments... Aug 7 21:22:11.311220152 (11h42m10.900565139s) [upload=12193]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:22:11.311224931 (11h42m10.900569918s) [upload=12193]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:22:11.311227586 (11h42m10.900572573s) [upload=12193]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:23:10.658643233 (11h43m10.24798821s) [upload=12193]: Done uploading segments... Aug 7 21:23:10.658662571 (11h43m10.248007558s) [upload=12193]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:23:11.571475028 (11h43m11.160820015s) [upload=12205]: Starting upload Aug 7 21:23:11.571532048 (11h43m11.160877035s) [upload=12205]: Done starting upload Aug 7 21:23:11.571582173 (11h43m11.16092716s) [upload=12205]: Uploading segments... Aug 7 21:23:11.571617942 (11h43m11.160962929s) [upload=12205]: Got next segment. Inline: true Aug 7 21:23:11.571621489 (11h43m11.160966486s) [upload=12205]: Waiting for error group managing segments... Aug 7 21:23:11.571655484 (11h43m11.161000471s) [upload=12205]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:23:11.571659531 (11h43m11.161004528s) [upload=12205]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:23:11.571662447 (11h43m11.161007444s) [upload=12205]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:24:10.926729728 (11h44m10.516074715s) [upload=12205]: Done uploading segments... Aug 7 21:24:10.926747603 (11h44m10.51609259s) [upload=12205]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:24:11.802089650 (11h44m11.391434637s) [upload=12218]: Starting upload Aug 7 21:24:11.802151558 (11h44m11.391496545s) [upload=12218]: Done starting upload Aug 7 21:24:11.802205020 (11h44m11.391549997s) [upload=12218]: Uploading segments... Aug 7 21:24:11.802240418 (11h44m11.391585406s) [upload=12218]: Got next segment. Inline: true Aug 7 21:24:11.802244236 (11h44m11.391589223s) [upload=12218]: Waiting for error group managing segments... Aug 7 21:24:11.802278040 (11h44m11.391623027s) [upload=12218]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:24:11.802282458 (11h44m11.391627445s) [upload=12218]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:24:11.802284602 (11h44m11.391629579s) [upload=12218]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:25:11.151652598 (11h45m10.740997575s) [upload=12218]: Done uploading segments... Aug 7 21:25:11.151674050 (11h45m10.741019037s) [upload=12218]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:25:17.859162022 (11h45m17.44850703s) [upload=12235]: Starting upload Aug 7 21:25:17.859247908 (11h45m17.448592915s) [upload=12235]: Done starting upload Aug 7 21:25:17.859314314 (11h45m17.448659301s) [upload=12235]: Uploading segments... Aug 7 21:25:17.859360473 (11h45m17.44870546s) [upload=12235]: Got next segment. Inline: true Aug 7 21:25:17.859365562 (11h45m17.44871056s) [upload=12235]: Waiting for error group managing segments... Aug 7 21:25:17.859407051 (11h45m17.448752058s) [upload=12235]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:25:17.859412402 (11h45m17.448757399s) [upload=12235]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:25:17.859415648 (11h45m17.448760646s) [upload=12235]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:26:17.209779892 (11h46m16.799124879s) [upload=12235]: Done uploading segments... Aug 7 21:26:17.209836961 (11h46m16.799181949s) [upload=12235]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:26:18.091653415 (11h46m17.680998403s) [upload=12251]: Starting upload Aug 7 21:26:18.091708220 (11h46m17.681053207s) [upload=12251]: Done starting upload Aug 7 21:26:18.091749139 (11h46m17.681094126s) [upload=12251]: Uploading segments... Aug 7 21:26:18.091784216 (11h46m17.681129203s) [upload=12251]: Got next segment. Inline: true Aug 7 21:26:18.091787672 (11h46m17.681132649s) [upload=12251]: Waiting for error group managing segments... Aug 7 21:26:18.091819754 (11h46m17.681164741s) [upload=12251]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:26:18.091823481 (11h46m17.681168468s) [upload=12251]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:26:18.091826015 (11h46m17.681170992s) [upload=12251]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:27:17.447643584 (11h47m17.036988571s) [upload=12251]: Done uploading segments... Aug 7 21:27:17.447669333 (11h47m17.03701433s) [upload=12251]: Upload finished. err: metaclient: context deadline exceeded Duration: 1m0.001176116s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [................X...................................X.......] duration: 1.908262505s FAIL (2025-08-07 21:27:17 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/1K") ======LOGS====== ]: Done uploading segments... Aug 7 21:13:55.370929874 (11h33m54.960274861s) [upload=12060]: Upload finished. err: <nil> Aug 7 21:14:45.614661829 (11h34m45.204006816s) [upload=12073]: Starting upload Aug 7 21:14:45.614700943 (11h34m45.20404593s) [upload=12073]: Done starting upload Aug 7 21:14:45.614745929 (11h34m45.204090917s) [upload=12073]: Uploading segments... Aug 7 21:14:45.614774204 (11h34m45.204119181s) [upload=12073]: Got next segment. Inline: true Aug 7 21:14:45.614777830 (11h34m45.204122817s) [upload=12073]: Waiting for error group managing segments... Aug 7 21:14:45.614809601 (11h34m45.204154578s) [upload=12073]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:14:45.614813057 (11h34m45.204158045s) [upload=12073]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:14:45.614815242 (11h34m45.204160219s) [upload=12073]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:14:45.911976434 (11h34m45.501321421s) [upload=12073]: Done uploading segments... Aug 7 21:14:45.911995080 (11h34m45.501340067s) [upload=12073]: Upload finished. err: <nil> Aug 7 21:15:14.898649385 (11h35m14.487994372s) [upload=12079]: Starting upload Aug 7 21:15:14.898694180 (11h35m14.488039167s) [upload=12079]: Done starting upload Aug 7 21:15:14.898741050 (11h35m14.488086027s) [upload=12079]: Uploading segments... Aug 7 21:15:14.898781948 (11h35m14.488126965s) [upload=12079]: Got next segment. Inline: true Aug 7 21:15:14.898789092 (11h35m14.488134079s) [upload=12079]: Waiting for error group managing segments... Aug 7 21:15:14.898847644 (11h35m14.488192641s) [upload=12079]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:15:14.898856401 (11h35m14.488201408s) [upload=12079]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:15:14.898860989 (11h35m14.488206026s) [upload=12079]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:15:15.228046082 (11h35m14.817391079s) [upload=12079]: Done uploading segments... Aug 7 21:15:15.228066881 (11h35m14.817411878s) [upload=12079]: Upload finished. err: <nil> Aug 7 21:16:25.941982337 (11h36m25.531327324s) [upload=12099]: Starting upload Aug 7 21:16:25.942029607 (11h36m25.531374594s) [upload=12099]: Done starting upload Aug 7 21:16:25.942075234 (11h36m25.531420221s) [upload=12099]: Uploading segments... Aug 7 21:16:25.942105222 (11h36m25.531450199s) [upload=12099]: Got next segment. Inline: true Aug 7 21:16:25.942108588 (11h36m25.531453575s) [upload=12099]: Waiting for error group managing segments... Aug 7 21:16:25.942142683 (11h36m25.53148767s) [upload=12099]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:16:25.942146340 (11h36m25.531491317s) [upload=12099]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:16:25.942148595 (11h36m25.531493582s) [upload=12099]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:16:26.215688283 (11h36m25.80503327s) [upload=12099]: Done uploading segments... Aug 7 21:16:26.215705086 (11h36m25.805050073s) [upload=12099]: Upload finished. err: <nil> Aug 7 21:17:38.285379669 (11h37m37.874724656s) [upload=12117]: Starting upload Aug 7 21:17:38.285441276 (11h37m37.874786323s) [upload=12117]: Done starting upload Aug 7 21:17:38.285507584 (11h37m37.874852611s) [upload=12117]: Uploading segments... Aug 7 21:17:38.285571005 (11h37m37.874915992s) [upload=12117]: Got next segment. Inline: true Aug 7 21:17:38.285575654 (11h37m37.874920641s) [upload=12117]: Waiting for error group managing segments... Aug 7 21:17:38.285618215 (11h37m37.874963243s) [upload=12117]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:17:38.285624958 (11h37m37.874969945s) [upload=12117]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:17:38.285629477 (11h37m37.874974474s) [upload=12117]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:17:38.561705487 (11h37m38.151050484s) [upload=12117]: Done uploading segments... Aug 7 21:17:38.561727969 (11h37m38.151072957s) [upload=12117]: Upload finished. err: <nil> Aug 7 21:18:46.880586061 (11h38m46.469931048s) [upload=12133]: Starting upload Aug 7 21:18:46.880627500 (11h38m46.469972487s) [upload=12133]: Done starting upload Aug 7 21:18:46.880681523 (11h38m46.47002651s) [upload=12133]: Uploading segments... Aug 7 21:18:46.880714476 (11h38m46.470059463s) [upload=12133]: Got next segment. Inline: true Aug 7 21:18:46.880718043 (11h38m46.47006302s) [upload=12133]: Waiting for error group managing segments... Aug 7 21:18:46.880752089 (11h38m46.470097076s) [upload=12133]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:18:46.880755715 (11h38m46.470100702s) [upload=12133]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:18:46.880758120 (11h38m46.470103107s) [upload=12133]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:18:47.149461450 (11h38m46.738806437s) [upload=12133]: Done uploading segments... Aug 7 21:18:47.149478763 (11h38m46.73882375s) [upload=12133]: Upload finished. err: <nil> Aug 7 21:19:44.345616276 (11h39m43.934961263s) [upload=12152]: Starting upload Aug 7 21:19:44.345660120 (11h39m43.935005098s) [upload=12152]: Done starting upload Aug 7 21:19:44.345731898 (11h39m43.935076885s) [upload=12152]: Uploading segments... Aug 7 21:19:44.345839653 (11h39m43.93518463s) [upload=12152]: Got next segment. Inline: true Aug 7 21:19:44.345844203 (11h39m43.93518919s) [upload=12152]: Waiting for error group managing segments... Aug 7 21:19:44.345886133 (11h39m43.93523111s) [upload=12152]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:19:44.345890932 (11h39m43.935235919s) [upload=12152]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:19:44.345893316 (11h39m43.935238303s) [upload=12152]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:19:44.666041542 (11h39m44.255386529s) [upload=12152]: Done uploading segments... Aug 7 21:19:44.666059567 (11h39m44.255404554s) [upload=12152]: Upload finished. err: <nil> Aug 7 21:20:10.176957983 (11h40m9.76630296s) [upload=12157]: Starting upload Aug 7 21:20:10.177009311 (11h40m9.766354298s) [upload=12157]: Done starting upload Aug 7 21:20:10.177070559 (11h40m9.766415546s) [upload=12157]: Uploading segments... Aug 7 21:20:10.177121496 (11h40m9.766466483s) [upload=12157]: Got next segment. Inline: true Aug 7 21:20:10.177125895 (11h40m9.766470882s) [upload=12157]: Waiting for error group managing segments... Aug 7 21:20:10.177160220 (11h40m9.766505207s) [upload=12157]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:20:10.177164087 (11h40m9.766509074s) [upload=12157]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:20:10.177166371 (11h40m9.766511359s) [upload=12157]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:20:10.463794245 (11h40m10.053139232s) [upload=12157]: Done uploading segments... Aug 7 21:20:10.463812280 (11h40m10.053157257s) [upload=12157]: Upload finished. err: <nil> Aug 7 21:20:47.295095407 (11h40m46.884440394s) [upload=12169]: Starting upload Aug 7 21:20:47.295162806 (11h40m46.884507793s) [upload=12169]: Done starting upload Aug 7 21:20:47.295214385 (11h40m46.884559372s) [upload=12169]: Uploading segments... Aug 7 21:20:47.295251876 (11h40m46.884596864s) [upload=12169]: Got next segment. Inline: true Aug 7 21:20:47.295255583 (11h40m46.88460057s) [upload=12169]: Waiting for error group managing segments... Aug 7 21:20:47.295291642 (11h40m46.884636629s) [upload=12169]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:20:47.295295450 (11h40m46.884640437s) [upload=12169]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:20:47.295297755 (11h40m46.884642742s) [upload=12169]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:20:47.581180532 (11h40m47.170525529s) [upload=12169]: Done uploading segments... Aug 7 21:20:47.581201993 (11h40m47.17054698s) [upload=12169]: Upload finished. err: <nil> Aug 7 21:21:39.503326248 (11h41m39.092671235s) [upload=12184]: Starting upload Aug 7 21:21:39.503368609 (11h41m39.092713586s) [upload=12184]: Done starting upload Aug 7 21:21:39.503416451 (11h41m39.092761438s) [upload=12184]: Uploading segments... Aug 7 21:21:39.503447480 (11h41m39.092792467s) [upload=12184]: Got next segment. Inline: true Aug 7 21:21:39.503451247 (11h41m39.092796234s) [upload=12184]: Waiting for error group managing segments... Aug 7 21:21:39.503485393 (11h41m39.09283038s) [upload=12184]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:21:39.503489370 (11h41m39.092834357s) [upload=12184]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:21:39.503491825 (11h41m39.092836802s) [upload=12184]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:21:39.826158170 (11h41m39.415503147s) [upload=12184]: Done uploading segments... Aug 7 21:21:39.826174040 (11h41m39.415519027s) [upload=12184]: Upload finished. err: <nil> Aug 7 21:22:46.427091203 (11h42m46.01643619s) [upload=12199]: Starting upload Aug 7 21:22:46.427135929 (11h42m46.016480916s) [upload=12199]: Done starting upload Aug 7 21:22:46.427188519 (11h42m46.016533496s) [upload=12199]: Uploading segments... Aug 7 21:22:46.427215882 (11h42m46.016560869s) [upload=12199]: Got next segment. Inline: true Aug 7 21:22:46.427219829 (11h42m46.016564806s) [upload=12199]: Waiting for error group managing segments... Aug 7 21:22:46.427252391 (11h42m46.016597378s) [upload=12199]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:22:46.427256048 (11h42m46.016601025s) [upload=12199]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:22:46.427268202 (11h42m46.016613189s) [upload=12199]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:22:46.705229475 (11h42m46.294574462s) [upload=12199]: Done uploading segments... Aug 7 21:22:46.705247530 (11h42m46.294592507s) [upload=12199]: Upload finished. err: <nil> Duration: 1.908287833s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 17 out of 60 is failed (threshold=3) [...........................................XXXXXXXXXXXXXXXXX] duration: 30.000423547s FAIL (2025-08-07 21:27:15 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.000441511s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 14 out of 60 is failed (threshold=3) [............................................X..XXXXXXXXXXXXX] duration: 30.001280954s FAIL (2025-08-07 21:27:06 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 30.001301293s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [.........................................X..XXXXXXXXXXXXXXXX] duration: 30.001233917s FAIL (2025-08-07 21:26:46 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== ext segment err: context deadline exceeded Aug 7 21:17:14.466505420 (11h37m14.055850407s) [upload=12105]: Done uploading segments... Aug 7 21:17:14.466521220 (11h37m14.055866207s) [upload=12105]: Upload finished. err: context deadline exceeded Aug 7 21:17:40.540477806 (11h37m40.129822823s) [upload=12118]: Starting upload Aug 7 21:17:40.540597054 (11h37m40.129942061s) [upload=12118]: Done starting upload Aug 7 21:17:40.540730389 (11h37m40.130075426s) [upload=12118]: Uploading segments... Aug 7 21:17:40.541043379 (11h37m40.130388366s) [upload=12118]: Got next segment. Inline: false Aug 7 21:17:40.541084667 (11h37m40.130429654s) [upload=12118]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:17:40.541090498 (11h37m40.130435485s) [upload=12118]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:18:09.894301553 (11h38m9.48364653s) [upload=12118]: Next segment err: context deadline exceeded Aug 7 21:18:09.894385774 (11h38m9.483730782s) [upload=12118]: Done uploading segments... Aug 7 21:18:09.894407416 (11h38m9.483752433s) [upload=12118]: Upload finished. err: context deadline exceeded Aug 7 21:18:51.953663291 (11h38m51.543008278s) [upload=12135]: Starting upload Aug 7 21:18:51.953718226 (11h38m51.543063213s) [upload=12135]: Done starting upload Aug 7 21:18:51.953825141 (11h38m51.543170128s) [upload=12135]: Uploading segments... Aug 7 21:18:51.953923359 (11h38m51.543268346s) [upload=12135]: Got next segment. Inline: false Aug 7 21:18:51.953958877 (11h38m51.543303864s) [upload=12135]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:18:51.953963656 (11h38m51.543308643s) [upload=12135]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:19:21.302189358 (11h39m20.891534335s) [upload=12135]: Next segment err: context deadline exceeded Aug 7 21:19:21.302254121 (11h39m20.891599108s) [upload=12135]: Done uploading segments... Aug 7 21:19:21.302274270 (11h39m20.891619267s) [upload=12135]: Upload finished. err: context deadline exceeded Aug 7 21:19:39.710866482 (11h39m39.300211459s) [upload=12149]: Starting upload Aug 7 21:19:39.710923851 (11h39m39.300268829s) [upload=12149]: Done starting upload Aug 7 21:19:39.710996290 (11h39m39.300341277s) [upload=12149]: Uploading segments... Aug 7 21:19:39.711114146 (11h39m39.300459133s) [upload=12149]: Got next segment. Inline: false Aug 7 21:19:39.711140085 (11h39m39.300485102s) [upload=12149]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:19:39.711144824 (11h39m39.300489811s) [upload=12149]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:20:09.053604686 (11h40m8.642949683s) [upload=12149]: Next segment err: context deadline exceeded Aug 7 21:20:09.053707913 (11h40m8.6430529s) [upload=12149]: Done uploading segments... Aug 7 21:20:09.053723724 (11h40m8.643068711s) [upload=12149]: Upload finished. err: context deadline exceeded Aug 7 21:20:24.058132378 (11h40m23.647477365s) [upload=12163]: Starting upload Aug 7 21:20:24.058185559 (11h40m23.647530546s) [upload=12163]: Done starting upload Aug 7 21:20:24.058265552 (11h40m23.647610529s) [upload=12163]: Uploading segments... Aug 7 21:20:24.058369542 (11h40m23.647714539s) [upload=12163]: Got next segment. Inline: false Aug 7 21:20:24.058412013 (11h40m23.647757s) [upload=12163]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:20:24.058418646 (11h40m23.647763663s) [upload=12163]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:20:53.409396630 (11h40m52.998741617s) [upload=12163]: Next segment err: context deadline exceeded Aug 7 21:20:53.409563509 (11h40m52.998908496s) [upload=12163]: Done uploading segments... Aug 7 21:20:53.409604537 (11h40m52.998949514s) [upload=12163]: Upload finished. err: context deadline exceeded Aug 7 21:21:23.928736473 (11h41m23.51808146s) [upload=12180]: Starting upload Aug 7 21:21:23.928792831 (11h41m23.518137818s) [upload=12180]: Done starting upload Aug 7 21:21:23.928856973 (11h41m23.51820195s) [upload=12180]: Uploading segments... Aug 7 21:21:23.929051695 (11h41m23.518396682s) [upload=12180]: Got next segment. Inline: false Aug 7 21:21:23.929103304 (11h41m23.518448291s) [upload=12180]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:21:23.929107933 (11h41m23.51845292s) [upload=12180]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:21:53.279642106 (11h41m52.868987093s) [upload=12180]: Next segment err: context deadline exceeded Aug 7 21:21:53.279858500 (11h41m52.869203477s) [upload=12180]: Done uploading segments... Aug 7 21:21:53.279873498 (11h41m52.869218485s) [upload=12180]: Upload finished. err: context deadline exceeded Aug 7 21:21:54.191662378 (11h41m53.781007366s) [upload=12188]: Starting upload Aug 7 21:21:54.191712274 (11h41m53.781057251s) [upload=12188]: Done starting upload Aug 7 21:21:54.191790404 (11h41m53.781135391s) [upload=12188]: Uploading segments... Aug 7 21:21:54.192007459 (11h41m53.781352446s) [upload=12188]: Got next segment. Inline: false Aug 7 21:21:54.192034671 (11h41m53.781379658s) [upload=12188]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:21:54.192038719 (11h41m53.781383706s) [upload=12188]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:22:23.533395958 (11h42m23.122740945s) [upload=12188]: Next segment err: context deadline exceeded Aug 7 21:22:23.533474659 (11h42m23.122819646s) [upload=12188]: Done uploading segments... Aug 7 21:22:23.533496701 (11h42m23.122841688s) [upload=12188]: Upload finished. err: context deadline exceeded Aug 7 21:23:08.442262622 (11h43m8.031607609s) [upload=12203]: Starting upload Aug 7 21:23:08.442325042 (11h43m8.031670019s) [upload=12203]: Done starting upload Aug 7 21:23:08.442385668 (11h43m8.031730655s) [upload=12203]: Uploading segments... Aug 7 21:23:08.442501099 (11h43m8.031846076s) [upload=12203]: Got next segment. Inline: false Aug 7 21:23:08.442542839 (11h43m8.031887826s) [upload=12203]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:23:08.442546706 (11h43m8.031891683s) [upload=12203]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:23:37.798210470 (11h43m37.387555457s) [upload=12203]: Next segment err: context deadline exceeded Aug 7 21:23:37.798367780 (11h43m37.387712767s) [upload=12203]: Done uploading segments... Aug 7 21:23:37.798381718 (11h43m37.387726705s) [upload=12203]: Upload finished. err: context deadline exceeded Aug 7 21:23:54.967168247 (11h43m54.556513234s) [upload=12215]: Starting upload Aug 7 21:23:54.967268208 (11h43m54.556613185s) [upload=12215]: Done starting upload Aug 7 21:23:54.967374321 (11h43m54.556719308s) [upload=12215]: Uploading segments... Aug 7 21:23:54.967600634 (11h43m54.556945621s) [upload=12215]: Got next segment. Inline: false Aug 7 21:23:54.967649888 (11h43m54.556994875s) [upload=12215]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:23:54.967657021 (11h43m54.557002029s) [upload=12215]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:24:24.312833601 (11h44m23.902178598s) [upload=12215]: Next segment err: context deadline exceeded Aug 7 21:24:24.313044925 (11h44m23.902389912s) [upload=12215]: Done uploading segments... Aug 7 21:24:24.313087757 (11h44m23.902432764s) [upload=12215]: Upload finished. err: context deadline exceeded Aug 7 21:24:25.196484464 (11h44m24.785829451s) [upload=12221]: Starting upload Aug 7 21:24:25.196569166 (11h44m24.785914153s) [upload=12221]: Done starting upload Aug 7 21:24:25.196697702 (11h44m24.786042719s) [upload=12221]: Uploading segments... Aug 7 21:24:25.196786912 (11h44m24.786131899s) [upload=12221]: Got next segment. Inline: false Aug 7 21:24:25.196818883 (11h44m24.78616387s) [upload=12221]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:24:25.196823522 (11h44m24.786168509s) [upload=12221]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:24:54.542928213 (11h44m54.1322732s) [upload=12221]: Next segment err: context deadline exceeded Aug 7 21:24:54.543143736 (11h44m54.132488713s) [upload=12221]: Done uploading segments... Aug 7 21:24:54.543169415 (11h44m54.132514402s) [upload=12221]: Upload finished. err: context deadline exceeded Aug 7 21:25:30.578713383 (11h45m30.16805837s) [upload=12239]: Starting upload Aug 7 21:25:30.578779109 (11h45m30.168124096s) [upload=12239]: Done starting upload Aug 7 21:25:30.578836729 (11h45m30.168181716s) [upload=12239]: Uploading segments... Aug 7 21:25:30.579017655 (11h45m30.168362642s) [upload=12239]: Got next segment. Inline: false Aug 7 21:25:30.579083341 (11h45m30.168428328s) [upload=12239]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:25:30.579091086 (11h45m30.168436073s) [upload=12239]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:25:59.907330605 (11h45m59.496675592s) [upload=12239]: Next segment err: context deadline exceeded Aug 7 21:25:59.907404857 (11h45m59.496749845s) [upload=12239]: Done uploading segments... Aug 7 21:25:59.907422762 (11h45m59.496767749s) [upload=12239]: Upload finished. err: context deadline exceeded Aug 7 21:26:16.857419024 (11h46m16.446764011s) [upload=12250]: Starting upload Aug 7 21:26:16.857477205 (11h46m16.446822182s) [upload=12250]: Done starting upload Aug 7 21:26:16.857572879 (11h46m16.446917856s) [upload=12250]: Uploading segments... Aug 7 21:26:16.857644405 (11h46m16.446989392s) [upload=12250]: Got next segment. Inline: false Aug 7 21:26:16.857669784 (11h46m16.447014771s) [upload=12250]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:26:16.857673381 (11h46m16.447018368s) [upload=12250]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:26:46.211380241 (11h46m45.800725218s) [upload=12250]: Next segment err: context deadline exceeded Aug 7 21:26:46.211514648 (11h46m45.800859625s) [upload=12250]: Done uploading segments... Aug 7 21:26:46.211531050 (11h46m45.800876037s) [upload=12250]: Upload finished. err: context deadline exceeded Duration: 30.001249378s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [.................X...................................X......] duration: 2.020542907s FAIL (2025-08-07 21:26:44 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/1K") ======LOGS====== ]: Done uploading segments... Aug 7 21:13:55.370929874 (11h33m54.960274861s) [upload=12060]: Upload finished. err: <nil> Aug 7 21:14:45.614661829 (11h34m45.204006816s) [upload=12073]: Starting upload Aug 7 21:14:45.614700943 (11h34m45.20404593s) [upload=12073]: Done starting upload Aug 7 21:14:45.614745929 (11h34m45.204090917s) [upload=12073]: Uploading segments... Aug 7 21:14:45.614774204 (11h34m45.204119181s) [upload=12073]: Got next segment. Inline: true Aug 7 21:14:45.614777830 (11h34m45.204122817s) [upload=12073]: Waiting for error group managing segments... Aug 7 21:14:45.614809601 (11h34m45.204154578s) [upload=12073]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:14:45.614813057 (11h34m45.204158045s) [upload=12073]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:14:45.614815242 (11h34m45.204160219s) [upload=12073]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:14:45.911976434 (11h34m45.501321421s) [upload=12073]: Done uploading segments... Aug 7 21:14:45.911995080 (11h34m45.501340067s) [upload=12073]: Upload finished. err: <nil> Aug 7 21:15:14.898649385 (11h35m14.487994372s) [upload=12079]: Starting upload Aug 7 21:15:14.898694180 (11h35m14.488039167s) [upload=12079]: Done starting upload Aug 7 21:15:14.898741050 (11h35m14.488086027s) [upload=12079]: Uploading segments... Aug 7 21:15:14.898781948 (11h35m14.488126965s) [upload=12079]: Got next segment. Inline: true Aug 7 21:15:14.898789092 (11h35m14.488134079s) [upload=12079]: Waiting for error group managing segments... Aug 7 21:15:14.898847644 (11h35m14.488192641s) [upload=12079]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:15:14.898856401 (11h35m14.488201408s) [upload=12079]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:15:14.898860989 (11h35m14.488206026s) [upload=12079]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:15:15.228046082 (11h35m14.817391079s) [upload=12079]: Done uploading segments... Aug 7 21:15:15.228066881 (11h35m14.817411878s) [upload=12079]: Upload finished. err: <nil> Aug 7 21:16:25.941982337 (11h36m25.531327324s) [upload=12099]: Starting upload Aug 7 21:16:25.942029607 (11h36m25.531374594s) [upload=12099]: Done starting upload Aug 7 21:16:25.942075234 (11h36m25.531420221s) [upload=12099]: Uploading segments... Aug 7 21:16:25.942105222 (11h36m25.531450199s) [upload=12099]: Got next segment. Inline: true Aug 7 21:16:25.942108588 (11h36m25.531453575s) [upload=12099]: Waiting for error group managing segments... Aug 7 21:16:25.942142683 (11h36m25.53148767s) [upload=12099]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:16:25.942146340 (11h36m25.531491317s) [upload=12099]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:16:25.942148595 (11h36m25.531493582s) [upload=12099]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:16:26.215688283 (11h36m25.80503327s) [upload=12099]: Done uploading segments... Aug 7 21:16:26.215705086 (11h36m25.805050073s) [upload=12099]: Upload finished. err: <nil> Aug 7 21:17:38.285379669 (11h37m37.874724656s) [upload=12117]: Starting upload Aug 7 21:17:38.285441276 (11h37m37.874786323s) [upload=12117]: Done starting upload Aug 7 21:17:38.285507584 (11h37m37.874852611s) [upload=12117]: Uploading segments... Aug 7 21:17:38.285571005 (11h37m37.874915992s) [upload=12117]: Got next segment. Inline: true Aug 7 21:17:38.285575654 (11h37m37.874920641s) [upload=12117]: Waiting for error group managing segments... Aug 7 21:17:38.285618215 (11h37m37.874963243s) [upload=12117]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:17:38.285624958 (11h37m37.874969945s) [upload=12117]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:17:38.285629477 (11h37m37.874974474s) [upload=12117]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:17:38.561705487 (11h37m38.151050484s) [upload=12117]: Done uploading segments... Aug 7 21:17:38.561727969 (11h37m38.151072957s) [upload=12117]: Upload finished. err: <nil> Aug 7 21:18:46.880586061 (11h38m46.469931048s) [upload=12133]: Starting upload Aug 7 21:18:46.880627500 (11h38m46.469972487s) [upload=12133]: Done starting upload Aug 7 21:18:46.880681523 (11h38m46.47002651s) [upload=12133]: Uploading segments... Aug 7 21:18:46.880714476 (11h38m46.470059463s) [upload=12133]: Got next segment. Inline: true Aug 7 21:18:46.880718043 (11h38m46.47006302s) [upload=12133]: Waiting for error group managing segments... Aug 7 21:18:46.880752089 (11h38m46.470097076s) [upload=12133]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:18:46.880755715 (11h38m46.470100702s) [upload=12133]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:18:46.880758120 (11h38m46.470103107s) [upload=12133]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:18:47.149461450 (11h38m46.738806437s) [upload=12133]: Done uploading segments... Aug 7 21:18:47.149478763 (11h38m46.73882375s) [upload=12133]: Upload finished. err: <nil> Aug 7 21:19:44.345616276 (11h39m43.934961263s) [upload=12152]: Starting upload Aug 7 21:19:44.345660120 (11h39m43.935005098s) [upload=12152]: Done starting upload Aug 7 21:19:44.345731898 (11h39m43.935076885s) [upload=12152]: Uploading segments... Aug 7 21:19:44.345839653 (11h39m43.93518463s) [upload=12152]: Got next segment. Inline: true Aug 7 21:19:44.345844203 (11h39m43.93518919s) [upload=12152]: Waiting for error group managing segments... Aug 7 21:19:44.345886133 (11h39m43.93523111s) [upload=12152]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:19:44.345890932 (11h39m43.935235919s) [upload=12152]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:19:44.345893316 (11h39m43.935238303s) [upload=12152]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:19:44.666041542 (11h39m44.255386529s) [upload=12152]: Done uploading segments... Aug 7 21:19:44.666059567 (11h39m44.255404554s) [upload=12152]: Upload finished. err: <nil> Aug 7 21:20:10.176957983 (11h40m9.76630296s) [upload=12157]: Starting upload Aug 7 21:20:10.177009311 (11h40m9.766354298s) [upload=12157]: Done starting upload Aug 7 21:20:10.177070559 (11h40m9.766415546s) [upload=12157]: Uploading segments... Aug 7 21:20:10.177121496 (11h40m9.766466483s) [upload=12157]: Got next segment. Inline: true Aug 7 21:20:10.177125895 (11h40m9.766470882s) [upload=12157]: Waiting for error group managing segments... Aug 7 21:20:10.177160220 (11h40m9.766505207s) [upload=12157]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:20:10.177164087 (11h40m9.766509074s) [upload=12157]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:20:10.177166371 (11h40m9.766511359s) [upload=12157]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:20:10.463794245 (11h40m10.053139232s) [upload=12157]: Done uploading segments... Aug 7 21:20:10.463812280 (11h40m10.053157257s) [upload=12157]: Upload finished. err: <nil> Aug 7 21:20:47.295095407 (11h40m46.884440394s) [upload=12169]: Starting upload Aug 7 21:20:47.295162806 (11h40m46.884507793s) [upload=12169]: Done starting upload Aug 7 21:20:47.295214385 (11h40m46.884559372s) [upload=12169]: Uploading segments... Aug 7 21:20:47.295251876 (11h40m46.884596864s) [upload=12169]: Got next segment. Inline: true Aug 7 21:20:47.295255583 (11h40m46.88460057s) [upload=12169]: Waiting for error group managing segments... Aug 7 21:20:47.295291642 (11h40m46.884636629s) [upload=12169]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:20:47.295295450 (11h40m46.884640437s) [upload=12169]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:20:47.295297755 (11h40m46.884642742s) [upload=12169]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:20:47.581180532 (11h40m47.170525529s) [upload=12169]: Done uploading segments... Aug 7 21:20:47.581201993 (11h40m47.17054698s) [upload=12169]: Upload finished. err: <nil> Aug 7 21:21:39.503326248 (11h41m39.092671235s) [upload=12184]: Starting upload Aug 7 21:21:39.503368609 (11h41m39.092713586s) [upload=12184]: Done starting upload Aug 7 21:21:39.503416451 (11h41m39.092761438s) [upload=12184]: Uploading segments... Aug 7 21:21:39.503447480 (11h41m39.092792467s) [upload=12184]: Got next segment. Inline: true Aug 7 21:21:39.503451247 (11h41m39.092796234s) [upload=12184]: Waiting for error group managing segments... Aug 7 21:21:39.503485393 (11h41m39.09283038s) [upload=12184]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:21:39.503489370 (11h41m39.092834357s) [upload=12184]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:21:39.503491825 (11h41m39.092836802s) [upload=12184]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:21:39.826158170 (11h41m39.415503147s) [upload=12184]: Done uploading segments... Aug 7 21:21:39.826174040 (11h41m39.415519027s) [upload=12184]: Upload finished. err: <nil> Aug 7 21:22:46.427091203 (11h42m46.01643619s) [upload=12199]: Starting upload Aug 7 21:22:46.427135929 (11h42m46.016480916s) [upload=12199]: Done starting upload Aug 7 21:22:46.427188519 (11h42m46.016533496s) [upload=12199]: Uploading segments... Aug 7 21:22:46.427215882 (11h42m46.016560869s) [upload=12199]: Got next segment. Inline: true Aug 7 21:22:46.427219829 (11h42m46.016564806s) [upload=12199]: Waiting for error group managing segments... Aug 7 21:22:46.427252391 (11h42m46.016597378s) [upload=12199]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:22:46.427256048 (11h42m46.016601025s) [upload=12199]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:22:46.427268202 (11h42m46.016613189s) [upload=12199]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:22:46.705229475 (11h42m46.294574462s) [upload=12199]: Done uploading segments... Aug 7 21:22:46.705247530 (11h42m46.294592507s) [upload=12199]: Upload finished. err: <nil> Duration: 2.020571512s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 13 out of 60 is failed (threshold=3) [...............................................XXXXXXXXXXXXX] duration: 1m0.001559397s FAIL (2025-08-07 21:26:43 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.001572973s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 16 out of 60 is failed (threshold=3) [............................................XXXXXXXXXXXXXXXX] duration: 30.001276373s FAIL (2025-08-07 21:26:35 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.001292194s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 5 out of 60 is failed (threshold=3) [......XX...........X.......................X.........X......] duration: 6.252971685s FAIL (2025-08-07 21:26:33 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/4M):: after 2e02304f76f64222b3d705a968d2ee9af3bf0a7208c5fd26b81e13fda594aae8 != before 104921413597176848b09bbcac0eaf1f8376611336d600010bba9ac44347d672 ======LOGS====== 3.946540688 (11h40m43.535885665s) [upload=12165]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:20:44.396290386 (11h40m43.985635373s) [seg_pos={0 0}, upload=12165]: Begin upload segment... Aug 7 21:20:44.396335673 (11h40m43.98568065s) [seg_pos={0 0}, upload=12165]: Done begin upload segment. Aug 7 21:20:44.396363115 (11h40m43.985708092s) [noise=true , address=104.36.82.2:4754 , piece_id=S3CMCUTB3TUV72SMF6633NFLPCMMR2WHAM73J4GLAEPCI5LECGKQ, seg_pos={0 0}, upload=12165]: Uploading piece... Aug 7 21:20:44.396372453 (11h40m43.98571744s) [noise=true , address=104.36.82.22:4521 , piece_id=FSYBN5FIGVGFYX3PZ5Y6G6NG4OTK7NPZOUUERUYSWV2ZK2U7Q2BA, seg_pos={0 0}, upload=12165]: Uploading piece... Aug 7 21:20:44.396739736 (11h40m43.986084723s) [noise=true , address=45.91.116.10:4098 , piece_id=R3KJ5V6S7BWSV3LPFFMO24ROS4WMSEIBABGKHPC7O7MRL5DVUIEA, seg_pos={0 0}, upload=12165]: Uploading piece... Aug 7 21:20:44.397049989 (11h40m43.986394976s) [noise=true , address=104.36.82.62:4435 , piece_id=VMK7QWCLX5QAKTSAQYQXAM47HQXZROULRGPSLCCS55VFBBF3Q3JA, seg_pos={0 0}, upload=12165]: Uploading piece... Aug 7 21:20:44.397325827 (11h40m43.986670814s) [noise=true , address=104.36.82.16:4326 , piece_id=H4U73XAABYVHJRDLX3QGVSPHIFVIAVI63PXJPYQFCCVBYZU74QEQ, seg_pos={0 0}, upload=12165]: Uploading piece... Aug 7 21:20:44.397596706 (11h40m43.986941733s) [noise=true , address=79.127.201.215:4354 , piece_id=GK4QLBAXCQ3ZBABFBMENNPTKVYR4S2OHF6SCMN2CSYTV64HXZYCQ, seg_pos={0 0}, upload=12165]: Uploading piece... Aug 7 21:20:44.397901489 (11h40m43.987246476s) [noise=true , address=104.36.82.34:4559 , piece_id=L4YP4DTEYZZPFF5Q7P67TOOZUCNS2BL4VX2EYZLX7NS6MM6EM6RQ, seg_pos={0 0}, upload=12165]: Uploading piece... Aug 7 21:20:44.398112963 (11h40m43.987457951s) [noise=true , address=104.36.82.59:4329 , piece_id=VETW6WNXY5OZOXEEZKF7QQ4AEOC36WIJWWBQ7W3J2LQ3VNRGBMKA, seg_pos={0 0}, upload=12165]: Uploading piece... Aug 7 21:20:44.398330409 (11h40m43.987675396s) [noise=true , address=104.36.82.46:4501 , piece_id=QV2ULRDKJWZNGPUJ5S66EZ7WGPJWRGNC527P4ZJCTJ3PBDV3KVDA, seg_pos={0 0}, upload=12165]: Uploading piece... Aug 7 21:20:44.398530302 (11h40m43.987875289s) [noise=true , address=104.36.82.45:4913 , piece_id=BZMRI73WGKSW635P527FSWUUBVTKUUKWGIZYOH5PSPKTR62E7ZXA, seg_pos={0 0}, upload=12165]: Uploading piece... Aug 7 21:20:44.398722059 (11h40m43.988067046s) [noise=true , address=104.36.82.61:4398 , piece_id=JL2A3FNDOK5IAHWDIJKVYRQTIUMLSK6THMLBZOXT3VIVTAQ4BJDA, seg_pos={0 0}, upload=12165]: Uploading piece... Aug 7 21:20:44.399272071 (11h40m43.988617058s) [noise=true , address=45.91.116.9:4269 , piece_id=4U5XKZOZ3XVHYH6ZS2B3VOQNJZFFX4MNTDVX3M3J5AIMNNONDQIA, seg_pos={0 0}, upload=12165]: Uploading piece... Aug 7 21:20:44.400049979 (11h40m43.989394966s) [noise=true , address=64.49.13.169:4705 , piece_id=P6HQ5FY5UKJIDVROT6UUQXXAZNO45MSDQZYONEFC7BLUOXRD2U3A, seg_pos={0 0}, upload=12165]: Uploading piece... Aug 7 21:20:44.401362511 (11h40m43.990707498s) [noise=true , address=104.36.82.16:4308 , piece_id=55KQJX7FOEMJZQZ4YPFWOZYTU5M4FR57JQVUBNOX3SEN4LKSDNGQ, seg_pos={0 0}, upload=12165]: Uploading piece... Aug 7 21:20:44.402286328 (11h40m43.991631315s) [noise=true , address=104.36.82.23:4450 , piece_id=AOSQPJBMRVJW4UVQPSL47YBR7JEYYACVX3FPDGJVLS2ZWV2FFN4A, seg_pos={0 0}, upload=12165]: Uploading piece... Aug 7 21:20:44.402739565 (11h40m43.992084552s) [noise=true , address=104.36.82.36:4633 , piece_id=SL7SSCVDNAC75DEQ54LRIHQKHLACTMAUDXZX4ETCY5U5TYYQNBCQ, seg_pos={0 0}, upload=12165]: Uploading piece... Aug 7 21:20:44.406751992 (11h40m43.996096979s) [noise=true , address=79.127.205.234:4585 , piece_id=6HC7GXLTE6RBYUXL3BU7ELFONIVSSZG7LDKZDXYQ4FH7NL2IPJNA, seg_pos={0 0}, upload=12165]: Uploading piece... Aug 7 21:20:44.407069309 (11h40m43.996414286s) [noise=true , address=104.36.82.4:4814 , piece_id=TG423PLOOSKYD5QG3HHLAYXLCA33WPJZFMQV5N2AXLBFJFQY25BA, seg_pos={0 0}, upload=12165]: Uploading piece... Aug 7 21:20:45.508808794 (11h40m45.098153781s) [upload=12165]: Next returned nil segment Aug 7 21:20:45.508823773 (11h40m45.098168761s) [upload=12165]: Waiting for error group managing segments... Aug 7 21:20:45.813770246 (11h40m45.403115233s) [noise=true , address=64.49.13.169:4705 , piece_id=P6HQ5FY5UKJIDVROT6UUQXXAZNO45MSDQZYONEFC7BLUOXRD2U3A, seg_pos={0 0}, upload=12165]: Done uploading piece. err: <nil> Aug 7 21:20:45.820526642 (11h40m45.409871629s) [noise=true , address=79.127.205.234:4585 , piece_id=6HC7GXLTE6RBYUXL3BU7ELFONIVSSZG7LDKZDXYQ4FH7NL2IPJNA, seg_pos={0 0}, upload=12165]: Done uploading piece. err: <nil> Aug 7 21:20:46.344838558 (11h40m45.934183545s) [noise=true , address=79.127.201.215:4354 , piece_id=GK4QLBAXCQ3ZBABFBMENNPTKVYR4S2OHF6SCMN2CSYTV64HXZYCQ, seg_pos={0 0}, upload=12165]: Done uploading piece. err: <nil> Aug 7 21:20:46.363721205 (11h40m45.953066192s) [noise=true , address=104.36.82.16:4326 , piece_id=H4U73XAABYVHJRDLX3QGVSPHIFVIAVI63PXJPYQFCCVBYZU74QEQ, seg_pos={0 0}, upload=12165]: Done uploading piece. err: <nil> Aug 7 21:20:46.372388488 (11h40m45.961733475s) [noise=true , address=104.36.82.16:4308 , piece_id=55KQJX7FOEMJZQZ4YPFWOZYTU5M4FR57JQVUBNOX3SEN4LKSDNGQ, seg_pos={0 0}, upload=12165]: Done uploading piece. err: <nil> Aug 7 21:20:46.423486692 (11h40m46.012831679s) [noise=true , address=104.36.82.45:4913 , piece_id=BZMRI73WGKSW635P527FSWUUBVTKUUKWGIZYOH5PSPKTR62E7ZXA, seg_pos={0 0}, upload=12165]: Done uploading piece. err: <nil> Aug 7 21:20:46.432743402 (11h40m46.022088389s) [noise=true , address=104.36.82.2:4754 , piece_id=S3CMCUTB3TUV72SMF6633NFLPCMMR2WHAM73J4GLAEPCI5LECGKQ, seg_pos={0 0}, upload=12165]: Done uploading piece. err: <nil> Aug 7 21:20:46.433213472 (11h40m46.022558449s) [noise=true , address=104.36.82.36:4633 , piece_id=SL7SSCVDNAC75DEQ54LRIHQKHLACTMAUDXZX4ETCY5U5TYYQNBCQ, seg_pos={0 0}, upload=12165]: Done uploading piece. err: <nil> Aug 7 21:20:46.481915712 (11h40m46.071260689s) [noise=true , address=104.36.82.61:4398 , piece_id=JL2A3FNDOK5IAHWDIJKVYRQTIUMLSK6THMLBZOXT3VIVTAQ4BJDA, seg_pos={0 0}, upload=12165]: Done uploading piece. err: <nil> Aug 7 21:20:46.498737416 (11h40m46.088082404s) [noise=true , address=104.36.82.4:4814 , piece_id=TG423PLOOSKYD5QG3HHLAYXLCA33WPJZFMQV5N2AXLBFJFQY25BA, seg_pos={0 0}, upload=12165]: Done uploading piece. err: <nil> Aug 7 21:20:46.620015213 (11h40m46.2093602s) [noise=true , address=45.91.116.9:4269 , piece_id=4U5XKZOZ3XVHYH6ZS2B3VOQNJZFFX4MNTDVX3M3J5AIMNNONDQIA, seg_pos={0 0}, upload=12165]: Done uploading piece. err: <nil> Aug 7 21:20:46.661576220 (11h40m46.250921197s) [noise=true , address=104.36.82.46:4501 , piece_id=QV2ULRDKJWZNGPUJ5S66EZ7WGPJWRGNC527P4ZJCTJ3PBDV3KVDA, seg_pos={0 0}, upload=12165]: Done uploading piece. err: <nil> Aug 7 21:20:46.700046554 (11h40m46.289391541s) [noise=true , address=45.91.116.10:4098 , piece_id=R3KJ5V6S7BWSV3LPFFMO24ROS4WMSEIBABGKHPC7O7MRL5DVUIEA, seg_pos={0 0}, upload=12165]: Done uploading piece. err: <nil> Aug 7 21:20:47.062793834 (11h40m46.652138821s) [noise=true , address=104.36.82.34:4559 , piece_id=L4YP4DTEYZZPFF5Q7P67TOOZUCNS2BL4VX2EYZLX7NS6MM6EM6RQ, seg_pos={0 0}, upload=12165]: Done uploading piece. err: <nil> Aug 7 21:20:47.062812619 (11h40m46.652157606s) [seg_pos={0 0}, upload=12165]: Segment reached optimal threshold of 14 pieces. Aug 7 21:20:47.063138753 (11h40m46.65248374s) [noise=true , address=104.36.82.59:4329 , piece_id=VETW6WNXY5OZOXEEZKF7QQ4AEOC36WIJWWBQ7W3J2LQ3VNRGBMKA, seg_pos={0 0}, upload=12165]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1dGY6BG4FULjuRwnEKd9wV4cZPVJsgo32oFHKZn5BgjxVaojbk): protocol: write tcp 188.166.225.170:35928->104.36.82.59:4329: use of closed network connection; write tcp 188.166.225.170:35928->104.36.82.59:4329: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35928->104.36.82.59:4329: use of closed network connection Aug 7 21:20:47.063191945 (11h40m46.652536932s) [noise=true , address=104.36.82.62:4435 , piece_id=VMK7QWCLX5QAKTSAQYQXAM47HQXZROULRGPSLCCS55VFBBF3Q3JA, seg_pos={0 0}, upload=12165]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12o9DvdWBWefvKcvWd62RS9pX7utf7KwbyDaYxfwdXtrMTvcUTh): protocol: rpc: dial tcp 104.36.82.62:4435: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 21:20:47.063317065 (11h40m46.652662052s) [noise=true , address=104.36.82.23:4450 , piece_id=AOSQPJBMRVJW4UVQPSL47YBR7JEYYACVX3FPDGJVLS2ZWV2FFN4A, seg_pos={0 0}, upload=12165]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XE5kZtVU6FeUR5NF2mhwypKiJahTUyt4qtRbo3PUJp6T5sdms): protocol: rpc: dial tcp 104.36.82.23:4450: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 21:20:47.063357091 (11h40m46.652702078s) [noise=true , address=104.36.82.22:4521 , piece_id=FSYBN5FIGVGFYX3PZ5Y6G6NG4OTK7NPZOUUERUYSWV2ZK2U7Q2BA, seg_pos={0 0}, upload=12165]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zWiwpaV58h8YKsAgw24fUWHBBYYyqF4UX6GJV316Uxo9Y7gt1): protocol: read tcp 188.166.225.170:35484->104.36.82.22:4521: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 21:20:47.063373783 (11h40m46.65271877s) [seg_pos={0 0}, upload=12165]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 21:20:47.063434599 (11h40m46.652779586s) [upload=12165]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 21:20:47.063438567 (11h40m46.652783554s) [upload=12165]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:20:47.363318446 (11h40m46.952663433s) [upload=12165]: Done uploading segments... Aug 7 21:20:47.363335478 (11h40m46.952680465s) [upload=12165]: Upload finished. err: <nil> Duration:* 6.252998225s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [............................................XXXXXXXXXXXXXXXX] duration: 30.001366076s FAIL (2025-08-07 21:26:30 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.001383739s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 12 out of 60 is failed (threshold=3) [................................................XXXXXXXXXXXX] duration: 1m0.000398778s FAIL (2025-08-07 21:26:23 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== [upload=12059]: Next segment err: context deadline exceeded Aug 7 21:14:44.664969041 (11h34m44.254314028s) [upload=12059]: Done uploading segments... Aug 7 21:14:44.665004269 (11h34m44.254349256s) [upload=12059]: Upload finished. err: context deadline exceeded Aug 7 21:14:45.557957097 (11h34m45.147302084s) [upload=12072]: Starting upload Aug 7 21:14:45.558041418 (11h34m45.147386395s) [upload=12072]: Done starting upload Aug 7 21:14:45.558153183 (11h34m45.14749817s) [upload=12072]: Uploading segments... Aug 7 21:14:45.558236202 (11h34m45.147581189s) [upload=12072]: Got next segment. Inline: false Aug 7 21:14:45.558263704 (11h34m45.147608691s) [upload=12072]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:14:45.558268123 (11h34m45.1476131s) [upload=12072]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:15:44.909553125 (11h35m44.498898112s) [upload=12072]: Next segment err: context deadline exceeded Aug 7 21:15:44.909693113 (11h35m44.49903809s) [upload=12072]: Done uploading segments... Aug 7 21:15:44.909705847 (11h35m44.499050834s) [upload=12072]: Upload finished. err: context deadline exceeded Aug 7 21:15:45.780163446 (11h35m45.369508433s) [upload=12088]: Starting upload Aug 7 21:15:45.780230324 (11h35m45.369575311s) [upload=12088]: Done starting upload Aug 7 21:15:45.780343460 (11h35m45.369688497s) [upload=12088]: Uploading segments... Aug 7 21:15:45.780437410 (11h35m45.369782397s) [upload=12088]: Got next segment. Inline: false Aug 7 21:15:45.780481765 (11h35m45.369826772s) [upload=12088]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:15:45.780488508 (11h35m45.369833535s) [upload=12088]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:16:45.135036416 (11h36m44.724381403s) [upload=12088]: Next segment err: context deadline exceeded Aug 7 21:16:45.135160053 (11h36m44.72450504s) [upload=12088]: Done uploading segments... Aug 7 21:16:45.135188378 (11h36m44.724533375s) [upload=12088]: Upload finished. err: context deadline exceeded Aug 7 21:16:46.020697803 (11h36m45.61004279s) [upload=12106]: Starting upload Aug 7 21:16:46.020754431 (11h36m45.610099448s) [upload=12106]: Done starting upload Aug 7 21:16:46.020843461 (11h36m45.610188448s) [upload=12106]: Uploading segments... Aug 7 21:16:46.020994882 (11h36m45.610339869s) [upload=12106]: Got next segment. Inline: false Aug 7 21:16:46.021030459 (11h36m45.610375446s) [upload=12106]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:16:46.021035288 (11h36m45.610380276s) [upload=12106]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:17:45.371432070 (11h37m44.960777057s) [upload=12106]: Next segment err: context deadline exceeded Aug 7 21:17:45.371471144 (11h37m44.960816142s) [upload=12106]: Done uploading segments... Aug 7 21:17:45.371492135 (11h37m44.960837122s) [upload=12106]: Upload finished. err: context deadline exceeded Aug 7 21:17:49.811942678 (11h37m49.401287665s) [upload=12122]: Starting upload Aug 7 21:17:49.812064030 (11h37m49.401409007s) [upload=12122]: Done starting upload Aug 7 21:17:49.812137160 (11h37m49.401482147s) [upload=12122]: Uploading segments... Aug 7 21:17:49.812396918 (11h37m49.401741895s) [upload=12122]: Got next segment. Inline: false Aug 7 21:17:49.812426735 (11h37m49.401771722s) [upload=12122]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:17:49.812430862 (11h37m49.401775849s) [upload=12122]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:18:49.169738786 (11h38m48.759083813s) [upload=12122]: Next segment err: context deadline exceeded Aug 7 21:18:49.169939078 (11h38m48.759284086s) [upload=12122]: Done uploading segments... Aug 7 21:18:49.169962594 (11h38m48.759307591s) [upload=12122]: Upload finished. err: context deadline exceeded Aug 7 21:18:50.060561583 (11h38m49.64990661s) [upload=12134]: Starting upload Aug 7 21:18:50.060675751 (11h38m49.650020808s) [upload=12134]: Done starting upload Aug 7 21:18:50.060772135 (11h38m49.650117122s) [upload=12134]: Uploading segments... Aug 7 21:18:50.061042412 (11h38m49.650387409s) [upload=12134]: Got next segment. Inline: false Aug 7 21:18:50.061084393 (11h38m49.65042944s) [upload=12134]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:18:50.061091987 (11h38m49.650437004s) [upload=12134]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:19:49.417978828 (11h39m49.007323815s) [upload=12134]: Next segment err: context deadline exceeded Aug 7 21:19:49.418015799 (11h39m49.007360786s) [upload=12134]: Done uploading segments... Aug 7 21:19:49.418037982 (11h39m49.007382959s) [upload=12134]: Upload finished. err: context deadline exceeded Aug 7 21:20:02.821763948 (11h40m2.411108945s) [upload=12155]: Starting upload Aug 7 21:20:02.821848960 (11h40m2.411193957s) [upload=12155]: Done starting upload Aug 7 21:20:02.821939103 (11h40m2.41128409s) [upload=12155]: Uploading segments... Aug 7 21:20:02.822080344 (11h40m2.411425331s) [upload=12155]: Got next segment. Inline: false Aug 7 21:20:02.822124998 (11h40m2.411469985s) [upload=12155]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:20:02.822132784 (11h40m2.411477771s) [upload=12155]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:21:02.173704394 (11h41m1.763049381s) [upload=12155]: Next segment err: context deadline exceeded Aug 7 21:21:02.173855945 (11h41m1.763200932s) [upload=12155]: Done uploading segments... Aug 7 21:21:02.173881643 (11h41m1.76322663s) [upload=12155]: Upload finished. err: context deadline exceeded Aug 7 21:21:03.041754970 (11h41m2.631099958s) [upload=12171]: Starting upload Aug 7 21:21:03.041871343 (11h41m2.63121633s) [upload=12171]: Done starting upload Aug 7 21:21:03.041935847 (11h41m2.631280834s) [upload=12171]: Uploading segments... Aug 7 21:21:03.042038162 (11h41m2.631383149s) [upload=12171]: Got next segment. Inline: false Aug 7 21:21:03.042070074 (11h41m2.631415061s) [upload=12171]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:21:03.042074442 (11h41m2.631419429s) [upload=12171]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:22:02.399093104 (11h42m1.988438091s) [upload=12171]: Next segment err: context deadline exceeded Aug 7 21:22:02.399277657 (11h42m1.988622644s) [upload=12171]: Done uploading segments... Aug 7 21:22:02.399305961 (11h42m1.988650948s) [upload=12171]: Upload finished. err: context deadline exceeded Aug 7 21:22:03.257202341 (11h42m2.846547328s) [upload=12189]: Starting upload Aug 7 21:22:03.257267245 (11h42m2.846612232s) [upload=12189]: Done starting upload Aug 7 21:22:03.257356857 (11h42m2.846701844s) [upload=12189]: Uploading segments... Aug 7 21:22:03.257444765 (11h42m2.846789742s) [upload=12189]: Got next segment. Inline: false Aug 7 21:22:03.257470033 (11h42m2.84681501s) [upload=12189]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:22:03.257474171 (11h42m2.846819158s) [upload=12189]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:23:02.611478992 (11h43m2.20082398s) [upload=12189]: Next segment err: context deadline exceeded Aug 7 21:23:02.611504952 (11h43m2.200849939s) [upload=12189]: Done uploading segments... Aug 7 21:23:02.611523989 (11h43m2.200868976s) [upload=12189]: Upload finished. err: context deadline exceeded Aug 7 21:23:11.364660428 (11h43m10.954005435s) [upload=12204]: Starting upload Aug 7 21:23:11.364761051 (11h43m10.954106038s) [upload=12204]: Done starting upload Aug 7 21:23:11.364863036 (11h43m10.954208023s) [upload=12204]: Uploading segments... Aug 7 21:23:11.364973286 (11h43m10.954318264s) [upload=12204]: Got next segment. Inline: false Aug 7 21:23:11.365036508 (11h43m10.954381495s) [upload=12204]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:23:11.365041447 (11h43m10.954386434s) [upload=12204]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:24:10.722815877 (11h44m10.312160864s) [upload=12204]: Next segment err: context deadline exceeded Aug 7 21:24:10.722857978 (11h44m10.312202965s) [upload=12204]: Done uploading segments... Aug 7 21:24:10.722871453 (11h44m10.31221644s) [upload=12204]: Upload finished. err: context deadline exceeded Aug 7 21:24:23.410577309 (11h44m22.999922296s) [upload=12220]: Starting upload Aug 7 21:24:23.410633838 (11h44m22.999978825s) [upload=12220]: Done starting upload Aug 7 21:24:23.410696377 (11h44m23.000041364s) [upload=12220]: Uploading segments... Aug 7 21:24:23.410808262 (11h44m23.000153239s) [upload=12220]: Got next segment. Inline: false Aug 7 21:24:23.410833469 (11h44m23.000178456s) [upload=12220]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:24:23.410837017 (11h44m23.000182004s) [upload=12220]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:25:22.762515918 (11h45m22.351860905s) [upload=12220]: Next segment err: context deadline exceeded Aug 7 21:25:22.762663662 (11h45m22.352008639s) [upload=12220]: Done uploading segments... Aug 7 21:25:22.762679401 (11h45m22.352024378s) [upload=12220]: Upload finished. err: context deadline exceeded Aug 7 21:25:23.709743833 (11h45m23.29908882s) [upload=12237]: Starting upload Aug 7 21:25:23.709835399 (11h45m23.299180386s) [upload=12237]: Done starting upload Aug 7 21:25:23.709934688 (11h45m23.299279675s) [upload=12237]: Uploading segments... Aug 7 21:25:23.710036063 (11h45m23.29938105s) [upload=12237]: Got next segment. Inline: false Aug 7 21:25:23.710068875 (11h45m23.299413862s) [upload=12237]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:25:23.710072732 (11h45m23.299417719s) [upload=12237]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:26:23.059097010 (11h46m22.648441997s) [upload=12237]: Next segment err: context deadline exceeded Aug 7 21:26:23.059154440 (11h46m22.648499427s) [upload=12237]: Done uploading segments... Aug 7 21:26:23.059167756 (11h46m22.648512743s) [upload=12237]: Upload finished. err: context deadline exceeded Duration: 1m0.000407715s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 13 out of 60 is failed (threshold=3) [..............................................X.XXXXXXXXXXXX] duration: 1m0.001345112s FAIL (2025-08-07 21:26:17 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 95846110 (11h35m45.585191127s) [upload=12089]: Uploading segments... Aug 7 21:15:45.995939870 (11h35m45.585284917s) [upload=12089]: Got next segment. Inline: true Aug 7 21:15:45.995949598 (11h35m45.585294615s) [upload=12089]: Waiting for error group managing segments... Aug 7 21:15:45.996009222 (11h35m45.585354249s) [upload=12089]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:15:45.996016616 (11h35m45.585361623s) [upload=12089]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:15:45.996022197 (11h35m45.585367204s) [upload=12089]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:16:45.347388198 (11h36m44.936733185s) [upload=12089]: Done uploading segments... Aug 7 21:16:45.347412374 (11h36m44.936757362s) [upload=12089]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:16:46.218888633 (11h36m45.80823362s) [upload=12107]: Starting upload Aug 7 21:16:46.218949108 (11h36m45.808294095s) [upload=12107]: Done starting upload Aug 7 21:16:46.219029552 (11h36m45.808374539s) [upload=12107]: Uploading segments... Aug 7 21:16:46.219157317 (11h36m45.808502334s) [upload=12107]: Got next segment. Inline: true Aug 7 21:16:46.219163238 (11h36m45.808508225s) [upload=12107]: Waiting for error group managing segments... Aug 7 21:16:46.219206872 (11h36m45.808551859s) [upload=12107]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:16:46.219212492 (11h36m45.80855747s) [upload=12107]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:16:46.219215178 (11h36m45.808560155s) [upload=12107]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:17:45.569887093 (11h37m45.15923208s) [upload=12107]: Done uploading segments... Aug 7 21:17:45.569902753 (11h37m45.15924774s) [upload=12107]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:17:46.442498934 (11h37m46.031843911s) [upload=12121]: Starting upload Aug 7 21:17:46.442561293 (11h37m46.03190628s) [upload=12121]: Done starting upload Aug 7 21:17:46.442612481 (11h37m46.031957468s) [upload=12121]: Uploading segments... Aug 7 21:17:46.442661796 (11h37m46.032006783s) [upload=12121]: Got next segment. Inline: true Aug 7 21:17:46.442665573 (11h37m46.03201056s) [upload=12121]: Waiting for error group managing segments... Aug 7 21:17:46.442697183 (11h37m46.03204217s) [upload=12121]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:17:46.442700580 (11h37m46.032045557s) [upload=12121]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:17:46.442704637 (11h37m46.032049624s) [upload=12121]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:18:45.798465603 (11h38m45.38781059s) [upload=12121]: Done uploading segments... Aug 7 21:18:45.798489779 (11h38m45.387834766s) [upload=12121]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:18:57.960074990 (11h38m57.549419977s) [upload=12139]: Starting upload Aug 7 21:18:57.960165033 (11h38m57.54951003s) [upload=12139]: Done starting upload Aug 7 21:18:57.960244845 (11h38m57.549589842s) [upload=12139]: Uploading segments... Aug 7 21:18:57.960294299 (11h38m57.549639286s) [upload=12139]: Got next segment. Inline: true Aug 7 21:18:57.960298928 (11h38m57.549643915s) [upload=12139]: Waiting for error group managing segments... Aug 7 21:18:57.960349855 (11h38m57.549694853s) [upload=12139]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:18:57.960354916 (11h38m57.549699903s) [upload=12139]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:18:57.960358112 (11h38m57.549703109s) [upload=12139]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:19:57.313173678 (11h39m56.902518665s) [upload=12139]: Done uploading segments... Aug 7 21:19:57.313193636 (11h39m56.902538623s) [upload=12139]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:20:04.133582941 (11h40m3.722927928s) [upload=12156]: Starting upload Aug 7 21:20:04.133643998 (11h40m3.722988985s) [upload=12156]: Done starting upload Aug 7 21:20:04.133712729 (11h40m3.723057726s) [upload=12156]: Uploading segments... Aug 7 21:20:04.133764127 (11h40m3.723109114s) [upload=12156]: Got next segment. Inline: true Aug 7 21:20:04.133769046 (11h40m3.723114033s) [upload=12156]: Waiting for error group managing segments... Aug 7 21:20:04.133838079 (11h40m3.723183066s) [upload=12156]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:20:04.133842868 (11h40m3.723187855s) [upload=12156]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:20:04.133845723 (11h40m3.72319071s) [upload=12156]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:21:03.481385970 (11h41m3.070730947s) [upload=12156]: Done uploading segments... Aug 7 21:21:03.481407009 (11h41m3.070751986s) [upload=12156]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:21:04.366237551 (11h41m3.955582538s) [upload=12172]: Starting upload Aug 7 21:21:04.366309369 (11h41m3.955654356s) [upload=12172]: Done starting upload Aug 7 21:21:04.366360317 (11h41m3.955705304s) [upload=12172]: Uploading segments... Aug 7 21:21:04.366406304 (11h41m3.955751291s) [upload=12172]: Got next segment. Inline: true Aug 7 21:21:04.366410022 (11h41m3.955755009s) [upload=12172]: Waiting for error group managing segments... Aug 7 21:21:04.366441823 (11h41m3.95578681s) [upload=12172]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:21:04.366445840 (11h41m3.955790827s) [upload=12172]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:21:04.366448265 (11h41m3.955793252s) [upload=12172]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:22:03.717958127 (11h42m3.307303114s) [upload=12172]: Done uploading segments... Aug 7 21:22:03.717976892 (11h42m3.307321879s) [upload=12172]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:22:11.311019027 (11h42m10.900364014s) [upload=12193]: Starting upload Aug 7 21:22:11.311074243 (11h42m10.90041923s) [upload=12193]: Done starting upload Aug 7 21:22:11.311123727 (11h42m10.900468714s) [upload=12193]: Uploading segments... Aug 7 21:22:11.311167090 (11h42m10.900512077s) [upload=12193]: Got next segment. Inline: true Aug 7 21:22:11.311183532 (11h42m10.900528519s) [upload=12193]: Waiting for error group managing segments... Aug 7 21:22:11.311220152 (11h42m10.900565139s) [upload=12193]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:22:11.311224931 (11h42m10.900569918s) [upload=12193]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:22:11.311227586 (11h42m10.900572573s) [upload=12193]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:23:10.658643233 (11h43m10.24798821s) [upload=12193]: Done uploading segments... Aug 7 21:23:10.658662571 (11h43m10.248007558s) [upload=12193]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:23:11.571475028 (11h43m11.160820015s) [upload=12205]: Starting upload Aug 7 21:23:11.571532048 (11h43m11.160877035s) [upload=12205]: Done starting upload Aug 7 21:23:11.571582173 (11h43m11.16092716s) [upload=12205]: Uploading segments... Aug 7 21:23:11.571617942 (11h43m11.160962929s) [upload=12205]: Got next segment. Inline: true Aug 7 21:23:11.571621489 (11h43m11.160966486s) [upload=12205]: Waiting for error group managing segments... Aug 7 21:23:11.571655484 (11h43m11.161000471s) [upload=12205]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:23:11.571659531 (11h43m11.161004528s) [upload=12205]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:23:11.571662447 (11h43m11.161007444s) [upload=12205]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:24:10.926729728 (11h44m10.516074715s) [upload=12205]: Done uploading segments... Aug 7 21:24:10.926747603 (11h44m10.51609259s) [upload=12205]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:24:11.802089650 (11h44m11.391434637s) [upload=12218]: Starting upload Aug 7 21:24:11.802151558 (11h44m11.391496545s) [upload=12218]: Done starting upload Aug 7 21:24:11.802205020 (11h44m11.391549997s) [upload=12218]: Uploading segments... Aug 7 21:24:11.802240418 (11h44m11.391585406s) [upload=12218]: Got next segment. Inline: true Aug 7 21:24:11.802244236 (11h44m11.391589223s) [upload=12218]: Waiting for error group managing segments... Aug 7 21:24:11.802278040 (11h44m11.391623027s) [upload=12218]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:24:11.802282458 (11h44m11.391627445s) [upload=12218]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:24:11.802284602 (11h44m11.391629579s) [upload=12218]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:25:11.151652598 (11h45m10.740997575s) [upload=12218]: Done uploading segments... Aug 7 21:25:11.151674050 (11h45m10.741019037s) [upload=12218]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:25:17.859162022 (11h45m17.44850703s) [upload=12235]: Starting upload Aug 7 21:25:17.859247908 (11h45m17.448592915s) [upload=12235]: Done starting upload Aug 7 21:25:17.859314314 (11h45m17.448659301s) [upload=12235]: Uploading segments... Aug 7 21:25:17.859360473 (11h45m17.44870546s) [upload=12235]: Got next segment. Inline: true Aug 7 21:25:17.859365562 (11h45m17.44871056s) [upload=12235]: Waiting for error group managing segments... Aug 7 21:25:17.859407051 (11h45m17.448752058s) [upload=12235]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:25:17.859412402 (11h45m17.448757399s) [upload=12235]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:25:17.859415648 (11h45m17.448760646s) [upload=12235]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:26:17.209779892 (11h46m16.799124879s) [upload=12235]: Done uploading segments... Aug 7 21:26:17.209836961 (11h46m16.799181949s) [upload=12235]: Upload finished. err: metaclient: context deadline exceeded Duration: 1m0.001370642s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway)

checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)